Michael Yuan

Dr Dobb’s Journal’s Jolt award is one of the longest running and most prestigious awards in the software industry. The 18th Jolt Award is now open for nominations:

http://www.joltnominations.com/Jolt/

Hint: If you develop tools for mobile applications, Jolt does have a category for mobile tools. I will participate in this category heavily this year.

To be qualified for Jolts, you have to submit your own product — there is a small submission fee ($50 for open source projects) to cover admin costs. As a Jolt judge for the past several years, I can tell you that none of the judges is paid. We are all volunteers contributing our time for free to go through the complicated software evaluation process. What do we get out of it? Well, we get exposure to the latest happenings in the industry and get to put “Jolts Judge” on our name cards.

What do you get out of it? Well, if your product wins Jolts, that gives you some major bragging right! Even if your product does not win, by participating in the process, you get Jolt judges (most of them are consultants influencing purchasing decisions) to look carefully at your product. If some of the judges like your product, they might blog about it or recommend it in their consulting work.

In the past several years, we got an overwhelming number of submissions every year. It is nearly impossible to evaluate each of them in detail, if the judges are not already using the product in his/her daily work. So, it will be enormously helpful if you can prepare a flash or live demo of your product to illustrate the key features that differentiates your product from the rest.

Fedora ships with SELinux (Security Enhanced Linux) enabled by default. Well, more security is a VERY GOOD THING if people know how to use it. But for a casual Linux user/admin, SELinux imposes a new set of unfamiliar runtime constraints beyond the standard *nix users/groups permissions. I learned this the hard way this weekend.

I recently installed Bugzilla on a fresh Fedora 7 box for my development team (well, I will write about my impression of Bugzilla versus JIRA in another post later …). After following the instructions closely to setup required Perl modules, MySQL database, Apache configuration, and SSH tunneling, I was able to hit Bugzilla web pages. However, when I tried to create an issue, Bugzilla complains that the notification email could not be sent. With the default “sendmail” option, Bugzilla complains:

/etc/mail/sendmail.cf: line 0: Permission denied

Error closing pipe to /usr/lib/sendmail

Hmm, that is weird since the /etc/mail/sendmail.cf file is globally readable. I can sudo into user apache and can read it just fine. Actually, I can invoke sendmail from the command line as user apache and sent out email … But nevermind, I will just try SMTP for email sending. But then Bugzilla complains:

Can’t connect to localhost

What? I can run telnet localhost 25 on the server, and send out email by manually interacting with the local SMTP server. Why would Bugzilla complain?

Without knowing anything about SELinux, I can only assume that the issue is with the Perl Email::Send module, which Bugzilla uses. So, I dived into my old Perl books and wrote several test scripts to see whether Email::Send works. They all worked fine from the command line console but all failed with the above two error messages when I tried them as CGI scripts.

Frustrated, I started to google for everything related to Bugzilla, Email::Send, Bugzilla, and Fedora 7. I spent hours reading, learned a lot about Perl, but still no luck. Then, I stumbled across this Red Hat Magazine article explaining how SELinux works. It says:

Its aim is to provide additional security to some of the more commonly used daemons: httpd, dhcpd, mailman, named, portmap, nscd, ntpd, portmap, mysqld, postgres, squid, syslogd, winbind, and ypbind by employing Mandatory Access Control (MAC) rules. These daemons run in their own domain such as httpd_t for httpd …

Duh! My SELinux configuration must have constrained the HTTPD from invoking sendmail or making outbound network connections! It is so obvious! I have no idea whether this strict security policy comes from Fedora 7 by default or is the result of checkbox I unknowingly chose at installation.

I disabled SELinux enforcement on the box, and Bugzilla works wonderfully now! I will leave it as disabled for now since my server runs inside a firewall and is only accessed via SSH tunneling. However, if we choose to setup a public server later, we will need to fix HTTPD security permissions and re-enable SELinux for sure.

In 20/20 hindsight, it is obvious that I should be able to spot the problem quickly if I searched for “SELinux” from start. However, due to my own ignorance of SELinux, I did not even know where to start. I believe this is one of the cases where I could really benefited from professional Linux support — a trained Linux admin would easily spot the problem and solve it for me in minutes. Lessons learned.