Saturday, June 27, 2009

E-mail notifications for SVN & Trac (SDI 07 part VI)

Much of the email I get is actually not from users directly, but generated by some automated system in response to some event. We are all used to getting these kind of emails - if we sign up on a new Internet sit, buy tickets online or somebody throws a sheep at us on Facebook.

Since email is nearly ubiquitous and has many powerful tools for sorting, filtering, archiving or processing messages, it is a natural channel for delivering such automated notifications.

Since we set up an email distribution system in the last 2 episodes of our series on startup software development infrastructure, we can make use of it as well to distribute event notifications from the key systems in our software development workflow, which are the issue tracking and source code version control systems - Trac tickets and svn respectively in our case.

To enable email notification for every change to a trac ticket which concerns us, we only need to edit the notification section in the Trac config file at /data/sdi07/trac/conf/trac.ini based on the build-in documentation on the Trac wiki under TracNotification - basically something like:
[notification]
smtp_enabled = true
smtp_server = localhost
smtp_from = trac@sdi.kugelfish.com
smtp_default_domain = sdi.kugelfish.com
This assumes that there is an email account for each active trac/svn user in the system, but more about that in a later posting.

Since the evolution of the codebase is of great concern to anybody on the team, we would also like to send out a notification message every time a new change is submitted, since any change to the latest state in the repository could potentially affect what anybody is working on.

Subversion has a generic mechanism to trigger customized user actions at particular stages of its operations. If they exist, particular hook scripts located in the server repository hierarchy are executed - e.g. the post-commit hook each time after a new version has been committed.

Since commit notification emails is such a common feature, we can choose from a variety of existing solutions - the simplest one being commit-email.pl from the svn contrib collection of hook scripts. In order to activate the hook, copy the script to /data/sdi07/svn/hooks/commit-email.pl and create a new file /data/sdi07/svn/hooks/post-commit with the following content:
#!/bin/sh
REPOS="$1"
REV="$2"
/data/sdi07/svn/hooks/commit-email.pl ${REPOS} ${REV} --diff n -h localhost -s "[svn commit] " swdev
Both files should again be owned by the apache uid/gid and have execute permission. This very basic notification script will send out an e-mail with a diff for all changes submitted in a new repository revision, to the swdev mailing list which we previously created. More sophisticated scripts are availalbe - .e.g svnnotify which allows many options to customize formatting and distribution of checkin notification emails.

Thursday, June 18, 2009

Archived Mailing Lists (SDI 07 Part V)

In the previous episode of our series on startup software development infrastructure, we have setup the basic email deliver system. The next step is to provide a solution for archived mailing lists.

Mailing lists have long been a backbone of the Internet community and there are mature solutions for managing very large-scale mailing lists. The classics are Listserv and Majordomo where users can manage their own subscriptions by sending emails with embedded commands and more recently Mailman has become very popular because of its web interface for managing mailing list settings for both users and administrators. Mailman even has a built-in web-based email archive.

Since we are targeting our solution for very small teams, we can get away with an even simpler approach. We expect our teams to only have a handful of members and where people joining and leaving are very rare events. In this case, we can simply use the mail alias functionality built into the email server. To maintain the web-based archive, we use MHonArc an email to html converter as a member on each of the lists.

Assuming our main software team maling list will simply be called swdev, we create the following three alias targets in a file /data/sdi07/lists/list-aliases:
swdev-archive: "|/usr/bin/mhonarc -add -outdir /data/sdi07/lists/swdev/ -title ’swdev mailing list archive’"
swdev : swdev-noarc, swdev-archive
swdev-noarc : user1, user2, user3
The first target will archive a copy of each email sent to it using the mhonarch archiver to a particular directory in our project subtree. The second one is the actual archived mailing list which consists of the archiver and the non-archived version of the same list, which contains all the human users.

Creating both the regular and noarc version of each list is not required but it can be convenient for sending social and administrative email, which may not need to be archived.
mkdir /data/sdi07/lists/swdev
chown -R apache:apache /data/sdi07/
postalias /data/sdi07/list-aliases
Prepares the new mailing list to be activated. Postfix will run the delivery command with the user/group ID of the alias file they are defined in. Generating the archive as the apache user, ensures that it can be displayed later on by the webserver which runs under this user ID.

To start delivery, we need to add our new alias file to the alias configuration in /etc/postfix/main.cfg:
alias_map = hash:/etc/aliases, hash:/var/proj/milinglists/list-aliases


And in order to make the mailing list archive visible on the website, we need to add the following section to the apache configuration file /etc/apache2/httpd.conf:
Alias /sdi07/lists/swdev "/data/sdi07/lists/swdev"
<Directory "/data/sdi07/lists/swdev">
AllowOverride None
Options -Indexes
</Directory>
After we send the first message to the ”swdev” mailing list, we can see the archive presented as /sdi07/lists/swdev/maillist.html on the local web server.

Tuesday, June 16, 2009

Local Email System (SDI 07 Part IV)

After setting up Subversion and Trac for managing source code, the next episode in our series on startup software development infrastructure is about email. Email or some other form of archived group communication is essential for a team to remain in sync on any of the important details of the project.

Using email for remote collaboration is about as old as the Internet itself and benefits from well established habits and usage pattern. Most open source projects are using mailing lists as their main or sole channel of communication, which means that most open-source software development tools are well integrated with an email centric work flow.

Systematically conducting all important technical discussions on archived mailing lists can bridge gaps in both space and time. Email can reach team members who are not here right now - traveling or in a remote location as well as future team members who can read up on old discussion threads to figure out why things were done a certain way.

These days when public Internet mail is dominated by spam, viruses and other malware, running an email service has become a major challenge and is best left to professional system administrators. For now, we can punt on the various security issues by running a private "close circuit" mail service only within and for the software development project. Most users today are using email clients which can connect to remote mail services using mailbox access protocols like POP or IMAP and can manage multiple disjoint mailboxes and email service accounts quite easily.

As the main engine or Mail Transfer Agent (MTA) of our private email system, we are using Postfix. Postfix is a modern replacement for sendmail - the grand-daddy of MTAs and is meant to be faster, more secure and more easy to administer than sendmail. It is clearly overkill for what we need, but since simple configurations are simple, we can as well use it and lay the foundtion for growing into a full-fledged mail service later on if needed.

In order to configure the close-circuit mail delivery system, we need to make only a few changes in the default postfix configuration file /etc/postfix/main.cfg:

# Configure local domain info
myhostname = sdi.kugelfish.com
mydomain = kugelfish.com

# Use domain instead of host name as origin
myorigin=$mydomain

# Accept email for those destinations
mydestination=$myhostname, localhost.$mydomain, locahost, $mydomain

# Reject email to all other destinations with an error
relay_transport = error:External delivery disabled
default_transport = error:External delivery disabled
After this configuration changes, we can active the mail delivery service as follows:
/etc/init.d/postfix start
rc-update add postfix default
In order to let users access their email, we need to run the necessary mailbox remote access protocols for which we use Dovecot, which supports both POP and IMAP in one package. All we need to change from the defaults in /etc/dovecot/dovecot.conf is to enable all possible protocol options - POP and IMAP in both plain and SSL versions:
protocols = imap imaps pop3 pop3s
before we can start the service as follows:
/etc/init.d/dovecot start
rc-update add dovecot default

With this configuration we have a basic local mail service for all Unix user accounts which are configured on the development infrastructure server. This is somewhat inconsistent with the services accessed through the http front-end (e.g. svn and trac) which use the htaccess file for user authentication. We will discuss at a later stage how to unify the user account information for all these services.

Saturday, June 6, 2009

Code Review App as a Service

I just noticed that Rietveld, one of the open-source code-review applications dicussed in an earlier post is now availalbe as a hosted virtual private application to any organization which uses the Google Apps service.

Since I use Google Apps for the @kugelfish.com domain, I tried out the new service by adding it to my domain from the link at this page. As for all virtualized apps, the administrator needs to create a new CNAME in the domains DNS records under which the new service will be mapped - e.g. reviews.kugelfish.com.

While there are many good arguments for an organization to use a hosted solution like Google Apps for providing their public email service, I am a bit more skeptical about outsourcing critical development infrastructure. Specially as in this case the SLA explicitly says that there is no SLA, since these are experimental labs applications - a step down even from the usual Google Beta label. But for organizations where code reviews are not yet a fully integrated and critical part of their development workflow, this is an intriguingly quick way to get started and hopefully become hooked on the practice of code reviews.