Monday, March 2, 2009

The other Benefit of Open-Source

Software development must be one of the fields where the gap between best practices and average practices is the widest. A poll in 2001 showed that only about two thirds of software development teams are using version control and only about one third use some kind of bug tracking system. C'mon people how many high-rise window cleaning crews are working without a safety harness?

Open-source projects with many collaborators distributed throughout the world generally need to adopt solid collaborative development practices and often build themselves the tools to support collaboration at such a large scale.

With the popularity of open-source software, an increasing number of people in the technical community have been exposed to the ways these projects operate and to the tools they use. Today, it is a lot harder to find fresh college grads who would not find it completely naturally to use version control, after all this is how you get the pre-release version of "insert-your-favorite-open-source-project-here". At the same time, they are naturally familiar with the idea of a release and that large, complex software systems don't just come together by themselves.

When I was in college, I don't think the word version control was ever mentioned. We learned to program in obscure and irrelevant languages - which is not necessarily a bad thing, since this helps to build a meta-level understanding of programming languages. I guess it was just assumed that those of us who would choose a career in software development would learn their trade on the job, once we got out into the industry. On the other hand, since not all industrial software projects are necessarily that well run, bad habits are propagated as much and as quickly as good ones. Since successful teams tend to stick together a lot longer than the ones which fail, maybe the bad habits spread even faster.

My first exposure to industrial software development was at a very reputable technology company. The kind of company, where you would expect using the most effective software development practices would be a given. It turns out it wasn't and each project had to figure it out for themselves - not uncommon in large companies. Our project ended up to be a classic death march: overly ambitious schedule, a team of fresh bodies assembled to quickly (hundreds of people towards the end), no particular method to the madness, builds started to take hours, dinners and week-ends at work became routine. Many of us were young and we made up in energy and enthousism what we lacked in experience - the few experienced software developers had either left in disgust, as nobody listened to them or kept a low profile, knowing well enough they couldn't influence much the inevitable course of events.. Yes, we had somehow heard that using version control (the company had even invented a few of them) was aparently useful and yes, documentation too - but we didn't have much of clue on how to put it all together.

When it had all come to and end, some of us refused to accept that this should really be the best way possible to do software development. If our current environment not teach us how to do it better, we had to look elsewhere for inspiration. We couldn't see how other companies were doing things better, but there certainly were a few open-source projects who seemed to be building software systems of comparable scale and complexity a lot more smoothly.

Open-source projects provide a unique insight into some very large and long running software development efforts - some of them like the Linux kernel development have gone on for decades and have produced millions of lines of working code. Most commercial software development projects are a lot smaller than that, but by adoptiong tools and practices which work for mega-projects like that, we can be reasonably certain that they will not run out of steam during the lifetime of our project. Furthermore, open-source can become a shared frame of reference on practical software development issues for professionals accross different organizations - hopefully helping to raise the standards of how software development is practiced throughout the industry.