Monday, October 13, 2014

Why Open-Source Software works

Many people, specially outside the technology industry are surprised that something like open-source software could exist, specially at the scale and sophistication it quite obviously does. Many vendors of proprietary - i.e. closed-source software, have long tried to discredit open-source as either suspicious and subversive "Commie-stuff" or as hobbyist toys and playthings, not fit for serious applications.

Looking at the development of human knowledge and culture throughout history, it should not be surprising that open-source software would exist and be thriving in the long run. On the contrary, closed-source software, like any idea or innovation that is kept secret essentially dies with its inventor, cannot influence new ideas and ongoing innovation and essentially becomes an evolutionary dead-end. In the short-lived word of technology, this can happen within decades rather than centuries, as historically no tech company has managed to stay dominant for more than a few decades at most - even the once mighty IBM, DEC or AT&T.

Unix and C, probably the most influential operating system and programming language respectively, were developed around 1970 at the AT&T Bell Laboratories. Since regulations more or less prevented the monopoly phone company to enter the computer business, it more or less openly shared it innovations in that area with the academic research community and thus they became part of the record of human knowledge and an influential source of innovation.

Some people argue that for example VMS, a competing operating system from Digital Equipment Corporation, was technically superior, but since the once mighty DEC went defunct in 1998 VMS is today pretty much irrelevant and ancient history.

Today, AT&T Bell Labs itself is not doing much better. Split into N parts by antitrust regulators and a brutally competitive market have turned it into a faint shadow of its former self.

Abandoned former Bell Labs research site in Holmdel, NJ - where I started my career (source Metropolis/Rob Dobi
We can only imagine how much poorer the state of the art of technology would be, if AT&T had been allowed to fully commercialize many of its inventions.

In the early days of computing, most software was open-source by necessity - for distribution, portability and because it was important for users to help each other in a yet immature and emerging field. Software was not yet a business and mostly written by users and/or researchers.  The research community considered access to source code as simply an essential and natural extension of academic freedom and the scientific method to the new field of software - i.e. the right to examine and investigate.

The emergence of software as a business clashed with this academic culture of openness, which lead to a series of high profile legal and cultural conflicts and led to the creation of a more formal and organized movement around Free and Open-Source Software - now with capital letters, organizations, foundations and its own legal frameworks and licenses (e.g. BSD license or GPL).

The impression that open-source is purely a playground for academia or a fringe of idealists and hobbyists is however largely wrong - at least for the large and significant open-source projects today. For example the most recent report on Linux kernel development from the Linux Foundation in 2013 shows that at least 80% of the contributors are payed for their work on the Linux kernel by their respective employers.

Specially in the areas of critical technical infrastructure like operating systems, programming languages, databases, web servers and other Internet infrastructure, open-source allows for a strange agreement of cooper-tition. While many companies critically rely on such infrastructure, very few could develop it all in house and very few would want to depend on a potential competitor for such critical parts of their infrastructure. So they rather allow some of their employees to work on what looks from the outside like a giant non-profit volunteer effort.

We are now again in a situation where an increasing number of software is written by technically sophisticated organizations which are not in the business of selling software, but rather products or services which is strongly dependent on software. I don't know if open-source software has in part undermined the prospects of software as a business or whether this shift in business focus has helped open-source software to become as relevant and significant as it is today.