Tuesday, February 24, 2009

The Thing we do - or why this Section is called Software Development

I do computer stuff for a living, my current job title is software engineer. I could also be called a programmer. But my favorite term to describe what we do is Software Development. It emphasizes on the evolutionary and incremental nature of the process. Because not unlike a Hollywood screen-play, software is typically not written in one shot, but developed over multiple iterations by various authors, based on feed-back from various sources.

Programming is a skill, a craft or - if done exceptionally well - maybe an art. It's the stuff that is thought in schools, that textbooks are about - on algorithms and data structures, programming languages etc. (Some Computer Science purists may argue that CS isn't really about programming, but that is a whole different debate.)

In the beginning, there used to be just programming. Computer were new, exciting, largely irrelevant in the grand scheme of things and there was no formal training yet for how to write programs for them. People from various backgrounds - scientists and engineers typically did whatever seemed to work and became the pioneers of a new field.

Somewhere around 1970 computers started to be considered important enough that the ad-hoc approaches seemed to look a bit inappropriate - specially since the results were not always very consistent. After a couple of high-profile flops with massive budget and schedule overruns, people started to talk about a software crisis.

The solution seemed to be to borrow some of the rigorous tools and methodology of engineering, which after all has been able to handle some massive and complex projects successfully. Ever since the days of building the pyramids or the great cathedrals of the middle ages all the way to the industrial age. Born was Software Engineering.

Even though the increased rigor and focus on process and methodology did certainly help some for the field to mature and be able to handle larger and larger software projects successfully - at least sometimes...

Yet still, software didn't seem work exactly like other things, engineers are used to work with. For one, it lacks the inherent physicality of what traditional engineering disciplines are dealing with. It can't be over-engineered and it lacks safety margins. On the flip-side, software doesn't age or break like all physical things.

Software is pure, disembodied complexity. Pure description - with a need to be nearly close to perfect. One might say that also about music or literature. But with an important difference, that software is consumed by computers. They don't have a sense of humor, no common sense at all and will execute by the letter, exactly what is spelled out in their program. Humans have an amazing ability to gloss over errors, get the jist of it, do what is meant not what is said. Computers can't do that - at least not yet. (For much better explanation on why software is hard, see this classic paper from 1987).

As software has permeated every imaginable aspect of life - from steering planes to online dating - some of the things, software is supposed to do for us has become a lot more touchy-feely than the things engineering methodology is particularly good for. While a a rigorous engineering approach is probably appropriate for something as critical and well specified as the fly-by-wire aircraft control system for a new passenger jet - but maybe not as much for building the latest social networking website. What would be the formal Software Requirements Specification for the next big social networking application anyway?

In recent years, old-school programming has had somewhat of a comeback. Capital Letter Software Engineering Processes and Methodologies with their intimidating IEEE, DoD or ISO standard numbers, are not the latest fashion any more - in favor or more light-weight and nimbler approaches (I am desperately tring to avoid the word agile here... ;-) )

Maybe a kind of dialectic synthesis between the "anything goes" approaches of the early days and the alphabet soup of government regulated software engineering processes, the new styles of software development openly acknoledge what most experienced practicioners have instinctively known all along:

That requirements do change and we often don't know them in advance. And even if we knew them, we still would often get things wrong the first few times. And sometimes only when we see the finished result in front of us, do we know that it may not be working or we simply don't like it.

But also to acknowledge that building something complex beyond our means,  requiring the collaboration of many and taking a long time to complete, does require a well organized, systematic and disciplined approach.

That's what I call software development.