Tuesday, February 24, 2009

Startups in the Age of Web 2.0

The current generation of web startups reminds me a lot of rock bands in the 60ies.

Before the transistor radio and the record player, there were many bands who toured the local dance-hall circuit, since life music was the only kind of entertainment possible. It was rather hard to get very rich as a musician. Records and radio changed the game - it became "winner takes it all". Suddenly the whole country could be swooning to the songs of a few long-haired boys from Liverpool for example - who became unimaginably rich and famous, where many unnamed musicians lost their income. Many of them played a mean guitar too - why was it the Beatles who became famous? But no matter who made it - the record companies always made money...

Today web technology is pretty well understood. Many people could build a pretty state of the art web application and to get started it takes about as many people as for starting a garage band. The web as a delivery platform has made it very cheap and easy to innovate with new service ideas, but not all of them will become the next Flickr, YouTube, Facebook or Twitter. What made those successful anyway among all the similar companies chasing the same ideas?

It's not their technology because unlike the first-gen web startups of the dot.com bubble who had to invent how to build a large-scale web-based service, this is pretty much a solved problem today. Something must have clicked with users (pun intended...) for those who took off big-time as opposed to all the very similar ones that didn't. Maybe some of the same reasons which made the Beatles to become superstars, rather than any of the hundreds of similar bands who remained in complete obscurity. And as in record companies' glory days, on the web, the owner of the large ad networks make money, no matter who becomes the next big thing.

The fate of a technology startup used to be at least somewhat decided by how well they solve some new technical challenges. As the barrier to entry and differentiation based on technical execution has become lower and lower, todays web service startups are not really determined as much by technology any more but by understanding the Zeitgeist and spotting the next trend of cool. At least in this particular field, the Triumph of the Nerds may have been short-lived.

The Thing we do - or why this Section is called Software Development

I do computer stuff for a living, my current job title is software engineer. I could also be called a programmer. But my favorite term to describe what we do is Software Development. It emphasizes on the evolutionary and incremental nature of the process. Because not unlike a Hollywood screen-play, software is typically not written in one shot, but developed over multiple iterations by various authors, based on feed-back from various sources.

Programming is a skill, a craft or - if done exceptionally well - maybe an art. It's the stuff that is thought in schools, that textbooks are about - on algorithms and data structures, programming languages etc. (Some Computer Science purists may argue that CS isn't really about programming, but that is a whole different debate.)

In the beginning, there used to be just programming. Computer were new, exciting, largely irrelevant in the grand scheme of things and there was no formal training yet for how to write programs for them. People from various backgrounds - scientists and engineers typically did whatever seemed to work and became the pioneers of a new field.

Somewhere around 1970 computers started to be considered important enough that the ad-hoc approaches seemed to look a bit inappropriate - specially since the results were not always very consistent. After a couple of high-profile flops with massive budget and schedule overruns, people started to talk about a software crisis.

The solution seemed to be to borrow some of the rigorous tools and methodology of engineering, which after all has been able to handle some massive and complex projects successfully. Ever since the days of building the pyramids or the great cathedrals of the middle ages all the way to the industrial age. Born was Software Engineering.

Even though the increased rigor and focus on process and methodology did certainly help some for the field to mature and be able to handle larger and larger software projects successfully - at least sometimes...

Yet still, software didn't seem work exactly like other things, engineers are used to work with. For one, it lacks the inherent physicality of what traditional engineering disciplines are dealing with. It can't be over-engineered and it lacks safety margins. On the flip-side, software doesn't age or break like all physical things.

Software is pure, disembodied complexity. Pure description - with a need to be nearly close to perfect. One might say that also about music or literature. But with an important difference, that software is consumed by computers. They don't have a sense of humor, no common sense at all and will execute by the letter, exactly what is spelled out in their program. Humans have an amazing ability to gloss over errors, get the jist of it, do what is meant not what is said. Computers can't do that - at least not yet. (For much better explanation on why software is hard, see this classic paper from 1987).

As software has permeated every imaginable aspect of life - from steering planes to online dating - some of the things, software is supposed to do for us has become a lot more touchy-feely than the things engineering methodology is particularly good for. While a a rigorous engineering approach is probably appropriate for something as critical and well specified as the fly-by-wire aircraft control system for a new passenger jet - but maybe not as much for building the latest social networking website. What would be the formal Software Requirements Specification for the next big social networking application anyway?

In recent years, old-school programming has had somewhat of a comeback. Capital Letter Software Engineering Processes and Methodologies with their intimidating IEEE, DoD or ISO standard numbers, are not the latest fashion any more - in favor or more light-weight and nimbler approaches (I am desperately tring to avoid the word agile here... ;-) )

Maybe a kind of dialectic synthesis between the "anything goes" approaches of the early days and the alphabet soup of government regulated software engineering processes, the new styles of software development openly acknoledge what most experienced practicioners have instinctively known all along:

That requirements do change and we often don't know them in advance. And even if we knew them, we still would often get things wrong the first few times. And sometimes only when we see the finished result in front of us, do we know that it may not be working or we simply don't like it.

But also to acknowledge that building something complex beyond our means,  requiring the collaboration of many and taking a long time to complete, does require a well organized, systematic and disciplined approach.

That's what I call software development.

Monday, February 23, 2009

Android Apps Reviews

After some ego-surfing, here are a few reviews on my Android apps, at least in the languages, I can understand...

BistroMath
NetMeter

NoiseAlert

AppEngine

I have been playing a bit with AppEngine lately. I is a cloud computing service geared towards running Python based web applications on Google's data-center infrastructure. Applications can only run short-lived http request handlers, which will be terminated if they exceed their CPU quota. On the other hand, many parallel instance of the same application can be spawned transparently if the load requires it.

AppEngine comes with its own proprietary non-relational data-store, which maps well to Django's object persistence model for example. And since Django is such a popular framework for web development in Python, it comes pre-packaged in the AppEngine API.

I think this is a very compelling solution for many new web applications since it takes a lot of pain out of building and scaling infrastructure. For those who don't believe that this is a significant problem, here is a very interesting presentation from the MySQL 04 conference about how LifeJournal scaled up from a hobby project on a single machine in a dorm room to high traffic website running on a whole cluster of machines.

One of the biggest strengths of web-apps seems to be the ability to innovate very quickly - throw some things out very quickly and see what sticks. Having a platform with a very low barrier to entry - no cost, no system administration headaches - might help to start some services which otherwise would never have made it off the ground. And should the new service become a runaway success - or even just being slashdotted, the web equivalent of a one-hit-wonder - then having an infrastructure which can scale seamlessly may just prove to be invaluable.

My biggest gripes with AppEngine is the lack of support for batch jobs and reliable database administration. Not everything can be easily implemented as short-lived web-requests - in particularly not backing up, restoring or upgrading and entire large data-store in an atomic or at least consistent fashion. The good news is that at least according to the roadmap, some of these problems are know and are being addressed.

And there is always the problem with lock-in to a single-vendor proprietary platform, which not everybody is comfortable with. I guess a Django based application, written to run on AppEngine, could be ported to run on a traditional server hosting, but any such transition would bring back all the infrastructure problems AppEngine was abstracting away in the first place.

Sunday, February 22, 2009

In Support of Code Reviews

Very few people doubt the usefulness of peer code reviews to improve software quality. Intuitively I would put it as one of the factors with the highest impact on quality, right up there with good design and thorough testing/validation.

Most projects I have worked on, did at least lip-service to code-reviews, but most failed to actually do it systematically. Not surprisingly since code-reviews done well are very time-consuming and put a huge burden on someone else other than the author of the code. The key to success would seem to minimize the hassle and burden put on the reviewer.

Some common ways to do that might be
  • Pair programming: the burden of review and credit for delivering the code are shared equally. While pair programming certainly seems to have many advantages and some very enthusiastic followers, it may be hard to implement in some places. While two pair of eyes certainly see more than one, it might also be helpful to get fresh pair of eyes do the review, who was not part of the design process.
  • Walk-through: the programmer presents the changes to the reviewer and walks him/her through the code, explaining his/her choices and answering questions. The risk here is that since the author is in driver seat, that the reviewer might be too easily coerced into accepting the author's point of view. Capturing the reviewer feedback and making sure it is properly incorporated is also particularly hard in informal approach. The need for shared face to face time for the review also puts constraints on author and reviewer and does not work very well for distributed teams or making use of unproductive time - e.g. on the train ride home.
  • Tool support: using a code-review support tool, the reviewer can at their own leisure look at the proposed changes and can add any comments and suggestions as annotations to the code changes presented by the tool. Author and reviewer can go back and forth with new versions and responding to each others comments until all the issues have been resolved. Such offline reviews, where author and reviewer never meet can be very efficient for both of them, but drawn out in time, as each takes their time to respond on each revision.
From my experience, a well designed collaborative code-review tool, tightly integrated with the version control system is the most successful way to implement effective code-reviews. Making code-reviews a natural part of the development process has some unexpected side-effects as well.
  • I find myself less tempted to cut corners (yes, I am naturally lazy!), if I know that my code will be reviewed line by line, causing me great embarrassment in front of my team-mates and peers.
  • If find that by having to spend significant amount of time carefully reviewing other people's code is a great learning experience. Looking closely at other developers code can often open up my eyes to new ways of doing something. For anybody who has mastered the craft of programming beyond what can be learned from books, looking at other people's code is one of the best ways to grow further as a programmer.

While most word-processors support reviews through incremental change tracking and embedding editorial contents into documents, there is no such support in most commonly used software development tools.

I think that the popularity of open-source projects has done a great deal in promoting and popularizing collaborative software development tools. It would be much harder today to find a software developer who has never heard of version control or issue tracking system and wouldn't think it quite normal to use them.

The good news is that collaborative code-review tools might be next. While 10 years ago, it would have been hard to find any references to code-review tools, a casual web-search now shows a few open-source and commercial options:
When the Android platform was released as an open-source project, the procedure to contribute changes includes code-reviews and approvals, by project members with committer privilege, integrating Gerrit a web-based collaborative code-review tool into its project hosting site.

Maybe a few years from now, a code-review system will be considered as natural and integral part of any project's development environment as we would consider today email, version control, issue-tracking or wiki/document sharing systems.

Cyrket

Cyrket.com provides web based interface to browse the Android Market. Sadly this is something which the official market site or the developer console still do not provide (and Google wouldn't even have to reverse-engineer their own protocol).

Here are the most recent ratings & comments on my apps:

Friday, February 20, 2009

Market Featured Application

BistroMath, the tip calculator is by far the most popular of my little Android applications. While introducing paid applications, the web interface has been revamped a bit. BistroMath even made it into the list of featured applications.

Friday, February 13, 2009

Party like its 1234567890

Today is party time for Unix geeks all over the world. Not because it's Friday the 13th or Valentine's day, but because today the Unix clock reaches the memorable state of '1234567890'. That is the number of seconds since 00:00 UTC on January 1st 1970.

To see when to celebrate in your timezone, run:

python -c 'import time; print time.ctime(1234567890)'