Friday, May 21, 2010

Thoughts on "Making Sense of Privacy and Publicity"

This year's SXSW keynote by danah boyd is probably one of the most insightful contributions to the debate on privacy and social networking. For those who have not yet seen it, the rough transcript can be found here. It puts a finger on so many important points, that it should be required reading for anybody who wants to work on consumer web services.

To summarize a key point: in real life things are usually not as simple as they seem. And that's bad news for the technocrats who typically build and run the virtual environments where social interactions are taking place online. Engineers and scientists like to simplify and standardize problems, apply Occam's razor, optimize systems along the dimensions of an assumed known quantitative model etc. The operators of today's large web properties study and analyze their users behavior and think to understand them better than the users understand themselves, but behind the user behavior observable from web logs are layers of significance and meaning which are completely hidden from this behavioral analysis.

In every society, people daily strike many a delicate balance between engagement and guardedness. We learn social rituals, what is appropriate in a given situation and what kind properties to expect from a certain context or environment. The boundaries of what is private and what is public are blurred. Some of the most private, intimate and confidential discussion happen in very public places - on park-benches on a warm summer night or in Washington D.C parking garages. In the offline world, experience allows us to judge how an environment will support our interactions - and in many jurisdictions, altering those properties through hidden means like microphones, telephoto lenses etc. is explicitly illegal as a violation of privacy rights.

In the online world things are a bit more tricky. We don't usually know as well how these virtual environments really behave and their properties are really easy to change by the people who build and run them. To make things worse, the Internet (almost) never forgets. While an intimate discussion years ago on park-bench has faded from the observable universe long ago, the same discussion done by IM may remain visible somewhere in a chat log forever. Users often assume rightfully that their daily lives are mundane enough that nobody will bother to explicitly track them through the digital noise. Safety by numbers and hiding in plain sight often works surprisingly well. Just because something is not explicitly block from access does not mean that the creator necessarily wants everybody to see it.

Assuming that users somehow find a delicate balance on how to operate within the virtual spaces formed by social networking and other web 2.0 richly interactive services, some of the worst things which the operators of these services can do to hurt their users is to change the rules on them. Since we do not know what kind of balance each user has found to make things work for themselves, it is hard to predict how any change might affect them. And since there is a lot of data in the system, changing the rules can even be retroactive: a spotlight suddenly shining into a corner of the virtual world where it was not supposed to, according to the expectations of the user.

Unfortunately the current best practice of software development is based on embracing change. The software as a service model allows to release early and release often, since nobody knows what will really resonate with users. In the end, we get services which are more sophisticated and more integrated with our daily lives than ever before, but at the cost of the eternal beta.

In order not to violate users sense of privacy, any changes which shift the fabric of our virtual online worlds that might affect the visibility or exposure of anything must be considered very carefully for unintended consequences.

As a fundamental principle, there should be no "ex post facto" or retroactive change to the visibility and exposure of anything without the most explicit and informed consent of the user. But even when operating diligently by such a high standard, accidents are bound to happen from misjudging the impact some changes may have in the user's world.

Ultimately as the Internet becomes more social, we need to better understand the social dynamics, conventions and rituals of its usage. But unfortunately that's not something that the introverted computer geeks who have so far built the foundations of the social web are particularly good at.

Thursday, May 20, 2010

From UGC to UCC

I have noticed, that a good part of the articles I read online have been suggested by members of my various social networks. Maybe it is a part of the true utility of social networks to be a platform for "User Curated Content".

While the web in its first phase tried to mirror the offline world, by moving every brick and mortar institution and service online, the so called web 2.0 promised a new world of participatory media, where everybody can create content. While digital media have drastically lowered the production costs, the web has driven distribution costs to near zero. Looking around on blogging sites, flickr, YouTube or other cornerstones of the "User Generated Content" revolution there are some seriously talented people out there! Some people have managed to make a mark, some even managed to make a living or become minor Internet celebrities in some field. Some other stuff is whimsical, funny or personal. There are unexpected viral hits or observers who happen to be at the right place at the right time and turn into citizen journalists. But the vast majority is just plain boring and inconsequential rubbish.

More so than ever, the problem has become discovery - i.e. finding the stuff that's relevant, interesting, worthwhile, stimulating, satisfying etc. at this point. When production and distribution of content is expensive as in traditional media, there are plenty of people whose primary role it is to make choices of what is being produced and distributed on behalf of their audience. They are called curators, editors, DJs, program directors, executive producers etc. and they are often the most most well know, prestigious (and feared/hated) people in their organization.

Curators make choices which works of art are on display and which ones are in storage, the ones at leading institutions even define what is considered art, based on what they acquire for their collections. Editors in chief decide what stories are being printed and define our perception of what is news. In a situation of scarce resources the difference between a curator and a censor are often only the nature of their intentions (educate and enlighten vs. oppress). On the Internet, the role of a curator is different. There is (near) infinite wall-space and everything which exists can be exposed - but because of that often not be seen or found by anybody in the sheer mass of stuff out there.

There are a few successful strategies for finding something in this giant heap of digital noise. Contextual search ranking revolutionized web-search in the late 90ies by creating algorithmic determinations of what is presumably more interesting or relevant to a particular question. Clustering algorithms can help find similar things to something we like and recommendation engines can suggest things which people like me have liked.

However, in a world dominated by the chatter of millions of undistinguished sources, search can often fail to find the nuggets in the trash heap. And recommendation engines only reinforce my current point of view. How can I learn, grow, be surprised and intrigued if I am only ever fed things recommended by people like me? Why should my taste and judgment be any good... I would rather get recommendations from people who are smarter, more knowledgeable, more stylish and more plugged in to a particular filed, but that's hard to decide by algorithmic means.

So we are back to curators. Or editors, guides, teachers, gurus, opinion-makers, trend-leaders, talent-scouts or whatever we want to call them. Relevance is no longer defined globally but based on people whose judgment we trust and respect. Social networks can be a source of such relevance, following the age-old patterns of world of mouth among friends and family. Or more powerful would be asymmetric social networks where we can find somebody whose judgment we respect and "follow them", without them necessarily having to know us.

In fact many blogger are in fact more editors or curators than creators of original content. This is even more so with micro-blogging systems like Twitter, which many consider the quintessential asymmetric social network. And Wikipedia is probably the most high profile project, where domain experts can live out their inner librarian and do so with great determination.

Given the importance of curators, I think there is still too much emphasis on content creation. The real challenge today is to mine the piles of digital trash for nuggets of gold which most certainly exist in numbers never seen before. We already have more content than we ever know what to do with, but there are not enough platforms and frameworks where people who would like to organize it could shine and be recognized. Part of the problem is that when it comes to derived works, copyright gets really murky and it's hard to say who should get credit (or even paid) for what. But it is time for online curators to get more respect and for librarians to step into the limelight. And this could easily be one of the next big things for online digital media platforms.

Tuesday, May 18, 2010

A day in the life of the Internet

Todays top suggestions on google.com search for "How do I" are:
  1. how do i delete my facebook account
  2. how do i find my ip address
  3. how do i get a passport
  4. how do i know if im pregnant
  5. how do i love thee
  6. how do i look
Out of which only #5 has a relatively straightforward answer:
... Let me count the ways.
I love thee to the depth and breadth and height
My soul can reach, when feeling out of sight
For the ends of Being and ideal Grace.
I love thee to the level of everyday's
Most quiet need, by sun and candle-light.
I love thee freely, as men strive for Right;
I love thee purely, as they turn from Praise.
I love thee with a passion put to use
In my old griefs, and with my childhood's faith.
I love thee with a love I seemed to lose
With my lost saints, --- I love thee with the breath,
Smiles, tears, of all my life! --- and, if God choose,
I shall but love thee better after death.

Sonnet 43, Elisabeth Barret Browning

Friday, May 7, 2010

IT != IT - the Case for a Differentiated Immigration Policy

The Swiss government recently reduced the quota for work-permits for applicants from so called 3rd-states - which typically means countries outside the EU and not covered by the free-trade treaties between the EU and Switzerland. After a highly publicized protest led by high-tech companies like Google, Microsoft and IBM, the Swiss government has rather quickly reverted its decision.

In the midst of a recession with higher than usual unemployment and increased levels of immigration from the EU following the free-trade agreements, the general mood in the population is not very supportive of any increase in immigration quotas. This is seen as yet another attempt by greedy corporation to undercut the Swiss standard of living by importing cheap labor from overseas - typically from south-east Asia in what is generally by called the IT or information technology sector. How can there be a shortage of IT labor, if almost everybody knows someone who is unemployed and supposedly somehow "in IT"?

As with most controversies, there may be some truth to the matter of companies trying to use immigration to depress labor costs, but the crucial core of the problem here is the ability for for companies who operate global R&D facilities in Switzerland (like the companies named above) to be able to attract the best possible talent in a particular field - regardless of skin-color or country of origin.

Since information technology has permeated about any aspect of not just business, but also increasingly personal live, the so called IT sector has become large and diverse that saying somebody is "in IT" is about as meaningful and descriptive as saying that somebody is working in an office.

The vast majority of IT jobs are about supporting and customizing systems and applications based on the specific needs of their users. The most visible IT workers are PC technicians or system administrators, which almost everybody knows first hand from their daily work. Or armies application developers which work on big in-house IT projects for large corporations like banks or insurance companies - either as employees or as contractors from large IT services companies like Accenture, IBM or Infosys on behalf of local clients. Most of what is going on here, is indeed not rocket science from a technical point of view. The key stake-holders are not technology companies, nor are they interested in technology per-se. They rather consider it a necessary evil, a cost center which they would like minimize. No matter how high-tech certain IT service providers are giving themselves for the public image, in reality they are trying to be as technically unspectacular and conventional as they can be in order to minimize the risks of implementing something which has basically been done before many times over in slightly different forms. As no two organizations are exactly the same, the IT systems used to support them are also slightly different which causes all this effort and duplication. For many of these jobs, the ability of communicating with the users of the system and understanding their application domain is a lot more important than raw technical skills and knowledge. It is rightfully debatable to what degree immigration vs. increased education and training should help resolve the general shortage in this still fast growing sector.

But there is also a very small segment of the IT industry, which is the true high-tech sector. This is where the technological innovation happens. These are the companies and people, why build the core pieces like operating systems, database engines, computer chips, programming frameworks or communication equipment. This is typically also where the value-add is most concentrated and companies like HP, Apple, Cisco, Intel, Microsoft, Oracle or Google have famously propelled their founders and investors into the top league of the worlds most wealthy people. This is where the coolest and sexist jobs are for the technically inclined, ambitious, talented and well educated in the IT workforce and the companies who are being seen as the avantguarde of technological innovation can typically get a pick of whom they want to hire globally.

This kind of jobs are also typically concentrated in a few select places around the world, most prominently in Silicon Valley, because this is where the necessary key talent can be found. However this is not because the people born in Santa Clara County are somehow smarter than the rest of the world, but because of migration. Silicon Valley has a share of foreign-born population way above the US average at about 40% (10% US average) an a rate of over 60% among engineers and scientists in the Valley. And this hides an equally significant domestic migration within the 300 million US population, where many with advanced degrees in engineering and science have moved to Silicon Valley from all over the country, if they want to play in the top league of their field.

Even though they pale in scale and importance next to Silicon Valley, there are a number of secondary clusters of high tech excellence around the world. Switzerland is reasonably well positioned with a strong tradition of industrial innovation going back to the industrial revolution in the 19th century, two technical universities who often appear among the highest non-anglosaxon institutions on many league-tables and a number of high-profile R&D labs. Some of them by domestic champions (pharma & machine industry) and some number of US high-tech companies (e.g. IBM, Google, Microsoft, Cisco) who have decided that Switzerland is a good place to hire some of the top talent, who for some reason doesn't want to move to Silicon Valley... It would be hubris for a country of barely 7 million to assume that a significant part of the worlds leading experts in any particular field could be produced domestically, no matter how strong a culture of excellence or how strong the confidence in the local education system.

Any organization which is at the same time highly specialized and world class must necessarily be able to recruit globally from the best talent possible in that particular field, whether it is a top-ranked symphony orchestra, a premier-league football club, an elite-university or a world-class industrial R&D lab.

The reasons why global companies have chosen Switzerland as a place for global R&D is only partly because of a strong technical tradition, good universities and some number of key talent already there, but also primarily because of the highly rated quality of life, including reliable public services, picturesque landscape, low personal income taxes, safety, stability and a generally pragmatic government. Basically a place where it is relatively easy to convince people to relocate to, who otherwise have plenty of choices, options and other offers.

In the grand scheme of things, these few world-class labs will only employ a few 100 to few 1000 highly educated specialists and as such have very little impact on the overall employment or immigration situation. But they generally contribute a disproportionate amount to the economic development through prestige or intensified interactions and networking with other local firms and universities. For the one key ingredient - the ability to attract top talent, they need to to be able to recruit internationally with minimal restrictions and interference. At this point it is up to the Swiss government to either leverage its current reasonably strong position in the competitive global knowledge economy or to snatch defeat from the jaws of victory. E.g. through misguided pandering to the populist right-wing on immigration and to the left in the form of misguided attempts at labor market protectionism.

Organizations who can credibly make the point that they are in that global league, recruiting for the top talent in their field, should be except from any restrictions and quotas. In addition there should generally be a priority visa category based on education, skills and experience compared to the best in their filed - similar to the "alient with extraordinary ability" visa in the US. To go a step further - anybody who graduates with distinction from any university in whatever to-N global league-table, should have the automatic pre-approved right to a work and residence permit should they choose to come. (To calm the shrieking voices of panic on the right: very few would actually come, since they typically have plenty of opportunities and lots of other good offers. )

Saturday, May 1, 2010

A Game Changer for Public Transportation Users

My favorite and most used app on Android isn't even an app, but rather a service. It is the Google Transit public transportation directions feature in Google maps, which can also be accessed more easily through the Maps application on Android - or on any other mobile platform which supports Google Maps for mobile.

In combination with the extremely dense and frequent network of public transportation in Switzerland, the transit directions on the phone offer a level of spontaneous mobility, which is generally associated with driving. When it came to using public transportation networks, people tended to know the routes by heart which they frequently travel (e.g. daily commute), while anything else required thorough planing by poring over books of printed time-table - an activity enjoyed only by the most hard-core train buffs.

The Swiss public transportation systems has always been particularly well integrated, across all providers and including everything from urban transit, buses, trains, boats to touristic mountain cable cars. The system-wide printed time-table the size of a phone-book has long been online and can be looked up at the SBB website. There is now also an Android app, Fahrplan CH - which acts as a query front-end to this site, but no official app yet as for the iPhone.

On the mobile maps application, finding next transit connections from the current locations is as easy as entering the destination, even a fuzzy one with some help from Google maps search & suggestions to find the exact destination address. The app shows the next few connections in an overview tab with total travel time and as a detailed list with connection times and description of each stop and carrier to take. A particular trip can also be viewed on the map, which is particularly useful for the first and last part which usually involves walking to and from the station or stop.

Unfortunate limitations of Google Transit is its model of treating each metro area in isolation, since the data usually comes from isolated transit authorities. Fortunately, Switzerland is represented as a single transit domain, but there is now way to display international directions - e.g. from a local address somewhere in Z├╝rich to a local address somewhere Paris, which would involve taking a bus or tram to the main station, the TGV to Paris and a metro or bus connection towards the final destination. The SBB website does show basic international connection, but not at the level of every possible subway stop in every possible city, since that data-set really doesn't exist in a single integrated form.

Moving to a new city a few month ago, really was a good test for how well on-the-go transit routing and trip planing works. We didn't know our way around and often left the house only knowing where we wanted to go but not how to best get there or how to get back. But this only works in areas with very dense coverage of public transportation, where we can be sure that there is probably a connection every 15minutes or so and there will always be a way back when we want to and service does not suddenly stop in the middle of the afternoon.

It does not take a lot of imagination that mobility itself would be a killer application for mobile devices and dynamic public transportation routing in areas with good service coverage is clearly delivering on this promise.