Wired on Friday:As Scott Rosenberg notes in his fresh and captivating book Dreaming in Code, major software products drive much of the developed world's economy, and yet the results of development projects are often over budget, full of bugs and sometimes dangerously unusable.
"If architects built buildings the way programmers build programs, the first woodpecker to turn up would destroy all of civilisation," goes the old coder's joke.
The real joke is that 50 years into the art of programming, nobody knows how to make software anywhere near as well as a townhouse. The question posed by many a midnight coder and her long-suffering manager is: why did we screw up again with the estimates? Why do we keep making those same mistakes?
To find out, Rosenberg spent several years at the Open Software Applications Foundation (Osaf).
The Osaf is an odd hybrid of many generations of Silicon Valley expertise. Founded by PC old-timer Mitch Kapor, staffed by recently rich ex-dotcomers, it was established as a non-profit organisation in San Francisco with the resources to pursue a dream: an open-source personal information manager that would let everyone organise their life the way they wanted.
If anyone knew how to push the software industry forward, it would be the hybrid vigour from this project.
You can guess the rest. The Osaf began in 2001, as so many dreams did, in the pit of the dotcom recession. Six years later, it has still to ship software, even as Rosenberg left to ship his book. As with (my now second favourite) classic of this genre, Fred Moody's Microsoft insider I Sing the Body Electronic, the author went to learn what goes right with software, and came out with a fine study in what goes wrong.
The challenge in documenting a flailing software project is not to becalm your reader along with the code. The pain of reading about months of dead ends and slow meanderings can be almost as frustrating on the page as it is in the cubicle farm.
There's a beautiful moment midway through the story where Rosenberg steps into his own narrative to note that self-respecting geeks should be, at this point, hurling the book at the wall in disgust at the Osaf's descent into obvious development mistakes.
He's right, and he's also right in noting that the amazing ability to spot others failing isn't enough. We all know the signs. It's reached the point where the golden rules of programming have become clichés:
Fred Brook, author of the Mythical Man Month, produced two of the most resilient adages: more coders do not accelerate project, but slow them down, and that there is no silver bullet - no one new coding fad would solve all ills. And we all know, in our hearts, that there is no short cut for programming for the future.
What we need to know is when we're wandering from these, the path that these warnings define.
Books like Rosenberg's do a great job at allowing us to step away from the keyboard and see our foibles as others might see them. His skill in portrayal lies not just in explaining the tribulations of the software project to the layman, it also lies in explaining them afresh to the seasoned - and therefore oblivious - hacker.
His mastery in picking just the right metaphor to pull an obscure coding controversy into the common world is unparalleled: I've never seen the difference between software's "back-end" and "front-end" portrayed better than in Rosenberg's analogy between the incomprehensible R2D2 and its fey, deferential interpreter, C3PO.
But even more impressive (at least, to me) is his ability to uncover anecdotes and connections that even the most oversurfing geek would not have heard. Even for me, an obsessive follower of the trivia of tech culture, there were genuine surprises in every chapter.
That said, Rosenberg does not tell all about his subjects. A lesser writer might have bemoaned how dull the players in the coding world are.
At the Osaf, as in so many Silicon Valley coding houses, characters in this story are, in fact, rather more flamboyant than Rosenberg coyly portrays them. Kapor's past management has stormier scenes than the genial guru-like figure presented here: the involvement of a central figure in the story, Rys McCusker, has had less polite handling at the hands of other geeks in the tech community - and a harder story that Rosenberg uncovers.
Rosenberg draws a veil over a lot, seemingly in deference to his "embedded" position as a reporter in the Osaf and in the San Francisco scene for a decade - and probably for many years hence.
Just as Kapor can't cut and run from the Osaf without losing his reputation, Rosenberg has too much tied up in this community to kiss and tell. The result can be tantalisingly as well as occasionally frustrating, even in his delicious asides.
He gives a beautiful, potted description of the fall of Douglas Engelbart's Xerox Parc project, (which created the mouse, as well as inspiring a generation of GUI designers), but appends it with a throwaway phrase about it foundering in the EST (Erhard Seminars Training) movement. The first modern PC clashed with Californian cult self-helpers? What was that about?
With an investigative book like this, you don't always want to be googling for the best dirt.
So perhaps I crave the trivia a little too much, but I crave good techie books even more. It's perhaps forgivable that such great literature on programming should happen so rarely: the mix of tech skills, explanatory abilities, and sheer determination to trail and document what from the outside looks as exciting as an accountant's meditation retreat, is rare among writers of Rosenberg's calibre. But we badly need those books to be written.
It's insights like these that might lead us to find out more about the dark art of programming, and perhaps yet save civilisation from the swiftly incoming woodpeckers.
