weblog.mattdorn.com

Some years ago I became aware of a software project called Chandler, a personal information manager/calendar/email client being developed by something called the Open Source Applications Foundation, which was started by by Mitch Kapor of Lotus fame. It appeared on my radar for two reasons. First, it seemed an unusually ambitious effort to compete in a space that had been dominated by proprietary offerings like Microsoft Outlook; second, it was being written in what had quickly become my programming language of choice, Python. When I learned that the project had attracted the full-time participation of industry heavyweights like Andy Hertzfeld, who played a key role in the development of the first Apple Macintosh, I was doubly interested.

It wasn’t long before Chandler lost my attention, though, as the hype surrounding it diminished amid complaints about its exceptionally slow progress. Occasionally I wondered, though: How could a project with so much industry muscle behind it, being written in a language renowned for its productivity, go so far off track? So when I read that a book-length treatment of the Chandler development process had been published, I couldn’t wait to get my hands on it.

Written by a fairly technically savvy layman, former Salon editor Scott Rosenberg, Dreaming in Code is an adept survey of the range of dysfunction that plagues even the most talented software development teams. His review of the available literature — ranging from Frederick Brooks’s The Mythical Man-Month (a book whose thesis furnished one of the few iron laws in the field) to Joel Spolsky’s rants on his popular "Joel on Software" blog — is comprehensive and informative, but probably the most insightful aspect of the book is its treatment of the aspects of software development that make it unique among disciplines.

For example, at the outset Rosenberg speaks of something he calls "software time":

Time really does seem to behave differently around the act of making software. When things go well, you can lose track of the passing hours in the state psychologists call "flow." When things go badly, you get stuck, frozen between dimensions, unable to move or see a way forward.

It is perhaps this quality, Roseberg suggests, that furnishes the foundation for Brooks’s Law, alluded to above, that "adding manpower to a late software project makes it later" — a seemingly paradoxical concept in most fields of endeavor.

Brooks provides further material for Rosenberg when he writes that "the programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination." The idealism implied by this description may be what’s behind a whole host of programming ills that lead software practitioners down a path of impracticality and ultimately cause projects to go far over budget, over schedule, or fail completely.

Among them is the "Not Invented Here Syndrome." The reusability of existing software components has long been one of several holy grails in the industry, and exponential increases in computing power as well as the rise of Open Source and the Internet have in many ways made it a reality. The problem is that most programmers exhibit a tendency to want to reinvent the wheel. On this matter Rosenberg quotes Larry Constantine at length:

Unfortunately, most programmers like to program. Some of them would rather program than eat or bathe. Most of them would much rather cut code than chase documentation or search catalogs or try to figure out some other stupid programmer’s idiotic work. … Other things being equal, programmers design and build from scratch rather than recycle.

Further complicating matters is that even the most talented programmers can undermine their own productivity by engaging in excessive "axe-sharpening," or refining their development environments and tool sets, rather than focussing on solving the problem at hand. Rosenberg offers a parable by Peter Drucker worth reproducing here:

A favorite story at management meetings is that of the three stonecutters who were asked what they were doing. The first replied, "I am making a living." The second kept on hammering while he said, "I am doing the best job of stonecutting in the entire country." The third one looked up with a visionary gleam in his eyes and said, "I am building a cathedral."

The third man is, of course, the true "manager." The first man knows what he wants to get out of the work and manages to do so. He is likely to give a "fair day’s work for a fair day’s pay." It is the second man who is a problem. Workmanship is essential, … but there is always a danger that the true workman, the true professional, will believe that he is accomplishing something when in effect he is just polishing stones or collecting footnotes.

Programmers by their nature (or perhaps by the nature of the technology of their tools) tend to be the second man.

Finally, Rosenberg notes, in spite of an abundance of empirical evidence about the difficulty of successfully completing a software project (the book’s epigraph is Donald Knuth’s observation that "software is hard"), "most of the programmers I have interviewed and gotten to know in the course of my research are consistently, and sometimes unaccountably, optimistic about their work. If they are Sisyphuses, they are happy ones."

So are there any constructive lessons to be carried away from Rosenberg’s cautionary tale?

Aside from implying that one ought to bring a greater awareness of the problems described above to software projects (e.g., counterbalancing programmers’ unaccountable optimism when planning a project), the book does provide some insights gleaned from a few instances of successful software projects.

For example, Watts Humphrey, who took over IBM’s software organization in 1966, instituted a culture of planning where not only were plans mandatory, but "had to be ‘bottom-up,’ derived from the experience and knowledge of the programmers who would commit to meeting them, rather than ‘top-down,’ imposed by executive fiat or marketing wish." The result was an organization that "did not miss a date for the next two and a half years."

In addition, after observing that a key problem in most software projects is the absence of a firm understanding of what it means to be "done," Rosenberg reports that in the rare cases he’s seen where software projects are successful from both a budget and schedule perspective, the success is "a by-product of iron-willed restraint — a choice firmly made and vocierfously reasserted at every challenge to limit a project’s scope. Where you find software success stories, you invariably find people who are good at saying no."