A few weeks ago, I met an old friend – Narain Gehani, once at Cornell with me, then at Bell Labs, and now at Stevens Institute of Technology – and he presented me with a book of reminiscences that he had written: “Bell Labs: Life in the Crown Jewel”. I enjoyed reading about Narain’s life in the 35 years since we last met; Narain tells a great story, and is a keen observer. However, one thing struck me about his experiences that he probably didn’t anticipate – the frequency with which he participated in projects aimed at delivering software products or software-infused products, products that ultimately didn’t achieve merited success in one marketplace or other (Surf n Chat and Maps r Us were the ones that eventually surfaced on the Web).
This is one aspect of software development that I rarely see commented on. We have all talked about the frequency of software-development project failure due to poor processes. But how about projects that by any reasonable standard should be a success? They produce high-quality software. The resulting solutions add value for the customer. They are well received by end users (assuming end users see them at all). They are clearly different from everything else in the market. And yet, looking back on my own experience as a software developer, I realize that I had a series of projects that were strikingly similar to Narain’s – they never succeeded in the market as I believe they should have.
My first project was at Hendrix Corporation, and was a word processing machine that had the wonderful idea of presenting a full page of text, plus room on the side for margins (using the simple trick of rotating the screen by 90 degrees to achieve “portrait” mode). For those who have never tried this, it has a magical effect on writing: you are seeing the full page as the reader is seeing it, discouraging long paragraphs and encouraging visual excitement. The word processor was fully functional, but after I left Hendrix it was sold to AB Dick, which attempted to fold it into their own less-functional system, and then it essentially vanished.
Next was a short stint at TMI, developing bank-transfer systems for Bankwire, Fedwire, etc. The product was leading-edge at the time, but the company was set up to compensate by bonuses that shrank as company size increased. As employee dissatisfaction peaked, it sold itself to Logica, key employees left, and the product apparently ceased to evolve.
At Computer Corporation of America, I worked on a product called CSIN. It was an appliance that would do well-before-its-time cross-database querying about chemical substances “in the field”, for pollution cleanup efforts. It turned out that the main market for the product was librarians; and these did not have the money to buy a whole specialized machine. That product sold one copy in its lifetime, iirc.
Next up was COMET/204, one of the first email systems, and the only one of its time to be based on a database – in this case, CCA’s wonderful MODEL 204. One person, using MODEL 204 for productivity, could produce a new version of COMET/204 in half the time it took six programmers, testers, and project managers to create a competing product in C or assembler. COMET/204 had far more functionality than today’s email systems, and solved problems like “how do you discourage people from using Reply All too much?” (answer: you make Reply the default, forcing them to do extra effort to do Reply All spam). While I was working on it, COMET/204 sold about 10 copies. One problem was the price: CCA couldn’t figure out how to price COMET/204 competitively when it contained a $60,000 database. Another was that in those days, many IT departments charged overhead according to usage of system resources – in particular, I/Os (and, of course, using a database meant more I/Os, even if it delivered more scalable performance). One customer kept begging me to hurry up with a new version so he could afford to add 10 new end users to his department’s system.
After that, for a short stint, there was DEVELOPER/204. This had the novel idea that there were three kinds of programming: interface-driven, program-driven, and data-driven. The design phase would allow the programmer to generate the user interface, the program outline, or the data schema; the programmer could then generate an initial set of code (plus database and interface) from the design. In fact, you could generate a fully functional, working software solution from the design. And it was reversible: if you made a change in the code/interface/data schema, it was automatically reflected back into the design. The very straightforward menu system for DEVELOPER/204 simply allowed the developer to do these things. After I left CCA, the company bundled DEVELOPER/204 with MODEL/204, and I was later told that it was well received. However, that was after IBM’s DB2 had croaked MODEL 204’s market; so I don’t suppose that DEVELOPER/204 sold copies to many new customers.
My final stop was Prime, and here, with extraordinary credit to Jim Gish, we designed an email system for the ages. Space is too short to list all of the ideas that went into it; but here are a couple: it not only allowed storage of data statically in user-specified or default folders, but also dynamically, by applying “filters” to incoming messages, filters that could be saved and used as permanent categorizing methods – like Google’s search terms, but with results that could be recalled at will. It scrapped the PC’s crippled “desktop metaphor” and allowed you to store the same message in multiple folders – automatic cross-referencing. As a follow-on, I did an extension of this approach into a full-blown desktop operating system. It was too early for GUIs – but I did add one concept that I still love: the “Do the Right Thing” key, which figured out what sequence of steps you wanted to take 90% of the time, and would do it for you automatically. Because Prime was shocked by our estimate that the email system would take 6 programmers 9 months to build, they farmed it to a couple of programmers in England, where eventually a group of 6 programmers took 9 months to build the first version of the product – releasing the product just when Prime imploded.
Looking back on my programming experiences, I realized that the software I had worked on had sold a grand total (while I was there) of about 14 copies. Thinking today, I realize that my experiences were pretty similar to Narain’s: as he notes, he went through “several projects” that were successful development processes (time, budget, functionality) but for one reason another never took off as products. It leads me to suspect that this type of thing – project success, product lack of success happens just as much as software development failure, and very possibly more often.
So why does it happen? It’s too easy to blame the marketers, who often in small companies have the least ability to match a product to its (very real) market. Likewise, the idea that the product is somehow “before its time” and not ready to be accepted by the market may be true – but in both Narain’s and my experience, many of these ideas never got to that point, never got tested in any market at all.
If I had to pick two factors that most of these experiences had in common, they were, first, barriers within the company between the product and its ultimate customers, and second, markets that operate on the basis of “good enough”. The barriers in the company might be friction with another product (how do you low-ball a high-priced database in email system pricing?) or suspicion of the source (is Bell Labs competing with my development teams?). The market’s idea of “good enough” may be “hey, gmail lets me send and receive messages; that’s good enough, I don’t need to look further”.
But the key point of this type of software failure is that, ultimately, it may have more profound negative effects on top and bottom lines, worker productivity, and consumer benefits in today’s software-infused products than development failure. This is because, as I have seen, in many cases the ideas are never adopted. I have seen some of the ideas cited above adopted eventually, with clear benefits for all; but some, like cross-referencing, have never been fully implemented or widely used, 25 or more years after they first surfaced in failed software. Maybe, instead of focusing on software development failure, we should be focusing on recovering our memory of vanished ideas, and reducing the rate of quality-product failure.
When I was at Sloan School of Management at MIT in 1980, John Rockart presented a case study of a development failure that was notable for one outcome: instead of firing the manager of the project, the company rewarded him with responsibility for more projects. They recognized, in the software development arena, the value of his knowledge of how the project failed. In the same way, perhaps companies with software-infused products should begin to collect ideas, not present competitive ideas, but past ideas that are still applicable but have been overlooked. If not, we are doomed not to repeat history, but rather, and even worse, to miss the chance to make it come out better this time.
