Recently there passed in front of me two sets of thoughts: one, an article by Marc Andreessen (once a Web software developer, now a VC) in the Wall Street Journal about how software is “eating the world”; one by a small development tools/services company called Thoughtworks about a new technique called Continuous Delivery.
Guess which one I think is more important.
Still, both make important points about changes in computing that have fundamental effects on the global economy. So let’s take them one at a time, and then see why I think Continuous Delivery is more important – indeed, IMHO, far more important. A brief preview: I think that Continuous Delivery is the technique that takes us into the Third Age of Software Development – with effects that could be even more profound than the advent of the Second Age.
To Eat the World, First Add Spices
A very brief and cursory summary of Andreessen’s argument is that companies selling software are not only dominant in the computer industry, but have disrupted (Amazon for books, Netflix for movies) or are “poised to disrupt” (retail marketing, telecom, recruiting, financial services, health care, education, national defense) major additional industries. The result will be a large proportion of the world’s industries and economy in which software companies will be dominant – thus, software will “eat the world”.
My quarrel with this argument is not that it overstates the case, but rather that it understates it. What Andreessen omits, I assume in the interest of conciseness, is that in far more industries and companies, software now is the competitive differentiator between companies, even though the companies’ typical pedigree is primarily “hardware” or “services”, and the spending on software is a small percentage of overall costs.
My favorite examples, of course, are the company providing audio and video for stadiums, which relies on software to synchronize the feeds to the attendees better than the competition, and Boeing, which would not be able to provide the differentiating cost-saving use of composites were it not for the massive amounts of software code coordinating and monitoring the components of the Dreamliner. In health care, handling the medical health record and support for the new concept of the medical home requires differentiating software to glue together existing tools and health care specialists, and allow them to offer new types of services. It isn’t just the solo software company that is eating the world; it’s the spice of software differentiation in the mostly non-software businesses that is eating these companies from within.
So Andreessen’s article signals a growing awareness in the global economy of the importance not just of computing, but of software specifically, and not just as a separate element of the economy, but as a pervasive and thoroughly embedded aspect of the economy whose use is critical to global economic success or failure. How far we have come, indeed, from the point 30-odd years ago when I suggested to Prof. Thurow of MIT that software competitive advantage would allow the US to triumph over the Japanese economic onslaught, and he quite reasonably laughed the idea to scorn.
I do have one nit to pick with Andreessen. He trots out the assertion, which I have heard many, many times in the past 35 years, that companies just can’t find the programming skills they need – and this time, he throws in marketing. Every time people have said this, even at the height of the Internet boom, I have observed that a large body of programmers trained in a previous generation of the technology is being overlooked. That sounds as if it makes sense; but, as anyone should know by now, it is part of the DNA of any decent programmer to constantly acquire new knowledge within and beyond a skill set – good programmers are typically good programmers in any arena, from Web-site design to cloud “program virtualization”.
Meanwhile, I was hearing as late as a year ago about the college pipeline of new programmers being choked off because students saw no future jobs (except in mainframes!), and I can attest that available, technically savvy marketers in the Boston area – a well-known breeding ground – are still thick on the ground, while prospective employers in the Boston area continue to insist that program managers linking marketing and programming have at least five years of experience in that particular job. It just won’t wash, Mr. Andreessen. The fault, as is often the case, is probably not mainly in the lack of labor, but in companies’ ideological blinders.
The Idea of Continuous Delivery
Continuous Delivery, as Thoughtworks presents it, aims to develop, upgrade, and evolve software by constant, incremental bug fixes, changes, and addition of features. The example cited is that of Flickr, the photo sharing site, which is using Continuous Delivery to change its production web site at the rate of ten or more changes per day. Continuous Delivery achieves this rate not only by overlapping development of these changes, but also by modularizing them in small chunks that still “add value” to the end user and by shortening the process from idea to deployment to less than a day in many cases.
Continuous Delivery, therefore, is a logical end point of the whole idea of agile development – and, indeed, agile development processes are the way that Thoughtworks and Flickr choose to achieve this end point. Close, constant interaction with customers/end users is in there; so is the idea of changing directions rapidly, either within each feature’s development process or by a follow-on short process that modifies the original. Operations and development, as well as testing and development, are far more intertwined. The shortness of the process allows such efficiencies as “trunk-based development”, in which the process specifically forbids multi-person parallel development “branches” and thus avoids their inevitable communication and collaboration time, which in a short process turns out to be greater than the time saved by parallelization.
I am scanting the many fascinating details of Continuous Delivery in the real world in order to focus on my main point: it works. In fact, as agile development theory predicts, it appears to work better than even other agile approaches. Specifically, Thoughtworks and Flickr report:
• A much closer fit between user needs and the product over time, because it is being constantly refreshed, with end-user input an integral part of the process;
• Less risk, because the small changes, introduced into production through a process that automatically minimizes risk and involves operations, decrease both the number of operational bugs and the time to identify and fix each; and
• Lower development costs per value added, as loosely measured by the percentage of developed features used frequently as opposed to rarely or never.
The Ages of Software Development
I did my programming time during the First Age of software development, although I didn’t recognize it as such at the time. The dominant idea then was that software development was essentially a bottleneck. While hardware and network performance and price-performance moved briskly ahead according to Moore’s Law, software developer productivity on average barely budged.
That mattered because in order to get actual solutions, end users had to stand in line and wait. They stood in line in front of the data center, while IT took as long as two years to generate requested new reports. They stood in line in front of the relatively few packaged application and infrastructure software vendors, while new features took 9 months to arrive and user wish lists took 1 ½ years to satisfy. They stood in line in front of those places because there was nowhere else to turn, or so it seemed.
I noted the death of the First Age and the arrival of the Second Age in the late ‘90s, when a major Internet vendor (Amazon, iirc) came in and told me that more than half of their software was less than six months old. Almost overnight, it seemed, the old bottlenecks vanished. It wasn’t that IT and vendor schedules were superseded; it was that companies began to recognize that software could be “disposable.” Lots of smaller-scale software could be developed independent of existing software, and its aggregate, made available across the Internet by communities, as well as mechanisms such as open source, meant that bunches of outside software could be added quickly to fill a user need. It was messy, and unpredictable, and much of the new software was “disposable” wasted effort; but it worked. I haven’t heard a serious end user complaint about 2-year IT bottlenecks for almost a decade.
Andreessen’s “software eating the world”, I believe, is a straightforward result of that Second Age. It isn’t just that faster arrival of needed software from the new software development approach allows software to start handling some new complex tasks faster than physical products by themselves – say, tuning fuel mixtures constantly for a car via software rather than waiting for the equivalent set of valves to be created and tested. It is also that the exponential leap in the amount of the resulting software means that for any given product or feature, software to do it is becoming easier to find than people or machines.
However, it appears clear that even in the Second Age, software as well as physical products retain their essential characteristics. Software development fits neatly into new product development. New products and services still operate primarily through more or less lengthy periods of development of successive versions. Software may take over greater and greater chunks of the world, and add spice to it; but it’s still the same world.
The real-world success of Continuous Delivery, I assert, signals a Third Age, in which software development is not only fast in aggregate, but also fast in unitary terms – so fast as to make the process of upgrade of a unitary application by feature additions and changes seem “continuous”. Because of the Second Age, software is now pervasive in products and services. Add the new capabilities, and all software-infused products/services -- all products/services – start changing constantly, to the point where we start viewing continuous product change as natural. Products and services that are fundamentally dynamic, not successions of static versions, are a fundamental, massive change to the global economy.
But it goes even further. These Continuous-Delivery product changes also more closely track changes in end user needs. They also increase the chances of success of introductions of the “new, new thing” in technology that are vital to a thriving, growing global economy, because those introductions are based on an understanding of end user needs at this precise moment in time, not two years ago. According to my definition of agility – rapid, effective reactive and proactive changes – they make products and services truly agile. The new world of Continuous Delivery is not just an almost completely dynamic world. It is an almost Agile World. The only un-agile parts are the rest of the company processes besides software development that continue, behind the scenes of rapidly changing products, to patch up fundamentally un-agile approaches in the same old ways.
And so, I admit, I think Thoughtworks’ news is more important than Andreessen’s. I think that the Third Age of Software Development is more important than the Second Age. I think that changing to an Agile World of products and services is far, far more important than the profound but more superficial changes that the software infused in products and services via the Second Age has caused and will cause.
Final Thoughts and Caveats
Having heralded the advent of a Third Age of Software Development and an Agile World, I must caution that I don’t believe that it will fully arrive any time in the next 2-3 years, and perhaps not for a decade, just as it took the Second Age more than a decade to reach the point of Andreessen’s pronouncements. There is an enormous amount of momentum in the existing system. It took agile development almost a decade, by my count, to reach the critical mass and experience-driven “best practices” it has achieved that made Continuous Delivery even possible to try out. It seems logical that a similar time period will be required to “crowd out” other agile new-product development processes and supersede yet more non-agile ones.
I should also add that, as it stands, it seems to me that Continuous Delivery has a flaw that needs to be worked on, although it does not detract from its superiority as a process. Continuous Delivery encourages breaking down features and changes into smaller chunks that reflect shorter-term thinking. This causes two sub-optimal tendencies: features that “look less far ahead”, and features that are less well integrated. To encapsulate this argument: the virtue of a Steve Jobs is that he has been able to see further ahead of where his customers were, and that he has been able to integrate all the features of a new product together exceptionally well, in the service of one, focused vision rather than many, diffused visions.
Continuous Delivery, as it stands, pushes the bar more towards more present-focused features that are integrated more as a politician aggregates the present views of his or her constituents. Somehow, Continuous Delivery needs to re-infuse the new-product development process with the ability to be guided at appropriate times by the strong views of an individual “design star” like Steve Jobs – else the Agile World will lose some of its ability to deliver the “new, new thing.” And it would be the ultimate irony if agile development, which aims to release programmers from the stultifying, counter-productive constraints of corporate development, wound up drowning their voices in a sea of other (end-user) views.
But who cares? In this latest of bad-news seasons, it’s really nice to look at something that is fundamentally, unreservedly good. And the Agile World brought about by Thoughtworks’ Continuous Delivery’s Third Age of software development, I wholeheartedly believe, is Good News.
No comments:
Post a Comment