The Other Agile Development: Thoughtworks and Continuous Delivery

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in agile development, agile new product development, and business agility over the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about agile development today.

The Importance of Continuous Delivery to Agile Development

One of the most enjoyable parts of writing about “the other” in general and “the other agile development” in particular is that it allows me to revisit and go more in-depth on cool new technologies. And this is a cool new technology if ever there was one.

Continuous Delivery, as Thoughtworks presents it, aims to develop, upgrade, and evolve software by constant, incremental bug fixes, changes, and addition of features (note: CD is not to be confused with Continuous Integration, which I hope to cover in a future blog post). The example cited is that of Flickr, the photo sharing site, which is using Continuous Delivery to change its production web site at the rate of ten or more changes per day. Continuous Delivery achieves this rate not only by overlapping development of these changes, but also by modularizing them in small chunks that still “add value” to the end user, as well as by shortening the process from idea to deployment to less than a day in many cases.

Continuous Delivery, therefore, is a logical end point of the whole idea of agile development – and, indeed, agile development processes are the way that Thoughtworks and Flickr choose to achieve this end point. Close, constant interaction with customers/end users is in there; so is the idea of changing directions rapidly, either within each feature’s development process or by a follow-on short process that modifies the original. Operations and development, as well as testing and development, are far more intertwined. The shortness of the process allows such efficiencies as “trunk-based development”, in which the process specifically forbids multi-person parallel development “branches” and thus avoids their inevitable communication and collaboration time, which in a short process turns out to be greater than the time saved by parallelization.

Now, let’s take a really broad view of Continuous Delivery. Unfortunately, blog posts are not great at handling graphs, so I’d like you to visualize in your head a graph with two axes, Features and Time. Over time, in each graph, the user’s need for features in a product and solution tends to go up at a fairly steady rate over time, as do consumer needs in general. What varies is how well the vendor(s) supply those needs.

The old model – as old as markets and technology – of what happened was this: somewhere between each version and the next, the disconnect between what the consumer wants and what the product delivers becomes too great, and at that point the vendor starts developing a new version, based on where the consumer is right now (plus a very minor projection into the future which we’ll ignore). For the most part, during this 6 month-2 year development process, the original spec does not change; so for 6 months to 2 years before another version comes out, no or few new features are added – but meanwhile, consumers start looking for new features on top of what they already wanted. The result is a stair-step progression, in which each new version takes the product only partway to meeting the consumer’s needs at that time, and the space between the user line and the vendor line represents lost sales and consumer frustration. However, since every other vendor is doing the same thing, no harm, no foul.

Now consider agile development. Agile development, remember, is about rapid delivery of incremental time-to-value, plus frequent changes to the spec based on end-user input. What that looks like in our graph, more or less, is a stairstep in which each step takes a shorter amount of time, and each “rise” is much closer to the level of user need at that time – but we’re still a little bit behind.

Conceptually, Continuous Delivery takes that idea almost as far as it can be taken. Now, our graph looks more like a squiggly product line overlapping the user need line. And here’s the key point: it actually goes above the user need line, just by a little, frequently.

How can that be, you say? Well, despite the way we disparage technology-driven products compared to need-driven ones, the fact remains that sometimes techies anticipate consumer needs. The typical way is that implicit in the design of the product are future features that the end user, with his or her tunnel vision on immediate frustrations, will never think of. It is the developer who suggests these to the user, not the other way around, or the developer who puts these in the product “for free”, understanding that since they are a logical technical evolution of the design, the user will see them as less strange and simpler to use. This may sound risky to the development manager, but in point of fact this is a minimal-risk kind of customer anticipation, with minimal impact on customer frustration even if it doesn’t pan out, and maximal impact on the consumer’s image of “the brand that anticipates my needs”.

One more variant on the graph: suppose we are talking about New Product Development (NPD) in general. Well, one thing about agile software development is that software is becoming an increasing part of competitive advantage in “hardware” and “services” across most industries. In other words, the development of a new “hardware” or “services” product now typically includes a fair amount of software, whose development is in-house, outsourced, or assigned to packaged-software vendors. In each of these cases, application of agile development processes produces a “mix” between the traditional-graph vendor line and the agile-development one. Visually, the “steps” between “rises” are no longer flat, but broken into little “mini-rises” and “ministeps” that take you a little closer to the user needs line. Continuous Delivery on software that is half of NPD effectively eliminates about half of the lost sales and customer frustration from the traditional approach.

Do you remember that I said: “CD takes the idea almost as far as it can be taken?” Well, the one thing agile development via CD doesn’t handle is a big jump in consumer needs because the consumer wants one of the features that is being developed over in the next county – “disruptive” technology. For instance, Apple really put a hurt on other user-interface vendors when it tweaked touch-screen technology for the iPhone. However, the software in other cell-phone and computer products at least allowed a more rapid partial response to the threat. So CD isn’t a cure-all – it just comes amazingly close to it. How to “go the final mile” is a discussion for a future blog post.

Let’s summarize: CD is an incredibly cool and incredibly useful technology, both to the vendor and to the consumer, because it results in a major increase in sales for the vendor and a major increase in satisfaction for the consumer. Moreover, because it’s also cheaper than traditional software development, both vendors and consumers see their costs decrease as the vendor’s use of CD in NPD rises (and for the typical business, that’s in addition to the cost savings and competitive advantage from more rapid development of better business-process software). Finally, because their needs are both satisfied and anticipated, customers become far more loyal, reducing business risk drastically.

Really minor nit: I note that CD is often applied strictly to the delivery stage of development. To my mind, extension to the entire process is appropriate, because delivery is the only major development stage (if we exclude operations after delivery) where the agile development methodology today is often applied minimally. In other words, “delivery” can mean one stage or the whole process, and in the real world if you make that one stage agile you are usually making the whole process agile – so it’s a good idea to emphasize the point of the whole exercise by equating continuous completion and continuous delivery.

The Relevance of Thoughtworks to Continuous Delivery

Thoughtworks is one of those smaller consultancies that took the Agile Manifesto’s ideas and ran with them while large development-tool vendors focused on other, ultimately far less effective, “best practices.” I have had my differences with Thoughtworks in the past, but always from a position of respect for an organization that clearly has agile development in its DNA to a greater extent than most. In the case of CD, a quick scan of the first page of a Google search on Continuous Delivery reveals no one else visibly applying it to the extent that Thoughtworks claims to be doing.

Does that mean that Thoughtworks is a stable vendor for the long term? One of the fascinating things about the agile-development market is that the question matters to a much lesser extent than in all previous markets. Look, in the early years some tried SCRUM and some tried extreme programming and some tried half-way solutions like slightly modified spiral programming, but it didn’t matter in the long run: the Wipros of the world have still done almost universally better than the startups focusing on Java or even folks like Cambridge Technology Partners. And that’s because agile development firms are, well, agile. It doesn’t just rub off on developers; some of it rubs off on managers, and strategists, and even, Ghu help us, on CEOs. They focus on changes; so, on average, they evolve more effectively. That’s as true of Thoughtworks as anyone else.

What isn’t true of most other agile development vendors, right now, is that Thoughtworks appears to have a significant edge in experience in the CD extension of agile development. That matters because, as I’ve said, something like Thoughtworks-type CD is the logical endpoint of agile development. So, if you want to get to maximal agile-development benefits sooner rather than later, it certainly seems as if Thoughtworks should be on your short list.

One point here requires elaboration: there is sometimes a misconception that outside providers are selling you agile-development services. That’s at least partially wrong. They are – or should be – fundamentally selling you agile-development training. They will make their money from being ahead of you in experience, and constantly selling you the improvements in agile development that their experience teaches them. Think of them as more like business-strategy consultants, always looking ahead to the new strategic idea and delivering that to you. Believe me, that’s just as valuable as running your data center – and is often more valuable than that.

Thus, Thoughtworks’ advantage in experience is not to be sneezed at. How well it will hold up over time, we will see. However, considering that the bulk of software development, even if it has adopted a modified version of SCRUM en masse, is still typically not very successful in embedding frequent user feedback into the typical project, I would say that Thoughtworks’ edge should last for at least a couple of years – an eon in the timeframe of the agile business.

Potential Uses of Thoughtworks-Type Agile Development for IT

An IT organization must crawl before it can walk; but it should also learn about walking before it tries to do so. That means that if IT is still at the early stages of adopting agile development, it should still apply CD to a “skunkworks” project, and if it doesn’t have such a project, it should create one.

Otherwise, this is not a “targets of opportunity” situation, but rather a “learn and merge” one. IT should bring CD on board as each project is ready for it, no faster, no slower. In my opinion, “ready” means that a project’s development process has (a) adequate process management tools specifically tuned to support agile development, thus allowing it to scale, (b) an adequate “store” of reusable infrastructure software to build on, so that moving to the next incremental feature is not too great a leap, and (c) an attitude from everyone involved that the first thing you do when you get something new is that you make it agile. That’s all. Print and ship.

Well, but today’s CD isn’t adapted to the peculiar needs of my development. Excuse me? You did say your process was agile, didn’t you? Believe me, CD is not only flexible but agile, and if you don’t know the difference, then you need far more help in achieving agility than you realize. What you’re really saying is that you don’t have an agile development process at all, because otherwise it would be straightforward to adapt your methodology to move steadily towards CD – using a vendor just speeds up the process change.

The Bottom Line for IT Buyers

I am always wary of being too enthusiastic about new technologies, because I remember a story in Gore Vidal’s Lincoln. A rich man has a tendency to exaggerate, and hires someone to nudge him at table when he does so. “How was your trip to Egypt?” “Amazing! Why, they have things called pyramids made of pure gold!” Nudge. “And they go a mile high!” Hard stamp on his foot, just as someone asks, “How wide?” He answers, in agony, “About a foot.” I worry that while any given technology’s value may seem a mile high, its real-world application will be about a foot wide.

That said, I really do see Continuous Delivery as a cool new technology that will have an impact, eventually, that will be a mile high and globe-wide. Here’s what I said in a previous post:

[[The real-world success of Continuous Delivery, I assert, signals a Third Age, in which software development is not only fast in aggregate, but also fast in unitary terms – so fast as to make the process of upgrade of a unitary application by feature additions and changes seem “continuous”. Because of the Second Age, software is now pervasive in products and services. Add the new capabilities, and all software-infused products/services -- all products/services – start changing constantly, to the point where we start viewing continuous product change as natural. Products and services that are fundamentally dynamic, not successions of static versions, are a fundamental, massive change to the global economy.

But it goes even further. These Continuous-Delivery product changes also more closely track changes in end user needs. They also increase the chances of success of introductions of the “new, new thing” in technology that are vital to a thriving, growing global economy, because those introductions are based on an understanding of end user needs at this precise moment in time, not two years ago. According to my definition of agility – rapid, effective reactive and proactive changes – they make products and services truly agile. The new world of Continuous Delivery is not just an almost completely dynamic world. It is an almost Agile World. The only un-agile parts are the rest of the company processes besides software development that continue, behind the scenes of rapidly changing products, to patch up fundamentally un-agile approaches in the same old ways.]]

But you don’t need to know about that. What you need to know is that for every IT organization that appreciates agile development, kicking the tires or adopting CD is a good idea, right now. I don’t even have to say it’s necessary, because that’s not the way an agile organization operates.

As for Thoughtworks, here’s what I think IT’s attitude should be. I have often trashed Sun for an ad that said “We built the Internet. Let us build your Internet.” I knew, based on personal experience, that this claim was, to say the least, exaggerated. Well, if Thoughtworks came to your door with a pitch that said “We built Continuous Delivery. Let us build your Continuous Delivery,” I would not only not trash them, I would encourage you to believe them, and consider doing as they request. The pyramid is that high. The materials with which it is built are that valuable. And my foot is still intact.