The Transition From Agile Development to the Agile Organization Is Beginning to Happen

Disclaimer:  I am now retired, and am therefore no longer an expert on anything.  This blog post presents only my opinions, and anything in it should not be relied on.

Recently, new publications from CA Technologies via Techtarget arrived in my in-box.  The surveys mentioned in them confirmed to me that the agile organization or agile business is beginning to be a Real Thing, not just vendor hype. 

Five years ago, I wrote but did not publish a book on the evidence of agile development’s benefits and implications for creation of a truly agile business or other organization.  Now, it appears not only that the theoretical foundation has been laid for implementation at least of agile processes in every major department of the typical business, but also that a significant subset of businesses now think that they have at implemented agile according to that foundation across pretty much all of the enterprise – yes, apparently in a few cases including legal (what does legal-department agility mean?  I have no idea, yet).

So what are the details of this evidence?  And what benefits of an agile organization seem to be proving themselves?

The Solid Foundation of Agile-Development Benefits

It is now approaching a truism that agile development delivers benefits compared to traditional software-development approaches.  My own survey during my brief re-up at Aberdeen Group 9 years ago suggested improvements in the 20-30 % range for project cost and speed, product quality, and customer satisfaction (with the obvious implication that it also decreased product-development risk by around that amount, as Standish Group studies also showed).  One striking fact was that it was achieving comparable results when compared to traditional approaches focused on cost and/or quality.

One CA pub (“Discover the Benefits of Agile:  The Business Case for a New Way to Work) extends these findings to agile development plus agile project management.  It says that a “summary of research” finds that agile delivered 29 % improvements in cost, 50 % in quality, 97 % in “productivity” (something like my “speed”), 400 % (!) in customer satisfaction, 470 (!!) in ROI (a proxy for revenue and profit), compared to the “least effective” traditional approaches. 

While this may sound like cherry-picking, my research showed that the most effective traditional approaches were not that much better than the least effective ones.  So I view this CA-cited result as the “practice effect”:  experience with agile development has actually increased its advantage over all traditional approaches – in the case of customer satisfaction and profitability, by really large amounts.

The Theoretical Case For Business Agility

Note, as I did back when I did my survey, that agile development often delivers benefits that show up in the top and bottom line of the success-critical-software-developing business, even before the rest of the organization attempts to go agile.  So why would it be important to go the rest of the way and make most or all of the organization agile?

The CA pub “The State of Business Agility 2017” plus my own work suggest that potential benefits of “agile beyond software development” fall into three areas: 

1.      Hard-nosed top and bottom line benefits:  That is, effects on revenue, cost, and margin (profit).  For example, better “innovation management” via agile project management goes to the top line and eventually to the bottom line, and in some cases can be measured.

2.      “Fuzzy” corporate benefits, including competitive advantage, quality, customer satisfaction, speed to act and change strategies, and reduction in negative risks (e.g., project or IT-infrastructure failures) and “fire drills”.

3.      “Synergy” benefits stemming from most of the corporation being on the same “agile page” with coordinated and communicating agile processes, including better collaboration, better/faster employee strategy buy-in, better employee satisfaction through better corporate information, and better “alignment between strategy and execution.”

The results of the CA business-agility pub survey suggest that most respondents understand many but not all of these potential benefits before they take the first step towards the agile business.  I would guess, in particular, that they don’t realize the possible positive effects on combating “existential risks”, such as security breaches or physical IT-infrastructure destruction, as well as the effects on employee satisfaction and better strategy-execution alignment.

The Extent of Agile Organizations and Their Realized Benefits

Before I begin, I should note two caveats about the CA-reported results.  The first is that respondents are in effect self-selected to be farther along in agile-organization implementation and more positive about it.  These are, if you will, among the “best and the brightest.”  So actual agile-organization implementations “on the ground” are certainly far less than the survey suggests.

Second, CA’s definition of “agile” leaves out an important component.  CA’s project-management focus makes part of its definition of agility to be holding time and cost in a project constant while varying project scope.  What that really means is that CA de-emphasizes the ability to make major changes in project aims at any point in the project.  In the agile-organization survey, this means an entire lack of focus on the ability to incrementally (and bottom-up!) change a strategy rather than just roll out a whole new one every few years.  And yet, “more agile” strategy change is at the heart of agility’s meaning and is business agility’s largest long-term potential benefit.

How far are we towards the agile organization?  By some CA-cited estimates, 83% of all businesses have the first agile-development step at least in their plans, and a majority of IT projects are now “agile-centric.”  Bearing in mind caveat (1) above, I note that 22% of CA-survey respondents say they are just focused on extending “agile” to IT as a whole, 17% are also working on a plan for full business agility, 19% have gotten as far as organizational meetings to determine agility-implementation goals, and 39% are “well underway” with rollout.  Ignoring the question about “momentum” in partial departmental implementations for a moment, I also note that 47 % say IT is agile, 36% that Project Management is, and marketing, R&D, operations/manufacturing (are they counting lean methodologies as agile?), and sales (!) are a few percentage points lower. 

Getting back to partial implementation, service/support seems to be the “new frontier.”  Surprisingly, corporate communications/PR is among the laggards even in implementation, along with accounting/finance, HR, and legal.  What I find interesting about this list is that accounting and legal are even in the conversation, indicating that people really are planning for some degree of “agile” in them.  And, of course, the CEO’s agility isn’t even in the survey – as I said 9 years ago, the CEO is likely to be the last person in the organization to “go agile.”  Long discussion, not relevant here.

How about benefits?  In the hard-nosed category, agile organizations increase revenue 37% faster and deliver 30% greater profit (an outside survey).  For the rest of benefits, there is far less concrete evidence – the CA business-agility survey apparently did not ask what business benefits respondents had already seen from their efforts.  What we can deduce is that most of the 39% of respondents who said they were “well underway” believe that they are already achieving the benefits they already understand, including most of the “fuzzy” and “synergy” benefits cited above.

Implications:  The Cat Is In The Details

At this point, I would ordinarily say to you the reader that you should move towards implementing an agile business/organization, bearing in mind that “the devil is in the details.”  Specifically, the CA surveys note that the complexity of the business and cultural/political opposition are key (and the usual) problems in implementation.  And, indeed, this would be a useful thing to know.

However, I also want to emphasize that there is a “cat” in the details of implementation:  a kind of Schrodinger’s Cat.  In quantum physics, as I understand it, different states of basic particles (e.g., exists, doesn’t exist) are entangled until we disentangle them (e.g., by “opening the box” and measuring them).  Schrodinger imagined entangled states of “cat inside the box/no cat inside the box”, so that we wouldn’t know whether Schrodinger’s Cat existed until we opened the box.  In the same way, we not only don’t know what and how much in the way of benefits we get until we examine the implementation details, we won’t know just how really agile we are until we “open the box.”

Why does that matter?  Because, as I have noted above, the really big long-term potential benefit of business agility, is, well, agility:  The ability to turn strategically, instantly, on a dime and ensure the business’ long-term existence, as well as comparative success, much more effectively.  Just because some departments seem to be delivering greater benefits right now, that doesn’t mean you have built the basic infrastructure to turn on a dime.

And so, the cat is in the details.  Please open the box by testing whether your implementation details allow you to change overall business strategies fast, and then, if there is no cat there, what should you do?

Why, change your agile-implementation strategy, of course.  Preferably fast.

Was Grant Agile?

As I write posts and white papers on agile software development and business agility, I find myself surprisingly often reminded of sayings by or about General Grant during the Civil War.  I am not recommending Grant as a model of agile thinking – I think we’ve had enough books about Jesus Christ being an ad man and Attilla the Hun being a master business strategist (although if you want repulsive characters who were good generals, Subodai the Mongol would be a much better choice).  I am, however, suggesting that Grant showed some interesting “agile” traits.

Let’s start with a remark by someone when Grant showed up to take over in Tennessee (I am paraphrasing): Grant didn’t come in with a lot of show and effort. And yet, when he came on the scene, everything seemed to start working like clockwork.  He seemed to have a plan and everything he did played into that plan.

The plan, by the way, was mostly not Grant’s. The idea of attacking downstream positions to free up the Union lines was General Thomas’.  The spontaneous attack on Southern positions at the end was not in anyone’s plan.  However, Grant adopted and adapted to realities on the ground and changes in plan immediately.  So, I suggest, Grant had (a) ability to reach out to a “customer” and change plans based on feedback – reactive agility – and (b) ability not just to adapt to unexpected change but also to incorporate it immediately in an overall strategy.

Now let’s reach back a little earlier, to the attack on Vicksburg.  Catton in his superbly written history states that Grant during the winter was trying various methods to get downstream, and while almost all of these didn’t pan out, the constant trying of different things had the effect of bewildering his opponent Pemberton so that Pemberton perceived himself as being under potential attack from many directions at once. As a result, when the real attack happened, Pemberton was constantly one step behind.  To me, that’s a bit like evolving the nature of a product constantly during development, compared to a competitor sticking to a fixed plan. 

However – and, to me, here’s a key point – Grant was proactive rather than reactive.  Not only was he constantly thinking about attacking; he was constantly thinking about changes in his attacks.  At Vicksburg, he had already committed to operating without a supply line once across the river; he then changed his plans in order to attack Jackson once he saw an opportunity, and then turned around once Pemberton finally came out and changed his attack plans again in order to attack and defeat Pemberton “in detail”.  So Grant could be proactively agile.

That Grant thought this way is apparently confirmed by an incident later in the war, at the Wilderness.  Lee was pressing Grant hard, and staff officers accustomed to previous Union commanders were panicked over whether Lee could destroy the Union army.  Grant blew up at them:  some of them seemed to think, he said, that Lee was going to do a giant somersault and attack the army from behind.  Instead of concentrating on what Lee was going to do to them, Grant said, they should concentrate on what they were going to do to Lee.

Now, I can’t say that Grant was completely agile – because he apparently sometimes did not spend enough time taking care of potential counter-moves by his opponents.  Thus, in Shiloh, he chose to hasten the arrival of reinforcements for an attack rather than be on the scene getting folks dug in to receive one before he had a chance to attack.

One final thought – and here I admit I’m really reaching.  An interesting characteristic of Grant’s attacks was that, given the chance, Grant would attack “up the middle”, but not to split the enemy’s lines, as Lee attempted to do with Pickett’s charge at Gettysburg. Rather, the main purpose was to hold the enemy in place as he then detached an equal force to hit the enemy on or behind one end, like a jab immediately followed by a roundhouse swing. If the enemy then detached too many to deal with problems on the end, the up-the-middle attack could then succeed, as in Tennessee. There is a vague analogy here to agile new-product development, in which one “fixes the competitor in place” with a strong product in an existing market, and then “hits the competitor on the end” with major innovative features, as the iPhone did.

I conclude that Grant was not a modern agile strategist, not fully.  However, compared to others in the Civil War, and many generals since, he thought in a much more agile way.  And note that it had nothing to do with how hard he worked.  He spent less effort than most, and achieved much more.  He was not efficient, he was effective.  As Lincoln once said about Grant, I can’t spare this general, he fights.

Data Virtualization: Users Grok What I Said 10 Years Ago

Ten years ago I put out the first EII (now data virtualization) report.  In it I said:

• ·         The value of DV is both in its being a database veneer across disparate databases, and in its discovery and storage of enterprise-wide global metadata
• ·         DV can be used for querying and updates
• ·         DV is of potential value to users and developers and administrators. Users see a far wider array of data, and new data sources are added more quickly/semi-automatically. Developers have “one database API to choke.” Administrators can use it to manage multiple databases/data stores at a time, for cost savings
• ·         DV can be great for mergers, metadata standardization, and as a complement to existing enterprise information architectures (obviously, including data warehousing)
• ·         DV is a “Swiss army knife” that can be used in any database-related IT project
• ·         DV is strategic, with effects not just on IT costs but also on corporate flexibility and speed to implement new products (I didn’t have the concept of business agility then)
• ·         DV can give you better access to information outside the business.

•  DV can serve as the “glue” of an enterprise information architecture.

I’m at DV Day 2012 (Composite Software) in NYC. Today, for the first time, I have heard not just vendors talking about implementing these things, but users actually doing them – global metadata, user self-service access to the full range of corporate data, development using SQL access, use for updating in operational databases, use for administration of at least the metadata management of multiple databases as well as archiving administrative tasks, use in mergers, use in “data standardization”, use as an equal partner with data warehousing, use in just about every database-related IT project, selling by as strategic to the business with citations of impacts on the bottom line through strategic products as well as “business agility”, use to access extra-enterprise Big Data across multiple clouds, and use as the framework of an enterprise information architecture.

I just wanted to say, on a personal note, that this is what makes being an analyst worthwhile.  Not that I was right 10 years ago. That, for once, by the efforts of everyone pulling together to make the point and elaborate the details, we have managed to get the world to implement a wonderful idea. I firmly believed that, at least in some small way, the world would be better off if we implemented DV. Now, it’s clear that that’s true, and that DV is unstoppable.

And so, I just wanted to take a moment to celebrate.

Business Agility and Business Theory in Sloan Management Review

One of the things that has been striking to me about the expanding field of business agility theory is how many things it can be applied to. I thought it might be interesting to share business agility theory applied to the major articles in a recent issue of Sloan Management Review. My apologies to the authors if I have distorted their message in any way.

Getting the Best From Corporate Functions

Campbell, Kunisch, and Muller-Stevens suggest that corporate functions “don’t get enough strategic direction from the CEO.” They suggest a four-step approach to fixing this:

1. For each function, define key sources of corporation value-add.
2. Review function strategies annually.
3. Develop a corporate initiatives matrix so function execs can see how their efforts relate to those of other functions and operating units.
4. Break out corporate services so they focus on unit support rather than control.

From an agility point of view, several things about this strike one. First, in an agile world, all corporate functions should be about service rather than control. Or, more exactly, they should view units (not the CEO) as their immediate customer and they should be mindful of the ultimate customer (the business’ actual customers). This is not to say that no “control” goes on; it just says that at the same time, the function and the unit are exchanging information, the unit seeking information to perform better, the function seeking information (and not just “deviance” information) in order to have the overall corporation perform better.

Second, why is the CEO getting in the middle of this in the first place? Hasn’t the CEO got enough to do? Is it really necessary to impose a process where the function is periodically checking back with the CEO to see if the function is doing the right thing? Shouldn’t we be instead encouraging better understanding of the ultimate customer in each function, and how to integrate with other functions and the unit to respond to and if possible anticipate the needs of the customer? I have been watching the field of agile marketing, and they propose an integrating function for marketers that allows functions and units to interact in just this way on a per-product basis. Since marketing typically does have something similar to the CEO’s strategic vision, plus somewhat of a direct line to the ultimate customer, that sounds like a better model to me.

On the plus side, this approach has the merit of at least establishing a closer link to the ultimate customer via the CEO. On balance, in terms of moving towards business agility, this is likely to be a plus. Business agility theory suggests that any other benefits are primarily a side-effect of that increased agility.

Investments Aimed at Revenue Do Better Than Those Focusing on Cost

Mithas, Tafti, Bardhan, and Goh report a study that suggests that IT investments (especially those aimed at new-revenue generation) offer better return than ad or R&D spending -- and that this return superiority is increasing over time. They conclude that corporate should favor IT spend aimed at new revenue over spend focused on costs.

This finding sounds suspiciously like my analysis (based on an Aberdeen worldwide survey) in 2009 of agile software development. Software continues to become integral to new-product differentiation in more and more industries. Based on the returns of switching to agile development in the survey, we should expect increasing agile software development to increase profits and decrease costs in new-product development, and only decrease costs in the other two categories of the business (risk management aka disaster handling, and ongoing operations). Therefore, permanently improving new-product development agility in IT will increase profitability more than operational initiatives (typically aimed at reducing costs) and risk management initiatives (typically for regulatory compliance or to reduce the negative risk of disaster). And, by the way, agility in whatever area tends to reduce the negative risk of disaster and increase the positive risk of unexpected additional earnings or “free” cost reductions.

Note, by the way, that business-process spending, if it connects to new-product development and the ultimate customer, can have a positive impact equal to that of investing in new-product development agility.

The study itself covered the years 1998-2003, well before agile development was present in significant amounts anywhere. My feeling is that what we are seeing is identification of one of the ways agility works: it maximizes the effectiveness of new-product investment that is already paying off in other ways, and therefore, subtly, shifts ever more investment into the “biggest bang for the buck.”

Growing Into New Countries Case by Case

Bingham and Davis find that companies use both direct and indirect approaches to the countries they enter, as well as starting off “soloing” or “seeding” by using local companies for part of the initial entry. They conclude that key determinants of success are the company’s initial skills in each, plus their ability to adapt their approach to a different requirement of the local market. They recommend, among other things, pre-training in the new countries and learning via local companies.

From a business agility point of view, the reaction would be, I suspect, this is news? Something is amiss in a globalizing company’s agility if they are not already incenting to focus on the next country entry, and the next, and changing to meet the needs of each. Instead of swinging the entire corporate superstructure ponderously around when the new target proves difficult, the corporation at every level should be ready to push their antennae out to the ultimate customer the moment the decision is made to move, and to achieve it by the path of least resistance – whether that be soloing or seeding – always remembering to keep a channel open past the local company to the ultimate customer. Business agility theory suggests that, usually, focusing on the change itself, rather than increasing profits or cutting risks with no change, leads to the most profit and the least risk.

Patent Agility

Kappos and Graham argue that common patents should be applied worldwide in order to speed patent issuance. By all means; such standards will simply speed issuance, not decrease patent quality, more or less. But agility theory sounds a cautionary note here: increased speed without increased effectiveness only provides one-quarter of agility’s benefits, at best, and at worst can actually double down investment on an un-agile process.

I would suggest that today’s patent process is over-balanced in the direction of the intermediate customer, the business, rather than the ultimate customer, the business’ customer, and this will simply exacerbate the problem – the problem being that there needs to be a better mechanism to allow patent offices to move in sync with both sets of customer. A half-measure may be to reward “offensive” (moving towards the future) patent applications more than “defensive” ones. A more proactive one might be to vary patent time frames as the balance shifts between the business’ needs and the ultimate consumer’s needs.

Major Change Requires Major Skepticism?

Johnson, Yip, and Hensmans study companies achieving successful major change (forced and unforced) and find that the best at it have a tradition of challenging the status quo “constructively” (a very vague term) and getting around corporate rigidities by establishing informal channels. Their specific recommendations include some odd ones, including non-intuitive promotions to “spread the worldview” of CEO successors and “encouraging tension.”

This is an area where business agility theory has no clear answers as of yet. System dynamics suggests that, no matter how well patched, corporate processes inevitably fail, sooner or later, catastrophically. Agility theory says that the time to failure can be extended by embedding constant incremental improvements in these processes; but it does not say that the time to failure can be extended indefinitely. It might possibly be that less-agile urges towards major change will indeed be better for the business overall by ensuring more frequent avoidance of this type of failure than more-agile urges towards incremental change that simply decrease this type of failure’s frequency.

What we can say, however, is that steps like “ensuring that decision making allows for [constructive] dissent” do very little if anything for business agility. In the agile business, it’s rarely a matter of “dissenting from company policy”; it’s more a matter among choosing among options for change, with the customer being often the ultimate arbiter. The customer doesn’t care whether you, the CEO, set a business policy or not; they want what they want, and your business policy is simply a constantly shifting guideline on how to respond, a guideline that you want your organization itself to evolve, rather than constantly feeding questions back to you.

Turn the Company’s Beliefs Into Strategic Success?

Goddard, Birkinshaw, and Eccles argue that the beliefs of its employees are a key company differentiator, and therefore should be consciously a part of the company’s strategy. They then suggest ways of evolving these “belief strategies” as needed via such tactics as discovering, discarding, marooning, and neutralizing (the details aren’t pertinent to this discussion). In a sense, this article puts the psychology of the company on the analyst’s couch, identifying the company’s early experiences and their manifestation in the company’s subconscious, and suggesting ways to make these conscious and so understand how to move forward from a frozen neurosis.

Business agility theory sounds an extremely cautionary note about this. One of the real dangers of focusing internally rather than on the ultimate consumer is that the neurosis will turn out to be narcissism: the belief that one’s image of the customer – an image reflecting rather one’s self – is the reality of the customer. Raiding other company’s “belief strategies” is a rather poor patch-up for this: other companies typically have the same problem.
This concern is particularly appropriate when you consider that a recent IBM CMO survey showed that the gap between the way the CEO and CMO viewed what the company should be and the actuality of the way its lower-level employees thought it was, was huge. In essence, the employees saw the company acting, in all sorts of ways, counter to what it wanted to send out as the corporate brand.

What agility theory would suggest instead, I would guess, is that businesses (a) reality-test their corporate function’s belief systems against employee and customer perceptions before they do anything about them, and (b) consider how agile the “belief strategies” remaining after this step are. In other words, how do we set up a process so that we are agile enough to reach the point of being in sync with a new belief strategy, and how do we evolve it agilely after that? If the organization does not do something like this, I would guess, a “new belief strategy” may in fact make the company even less agile, by reinforcing corporate narcissism.

New Business Model or More Agile Customer Interaction Process?

Amit and Zott argue that developing a new business model (which they define as a new system of connecting with the customer) has a much better return on investment today than new-product development. They then go through ways of creating the best business model (not pertinent here), and close by suggesting some fascinating questions (excerpted) that the business should ask itself when creating a new business model:

1. “What perceived needs can be satisfied through the business model design?
2. Who should perform each of the activities in the business model, the company, a partner, or the customer?
3. How is value created through the novel business model for each of the participants?”

The reason these questions are fascinating to me is that they create a new process that specifically requires that the business try to put itself into the mind of the customer (or partner) and ask what value they see. That is fundamentally different from imagining that the customer is like yourself; and it is fundamentally more agile. Bravo to the authors for intuitively understanding the power of such an approach.

The other key point they are making, however, is undercut by their suggested use case – electronic publishing. It seems to me as I examine that use case that what they mean by new business model development and what I mean by new-product development are, at the least, extremely difficult to divide into two separate categories. What they would call a new electronic publishing business model, I think, I would call new electronic-publishing product development that includes a new business model. Had one started with the product and evolved agilely to the new business model that it implies, what would be the difference in the end? And that is what agile new-product development could, if enfolded in an agile cross-product business strategy, have done.

How To Never Become a Better Leader

It was as I read this article that the idea that agile is better struck me with particular force. Toegel and Barsoux argue that incipient leaders should “recognize and manage their strongest tendencies.” Don’t be too composed; don’t be too excited. Don’t be too much of an extrovert; don’t be too introverted. Don’t innovate wildly in all directions; don’t be too conventional. Don’t be too tough-minded dealing with others; don’t be too considerate. Don’t be too perfectionistic and detail-oriented in deciding something; don’t just make decisions immediately. Understand when you are leaning too much one way or another, and then correct.

OK, rather than critique the authors point by point, let me just say what I think business agility would recommend, and recommend with great force: teach the leader how to think agile. Whatever the flaw, it has a counteracting force. Too composed? It’s not a question of being composed, it’s a question of being focused on change. It’s not a question of whether you panic on hearing bad news, it’s a question of whether you immediately pass on to a positive plan to not only deal with the unwelcome change, but make hay out of it. Too introspective? Focusing on change means reaching out to the ultimate customer and others to get the next feedback as you move toward change in a very structured format. Your need to be alone will still be satisfied; your increasing interaction with others will be encouraged and rewarded.

Too innovative? But if you innovate too much before you see the customer, most of it is wasted. Too inclined to make decisions quickly? Then you’ll just have to keep undoing them when you get your feedback from the customer – which will make the decision-feedback loop for you shorter as you try to solve the problem by seeing the customer faster.

The final dichotomy is tough vs. tender; and it gives me another opportunity for talking about a technique I learned back at Prime to ensure the very best agile customer interaction. It’s called “reflection” – no, not that reflection. What happens is that when someone says something to you, you reflect back to them first of all your understanding of what they were trying to say – no shortcuts, no imposition of what you wanted them to say, put yourself in their shoes saying it, and preferably with the feelings they had when they were saying it – and only then say something in direct response to what they said. It’s awkward at first, but over time you learn what works and can telescope your response. Just never try to skip a step at first – because you will inevitably put your own spin on what they are saying, as you have been doing all your life.

Now think about either the tough or tender, but in each case agile, leader applying this technique. There’s no point in being brutal, because what really matters is the change you want to accomplish. Instead, you understand the person’s concerns via reflection, and show them how the change is safer for them, and then expect them to get to it, because they like safe change. Likewise, if you are tender, the very best thing for the person is to do safe change along with you, and it’s a very positive message to give. What feelings to hurt? You’re offering them a way to succeed, and showing that they get the best marks by collaborating with you as you both seek out the best, as yet undetermined, safe change.

Have you noticed the agile way is not “don’t do this, don’t do this”? It’s “make this change, it’s going to be the best thing you could hope for.” Best of all, that’s almost certainly true, if they last over the long run, and probably true in the short run.

Project Baron or Agile Federation?

I’m just going to comment on this one briefly. Gann, Salter, Dodgson, and Phillips argue that organizations of several little very agile groups off doing generally the same things must make sure that corporate serves their interests and integrates them effectively according to the degree they are presently tightly or loosely interconnected. My agile thought here is that the “corporate-barony” dichotomy here is artificial. These are relatively agile groups; and the corporate “brand” should be an evolving umbrella, not a straitjacket constraining each barony to integrate effectively with the others. The “inefficiency” when some groups do not share corporate resources with others is far less concerning than the remaining lack of agility because these groups are not tapping each other for additional sources of insight about their customers. Time to develop your own contacts with the whole gamut of customers to figure out just how far short they are failing, and give them feedback. That’s what corporate is really selling in these situations: cross-pollination. Let’s decrease the control emphasis and increase the corporate information – not resource – value prop to the barony.

Lego as a Model of Involving Customers in Product Development

Antorini, Muniz, and Askildsen hold up the Lego Group as a model of how to save the company by getting your enthusiastic customers involved in designing the extensions they want, and feeding them back into your new product development. From an agile point of view, what’s not to like? Just one thing. It appears from the description that this is reactive agility. Customer comes up with an idea, the company decides if it’s good, if so flows the part that applies into new-product development. Again, just one problem: how does the company get ahead of the customer?

Because, remember, new-product development at Lego is still not like agile software development. It’s more like “hybrid” open-source plus internal development. My studies suggest that because the areas it can improve are limited and it’s not proactively agile, this approach typically returns about one-half or less of the benefits of the average agile development organization.

So how would Lego become more agile? By making its new-product development more agile. The remark by one exec is a dead giveaway: we “hold a dialog with the customer.” It’s clear that it’s a tilted dialog: the customer tells Lego a lot, Lego doesn’t counter every customer suggestion with a suggestion for the customer. Instead, he or she has to wait for the next product release, and hope. What you’d like is to unleash the creativity of the development team in making the new products “orthogonal” – simple and powerful to use – and speed up responsiveness to all customers, not just the enthusiasts, by more frequent incremental product releases.

Final Thought

Rather than go through the last article, which doesn’t add anything in terms of agile theory, I thought I’d make an overall comment. It seems to me that many of these researchers were perhaps not thinking in as agile a way as they might. For example, many of these insights are based on rigid surveys and existing data. Certainly that’s understandable – obviously, agile theory is not widely applied anywhere except in agile software development.

Still, it does make me understand better why I like the customer survey/interview techniques I stumbled on 15 years ago. They let the customer provide an insight into his or her wants or needs, by telling about the product I’m surveying about as a story of a solution to his or her needs. They are open-ended, allowing the customer to take the questionnaire in a direction not contemplated initially, and allowing me to adjust agilely to that change in direction. They specifically ask for important information outside the bounds of the questionnaire. In effect, my research is a bit more agile because I am acting more in an agile manner.

All those who are researching business value-add, might this approach be of help in understanding your work through the lens of business agility? Or do you have agile suggestions of your own? I don’t know the answer. I’m just trying to be agile.

A Draft Agile Business Manifesto (& Agile Marketing Manifesto)

More than a decade ago, the experience of several far-seeing developers drove an Agile Development Manifesto that has more than stood the test of time. Now, finally, other areas of the business are considering equivalent follow-on manifestoes of their own – in particular, an Agile Marketing Manifesto. In order to insure that the fundamental insights of the original Agile Manifesto are not lost, I would like to propose a similar set of principles for the agile business in general – principles within which efforts such as an agile marketing manifesto should be expected to fit.

Because “the old ways” are much more entrenched in the business as a whole than in the software development arm of an organization, I set these principles up in opposition to what I view as some of the key principles of business as we have known it – or at least the ones that the truly agile business contradicts. And, of course, I have a bit of fun in doing so.

I also believe that these principles will also serve for a draft agile marketing manifesto. The reasons why will become apparent later.

Agile Organization Goals: Fast, Effective Change
The old principle 1 (monetary goal): In the long run, the most successful are most likely to survive.

Agile principle 1 (fast, effective change goal): The successful thrive; the agile survive. In the long run, survive beats thrive.

Remember that in an agile organization, the focus is on maximizing the speed and effectiveness of change, not on maximizing short-run or long-run success as defined in monetary terms. Experience shows that in many development organizations, agile teams focused on speed of customer-coordinated change rather than cutting costs, improving quality, or speeding up development do better in the short and long run in monetary terms (costs, revenues, risk profile). This principle asserts that even if an organization is more successful in the short run by focusing on maximizing profit, it is less likely to survive in the long run, and less successful if it does survive, than the truly agile organization.

Agile Coopetition
The old principle 2: What does not kill you makes you stronger.

Agile principle 2 (coopetition): What tries to kill us and fails makes us weaker and more agile – and that’s good.

Recent research suggests that humans benefit most from eating foods that give us “moderate poisoning.” That is, they do not kill us, but they do weaken some existing systems. In order to adapt to these weaknesses, the body grows new capabilities – physical and mental – that let us handle a wider range of situations – that make us “survive by fitting” a wider range of environments. By contrast, foods with no poisoning capabilities offer much less benefit, and foods that cripple us offer no benefit at all.

There is a reasonable analogy here with competition vs. what is now called coopetition. The non-agile business assumes that one “wins” in the long term by crushing the competition, and that therefore the company emerges from the process stronger. This is doubtful; in the long-run, over-success in competition leads to monopoly behavior that decreases innovation and makes “crossing the chasm” much more difficult.

The agile business seeks to create “safe” competition by combining competition with cooperation – the same business that competes with you in one area cooperates with you in another area, and neither side expects to win overall in the long run. Rather, the relative weakness in one area of one of the two vendors will lead either to new capabilities or to redefinition of its focus on that side, using the increasing weakness of a particular area as an indicator of the need to move to something different. In the short run, that competition makes the agile vendor weaker. In the long run, the safety of that “competition” leads to increased agility, which leads to long-term survival and success. Meanwhile, cooperation in another area expands the solutions that the firm can offer and provides additional customer feedback.

Agile Culture: Happiness is Customer-Coordinated Change
The old principle 3: Greed is good.

Agile principle 3: Greed is bad. Customer-coordinated change is good.

Immortalized as the “invisible hand”, the pure form of the old principle 3 asserted that focusing on maximizing profit was good for the individual employee, good for the company, good for the consumer, and ultimately good for the economy and society. The agile principle asserts that focusing on speed and effectiveness of “safe” change is better for the individual employee, better for the consumer, and ultimately better for the economy and society in both short-term and long-term monetary and “social welfare” terms.

This is not a simple matter of “doing well by doing good.” Rather, the agile business recognizes that effective change must be constantly and firmly grounded in consumer reality, and that treating the consumer as an infinitely malleable toy to be squeezed of every last drop of money has bad long-term effects not only on survivability but also on corporate culture and thence on corporate success and “social welfare.”

Agile Strategy: Focus on Who
The old principle 4: If you don’t know where you’re going, any road will take you there.

Agile principle 4: If you don’t know who you will meet, you don’t know where you’re going.

As shown in such works as “Marketing Myopia”, the focus of a traditional firm is to accept the given of a global mass “market” as it is now, and try to determine just what market within that global one fits the firm best, and how the firm can steadily increase its profits within and by extensions of that market. The agile strategist focuses on a global market full of individual consumers with common lifecycles and fine-grained differences, and seeks to capture as much as possible of the lifecycle and time allotment of the largest group of such consumers as their needs evolve over time. When the agile firm talks about “continuous delivery” of what the customer wants, that is what it means. The traditional firm focuses on the visible, static market of an undifferentiated mass of needs; the agile firm focuses on understanding and meeting the inevitably changing individual needs of each of a class of individual consumers.

The agile firm believes that such a strategy gives it two key advantages over the traditional firm: it grounds the agile firm in reality rather than its own wishes about what the customer “should” be like in order to maximize firm profit, and it focuses the agile firm on striving toward the most effective kind of long-run strategy: cradle-to-grave, comprehensive, open-ended customer support.

Agile Target Market: Learners
The old principle 5: People want solutions

The new principle 5: People buy emotions

Agile principle 5: People buy learning

This agile principle can be seen as a corrective to the extremes of the old and new principles. It is true that in the short run, people will go out and buy a solution for an immediate problem, and that given extra money with which to buy things, people will then choose a solution that makes them feel better, or even an “emotional trigger” without any usefulness at all. However, in the long run, people usually continue buying in a particular area because it continues to relate to the ongoing reality of their lifecycle. Moreover, the best kind of customer buys because he or she enjoys the open-ended pleasure of constantly learning more in a “safe” (usually successful) way.

The alternative customer is the addict: one that develops such a need for an offering that the addict is willing to continue to spend on it forever, as it becomes less and less useful. As industries like tobacco and alcohol have shown, the addict can be a successful short-term market. However, firms in such industries tend to be unusually un-agile – reflecting their customers. Moreover, these industries react to the inevitable challenges of evolving customer needs by attempting to “game the system” and prevent these changes from happening – increasing the costs to society as a whole, as well as the impact to the firm of the inevitable failure to prevent change. The tobacco and alcohol industries are among the least successful over the last 60 years. By contrast, the computer software industry, which ranks among the most agile, is close to the most successful, and software companies that focus on the consumer rather than the business more successful still (see agile principle 4).

There is another implication of agile principle 5: the agile business should never simply react to the customer, but rather constantly alternate getting ahead of the customer and reacting to the customer. Helping a person to learn forever means constantly generating new, follow-on, useful things for the customer to learn, as the customer moves through the stages of a life. The customer, in return, tells you which of these innovations is desired and what his or her own innovations are.

This Is Also A Draft Agile Marketing Manifesto
The reason that these principles apply verbatim to agile marketing, I believe, is because it is probable that in the agile business, marketing will become a “loosely-coupled CEO”. That is, agile marketing’s primary job will be to integrate the constant iteration of new products into a long-term, effective business strategy.

This is not to say that this is marketing’s present job, nor that those trying to achieve agile marketing do integration very well right now. Rather, it says that marketing is the only existing business function that can carry out this necessary task effectively in the agile organization. The other functions are too focused on particular parts of the overall business’ agile process. The CEO simply does not have the capability of handling all the integrating tasks that must be done.

Rather, individual marketers, empowered by their unique focus on long-term customer need evolution, must connect and coordinate the enthusiastically wandering dogs of the other agile business functions. Meanwhile, the CEO imposes a single long-term personality on the agile marketers’ work, giving the results a touch of individuality that appeals to the individual consumer – much the same role that a great soloist plays in giving an old war-horse of a classic musical piece new life, or the kind of role that Steve Jobs occasionally played toward the end.

Thus, agile marketing in the long run will be effectively carrying out the agile organization’s principles – but in an agile way, in which the customers are equally the other agile business functions and the firm’s actual customers.

The Long Road Ahead
I say this is a draft manifesto; but, to me, it is also a platform with which to challenge other upcoming manifestos. I believe that if another manifesto contradicts or ignores these principles, there may very well be fundamental problems with that manifesto. If so, these principles give me (and, I hope, others) a basis for criticizing and, if necessary, rejecting such a manifesto. We are trying to create manifestoes at a much earlier – dangerously early – stage of our understanding of business agility. It may be OK not to get a manifesto right; but it is very important not to get it irretrievably wrong.

This is why I am so dogmatic about what goes into an agile manifesto. If the experience of agile software development and its alternatives teaches us anything, it is that if you think agile, your process becomes more and more agile and more and more successful; while if you don’t, it doesn’t. There doesn’t seem to be much of a middle ground. And it can be hard to wrap your mind around such a different way of thinking, and therefore easy to backslide subtly but permanently.

I will try in the next few weeks to flesh out the agile marketing “thought experiment” on which these suggested principles are based. In the meanwhile, enjoy – or not.

Agile principle 1 (fast, effective change goal): The successful thrive; the agile survive. In the long run, survive beats thrive.

Agile principle 2 (coopetition): What tries to kill us and fails makes us weaker and more agile – and that’s good.

Agile principle 3: Greed is bad. Customer-coordinated change is good.

Agile principle 4: If you don’t know who you will meet, you don’t know where you’re going.

Agile principle 5: People buy learning

Agility and Sustainability: Complement or Clash?

It seems to me that both the concept of “agility” and the concept of “sustainability” have reached the point of real-world use where they will begin to intersect, especially in businesses, and where we will need to think about whether one should be subordinated to the other – because sustainability projects may detract from efforts towards agility, or agility efforts might detract from moves toward sustainability. Or, it may be that, fundamentally, the two complement each other, and can act in synergy. Since agility, at least, has proven its long-term worth in bottom-line results, and sustainability may turn out to be as necessary as the air we wish to breathe, the answer matters.

What follows is very far removed from daily reality. If business agility is fully achieved in a business – and we are so far from such a business that it is hard to say if it will ever exist – and if sustainability really takes over as a global business strategy and economic system – and we still do not know how such a system can function in the long term – then the relationship between the two will be a major issue. Still, some sort of answer would be helpful right now, so that as these two business and societal cultures lurch slowly into possible ubiquity, we can smooth the way for either/both. All we can do is paint with a broad brush; well, let’s get to it.

Setting the Stage: Similarities and Differences

Let’s start with business agility in its extreme form. Such a business (or society) is not focused on success as measured in money or power, but rather on changing rapidly and effectively. The reason this works is that all our institutions and thoughts bring us back to considering success – what we need from agility is a reward for constantly moving on from success. What grounds the agile business is constant negotiation with its customers and the environment as a whole, in which first one, then the other takes the lead in saying what to do next.

The mind shift described here is not as great as it sounds. In any given business, people want to learn, as long as it’s “safe”: it’s accepted by the people around you, people value you for being able to change as well as for specific accomplishments, you have some idea what to change to (i.e., the customer or end user gives you lots of suggestions), and the success rate of this type of thinking is high.

Such an approach has not only enormous positive side-effects on traditional but real measures of success – revenue, cost reduction, profit, customer satisfaction, power – but also great long-term psychic rewards – constant learning with constant positive feedback that what you are doing matters to other people as well as you, which can in some cases only end with the end of your life. In other words, once introduced, business agility can feed on itself – as long as it can get past the initial major mind and culture shift, which involves new metrics, new daily work routines, and baffling encounters with other parts of the company who are doing the same agility thing in a different way. But it all works out, because the work is more fun and because the new agile you starts treating those other parts of the company as customers.

Now let’s switch to the extreme form of sustainability. Wikipedia’s definition continually circles back to: preservation of a system’s ability to endure. More practically, this means: don’t put a system into such a state that it will break down. Usually, breakdown is done by scaling beyond the system’s ability to be patched, or destroying enough of the system’s infrastructure that it can no longer function. So, in my own definition, sustainability means creating a system that will not scale beyond a certain limit, forever, and will not destroy necessary infrastructure. All that we do, all the ways that we grow and change, are forever bounded by those limits – so a sustainable system is one in which not only are limits enforced, but the system itself encourages no violation of those limits. And yet, how do you do that without eventually banning all important change?

Well, strictly speaking, you can’t. And yet, you can indeed get so close to that, that we might well defer the problem until the end of the universe or thereabouts. The answer is equal parts “flight to the huge” and “flight to the virtual.” By that I mean, there are enough physical resources of certain types in the universe (like sunlight) that are beyond our capacity to exhaust for a very long time, and there is enough computer capacity even now that we can shift the things we do from physical to virtual for a very long time, and when we combine the two we see that most other limits that we are approaching can be handled for a huge amount of time by shifting them into the huge and virtual buckets as fast as we ramp up our systems.

The obvious example is putting carbon in the atmosphere. In a fairy tale world, we figure out how to capture and store sunlight with maximum effectiveness and level out its use, shifting from fossil fuels to handle our expanding use of energy as we increase in numbers and prosper. However, that’s not enough; we also need to wipe out that increase of energy usage. And so, we shift from physical to virtual, changing the content of our physical machines to include greater and greater amounts of software, avoiding the need for physical machines by accomplishing much of their tasks by software. This is not about making the physical worker useless before an omnipotent machine; this is about 6 billion people with 6 billion “virtual machines” doing more than what 5 billion people with 1 billion “virtual machines” were doing before, but with the same number of real machines and the same energy usage, ad infinitum. That’s “close enough” sustainability: you are not trapped in a pond, circling the same shores forever, but always flowing down a river that appears to be forever widening and changing, with enormous shores on either side and a riverbed that never deepens.

And now we can see the similarities and the differences between this type of sustainability and this type of agility. Here are the similarities:

• Both are all-encompassing mind sets, cultures, and processes that are clearly quite different from the ones we have now.

• Both are likely to be more effective – help us to survive more, help us to be better off physically and psychologically – than what we are doing today, in the long term.

• Both allow for growth and change – the growth and change that has brought us this far.

But, here are the differences:

• Agility is a more positive message: there’s always something to learn, something better to do; while sustainability is a more negative message: there are limits, and life is about staying within those limits.

• Agility is open and more uncertain, you assume that there will always be new things to do, but you have no idea what they are; sustainability is closed and more certain, as the things that matter are avoiding breakdowns, which are typically bounded problems that are physical, definable, and measurable.

• Fundamentally, agility focuses on changing and ignores survivability, and so improves your chance to survive; sustainability limits change in the name of survival, and who knows what that does to your ability to grow and change?

In other words, agility and sustainability can be seen as rivals for the better business, culture, and society of the future, which can all too easily either undercut the other in their competition for being the One True Way, or fight so fiercely that neither is achieved. And, as I noted in the introduction, either kind of “clash” outcome will eventually matter a lot.

Initial Thoughts on Synergy

It seems to me that there are two approaches that could ensure that, despite this, both agility and sustainability can thrive, and perhaps even reinforce each other. I call them the tactical and the strategic.

The tactical approach tries to manage combining agility and sustainability on a case-by-case basis. This means, for example, setting the boundaries of sustainability very clearly, so that they are intuitive for all, for the business as well as the society, and enforcing them strongly when clearly violated, but allowing endless agility within those boundaries. Carbon pricing is an example of the tactical approach: if effectively administered, it constantly sets clear monetary boundaries on carbon use, within a sustainable carbon yearly limit. Do that, says the tactical approach, and then let the business be as agile as it wishes.

The strategic approach, by contrast, seeks to create an integrated mindset combining agility and sustainability. One way to do this that I view as feasible is to get people used to the idea that trying to do the impossible is not the most effective way of doing things. As I have previously written, “working smarter, not harder” comes down to “instead of trying the impossible, be smart and focus your (in this case, change ideas) on things that are possible.” Knowing short-term and long-term limits becomes an essential part of the really agile business, because it allows people to increase the number of new ideas that are really good ones. But then, the strategic approach involves yet a third mind shift, to trying to work smarter instead of harder and seeing limits as positive idea-producers; how many mind shifts can we do, anyway?

Summing Up

My overall conclusion is brief: Tentatively, I lean towards taking a strategic approach to melding agility and sustainability, and applying it wherever frictions between the two crop up; but my main conclusion is that we had better start thinking more about tactical and strategic approaches, and how to do them, right now. It may be that the successes of agility will lead to a mindset that treats sustainability as yet another source of new ideas, and so we muddle through to survival; but I wouldn’t count on it. Much better to be proactive even when agility doesn’t call for it, and think about how to combine carbon metrics and agility metrics. In the meanwhile, I’ll continue preaching agility – because it’s fun – and keeping a wary eye on sustainability – because we may need it sooner than we think.

That’s as far as I can get. Ideas, anyone?

Information Agility and Data Virtualization: A Thought Experiment

In an agile world, what is the value of data virtualization?

For a decade, I have been touting the real-world value-add of what is now called data virtualization – the ability to see disparate data stores as one, and to perform real-time data processing, reporting, and analytics on the “global” data store – in the traditional world of IT and un-agile enterprises. However, in recent studies of agile methodologies within software development and in other areas of the enterprise, I have found that technologies and solutions well-adapted to deliver risk management and new product development in the traditional short-run-cost/revenue enterprises of the past can actually inhibit the agility and therefore the performance of an enterprise seeking the added advantage that true agility brings. In other words: your six-sigma quality-above-all approach to new product development may seem as if it’s doing good things to margins and the bottom line, but it’s preventing you from implementing an agile process that delivers far more.

In a fully agile firm – and most organizations today are far from that – just about everything seems mostly the same in name and function, but the CEO or CIO is seeing it through different eyes. The relative values of people, processes, and functions change, depending no longer on their value in delivering added profitability (that’s just the result), but rather on the speed and effectiveness with which they evolve (“change”) in a delicate dance with the evolution of customer needs.

What follows is a conceptualization of how one example of such a technology – data virtualization – might play a role in the agile firm’s IT. It is not that theoretical – it is based, among other things, on findings (that I have been allowed by my employer at the time, Aberdeen Group, to cite) about the agility of the information-handling of the typical large-scale organization. I find that, indeed, if anything, data virtualization – and here I must emphasize, if it is properly employed – plays a more vital role in the agile organization.

Information Agility Matters

One of the subtle ways that the ascendance of software in the typical organization has played out is that, wittingly or unwittingly, the typical firm is selling not just applications (solutions) but also information. By this, I don’t just mean selling data about customers, the revenues from which the Facebooks of the world fatten themselves on. I also mean using information to attract, sell, and follow-on sell customers, as well as selling information itself (product comparisons) to customers.

My usual example of this is Amazon’s ability to note customer preferences for certain books/music/authors/performers, and to use that information to pre-inform customers of upcoming releases, adding sales and increasing customer loyalty. The key characteristic of this type of information-based sale is knowledge of the customer that no one else can have, because it involves their sales interactions with you. And that, in turn, means that the information-led sell is on the increase, because the shelf life of application comparative advantage is getting shorter and shorter – but comparative advantage via proprietary information, properly nurtured, is practically unassailable. In the long run, applications keep you level with your competitors; information lets you carve out niches in which you are permanently ahead.

This, however, is the traditional world. What about the agile world? Well, to start with, the studies I cited earlier show that in almost any organization, information handling is not only almost comically un-agile but also almost comically ineffective. Think of information handling as an assembly line (yes, I know, but here it fits). More or less, the steps are:

1. Inhale the data into the organization (input)

2. Relate it to other data, including historical data, so the information in it is better understood (contextualize)

3. Send it into data stores so it is potentially available across the organization if needed (globalize)

4. Identify the appropriate people to whom this pre-contextualized information is potentially valuable, now and in the future, and send it as appropriate (target)

5. Display the information to the user in the proper context, including both corporate imperatives and his/her particular needs (customize)

6. Support ad-hoc additional end-user analysis, including non-information-system considerations (plan)

The agile information-handling process, at the least, needs to add one more task:

7. Constantly seek out new types of data outside the organization and use those new data types to drive changes in the information-handling process – as well, of course, as in the information products and the agile new-information-product-development processes (evolve).

The understandable but sad fact is that in traditional IT and business information-handling, information “leaks” and is lost at every stage of the process – not seriously if we only consider any one of the stages, but (by self-reports that seem reasonable) yielding loss of more than 2/3 of actionable information by the end of steps one through six. And “fixing” any one stage completely will only improve matters by about 11% -- after which, as new types of data arrive, that stage typically begins leaking again.

Now add agile task 7, and the situation is far worse. By self-reports three years ago (my sense is that things have only marginally improved), users inside an organization typically see important new types of data on average ½ year or more after they surface on the Web. Considering the demonstrated importance of things like social-media data to the average organization, as far as I’m concerned, that’s about the same as saying that one-half the information remaining after steps one to six lacks the up-to-date context to be truly useful. And so, the information leading to effective action that is supplied to the user is now about 17% of the information that should have been given to that user.

In other words, task 7 is the most important task of all. Stay agilely on top of the Web information about your customers or key changes in your regulatory, economic, and market environment, and it will be almost inevitable that you will find ways to improve your ability to get that information to the right users in the right way – as today’s agile ad-hoc analytics users (so-called business analysts) are suggesting is at least possible. Fail to deliver on task 7, and the “information gap” between you and your customers will remain, while your information-handling process falls further and further out of sync with the new mix of data, only periodically catching up again, and meanwhile doubling down on the wrong infrastructure. Ever wonder why it’s so hard to move your data to a public cloud? It’s not just the security concerns; it’s all that sunk cost in the corporate data warehouse.

In other words, in information agility, the truly agile get richer, and the not-quite-agile get poorer. Yes, information agility matters.

And So, Data Virtualization Matters

It is one of the oddities of the agile world that “evolution tools” such as more agile software development tools are not less valuable in a methodology that stresses “people over processes”, but more valuable. The reason why is encapsulated in an old saying: “lead, follow, or get out of the way.” In a world where tools should never lead, and tools that get out of the way are useless, evolution tools that follow the lead of the person doing the agile developing are priceless. That is, short of waving a magic wand, they are the fastest, easiest way for the developer to get from point A to defined point B to unexpected point C to newly discovered point D, etc. So one should never regard a tool such as Data Virtualization as “agile”, strictly speaking, but rather as the best way to support an agile process. Of course, if most of the time a tool is used that way, I’m fine with calling it “agile.”

Now let’s consider data virtualization as part of an agile “new information product development” process. In this process, the corporation’s traditional six-step information-handling process is the substructure (at least for now) of a business process aimed at fostering not just better reaction to information but also better creation and fine-tuning of combined app/information “products.”

Data virtualization has three key time-tested capabilities that can help in this process. First is auto-discovery. From the beginning, one of the nice side benefits of data virtualization has been that it will crawl your existing data stores – or, if instructed, the Web – and find new data sources and data types, fit them into an overall spectrum of data types (from unstructured to structured), and represent them abstractly in a metadata repository. In other words, data virtualization is a good place to start proactively searching out key new information as it sprouts outside “organizational boundaries”, because they have the longest experience and the best “best practices” at doing just that.

Data virtualization’s second key agile-tool capability is global contextualization. That is, data virtualization is not a cure-all for understanding all the relationships between an arriving piece of data and the data already in those data stores; but it does provide the most feasible way of pre-contextualizing today’s data. A metadata repository has been proven time and again to be the best real-world compromise between over-abstraction and zero context. A global metadata repository can handle any data type thrown at it. For global contextualization, a global metadata repository that has in it an abstraction of all your present-day key information is your most agile tool. It does have some remaining problems with evolving existing data types; but that’s a discussion for another day, not important until our information agility reaches a certain stage that is still far ahead.

Data virtualization’s third key agile-tool capability is the “database veneer.” This means that it allows end users, applications, and (although this part has not been used much) administrators to act as if all enterprise data stores were one gigantic data store, with near-real-time data access to any piece of data. It is a truism in agile development that the more high-level the covers-all-cases agile software on which you build, the more rapidly and effectively you can deliver added value. The database veneer that covers all data types, including ones that are added over time, means more agile development of both information and application products on top of the veneer. Again, as with the other two characteristics, data virtualization is not the only tool out there to do this; it’s just the one with a lot of experience and best practices to start with.

And, for some readers, that will bring up the subject of master data management, or MDM. In some agile areas, MDM is the logical extension of data virtualization, because it begins to tackle the problems of evolving old data types and integrating new ones in a particular use case (typically, customer data) – which is why it offers effectively identical global contextualization and database-veneer capabilities. However, because it does not typically have auto-discovery and is usually focused on a particular use case, it isn’t as good an agile-tool starting point as data virtualization. In agile information-product evolution, the less agile tool should follow the lead of the more agile one – or, to put it another way, a simple agile tool that covers all cases trumps an equivalently simple agile tool that only covers some.

So, let’s wrap it up: You can use data virtualization, today, in a more agile fashion than other available tools, as the basis of handling task 7, i.e., use it to auto-discover new data types out on the Web and integrate them continuously with the traditional information-handling process via global contextualization and the database veneer. This, in turn, will create pressure on the traditional process to become more agile, as IT tries to figure out how to “get ahead of the game” rather than drowning in a continuously-piling-up backlog of new-data-type requests. Of course, this approach doesn’t handle steps 4, 5, and 6 (target, customize, and plan); for those, you need to look to other types of tools – such as (if it ever grows up!) so-called agile BI. But data virtualization and its successors operate directly on task 7, directly following the lead of an agile information-product-evolution process. And so, in the long run, since information agility matters a lot, so does data virtualization.

Some IT-Buyer Considerations About Data Virtualization Vendors

To bring this discussion briefly down to a more practical level, if firms are really serious about business agility, then they should consider the present merits of various data virtualization vendors. I won’t favor one against another, except to say that I think that today’s smaller vendors – Composite Software, Denodo, and possibly Informatica – should be considered first, for one fundamental reason: they have survived over the last nine years in the face of competition from IBM among others.

The only way they could have done that, as far as I can see, is to constantly stay one step ahead in the kinds of data types that customers wanted them to support effectively. In other words, they were more agile – not necessarily very agile, but more agile than an IBM or an Oracle or a Sybase in this particular area. Maybe large vendors’ hype about improved agility will prove to be more than hype; but, as far as I’m concerned, they have to prove it to IT first, and not just in the features of their products, but also in the agility of their data-virtualization product development.

The Agile Buyer’s Not So Bottom Line

Sorry for the confusing heading, but it’s another opportunity to remind readers that business agility applies to everything, not just software development. Wherever agility goes, it is likely that speed and effectiveness of customer-coordinated change are the goals, and improving top and bottom lines the happy side-effect. And so with agile buyers of agile data-virtualization tools for information agility – look at the tool’s agility first.

To recap: the average organization badly needs information agility. Information agility, as in other areas, needs tools that “follow” the lead of the person(s) driving change. Data virtualization tools are best suited for that purpose as of now, both because they are best able to form the core of a lead-following information-product-evolution toolset, and because they integrate effectively with traditional information-handling and therefore speed its transformation into part of the agile process. Today’s smaller firms, such as Composite Software and Denodo, appear to be farther along the path to lead-following than other alternatives.

Above all: just because we have been focused on software development and related organizational functions, that doesn’t mean that information agility isn’t equally important. Isn’t it about time that you did some thought experiments of your own about information agility? And about data virtualization’s role in information agility?

The Other Agile Development: XtremeEDA and Product Development

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in agile development, agile new product development, and business agility over the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about agile development today.

The Importance of Continuous Delivery to Agile Development

I have taken a hiatus from this series over the last two weeks, but I knew one thing that I wanted to tackle: product development that included hardware, or New Product Development (NPD) solutions. And I have to say that, in general, the situation is distressing.

It’s been more than ten years since the Agile Manifesto, and in that time there have certainly been vocal proponents of agile NPD. The lean movement, for one, keeps claiming that “lean” has incorporated the tenets of agile software development, that “lean” and “agile” are complementary, and that this is a methodology that is being applied to NPD in general.

Looking at their and other evidence, I am not at all convinced. There is little or no evidence that their proponents are intensively applying agile software development techniques and methodologies to actual hardware development, whether you’re talking about engineering a plane or designing and fabricating a chip. There is a persistent tendency in what I have seen in the agile/lean movement to treat just-in-time leanness and fast-design-change-from-user-feedback agility as equally important (or, treat lean as more important than agile), rather than carefully considering what it would mean to have super-lean NPD give way when agility demands (as, in agile software development, it often does) temporary use of extra resources plus Gantt charts that are not time-optimized. Finally, I cannot find clear evidence of forward thinking on how, if we are going to maintain that agile hardware development is not appropriate in many hardware-design cases, agile and non-agile will integrate effectively. It is clear from some of the comments I have seen that many hardware engineers are bewildered by the seeming Wild West of agile software development, and in no mood to face an overall schedule in which they depend on the changing specs and indefinite deadlines of the software side.

And yet, I was able to find one small but amazingly perceptive example of another kind of thinking about NPD. XtremeEDA Corporation, and in particular Neil Johnson there, is just what I’m talking about.

If you want to see an example, try submit2011.agilealliance.org/files/session_pdfs/applying%20agile%20to%20ic%20development.pdf. There it all is: How to turn ASIC development into agile; how to coordinate with agile software developers; how to use more customer feedback; and the message “to heck with minimal just-in-time resources and up with ‘one person owns a task.’” He even – be still, my heart – quotes Ken Schwaber, whom I knew, before his days as Mr. Scrum and as one of the original Manifesto signers, as the vendor of a superb rapid application development tool that basically “got” agile (alas, it was only for the AS/400).

Even I don’t want to overstate the relevance of agile development approaches to hardware NPD. There are still constraints, not the least of which is the fact that producing a hardware prototype or even a functioning sub-product prototype is in many cases a costly and time-consuming process that we cannot yet completely accomplish as swiftly as compile-load-go.

And yet, what Mr. Johnson makes clear is that, in the real world, agile hardware development and integrated, totally agile joint hardware/software NPD still do better than the traditional counterparts, in time, in money, and in customer satisfaction. And that’s just one product iteration. You tell me what rapid, incremental, customer-pleasing hardware improvements would mean for your bottom line.

Isn’t it time that not only IT but also the CAD and manufacturing side gave it some thought?

The Relevance of XtremeEDA to Agile NPD

I don’t want to misrepresent the imminence of agile NPD, so these next two sections will be short. Neil Johnson is one person in XtremeEDA, and clearly the firm doesn’t make a big point of his efforts, I am guessing in order not to spook its traditional customers for consulting and ASICs. I am sure that XtremeEDA does a fine job with these tasks, and that if you want to use them you need never run the risk (?????) of agile hardware development, or even hear about it, if you don’t want.

That said, I am also sure that if anybody does want to see how it’s really done, XtremeEDA would be very pleased to have someone like Neil show them, and any fees involved have got to be minimal considering the advantage to be gained. So what, bottom line, could XtremeEDA potentially offer if you go that route?

First, hands-on experience – as in the url to which I just sent you. Second, creative thinking about how to apply this experience and the resulting insights to a wide variety of (at least in the chip area) hardware development projects, fab and otherwise. Third, both experience and creative thinking in how to mesh embedded-software and hardware agile development. Fourth, consulting and support for your efforts. Above all, fifth, suggestions on scaling agile hardware and hardware/software NPD.

The point of this exercise in imagining how xTremeEDA could help you is to lead you to the next question: How will this information make my NPD – specifically, NPD that includes hardware, like a cell phone or a child’s toy – agile? And the answer, in case I haven’t said it sufficiently, is that agile is, more than anything else, a process and a culture. You process NPD that way, you think NPD that way. If you are really doing agile NPD right, NPD agility comes with the package. If you use information from someone like XtremeEDA to implement agile hardware development and integration between your agile software development and agile hardware development, your NPD will be agile.

Potential Uses of XtremeEDA-Type Agile NPD for IT and NPD

In this case, IT and NPD are two separate things, and often, in the case of hardware development, two separate organizations. On the IT side, if there is a separate NPD organization, this means evangelism. It may not work – but with someone who has walked the hardware walk at your side, like XtremeEDA, you may actually get listened to. Of course, if the NPD organization is folded under a joint hardware-plus-software IT organizatio, it’s more a matter of top-down commitment from the CIO (and, hopefully, the CEO), and then using someone like XtremeEDA to hand-hold and calm the rightful fears of experienced hardware engineers.

If we’re talking about an overall NPD organization to whom software development is both important and an annoying pest, on the other hand, you may want to consider initially leaving IT and/or agile software development organizations out of the conversation completely. That will avoid the usual antagonisms that surface between hardware and software teams, and it will also make the organization squarely face the fact that pure agile hardware development works – and there’s a hardware geek like someone at XtremeEDA to tell them so. Once the hardware-only implementation has shown the doubters, and worriers begin to turn into enthusiasts, you can let loose the software and hardware sides on each other, and they will automagically start practicing their getting-user-feedback skills on each other. An integrating process tool is great, but don’t forget that these folks may well do some of the integration on their own.

The Bottom Line for IT Buyers

I freely admit that an XtremeEDA approach to NPD may turn out to be of little value to most companies in the next 2-3 years. That, however, is not because it intrinsically has no value. Rather, it is because I am fully confident that most companies with significant hardware in their products will not figure out how to walk the walk before 2015. They will probably not be in a hurry, because they will rightly figure that most of their competitors will take the same attitude. Into the pool? You first. I live in hope; but I’ve seen too much.

However, when – not if – agile NPD really begins to take over, I urge you to consider carefully the further implications of truly agile NPD. My survey results, as well as experiences when software is the only product of the company, suggest that NPD is the absolute biggest bang for the buck for the business in implementing business agility, leading to permanent, 10-35% improvements in revenues, ROI, and customer satisfaction (not to mention quality), and similar reductions in costs, compared to what they would have been had you not implemented agile. Today, in many companies, agile software development is really about competitive advantage. In NPD, it is about the top and bottom lines.

Moreover, because NPD is always at the core of company success, agile NPD leaks into other areas of the company, far more than agile software development. Marketers cannot help being more agile. CEO strategies, given new power to change product directions fast, become more nimble, and the CEO begins to think more in terms of strategy change than enforcing execution. Even finance – well, maybe someday.
If, however, you really are a visionary, and would like to do it now, then I beg you not to talk about it publicly. What? XtremeEDA? Oh, they’re just here to help us out with some ASICs. Nothing to see here. These aren’t the droids you’re looking for. And your company’s success? Oh, just lucky, I guess. Fun, isn’t it?

Yes, it is. And XtremeEDA-type agile product development just happens to be quite profitable, too. Just a coincidence, I’m sure. Yeah, right.