We Have Blamed the Vendors and IT; Is It Now Time to Blame the Business Types?

Reflecting on the recent financial reports of the giants of computing infrastructure – companies like IBM, HP, Oracle, and Microsoft – I am struck by the continuation of a troubling trend:  The technology and its usefulness is moving ahead as fast as, if not faster than, ever before; IT has made some striking adjustments; but the revenues to vendors from this, and therefore IT spend, are flat if not falling. Why?

Let’s see.  In databases, Oracle and IBM are almost flat, despite (at least in IBM’s case) two technologies in two years that potentially deliver performance and price-performance gains well ahead of Moore’s Law.  I find plenty to criticize in Windows 8, but its implementation with regard to business needs, in Windows and Office usage, should deliver benefits comparable to that of the introduction of the mouse-based user interface – and Microsoft enterprise revenues are flat.  Cloud technology is not replacing, but rather complementing, in-house architectures, and therefore should require, initially, more rather than less total spending – and yet, increases in cloud spending are instead paired with overall flat or lower user computing spend.  No, I don’t buy the “public cloud saves money” rationale – the benefits come later rather than sooner, and involve mainly flexibility and to a much lesser extent multi-tenancy and grid savings, especially when many users are employing multiple cloud providers that must be coordinated.  And the IBM and Oracle/SPARC data suggest the biggest hardware decline is in Unix/Linux hardware, the darling of the cloud set, not in mainframes or PCs. 

In the past, a typical explanation for this has been the need for the business to cut back IT spend in response to decreasing profits.  Of course, that did not prevent IT and computing from becoming a larger and larger part of overall business spending during the 1980s and 1990s.  In the early 2000s, IT spend actually decreased proportionally to the cost-cutting of the rest of the business – and yet, despite an initially tepid overall business recovery, computing spending from about 2003 to 2008 rebounded and grew quite briskly. So this pattern of user computing spending is unprecedented in three ways:  its length, its unremitting focus on cost-cutting apparently comparable in size to the rest of the business, and its seeming inability to reflect major technology advances with a good claim to deliver major cost or other benefits to the rest of the business.

The Blame Game

In the past, I have found, when things go wrong in computing spending the first reaction is to blame the vendors.  In this case, the obvious critique is that they have failed to communicate the benefits of the technology, cost and otherwise.  Except that, as I can attest from my experience as an analyst, both the technology advances and the communications from the vendors are as good as, if not in many cases better than, those of most or all of the past.  Big Data, for example, is not pure hyperbole, and vendors have done a reasonable job by traditional standards of highlighting the ability of targeted Big-Data analytics to explain and bind the customer as never before, in a cost-effective way. 

OK, then the next line of analysis is to blame IT, typically for failing to align themselves with the business’ strategy, or (when times are tough) identify ways of lifting the dead hand of older systems and other seemingly outdated costs.  To this, I have one response:  agile IT.  As never before, IT is aligning itself with agile development, by supporting a software lifecycle that includes operational feedback integrated with development, and by encouraging automation of functions that provides a basis for rapid response and identification of architectural problems and opportunities at the administrator level.  As my studies have shown, these plus the increasingly agile nature of the software that IT makes available to the business translate into major cost-cutting and major benefits beyond what traditional IT approaches can deliver.  And as for aligning itself with business strategy, IT is well aware of Big Data as a hot topic, hence the sudden and somewhat odd demand for Hadoop experts.  No, whatever IT may have done in the past, it seems clear that it has upped its game.

OK, then, who does that leave?  And yet, how can we blame the business types?  Aren’t these the same folk who drove those increases in IT percentage of spending over the 1980s and 1990s?  Aren’t these the most receptive listeners when we talk about the power of embedded analytics to improve business processes and the importance of analyzing the “customer of one”?

There’s Something Not Going On Here …

To understand why there might be cause for concern and, yes, for blame in business strategy – driven by the CEO, the CFO, and their staff – let’s look in very broad-brush terms at what has been going on, not just for the last five years, but for the last three decades.  Here are a few of the highlights, as suggested by various studies:

1.       Advances in productivity have been accruing 90% to the CEO – ½ in his income as CEO, and ½ in his increases in wealth as an investor.  While this still represents a fairly small percentage of overall expense, it does unnecessarily increase the focus on cutting costs for all except the CEO.

2.       Hedge funds that sometimes act as turnaround artists or “shadow banks” have taken a good 90% or so of the funds’ investment profits for themselves, and thus receive compensation not just in the tens of millions of dollars but, at the top, in billions of dollars, despite the fact that they deliver less to the investor than an index fund in most if not all cases.  While it is difficult to see direct effects of this “fleecing of the rich” on corporate strategies, one significant effect is to increase the fleeced CEO’s focus on growing profits uber alles, to at least get some return from their investments – with the Googles of the world the few happy exceptions.

3.       The present long-lasting recession/stagnation has effectively removed perhaps 10% of the workforce from work over a long period of time.  This has the effect, since businesses often discriminate against the long-term underemployed, of creating a permanent gap between “potential” GNP and real GNP.  To put it another way, even if the economy grows at a reasonable pace, it will still be much smaller it should be.  That means smaller markets and, again, more relative emphasis on cost-cutting to achieve profit growth rather than growing revenues to grow profits.

There are three concerns about this excessive focus on cost-cutting.  The first, which probably is not serious right now but is far more troubling than five years ago, is that cost-cutting eventually runs out as a strategy if revenues continue to be flat (and, don’t forget, revenues are pretty flat for the bulk of businesses, hence the 1% growth in GNP over the first part of this year).  The second concern is that this cost-cutting, applied economy-wide (as it seems to be doing), reinforces and cements smaller markets by eliminating the part of the market funded by the now underemployed.  Ordinarily, recessions are too short for this to happen; but it seems to be happening now, as new jobs continue to make no dent in the workforce/working-age-population ratio.

The third concern, which I for one find the most troubling, is the possibility that reflexive cost-cutting becomes the answer to every situation, the reflex of the business.  If that is true, it would suggest that businesses are blowing opportunities for savvy increases in computing technology spend because it’s all about cost-cutting, not strategic investment. 

Blame or Not, What Might Business Strategists Do Better?

First, I would suggest that businesses set up a long-term plan and process for using analytics to better understand and improve the relationship with the customer.  This means acquiring data virtualization software and the like, to allow aggressive searching out and combination of all the new customer information constantly being created.  It also means buying the new infrastructure tools (database and information architecture) that can scale to handle aggregation of social-media data on the Web and in-house customer-experience data. 

Second, I would argue (not that I expect many businesses to listen) that in order to grow revenues as well as profits (and probably grow profits faster), business types need to acquire and foster use of agile tools and processes – such as agile marketing, with its emphasis on data-driven understanding of the customer and prospect rather than the anecdotal, top-down opinion that has been all too common in the past.  As I noted in a recent blog post, an agile business is a possibility, if the CEO and his lieutenants really commit to it – and it typically means not only the CEO using a Scrum-type planning process, but also constant modification and constant feedback up and down the organization about plans.  I should also note that an agile process specifically builds in slack for learning and review, and yet (according to my studies) it produces much better cost-saving and profit results than a “cram as much work as possible in, be as cost-efficient as possible” approach, for the CEO as well as the developer.

And then there is the really controversial stuff. Sustainability efforts are, by and large, really underperforming, as a recent article in www.thinkprogress.com confirms my impression that gains in the US in carbon emissions are effectively coming from offshoring the problem.  Businesses need to acquire effective global carbon-tracking software, and use it.  Businesses need to join to push shared quality regulations that will move all towards new computer and software technology that is more productive, rather than clinging to the infrastructure of the past that in the long run costs more, as in the electrical/energy grid vs. software-coordinated regional combined solar/wind – and rather than enabling the block-all-regulation political excesses of the U.S. Chamber of Commerce and its ilk.

I’m not saying that all this is doable, or that the case for questioning the approach of the overall business to computing technology is clear.  I am saying that it’s time we recognize the seriousness of the overall problem of which computing spend is a part, and we stop blaming the usual suspects.  Five years is enough.  Thirty years is enough.  In the long run, a cost-cutting strategy is not enough – and the long run is beginning to arrive.   

 

Tiempo Development and the Agile Business

I have heard a future of business organization, and it sounds as if it works.

After ten years of hearing companies only begin to appreciate agile development and then agile marketing, I certainly was not expecting to hear a company organize its management, its strategy, and its planning around a company-wide agile process. But when Tiempo Development, billing itself as yet another agile development outsourcer, gave me a telebriefing last week, I asked the obligatory question and was floored to hear that they had, indeed, done so. 

The Beginning Of The Epic

I am sure that, like many another agile-development startup, Tiempo asked themselves why they couldn’t apply agile principles to management – and yet, others I have talked to stopped there.  Tiempo, however, when faced with the need to begin doing better planning and budgeting as the business scales, seems to have had the guts to take a leap of faith and try to do it via a variant of Scrum, complete with Kanban-type lean scheduling, plus the “epics” and “stories” beloved of agile marketing.  In other words, there’s a five-year plan and budget (and, of course, in an agile organization that’s not set in stone) devolving into yearly, quarterly, monthly, and weekly epics and stories. 

Revisions to the plans are quarterly, flowing forward into a new rolling five-year plan. As in Scrum, daily task sessions and weekly reviews communicate widely, and change flows from below as well as above.  “Customer” feedback comes in laterally and frequently from project customers, as well as from above and below on the plans and their implementation.  To put it another way, the concentrated but broad communication of Scrum ensures the maximum of customer-driven evolution, while the five-year horizon ensures that long-term strategic considerations engage in a delicate dance with short-term customer –driven product specs.

We are long past the time when companies would ask whether agile development scales.  Still, the question should be asked:  how well does a Tiempo-style “Scrumban” management approach scale?  The apparent answer after about 1 1/2 years of practice is:  Quite well, so far.  The reason, I would suggest, lies in the initial establishment of five-year company goals in terms of “rocks”.  That is, the implementing CEO picked about five goals that formed what he saw as the fundamental things that the company needed to achieve over the next five years – the “rocks” in both the foundational and the risk sense. 

Again, as noted, the “rocks” are not made of concrete; they evolve over time, fairly frequently, as the company evolves.  Think of them as “virtual rocks”.  They therefore combine long-term perspective; frequent review based on customer and employee feedback; and a broad perspective, not only for the CEO but also in terms of employee understanding of (and buy-in to) strategy. Plug in more business, plug in new areas, plug in new employees.

To Understand Agile Management’s Effectiveness, Redefine Success

What does Tiempo report as evidence of success in agile management?  Interestingly, they start first with better communications between managers and “line” developers, followed by quick surfacing of bad news for rapid handling, and then “empowering” the employees.  It was only when I probed deeper that the follow-on effects emerged:  more rapid development without other sacrifices, leading to better profitability; greater customer satisfaction, leading to greater follow-on business and willingness of large enterprise customers to trust Tiempo in new areas; and a strong growth path that has the CEO more concerned with avoiding lack of focus via too many new areas than with short-term risks.

It was not a given that this would be so, but in this respect agile management is mirroring agile development.  My survey four years ago found a persistent pattern of comparative focus on change rather than cost, speed, profit, quality, or (downside) risk leading to greater cost, speed, profit, quality, and customer satisfaction as well as “upside” risk, while “downside” risk decreases – all compared to attitudes and processes that do focus on these things.  That Tiempo is focusing on the employee and customer satisfaction parts of the effect is what we should expect if Tiempo is “talking the agile talk and walking the agile walk.” And so, if agile management is indeed like agile development, out of this process we should indeed expect better top-line and bottom-line results that are likely to endure.

The problem is, of course, metrics that will establish these things – because the merits of agile management are not as easy to prove as those of agile development.   A fluke, a Board of Directors and the stock market will call the first agile management bottom-line results, caused by unique circumstances and remnants of past practices.  All these daily and weekly meetings, are they really doing any more than past business strategy fads?  I believe strongly that the answer is resoundingly yes. But I can’t do a survey like the agile development one to prove the point, because there are no obvious speed, cost, and risk-reduction metrics for the CEO’s actions, just customer and employee satisfaction, which have proven misleading in the past.

And so, we are back to CEO “culture”, just as we had to deal with IT and business management “cultures” that were highly resistant to the touchy-feely feel-good aspects of agile development.  Remember, CEOs are the byproduct of a selection process that emphasizes enormous, visible hard work and schmoozing as well as hard-nosed attention to the bottom line, and rewards these by surrounding the CEO with those who understand that their future depends on giving the CEO positive reinforcement, as well as with compensation that most fair-minded outside observers concede is well beyond the demonstrated relative effectiveness of the CEO.  In other words, power and money.  I once took a survey at B school that asked whether I wanted to make money, have power, do good for people, or accomplish something; it turned out that those who answered power or money got much more of both.

I mention this because (and this is only my impression) what really got the CEO excited, talking to me, was his own personal satisfaction – that is, the “accomplish something” part of the B school survey.  And yet, there was no reason he wouldn’t wind up with just as much power and money as my classmates who answered “power” or “money”.  In other words, in the agile world, you can make as much (or more, since the organization should be more successful) money and have as much power, but work less (remember, you’re not running around trying to single-handedly move an ocean liner of an organization or sell the world on your product) and get more real satisfaction out of it, because you can point to shared concrete projects finished successfully, not just praise from those who wish to share in your money.  And so, if the CEO and upper management can make the cultural shift, they may well find that they are better off in every way, including personal satisfaction – another agile paradox.

In fact, it turns out that the creators of agile processes may have been craftier than they knew in calling projects and project groupings “stories” and “epics”.  It speaks to something Tolkienesque in all of us.  A study recently reported in MIT Technology Review suggests that we don’t have fixed memories like snapshots of events, we have stories of the past, and each time we revisit them, we can change them, e.g., to fit new understandings of the past, a new narrative arc.  When I do TCO studies, it’s not all about yes/no, 1-5, and numbers; I ask the interviewee to tell his or her story, the story of him or her in the project.  Set the stage for me, what was your environment; tell me how the project unfolded; tell me what you found were the results; and only now tell me how this affected the TCO that you find justifies the project.  I have found the insights from such an approach far richer and better founded than those from standard surveys, even when the data is sparser.

In an agile business such as Tiempo’s, it sounds as if everyone becomes a hero or heroine in a shared epic that never ends, always new and unfolding.  Oh, and by the way, it should perform better than an old-style business.

Caveats and Not So Bottom Line

I have always been a bit dissatisfied with the analyst’s usual disclaimer:  Your mileage may vary.  Yes, and your hair may grow longer or get cut shorter.  In this case, my real disclaimer is not that agile management may or may not apply to your organization, but rather that I don’t have the figures or real-world use cases to support my belief that Tiempo’s agile-management experience applies well to most medium-sized and large-sized enterprises and organizations. 

Will Tiempo continue scaling?  Does the agile management model appropriate to a small-to-medium-sized software-development outsourcer apply to most other types of business – not the traditional “hardware” manufacturing firm that forms the “unconscious” of all businesses today, and which as far as I can tell is a small fraction of today’s businesses, but the rest of them?  What I see and hear says that it should; but until it happens, there are afaik no decent metrics to show it is working.  Your mileage will probably not vary, but you won’t know the mileage for sure until the end of your first long trip.

No now we can proceed to my not so bottom line – because, remember, agile is about focusing on change, and the bottom line will follow.  Agile development, by now, has thousands of successful use cases. New product development is getting there, and there is now a “hardware” (silicon foundry) use case, as cited in a blog post a while ago.  Agile marketing is approaching its thousandth use case – I note that in a little over a year, the Bay Area agile marketing group has gone from zero members to well over a thousand; and IT is doing its best to keep up.  And now, we have an agile management – and hence agile business – use case. It’s time to start shifting the burden of proof.  Instead of wondering why you should be an agile developer/marketer/business, you should be demanding good reasons why you shouldn’t be.

I have heard a future of business organization, and it sounds as if it works.   Please stop asking about the mileage and start installing the tires.

2012 Election: Big Data Makes It Easy to Shoot Yourself In The Foot

A recent post in MIT Technology Review about the use of Big Data by the parties in the 2012 election offers fascinating insights into what works – and what doesn’t.  Both sides had operations specifically designed to glean new insights about voters and apply them to helping their candidate win, and the article interviews the director of each effort shortly after the election.  As a result, we see long-term Big Data projects whose results have just been tested against the reality of the voter “market” and whose project reviews are relatively untainted by post-project “spin.”

I should note that I view the article’s author’s grandiose claims about the meaning of the Obama campaign’s relative success in Big Data analytics to be overblown.  Luckily, many of those claims take up only the first 20% of the article. The rest provides a fascinating insight into two approaches to Big Data, one of which demonstrably worked better than the other.

Problem #1: Big Data as a “Hot Topic” vs. as a Strategy

The first contrast between the campaigns was that the Obama one chose to bring the analytical systems in-house, whereas the Romney campaign chose to use existing software and hardware run by someone else.  In fact, the Obama campaign specifically committed major bucks to a Vertica solution way back in 2009.  This begs the question, however – why did the two campaigns take such different approaches? After all, the Romney campaign actually started quite early, around the same time, and had no real constraints on money spent.

The answer, I think, is that while Obama viewed in-depth analytics and trolling the Internet for more voter insights as a strategy, while for the Romney campaign, the concept of Big Data was fuzzier, more of a “hot topic” type of strategy.  We have all seen CEOs who have said, “Everyone’s talking about this new technology. We must therefore do it.”  The flavor of Romney-campaign thinking, as reported in the article, was, “Big Data is clearly effective at the corporations we look at; we must therefore get something that purports to do Big Data and use it somehow to understand voters better.”

The result was a cautionary tale.  The Romney system involved a third party merging two sets of legacy-application data repeatedly.   As a result, breakdowns occurred in the stress of the late campaign, and analytics was unnecessarily delayed, delaying candidate responses. The Obama campaign suffered from no such problems. It is not clear how much the Obama campaign’s faster response time helped (although it apparently helped change Obama’s style after the first debate).  However, it is clear that the Romney campaign’s fuzzy and delayed analytics helped lead to Problem #2.

Problem #2:  Hearing the “Customer of One” vs. Hearing What You Want To Hear

One of the great shocks of the campaign was that, in an unprecedented way, the Romney campaign and the Republican party as a whole deluded themselves about what was going to happen.  And yet, the campaign’s Big Data analysis mined a long tradition of understanding voter blocs, and could be supposedly cross-checked against a broad array of pollsters – most of whom were themselves deluded by their political biases.

As it turned out, even the closest estimate underestimated Obama’s margin by about 0.5%.  Moreover, on election day, the Romney campaign actually thought they had a better than even chance of winning. However, as it turned out, to win they would have had to amass more than 5.3% more of the vote in a couple of battleground states than they did, at minimum. To put it another way, they probably would have had to change the vote percentages by more than 5% nationwide to win.  They weren’t even in the same ballpark.

By the way, for Nate Silver fans, he was off by about 1.5%.

There appeared to be two key insights that made this election different.  First, perhaps one-third of people voting could only be reached for polling via cell phone – and they voted in a significantly different way.  Second, only perhaps 5% of the electorate reached the Labor Day start of the campaign with a changeable vote – again, a major difference from 20-40 years ago.  Gaffes, debates, Hurricane Sandy – it didn’t make that much difference. 

Given this, the Romney campaign’s one faint hope was voter turnout – unprecedented turnout for him in battleground states. And yet, there was not the slightest appreciation of this in the campaign.  Instead, the Romney campaign or related groups spent enormous sums on TV ads in the obvious markets, to the point where one station in Ohio was running more ads than programming. The crime wasn’t that this was wasted money; it was that relatively little money was spent on “get out our vote.”

In fact, most of the Romney campaign Big Data folks’ attention was on a series of seemingly baffling Obama ads in small markets targeting small demographics.  As a result, by election day, the Big Data folks were suffering from the “fog of war”, trying to counter something they did not understand.

Again, by contrast, the Obama campaign was focused on a new insight garnered from their use of Big Data to identify the “customer of one”.  More specifically, they realized that although most votes were unchangeable, smaller pockets of “conservative” voters could actually change their mind if policies they valued and prioritized were brought to their attention. These could be reached by, say, social-program-highlighting ads aimed at conservative women in more rural Ohio markets.

An under-appreciated part of these ads was that they contained real content.   After all, “conservative” voters were being saturated with generic Romney ads.  To succeed, the Obama ads needed to point to real, verifiable programs.

And now we come to Problem #3.

Problem #3, the Endgame: Big Data Tunnel Vision vs. a Loosely-Coupled Strategy

The end of every political campaign is “get out the vote” – maximize the number of ones supporters who actually vote on election day (and, of course, the increasing percentage who vote before then). And this is where the most startling difference between the campaigns’ approach to Big Data emerges.

It is apparent from various reports that on election day, the Romney campaign was prepared to focus its “turn out the vote” efforts on key demographics in battleground states, fine-tuned by what their Big Data analytics was telling them.  And then, right when voter turnout efforts were supposed to start, their feed from their system went down.  And it stayed down, for most of the day.

But what almost passes belief is what the campaign did about that.  Local offices were begging to go out and do “get out the vote” efforts anyway.  Instead, they were apparently told to wait until the system came up again.  Yes, theoretically, unfocused efforts could have done more harm than good.  Practically, however, it was overwhelmingly likely that “feet on the street” would have instead achieved slightly higher Romney turnout. And not only national but local campaign coordinators failed to realize that.

The problem, it appears, was that no one had autonomy, and all were focused on the Big Data part of the “get out the vote” strategy.  By contrast, the Obama campaign – partly due to an existing Internet-enabled strategy – granted local offices the ability to act proactively, and the Big Data “customer of one” focus was only part of a broader effort in battleground states to get out key Obama demographics, which were already well understood pre-Big Data.

Again, it is worth noting that this made little difference “on the day.” It is possible that it made a difference to several House seats; but not enough to overcome the effects of the 2010 election and its resulting Republican-dominated redistricting.  Note that there was apparently the biggest difference between the vote for each party (Democrats +1%) and the House seat allocation (Republicans +4%) ever recorded.

Implications for Organization Big-Data Use

We have seen, above, how the Romney organization used Big Data to shoot themselves in the foot – and yet, their strategies were superficially reasonable and well aligned with the practices in many businesses.  The immediate recommendations for IT and the enterprise are likewise relatively straightforward:

1.       Treat Big Data as a strategy, not as a “hot topic.”  Understand that for it to be successful, it must provide greater depth and a more accurate view of the customer and of one’s own organization, and those insights need to be translated to fine-tuning of strategies sooner rather than later.

2.       Focus on Big Data’s ability to understand customers not only more deeply as global “types” but in finer-grained groups, and ensure that the organization accepts a more realistic view of the customer.  Bluntly, the Romney campaign started believing their own propaganda; where have we seen that before?  And one reason was that Big Data was not telling them any different.

3.       Adopt an agile marketing strategy that does not hang on command-and-control top-down implementation. Agile marketing has a great respect for in-depth data.  However, it also has a great respect for the way that data reflects or fails to reflect actual customers, and customers who are constantly changing.

  And that brings us to our final point.  As I’ve noted, the success or failure of Big Data efforts turned out to matter surprisingly little to the outcome of this particular election.  However, Republican post-mortem efforts show that their failure to understand its implications has drastically delayed – and we are talking more than a decade here – their adaptation to changing American demographics.  They have created a party culture that makes it extremely difficult for them to move to anything more than permanent minority status nationally, because it carefully widened the divide with groups such as Hispanics in the name of older white-male “get out the vote.”

The point I am trying to make is that if Big Data cements an organization’s existing un-agile strategies in place, it is doing just as much harm as good – even though, for now, things are going better than ever.  The real value of Big Data, done well, is that it not only enables you to understand your customer better, but it also enables your organization to fit itself better to the customer – bottom-up as much as top-down – and both of them to understand how the customer is changing, not just what the customer is like now.

So what is it going to be?  Are you going to use Big Data to shoot yourselves in the foot, or to deliver better strategies, better implemented?  Inquiring Republicans want to know.