IBM Acquires Weather Company IT: Into the Data Unknown

The recent IBM acquisition of much of The Weather Company (TWC) – effectively, everything but The Weather Channel – is an odd duck.  I can’t remember anything quite like it in the computer industry before now.  Indeed, I suspect that IBM does not yet fully understand where in the search for better analytics via Big Data this new addition will take it.  To put it another way, IBM is stepping into an area where the ultimate use of the acquired data is to a surprising extent unknown.

The Potential Benefits of Launching Into the Data Unknown

As I noted in a previous piece about the initial IBM-TWC partnership, some core benefits are easy to see, and the acquisition simply extends them.  The infusion of analytics with weather data allows fine-tuning of customer and supplier behavior analysis and prediction – if it’s raining cats and dogs, the retail store may see more or less traffic than on a sunny day, and extreme weather will inevitably lead to decreased sales and deliveries.  Of equal long-term potential is the use of climate change science to avoid the “if this goes on” approach to weather forecasting that becomes increasingly unable to anticipate both secular year-round warming and increased occurrence of extreme events.  By taking over these functions, IBM allows its customers to drive weather-data acquisition as well as analytics in directions that a TWC-owned platform would likely not have done.

However, according to its presentation of the acquisition, IBM also views the TWC platform as a proven approach for leveraging Big Data in general for new insights, extending to the sensor-driven Web aka the Internet of Things (IoT) the innovative “cognitive” insights of IBM Watson.  I would argue, in fact that the TWC IT acquisition adds less – and more.  Less, because TWC is not on the cutting edge of the Fast Data “new technology wave” that I have written about previously.  More, because if and when it is combined with the raw “real-time sensor” data that governments collect about the weather and related location information (e.g., habitations, flora, fauna, and topography), it provides a simple “location state” of the individual customer or car or delivery truck that can allow as-yet-undefined risk reduction (avoiding the high-water spots in the road), “state of mind” assessment for selling purposes, and  services (rerouting based on likely washouts).

An Odd Way to Slice a Duck

Certain parts of the TWC acquisition put IBM effectively into areas that are not necessarily a good long-term fit – goodness of fit will depend on IBM’s execution of the acquisition. 

For one thing, acquiring the TWC web sites makes IBM effectively a media company.  Hurricane Sandy, in particular, created large followings for the blogs of certain weather forecasters.  While IBM can pretend that these are merely avocations for TWC personnel focused on The Weather Channel, the fact is that they represent an important news and analysis source for a significant segment of the public.  Whatever IBM does with the link between TWC content suppliers and the web sites will indicate whether IBM is going to run the media part of its slice of TWC into the ground or allow the weather forecasters and analysts that remain with TWC to continue leveraging their clout as experts into eyeballs and advertising revenue. 

And the question of who goes where brings up another area where IBM is venturing into new and not necessarily compatible territory:  weather and climate expertise.  As a recent IBM presentation citing weather data’s usefulness to insurance companies shows, applications to mobile car users that can reduce customer risks (avoiding or sitting out a storm) certainly have an upside to them, but what is not clear is whether the insurance company can use such a mobile app frequently enough to reduce overall customer risk substantially (and thereby increase profit margins). 

An obvious case where weather warnings may have an impact on customer risk profiles is a “black swan”, a seemingly unlikely event such as Hurricane Sandy.  However, in that case individual customer advice is likely to have only minor effect – rather, the weather experts must recognize the need for and drive proactive, outside-the-company warnings that ensure lots of customers take care.  TWC has such experts – but where is the place for them in IBM’s new weather-related organization?  Perhaps Watson will have such “domain knowledge” in the future; but it does not have it today, and so “weather analytics” badly needs such expertise.  And then weather analytics needs climate domain knowledge and climate change expertise, else predictive analytics applied to raw weather data will be far too frequently wrong.

The User Bottom Line:  Going Into the Data Unknown Is Good

I have dwelt on my concerns about IBM’s ability to get the most out of its new oddly sliced duck.  The fact remains, however, that, like Watson, a massive chunk of weather data is important not for its immediate applications, but for the ways in which users will drive new types of insights with it.  Just as Watson has spawned new “cognitive” ways to analyze data, driven by user use cases, so the early adopters of weather data will generate use cases that will suggest other applications that we cannot foresee.  The savvy user, therefore, will pick the right time and the right use case to begin to use weather data – and IBM’s TWC capabilities are an excellent place to start.

And if IBM figures out how to leverage TWC effectively in the areas that I have cited, so much the better for the user.  That the weather data will also lead IBM to get even more serious about climate change, I suppose, is too much to hope.  But hope I will.

We Have Blamed the Vendors and IT; Is It Now Time to Blame the Business Types?

Reflecting on the recent financial reports of the giants of computing infrastructure – companies like IBM, HP, Oracle, and Microsoft – I am struck by the continuation of a troubling trend:  The technology and its usefulness is moving ahead as fast as, if not faster than, ever before; IT has made some striking adjustments; but the revenues to vendors from this, and therefore IT spend, are flat if not falling. Why?

Let’s see.  In databases, Oracle and IBM are almost flat, despite (at least in IBM’s case) two technologies in two years that potentially deliver performance and price-performance gains well ahead of Moore’s Law.  I find plenty to criticize in Windows 8, but its implementation with regard to business needs, in Windows and Office usage, should deliver benefits comparable to that of the introduction of the mouse-based user interface – and Microsoft enterprise revenues are flat.  Cloud technology is not replacing, but rather complementing, in-house architectures, and therefore should require, initially, more rather than less total spending – and yet, increases in cloud spending are instead paired with overall flat or lower user computing spend.  No, I don’t buy the “public cloud saves money” rationale – the benefits come later rather than sooner, and involve mainly flexibility and to a much lesser extent multi-tenancy and grid savings, especially when many users are employing multiple cloud providers that must be coordinated.  And the IBM and Oracle/SPARC data suggest the biggest hardware decline is in Unix/Linux hardware, the darling of the cloud set, not in mainframes or PCs. 

In the past, a typical explanation for this has been the need for the business to cut back IT spend in response to decreasing profits.  Of course, that did not prevent IT and computing from becoming a larger and larger part of overall business spending during the 1980s and 1990s.  In the early 2000s, IT spend actually decreased proportionally to the cost-cutting of the rest of the business – and yet, despite an initially tepid overall business recovery, computing spending from about 2003 to 2008 rebounded and grew quite briskly. So this pattern of user computing spending is unprecedented in three ways:  its length, its unremitting focus on cost-cutting apparently comparable in size to the rest of the business, and its seeming inability to reflect major technology advances with a good claim to deliver major cost or other benefits to the rest of the business.

The Blame Game

In the past, I have found, when things go wrong in computing spending the first reaction is to blame the vendors.  In this case, the obvious critique is that they have failed to communicate the benefits of the technology, cost and otherwise.  Except that, as I can attest from my experience as an analyst, both the technology advances and the communications from the vendors are as good as, if not in many cases better than, those of most or all of the past.  Big Data, for example, is not pure hyperbole, and vendors have done a reasonable job by traditional standards of highlighting the ability of targeted Big-Data analytics to explain and bind the customer as never before, in a cost-effective way. 

OK, then the next line of analysis is to blame IT, typically for failing to align themselves with the business’ strategy, or (when times are tough) identify ways of lifting the dead hand of older systems and other seemingly outdated costs.  To this, I have one response:  agile IT.  As never before, IT is aligning itself with agile development, by supporting a software lifecycle that includes operational feedback integrated with development, and by encouraging automation of functions that provides a basis for rapid response and identification of architectural problems and opportunities at the administrator level.  As my studies have shown, these plus the increasingly agile nature of the software that IT makes available to the business translate into major cost-cutting and major benefits beyond what traditional IT approaches can deliver.  And as for aligning itself with business strategy, IT is well aware of Big Data as a hot topic, hence the sudden and somewhat odd demand for Hadoop experts.  No, whatever IT may have done in the past, it seems clear that it has upped its game.

OK, then, who does that leave?  And yet, how can we blame the business types?  Aren’t these the same folk who drove those increases in IT percentage of spending over the 1980s and 1990s?  Aren’t these the most receptive listeners when we talk about the power of embedded analytics to improve business processes and the importance of analyzing the “customer of one”?

There’s Something Not Going On Here …

To understand why there might be cause for concern and, yes, for blame in business strategy – driven by the CEO, the CFO, and their staff – let’s look in very broad-brush terms at what has been going on, not just for the last five years, but for the last three decades.  Here are a few of the highlights, as suggested by various studies:

1.       Advances in productivity have been accruing 90% to the CEO – ½ in his income as CEO, and ½ in his increases in wealth as an investor.  While this still represents a fairly small percentage of overall expense, it does unnecessarily increase the focus on cutting costs for all except the CEO.

2.       Hedge funds that sometimes act as turnaround artists or “shadow banks” have taken a good 90% or so of the funds’ investment profits for themselves, and thus receive compensation not just in the tens of millions of dollars but, at the top, in billions of dollars, despite the fact that they deliver less to the investor than an index fund in most if not all cases.  While it is difficult to see direct effects of this “fleecing of the rich” on corporate strategies, one significant effect is to increase the fleeced CEO’s focus on growing profits uber alles, to at least get some return from their investments – with the Googles of the world the few happy exceptions.

3.       The present long-lasting recession/stagnation has effectively removed perhaps 10% of the workforce from work over a long period of time.  This has the effect, since businesses often discriminate against the long-term underemployed, of creating a permanent gap between “potential” GNP and real GNP.  To put it another way, even if the economy grows at a reasonable pace, it will still be much smaller it should be.  That means smaller markets and, again, more relative emphasis on cost-cutting to achieve profit growth rather than growing revenues to grow profits.

There are three concerns about this excessive focus on cost-cutting.  The first, which probably is not serious right now but is far more troubling than five years ago, is that cost-cutting eventually runs out as a strategy if revenues continue to be flat (and, don’t forget, revenues are pretty flat for the bulk of businesses, hence the 1% growth in GNP over the first part of this year).  The second concern is that this cost-cutting, applied economy-wide (as it seems to be doing), reinforces and cements smaller markets by eliminating the part of the market funded by the now underemployed.  Ordinarily, recessions are too short for this to happen; but it seems to be happening now, as new jobs continue to make no dent in the workforce/working-age-population ratio.

The third concern, which I for one find the most troubling, is the possibility that reflexive cost-cutting becomes the answer to every situation, the reflex of the business.  If that is true, it would suggest that businesses are blowing opportunities for savvy increases in computing technology spend because it’s all about cost-cutting, not strategic investment. 

Blame or Not, What Might Business Strategists Do Better?

First, I would suggest that businesses set up a long-term plan and process for using analytics to better understand and improve the relationship with the customer.  This means acquiring data virtualization software and the like, to allow aggressive searching out and combination of all the new customer information constantly being created.  It also means buying the new infrastructure tools (database and information architecture) that can scale to handle aggregation of social-media data on the Web and in-house customer-experience data. 

Second, I would argue (not that I expect many businesses to listen) that in order to grow revenues as well as profits (and probably grow profits faster), business types need to acquire and foster use of agile tools and processes – such as agile marketing, with its emphasis on data-driven understanding of the customer and prospect rather than the anecdotal, top-down opinion that has been all too common in the past.  As I noted in a recent blog post, an agile business is a possibility, if the CEO and his lieutenants really commit to it – and it typically means not only the CEO using a Scrum-type planning process, but also constant modification and constant feedback up and down the organization about plans.  I should also note that an agile process specifically builds in slack for learning and review, and yet (according to my studies) it produces much better cost-saving and profit results than a “cram as much work as possible in, be as cost-efficient as possible” approach, for the CEO as well as the developer.

And then there is the really controversial stuff. Sustainability efforts are, by and large, really underperforming, as a recent article in www.thinkprogress.com confirms my impression that gains in the US in carbon emissions are effectively coming from offshoring the problem.  Businesses need to acquire effective global carbon-tracking software, and use it.  Businesses need to join to push shared quality regulations that will move all towards new computer and software technology that is more productive, rather than clinging to the infrastructure of the past that in the long run costs more, as in the electrical/energy grid vs. software-coordinated regional combined solar/wind – and rather than enabling the block-all-regulation political excesses of the U.S. Chamber of Commerce and its ilk.

I’m not saying that all this is doable, or that the case for questioning the approach of the overall business to computing technology is clear.  I am saying that it’s time we recognize the seriousness of the overall problem of which computing spend is a part, and we stop blaming the usual suspects.  Five years is enough.  Thirty years is enough.  In the long run, a cost-cutting strategy is not enough – and the long run is beginning to arrive.   

 

Energy Usage and IT: Is the Name Not a Clue?

At the IBM Systems and Technology Group analyst briefing two days ago, IBM displayed three notable statistics:

1. The global amount of information stored has been growing at 70-100% per year the last 5 years, with the result that the amount of storage has been growing by 20-40% per year;

2. The amount of enterprise expenditures for datacenter power/cooling has grown by more than 10-fold over the last 15 years, with the result that these expenditures are now around 16% of system TCO – equal to the cost of the hardware, although well below the also-rising costs of administration;

3. Datacenter energy usage has doubled over the last five years.

These statistics almost certainly underestimate the growth in computing’s energy usage, inside and outside IT. They focus on infrastructure in place 5 years ago, ignoring a highly likely shift to new or existing data centers in developing countries that are highly likely to be more energy-inefficient. Also, they ignore the tendency to shift computing usage outside of the data center and into the small-form-factor devices ranging from the PC to the iPhone that are proliferating in the rest of the enterprise and outside its virtual walls. Even without those increases, it is clear that computing has moved from an estimated 2 % of global energy usage 5 years ago to somewhere between 3 and 4%. Nor has more energy usage in computing led to a decrease in other energy usage – if anything, it has had minimal or no effect at all. In other words, computing has not been effectively used to increase energy efficiency or decrease energy use by more than marginal amounts – not because the tools are not beginning to arrive, but rather because they are not yet being used by enterprises and governments to monitor and improve energy usage in an effective enough way.

And yet, there have been voices – mine among them – pointing out that this was a significant problem, and that there were ways to move much more aggressively, since the very beginning. I remember giving a speech in 2008 to IT folks, in the teeth of the recession, stressing that the problem would only get worse if ignored, that doing something about it would in fact have a short payback period, and that tools for making a major impact were already there. Here we are, and the reaction of the presenters and audience at the STG conference is that the rise in energy usage is no big deal, that datacenters are handling it just fine with a few tweaks, and that IT should focus almost exclusively on cutting administrative costs.

All this reminds me of a Robin Williams comedy routine after the Wall Street implosion. Noting the number of people blindly investing with Bernard Madoff, pronounced “made off” as in “made off with your money”, Robin simply asked, “Was the name not a clue?” So, I have to ask, “energy usage”: is the name not a clue? What does it take to realize that this is a serious and escalating problem?

The Real Danger

Right now, it is all too easy to play the game of “out of sight, out of time, out of mind.” Datacenter energy usage seems as if it is easily handled over the next few years. Related energy usage is out of the sight of corporate. Costs in a volatile global economy that stubbornly refuses to lift off (except in “developing markets” with lower costs to begin with), not to mention innovations to attract increasingly assertive consumers, seem far more urgent than energy issues.

However, the metrics we use to determine this are out of whack. Not only do they, as noted above, ignore the movement of energy usage to areas of lower efficiency, but they also ignore the impact of the Global 10,000 moving in lockstep to build on instead of replacing existing solutions.

Let’s see how it has worked up to now. Corporate demands that IT increase capabilities while not increasing costs. The tightness of the constraints and the existence of less-efficient infrastructure causes IT to increase wasteful scale-out computing almost as much as fast-improving scale-up computing, and also to move some computing outside the data center – e.g., Bring Your Own Device – or overseas – e.g., to an available facility in Manila that is cheaper to provision if it is not comparably energy-optimized at the outset. Next year, the same scenario plays out, only with even greater costs from rebuilding from scratch a larger amount of existing inefficient physical and hardware infrastructure. And on it goes.

But all this would mean little – just another little cost passed on to the consumer, since everyone’s doing it – were it not for two things; two Real Dangers. First, the same process impelling too-slow dealing with energy inefficiency is also impelling a decreasing ability of the enterprise to monitor and control energy usage in an effective way, once it gets around to it. More of the energy usage that should be under the company’s eye is moving to developing countries and to employees/consumers using their own private energy sources inside the walls, so that the barriers to monitoring are greater and the costs of implementing monitoring are higher.

Second – and this is more long-term but far more serious – shifts to carbon-neutral economies are taking far too long, so that every government and economy faces an indefinite future of increasing expenditures to cope with natural disasters, decreasing food availability, steadily increasing human and therefore plant/office/market migration, and increasing energy inefficiency as heating/cooling systems designed for one balance of winter and summer are increasingly inappropriate for a new balance. While all estimates are speculative, the ones I think most realistic indicate that over the next ten years, assuming nothing effective is done, the global economy will reach underperformance by up to 1% per year due to these things, and up to double that by 2035. That, in turn, translates into narrower profit margins due primarily both to consumer demand underperformance and rising energy and infrastructure maintenance costs, hitting the least efficient first, but hitting everyone eventually.

The Blame and the Task

While it’s easy to blame the vendors or corporate blindness for this likely outcome, in this case I believe that IT should take its share of the blame – and of the responsibility for turning things around. IT was told that this was a problem, five years ago. Even had corporate been unwilling to worry about the future that far ahead, IT should at least have considered the likely effects of five years of inattention and pointed them out to corporate.

That, in turn, means that IT bears an outsized responsibility for doing so now. As I noted, I see no signs that the vendors are unwilling to provide solutions for those willing to be proactive. In the last five years, carbon accounting, monitoring within and outside the data center, and “smart buildings” have taken giant leaps, while solar technologies at whatever cost are far more easily implemented and accessed if one doesn’t double down on the existing utility grid. Even within the datacenter, new technologies were introduced 4 years ago by IBM among others that should have reduced energy usage by around 80% out of the box – more than enough to deliver a decrease instead of a doubling of energy usage. The solutions are there. They should be implemented comprehensively and immediately, as, by and large, has not been done.

Alternate IT Futures

I am usually very reluctant to criticize IT. In fact, I can’t remember the last time I laid the weight of the blame on them. In this case, there are many traditional reasons to lay the primary blame elsewhere, and simply suggest that IT look to neat new vendor solutions to handle urgent but misdirected corporate demands. But that begs the question: who will change the dysfunctional process? Who will change a dynamic in which IT claims cost constraints prevent it from “nice to have” energy tools, while corporate’s efforts to respond to consumer “green” preferences only brush the surface of a sea of energy-usage embedded practices in the organization?

Suppose IT does not take the extra time to note the problem, identify solutions, and push for moderate-cost efforts even when strict short-term cost considerations seem to indicate otherwise. The history of the past five years suggests that, fundamentally, nothing will change in the next five years, just as in the past five, and the enterprise will be deeper in the soup than ever.

Now suppose IT is indeed proactive. Maybe nothing will happen; or maybe the foundation will be laid for a much quicker response when corporate does indeed see the problem. In which case, in five years, the enterprise as a whole is likely to be on a “virtuous cycle” of increasing margin advantages over the passive-IT laggards.

Energy usage. Is the name not a clue? What will IT do? Get the clue or sing the blues?

Don't Cloud the Computing Carbon Emissions Reduction Issue

I recently read a post by Jon Koomey, Consulting Professor at Stanford, at www.climateprogress.org, called “4 reasons why cloud computing is efficient”. He argues (along with some other folks) that cloud computing – by which he apparently means almost entirely public clouds – is much more beneficial for reducing computing’s carbon emissions than the real-world alternatives. As a computer industry analyst greatly concerned by carbon emissions, I'd like to agree with Jon; I really would. However, I feel that his analysis omits several factors of great importance that lead to a different conclusion.

The study he cites compares the public cloud -- not a private or hybrid cloud -- to "the equivalent". It is clear from context that it is talking about a "scale-out" solution of hundreds and thousands of small servers, each with a few processors. This is, indeed, typical of most public clouds, and other studies have shown that in isolation, these servers do indeed have a utilization rate of perhaps 10-20%. However, the scale-up hundreds-of-processors servers that are a clear alternative, and which are typically not used in public clouds (but are often used in private clouds), have a far better record. The most recent mainframe implementations, which support up to a thousand "virtual machines", achieve utilization rates of better than 90% -- a three times better carbon efficiency than the public cloud, right up front.

The second factor Jon omits is the location of the public cloud. According to Carol Baroudi, author of "Green IT For Dummies", only one public cloud site that she studied is located in an area that has a strong record of electricity that is carbon-emission-light (Oregon). The others are in areas where the energy is "cheaper" because of fossil fuel use. That may change; but you don't move a public cloud data center easily, because the petabytes of data stored there to deliver high performance to nearby customers doesn't move easily, even over short distances. Corporate data centers are more movable, because the data storage sizes are smaller and they have extensive experience with "consolidation". While until recently most organizations were not conscious of the carbon-emission effects of their location, it appears that companies like IBM are indeed more conscious of this concern than most public cloud providers.

The third factor that Jon omits is what I call "flight to the dirty". High up-front costs of more efficient scale-up servers leads unconsciously to use of less energy-efficient scale-out servers. Controls over access to public and private clouds and data centers, and visibility of their costs, moves consumer and local computing onto PCs and smartphones. Apparent cheapness of labor and office space in developing nations leads companies to rapidly implement data centers and computing there using existing energy-inefficient and carbon-wasting electrical supplies. All of these "carbon inefficiencies" are not captured in typical analyses.

Personally, I come to three different conclusions:

1. The most carbon-efficient computing providers use scale-up computing and integrated energy management, and so far most if not all of those are private clouds.

2. The IT shops that are most effective at improving carbon efficiency in computing monitor energy efficiency and carbon emissions use not only inside but outside the data center, and those inevitably are not public clouds.

3. Public clouds, up to now, appear to be "throwing good money after bad" in investing in locations that will be slower to provide carbon-emission-light electricity -- so that public clouds may indeed slow the movement towards more carbon-efficient IT.

A better way of moving computing as a whole towards carbon-emission reductions is by embedding carbon monitoring and costing throughout the financials and computers of companies. Already, a few visionary companies are doing just that. Public cloud companies should get on this bandwagon, by making their share of carbon emissions transparent to these companies (and by doing such monitoring and costing themselves). This should lead both parties to the conclusion that they should either relocate their data centers or develop their own solar/wind energy sources, that they should move towards scale-up servers and integrated energy management, and that they should not move to less costly countries without achieving energy efficiency and carbon-emission reduction for their sites up front.