The Other Sad Task of Combating Climate Change

Writing this kind of blog post tends to freeze my brain.  I find it astounding sometimes to be trying to present in a logical and calm fashion a description of horrors.  But there it is.

I noticed in perusing comments in various climate-change-related web sites that even among fairly well-informed folks there seems to be a misperception, which runs something like this:  The job of combating climate change is about slowing carbon emissions, preferably as quickly as possible.  As I understand it, that is half right.  There is another distinct task:  leaving at least a significant amount of carbon-emitting “fossil fuels” in the ground – forever, or at least for the next 100-1000 years.  Moreover, that task assumes that we do not discover major new sources of oil, natural gas, and coal.  If we do, then we need to leave the equivalent of a significant amount of present “reserves” plus all reserves discovered in the future in the ground.

One implication of this:  we need to understand that there is a Hard Stop somewhere in the future, a point beyond which we dare not use even one milligram more of fossil fuels.  If we cut carbon emissions drastically in the near future, and keep them cut, that Hard Stop almost certainly will never arrive – instead, we will suffer various degrees of what Joe Romm calls “Hell and High Water”, involving at worst the decimation (not in the Roman sense – in the sense that 9/10ths of humanity will die, mostly of starvation, disease, and poisoned air) of humankind.  If we continue on the present path of fossil-fuel use increases and minor moves towards “sustainability”, the so-called “business as usual”, that Hard Stop may even arrive by the end of this century.  That Hard Stop represents absolutely no further use of fossil fuels because the alternative might be the end of all life on earth, forever.

How can I say this?  How can I not be wildly exaggerating, in Mark Twain’s sense (“the reports of my death are wildly exaggerated”)?

Keystone and Game Over

It may strike people as odd that there is such an environmental furor over one oil pipeline project in the US (the Keystone XL proposal).  Here’s a frequently cited quote (paraphrased) by Dr. James Hansen on the subject:  “If Keystone XL goes forward, then it’s game over for the climate.”  Most people, my sense is, read that as meaning that some form of “Hell and High Water” becomes inevitable.  I believe that instead, he is also referring to a previous quote (in, I believe, his book Storms of My Grandchildren, and also paraphrased):  “If we use all our present reserves of coal and oil, there is a significant chance of a runaway greenhouse effect.  If we also use all our tar sands and oil shale, I view the runaway greenhouse effect as likely.” Before I explain my understanding of this, let’s note that Keystone XL transports oil derived from Canadian tar sands to US ports for export abroad.

What’s a runaway greenhouse effect?  If we look at Venus, we see a planet with extreme heat and with acid rain that dissolves any life forms that might exist in the air, and then evaporates before it reaches the surface.  However, if Venus had no atmosphere, there would be no extreme heat and no acid rain.  Instead the temperature would be a significant distance below the “runaway point” (estimated by Dr. Hansen at somewhere around 62 degrees Fahrenheit, iirc). Carbon or other substances in the atmosphere reflect light-generated heat bounced from the surface back to the surface again, trapping it – and also increasing the acidity of water (again, as I understand it).  If Earth passes that “runaway point”, then we will become like Venus.

Now, Earth without an atmosphere would be far below the temperature of Venus – below freezing, actually.  The atmosphere adds one layer of carbon-based reflection or “trapping” of heat (yes, I realize I’m simplifying drastically).  Life itself – all life, especially vegetable – adds another.  Life is carbon-based, and it creates a carbon cycle that emits carbon to the atmosphere, and then absorbs it in non-organic matter when it returns, via a process called “weathering” that deposits much of the carbon returned into the oceans. In ordinary times, this creates a way of handling perturbations in carbon emissions so that one returns eventually to somewhat of a “steady state”. And that “steady state” is still clearly under the “runaway point.”

Now here is where we get to the importance of leaving some fossil fuels in the ground.  Because we have seen “Hell and High Water” in the past, and life has been decimated but survived.  But what are fossil fuels, really?  Primarily the carbon deposited in the ground by life – especially vegetable life – over the last up to a billion years or so.  Now compare this episode of carbon emissions to all past episodes.  We have seen surges in carbon emissions from the Milankovitch cycle before, and from long-term underwater eruptions that bring new carbon up from the Earth’s core to the air.  We have seen methane spurts due to accompanying thawing of places like the Arctic that may have made the temperature rises and carbon in the atmosphere more extreme.  What we have never seen before is taking all the stored carbon for hundreds of millions of years and injecting it into the atmosphere over what could turn out to be a period of 200 years (and carbon has a half-life of perhaps 100 years in the atmosphere). 

And Hansen’s best estimate is that use of all of that stored carbon over a period even of much longer than 200 years is likely to bring on a runaway greenhouse effect.  This is because the ocean is the primary way of restoring equilibrium to the system, and at some point before we use up all that carbon, if we do it fast enough now, the ocean stops being able to absorb as much carbon (apparently, according to Wikipedia, because of the slowing of the “biological pump”) – and carbon coming down from the atmosphere cycles right back up again.  And so, once that point is reached, carbon doesn’t cycle very much back into the ocean – it goes on accumulating in the atmosphere, for a thousand years or more, until the ocean begins to regain its ability to absorb carbon. Thus, as we get close to the “runaway point”, we can’t just slow carbon emissions down to a point at which as much carbon is returning as is being emitted – the only point at which that is true is near-zero emissions, a Hard Stop.

Now, hopefully, you begin to see why it’s important to leave significant amounts of fossil fuels in the ground for at least 100-1000 years:  it keeps us away from that Hard Stop, and hence that “runaway point.”  It keeps us away from the ultimate horror.

So why is Keystone XL so critical to this?  There is at present no real market for tar sands oil.  There are very high up-front costs, which only the Canadian government has taken on so far – and no one appears likely to, in the immediate future, if the Canadians don’t succeed.  The only realistic way of getting that oil from inland Canada to a decent market, it appears, is to add to existing pipelines and send it to ports in the southern US – all other routes appear to involve too-large costs and times of building new infrastructure – and further carbon emissions from tar sands oil will be minimal. If Keystone XL goes through, it appears likely that a significant portion of the world’s tar sand oil will be emitted over the next 40 years – if not, not.

That’s why Keystone XL matters.  That’s why Hansen has been campaigning for several years to stop most worldwide production of coal, as the least painful way of avoiding the “significant chance” of a runaway greenhouse effect. That’s why people need to think about handling climate change today as not simply a matter of adaptation to “Hell and High Water” or slowing down carbon emissions by a couple of percentage points per year right now.  It has now reached the point where we need to face the idea of effectively never using some of those fossil fuels – not just letting the market assume that using it all is OK.

Action Items and Dyslogy

What our sad other task of facing climate change amounts to, therefore, imho, is not only to stop Keystone XL in its tracks.  It amounts to making sure that Keystone and its ilk never happen.  It also amounts to trying to ensure, with each future use of fossil fuels, that a comparable amount of reserves is made unusable, effectively, forever (or until 1000 years from now, whichever comes first).  And it means keeping an eye on new sources of fossil fuels, to limit their use sharply forever.

And if we fail?  I suppose we can write a eulogy for life on Earth.  Except that writing a eulogy for a species that ended life forever seems a bit off, somehow.  The opposite of Utopia is dystopia; I guess we should write a dyslogy.  Someone recently passed me the end of a Swinburne poem that seems to fit – it even includes the sea rise that’s an initial stage.  I have changed one word.

Here death may deal not again for ever;

       Here change may come not till all change end.

From the graves they have made they shall rise up never,

       Who have left nought living to ravage and rend.

Earth, stones, and thorns of the wild ground growing,

       While the sun and the rain live, these shall be;

Till a last wind's breath upon all these blowing

               Roll the sea.

Till the slow sea rise and the sheer cliff crumble,

       Till terrace and meadow the deep gulfs drink,

Till the strength of the waves of the high tides humble

       The fields that lessen, the rocks that shrink,

Here now in his triumph where all things falter,

       Stretched out on the spoils that his own hand spread,

As a god self-slain on his own strange altar,

               Man [Swinburne – Death] lies dead.

GoodData and Grey's Anatomy: It's Not Always the Vendor's Fault

Recently I received information about an interesting new vendor called GoodData – it appears that they are specializing in BI-type user interfaces for function-specific dashboards (of course, in the cloud and involving cross-database information display).  One of these was GoodSupport Bash, a “bash-up” of customer-support metrics and KPIs (Key Performance Indicators).  And that caused me to reflect on the idea of applying these types of “comprehensive” metrics to customer support.

Customer-support technology has a relatively long software-vendor history:  I remember first seeing the idea in action back in 1982, when Computer Corp. of America’s email system was used to track customer inquiries. Our forms capability allowed us to enter the call info in an email, send it for resolution to the appropriate techie, and then see the “email trail” as the problem was resolved.  Like many small companies, we had a reputation for excellent customer support because the techies were developers who had spare time to actually solve problems and the clout to change the code base to do so.  In fact, I was once told that one “guru” would go to install at new sales and when the customer asked for changes, he would make them without documentation or changing the main code base, so that the next version overrode the customization – it made the customer very happy initially, and very annoyed a year later.

However, in the early 2000s, when offshoring came in, major-vendor customer support was one of the first to move, and the semi-mature customer-support software of the day now had to define support personnel tasks very carefully, as the new Indian and Filipino respondents had some difficulties with American English and more difficulties with being up to date with software technology.  The resultant upheaval in customer support has never really settled down again, imho.  Companies like Microsoft would really like to move away from live-person customer support entirely, but the complexity of the technology means that users keep having to ask questions of live representatives who can be forced to pay attention.  For example, it was not until I got to a live representative that I realized that the Windows 8 version of Windows Media Player does not support DVD playing out of the box (according to PC World, you need the $10 Media Pro or some such).

Well, this in turn meant that phone-based customer support had to be “optimized” in some way, else waits of 2 hours and unresponsive representatives would cause a significant black eye for the company providing the support for its products.  And so, solutions like GoodSupport Bash allow companies to “take the temperature” of their customer support according to all sorts of criteria, like speed to reach a representative, time taken by the representative, and whether the representative is taking the opportunity for up-sell or cross-sell.

Except that I believe that the results of this kind of fine-tuning according to various management theories can sometimes actually be counter-productive – and the fault lies not with the customer-support software vendor.

Grey’s Anatomy and Ice Removal

I must confess that Grey’s Anatomy is one of my guilty pleasures, and I often find it difficult to watch without rotfl.  To me, GA is the ultimate in actor torture.  In just about every episode, at least one and often several actors and actresses must somehow make credible a complete 360-degree turn in their characters, with the words they are given to do this stretching the limits of belief that anyone could talk in this way.  As character after character changes partners or sexual orientation on the verge of a marriage and in the middle of a surgery, I can almost imagine the reaction as that actor or actress sees the week’s script – bracing themselves, and yet unable to anticipate the next contortion.  Although there are limits:  no one has taken a sudden interest in animal husbandry – yet.

Anyway, in this episode, doctors facing a hostile takeover of the hospital went to find out what this new highly-efficient corporation was doing, and posed as patients.  The doctor started reading from a scripted set of questions, and when the “patients” attempted to derail them, started the script over again until they could get it done.  This was done, we are assured, in the name of standardized efficiency.  You will recognize the similarity to some customer support experiences.

However unlikely the situation, the GA actors and actresses, trained to make the unlikely plausible, made it very clear how customer support driven by metrics appears to the customer.  The caller reporting something that often is unusual must be fit into a series of questions that attempts to cover only the usual.  Callers who do not fit are effectively ignored – you could share the frustration as the “patient” attempting to vaguely describe symptoms is asked to spend much time talking about things that are not problems, and the feeling that no one is listening.  At one point, the “patient” confesses that he is a doctor, pretends he’s already a new part of the company, and asks why these procedures; the “representative” assures him in a scripted manner that the procedures are more effective – except it’s clear that he must answer that way, no matter what he thinks, or lose his job.

Sound implausible?  Well, I actually went through a similar experience the other day, when I tried to see if a national gutter specialist could provide snow and ice removal on an emergency basis.  Sure enough, I wound up with a representative who took some time to realize that I was not calling about gutter cleaning, finally said that the company did not provide the services, and then spent three minutes attempting to upsell me on a bigger gutter cleaning contract, despite the fact that (a) I said twice that I was not going to do so, and (b) I had just been told that they didn’t do something else I needed.  It was clear that this was one of the metrics by which he was judged:  did you run through all the questions?  Did you upsell? You can imagine what I felt about the company afterwards – and it wasn’t the representative’s fault.

Sloan Management Review, iirc, had some interesting data on the effect of this type of thing on customer loyalty.  Often, the most loyal or biggest-spending before the disappointment were the quickest to jump ship – they felt “entitled”, as it were.  But even the truly loyal needed customization that truly paid attention to their needs, else they too found it difficult to stay.  Branding only goes so far.

My point here, however, is that none of this was the customer-support software vendor’s fault.  On the contrary, whether the company’s support strategy is good or bad, the typical indication from users is that something like GoodData Support Bash will make it more efficient.  If the strategy is bad, however, it may also make it less effective, or more harmful.

The Bottom Line:  Metrics Is As Metrics Does

It is now a truism of management theory that metrics create behavior – employees game everything to their own advantage, or to minimize their own disadvantage.  And so, users of customer-support software like GoodData Support Bash badly need to remember that the most effective use of any customer-facing tool is not to create representatives who will deliver the company’s idea of customer happiness via efficiency, but to find analytics to understand the customer better.  Only after that knowledge is gained and the company has used that knowledge to improve customer support responsiveness to real customer needs should efficiency metrics be considered.

To put it more concretely:  among the metrics in your dashboard, is at least one telling you honestly the degree of customer satisfaction with the interaction?  And are you mining data from that metric telling you what’s good and bad about your present solutions?  If not, are you really making things better with your improvements in customer-support efficiency?

Or are you making them worse?

Life in a Software World

Various tech-industry “visionaries” have proclaimed that we are entering an “age of software”, pointing to the importance of software to today’s solutions and the world economy.  By and large, I agree that “all things software” is more and more a differentiator among companies, a focus of work, and a prevalent element of life outside of work.  However, it seems to me that no one has fully defined just what life in a “software world” will mean – what it has meant for those who participated in its formative stages.

Below, I lay out five things that I believe are worth considering as unique characteristics of enterprises in a “software world”.  They are my own point of view, based on 12 years in firms as a software developer informed by the business theory I learned at Sloan, plus 22 years as a computer industry analyst.  Everyone will have his or her own list; I’d just like these potential aspects of the “software world” to be considered along with the rest of those lists.

Project Management, Not Manufacturing

In a typical manufacturing firm, the norm was and is production of pieces of hardware.  In a so-called “services” firm, the norm is delivery of pre-designed service “solutions”, and therefore there is a reasonable analogy to manufacturing.  In a software firm, the cost of actually creating copies of a software program is pretty close to zero.  Instead, the focus is on creation of the next version or fixing bugs in this version.  In the manufacturing/service firm, the focus is often on optimizing the process of producing the same thing over and over.  In the software firm, the focus is on developing new features so customers don’t walk away.  And so, these firms seek to optimize new-product development – that’s the critical success factor for a software firm.  Does anyone suppose that if Google had simply cut the costs of servicing its search engine to the bone as its main focus, it would have survived, much less thrived?

But if ongoing success is a matter of new-product development, then it follows that success comes from successful development-project management, not optimization of the supply chain.  Apple grows profits and revenues while most if not all hardware/service companies achieve only flat revenues and lesser profit growth, and clearly the iPhone and iPad rather than Mac production optimization explain this. 

In fact, I believe that this changeover is also happening in so-called manufacturing and service companies.  50 years ago, it was still possible to say that the bulk of workers in a company were production workers, with support staff a poor second.  Today, pick any company and you have far more people in clerical and new-product development plus management roles, and those “clerical” and “managerial” tasks like administrative assistant, web designer, and product marketer are to a much greater degree about producing new solutions.  And so, the management of innovation projects is becoming much more important in traditional firms.

Accounting:  No Inventory, No Capital Stock

All right, that’s a bit of an overstatement, but not much.  Software requires very little if any hardware to produce these days:  just download it from a web site a small chunk of whose servers you have leased from a public cloud.  That cost is there no matter how many copies are downloaded of the single stored copy there.  What widgets of inventory?  What capital stock of hardware-producing machines and factory buildings? What FIFO vs. LIFO?

It seems to me that this software world therefore puts our favorite financial metrics out of whack.  Is inventory turnover really telling us that the firm is about to fail?  How about sales turnover?  If we are investing more in capital than in labor, aren’t we optimizing a non-existent manufacturing process rather than new-product development?  So are our metrics telling us, not that a firm is succeeding wildly, but that it is headed for failure?  And how do we tell what a successful new-product investment strategy is from our books?  For the last 20 years, since the Harvard Business Review suggested computer companies’ futures were in services rather than hardware, IBM has been focusing on both innovation and services rather than hardware, with strong metrics and some profit growth – but, especially lately, revenues have been flat overall, with only software showing strong growth over the entire period, and services effectively beginning to prosper only when IBM delivers innovative software. 

I confess that I have no strong sense for what the new accounting and the new financial metrics should be.  There needs to be some way to measure new-product development success and detect failure, and isolate it in the company’s books – but my textbooks tell me that projecting the revenues from software development is highly speculative.  Maybe so; or maybe Microsoft does have at least some handle on how many copies a new version of Windows will sell, so it’s not as bad as all that.

It’s About Growth, Not Optimization

One of the biggest shocks in the two software companies I worked in for 4+ years was the way that expectations escalated.  Every year was the baseline for the next year, no matter how good, and I was expected to do more in some way:  produce more code, do more tasks, whatever.  In fact, after the first two companies I started to assume a general curve, in which the 3^rd year was necessarily disappointing to my bosses – who, by the way, were never the same at the end of the year as they were at the beginning – and sometimes I wondered if the fourth simply laid the grounds for the end of my employment.  It wasn’t that I wasn’t producing; it was that I wasn’t producing that much more. And, in point of fact, until a change of strategy my fourth year there, it looked as if Aberdeen Group was going the same way.

Recently, I met a car salesman who, as in many traditional jobs, had been there for many years.  It was clear that he was not constantly faced with rising expectations; the fact that he continued to excel compared to others was reason enough to keep him.  Clearly, an auto dealer is not yet a software company. Why the difference, I keep asking myself? 

I would suggest that in a software world, optimization of the machines that support a process simply does not speed up software development significantly – and yet, managers want to grow both revenue and profits.  The obvious answer:  everyone must do more. No matter that it does not fit the mold of unpredictable new-software development, or that it is a one-size-fits-all approach to labor; it must focus on growth of production rather than accepting optimized production, in order to conform to the expectations of the firm.

I don’t say this is good or bad, although I have my opinions.  It does suggest to me, however, some reasons why agile software development seems so inefficient and yet produces such great bottom-line results.  That is, agile development removes the ability to demand more from each person – it changes the metric from lines of code per day or some such to customer satisfaction.  And so, the company can have its bottom-line growth without efficient, ineffective programmer optimization. Because today, more and more companies, like software companies, are thinking dynamic, not static:  seeking growth in unpredictable markets with changing consumers, not seeking to optimize the supply chain or manufacturing process in more stable markets.

It’s About the New New Thing, Not the Success

In some ways, this is a repetition of my first assertion.  In a software world, especially a “virtual” one, lack of hardware means relative lack of chains to keep the customer attached – or to make the customer feel trapped.  It amuses me to see the venom attached to Microsoft Office as a boat anchor for innovation, when in point of fact it is far more innovative than, say, the gas station or the fashion industry.  And that’s the point:  software firms have a significant amount of new content in each version of each product, because they have to.  They don’t focus on success by standing still or recycling:  they focus on adding features.

I see this as a real problem for our present foundations of microeconomics.  I believe that Paul Krugman wrote recently that static manufacturing firms producing widgets in microeconomics indeed did badly as a model of real firms and markets, but that was OK, because it captured the essentials needed for effective macroeconomics. I question whether that is still true in a software world.

Specifically, it is a recipe for underperformance of macro-economies. For example, over-investment in capital that represents the capital stock necessary for optimizing a manufacturing process and under-investment in labor that represents new-product development may show decreased costs, but it also does less well at meeting consumer needs, and hence static IS-LM curves are achieving less sales at a price than they could – the economy is under-performing not just relative to “potential GDP”, but also “potential GDP” at full customer satisfaction.  Or so I wonder.  This wasn’t a clear problem when we had no alternatives; but now we have a software world.

Flexibilists Rule

This is perhaps the most speculative of my suggestions – and I realize that’s saying a lot.  There’s been a lot of attention paid to Brynjolffsohn’s assertion that increased automation via computing has meant the loss of jobs, even including knowledge workers, and it’s not clear if and when that will reverse.  I would suggest that while that may very well be true of manufacturing and service/support jobs, it should not be true of new-product development jobs in a software world, and, in the best software firms, like Google, it isn’t true. 

For one thing, software automation is very far from replacing programmers.  This is something I’ve been arguing for thirty years:  programming is partly creation of new mathematics, and that’s something that we are not near automating (and yes, that’s very distinct from the “I get human semantics” analytics of Watson).  For another thing, programming in a software world is usually about development of new features, i.e., new-product development, and it makes sense to invest there rather than in non-existent inventory management.

I say, it should be true, but I recognize that in many firms, even software ones, programmers are thought of as disposable and capital investment as to be preferred – e.g., the offshoring of development that is still ongoing, even in areas where the supposed lowered wages are pretty much vanished.  That is partly because, I think, these firms tend to think of programmers, and marketers, and so on as narrow specialists, so that as technology and so on changes, it becomes necessary to hire someone who knows the latest language.  That is completely untrue of most of the programmers I know; in an environment free of threat to their jobs and with a little time to burn, they are the most avid consumers of new programming technologies, and a wide variety of other things.  They are flexibilists, not specialists.

The same, it seems to me, can be said of the agile marketing movement, which is – surprise, surprise – strongly associated with the software industry and software-heavy firms.  Agile marketing is more and more about flexible creation and delivery of new product – in fact, a continuous feedback loop of such creation.  And testimonials from practitioners echo those from agile programming shops:  it does better, and people stay around longer.  In a software world, or at least in a better one than we have today, flexibilists rule.

Conclusion

Them’s my preliminary thoughts.  Reactions?

The Smartphone as TV Remote: It's About Computing

Whilst driving the other day I overheard some NPR “experts” decrying the state of the cable/TV industry, and citing as an example that one’s elderly relative could use the Internet but not the TV remote.  Immediately, I asked myself a naïve question:  Why can’t the smartphone be a remote?

And so, I did my usual due diligence, and discovered that there are indeed some apps out there that purport to do just that:  allow the smartphone to switch cable channels, and the like.  But I also found that there are some surprising limitations at present, which seem as if they could be overcome in a straightforward fashion if someone really puts an effort into it. And it seems well worth the effort for some entrepreneurial person or business.

Let me therefore state the problem as I see it, the solution and why the smartphone should be part of it, and the things that need to be accomplished.

The Cable Navigation Problem

Over the years (as the “experts” noted) access to TV channels has become dominated by 2-4 cable companies, which decide to some extent what you see.  Initially, their offerings proliferated to 200-odd distinct channels.  Over the last few years, according to the “experts”, the cable companies have discovered that there’s money in sitcoms and dramas whose characters actually evolve over time, because a loyal band of watchers creates buzz over the Internet about the characters – but those series are expensive, and so the rest of the shows per channel must cut expenses to the bone.  In practice, that means thinly disguised reality shows (and, secondarily, reruns; but those have a half-life on the order of 2-3 years).

The result is almost stupefying:  almost universal prime-time programming of reruns, rehashes (true murder mysteries resliced ad nauseam), and reality shows in the oddest of places.  The Learning Channel does not feature learning.  The Discovery Channel does not feature science, and the Science Channel is going the same way with science fiction (read:  horror movie) reruns.  Animal Planet spends endless hours with real-life searchers for Bigfoot; National Geographic Channel does the same with prison reality shows.  Biography Channel?  Not really.  It isn’t just that these bear no relation to their name and what they originally set out to do; it’s also that they bear a great deal of resemblance to each other.  And so, counterintuitively, the TV watcher seeking something new needs to do a lot of searching. In fact, the watcher needs to do a lot of digging when he or she finds an innovative show, as the TV screen provides barely enough information to be misleading:  “Dangerous Attractions:   Relationships that test the limits of …” turns out to be about dogs and cats being pals.

At the same time, the goal of a “universal remote” still seems far off, if the cable company and TV manufacturer offerings are any guide.  Often TVs have their own remotes, which do things that the universals don’t, like handle various types of HD display.  If there is interference, the TV may turn on and the cable not, or vice versa – and what’s happened is not that obvious.  If one accidentally starts talking to the TV rather than the cable, changing the channel knocks the viewer out of the cable universe entirely.  And don’t get me started on attached DVD players.

So there are two basic problems:

1. 1.       Turning TV and cable on is typically a complicated pain in the neck.
2. 2.       It’s hard to find the right show, because it takes channel surfing through 200-odd channels with no good descriptions to hand.

A Proposed Smartphone Solution

The solution seems straightforward:  a flexible piece of software on a remote that gives you one button to push to turn both TV and cable on and off, the usual channel-changing keys, and the ability to semi-automatically display adequate information about the shows, singly or summarized in the traditional TV-Guide timeline.  Oh, and the ability to share short comments about shows as they unfold would be nice.

Yes, but what’s the form factor?  Today, the universal remote is a single-purpose, inflexible piece of extra junk.  What’s needed is something truly programmable, easy to upgrade, handling all the little subcases that universal remotes vainly try to do now, plus be used for other tasks yet to be determined.  In other words, something like the smartphone or laptop.

Those of you who follow my blog posts are aware that I have spent a great deal of effort pointing out the things that the smartphone will probably never do, that make it unlikely to take over the computing world.  However, in the TV-remote case, these are in fact virtues.  We don’t need acres of space to see a blog post or long essay on a show, just enough information to tell if it’s really new and interesting to the viewer.  We don’t need to write that long essay, either, just to share a few pithy comments.  Now that, imho, the Samsung Galaxy SIII and other Androids have fixed the iPhone-type interface that was likely to put you in an odd part of a task with no obvious way to escape, the user interface to do both turn-off/turnon and search/texting is at least adequate.  And it’s just easier to lug the smartphone into and out of the family room than do the same for a laptop.

State of the Art

Anyway, I had reasoned this far when I looked at available offerings from folks like Dijjit and Beacon, and it became clear that there are some pesky details that still need to be solved. It appears that some cable boxes and TVs talk WiFi or Bluetooth, and some talk infrared in their own code.  Thus, not only does the smartphone, TV, and cable need to connect via WiFi to a home network for one-button “on” to work in some cases, but the smartphone actually has to generate infrared signals in multiple codes in some cases.  The result:  a “dongle” on the smartphone or an attached “rock” a la Beacon for channel surfing, and no one seems to handle the full case of cable/TV on/off. 

Then there’s the nagging question about what to display about the channels one surfs.  It seems that apps are only at the point where a raw download of some cable channels’ TV guides is supported, and even those are designed for less detail than a smartphone user – or a user in general – needs.  And, of course, there is no connection between tweeting and channel surfing; but I regard that as a minor inconvenience, since when you tweet you’re going to be typically stuck on a particular cable channel anyway.

Last and least, the vendors seem to be doing iPhone first and then Android – perfectly understandable, given market sizes, but it does delay (again, imho) user friendliness of the smartphone “remote”.

All of these things seem straightforward to fix, with the possible exception of the infrared communications.  It just takes time, and will on the part of app and smartphone vendors …

The Bottom Line: The Potential Lies in the Computing

The key to the potential superiority of the smartphone over traditional TV remotes, I believe, comes down to the fact that its pedigree is in the software-dominated software-hardware partnership of computing.  A TV’s interface is just plain stodgy, and in attempting to make it a veneer for both shows and the Internet, cable companies are forever playing catch-up, and forever falling further behind.  By contrast, the main difficulties from the smartphone side are ones caused by the rigid interfaces of the cable/TV hardware combo. 

And yet, the cable companies badly need to attract the generation in its 20s right now – which by all accounts is barely buying cable at all.  The prices are certainly too high for a large proportion of this generation, which has been hit hard by the recession; but there is nothing showing their buying behavior turning around soon, and at a certain point, they may conclude they don’t need cable at all; and that could be a death knell.  A smartphone remote may seem like an odd place to start; but if it means that this generation finds it interesting to tweet on programs, there is real hope that cable will not destroy its seed corn.  As for advertisers, if there is a better connection between the Internet and cable, some of them may well start spending on smartphone ads connected with cable ads – the tail wagging the dog, so to speak.  And that should slow the bleeding of ad dollars from TV …

Or maybe not.  In any case, it seems to me that we do too much contrasting between smartphones and the computing industry, when it seems to me that the real difference is between these two, who share via Steve Jobs the same software-driven computing DNA, and traditional media – and yes, by now cable is traditional, not too long after actors and actresses started seeing it as more interesting than the movies. I’d love to see the smartphone as TV remote; I think it’s an idea whose time has come, and I would dearly love to deep-six the five remotes (two cable, two TV, one DVD) for two TVs I presently have to have.  But even if it doesn’t, similar ideas are happening all the time, for both the PC and smartphone/tablet form factors.  Vive la difference et l’identite! Vive le smartphone remote!

Why Can't the Mainframe (and the PC) Stay Decently Dead?

For more than 23 years, as an analyst, I have been watching various industry observers suggest that the end of the mainframe may be nigh.  For nearly that long, I have been hearing voices proclaiming the nearing death of the PC. For the mainframe, it was the proprietary architecture, the lack of developers, the aging of administrators, or the rise of scale-out.  For the PC, it was the expense of personal storage and its administration compared to the “network computer”, the lack of Web savvy, the large form factor compared to the smartphone, and the unaptness to touch gestures.

So what did the 2012 markets say?  Astonishingly, after a 10-20 % dip in revenues in the first 3 quarters, IBM saw a 56% mainframe-revenue jump in the fourth quarter, to overall revenue growth in 2012, better than the decreasing revenues of its Unix/Linux (System p) and Windows (System x) alternatives.  This extended the mainframe’s 3-year streak of revenue (and therefore income) growth. 

IBM was not alone in seeing these results.  HP apparently saw greater single-digit decreases in Unix/Linux than in PC revenues.  Anecdotal evidence suggests that Oracle/Sun Unix/Linux revenues continued to decrease.  And, of course, touch screen in a PC form factor had only just begun to arrive at the end of 2012 (with Microsoft Office in a hybrid tablet/PC configuration only apparently arriving some time in 2013) – so the PC was competing with smart phones and tablets with one hand tied behind its back, so to speak.

In fact, if 2012 was proclaiming anything, it was suggesting the eventual death of Unix/Linux (no, I don’t believe that either).

In an era in which IT is apparently content to spend the same amount each year on computing, much of it on saving costs – thus increasing computer maker profits but keeping revenues flat – it is not likely that any of today’s form factors, including the smart phone, is going to attain the 50%-+ growth rates that we have seen in the past.  Thus, we may have seen the popping of the Apple stock-market bubble. And therefore, in the near future, I assert that neither the mainframe nor the PC is going into terminal decline – on the contrary, they should prosper modestly, as the mainframe has done in the last 3 years.

So why will all the cited disadvantages of mainframe and PC not lead to steady or sudden terminal decline in the next 2-3 years, but rather to sober growth?  Let’s take each in turn.

The Mainframe Is Fully Reinvented

When I first suggested to IBM that the mainframe needed to become a “hub” in the sense of a fully networked node of especial prowess in certain types of workload rather than in the sense of “either mainframe or something else but not both”, it was only about five years ago.  Since then, the process of turning the mainframe into something that is a full participant in the enterprise architecture has been pretty completely achieved.  All of IBM’s major enterprise software, from administration to security to data management to development, now is on approximately the same track inside and outside the mainframe, and most apps can easily and even dynamically move between mainframe and non-mainframe servers.  As a result, users find it far easier to employ the mainframe flexibly for its strengths in robustness, security, and high-end transactional scale-up. 

Imho, IBM’s mainframe plans for 2013 contain no such dramatic transformations – and don’t need to.  The “bridging” architectures of zEnterprise and (eventually) of PureSystems form a completely adequate foundation for future elaborations in mainframe-including enterprise architectures.  To the end user, developer, and administrator, the mainframe if needed can appear more or less as a transparent part of a fully modern overall enterprise architecture. My only caveat, as noted before, is that the mainframe fails to support Windows in public cloud architectures adequately (again, imho) – but that simply limits growth, it does not portend decline.

And so, the old objections begin to melt away.  Few argue that the mainframe’s proprietary architecture cannot keep pace just as well in the near future with the evolution of hardware/software technology. IBM cites reports that a new generation of system administrators is arriving, with adequate skills, and therefore some new buys are even showing a preference for z/OS up front rather than Linux.  While the mainframe will never rival the hundreds of thousands of apps for Apple’s OS or Android out of the box, there are few major barriers to software today, either. 

To put it in a nutshell:  the mainframe is attracting new customers, even in the US, because the old knocks on the mainframe no longer apply, leaving it to leverage its strengths in a particular segment of the market that will grow with the growth of usage of Big Data.

The PC Keeps On Doing What It Does Best

The argument for the PC’s continued success is of a different sort from that of the mainframe.  I say, rather, that the doomsayers of the PC have not failed to appreciate adequately its long-term strengths.

Let’s start with the “network computing” argument.  Yes, as with the “dumb terminal” before it, the “network computer” is cheaper than the PC.  However, one value of the PC has been its ability to store personal data and applications, whether as a “home within an office” that allows the end user to generate his or her own Powerpoints and spreadsheets, or as a bridge between home and office to allow work wherever.  That is precisely why the countervailing trend of BYOD (Bring Your Own Device) – which is just as much if not more about laptops than smartphones and tablets – continues to ensure a major presence of PCs both at home and at the office.

As for Web savvy, once again the smartphone’s evolution has proven the value of personal physically-next-to-the-user storage – whether it be stored song downloads or phone logs – and hence the smartphone is turning inevitably into a PC – but one in which the small vs. large screen carves up turf between the PC blogger and the smartphone tweeter.  And, of course, my experience with Windows 8, wrong-headed as I believe some of Microsoft’s decisions are (e.g., crippling the desktop screen out of the box), shows that touch not only is valuable to the PC’s core word-processing and navigating/data-organization skills, but that it is not as easily applied to the smartphone’s small screen without a comparable, complementary typing feature (hunt-and-peck still doesn’t do it). And so, not only in terms of Web usage but also in terms of end-user-friendliness, the PC still holds its share of the market; and, for the same reasons, should continue to do so.

Note that I say that the relatively large form factor is a plus rather than a minus.  I assert that there is a “finger limit” in which the typing necessary to create large-scale amounts of real content can only be accomplished on a sufficient-sized screen.  Call it semantics, call it deep analysis, call it whatever you want, 50 lines is about the minimum for a decent blog post that does not amount to dipping your finger into a very large pool – and, as in this post, 2-3 old-style pages are more like a comprehensive look at a subject.  Note that folks like Paul Krugman use shorter blog posts as “riffs” to serve as the basis for a comprehensive multi-page paper, not as the definitive word on a subject.  A mashup is no substitute.  And so, the PC’s combination of personal storage and relatively large screen/keyboard should see continued modest growth in demand, in whatever form, over the next 2-3 years – depending, of course, on the world economy.

Sacrificing Some of the Future By Discarding the Present

I am long since resigned to periodic outbursts of “the death of …”, which sometimes are justified (I for one don’t lament the effective death of the dumb terminal, having had the dubious joy of programming for it).  What I do object to is the way in which these persuade many people in and out of the industry that the architectures in question are indeed dead, which means we don’t need to think about them, which means we don’t leverage their unique strengths in the next generation of the world-wide web of computing. 

Specifically, the mainframe pushes the limits of scale-up computing.  While scale-out grid solutions may indeed have achieved some notable successes (I understand that Google has made notable advances by thousands-of-servers divide-and-conquer applied to language comprehension and translation), on a per-processor basis scale-up’s tight integration continues to offer frequent performance advantages over scale-out’s loosely-coupled networking. 

On the PC end, the latest tablet/smartphone user interfaces are crippled by a lack of appreciation of foldering’s static personal data organization as a complement to the Web’s dynamic search-based organization. Moreover, we continue to move away from the idea of a “virtual end-user space” in which the full functionality and data of one’s own PC is available anywhere, any time, whether we are connected to the Web or not (and this will be needed because we are still quite a ways away from always-available and always-fully-functional/personalized Web avatars). 

Most of us remember the Monty Python routine in Holy Grail where one person attempts to fob off an alive person on a “dead collector”, while the corpse in question protests “I’m not dead yet!” To which the seller keeps replying “Shut up.”  For very good reasons, the mainframe and the PC refuse to stay decently dead.  Could we please not keep telling them to shut up?

The Sad Implications of Two 2012 Climate Change Surprises

In reading postings in one of my favorite blogs recently (Neven’s superlative Arctic Sea Ice blog), it occurred to me that I hadn’t tried to summarize in my mind the two major surprises that scientists recognized in climate change last year.  What follows is my take on these surprises:

1.      Arctic methane is venting much faster than expected; and
2.        The weather effects of global warming are stronger than expected.

What are the implications?  Let’s take them one at a time.

Arctic Sea Ice and Methane

It still amazes me that most people did not see the likelihood that Arctic sea ice was going to take a nosedive to near zero in the 2013-2016 period, because I only had to apply some basic math when I first took a look in 2009:  exponential and normal curves.

Until recently, there was only one serious attempt to assess the volume of Arctic sea ice:  Maslowski’s PIOMAS model.  As I understand it, people tended to dismiss the model because Maslowski said:  this is the way I model Arctic sea ice dynamics, and therefore the sea ice volume should change over time in this and this way; and because he asserted without proof in his model that Arctic Ocean water temperature changes had a major role in increased melting over time.  However, when I saw that this model was constantly reality-tested by on-the-site sampling, even though each sample was of a small part of the overall Arctic Ocean, I realized that the long-term trends there were likely to be true.  And what Maslowski’s model showed, at any time of the year, was an exponentially decreasing Arctic sea ice volume.

By the way, recent Cryosat observations have definitively shown that, if anything, Maslowski’s model has underestimated the rate of volume decrease.

So why, I asked myself, do I not see corresponding decreases in Arctic sea ice area and extent?  As I looked at the dynamics of Arctic sea ice, I realized that that could only happen if there was a uniform distribution of Arctic sea ice thickness from, say, zero to twice the average thickness (at any time of year).  But what was really going on was that a certain percentage of the sea ice survived between years to become second year ice, third, and so on; but cue to currents, the average age of Arctic sea ice back in 1980 was five years – sooner or later, ice frozen at one end of the Arctic would reach the other end and head south into warmer waters, there to inevitably unfreeze.  Instead, thickness (with a little adjustment for the age of the ice) had much more of a normal distribution around the average. And that, in turn, meant that accelerated volume losses at, say, minimum would only show up in area and extent when we reached the fat part of the curve – which, it is now apparent, occurred in 2012.

By the way, the same logic also says that volume and the rest will not go to zero somewhere around 2014 – 2015; we will have reached the other thin end of the distribution, and the exponential decrease in volume will flatten out, exponentially.  That’s why I fully expect to see around 1-5% of the ice remaining at minimum until sometime around 2016-2020.

So, as I’ve said, I expected that Arctic sea ice would begin to obviously disappear around now, and I expected the climate change implications of this – including the fact that Russian-Arctic-Continental-shelf methane “clathrates” would begin to release their methane.  What I (and apparently others) did not expect was the scale of that release.  A Russian sampling of methane bubbling to the surface found huge pockets of the stuff – hundreds of times more than research had suggested might be the case.

Before I go on to discuss this, let me cycle back to the (expected) implications of Arctic sea ice melt.  Today’s models simply do not include melt to near zero at minimum in the 2013-16 time period, and a likely follow-on melt to near zero at all times of the year between 2035 and 2045. This in turn, will not directly lead to more carbon emissions.  What it will do is decrease Arctic Ocean albedo (from off-white-reflection to dark-blue-absorption of heat from the sun during the spring-summer-fall), and therefore warm up the Arctic Ocean portion of global and ocean temperature.  This already is 15-20 degrees Fahrenheit above normal during the summer; we are talking another 25-35 degrees by 2045, taking it to 20-25 degrees during the winter. 

The increased water and air temperature should therefore (a) accelerate methane clathrate melt, including that of the deeper waters nearer the North Pole, and (b) both cool and warm winter temperatures of more temperate zones – with the “warm” predominating over time.  How can (b) be?  Well, warmer Arctic air has more energy, and therefore pushes south against the “jet stream” more strongly, creating weather in which unusually cold Arctic air reaches further south periodically.  However, that same Arctic air is steadily warming over time, to the point where by 2050 it should be as warm as or warmer than southern winter air was in, say, 1980. 

Implications of the Methane Surprise

Part of the problem with assessing methane’s implications is that most if not all scientists have not factored in 2015-2045 Arctic sea ice melt’s implications for temperatures just a bit further south.  Because methane clathrate melt should be understood as part of a “double whammy” for methane – Arctic methane melt at the same time as permafrost melt.

Since (up to a point) methane has a very short half-life in the atmosphere (say, 8-10 years), you need a much greater rate of methane release into the atmosphere than carbon (all right, methane includes carbon too, but methane has a much greater effect on global warming per ppm than carbon dioxide) to achieve a comparable warming effect over time. And yet, studies of 55 million years ago, when the rate of warming was much less, indicates that methane had a major role in causing what Joe Romm at www.climateprogress.com calls “Hell and High Water”, with 90% of species wiped out.

Scientists have made a persuasive case for the idea that, even if methane clathrates are starting to melt and permafrost at the same time (i.e., even if we factor in Arctic sea ice melt), methane emissions will not reach a “danger point” where their effects in the atmosphere will rival that of carbon emissions any time soon – and therefore will avoid the main danger time of carbon emissions, before the lack of fossil-fuel reserves begins to decrease those emissions on its own). The problem is that the Russian observations indicate that those reassurances are based on assumptions about the rate of methane clathrate “bubble” occurrence that far underestimate their rate and/or amount.

So what, then, are the likely implications of this methane surprise?  As far as I can see, there are no “likely” implications, because the range of possible methane “bubble” rates, and therefore emission rates, over the next 40 years is so wide.   Nor is it clear to me, given that these emissions are occurring in such a localized northern area, just how wide an effect on global warming there will be.  However, my best guess is that over the next 40 years there will be a significant, localized effect:  Methane emissions will increase Russian and lower-Arctic-sea heat retention over what it would have been by perhaps 25%, with a corresponding increase in Arctic average temperature and Russian permafrost methane/carbon release.  This, in turn, may add perhaps ½ degree Celsius to global warming over the next 40 years – and, of course, will add a comparable amount over the 50 years following, at least – always remembering that a major fraction of methane released turns into carbon dioxide, and hangs around in the atmosphere for a hundred years or so on average.  In other words, the major effect of the surprise may be a more long-term one:  arrival of a ½ - 1 degree Celsius additional increase in “thousand-year” global warming now rather than later, when it would have less practical effect.   

The Weather Effect Surprise and Implications

Taking my cue from James Hansen and Joe Romm, I had guesstimated in 2009 that we in the US would first see constant, undeniable reminders that global warming is real in the 2020-2025 timeframe.  These reminders would include not only “hundred-year” hurricane-type wind speeds and scorching summers that created Dust-Bowl conditions in many areas, but also an overall burden of disasters that reached 0.1% of GDP even for a country like the US.  I believe it was Heidi Cullen that imagined NYC missing a massive hurricane in 2017 and getting one in 2041, by which time the city was prepared and the sewers did not back up and overflow, causing hundreds of thousands of deaths from disease.

But, as Joe Romm noted in his blog, the things that should have been expected in 2020-2025 seem to be happening in 2011-2012, ranging from devastating Australian rainstorms to stupendous Russian wildfires to Hurricane Sandy and its 13-foot storm surge (still short of the 20-odd-foot storm surge that might cause the sewer outlets to be closed and the sewers to back up, but enough to flood the subways and make downtown Manhattan, Queens, and Staten Island disaster areas).  It appears that the monetary effects of disasters globally, according to insurers like Munich Re, are 10 times what they were a decade ago, and there is no reason why they should not continue to double or triple by 10 years from now – meaning that the timetable for effects on global GDP should perhaps be moved up by 3-5 years.

The surprise is not that global warming is happening faster than predicted – globally, 2012 was actually about average for the last decade, which in turn means that it was one of the “dips” in our steady, accelerating global temperature ascent. The surprise is that the effects on weather were larger than expected.  What I suspect is that forecasts simply assumed that certain catastrophic events would happen more frequently than expected, but could not predict that these catastrophes would spread to areas where they had not before (devastating tornadoes in western Massachusetts), areas that were less adapted to a new set of weather patterns. We are reaching the point where we have not only extremes of existing weather patterns, but also new climates that produce new weather patterns.

And so, I also suspect that the effects of the “weather is changing faster than we expect” surprise are, like the methane surprise, bad but unpredictable.  I anticipate that the Nino/Nina cycle and North Atlantic Oscillation patterns that have driven weather around here since time immemorial (i.e., the last 5,000 years) are changing, but will manifest first in longer versions of the extremes of this cycle – and that’s a total guess.  Certainly, an extended Nino would mean even greater Dust Bowl conditions and summer heat over a great extent of the US for longer than ever before, even leaving out the effects of the ongoing global temperature warming.  Initial predictions show the US except the Northeast and Pennsylvania in catastrophic drought conditions in 2050 – is it possible it could happen before then?

Boy, I hope not. But, as a clueless Presidential candidate noted in 2008, hope is not a plan.

Conclusions

Overall, oddly enough, the implications of these surprises for me are not great.  I concluded in 2009 that we desperately needed to cut carbon emissions in absolute terms by 40% 2010-2020, and another 40% 2020-2030.  Since then, with extremely minor exceptions, all major countries in the world have utterly wasted their time in that regard.  In fact, my definition of functional insanity is to see oil companies and countries seeing Arctic sea ice melt as an opportunity for increased drilling of fossil-fuel carbon pollutants, and the United States seeing a Keystone XL pipeline that solidifies tar-sand drilling that sharply increases the likelihood of the end of all life on Earth as an opportunity worth considering, much less actually being relatively close to implementing it.

So, to my eyes, the horrible effects of the two 2012 surprises simply speed up what’s coming and increase its bad effects in relatively minor ways, and that will be worth it if people wake up now and start doing something globally effective.  Except that there’s little sign as yet that people and their leaders are even beginning to understand the urgency of an adequate scale of action.

I wonder what new surprises 2013 will bring?  I could really use some good news.

2012 Election: Big Data Makes It Easy to Shoot Yourself In The Foot

A recent post in MIT Technology Review about the use of Big Data by the parties in the 2012 election offers fascinating insights into what works – and what doesn’t.  Both sides had operations specifically designed to glean new insights about voters and apply them to helping their candidate win, and the article interviews the director of each effort shortly after the election.  As a result, we see long-term Big Data projects whose results have just been tested against the reality of the voter “market” and whose project reviews are relatively untainted by post-project “spin.”

I should note that I view the article’s author’s grandiose claims about the meaning of the Obama campaign’s relative success in Big Data analytics to be overblown.  Luckily, many of those claims take up only the first 20% of the article. The rest provides a fascinating insight into two approaches to Big Data, one of which demonstrably worked better than the other.

Problem #1: Big Data as a “Hot Topic” vs. as a Strategy

The first contrast between the campaigns was that the Obama one chose to bring the analytical systems in-house, whereas the Romney campaign chose to use existing software and hardware run by someone else.  In fact, the Obama campaign specifically committed major bucks to a Vertica solution way back in 2009.  This begs the question, however – why did the two campaigns take such different approaches? After all, the Romney campaign actually started quite early, around the same time, and had no real constraints on money spent.

The answer, I think, is that while Obama viewed in-depth analytics and trolling the Internet for more voter insights as a strategy, while for the Romney campaign, the concept of Big Data was fuzzier, more of a “hot topic” type of strategy.  We have all seen CEOs who have said, “Everyone’s talking about this new technology. We must therefore do it.”  The flavor of Romney-campaign thinking, as reported in the article, was, “Big Data is clearly effective at the corporations we look at; we must therefore get something that purports to do Big Data and use it somehow to understand voters better.”

The result was a cautionary tale.  The Romney system involved a third party merging two sets of legacy-application data repeatedly.   As a result, breakdowns occurred in the stress of the late campaign, and analytics was unnecessarily delayed, delaying candidate responses. The Obama campaign suffered from no such problems. It is not clear how much the Obama campaign’s faster response time helped (although it apparently helped change Obama’s style after the first debate).  However, it is clear that the Romney campaign’s fuzzy and delayed analytics helped lead to Problem #2.

Problem #2:  Hearing the “Customer of One” vs. Hearing What You Want To Hear

One of the great shocks of the campaign was that, in an unprecedented way, the Romney campaign and the Republican party as a whole deluded themselves about what was going to happen.  And yet, the campaign’s Big Data analysis mined a long tradition of understanding voter blocs, and could be supposedly cross-checked against a broad array of pollsters – most of whom were themselves deluded by their political biases.

As it turned out, even the closest estimate underestimated Obama’s margin by about 0.5%.  Moreover, on election day, the Romney campaign actually thought they had a better than even chance of winning. However, as it turned out, to win they would have had to amass more than 5.3% more of the vote in a couple of battleground states than they did, at minimum. To put it another way, they probably would have had to change the vote percentages by more than 5% nationwide to win.  They weren’t even in the same ballpark.

By the way, for Nate Silver fans, he was off by about 1.5%.

There appeared to be two key insights that made this election different.  First, perhaps one-third of people voting could only be reached for polling via cell phone – and they voted in a significantly different way.  Second, only perhaps 5% of the electorate reached the Labor Day start of the campaign with a changeable vote – again, a major difference from 20-40 years ago.  Gaffes, debates, Hurricane Sandy – it didn’t make that much difference. 

Given this, the Romney campaign’s one faint hope was voter turnout – unprecedented turnout for him in battleground states. And yet, there was not the slightest appreciation of this in the campaign.  Instead, the Romney campaign or related groups spent enormous sums on TV ads in the obvious markets, to the point where one station in Ohio was running more ads than programming. The crime wasn’t that this was wasted money; it was that relatively little money was spent on “get out our vote.”

In fact, most of the Romney campaign Big Data folks’ attention was on a series of seemingly baffling Obama ads in small markets targeting small demographics.  As a result, by election day, the Big Data folks were suffering from the “fog of war”, trying to counter something they did not understand.

Again, by contrast, the Obama campaign was focused on a new insight garnered from their use of Big Data to identify the “customer of one”.  More specifically, they realized that although most votes were unchangeable, smaller pockets of “conservative” voters could actually change their mind if policies they valued and prioritized were brought to their attention. These could be reached by, say, social-program-highlighting ads aimed at conservative women in more rural Ohio markets.

An under-appreciated part of these ads was that they contained real content.   After all, “conservative” voters were being saturated with generic Romney ads.  To succeed, the Obama ads needed to point to real, verifiable programs.

And now we come to Problem #3.

Problem #3, the Endgame: Big Data Tunnel Vision vs. a Loosely-Coupled Strategy

The end of every political campaign is “get out the vote” – maximize the number of ones supporters who actually vote on election day (and, of course, the increasing percentage who vote before then). And this is where the most startling difference between the campaigns’ approach to Big Data emerges.

It is apparent from various reports that on election day, the Romney campaign was prepared to focus its “turn out the vote” efforts on key demographics in battleground states, fine-tuned by what their Big Data analytics was telling them.  And then, right when voter turnout efforts were supposed to start, their feed from their system went down.  And it stayed down, for most of the day.

But what almost passes belief is what the campaign did about that.  Local offices were begging to go out and do “get out the vote” efforts anyway.  Instead, they were apparently told to wait until the system came up again.  Yes, theoretically, unfocused efforts could have done more harm than good.  Practically, however, it was overwhelmingly likely that “feet on the street” would have instead achieved slightly higher Romney turnout. And not only national but local campaign coordinators failed to realize that.

The problem, it appears, was that no one had autonomy, and all were focused on the Big Data part of the “get out the vote” strategy.  By contrast, the Obama campaign – partly due to an existing Internet-enabled strategy – granted local offices the ability to act proactively, and the Big Data “customer of one” focus was only part of a broader effort in battleground states to get out key Obama demographics, which were already well understood pre-Big Data.

Again, it is worth noting that this made little difference “on the day.” It is possible that it made a difference to several House seats; but not enough to overcome the effects of the 2010 election and its resulting Republican-dominated redistricting.  Note that there was apparently the biggest difference between the vote for each party (Democrats +1%) and the House seat allocation (Republicans +4%) ever recorded.

Implications for Organization Big-Data Use

We have seen, above, how the Romney organization used Big Data to shoot themselves in the foot – and yet, their strategies were superficially reasonable and well aligned with the practices in many businesses.  The immediate recommendations for IT and the enterprise are likewise relatively straightforward:

1.       Treat Big Data as a strategy, not as a “hot topic.”  Understand that for it to be successful, it must provide greater depth and a more accurate view of the customer and of one’s own organization, and those insights need to be translated to fine-tuning of strategies sooner rather than later.

2.       Focus on Big Data’s ability to understand customers not only more deeply as global “types” but in finer-grained groups, and ensure that the organization accepts a more realistic view of the customer.  Bluntly, the Romney campaign started believing their own propaganda; where have we seen that before?  And one reason was that Big Data was not telling them any different.

3.       Adopt an agile marketing strategy that does not hang on command-and-control top-down implementation. Agile marketing has a great respect for in-depth data.  However, it also has a great respect for the way that data reflects or fails to reflect actual customers, and customers who are constantly changing.

  And that brings us to our final point.  As I’ve noted, the success or failure of Big Data efforts turned out to matter surprisingly little to the outcome of this particular election.  However, Republican post-mortem efforts show that their failure to understand its implications has drastically delayed – and we are talking more than a decade here – their adaptation to changing American demographics.  They have created a party culture that makes it extremely difficult for them to move to anything more than permanent minority status nationally, because it carefully widened the divide with groups such as Hispanics in the name of older white-male “get out the vote.”

The point I am trying to make is that if Big Data cements an organization’s existing un-agile strategies in place, it is doing just as much harm as good – even though, for now, things are going better than ever.  The real value of Big Data, done well, is that it not only enables you to understand your customer better, but it also enables your organization to fit itself better to the customer – bottom-up as much as top-down – and both of them to understand how the customer is changing, not just what the customer is like now.

So what is it going to be?  Are you going to use Big Data to shoot yourselves in the foot, or to deliver better strategies, better implemented?  Inquiring Republicans want to know.