Lessons of Moore's-Law History 3: The Consumer is the Chasm

In the previous two blog posts, we have seen (hopefully!) the importance of betting on technologies and suppliers that most closely resemble the silicon-transistor-chip evolution process, and the need to bet on suppliers adhering to that process that emphasize infrastructure software combining openness to technology evolution with support for existing systems – but we have not dealt with the “chasms” that occur when Moore’s-Law evolution results in new, radically different higher-level technologies. 

Here I am combining two books popular back in the 1990s:  Crossing the Chasm, by Geoffrey Moore (no relation), which talked about how innovative startups bridged the gap between early techie adopters and the bulk of the market; and The Innovator’s Dilemma, by Clayton Christensen, which argued that “disruptive innovation” driven by new technologies would over time change existing markets profoundly, usually to the detriment of larger, well-entrenched companies presently dominant in an industry.  In an industry like the computer industry (including software and services), where Moore’s Law drives a more rapid pace of underlying improvement to platforms, entrenched companies must often react to disruptive innovation by “crossing the chasm” rapidly with their customers to the new technology, else they risk losing their markets and their ability to survive.

When A Company Is Ultimate-Customer-Blind

The 1990s and early 2000s gave us a good set of data on who succeeded and who failed in “crossing the chasm.”  The results were surprising to many.  It seemed reasonable that a Sun or an Oracle would surpass a Microsoft, that a Digital Equipment would beat a Novell, that an IBM would dominate a Compaq or a Dell in the PC market.  And yet, here we are 25 years later, and by no stretch of the imagination can we say that these things happened.

It seems to me that a common theme of this litany of successes and failures at adapting to new Moore’s-Law technologies is that companies that sold to the business market failed far more often than companies that sold to the consumer market.  Sun, which did a superb job jumping from the workstation to the server market, failed among other things to note consumer-driven Linux/Windows open-source software that was undercutting its prices and playing better with the server farms that later led to public clouds.  IBM’s initial great Charlie-Chaplin PC marketing fell before the next-day delivery and Intel allegiance of Dell and Compaq.  On a smaller scale, we saw Informix’s Oracle-protected VAR channel fail fatally when Informix failed to reach through that channel to detect that the servers it was shipping to VARs were no longer being bought at a comparable rate by the ultimate customers. 

On a longer time scale, the proprietary chip sets of IBM, HP, and Sun/Oracle have steadily lost ground compared to Intel markets.  It may seem logical that IBM over the last few years is walking away from the PC, starting with its consumer PCs/laptops; but that approach has steadily given it less and less sense of trends in the consumer market, and that, Moore’s-Law history would suggest, is dangerous.  It was once assumed that one’s employees would buy consumer versions of what a business-focused computer company offered; but, starting in the 1980s, the opposite has appeared true:  Word/Lotus, Excel, presentation software, laptops, smartphones, and social media have all been imported into businesses before and despite corporate standards.  If a computer company wishes to handle a disruptive innovation, Moore’s-Law history suggests, it must immediately sense the consumer movement, and then follow the consumer across the chasm, and quickly.

The Top Line is Now the Bottom Line

From 2008 on until fairly recently, a focus on cutting costs allowed companies like IBM, Oracle, and HP to look good to Wall Street although revenues were slightly down to slightly up.  Now, however, the contrast with consumer-focused companies like Google, Amazon, and Apple that continue to grow revenues by leaps and bounds is becoming all too obvious, and the chasm to cross to support the new mobile and cloud technologies is becoming wider by the day.  Generalizing to other industries, it is not enough to keep in sync with a Moore’s-Law-related process and focus on flexible, forward-compatible software.  One should also focus on growing top-line revenues by crossing the chasm with the customer. 

That means aligning platforms with consumer-successful companies like Intel.  That means aligning strategies with Google's, Amazon's, and Apple’s smartphone and cloud technology changes.  And that means far better coordination with and analytics about ultimate end-user customers – e.g., agile marketing.

Moore’s Law won’t last forever – it probably won’t last more than 5-10 years longer.  But the lessons of its history will last much longer than that.  Caveat seller.

HP: Memristors And Things Past

As noted in the latest issue of MIT Technology Review, HP’s research arm is making a major bet on memristors, seeking to deliver a major performance boost beyond the traditional processor/memory/disk hierarchy.  More specifically, as presented in the Review, HP would replace memory, SSDs, and disk with “flat” memristor memory, and then communicate within the new processor-memory architecture using optical-fiber-type light-speed technology rather than today’s slower-electron-speed technology.  In the Review, HP projects initial product ship as next year, although it appears from Wikipedia that 2018 is a likelier date.

It is exceptionally difficult to tell whether the resulting products would really outdistance SSDs and cables over the long haul, because HP is making its bet at an unusually early stage in the technology’s development.  However, my skepticism is not of the technology, nor of its potential to deliver major and scalable performance benefits beyond the transistors beloved of Moore’s Law.  Rather, I believe that a critical success factor for HP will be delivery of an operating system (HP has christened the concept The Machine, and the operating system Machine OS) plus the attendant compilers and interpreters.  To put it bluntly, HP has been late to realize that it needs software development savvy in its DNA, and past experience with HP software development suggests that even if the technology pans out, HP will probably fail at giving business and consumer customers a reason to insert Machine OS in place of the old-architecture operating systems.

Let’s dig further into the details.

More On Memristors

As Wikipedia notes, it is not clear even now that what HP is touting is indeed, strictly speaking, a memristor.  This matters only if, as fab facilities are developed and the technology tries to scale like the transistor, it turns out that HP’s technology is not only not a memristor but also cannot scale as the memristor should.  That’s one unusual technology risk for HP.

The second technology risk is that the actual “unit form” of the memristor memory has also not yet been specified and verified.  Apparently, a “crossbar latch” approach has been tested at a small scale with good results, but it is not certain whether that’s the best approach or if it will scale appropriately to memory chips.  If it does, HP estimates that 1 cubic centimeter can contain 1 petabit of memory (all accessible at the same tens-of-nanoseconds speed).

Other details of the memristor seem less relevant at this time.  For example, a true memristor effectively keeps a “memory” of past changes of state, so that, as with a disk but unlike with main memory, if you turn off the power and then turn it on again, the memristor will be able to recover its value at the time of shutoff (or previous values).  This type of “remembrance” does not seem important in computations, although it certainly might help with machine-learning experiments.

Remembering Itanium

Again citing MIT Technology Review, initial plans are to create a “Linux over memristors” OS first, called Linux ++, which would not be a “clean slate” optimization for the new “flat” memory, and then a version called Carbon that would be optimized for flat memory and open sourced.  Left unexamined are such questions as “Do I need to recompile my Linux apps so that they’ll work on Linux ++, as was the case between Red Hat and Novell Linux?”  If most apps must be recompiled, business use of the new OSes is probably in most cases a non-starter, and even cloud providers might question the business case.

We have been here before.  When HP committed to Intel’s Itanium line, I remember from my briefings that they did not at all appreciate the importance of having well-tuned compilers and interpreters with minimal recompilation involved from the get-go.  I believe that Itanium’s variable-length-word instruction set approach was a very good idea at the time, but without a clear transition strategy, HP’s effort to implement it was pretty close to doomed at the start:  it could never gain critical mass in the market. 

I see no indication yet that HP has the software-development prowess to succeed in an even larger app conversion.  This is especially true because HP’s tradition since the early ‘90s has by and large been “let customers pick other companies’ software where needed”, which is fine until you really need in-house software to boost your new hardware technology, and quickly.

Is The Era of the Hardware And/Or Services Critical Success Factor Over?

It is generally understood that HP is not in a great strategic position these days, having decided to split PC computing from the server side as two separate companies and with a small but consistent decline in revenues (together with a larger early dip in profits) in its immediate past.  I do not, however, view HP as wildly less successful than other of the traditional hardware/services-biased companies, including IBM.  There does, however, seem to be a continuum ranging from purer hardware/services offerings to predominantly software-related competitive advantage, and it correlates with revenue trends.  HP is down more than IBM, which does less well than Oracle, which does less well than Microsoft.  Apple may seem to be an outlier, with its smart-phone hardware, but the differentiator that allowed Apple to croak Nokia’s hardware focus was the swipe touch screen – software all the way. 

If I am right, then HP really needs to do something about innovative-software development.  If it were me, I’d bet the farm on a better form of agile development than other large firms are practicing, plus meshing agile marketing with agile software development in a major way.  Despite welcoming some major agile-development upgrades in firms like IBM, I believe there’s plenty of room for additional goodness.

Failing that, it’s hard to see how HP turns itself around, as one company or two, via the memristor route or something less risky, in the next few years.  To misquote Santayana, even those who have hardware that remembers for them are doomed to repeat software-driven history – and that ain’t good.

We Have Blamed the Vendors and IT; Is It Now Time to Blame the Business Types?

Reflecting on the recent financial reports of the giants of computing infrastructure – companies like IBM, HP, Oracle, and Microsoft – I am struck by the continuation of a troubling trend:  The technology and its usefulness is moving ahead as fast as, if not faster than, ever before; IT has made some striking adjustments; but the revenues to vendors from this, and therefore IT spend, are flat if not falling. Why?

Let’s see.  In databases, Oracle and IBM are almost flat, despite (at least in IBM’s case) two technologies in two years that potentially deliver performance and price-performance gains well ahead of Moore’s Law.  I find plenty to criticize in Windows 8, but its implementation with regard to business needs, in Windows and Office usage, should deliver benefits comparable to that of the introduction of the mouse-based user interface – and Microsoft enterprise revenues are flat.  Cloud technology is not replacing, but rather complementing, in-house architectures, and therefore should require, initially, more rather than less total spending – and yet, increases in cloud spending are instead paired with overall flat or lower user computing spend.  No, I don’t buy the “public cloud saves money” rationale – the benefits come later rather than sooner, and involve mainly flexibility and to a much lesser extent multi-tenancy and grid savings, especially when many users are employing multiple cloud providers that must be coordinated.  And the IBM and Oracle/SPARC data suggest the biggest hardware decline is in Unix/Linux hardware, the darling of the cloud set, not in mainframes or PCs. 

In the past, a typical explanation for this has been the need for the business to cut back IT spend in response to decreasing profits.  Of course, that did not prevent IT and computing from becoming a larger and larger part of overall business spending during the 1980s and 1990s.  In the early 2000s, IT spend actually decreased proportionally to the cost-cutting of the rest of the business – and yet, despite an initially tepid overall business recovery, computing spending from about 2003 to 2008 rebounded and grew quite briskly. So this pattern of user computing spending is unprecedented in three ways:  its length, its unremitting focus on cost-cutting apparently comparable in size to the rest of the business, and its seeming inability to reflect major technology advances with a good claim to deliver major cost or other benefits to the rest of the business.

The Blame Game

In the past, I have found, when things go wrong in computing spending the first reaction is to blame the vendors.  In this case, the obvious critique is that they have failed to communicate the benefits of the technology, cost and otherwise.  Except that, as I can attest from my experience as an analyst, both the technology advances and the communications from the vendors are as good as, if not in many cases better than, those of most or all of the past.  Big Data, for example, is not pure hyperbole, and vendors have done a reasonable job by traditional standards of highlighting the ability of targeted Big-Data analytics to explain and bind the customer as never before, in a cost-effective way. 

OK, then the next line of analysis is to blame IT, typically for failing to align themselves with the business’ strategy, or (when times are tough) identify ways of lifting the dead hand of older systems and other seemingly outdated costs.  To this, I have one response:  agile IT.  As never before, IT is aligning itself with agile development, by supporting a software lifecycle that includes operational feedback integrated with development, and by encouraging automation of functions that provides a basis for rapid response and identification of architectural problems and opportunities at the administrator level.  As my studies have shown, these plus the increasingly agile nature of the software that IT makes available to the business translate into major cost-cutting and major benefits beyond what traditional IT approaches can deliver.  And as for aligning itself with business strategy, IT is well aware of Big Data as a hot topic, hence the sudden and somewhat odd demand for Hadoop experts.  No, whatever IT may have done in the past, it seems clear that it has upped its game.

OK, then, who does that leave?  And yet, how can we blame the business types?  Aren’t these the same folk who drove those increases in IT percentage of spending over the 1980s and 1990s?  Aren’t these the most receptive listeners when we talk about the power of embedded analytics to improve business processes and the importance of analyzing the “customer of one”?

There’s Something Not Going On Here …

To understand why there might be cause for concern and, yes, for blame in business strategy – driven by the CEO, the CFO, and their staff – let’s look in very broad-brush terms at what has been going on, not just for the last five years, but for the last three decades.  Here are a few of the highlights, as suggested by various studies:

1.       Advances in productivity have been accruing 90% to the CEO – ½ in his income as CEO, and ½ in his increases in wealth as an investor.  While this still represents a fairly small percentage of overall expense, it does unnecessarily increase the focus on cutting costs for all except the CEO.

2.       Hedge funds that sometimes act as turnaround artists or “shadow banks” have taken a good 90% or so of the funds’ investment profits for themselves, and thus receive compensation not just in the tens of millions of dollars but, at the top, in billions of dollars, despite the fact that they deliver less to the investor than an index fund in most if not all cases.  While it is difficult to see direct effects of this “fleecing of the rich” on corporate strategies, one significant effect is to increase the fleeced CEO’s focus on growing profits uber alles, to at least get some return from their investments – with the Googles of the world the few happy exceptions.

3.       The present long-lasting recession/stagnation has effectively removed perhaps 10% of the workforce from work over a long period of time.  This has the effect, since businesses often discriminate against the long-term underemployed, of creating a permanent gap between “potential” GNP and real GNP.  To put it another way, even if the economy grows at a reasonable pace, it will still be much smaller it should be.  That means smaller markets and, again, more relative emphasis on cost-cutting to achieve profit growth rather than growing revenues to grow profits.

There are three concerns about this excessive focus on cost-cutting.  The first, which probably is not serious right now but is far more troubling than five years ago, is that cost-cutting eventually runs out as a strategy if revenues continue to be flat (and, don’t forget, revenues are pretty flat for the bulk of businesses, hence the 1% growth in GNP over the first part of this year).  The second concern is that this cost-cutting, applied economy-wide (as it seems to be doing), reinforces and cements smaller markets by eliminating the part of the market funded by the now underemployed.  Ordinarily, recessions are too short for this to happen; but it seems to be happening now, as new jobs continue to make no dent in the workforce/working-age-population ratio.

The third concern, which I for one find the most troubling, is the possibility that reflexive cost-cutting becomes the answer to every situation, the reflex of the business.  If that is true, it would suggest that businesses are blowing opportunities for savvy increases in computing technology spend because it’s all about cost-cutting, not strategic investment. 

Blame or Not, What Might Business Strategists Do Better?

First, I would suggest that businesses set up a long-term plan and process for using analytics to better understand and improve the relationship with the customer.  This means acquiring data virtualization software and the like, to allow aggressive searching out and combination of all the new customer information constantly being created.  It also means buying the new infrastructure tools (database and information architecture) that can scale to handle aggregation of social-media data on the Web and in-house customer-experience data. 

Second, I would argue (not that I expect many businesses to listen) that in order to grow revenues as well as profits (and probably grow profits faster), business types need to acquire and foster use of agile tools and processes – such as agile marketing, with its emphasis on data-driven understanding of the customer and prospect rather than the anecdotal, top-down opinion that has been all too common in the past.  As I noted in a recent blog post, an agile business is a possibility, if the CEO and his lieutenants really commit to it – and it typically means not only the CEO using a Scrum-type planning process, but also constant modification and constant feedback up and down the organization about plans.  I should also note that an agile process specifically builds in slack for learning and review, and yet (according to my studies) it produces much better cost-saving and profit results than a “cram as much work as possible in, be as cost-efficient as possible” approach, for the CEO as well as the developer.

And then there is the really controversial stuff. Sustainability efforts are, by and large, really underperforming, as a recent article in www.thinkprogress.com confirms my impression that gains in the US in carbon emissions are effectively coming from offshoring the problem.  Businesses need to acquire effective global carbon-tracking software, and use it.  Businesses need to join to push shared quality regulations that will move all towards new computer and software technology that is more productive, rather than clinging to the infrastructure of the past that in the long run costs more, as in the electrical/energy grid vs. software-coordinated regional combined solar/wind – and rather than enabling the block-all-regulation political excesses of the U.S. Chamber of Commerce and its ilk.

I’m not saying that all this is doable, or that the case for questioning the approach of the overall business to computing technology is clear.  I am saying that it’s time we recognize the seriousness of the overall problem of which computing spend is a part, and we stop blaming the usual suspects.  Five years is enough.  Thirty years is enough.  In the long run, a cost-cutting strategy is not enough – and the long run is beginning to arrive.   

 

Why Can't the Mainframe (and the PC) Stay Decently Dead?

For more than 23 years, as an analyst, I have been watching various industry observers suggest that the end of the mainframe may be nigh.  For nearly that long, I have been hearing voices proclaiming the nearing death of the PC. For the mainframe, it was the proprietary architecture, the lack of developers, the aging of administrators, or the rise of scale-out.  For the PC, it was the expense of personal storage and its administration compared to the “network computer”, the lack of Web savvy, the large form factor compared to the smartphone, and the unaptness to touch gestures.

So what did the 2012 markets say?  Astonishingly, after a 10-20 % dip in revenues in the first 3 quarters, IBM saw a 56% mainframe-revenue jump in the fourth quarter, to overall revenue growth in 2012, better than the decreasing revenues of its Unix/Linux (System p) and Windows (System x) alternatives.  This extended the mainframe’s 3-year streak of revenue (and therefore income) growth. 

IBM was not alone in seeing these results.  HP apparently saw greater single-digit decreases in Unix/Linux than in PC revenues.  Anecdotal evidence suggests that Oracle/Sun Unix/Linux revenues continued to decrease.  And, of course, touch screen in a PC form factor had only just begun to arrive at the end of 2012 (with Microsoft Office in a hybrid tablet/PC configuration only apparently arriving some time in 2013) – so the PC was competing with smart phones and tablets with one hand tied behind its back, so to speak.

In fact, if 2012 was proclaiming anything, it was suggesting the eventual death of Unix/Linux (no, I don’t believe that either).

In an era in which IT is apparently content to spend the same amount each year on computing, much of it on saving costs – thus increasing computer maker profits but keeping revenues flat – it is not likely that any of today’s form factors, including the smart phone, is going to attain the 50%-+ growth rates that we have seen in the past.  Thus, we may have seen the popping of the Apple stock-market bubble. And therefore, in the near future, I assert that neither the mainframe nor the PC is going into terminal decline – on the contrary, they should prosper modestly, as the mainframe has done in the last 3 years.

So why will all the cited disadvantages of mainframe and PC not lead to steady or sudden terminal decline in the next 2-3 years, but rather to sober growth?  Let’s take each in turn.

The Mainframe Is Fully Reinvented

When I first suggested to IBM that the mainframe needed to become a “hub” in the sense of a fully networked node of especial prowess in certain types of workload rather than in the sense of “either mainframe or something else but not both”, it was only about five years ago.  Since then, the process of turning the mainframe into something that is a full participant in the enterprise architecture has been pretty completely achieved.  All of IBM’s major enterprise software, from administration to security to data management to development, now is on approximately the same track inside and outside the mainframe, and most apps can easily and even dynamically move between mainframe and non-mainframe servers.  As a result, users find it far easier to employ the mainframe flexibly for its strengths in robustness, security, and high-end transactional scale-up. 

Imho, IBM’s mainframe plans for 2013 contain no such dramatic transformations – and don’t need to.  The “bridging” architectures of zEnterprise and (eventually) of PureSystems form a completely adequate foundation for future elaborations in mainframe-including enterprise architectures.  To the end user, developer, and administrator, the mainframe if needed can appear more or less as a transparent part of a fully modern overall enterprise architecture. My only caveat, as noted before, is that the mainframe fails to support Windows in public cloud architectures adequately (again, imho) – but that simply limits growth, it does not portend decline.

And so, the old objections begin to melt away.  Few argue that the mainframe’s proprietary architecture cannot keep pace just as well in the near future with the evolution of hardware/software technology. IBM cites reports that a new generation of system administrators is arriving, with adequate skills, and therefore some new buys are even showing a preference for z/OS up front rather than Linux.  While the mainframe will never rival the hundreds of thousands of apps for Apple’s OS or Android out of the box, there are few major barriers to software today, either. 

To put it in a nutshell:  the mainframe is attracting new customers, even in the US, because the old knocks on the mainframe no longer apply, leaving it to leverage its strengths in a particular segment of the market that will grow with the growth of usage of Big Data.

The PC Keeps On Doing What It Does Best

The argument for the PC’s continued success is of a different sort from that of the mainframe.  I say, rather, that the doomsayers of the PC have not failed to appreciate adequately its long-term strengths.

Let’s start with the “network computing” argument.  Yes, as with the “dumb terminal” before it, the “network computer” is cheaper than the PC.  However, one value of the PC has been its ability to store personal data and applications, whether as a “home within an office” that allows the end user to generate his or her own Powerpoints and spreadsheets, or as a bridge between home and office to allow work wherever.  That is precisely why the countervailing trend of BYOD (Bring Your Own Device) – which is just as much if not more about laptops than smartphones and tablets – continues to ensure a major presence of PCs both at home and at the office.

As for Web savvy, once again the smartphone’s evolution has proven the value of personal physically-next-to-the-user storage – whether it be stored song downloads or phone logs – and hence the smartphone is turning inevitably into a PC – but one in which the small vs. large screen carves up turf between the PC blogger and the smartphone tweeter.  And, of course, my experience with Windows 8, wrong-headed as I believe some of Microsoft’s decisions are (e.g., crippling the desktop screen out of the box), shows that touch not only is valuable to the PC’s core word-processing and navigating/data-organization skills, but that it is not as easily applied to the smartphone’s small screen without a comparable, complementary typing feature (hunt-and-peck still doesn’t do it). And so, not only in terms of Web usage but also in terms of end-user-friendliness, the PC still holds its share of the market; and, for the same reasons, should continue to do so.

Note that I say that the relatively large form factor is a plus rather than a minus.  I assert that there is a “finger limit” in which the typing necessary to create large-scale amounts of real content can only be accomplished on a sufficient-sized screen.  Call it semantics, call it deep analysis, call it whatever you want, 50 lines is about the minimum for a decent blog post that does not amount to dipping your finger into a very large pool – and, as in this post, 2-3 old-style pages are more like a comprehensive look at a subject.  Note that folks like Paul Krugman use shorter blog posts as “riffs” to serve as the basis for a comprehensive multi-page paper, not as the definitive word on a subject.  A mashup is no substitute.  And so, the PC’s combination of personal storage and relatively large screen/keyboard should see continued modest growth in demand, in whatever form, over the next 2-3 years – depending, of course, on the world economy.

Sacrificing Some of the Future By Discarding the Present

I am long since resigned to periodic outbursts of “the death of …”, which sometimes are justified (I for one don’t lament the effective death of the dumb terminal, having had the dubious joy of programming for it).  What I do object to is the way in which these persuade many people in and out of the industry that the architectures in question are indeed dead, which means we don’t need to think about them, which means we don’t leverage their unique strengths in the next generation of the world-wide web of computing. 

Specifically, the mainframe pushes the limits of scale-up computing.  While scale-out grid solutions may indeed have achieved some notable successes (I understand that Google has made notable advances by thousands-of-servers divide-and-conquer applied to language comprehension and translation), on a per-processor basis scale-up’s tight integration continues to offer frequent performance advantages over scale-out’s loosely-coupled networking. 

On the PC end, the latest tablet/smartphone user interfaces are crippled by a lack of appreciation of foldering’s static personal data organization as a complement to the Web’s dynamic search-based organization. Moreover, we continue to move away from the idea of a “virtual end-user space” in which the full functionality and data of one’s own PC is available anywhere, any time, whether we are connected to the Web or not (and this will be needed because we are still quite a ways away from always-available and always-fully-functional/personalized Web avatars). 

Most of us remember the Monty Python routine in Holy Grail where one person attempts to fob off an alive person on a “dead collector”, while the corpse in question protests “I’m not dead yet!” To which the seller keeps replying “Shut up.”  For very good reasons, the mainframe and the PC refuse to stay decently dead.  Could we please not keep telling them to shut up?

Hey, Was I Actually Right About HP and Oracle?

HP's outlook won't be all that bad only because analysts are expecting Armageddon. – Larry Dignan, CNET, Nov. 21, 2011

Oracle Corp. /quotes/zigman/76584/quotes/nls/orcl ORCL +3.54% will likely beat analysts' estimates when it reports its fourth-quarter on Thursday. But investors will probably take their cue from what's expected to be a strong outlook … Big companies are investing heavily in Oracle's new generation of database software, as well as hardware it introduced since acquiring Sun Microsystems last year. – Steven Jones, MarketWatch, June 21, 2011.

I will stick out my neck and predict that HP will not implode over the next 3 years, and it will not fall behind IBM in revenues either, barring a truly epochal acquisition by IBM. – Wayne Kernochan, Infostructure Associates, blog post, May 17, 2011

.Sun’s UltraSPARC architecture has already shown signs of falling behind, and it appears that Oracle is not committing the necessary resources or expertise to play catch-up with IBM and Intel. … Exadata  … unless it is acquired by a leading-edge top-three-in-technology processor technology vendor such as Intel or IBM, … will start falling behind in scalability … It seems likely that within the next 3 years, Oracle will either load additional pricing on its locked-in customers to support another acquired platform, or deliver less and less relative price/performance … It may make sense for Oracle’s customers to move to a vendor such as IBM even if it will be painful in the short term. If my outlook for Oracle’s pricing and technology over the next 3 years is correct, the cost inefficiencies compared to others won’t get better, they’ll get worse; and the degree of database lock-in will stay the same or worsen, too, as additional Oracle implementations occur. – Wayne Kernochan, Pund-IT Review, Sept. 2011

Oracle’s hardware business will continue to look troubled while the Sun portion continues to shrink – Erik Hessedahl, All Things D, June 18, 2012

Oracle hardware revenues decrease by 10% in constant currency over 9 months, total revenues up about 3% for fiscal 2012 and 1% for 4Q12 - Oracle earnings press releases Mar/Jun 2012

Desktop units were up 5%, revenue down 3% yty for 2Q12, revenue run rate for fiscal 2012 about $121B – HP earnings press release 5/23/12

Revenue flat for 1Q12 yty, revenue run rate for fiscal 2012 about $99B – IBM earnings press release, 4/17/12

The Other BI: HP Vertica and Columnar Databases

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about BI today.

The Importance of Vertica-Type Columnar Database Technology to BI

Last year, I wrote a blog post saying that it was likely that HP would underestimate the columnar database technology in Vertica, and if so they were missing a major opportunity. In the last year, HP has been pretty quiet about Vertica, but I have partially changed my mind, to the point where I want to call attention to Vertica as a less visible candidate for IT buyers to get the full benefits of columnar database technology over the next 2-3 years.

Let’s start with columnar technology. Here, I want to go more in-depth into Vertica’s core technology than usual, because it’s an excellent way to begin to see the benefits of columnar beyond traditional row-oriented databases.

The original idea of Vertica was to recast the relational database to focus on the (data warehousing) case where there are few if any updates. The redesign started with the idea that the data should be stored in "columns" rather than rows; the details of this are that the columns themselves (because they don't have to follow relational dogma) can be stored in a highly compressed format, with lots of compression techniques like inverted list, bit-mapped indexing, and hashing, as appropriate. Thus, (a) the database can use the column format to zero in faster on the data that the query is gathering, (b) because the data is compressed an average of 10 times (according to Vertica), more data can be crammed into main memory for faster processing. Result: a claimed 10-100 times speedup in performance, comparable to in-memory databases but far more scalable. It also means the database can handle at least 10 times more data (say, 100 terabytes instead of 5) with the same performance for a given query; or that the data center can use an order of magnitude less storage.

Now, all this does not come without a cost, and the typical cost would at first seem to be speed of updating. That is, the column storage format requires more revision of the data stored on disk when an update arrives, so update is slower. But this is counteracted by the ability to load more of localized data at once into main memory in a compressed form, for faster in-memory updating. Only at update frequencies typical of old-style operational online transaction processing (OLTP) does the row-oriented relational database have a clear edge.

The elaboration of the design in Vertica is that the basic data is also stored as "projections" (aka materialized views). That is, a set of columns in a tuple is stored one (relational) way; each column also shows up in a projection, but the projection is cross-tuple (one from tuple A, one from tuple B, etc.). This accomplishes two things: one, it gives an alternative way of querying which may be faster than basic storage, and two, it gives redundancy and therefore robustness, in a similar way to RAID 5 (projections can be "striped" across disks).

Now, here's where things get really interesting. Practically speaking, today, in data-warehousing-type databases, updates via "load windows" are becoming more and more frequent, to the point where data is pretty up-to-date and updates are a bigger part of data warehousing. To keep "write locks" from gumming up performance (especially with column update being slower), Vertica splits the storage into a write-optimized column store (WOS; effectively, a cache) and a Read-optimized Column Store (ROS). Periodically, the WOS becomes the ROS. So the write locks for the updates only interfere with reads when there’s a mass update. At the same time, such a mass update can re-store whole chunks of the ROS for optimum storage efficiency. Moreover, to gain currency, the query can be carried out across the ROS and WOS. And, because there is all this redundancy, there is no need for logs—another performance improvement. Note that because of its redundancy, Vertica doesn't need to do roll-back/roll-forward nor backup/restore.

The net of all this for IT buyers is that columnar databases in general, and Vertica in particular, should be able to deliver on average much better performance than traditional relational databases in the majority of not-highly-update-intensive cases, due mostly to its compression abilities, and that addition of other technologies like in-memory technology to both alternatives will not alter this superiority.

The Relevance of HP Vertica to BI

This kind of approach cries out for integration with or development of sophisticated admin tools, expansion beyond data warehousing and analytics to “mixed” transactions in competition with the noSQL fad, better programming tools to build up a war chest of business/industry customized solutions, and using a relational database as an OLTP complement. The resulting data-management platform would be a solid alternative for all sizes of enterprise to the “relational fits all” or “let the thousand flowers bloom” strategies of most organizations.

Once this platform is in place, it needs to become the keystone of enterprise architectures, not just an analytics or business intelligence “super-scaling” engine. That means adding integration with semi-structured and unstructured data. It also means adding major functionality for handling content, and integration with storage software for additional performance optimization. And so, anticipating that HP would not do this, I criticized the HP acquisition of Vertica last year.

Well, two things happened: HP did more than I thought it would, and competitors did less. HP bought a company called Autonomy, which added semi-structured/unstructured data support. Necessarily, this takes Vertica beyond pure data-warehousing-style analytics into a more update-intensive world, and HP’s redirection of Mercury Interactive towards agile ALM (application lifecycle management) associated Vertica with better programming tools. Meanwhile, SAP took its eye off Sybase IQ with its focus on HANA, IBM at least temporarily walked away from its Netezza semi-columnar database technology, and Oracle’s columnar-optional appliance ran into questions about its long-term hardware growth path. In other words, the result of half a loaf from HP and less than half a loaf from everyone else is that Vertica is moving towards leadership status in delivering columnar database technology to all scales of BI and analytics.

Meanwhile, of course, only the deluded think that HP will suddenly vanish, while database technology and the rest of the new software embed themselves ever deeper in HP’s DNA. HP Vertica is going to be around for quite a while; and it will be an attractive option for quite a while.

Potential Uses of Vertica-Type Columnar-Based BI for IT

The use cases of a columnar database IT is straightforward. IT should use a columnar database in new projects as an alternative or complement to a traditional relational database, unless the operations are update-intensive, in which case row-oriented relational is preferred. As a complement, columnar databases operate on a “switching” basis, in which an overall engine decides which queries should be allocated to row-oriented, which to columnar, usually on the basis of whether two or more of the “fields” involved in an operation can be compressed highly by using a columnar format. Oracle (and, until recently, IBM Netezza) takes this approach; but IT can also do its own switching mechanism.

And that’s it. Over the next 2-3 years, if not already, columnar can scale as high as querying, can integrate with as many data types and upper-level tools and applications, and can evolve to greater performance/scalability just as rapidly as the traditional row-oriented database. In the long run, in a lot of use cases, and sometimes in the short run, that favors Vertica-type columnar.

However, right now, columnar requires in some cases to “grow into” its assigned role in a new project, by adding administrative tools for particular cases. Therefore, in most applications where 24x7 operation and an adequate level of customer response time is business-critical, relational row-oriented should still be preferred. That should leave plenty of analytical and other BI uses for which Vertica-type columnar database software will deliver an important performance advantage.

The Bottom Line for IT Buyers

Over the next few years, IT buyers can take one of two views: the author of this blog post is prescient, columnar will replace row-oriented in the majority of new applications in BI and other areas, and we should include columnar in all our short lists from now on; or, the author of this blog post is wrong about the future, but columnar is useful for some things right now, and trying to standardize on one database is a fool’s game that we no longer bother to try to play. If IT buyers hold the second view, then they should be focused on applying columnar to analysis of huge amounts of structured data with “sparse” fields where high compression is achievable – like five-field customer names (Mr. John Taylor Jakes, Jr.) and product codes. Spend the resulting improvements on increased performance, lowered storage costs, or both.

Again, this is not a matter of a pre-short list, unless you have a “gray area” BI project involving somewhat update-intensive or somewhat business-critical little-downtime apps, in which case you want to wait for columnar to evolve a little. In all other cases, HP Vertica should go on the short list along with the obvious others, like Sybase IQ. Right now, Vertica appears to be ahead both in some of the needed features to adapt to new analytics needs and in speed of evolution. One never knows – but over the next year, that leadership role may continue.

Above all, IT buyers should not listen to any FUD from traditional relational vendors suggesting that this is yet another new technology, like object databases, that will eventually fall to earth with a thud. Columnar database technology proved its superiority in many situations long ago in the non-relational world, with CCA’s Model 204, and has found uses continuously since then, like bit-mapped indexing. Most times there’s a fair BI matchup, as with some of the TPC benchmarks of the last seven years, columnar comes out well ahead. Under whatever name, columnar database technology is not going away. Therefore, its markets will continue to grow relative to row-oriented relational. For IT buyers, acquiring columnar BI solutions like HP’s Vertica is simply being smart and getting a little ahead of the curve.