A Short Look Back at 2016

I have found very little in the last month and a half to add to previous posts.  CO2 continues its alarming rise, to what will in all likelihood be more than 406 ppm (yearly average) by end of year, and while global temperatures have ended their string of monthly records, we are still on course for a 1.2 degree C rise since 1850, about 0.3 degrees of it in the last 2 ½ years.  The big news in the Arctic (and Antarctic) is an unprecedented low in sea ice extent/area, plus record high temps in the Arctic in December – but that continues a smaller trend evident in most of the first half of 2016. 

Meanwhile, on the computing side, relatively little real-world innovation happened this year.  In-memory computing continued its steady rise in both performance and applicability, with smaller companies taking more of a lead as compared to 2015.  While blockchain technology and quantum computing made a big news splash early in the year, careful reading of “use cases” shows that real-world implementations of blockchain are thin on the ground or non-existent, as most companies try to figure out how best to make it work, while quantum computing is clearly far from real-world usefulness as of yet.

The big news in both areas, alas, is therefore the election of Donald Trump as President.  In the climate change area, as I predicted, he is proving to be an absolute disaster, with nominees for at least six posts that are climate change deniers with every incentive to make the American government a hindrance rather than a help in efforts to change “business as usual.” 

In the computing area, we see the spectacle of some large computing firms offering their services in public to Trump, an unprecedented move based on the calculation that while being seen as cooperative may not bring any benefits, failure to act in this way may cause serious problems for the firm.  Thus, we see Silicon Valley execs whose workforces are not at all enthused about Trump acting in meetings with him as if he offers new business opportunities, and IBM’s CEO announcing ways in which IBM technology can aid in achieving his presumed goals. 

It may seem odd to give such prominence to the personality of the President in assessing either climate change or the computing industry.  The fact is, however, that all of the moves I have cited are unprecedented, and derive from Trump’s personality.  To fail to consider this in assessing the likely long-run effects of the “new abnormal” in both the sustainability field and the computing industry is, imho, a failure to be an effective computer industry analyst.  And while no one likes a perpetually downbeat analyst, one that continually predicts rosy outcomes in this type of situation is simply not worth listening to.

I look back on 2016, and I see little that is permanent to celebrate – although the willingness of the media to begin to report on and accept climate change is, however temporary, worth noting.  I wish I could say that there is hope for better things in 2017; but as far as I can see, there isn’t.

IBM's Quantum Computing Announcement: The Wedding of Two Wonderful Weirdnesses

Yesterday, IBM announced that the science of quantum computing had advanced far enough that IBM could now throw open to researchers a quantum computer containing 5 quantum bits, or “qubits.”  This announcement was also paired with striking claims that “[w]ith Moore’s Law running out of steam [true, imho], quantum computing will be among the technologies that we believe will usher in a new era of innovation across industries.”  Before I launch into the fun stuff, I have to note that based on my understanding of quantum physics and computing’s algorithmic theory, quantum computing appears pretty unlikely to spawn major business innovations in the next 5 years, and there is no certainty that it will do so beyond that point. 
 

Also, I suppose I had better state my credentials for even talking about the subject with some degree of credibility at all.  I have strange reading habits, and for the last 30 years I have enjoyed picking up the odd popularization of quantum physics and trying to determine what it all meant.  In fact, a lot of the most recent physics news has related to quantum physics, and this has meant that the latest presentations are able to a much greater degree to fit quantum physics into an overall physics theory that has some hope of making some sense of quantum physics’ weirdnesses.  As far as theory of algorithms goes, I have a much greater background, due to my luck in taking courses from Profs. Hopcroft and Hartmanis, two of the true greats of computing science and applied mathematics in their insights and their summarizations of the field of “theory of algorithms and computing.”  And, of course, I have tried to keep an eye on developments in this field since.

So where’s the fun?  Well, it is in the wedding of the two areas listed above – quantum physics and theory of algorithms – that produces some mind-bending ways of looking at the world.  Here, I’ll only touch on the stuff relevant to quantum computing. 

The Quantum Physics of Quantum Computing

I think it’s safe to say that most if not all of us grew up with the nice, tidy world of the atom with its electrons, protons, and neutrons, and its little solar system in which clouds of electrons at various distances orbited a central nucleus.  Later decomposition of the atom all the way to quarks didn’t fundamentally challenge this view – but quantum physics did and does.

Quantum physics says that at low energy levels, the unit is the “quantum,” and that this quantum unit is not so much like a discrete unit but as to a collection of probabilities of that particle’s “state” summing to one.  As we combine more and more quanta and reach higher levels of energy in the real world, the average behavior of these quanta comes to dominate, and so we wind up in the world that we perceive, with seemingly stable solar-system atoms, because the odds of the entire atom flickering out of existence or changing fundamentally are so small.

Now, some quanta change constantly, and some don’t.  For those that don’t, it is useful for quantum computing purposes to think of their collection of probabilities as a superposition of “states”.  One qubit, for example, can be thought of as a superposition of two possible states of spin of an electron – effectively, “0” and “1”.  And now things get really weird:  Two quanta can become “entangled”, either by direct physical interaction with other quanta or (according to Wikipedia), at least conceptually, by the act of our measuring what really is the state of the qubit.  To put it another way, the moment we measure the qubit, the qubit tells us what its value really is – 0 or 1 – and it no longer is a collection of probabilities.  [ Please bear in mind that this is a model of how quanta behave, not their actual behavior -- nobody knows yet what really happens]

It’s quantum entanglement by direct physical interaction pre-measurement that really matters to quantum computing – and here things get even weirder.  Say you have two qubits entangled with each other.  Then quantum entanglement means that no matter how far apart the two qubits are, when you measure them simultaneously the discovered value of one is some function of the discovered value of the other (for example, the two will yield the same value) – what Einstein called “spooky action at a distance.”  Physicists have already shown that if there is any communication going on at all between two qubits, it happens at hundreds of times the speed of light. 

The meaning of this for quantum computing is that you can create a computer with two qubits, and that quantum computer will have four states (actually, in some cases as many as 16), each with its own probability.  You can then perform various operations in parallel on each qubit, changing each state accordingly, and get an output (effectively, measuring the qubits).  That output will have a certain probability of being right.  Recreate the qubits and do it again; after several iterations, you very probably have the right answer.

One final important fact about qubits:  After a while, it is extraordinarily likely that they will decohere – that is, they will lose their original states.  One obvious reason for this is that in the real world, quanta are wandering around constantly, and so when they wander by a qubit during a computation they change that qubit’s stored state.  This is one of the things that make creating an effective quantum computer so difficult – you have to isolate your qubits carefully to avoid them decohering before you have finished your computation.

The Algorithmic Theory of Quantum Computing

 So now, hopefully, you are wondering how this strange quantum world of probabilities at a distance translates to faster computing.  To figure this out, we enter the equally strange world of the theory of algorithms.

The theory of algorithms begins with Godel’s Incompleteness Theorem.  Kurt Godel is an unsung giant of mathematics – one Ph.D. student’s oral exam consisted in its entirety of the request “Name five theorems of Godel”, the point being that each theorem opened up an entirely new branch of mathematics.  In terms of the theory of algorithms and computing, what Godel’s Incompleteness Theorem says, more or less, that there are certain classes of problems (involving infinity) for which one can never create an algorithm (solution) that will deliver the right answer in all cases.  Because there is a one-to-one mapping between algorithms and mathematical solutions, we say such problems are unsolvable.  David Hilbert the mathematician, at the start of the 20^th century, posed 20 hard problems for mathematicians to solve; several of them have been proven to be unsolvable, and hence uncomputable.

By the 1960s, people had used similar methods to establish several more classifications of problem “hardness”.  The most important one was where the minimum time for an algorithm was exponential (some constant times 2**n, where n is [more or less] the most computationally expensive operation in the algorithm) rather than polynomial (some constant times n ** q, q less than 3).  In the typical case, once n got above 30 or 40, exponential-minimum-time problems like Presburger arithmetic were effectively unsolvable in any time short of the ending of the universe.

It has proved extraordinarily hard to figure out where the boundary between polynomial and exponential algorithms lies.  One insight has been that if one pursues all the possible solutions to some problems on some imagined computer offering infinite parallelism, they can be solved in polynomial time – the “NP” class of problems.  Certain real-world problems like the traveling salesman problem (figure out the minimum-cost travel sequencing to n destinations for any traveling salesperson) are indeed NP-solvable.  But no one can figure out a way to determine if those problems are solvable in polynomial time without such parallelism – whether P=NP.  In any case, it would be a major boon if some computing approach allowed P-time solution of at least some of those problems.

But even if we can’t find such an approach via quantum computing, it would still be very worthwhile for all classifications of algorithms (except unsolvable, of course!) to find a way to get to an “approximate” solution faster.  Thus, for example, if we can come to a solution to the traveling salesman problem that 99 times out of 100 is within a small distance of the “real” best solution, and do so in polynomial time on average, then we will be happy with that.  And thus, despite its probabilistic nature – or possibly because of it – quantum computing may help with both the weird world of “presently effectively unsolvable” exponential-time problems and with the more normal world of speeding up solvable ones.

How All This Applies To Today’s Quantum Computing

The world of qubits as it exists today appears to have two advantages over “classical” computers in attempting to speed up computation:

1.       Greater parallelism.  That is, you don’t sequence the computations and output; instead, within each and every qubit, they are performed pretty much at the exact same time.

2.       The ability to get “close enough” probabilistically.  As noted above, you can choose how much likelihood of being right you want, and as long as it isn’t absolute certainty, the number of iterations needed to get to a “good enough” answer on average should be less than the number of computations in classical computing.

However, there are two significant disadvantages, as well:

(a) The “sweet spot” of quantum computing is the case where the number of inputs (my interpretation:  the number of needed “states” in the quantum computer) is equal to the number of outputs needed to reach a “good enough” answer.  Outside of that “sweet spot”, quantum computing must use all sorts of costly contortions, e.g., repetition of the entire computation the same number of times when the number of states does not cover all the possible outcomes you are examining to find the right answer.

(b) When you need a very high probability or certainty of getting the right answer, or, failing that, you need to know how likely you are to be very close to the correct answer, you need a lot more runs of your quantum computer.

To see how this might work out in practice, let’s take the “searching the database” problem and use (more or less) Grover’s Algorithm (no Sesame Street jokes, please).  We have 1023 different values in 1023 different slots in our database, and we want to find out if the number “123456” is in the database, and if so, where (sorry, it’s a small database, but no bit-mapped indexing allowed).  We declare that each possibility is equal, so there is a 1/1024 chance of each outcome.  We use 10 qubits to encode each such possibility as an equally probable “state”.

The classical computing algorithm will find-and-compare all 1023 values before coming to a conclusion.  Grover’s Algorithm states that on average, the quantum computer will take sqrt(1024) or about 33 iterations before (on average) it will have found the right answer/location.  Score one for quantum computing, right?

Well, leave aside for a moment the flaws of my example and the questions it raises (e.g., why can’t a computer do the same sort of probabilistic search?) Instead, let’s note that it appears that in this example the disadvantages of quantum computing are minimized.  I configured the problem to be in quantum computing’s algorithmic “sweet spot”, and effectively only asked that there be a greater than 50% probability of having the right answer.  If all you want to know is where the data is located, then you will indeed on average take 33 iterations to find that out.  If you want to be sure that the number is or isn’t in the database, then your best solution is the classical one – 1024 find-and-compares, without the added baggage of restarting that quantum computing brings with it.

And I haven’t even mentioned that so far, we’ve only scaled up to 5 qubits available for researchers, and that as we scale further and our computations take longer, decoherence becomes even more of a problem. 

So that’s why I dropped that downer of a conclusion above.  It’s early days yet; but there’s a lot of difficult work still to come in scaling quantum computing, with no guarantee of success, and the range of long-term use cases still appears less than broad – and possibly very narrow.

Oh, Well, Fun Take-Aways

So while we wait and wait, here are some wonderful weirdnesses of the wedding of quantum physics and theory of algorithms to think about:

·         You can’t be in two places at once.  Not.  Simply create two quantum copies of yourself in your present “state.”  Keeping them isolated (details, details), move one of them halfway around the world.  Measure both simultaneously.  They will both be identical at that moment in time.

·         Things are either true or false.  Not.  Things are either true, or they are false, or whether they are true or false is undecidable.

·         Explanations of quantum computing are often incoherent, and whether they convey anything of value is undecidable.

IBM and Blockchain: Limited But Real Value

Last week, IBM announced an expansion of its blockchain capabilities to fully support secure networks for whatever uses its customers find for the blockchain technology.  Key features include support on IBM Cloud for developers, management of the environment aimed at rapid deployment via IBM Bluemix, and enhancements to the Linux Foundation open-source HyperLedger Project code for those seeking to implement blockchain use cases at least partly outside the IBM software/services umbrella.  My initial take is that IBM is laying a solid foundation for blockchain use.  More important, I find that it is likely that, with help like this, customers of blockchain technology may well find that it delivers real value-add in limited but nevertheless concrete ways.  That is, blockchain implementations with foundations such as IBM’s can deliver much less than what its most enthusiastic proponents would claim, but can provide it more reliably and immediately than, say, data lakes at this stage in its lifecycle.

It Begins With the Blockchain Technology

At present, the word “blockchain” is surrounded by clouds of hype and techie jargon that make its nature unusually difficult to understand.  However, reading between the lines, it appears that blockchain technology has two key new approaches:

1.       A “blockchain” data unit that lends itself flexibly to use in medium-scale applications involving many-to-many business-to-business processes, such as “smart contract” management or business-to-business asset exchange involving “clearing” and the like.

2.       A fully-replicated distributed database that allows each participant a full view of the “state” of the “blockchain” data and automates some of the security involved, e.g., the ability to block a single “bad actor” from altering the data by requiring “consensus” among all copies of the replicated data as to the validity of a new transaction.

More specifically, a typical “blockchain” implementation creates a string of data together with the transactions on it, each part of the string being a “block” containing a transaction and the resulting data, and each block containing a “hash” of the previous block, in effect a backward pointer to the transaction/data block previous in time/sequence to this one.  Thus, each blockchain contains its own readily accessible “audit trail”, copied to every node and hence every user.

On the networking side, as noted, the distributed database uses pure peer-to-peer networking with no central manager or “master record”.  Thus, more or less, according to the description in Wikipedia, if one transaction comes in on one node saying the resulting block should be as follows and another disagrees, all nodes need to look at what all the other nodes are saying (step 1) and determine what the “consensus” is (step 2).  They can then all independently create a new block, because they will all come to the same conclusion.  The result is that if one “bad actor” comes in and tries to corrupt the data, the consensus will reject that attempt every time it happens, no matter what the alias of the bad actor or how many times the bad actor tries, or even if the bad actor keeps switching the node of access.

The limitations of blockchain technology also come from these two new approaches.  For one thing, the “blockchain” data unit’s use of a “chain” approach to data access means that the database is simply not as scalable in most if not all cases.  There is a reason that most databases optimize direct access to each data unit:  the time taken to go backwards through the whole chain as a series of “reads” is much greater than the time for going directly to the data unit at the end of the chain.  IBM clearly signals this limitation on scalability by noting its ability to scale to “thousands of users” – laughably small for a consumer application, but quite appropriate for many business-to-business applications.

On the networking side, the requirement of “consensus” can significantly slow the process of committing a transaction – after all, one of the participants may not be responsive at one time or another.  Therefore, blockchain technology is less appropriate where the fastest possible transaction performance is needed, and more appropriate in automating existing approaches such as “clearing” (netting out transactions between businesses and then having each participant pay/receive the differences), which can take up to 3 days in some banking apps.

What Users May Wish to Watch Out For

As noted above, blockchain technology can deliver real benefits, particularly in security, speed-to-perform, and flexibility/ease of implementation and upgrade, for business-to-business many-to-many applications – but it probably will require extra care in implementation.  One area that I believe users should consider carefully is the long-term scalability of the blockchain technology.  It should be possible to get around the limitations of a blockchain data unit by writing to a “blockchain” veneer that looks like a blockchain but acts like a regular database (in fact, this is what a relational database does at the storage level).  However, this means that the implementation must use such a veneer religiously, and avoid third-party code that “hard-codes” blockchain access.

A second implementation detail that I recommend is pre-defining the blockchain technology’s links to existing systems.  In many cases blockchain data will need to be kept in sync with parallel updates going on in IT’s existing databases.  It’s not too early to begin to think how you’re going to do that.  Moreover, the blockchain approach is not a security cure-all, and therefore requires careful integration with existing security systems.

A third implementation “key decision” is whether to use some version of Bitcoin in implementing a blockchain solution.  I would recommend against this.  First, digital currencies modeled after Bitcoin have been shown in the real world to be vulnerable to “gaming the system”; in economic terms, unlike a good paper currency, it does not serve as a good “store of value”.  Second, Bitcoin has scaled beyond the “thousands of users” consensus needed for the business-to-business apps discussed here – but, in order to do so, it is now using a very significant part of the resources of the World Wide Web.  If you use Bitcoin, you must connect to that very large Bitcoin implementation, which will mean that your network performance may well be adversely impacted by constant demands for consensus from the Internet.

A Different Viewpoint

Interestingly, Steve Wilson of Constellation Research has recently come out with a strongly worded argument against business use of blockchain technology, with two key arguments:  (1) It actually adds nothing to corporate security; and (2) It does nothing that the business’ existing database can’t do. 

While I find myself in agreement with many of his technical viewpoints, I find that he overstates the effectiveness in the use case I have described above of present corporate security and of existing business databases.  Over the years, such functions as contract management and “episode management” in health care have been relatively ill-served by being lumped in with the rest of corporate security and implemented typically in relational databases.  The blockchain model is more like an “accounting ledger”, and the ability to use such a model in business-to-business interactions as well as for security inside and outside each participating business is a key long-term source of usefulness for today’s businesses.  To put it another way, the trend in business is away from the “one size fits all” database and towards “databases for specific purposes” that can add additional “domain” security and “ease of development”.

All in all, as long as businesses are savvy about implementation, there does appear to be an important use case for blockchain technology.  In which case, support like IBM’s is right on the mark.

IBM Acquires Weather Company IT: Into the Data Unknown

The recent IBM acquisition of much of The Weather Company (TWC) – effectively, everything but The Weather Channel – is an odd duck.  I can’t remember anything quite like it in the computer industry before now.  Indeed, I suspect that IBM does not yet fully understand where in the search for better analytics via Big Data this new addition will take it.  To put it another way, IBM is stepping into an area where the ultimate use of the acquired data is to a surprising extent unknown.

The Potential Benefits of Launching Into the Data Unknown

As I noted in a previous piece about the initial IBM-TWC partnership, some core benefits are easy to see, and the acquisition simply extends them.  The infusion of analytics with weather data allows fine-tuning of customer and supplier behavior analysis and prediction – if it’s raining cats and dogs, the retail store may see more or less traffic than on a sunny day, and extreme weather will inevitably lead to decreased sales and deliveries.  Of equal long-term potential is the use of climate change science to avoid the “if this goes on” approach to weather forecasting that becomes increasingly unable to anticipate both secular year-round warming and increased occurrence of extreme events.  By taking over these functions, IBM allows its customers to drive weather-data acquisition as well as analytics in directions that a TWC-owned platform would likely not have done.

However, according to its presentation of the acquisition, IBM also views the TWC platform as a proven approach for leveraging Big Data in general for new insights, extending to the sensor-driven Web aka the Internet of Things (IoT) the innovative “cognitive” insights of IBM Watson.  I would argue, in fact that the TWC IT acquisition adds less – and more.  Less, because TWC is not on the cutting edge of the Fast Data “new technology wave” that I have written about previously.  More, because if and when it is combined with the raw “real-time sensor” data that governments collect about the weather and related location information (e.g., habitations, flora, fauna, and topography), it provides a simple “location state” of the individual customer or car or delivery truck that can allow as-yet-undefined risk reduction (avoiding the high-water spots in the road), “state of mind” assessment for selling purposes, and  services (rerouting based on likely washouts).

An Odd Way to Slice a Duck

Certain parts of the TWC acquisition put IBM effectively into areas that are not necessarily a good long-term fit – goodness of fit will depend on IBM’s execution of the acquisition. 

For one thing, acquiring the TWC web sites makes IBM effectively a media company.  Hurricane Sandy, in particular, created large followings for the blogs of certain weather forecasters.  While IBM can pretend that these are merely avocations for TWC personnel focused on The Weather Channel, the fact is that they represent an important news and analysis source for a significant segment of the public.  Whatever IBM does with the link between TWC content suppliers and the web sites will indicate whether IBM is going to run the media part of its slice of TWC into the ground or allow the weather forecasters and analysts that remain with TWC to continue leveraging their clout as experts into eyeballs and advertising revenue. 

And the question of who goes where brings up another area where IBM is venturing into new and not necessarily compatible territory:  weather and climate expertise.  As a recent IBM presentation citing weather data’s usefulness to insurance companies shows, applications to mobile car users that can reduce customer risks (avoiding or sitting out a storm) certainly have an upside to them, but what is not clear is whether the insurance company can use such a mobile app frequently enough to reduce overall customer risk substantially (and thereby increase profit margins). 

An obvious case where weather warnings may have an impact on customer risk profiles is a “black swan”, a seemingly unlikely event such as Hurricane Sandy.  However, in that case individual customer advice is likely to have only minor effect – rather, the weather experts must recognize the need for and drive proactive, outside-the-company warnings that ensure lots of customers take care.  TWC has such experts – but where is the place for them in IBM’s new weather-related organization?  Perhaps Watson will have such “domain knowledge” in the future; but it does not have it today, and so “weather analytics” badly needs such expertise.  And then weather analytics needs climate domain knowledge and climate change expertise, else predictive analytics applied to raw weather data will be far too frequently wrong.

The User Bottom Line:  Going Into the Data Unknown Is Good

I have dwelt on my concerns about IBM’s ability to get the most out of its new oddly sliced duck.  The fact remains, however, that, like Watson, a massive chunk of weather data is important not for its immediate applications, but for the ways in which users will drive new types of insights with it.  Just as Watson has spawned new “cognitive” ways to analyze data, driven by user use cases, so the early adopters of weather data will generate use cases that will suggest other applications that we cannot foresee.  The savvy user, therefore, will pick the right time and the right use case to begin to use weather data – and IBM’s TWC capabilities are an excellent place to start.

And if IBM figures out how to leverage TWC effectively in the areas that I have cited, so much the better for the user.  That the weather data will also lead IBM to get even more serious about climate change, I suppose, is too much to hope.  But hope I will.

In-Memory Computing Summit 2: Database Innovation

In the late 1990s, my boss at the time at Aberdeen Group asked me a thought-provoking question:  Why was I continuing to cover databases?  After all, he pointed out, it seemed at first glance like a mature market – because the pace of technology innovation had slowed down and nothing important seemed to be on the horizon.  Moreover, consolidation meant that there were fewer and fewer suppliers to cover.  My answer at the time was that users in “mature” markets like the mainframe still needed advice on key technologies, which they would not find because analysts following my boss’s logic would flee the field. 
However, as it turned out, this was not the right reason at all – shortly thereafter, the database field saw a new round of innovations centering on columnar databases, data virtualization, and low-end Linux efforts that tinkered with the ACID properties of relational databases.  These, in turn, led to the master data management, data governance, global repository, and columnar/analytic database technologies of the late 2000s.  In the early 2010s, we saw the Hadoop-driven “NoSQL” loosening of commit constraints to enable rapid analytics on Big Data by sacrificing some data quality.
As a result, the late-1990s “no one ever got fired for buying Oracle” mature market wisdom is now almost gone from memory – and databases delivering analytics are close to the heart of many firms’ strategies.  And so, it seems that the real reason to cover databases today is that their markets, far from being mature, are rapidly spawning new entrants and offering many technology-driven strategic reasons to upgrade and expand.
The recent 2015 In-Memory Computing Summit suggests that a new round of database innovation, driven by the needs listed in my first post, is bringing changes to user environments especially in three key areas:

1.       Redefinition of data-store storage tiering, leading to redesign of data-access software;

2.       Redefinition of “write to storage” update completion, allowing flexible loosening of availability constraints in order to achieve real-time or near-real-time processing of operational and sensor-driven Big Data; and

3.       Balkanization of database architectures, as achieving Fast Data turns out to mean embedding a wide range of databases and database suppliers in both open source and proprietary forms.

Real-Time Flat-Memory Storage Tiering:  Intel Is a Database Company

All right, now that I’ve gotten your attention, I must admit that casting Intel as primarily a database company is a bit of an exaggeration.  But there’s no doubt in my mind that Intel is now thinking about its efforts in data-processing software as strategic to its future.  Here’s the scenario that Intel laid out at the summit:
At present, only 20% of a typical Intel CPU is being used, and the primary reason is that it sits around waiting for I/O to occur (i.e., needed data to be loaded into main memory) – or, to use a long-disused phrase, applications running on Intel-based systems are I/O-bound.  To fix this problem, Intel aims to ensure faster I/O, or, equivalently, the ability to service the I/O requests of each of multiple applications running concurrently, and do it faster.  Since disk does not offer much prospect for I/O-speed improvement, Intel has proposed a software protocol standard, NVRAM(e), for flash memory.  However, to ensure that this protocol does indeed speed things up adequately, Intel must write the necessary data-loading and data-processing software itself.
So will this be enough for Intel, so that it can go back to optimizing chip sets? 
Well, I predict that Intel will find that speeding up I/O from flash storage, which treats flash purely as storage, will not be enough to fully optimize I/O.  Rather, I think that the company will also need to treat flash as an extension of main memory:  Intel will need to virtualize (in the old sense of virtual memory) flash memory and treat main memory and flash as if they were on the same top storage tier, with the balancing act between faster-response main memory and slower-response flash taking place "beneath the software covers."  Or, to coin another phrase, Intel will need to provide virtual processors handling I/O as part of their DNA.  And from there, it is only a short step to handling the basics of data processing in the CPU -- as IBM is already doing via IBM DB2 BLU Acceleration.

Real-Time Flat-Memory Storage Tiering:  Redis Labs Discovers Variable Tiering

"Real time" is another of those computer science phrases that I hate to see debased by marketers and techie newbies.  In the not-so-good old days, it meant processing and responding to every single (sensor) input in a timely fashion (usually less than a second) no matter what.  That, in turn, meant always-up and performance-optimized software aimed specifically at delivering on that "no matter what" guarantee.  Today, the term seems to have morphed into one in which basic old-style real-time stream data processing (e.g., keep the car engine running) sits cheek-by-jowl with do-the-best-you-can stream processing involving more complex processing of huge multi-source sensor-data streams (e.g., check if there's a stuck truck around the corner you might possibly bash into).  The challenge in the second case (complex processing of huge data streams) is to optimize performance and then prioritize speed-to-accept-data based on the data's "importance".
I must admit to having favorites among vendors based on their technological approach and the likelihood that it will deliver new major benefits to customers, in the long run as well as the short. At this conference, Redis Labs was my clear favorite.  Here's my understanding of their approach:
The Redis architecture begins with a database cluster with several variants, allowing users to trade off failover/high availability with performance and maximize main memory and processors in a scale-out environment.  Then, however, the Redis Labs solution focuses on "operational" (online real-time update/modify-heavy sensor-driven transaction processing).  To do this of course, the Redis Labs database puts data in main memory where possible.  Where it is not (according to the presentation), the Redis Labs database  treats flash as if it were main memory, mimicking flat-memory access on flash interfaces.  As the presenter put it, at times of high numbers of updates, flash is main memory; at other times, it's storage.
The Redis Labs cited numerous benchmarks to show that the resulting database was the fastest kid on the block for "operational" data streams.  To me, that's a side effect of a very smart approach to performance optimization that crucially includes the ideas of using flash as if it were main memory and of varying the use of flash as storage, meaning that sometimes all of flash is the traditional tier-2 storage and sometimes all of flash is tier-1 processing territory.  And, of course, in situations where main memory and flash are all that is typically needed, for processing and storage, we might as well junk the tiering idea altogether:  it's all flat-memory data processing.

memSQL and Update Completion

Redis Labs' approach may or may not complete the optimizations needed for flash-memory operational data processing.  At the Summit, memSQL laid out the most compelling approach to the so-called "write-to-storage" issue that I heard. 
I first ran into write-to-storage when I was a developer attending meetings of the committee overseeing the next generation of Prime Computer's operating system.  As they described it, in those days, there was either main memory storage, which vanished whenever you turned off your PC or your system crashed, and there was everything else (disk and tape, mostly) that kept this information for a long time, whether the system crashed or not.  So database data or files or anything that was new or changed didn't really "exist" until it was written to disk or tape.  And that meant that further access to that data (modifications or reads) had to wait until the write-to-disk had finished.  Not a major problem in the typical case; but where performance needs for operational (or, in those days, online transaction processing/OLTP) data processing required it, write-to-storage was and is a major bottleneck.
memSQL, sensibly enough, takes the "master-slave" approach to addressing this shortcoming.  While the master software continues on its merry way with the updated data in main memory, the slave busies itself with writing the data to longer-term storage tiers (including flash used as storage).  Problem solved?  Not quite.
If the system crashes after a modification has come in but before the slave has finished writing (a less likely occurrence, but still possible), then both the first and second changes are lost.  However, in keeping with the have-it-your-way approach of Hadoop, memSQL allows the user to choose a tradeoff between performance speed and what it calls a "high availability" version.  And so, flash plus master-slave processing plus a choice of "availability" means that performance is increased in both operational and analytical processing of sensor-type data, the incidence of the write-to-storage problem is decreased, and the user can flexibly choose to accept some data loss to achieve the highest performance, or vice versa.

The Balkanization of Databases:  The Users Begin To Speak

Balkanization may be an odd phrase for some readers, so let me add a little pseudo-history here.  Starting a bit before 100 BC, the Roman Empire had the entire region from what is now Hungary to northern Greece (the "Balkans") under its thumb.  In the late 400s, a series of in-migrations led to a series of new occupiers of the region, and some splintering, but around 1450 the Ottoman Empire again conquered the Balkans.  Then, in the 1800s, nationalism arrived, Romantics dug up or created rationales for much smaller nations, and the whole Balkan area irretrievably broke up into small states.  Ever since, such a fragmentation has been referred to as the "Balkanization" of a region.
In the case of the new database scenarios, users appear to be carrying out similar carving out of smaller territories via a multitude of databases.  One presenter's real-world use case involved, on the analytics side, HP Vertica among others, and several Hadoop-based databases including MongoDB on the operational side.  I conclude that, within enterprises and in public clouds, there is a strong trend towards Balkanization of databases.
That is new.  Before, the practice was always for organizations to at least try to minimize the number of databases, and for major vendors to try to beat out or acquire each other.  Now, I see more of the opposite, because (as memSQL's speaker noted) it makes more sense if one is trying to handle sensor-driven data to go to the Hadoop-based folks for the operational side of these tasks, and to the traditional database vendors for the analytical side.  Given that the Hadoop-based side is rapidly evolving technologically and spawning new open-source vendors as a result, it is reasonable to expect users to add more database vendors than they consolidate, at least in the near term.  And in the database field, it's often very hard to get rid of a database once installed. 

SAP and Oracle Say Columnar Analytics Is "De Rigueur"

Both SAP and Oracle presented at the Summit.  Both had a remarkably similar vision of "in-memory computing" that involved primarily columnar relational databases and in-main-memory analytics.
In the case of SAP, that perhaps is not so surprising.  SAP HANA marketing has featured its in-main-memory and columnar-relational technologies for some time.  Oracle’s positioning is a bit more startling:  in the past, its acquisition of TimesTen and its development of columnar technologies had been treated in its marketing as a bit more of a check-list item -- yeah, we have them too, now about our incredibly feature-full, ultra-scalable traditional database ...
Perhaps the most likely answer why both Oracle and SAP were there and talking about columnar was that for flat-memory analytics, columnar's ability to compress data and hence fit it in the main-memory and/or flash tier more frequently trumps traditional row-oriented relational strengths where joins involving less than 3 compressible rows are concerned.  Certainly, the use case cited above where HP Vertica's columnar technology was called into service makes the same point.
And yet, the rise in columnar's importance in the new flat-memory systems also reinforces the Balkanization of databases, if subtly.  In Oracle's case, it changes the analytical product mix.  In SAP's case, it reinforces the value of a relatively new entrant into the database field.  In HP's case, it brings in a relatively new database from a relatively new vendor of databases that is likely to be new to the user or relatively disused before.  Even within the traditionally non-Balkanized turf of analytical-database vendors some effective Balkanization is beginning to happen, and one of its key driving forces is the usefulness of columnar databases in sensor-driven-data analytics.

A Final Note:  IBM and Cisco Are Missing At the Feast But Still Important

Both IBM's DB2 BLU Acceleration and Cisco Data Virtualization, imho, are important technologies in this Brave New World of flat-memory database innovation; but neither was a presenter at the Summit.  That may be because the Summit was a bit Silicon-Valley heavy but I don't know for sure.  I hope to give a full discussion at some point of the assets these products bring in the new database architectures, but not today.  Hopefully, the following brief set of thoughts will give an idea of why I think them important.
In the case of IBM, what is now DB2 BLU Acceleration anticipated and leapfrogged the Summit in several ways, I think.  Not only did BLU Acceleration optimize main-memory and to some extent flash memory analytics using the columnar approach; it also optimized the CPU itself.  Among several other valuable BLU Acceleration technologies is one that promises to further speed update processing and, hence, operational-plus-analytic columnar processing.  The only barrier -- and so far, it has proved surprisingly high -- is to get other vendors interested in these technologies, so that database "frameworks" which offer one set of databases for operational and another for analytical processing can incorporate "intermediate" choices between operational and analytic, or optimize operational processing yet further.
In the case of Cisco, its data virtualization capabilities offer a powerful approach to creating a framework for the new database architecture along the lines of a TP monitor -- and so much more.  The Cisco Data Virtualization product is pre-built to optimize analytics and update transactions across scale-out clusters, so is well acquainted with all but the very latest Hadoop-based databases, and has excellent user interfaces.  It can also serve as a front end to databases within a slot/"framework", or as a gateway to the entire database architecture.  As I once wrote, this is amazing "Swiss army knife" technology -- there's a tool for everything.  And for those in Europe, Denodo’s solutions are effective in this case as well.
I am sure that I am leaving out important innovations and potential technologies here.  That's how rich the Summit was to a database analyst, and how exciting it should be to users.
So why am I a database analyst, again?  I guess I would say, for moments like these.

Lessons of Moore's-Law History 3: The Consumer is the Chasm

In the previous two blog posts, we have seen (hopefully!) the importance of betting on technologies and suppliers that most closely resemble the silicon-transistor-chip evolution process, and the need to bet on suppliers adhering to that process that emphasize infrastructure software combining openness to technology evolution with support for existing systems – but we have not dealt with the “chasms” that occur when Moore’s-Law evolution results in new, radically different higher-level technologies. 

Here I am combining two books popular back in the 1990s:  Crossing the Chasm, by Geoffrey Moore (no relation), which talked about how innovative startups bridged the gap between early techie adopters and the bulk of the market; and The Innovator’s Dilemma, by Clayton Christensen, which argued that “disruptive innovation” driven by new technologies would over time change existing markets profoundly, usually to the detriment of larger, well-entrenched companies presently dominant in an industry.  In an industry like the computer industry (including software and services), where Moore’s Law drives a more rapid pace of underlying improvement to platforms, entrenched companies must often react to disruptive innovation by “crossing the chasm” rapidly with their customers to the new technology, else they risk losing their markets and their ability to survive.

When A Company Is Ultimate-Customer-Blind

The 1990s and early 2000s gave us a good set of data on who succeeded and who failed in “crossing the chasm.”  The results were surprising to many.  It seemed reasonable that a Sun or an Oracle would surpass a Microsoft, that a Digital Equipment would beat a Novell, that an IBM would dominate a Compaq or a Dell in the PC market.  And yet, here we are 25 years later, and by no stretch of the imagination can we say that these things happened.

It seems to me that a common theme of this litany of successes and failures at adapting to new Moore’s-Law technologies is that companies that sold to the business market failed far more often than companies that sold to the consumer market.  Sun, which did a superb job jumping from the workstation to the server market, failed among other things to note consumer-driven Linux/Windows open-source software that was undercutting its prices and playing better with the server farms that later led to public clouds.  IBM’s initial great Charlie-Chaplin PC marketing fell before the next-day delivery and Intel allegiance of Dell and Compaq.  On a smaller scale, we saw Informix’s Oracle-protected VAR channel fail fatally when Informix failed to reach through that channel to detect that the servers it was shipping to VARs were no longer being bought at a comparable rate by the ultimate customers. 

On a longer time scale, the proprietary chip sets of IBM, HP, and Sun/Oracle have steadily lost ground compared to Intel markets.  It may seem logical that IBM over the last few years is walking away from the PC, starting with its consumer PCs/laptops; but that approach has steadily given it less and less sense of trends in the consumer market, and that, Moore’s-Law history would suggest, is dangerous.  It was once assumed that one’s employees would buy consumer versions of what a business-focused computer company offered; but, starting in the 1980s, the opposite has appeared true:  Word/Lotus, Excel, presentation software, laptops, smartphones, and social media have all been imported into businesses before and despite corporate standards.  If a computer company wishes to handle a disruptive innovation, Moore’s-Law history suggests, it must immediately sense the consumer movement, and then follow the consumer across the chasm, and quickly.

The Top Line is Now the Bottom Line

From 2008 on until fairly recently, a focus on cutting costs allowed companies like IBM, Oracle, and HP to look good to Wall Street although revenues were slightly down to slightly up.  Now, however, the contrast with consumer-focused companies like Google, Amazon, and Apple that continue to grow revenues by leaps and bounds is becoming all too obvious, and the chasm to cross to support the new mobile and cloud technologies is becoming wider by the day.  Generalizing to other industries, it is not enough to keep in sync with a Moore’s-Law-related process and focus on flexible, forward-compatible software.  One should also focus on growing top-line revenues by crossing the chasm with the customer. 

That means aligning platforms with consumer-successful companies like Intel.  That means aligning strategies with Google's, Amazon's, and Apple’s smartphone and cloud technology changes.  And that means far better coordination with and analytics about ultimate end-user customers – e.g., agile marketing.

Moore’s Law won’t last forever – it probably won’t last more than 5-10 years longer.  But the lessons of its history will last much longer than that.  Caveat seller.

IBM-Weather Company Alliance: The Benefits and Limitations of Weather-Driven Analytics

Make no mistake about it, combining IBM and The Weather Company (TWC) is about as good as it gets for companies looking to improve their responsiveness to today’s extremes of weather.  This is not just because IBM’s Watson Analytics promises to add in-depth business insights and additional predictability to your basic average weather forecast.  In fact, The Weather Company is if anything the more important partner, because it has established a solid track record for anticipating some of the more unusual effects of such events as Hurricane Sandy – and Jeff Masters of Weather Underground is an effective analyst with a good appreciation of the effects of climate change on weather patterns.

And yet, the recent announcement of the alliance also serves to show how today’s business analytics is severely handicapped in its ability to incorporate likely future changes in climate and their effect on business for all but the next few days.  To put it another way, IBM/TWC can help a Boston-based firm anticipate and adjust on the fly to a 2-foot snow 5 days from now; but to make the necessary repairs to buildings to avoid roof collapses and leaks, and to make plans to truck snow to more remote locations because local “snow farms” can’t handle so much snow, when there has never been a need before – well, that IBM/TWC probably won’t help with.   However, increasingly, that’s where much of the additional weather-related business cost these days resides.  (And of course, what goes for IBM/TWC goes for other analytics solutions in spades)

The real problem here is the lack of a global climate model that operates on the basis of the most likely climate changes over the next 30 years, not just the ones that can be verified by conservative scientific analysis.  To take one example, we know that melting permafrost is likely to have a significant effect in warming the globe by releasing carbon and methane over the next 30 years, but since we can’t rule out the possibility that it will have no effect, today’s climate models do not include permafrost melting effects. 

As a result, even if Watson were to incorporate such a model in its projections of weather over the next couple of years at business locations, it would be very likely to fail to anticipate the extremes of weather that would actually occur.  Thus, Heidi Cullen, another meteorologist savvy about climate change, wrote a book a few years back in which she projected that NYC would see a major hurricane with a dangerous rise in the water level that threatened the city – but with a near-miss in 2017 and with a greater Hurricane Sandy equivalent happening in 2041.

A Quick Fix

So what’s a poor business to do?  Well, as of now, someone within the organization needs to develop enough expertise in today’s climatology to be able to create an informal “model” based on current research that can project possible regional climate changes for business and sales locations over the next five years – assuming non-zero global-warming effects of “known unknowns.”  Then, the business needs to run “what-if” weather scenarios that include these new possible extreme weather events – high winds, new hurricane locations, extreme rain and snow, extreme drought – and also their ancillary effects – water shortages, stressed utilities, difficulties getting to stores and work, impoverished consumers.  TWC’s expertise would be critical in determining the “penumbra” of related weather – geographical size and track of an extreme storm, frequency/length of extreme droughts – while IBM’s analytics would scrutinize the unfolding scenario for business impacts of ancillary effects.  This approach would do a far better job of adequately preparing the firm for the medium term (say, 2-5 years). 

Ah, but then there’s the longer term, 5-35 years out.  What prepares the firm for medium-term adaptation may very well prove to be more costly than adaptation that gives up certain locations as lost causes – and far, far more costly than the firm taking its fair part in effective global mitigation.

Getting Specific

What follows is necessarily speculative and US-centric, but should give an example of both medium-term and long-term business thinking.  Today’s “new climate” as it plays out over the next 2 ½ years is characterized by a modified “el Nino/la Nina” cycle and a new Arctic impact on weather, especially in winter.  High temperatures along various parts of the Pacific have attenuated el Nino and delayed its onset, but over the next 2 years, and possibly for longer, we should be in an el Nino phase characterized by higher temperatures than normal, especially in the western US.  At the same time, extensive warming of the Arctic combined with a mild el Nino should lead to significantly higher Arctic temperatures in the summer (and winter), and may redirect the cold and “extreme snowfall” of this winter to the European side of the Atlantic.

However, at the same time, Greenland ice melting has been putting a fresh-water “cap” on the area where the Gulf Stream dives down to the deep ocean, slowing and weakening the “force of flow” of the Gulf Stream.  At the same time, the greater increase in winter temperatures in the Arctic due to global warming compared to the continental US has made it easier for Arctic cold to “punch through” the jet stream and park over the eastern US for weeks at a time.  The increased warmth of that Arctic air means increased incidence of “extreme snowfalls” either in the eastern US or (in the other phase of the “Arctic oscillation”) in northern Europe.

Combining the two effects above with ongoing global warming, and we might project winters similar to the last 2 in the eastern US, but with a shorter snow season and slightly warmer temperatures.  The southern and western US may see significantly higher summer temperatures, and we cannot anticipate that the drought in the California area will be overcome by enough rain to replenish reservoirs, especially with little or no snowpack in the Sierras.  Finally, we might anticipate that there may be another out-of-season Hurricane Sandy or 75-80-mph nor’easter in the Northeast, with higher-wind-speed hurricanes in the South.  Overall, the globe should see a faster increase in temperatures year-round, leading to record high temperatures over much of the globe.  Businesses should review not only their dependence on water but also the reliability and cost-effectiveness of their air conditioning.

Beyond that, in the next 5-35 years, another main actor is sea level rise.  It is plausible that on average, global sea level will rise almost a foot over the next 35 years, and the slowing of the Gulf Stream means that sea level rise along the East Coast may even double that.  Sooner than expected, then, cities such as Miami and perhaps even Boston are under threat, and increased wave levels during more-violent storms mean faster coastal erosion and destruction.  Businesses should rethink their investments in locations less than 25 feet above sea level, in areas of high drought (western and possibly southern US), and in areas requiring high levels of air conditioning costs.  Analytics such as that available via IBM/TWC plus “what-if” scenarios should allow fine-tuning these shifts in investments within regions.

The Bottom Line

One line credited, I believe, to an IBM employee way back when, said of programs in general what could well be said of “business as usual” weather-related analytics:  “Garbage in, garbage out.”  In other words, if you assume that future weather will be like the past, with today’s climate change, that assumption is untrue “garbage”; and the result of your analytics will be either a 5-day horizon that gives you that long to make major changes in your current setup at various locations, or a very misleading guide to your multi-year strategy’s chance of avoiding major unnecessary weather-related costs – a medium-term forecast that is “garbage” as far as effective planning is concerned.

I have suggested one approach to overcome this problem; but, really, the first step is to recognize that dealing “just in time” with a weather problem is far less than half the battle, and to do something about it.  If you do create an approach along the lines I have suggested, however, IBM/TWC is an exceptional partner, because TWC “gets” climate change and because IBM via Watson and the like can dig deeper for related business effects, once the data it is fed is no longer “garbage.”

One final thought:  I and others like Joe Romm have said that mitigation (avoidance of further carbon emissions and hence far greater climate change) was and is far less expensive than adaptation such as the steps we have talked about here.  Unfortunately, in the 5-7 years since this point was made, most businesses have failed to have much positive impact on mitigation, either in their investments or politically.  As a result, the costs of mitigation have not increased significantly, but the costs of the adaptation that has now become necessary have risen quite a bit.  To put it another way, ignoring mitigation has guaranteed much more in the way of adaptation costs – and we may anticipate the same happening over the next 5-35 years, unless most businesses change both their political and investment practices in a much more drastic way.     

Data Virtualization: Sorry, Forrester, It Seems I Disagree

According to Barry Brunelli of Techtarget, a recent Forrester Research report places IBM and Informatica at the top of the heap, ahead of Composite Software and Denodo – and I disagree.  I do, in fact, have a lot more respect for the data management folks at Forrester than I do for their development folks, who produced a  report a couple of years back with a very poor (imho) understanding of the nature of agile development.  And I do believe that Forrester deserves credit, compared apparently to Gartner, for recognizing both the increasing importance and the ongoing potential for business benefits of data virtualization.  However, I would continue to put Composite Software (now under Cisco) and Denodo ahead of IBM and Informatica in functionality, fit to customer need, and ongoing value-add in the immediate future.  Why?

The Importance Of Paying One’s Dues In Data Virtualization

The understanding of Composite’s and Denodo’s advantages begins with the fact that since the beginning, it has often been confused with a technology originally called EAI, or Enterprise Application Integration.  Both integrate data; but they start from a foundation that aims data integration at very different purposes.  EAI originally aimed (and still does, in some cases), to pass data between two or more enterprise applications, such as SAP and Oracle Apps.  As a result, they created gateways that converted this (usually bulk) data to a common format, and then retranslated as necessary to pass to the target enterprise app.  As it turned out, this conversion to a common format is exactly what is needed to provide a front end to handle data streaming to a data warehouse – and thus, EAI and ETL (extract, transform, load) tools share a fair amount of functionality.  However, there is no sense of urgency about this conversion; it is for populating a database, not for immediately providing an answer to a query.

By contrast, data virtualization from the start aimed to provide querying (and, eventually, updates) across multiple databases and data management tools.  That, in turn, meant leaving most of the data on the device on which it already resided, and converting and combining only those parts of the data needed for a result – and so, high-performance querying became part of the package from the get-go.  Moreover, figuring out how to optimize queries in this way effectively takes quite a while, and new data types (e.g., social media, Hadoop) and data stores (e.g., data from multiple clouds) keep coming along and must be handled.

As I recall, Composite Software have been continually refining their software since at least 2003.  IBM originally had a matching product (now apparently part of InfoSphere).  However, in the mid-2000s, IBM chose to focus on the newly-acquired Ascential (more of an EAI-type product) instead, and only recently have they begun to re-focus on data virtualization technologies, with the acquisition of an unstructured-data virtualization company and with increased (and welcome!) attention paid, notably during the recent Information Management conference.  Based on my last conversations with IBM, I suspect that they have a fair amount of work still to do to upgrade the unstructured-data acquisition’s cross-database querying with many more use cases, from cloud data to object, streaming/sensor, data-warehouse, and IMS/Informix data types – not to mention integrating it with Master Data Management, operational-data querying needs, and features such as information governance.  And, of course, I’ve left out such newer functionality as cross-database updates, cross-database access control, developer support, and administrator support.

Informatica, apparently, is starting from behind what IBM has.  For most of the last decade, it has been playing in the EAI and “data integration” (including ETL) space, but only over the last two or three years has it publicized its “data virtualization” capabilities – nor it is clear where it got its cross-database querying chops.  Certainly, most of the smaller players from 10 years ago are already acquired, and suffering under the negligent hand of their masters – Oracle, for example, acquiring an already-neglected AquaLogic product with its buyout of BEA.  In similar fashion, SAP has wound up with a Sybase-acquired product, and Red Hat with the granddaddy of data virtualization, MetaMatrix.  In any case, large marketing claims do not substitute for a demonstrated pedigree of functional development.

Lessons For Users

So where do I view Forrester as having gone wrong, and how can IT buyers avoid buying less than the needed functionality?  I don’t know for sure, but I suspect that underlying the Forrester take was (a) confusion between EAI-type and data-virtualization-type “data integration” as well as a misunderstanding of what “data virtualization” really means, and (b) a subconscious belief that when a large and a small company say they have something, typically the large company wins because of breadth of features and support.

Let’s take the confusion first.  I am one who wonders if “data virtualization” hasn’t caused as much confusion as attention. Originally, the technology was called Enterprise Information Integration, which at least gets across the idea that the technology delivers value-add (timely, cross-data-type-contexted “information”).  “Data virtualization”, however, suggests that the main value of the technology, like that of storage and server virtualization, is to provide a single view that allows better load balancing.  On the contrary, data virtualization products also provide the basis for global metadata repositories, distributed master data management data-store query optimization, cross-the-hybrid-cloud data discovery, developer data abstraction for longer-lasting code, single-key cross-database administration, and semi-automated data governance, not to mention cross-cloud querying.  Given these additional features, users, unlike Forrester, must carefully probe whether vendors aside from Composite Software and Denodo are really walking the walk.

For the same reason, (b) doesn't apply:  you can’t simply feel comfortable with the large company’s features and support, because the features and support may very well not cover the types of things that data virtualization does well out of the box.  To put it another way, at present, IBM and Informatica have excellent and extensive data-integration and EAI features; but trying to do flexible data management, Web data discovery and ad-hoc querying, near-realtime data warehousing, and global metadata repositories for data governance without a well-optimized data virtualization product is like trying to fight with one hand tied behind one’s back. 

Data virtualization now matters more than ever to you, the IT buyer.  Forrester admits it, IBM admits it, and it seems that folks like Microsoft are now beginning to admit it.  If you don’t get 90% of the potential benefit because someone told you to use flawed criteria, you will therefore be missing out on the things that make companies like Qualcomm achieve real value-add, not just now but well into the future.  Whether I’m right about Forrester or not, the important thing is not to sell data virtualization short. Now, go out and kick those tires – the right way.