Data Virtualization: Sorry, Forrester, It Seems I Disagree

According to Barry Brunelli of Techtarget, a recent Forrester Research report places IBM and Informatica at the top of the heap, ahead of Composite Software and Denodo – and I disagree.  I do, in fact, have a lot more respect for the data management folks at Forrester than I do for their development folks, who produced a  report a couple of years back with a very poor (imho) understanding of the nature of agile development.  And I do believe that Forrester deserves credit, compared apparently to Gartner, for recognizing both the increasing importance and the ongoing potential for business benefits of data virtualization.  However, I would continue to put Composite Software (now under Cisco) and Denodo ahead of IBM and Informatica in functionality, fit to customer need, and ongoing value-add in the immediate future.  Why?

The Importance Of Paying One’s Dues In Data Virtualization

The understanding of Composite’s and Denodo’s advantages begins with the fact that since the beginning, it has often been confused with a technology originally called EAI, or Enterprise Application Integration.  Both integrate data; but they start from a foundation that aims data integration at very different purposes.  EAI originally aimed (and still does, in some cases), to pass data between two or more enterprise applications, such as SAP and Oracle Apps.  As a result, they created gateways that converted this (usually bulk) data to a common format, and then retranslated as necessary to pass to the target enterprise app.  As it turned out, this conversion to a common format is exactly what is needed to provide a front end to handle data streaming to a data warehouse – and thus, EAI and ETL (extract, transform, load) tools share a fair amount of functionality.  However, there is no sense of urgency about this conversion; it is for populating a database, not for immediately providing an answer to a query.

By contrast, data virtualization from the start aimed to provide querying (and, eventually, updates) across multiple databases and data management tools.  That, in turn, meant leaving most of the data on the device on which it already resided, and converting and combining only those parts of the data needed for a result – and so, high-performance querying became part of the package from the get-go.  Moreover, figuring out how to optimize queries in this way effectively takes quite a while, and new data types (e.g., social media, Hadoop) and data stores (e.g., data from multiple clouds) keep coming along and must be handled.

As I recall, Composite Software have been continually refining their software since at least 2003.  IBM originally had a matching product (now apparently part of InfoSphere).  However, in the mid-2000s, IBM chose to focus on the newly-acquired Ascential (more of an EAI-type product) instead, and only recently have they begun to re-focus on data virtualization technologies, with the acquisition of an unstructured-data virtualization company and with increased (and welcome!) attention paid, notably during the recent Information Management conference.  Based on my last conversations with IBM, I suspect that they have a fair amount of work still to do to upgrade the unstructured-data acquisition’s cross-database querying with many more use cases, from cloud data to object, streaming/sensor, data-warehouse, and IMS/Informix data types – not to mention integrating it with Master Data Management, operational-data querying needs, and features such as information governance.  And, of course, I’ve left out such newer functionality as cross-database updates, cross-database access control, developer support, and administrator support.

Informatica, apparently, is starting from behind what IBM has.  For most of the last decade, it has been playing in the EAI and “data integration” (including ETL) space, but only over the last two or three years has it publicized its “data virtualization” capabilities – nor it is clear where it got its cross-database querying chops.  Certainly, most of the smaller players from 10 years ago are already acquired, and suffering under the negligent hand of their masters – Oracle, for example, acquiring an already-neglected AquaLogic product with its buyout of BEA.  In similar fashion, SAP has wound up with a Sybase-acquired product, and Red Hat with the granddaddy of data virtualization, MetaMatrix.  In any case, large marketing claims do not substitute for a demonstrated pedigree of functional development.

Lessons For Users

So where do I view Forrester as having gone wrong, and how can IT buyers avoid buying less than the needed functionality?  I don’t know for sure, but I suspect that underlying the Forrester take was (a) confusion between EAI-type and data-virtualization-type “data integration” as well as a misunderstanding of what “data virtualization” really means, and (b) a subconscious belief that when a large and a small company say they have something, typically the large company wins because of breadth of features and support.

Let’s take the confusion first.  I am one who wonders if “data virtualization” hasn’t caused as much confusion as attention. Originally, the technology was called Enterprise Information Integration, which at least gets across the idea that the technology delivers value-add (timely, cross-data-type-contexted “information”).  “Data virtualization”, however, suggests that the main value of the technology, like that of storage and server virtualization, is to provide a single view that allows better load balancing.  On the contrary, data virtualization products also provide the basis for global metadata repositories, distributed master data management data-store query optimization, cross-the-hybrid-cloud data discovery, developer data abstraction for longer-lasting code, single-key cross-database administration, and semi-automated data governance, not to mention cross-cloud querying.  Given these additional features, users, unlike Forrester, must carefully probe whether vendors aside from Composite Software and Denodo are really walking the walk.

For the same reason, (b) doesn't apply:  you can’t simply feel comfortable with the large company’s features and support, because the features and support may very well not cover the types of things that data virtualization does well out of the box.  To put it another way, at present, IBM and Informatica have excellent and extensive data-integration and EAI features; but trying to do flexible data management, Web data discovery and ad-hoc querying, near-realtime data warehousing, and global metadata repositories for data governance without a well-optimized data virtualization product is like trying to fight with one hand tied behind one’s back. 

Data virtualization now matters more than ever to you, the IT buyer.  Forrester admits it, IBM admits it, and it seems that folks like Microsoft are now beginning to admit it.  If you don’t get 90% of the potential benefit because someone told you to use flawed criteria, you will therefore be missing out on the things that make companies like Qualcomm achieve real value-add, not just now but well into the future.  Whether I’m right about Forrester or not, the important thing is not to sell data virtualization short. Now, go out and kick those tires – the right way.

Climate Change: The Winter of Our Discontent

Usually, I try to get right to the point in these blog posts, but in this case I’d like to take a side trip into the phrase “the winter of our discontent.”  Trust me, it relates to climate change.

The phrase comes from the very beginning of Shakespeare’s play Richard III, in which Richard himself sets the theme of the entire play – the denouement of the entire cycle of history plays – by saying:

“Now is the winter of our discontent/Made glorious summer by this sun of York.”

Richard then goes on to reveal that he does not share this happiness, cannot share it, and intends to destroy it.

I have always felt that interpretations of this have failed to consider one reasonably likely interpretation.  Over and over throughout the history cycle, once the rightful king (Richard lII) is deposed, people choose to start wars instead of trying to reestablish peaceful order.  As a final result, this produces a Richard III – a war hero with PTSD; a man whose “withered [right] arm” could just as easily be seen as a massively over-developed left arm, which in England then would have made him an especially effective fighter but also made him a sinister (Latin for left-handed), distorted monster to others; a man whose sense of humor is a desperate and failing attempt to overcome these demons.  Now let’s re-read:  “Now is the winter of our discontent [true, for Shakespeare] made glorious summer by this sun [untrue, for Shakespeare] OF YORK.”  Utterly sarcastic, that the son of one of the chief war-makers, a son whose job as king seems to be living it up, should solve the whole problem of disordered war.

By the way, the interpretation goes on all the way to just about the end, as this embodiment of English war destroys or wounds everyone around him, until even the kingship of England is less important to him than simply going on destroying – but with a sense of the absurdity of it all:  “A horse [to escape and fight again], a horse, MY KINGDOM [despairing irony] for a horse.” This, perhaps Shakespeare is saying, is the tragic solution to an unnecessary attempt to solve the original problem by deposition of the rightful king – to let those who have made a living from perpetual war destroy each other, and clear the path for a new peacemaker and a reversion to order.  Also by the way, Alison Weir estimates iirc that most of the nobility of England was killed in the Wars of the Roses, paving the way for Tudor upstarts to utterly dominate the peerage.

So what does this have to do with climate change?

Media Summer

OK, try this rephrasing of Shakespeare:  “Now is the winter of our discontent/Made glorious summer by the sun OF MEDIA COVERAGE.”  I trust you can hear my sarcasm.

Over the last few months, suddenly the media coverage of the news seems to be taking climate change seriously and producing many hopeful signs, where it rarely did before.  We have Years of Living Dangerously on Showtime garnering attention, sustainability being connected to the climate change movement, Bill McKibben being heard with greater attention, a spate of dire reports with notes that mitigation will be far cheaper from the IPCC, a greater focus on solar and wind technology, reports of isolated successes such as Germany’s ability one day to supply 74% of its needs from renewables, and on and on.  Where media coverage of climate change science last year, especially, was atrocious winter, now it seems as if hopeful signs of upcoming media-reported technological-fix summer abound.

And yet, if we look at the facts on the ground, things continue to get worse.  In yearly average, the rate of CO2 atmospheric growth is steadily going up, with the second largest rate of growth since measurements started in the 1950s.  China, now the largest carbon polluter (although not per capita – that’s still the US) promises massive solar investments and actually ramps up coal use to the point of unbelievable air pollution in major cities.  In the US, business talks sustainability and solar/wind technological-fix projects abound, but any decrease in US carbon pollution, data suggest, is counterbalanced by business investments that effectively export that pollution abroad.  Meanwhile, an el Nino “Kelvin wave” greater even than the one in 1998 that produced a previous spike in global temperatures is probably heralding weather patterns starting around July that give a heavy push to Arctic sea ice and permafrost melt, which in turn would mean more violent weather, warmth, and ocean level rise sooner than now expected.

Above all, the machinery of business and infrastructure, like the machinery of war in Shakespeare’s day, continues to grind on and produce a heavy counter-pressure to any significant change.  Businesses and hence nations continue to talk about developing Arctic oil and gas resources.  Despite serious questions about fracking’s effects and studies indicating that its improvement over oil and coal in carbon pollution is much more minimal than originally thought, fracking and natural-gas infrastructure continue to proliferate.

No, I’m not a Richard III.  But I can recognize the similarities between Shakespeare’s play and this situation.  Now is the winter of our discontent/Made glorious summer – no, it isn’t. 

Vocal Spring

At this point, inevitably, a reaction to these points says, you can’t just be negative; you have to encourage people in positive efforts.  That is true; but it’s just as true that if we do not clearly say that these positive efforts are nowhere near having significant impacts, at the same time as we applaud the positive effort, people suffer “compassion fatigue” – encouraging the next project, and the next, but getting tired of pouring their energies into things and still having little impact on the problem.

But there’s another reason to pair encouragement and honest realism.  I don’t have the exact quote from Keynes handy, but many are familiar with his saying, “In the long run, we are all dead.”  Few, however, understand what he meant.  Here’s my translation:  We [economists] set ourselves too easy a task if we say [however truthfully] that in the long run, these troubles will pass, the sun will shine again [and the economy will be back to functioning normally].  In the long run, we are all dead.  [In the meanwhile, we will all have unnecessarily suffered, because the economist did not identify a better solution and push the government to implement it.]

In other words, if we all simply say, these efforts are enough, rather than honestly facing the fact that as of yet they aren’t, we will unnecessarily be condemning ourselves and our grandchildren to greater suffering before we are dead, and we will be inevitably condemning our great- and great-great-grandchildren to far greater suffering and death, well before their “long run” arrives.

I do not want a real-world Hellish summer with High Water (Joe Romm’s phrase).  I do not want a Silent Spring with few or no species left, even human.  But if we are to get to a real Spring of hope, much less minimizing the disastrous impact of climate change, we must tackle how we are failing; we must not set ourselves too easy a task. 

Now should the winter of our discontent/Be yet made glorious spring, by toil, by honesty.  What do you know, I think it scans.

Has Anyone Actually Bothered to Read Feldstein???

The debate over the accuracy of Thomas Piketty’s book on wealth inequality now often cites Martin Feldstein’s Wall Street Journal piece – but commentary on Feldstein’s statements seems to me not to analyze what Feldstein wrote, to the point where I question whether anyone actually has read carefully what Feldstein said.  I have – and, frankly, my jaw dropped at both the apparently poor quality of the analysis and the seemingly huge disconnect with what things are like in the real world.

So, since no one else per Google seems to be doing so, let me set forth Feldstein’s core argument and why it seems not only flawed but wrong-headed.  Feldstein says that (a) wealth disparities in the US are much less than in Piketty’s analysis, and (b) there has been no increase in these disparities over the last 35 years.  Let’s take each of Feldstein’s core supporting arguments in turn.

First, “Ever-increasing inequality –true if people lived forever; but they don’t.  Cumulative effect diluted by estate taxes and sharing among children/grandchildren”.  To a first approximation, 40-50% of the US has negative or zero net worth, 49.9% has positive net worth (wealth) primarily in housing, and 0.1% has wealth primarily in stocks and bonds – those being the wealthy. As the last few years have shown, the trend has been towards a slightly greater percentage of the US having zero or negative net worth over the last 35 years, while housing over that period (if we recognize that for most of the 49.9%, housing is also for shelter, and hence has major expenses) has little net return, and per Piketty the wealthy are seeing 7% return on investment – while if they had invested in Vanguard S & P 500 index funds during that period, they would have seen about 8.5% return on investment. 

So let’s assume (reasonably), on average, that the megarich earn this return for 40 years before they die.  At, let’s say, a 6% investment advantage over the rest of us, the megarich have 10.2 times the net worth ratio at the end as they have at the beginning.  Now slice this by 35% under the old estate tax formulas, adjusted to reflect tax-avoidance strategies, or 25% under the new ones.  You’re still up by 6.7 or 7.7 times after those forty years compared to the rest of us.  Even with dilution by (on average) 2.2 beneficiaries, each beneficiary starts at 3 or 3.5 times where his or her predecessor did relative to the non-wealthy – the only thing that changes is a slow expansion of the “very rich” category, from, say, 0.09% over that 40 years to 0.11%. 

Think about it:  all that “dilution” simply means that every year, 1/40 of the total rich get a 70 % penalty if you worry about multiple beneficiaries, or 30% if not (which makes more sense), meaning that the rich get less than 25 % of their relative increase lopped off by estate-tax effects.  In other words, dilution doesn’t get close to affecting the fact of ever-increasing inequality.

Second, “total wealth grows over time roughly in proportion to total income [3.2% per year].” Feldstein cites problematic evidence:  Federal-Reserve-calculated flow-of-funds since 1960.  He doesn’t cite any details, so I have to guess at the best scenario for his argument, but clearly using Federal Reserve data meant for an entirely different purpose and not vetted against Census wealth figures is really questionable. 

More importantly, this says nothing about the relative share of that supposed increase going to the wealthy.  Consider, for example, the median wage ($30,000) versus reported dividend income from $100 million in an index fund -- $2.4 million.  We know that for the bottom 80%, over the last 35 years, real income has been flat or lower, and for the top 0.1%, it has increased more than 10 times.  We also know that 90% of the increase in productivity in the last 35 years has effectively accrued to the wealthy.  Either way, it says that even if the Fed is right about the rate of growth of wealth and income, 90-120% of that growth is going to the wealthy.  Again, increasing inequality, and apparently ever-increasing inequality.

And note that since the figures are since 1960, 30% of Feldstein’s analysis takes place in a world where inequality is not increasing – stacking the deck, as it were.  Finally, note that if the super-rich were investing in stocks, and therefore reported only dividends as income (until end of life, if not later), and dominated the wealth distribution, then you would see total income and wealth increase in tandem in the same way.

Third, Feldstein gets to the part where he is arguing that the appearance of increasing inequality is an artifact of the wealthy reporting more of their income – not wealth, income.  He says, more or less, that  with the passage of changes around 1980 plus tax reform in 1986 there was a strong incentive to shift investments from low-yielding tax-exempt investments to higher-yielding taxable investments.  This is ridiculous:  bonds were and are priced to yield the same net return on taxed and tax-exempt investments for the wealthy who buy them, and so we can be sure that something like ½ of the bond income was already reported – and, of course, this was a smaller proportion than stocks.  In fact, for the wealthy, over time, it makes no sense except for diversification to go for (taxable or otherwise) bonds instead of stock index funds.  And so, by any stretch of the imagination, the effect would apply to no more than 15% of the wealthy’s investments – a one-time effect over thirty-five years, for a net bump of 0.42%.   And, of course, what about what’s happened since then?

Fourth, Feldstein cites the income tax rate being lowered from 50% to 28%:  result, more taxable income reported (I guess, from the wealthy, although why this doesn’t apply in the same proportion to the rest of the top 10% I haven’t a clue).  This greater taxable income, claims Feldstein, comes from more work incentive, more payment as salaries, and reducing deductions. 

Oh, really?  On what planet is this guy living?  Over the last 30 years, we have seen more payment as stock options, continued heavy use of deductions, and (here we go again with the flawed “raise the income, they work less” income effect on labor) that lowering of tax rates had no effect on either the work incentives of the wealthy or, for that matter, those with subzero net wealth (40-50% of the whole).  If anyone was affected, it was the rest of the top 10%.

Fifth (again with the taxable income), Feldstein sees a “shift from business income to subchapter S” – i.e., what was reported before as business profits is now being reported as income to the wealthy from new subchapter S corporations.  Is Feldstein seriously pretending that most CEOs are the owners of their businesses, or that most wealth involves small corporations suitable for subchapter S?  Even if he isn’t, he again conflates wannabe wealthy who are getting high income to get into the charmed circle with those who are already wealthy and are getting most of their income from stocks.  And, of course, this is a one-shot effect, even if it existed. 

Sixth, Feldstein gets back partially to wealth.  “Calculations of income [and, I have to guess, wealth]  exclude Social Security [actuarial value], health benefits, and food stamps”, says Feldstein, resulting in much more in the way of relative income gains (and, apparently, relative wealth gains) to the non-wealthy, and much less to the wealthy.  By all means, Feldstein, undercut your previous argument about income rising at the same rate as wealth.  But don’t count Social Security, health benefits, and food stamps as wealth.  These mainly accrue to the bottom 50%, and have done so since the 1930s and 1960s, respectively.  Negative wealth is not the same as positive wealth.  With negative wealth, you spend all of the benefits you get when you get them, leaving you still in the general category of “negative to zero wealth”.  So trying to create a giant sum and saying, well, by decreasing negative wealth, this increases the relative share of the wealth of the bottom 50%, is not even close to a good way of calculating net worth.

Let’s run this one into the ground.  What happens if you take away SS, food stamps, and health benefits?  A lot of those people die – as they did before 1933.  In that case, they aren’t counted as part of total wealth and total income.  A reasonable figure might be a 50% difference, when these benefits are used, i.e., primarily after retirement – remember, only 1 in 10 potential SS beneficiaries lived to retirement age in 1933.  And yet, at that point, the negative-net-worth folks were still apparently around 40-50%.  And so, the maximum 25% reduction in the debt of the negative-wealth non-wealthy is partially to wholly counterbalanced by a 10% increase in the retired negative-to-zero-net-worth percentage of the population.  We may also note that the use of an overall actuarial value almost certainly ignores that the poor die much younger than the wealthy.

Then Feldstein makes the jaw-dropping undocumented assertion that this putting of SS and health benefits into the income column results in a much smaller rise in relative income of the wealthy.  Let’s go back to our previous example:  Adding on today’s median yearly (actuarial) Social Security (and probably health benefits), and factoring in the fact that this would be received by 1 in 2 of today’s median workers who survive to retirement age and in old age are not receiving other income, would result in income in today’s terms of a maximum of $60K instead of $30K.  So now, we are talking about comparing $2.4 million in income from the $100-millionaire to $60K instead of $30K.  That’s a smaller ratio, but certainly not a smaller rise.  And it is by no means significant. 

[extended note:  health benefits includes benefits during work days (through much of this period, available only to ½ of Americans, and decreasing over time) and Medicare after retirement – which is not guaranteed to rise with inflation.  Thus, translating this into money, especially without factoring in the 1/3 share of company health insurance that the large corporations are asking workers to pony up, means that health benefits run around $7,000 per year or less before retirement, and more later – but probably not the equivalent of $7,000 more per year.  Then we have “pensions”, apparently including 401(k)s and IRAs. Obviously, company pensions these days are worth far less as an annuity than promised, and both IRAs and 401(k)s suffer from both eventual income-tax application for those who actually have significant amounts in them.  Moreover, IRAs and 401(k)s have had 2 ½ % fees and poor diversification choices, leading to a 3.5-4 % investment penalty relative to a Vanguard fund, or perhaps a 2% return in real terms, if you’re lucky, over an average of about 20 years.  In effect, this allows a withdrawal of maybe $10,000 a year over your 10 years of life after retirement – and that’s optimistic.  Total: $14,000 plus $7,000 plus $9,000, or $30,000 on top of your median wage.  Again, we are moving after-retirement income to working days, and comparing it to rich whose investment income operates at the same pace before and after retirement.]

Then Feldstein completes the argument by saying “Count SS, health benefits, and pensions in wealth figures”.  I’m sorry, but this person is really clueless.  I just disposed of that one.

But really, we could have skipped the entire analysis above and applied some simple real-world observations.  This is one case where examples really are representative of the whole, because there are pretty few wealthy – I estimate 150,000 people (not households) with more than 8 million in net worth, and so looking at the top 10 or 100 billionaires gives a pretty good picture of that group.  So we have, in the top few, the Waltons, Trump, and the Koch brothers, all of whom inherited plenty of wealth, and increased it.  We also have the upper ranks  dominated by CEOs of major corporations and hedge fund upper management – and it should be clear that both have seen compensation including stocks (a large proportion, for CEOs) grow by 10-30 times over the last 35 years – in contrast to almost all of the rest of the population.  Or I can cite the experience of an acquaintance who is seeing far more income and a far faster increase in net worth from a multi-million-dollar inheritance converted to index funds than from previous periods of top-0.2% job income and little investment income.

The kindest thing I can say about Feldstein’s analysis is that it seems to take what he did as part of Reagan’s economic team, using unexamined assumptions and figures available at the time, and assume that these figures and assumptions can be extended beyond 1992.  The nature of the corporation, the types of and returns from investment, and government taxation (or lack thereof) were changing as he served, and have changed dramatically since then, and yet there is no real trace of these changes in his argument.  Instead, we have conservative shibboleths justified by 1980s statistical analysis that even then was not satisfactory.

So don’t, as Prof. deLong does, say that it’s a good analysis with a poor WSJ title.  Let’s lay the blame where it really is:  Martin Feldstein did it to himself.