Groucho Marx Syndrome Lives

Over the years, I have discovered a psychological trait that I call Groucho Marx Syndrome. Here’s the way it goes:

In the movie Duck Soup, Prime Minister Groucho Marx has been insulted by the leader of a neighboring country, and has slapped him in return, putting both countries on the brink of war. He is told that the leader will come to apologize, and agrees to meet him. Then he thinks out loud (from my memory of the movie), “I will hold out my hand in friendship, and we’ll shake. But suppose he doesn’t? That would be a fine thing, wouldn’t it, me holding out my hand and his refusing to shake it? What kind of cad would do such a thing? Me, holding out my hand and him refusing to shake it? How dare he! How dare he refuse to shake the hand of friendship!” And just then the leader walks up and Groucho shouts, “So, you refuse the hand of friendship? You cad!” and slaps the leader again. War ensues.

The point of Groucho Marx Syndrome is that certain people are prone to convince themselves that other people not only will do something bad to them, but have already done so – even when those other people haven’t, and won’t, do any such thing. Over the years, I have seen many worriers lapse into instances of Groucho Marx Syndrome – say, when they anticipate being fired and are hostile to the boss who they believe has already arranged to fire them, thereby causing the firing. But there are also people who are in the habit of convincing themselves that, behind their backs, most other people are spending their time undercutting them -- even to the point of saying to others “I know you made sure my application was turned down” when the application has not yet actually been decided on. For those people, behavior like that of Groucho Marx is habitual, is a Syndrome.

Over the years, I haven’t seen as much evidence of Groucho Marx Syndrome in public figures. Today, however, I was delighted and distressed to come across what seems to me be an excellent example of this psychological problem. Here’s the relevant quote from Paul Krugman’s blog:

“… one of Germany’s Wise Guys Men has lashed out at me in Handelsblatt over my criticism of Axel Weber:

Wolfgang Franz, who heads the German government’s economic advisory panel known as the Wise Men, tore into Krugman — and the US — in an op-ed in the German business daily Wednesday, titled “How about some facts, Mr. Krugman?”

“Where did the financial crisis begin? Which central bank conducted monetary policy that was too loose? Which country went down the wrong path of social policy by encouraging low income households to take on mortgage loans that they can never pay back? Who in the year 2000 weakened regulations limiting investment bank leverage ratios, let Lehman Brothers collapse in 2008 and thereby tipped world financial markets into chaos?” he wrote.

Now, what may not be obvious from this quote is that Paul Krugman is not a representative of the United States; far from it. Moreover, he was in some respects a prescient critic of some of the actions that Mr. Franz cites (in the case of encouraging low income households to take on mortgage loans that they can never pay back, it appears that Mr. Franz has his facts wrong, as Dr. Krugman notes). By contrast, I can find no obvious record in Mr. Franz’s publications of such anticipatory criticism, and a CNBC interview on Aug. 23, 2008 finds Mr. Weber not only apparently oblivious to the problems cited by Mr. Franz but also completely wrong about the medium-term and long-term outlook for the European economy. But that isn’t the real point here.

What Mr. Franz appears to be saying is, “The United States government is about to come in and tell us how to be prudent economically. How dare they? Aren’t they the folks who caused this mess? Aren’t they the folks who preached at the whole world that they knew better than everyone else? And now they’re coming here and telling us, who were ever so conservative economically, that we are doing it all wrong. What gall! They show up and tell us we’re running things wrong! Well, we won’t stand for it! We’re going to tell them off, right now!”

The rest of Mr. Franz’s editorial, translated by Bing, as I read it, appears to show some of the same style of thinking. Continue debt-backed investment and delay spending cutbacks? But everyone knows the US is just going to delay cutting back until too late; so they’re already too late, so we should cut back now. What about the continuing effects of German cutbacks on European policy and thereby on the rest of the world? Clearly we’re talking about our effects on the US. Our trade with them is miniscule; so there won’t be any effects, so there haven’t been any effects, so we have refuted that argument.

Just to reiterate:

(1) Paul Krugman is not the US government.
(2) By and large, there are different people running the US government’s economic policies and/or with a different message than those who were running it in 1980-2008.
(3) Mr. Weber has responsibility for the European as well as the German economy, and will have more responsibility for Europe in the near future as head of the ECB, which is apparently what Dr. Krugman is addressing.
(4) Today’s US government has not, as far as I can tell, officially agreed with Paul Krugman’s criticisms or recommendations yet.

With all undue respect, Mr. Franz appears to show strong signs of Groucho Marx Syndrome. The consequences of this Syndrome, between two people in ordinary life, are unnecessary ruptures and fights that benefit no one, least of all the person with the Syndrome. But, as in Duck Soup, some of these people represent nations. Let us hope that Groucho Marx Syndrome is not widespread among the actors in the present economic crisis.

Or economic war could ensue.

The Last 50 Years: Not What I Expected

For some reason, I was remembering today the predictions of the early 1990s that “all the great ideas have been found” already, and that little remained to be discovered. I asked myself, compared to 50 years ago when I first encountered various scientific and social-science disciplines, how far we had come since then? What had I expected, and what did we get? What was new and made me rethink things? What had made a big splash but was still not really revolutionary enough? Here’s my list.
Physics. I hoped for fusion as a new power source by now. I hoped for new containment of antimatter as a rocket fuel. I hoped for a full explanation of quantum phenomena. I can’t say any of these things has arrived. Compared to them, greater knowledge of subatomic particles and the beginnings of the Big Bang, better Grand Unified Theories, and the like seem like small change. Maybe in the next 50 years …
Environment. Of all the areas of scientific inquiry, this I believe has brought forth the most astonishing new ideas. Despite the threat of climate change, I am happy I have lived to learn the subtlety and variety of the chemical cycle between earth, sky, and life, now and in the past. In the 1960s, none of this was apparent; there was no concept of ecosystem, and the effects of climate on everything were barely beginning to be understood. I don’t care if we still can’t predict long-term weather well; what has been discovered is fascinating.
Biology. I give this mixed reviews. It seemed apparent to everyone in the 1960s that so much about the human body remained to be discovered. Now we have sequenced the genome and are beginning to identify artificial proteins, and yet the effects on disease treatments and our knowledge of the human brain are still slow to arrive. Granted, knowledge of our DNA ancestry is a great new window into prehistory and anthropology; nevertheless, this belongs more to applications than to fundamental new ideas.
Economics. Since my first economics class in college, I have been hoping for insights into how economies evolve with technological change. Yes, the theory of productivity has advanced; but not to the point where policy-makers can exert much control over it. The one area where I feel a surprising amount of good new ideas have come forth is international economics, where I believe there is a far greater understanding of factors such as exchange rates, an understanding that does not prevent horrible world-wide crashes but gives a smart policy maker the chance to avert them. We appear to know much more about gold as a standard, about fixed exchange rates, and about central-bank policy than we did back in the 1960s, or even the 1980s. Alas, the intervention of politics and the interests of the rich into economic research have prevented much more from being done.
Mathematics. I confess to being spoiled by the efforts of giants such as Godel before I was born in discovering the concept of unknowability. While chaos theory and networking theory have been fascinating, I think that the work of the 1970s in applied mathematics, in exploring undecidability and “effective” uncomputability (NP-complete algorithms), not to mention “problem solution lower bounds”, are surprisingly profound and should rank among the top new ideas in mathematics in the last 50 years – certainly not something I would have expected, since computers were barely visible to me in 1960. Of course, since I have spent my life since then in efforts related to computer science, I may be prejudiced; but I still believe that those who learn about unknowability grapple with a shift in perception almost as significant as that faced by those looking at Einstein’s theory of relativity in the early 1900s.
Computer Science. As Sesame Street would say, one of these things is not like the others … but I throw it in only to discard it. Yes, the advent of computers has changed our way of life profoundly over the last 50 years in ways that the sciences listed above cannot match; but really new ideas that profoundly alter the way one looks at the world? Unless you include the undecidability results of the 1970s – and those I prefer to classify under mathematics – I don’t think so. Nanotechnology? Application of lasers? Email? The Web? As technologies, their effective is pervasive; but I have never felt they offered a fundamental alternation in my view of the world. Maybe if artificial intelligence or linguistics had advanced further, those areas would provide new insights – certainly the idea that intuitive thought finds exceptions more powerful as learning aids than corroborations is interesting. But no; I don’t see any blockbusters here.
Looking over this list, I am struck by how wrong I was about the direction from which fundamental new scientific ideas would come over the last 50 years. It makes me very cautious about treating science as leading steadily into new technology, and technology as leading steadily into productivity and better living standards. I could be happily underestimating; I could be unhappily overestimating. One thing, though: the arrival of fundamental ideas has not flagged, and I can hope to be pleasantly surprised for the next 50 years – or whatever.

EMC World: The Dedupe Revolution

What, to my mind, was the biggest news out of EMC World? The much-touted Private Cloud? Don’t think so. The message that, as one presenter put it, “Tape Sucks”? Sorry. FAST Cache, VPLEX, performance boosts, cost cuts? Not this time. No, what really caught my attention was a throw-away slide showing that almost a majority of EMC customers have already adopted some form of deduplication technology, and that in the next couple of years, probably a majority of all business storage users will have done so.
Why do I think this is a big deal? Because technology related to deduplication holds the potential of delivering benefits greater than cloud; and user adoption of deduplication indicates that computer markets are ready to implement that technology. Let me explain.
First of all, let’s understand what “dedupe”, as EMC calls it, and a related technology, compression, mean to me. In its initial, technical sense, deduplication means removing duplicates in data. Technically, compression means removing "noise" -- in the information-theory sense of removing bits that aren’t necessary to convey the information. Thus, for example, removing all but one occurrence of the word “the” in storing a document would be deduplication; using the soundex algorithm to represent “the” in two bytes would be compression.
However, today popular “compression” products often use technical-deduplication as well; for example, columnar databases compress the data via such techniques as bit-mapped indexing, and also de-duplicate column values in a table. Likewise, data deduplication products may apply compression techniques to shrink the storage size of data blocks that have already been deduplicated. So when we refer to “dedupe”, it often includes compression, and when we refer to compressed data, it often has been deduplicated as well. To try to avoid confusion, I refer to “dedupe” and “compress” to mean the products, and deduplication and compression to mean the technical terms.
When I state that there is an upcoming “dedupe revolution”, I really mean that deduplication and compression combined can promise a new way to improve not only backup/restore speed, but also transaction processing performance. Because, up to now, “dedupe” tools have been applied across SANs (storage area networks), while “compress” tools are per-database, “dedupe” products simply offer a quicker path than “compress” tools to achieving these benefits globally, across an enterprise.
These upcoming “dedupe” products are perhaps best compared to a sponge compressor that squeezes all excess “water” out of all parts of a data set. That means not only removing duplicate files or data records, but also removing identical elements within the data, such as all frames from a loading-dock video camera that show nothing going on. Moreover, it means compressing the data that remains, such as documents and emails whose verbiage can be encoded in a more compact form. When you consider that semi-structured or unstructured data such as video, audio, graphics, and documents makes up 80-90% of corporate data, and that the most “soggy” data types such as video use up the most space, you can see why some organizations are reporting up to 97% storage-space savings (70-80%, more conservatively) where “dedupe” is applied. And that doesn’t include some of the advances in structured-data storage, such as the columnar databases referred to above that “dedupe” columns within tables.
So, what good is all this space saving? Note the fact that the storage capacity that users demand has been growing by 50-60 % a year, consistently, for at least the last decade. Today’s “dedupe” may not be appropriate for all storage; but where it is, it is equivalent to setting back the clock 4-6 years. Canceling four years of storage acquisition is certainly a cost-saver. Likewise, backing up and restoring “deduped” data involves a lot less to be sent over a network (and the acts of deduplicating and reduplicating during this process add back only a fraction of the time saved), so backup windows and overhead shrink, and recovery is faster. Still, those are not the real reasons that “dedupe” has major long-term potential.
No, the real long-run reason that storage space saving matters is that it speeds retrieval from disk/tape/memory, storage to disk/tape/memory, and even processing of a given piece of data. Here, the recent history of “compress” tools is instructive. Until a few years ago,the effort of compressing and uncompressing tended to mean that compressed data actually took longer to retrieve, process, and re-store; but, as relational and columnar database users have found out, certain types of “compress” tools allow you to improve performance – sometimes by an order of magnitude. For example, recently, vendors such as IBM are reporting that relational databases such as DB2 benefit performance-wise from using “compressed” data. Columnar databases are showing that it is possible to operate on data-warehouse data in “compressed” form, except when it actually must be shown to the user, and thereby get major performance improvements.
So what is my vision of the future of “dedupe”? What sort of architecture are we talking about, 3-5 years from now? One in which the storage tiers below fast disk (and, someday, all the tiers, all the way to main memory) have “dedupe”-type technology added to them. In this context, it was significant that EMC chose at EMC World to trumpet “dedupe” as a replacement for Virtual Tape Libraries (VTL). Remember, VTL typically allows read/query access to older, rarely accessed data within a minute; so, clearly, deduped data on disk can be reduped and accessed at least as fast. Moreover, as databases and applications begin to develop the ability to operate on “deduped” data without the need for “redupe”, the average performance of a “deduped” tier will inevitably catch up with and surpass that of one which has no deduplication or compression technology.
Let’s be clear about the source of this performance speedup. Let us say that all data is deduplicated and compressed, taking up 1/5 as much space, and all operations can be carried out on “deduped” data instead of its “raw” equivalents. Then retrieval from any tier will be 5 times as fast and 5 times as much data can be stored in the next higher tier for even more performance gains. Processing this smaller data will take ½ to 1/5 as much time. Adding all three together and ignoring the costs of “dedupe”/”redupe”, a 50% speedup of an update and an 80% performance speedup of a large query seems conservative. Because the system will only need “dedupe”/”redupe” rarely, “dedupe” when the data is first stored and “redupe” whenever the data is displayed to a user in a report or query response, and because the task could be offloaded to specialty “dedupe”/”redupe” processors, on average “dedupe”/redupe” should add only minimal performance overhead to the system, and should subtract less than 10% from the performance speedup cited above. So, conservatively, I estimate the performance speedup from this future “dedupe” at 40-70%.
What effect is this going to have on IT, assuming that “the dedupe revolution” begins to arrive 1-2 years from now? First, it will mean that, 3-5 years out, the majority of storage, rather than a minority replacing some legacy backup, archive, or active-passive disaster recovery storage, will benefit from turning the clock back 4-6 years via “dedupe.” Practically, performance will improve dramatically and storage space per data item will shrink drastically, even as the amount of information stored continues its rapid climb – and not just in data access, but also in networking. Data handled in deduped form everywhere within a system also has interesting effects on security: the compression within “dedupe” is a sort of quick-and-dirty encryption that can make data pilferage by those who are not expert in “redupe” pretty difficult. Storage costs per bit of information stored will take a sharp drop; storage upgrades can be deferred and server and network upgrades slowed. When you add up all of those benefits, from my point of view, “the dedupe revolution” in many cases does potentially more for IT than the incremental benefits often cited for cloud.
Moreover, implementing “dedupe” is simply a matter of a software upgrade to any tier: memory, SSD, disk, or tape. So, getting to “the dedupe revolution” potentially requires much less IT effort than getting to cloud.
One more startling effect of dedupe: you can throw many of your comparative TCO studies right out the window. If I can use “dedupe” to store the same amount of data on 20% as much disk as my competitor, with 2-3 times the performance, the TCO winner will not be the one with the best processing efficiency or the greatest administrative ease of use, but the one with the best-squeezing “dedupe” technology.
What has been holding us back in the last couple of years from starting on the path to dedupe Nirvana, I believe, is customers’ wariness of a new technology. The EMC World slide definitively establishes that this concern is going away, and that there’s a huge market out there. Now, the ball is in the vendors’ court. This means that all vendors, not just EMC, will be challenged to merge storage “dedupe” and database “compress” technology to improve the average data “dry/wet” ratio, and “dedupify” applications and/or I/O to ensure more processing of data in its “deduped” state. (Whether EMC’s Data Domain acquisition complements its Avamar technology completely or not, the acquisition adds the ability to apply storage-style “dedupe” to a wider range of use cases; so EMC is clearly in the hunt). Likewise, IT will be challenged to identify new tiers/functions for “dedupe,” and implement the new “dedupe” technology as it arrives, as quickly as possible. Gentlemen, start your engines, and may the driest data win!

Memories of The Other Type of Software Failure

A few weeks ago, I met an old friend – Narain Gehani, once at Cornell with me, then at Bell Labs, and now at Stevens Institute of Technology – and he presented me with a book of reminiscences that he had written: “Bell Labs: Life in the Crown Jewel”. I enjoyed reading about Narain’s life in the 35 years since we last met; Narain tells a great story, and is a keen observer. However, one thing struck me about his experiences that he probably didn’t anticipate – the frequency with which he participated in projects aimed at delivering software products or software-infused products, products that ultimately didn’t achieve merited success in one marketplace or other (Surf n Chat and Maps r Us were the ones that eventually surfaced on the Web).

This is one aspect of software development that I rarely see commented on. We have all talked about the frequency of software-development project failure due to poor processes. But how about projects that by any reasonable standard should be a success? They produce high-quality software. The resulting solutions add value for the customer. They are well received by end users (assuming end users see them at all). They are clearly different from everything else in the market. And yet, looking back on my own experience as a software developer, I realize that I had a series of projects that were strikingly similar to Narain’s – they never succeeded in the market as I believe they should have.

My first project was at Hendrix Corporation, and was a word processing machine that had the wonderful idea of presenting a full page of text, plus room on the side for margins (using the simple trick of rotating the screen by 90 degrees to achieve “portrait” mode). For those who have never tried this, it has a magical effect on writing: you are seeing the full page as the reader is seeing it, discouraging long paragraphs and encouraging visual excitement. The word processor was fully functional, but after I left Hendrix it was sold to AB Dick, which attempted to fold it into their own less-functional system, and then it essentially vanished.

Next was a short stint at TMI, developing bank-transfer systems for Bankwire, Fedwire, etc. The product was leading-edge at the time, but the company was set up to compensate by bonuses that shrank as company size increased. As employee dissatisfaction peaked, it sold itself to Logica, key employees left, and the product apparently ceased to evolve.

At Computer Corporation of America, I worked on a product called CSIN. It was an appliance that would do well-before-its-time cross-database querying about chemical substances “in the field”, for pollution cleanup efforts. It turned out that the main market for the product was librarians; and these did not have the money to buy a whole specialized machine. That product sold one copy in its lifetime, iirc.

Next up was COMET/204, one of the first email systems, and the only one of its time to be based on a database – in this case, CCA’s wonderful MODEL 204. One person, using MODEL 204 for productivity, could produce a new version of COMET/204 in half the time it took six programmers, testers, and project managers to create a competing product in C or assembler. COMET/204 had far more functionality than today’s email systems, and solved problems like “how do you discourage people from using Reply All too much?” (answer: you make Reply the default, forcing them to do extra effort to do Reply All spam). While I was working on it, COMET/204 sold about 10 copies. One problem was the price: CCA couldn’t figure out how to price COMET/204 competitively when it contained a $60,000 database. Another was that in those days, many IT departments charged overhead according to usage of system resources – in particular, I/Os (and, of course, using a database meant more I/Os, even if it delivered more scalable performance). One customer kept begging me to hurry up with a new version so he could afford to add 10 new end users to his department’s system.

After that, for a short stint, there was DEVELOPER/204. This had the novel idea that there were three kinds of programming: interface-driven, program-driven, and data-driven. The design phase would allow the programmer to generate the user interface, the program outline, or the data schema; the programmer could then generate an initial set of code (plus database and interface) from the design. In fact, you could generate a fully functional, working software solution from the design. And it was reversible: if you made a change in the code/interface/data schema, it was automatically reflected back into the design. The very straightforward menu system for DEVELOPER/204 simply allowed the developer to do these things. After I left CCA, the company bundled DEVELOPER/204 with MODEL/204, and I was later told that it was well received. However, that was after IBM’s DB2 had croaked MODEL 204’s market; so I don’t suppose that DEVELOPER/204 sold copies to many new customers.

My final stop was Prime, and here, with extraordinary credit to Jim Gish, we designed an email system for the ages. Space is too short to list all of the ideas that went into it; but here are a couple: it not only allowed storage of data statically in user-specified or default folders, but also dynamically, by applying “filters” to incoming messages, filters that could be saved and used as permanent categorizing methods – like Google’s search terms, but with results that could be recalled at will. It scrapped the PC’s crippled “desktop metaphor” and allowed you to store the same message in multiple folders – automatic cross-referencing. As a follow-on, I did an extension of this approach into a full-blown desktop operating system. It was too early for GUIs – but I did add one concept that I still love: the “Do the Right Thing” key, which figured out what sequence of steps you wanted to take 90% of the time, and would do it for you automatically. Because Prime was shocked by our estimate that the email system would take 6 programmers 9 months to build, they farmed it to a couple of programmers in England, where eventually a group of 6 programmers took 9 months to build the first version of the product – releasing the product just when Prime imploded.

Looking back on my programming experiences, I realized that the software I had worked on had sold a grand total (while I was there) of about 14 copies. Thinking today, I realize that my experiences were pretty similar to Narain’s: as he notes, he went through “several projects” that were successful development processes (time, budget, functionality) but for one reason another never took off as products. It leads me to suspect that this type of thing – project success, product lack of success happens just as much as software development failure, and very possibly more often.

So why does it happen? It’s too easy to blame the marketers, who often in small companies have the least ability to match a product to its (very real) market. Likewise, the idea that the product is somehow “before its time” and not ready to be accepted by the market may be true – but in both Narain’s and my experience, many of these ideas never got to that point, never got tested in any market at all.

If I had to pick two factors that most of these experiences had in common, they were, first, barriers within the company between the product and its ultimate customers, and second, markets that operate on the basis of “good enough”. The barriers in the company might be friction with another product (how do you low-ball a high-priced database in email system pricing?) or suspicion of the source (is Bell Labs competing with my development teams?). The market’s idea of “good enough” may be “hey, gmail lets me send and receive messages; that’s good enough, I don’t need to look further”.

But the key point of this type of software failure is that, ultimately, it may have more profound negative effects on top and bottom lines, worker productivity, and consumer benefits in today’s software-infused products than development failure. This is because, as I have seen, in many cases the ideas are never adopted. I have seen some of the ideas cited above adopted eventually, with clear benefits for all; but some, like cross-referencing, have never been fully implemented or widely used, 25 or more years after they first surfaced in failed software. Maybe, instead of focusing on software development failure, we should be focusing on recovering our memory of vanished ideas, and reducing the rate of quality-product failure.

When I was at Sloan School of Management at MIT in 1980, John Rockart presented a case study of a development failure that was notable for one outcome: instead of firing the manager of the project, the company rewarded him with responsibility for more projects. They recognized, in the software development arena, the value of his knowledge of how the project failed. In the same way, perhaps companies with software-infused products should begin to collect ideas, not present competitive ideas, but past ideas that are still applicable but have been overlooked. If not, we are doomed not to repeat history, but rather, and even worse, to miss the chance to make it come out better this time.

Whither Hybrid Columnar?

Oracle’s announcement of its “hybrid columnar compression” option for its Exadata product last summer clearly relates to the renewed attention paid to columnar databases over the last year by columnists such as Merv Adrian and Curt Monash. This example of a “hybrid” between columnar and row-oriented technology makes life yet more complicated for the IT buyer of data warehousing and business intelligence (BI) solutions. However, there does seem to be some agreement between the advocates of columnar and row-oriented that sheds some light on the appropriate places for each – and for hybrid.

Daniel Abadi in an excellent post summarizes the spectrum. If I understand his taxonomy correctly, row-oriented excels in “random reads” where a single record with more than 2-3 fields is accessed (or for single inserts and updates); columnar excels for queries across a large number of records whose columns (or “fields”) contain some commonality that makes for effective compression. The hybrids attempt to achieve 80% (my figure, plucked from the air) of the performance advantages of both.

To follow this thought a bit further, Daniel divides the hybrid technologies into block-oriented or “PAX”, fractured mirror, and fine-grained. The PAX approach stores in column format for particular disk blocks; the fractured mirror operates in a real-world disaster recovery environment and treats one copy of the data store as row-oriented, the other as column-oriented, sending transactions to either as appropriate; the fine-grained hybrid is similar to PAX, but compresses the columns of particular fields of particular tables, not a whole disk block. Oracle appears to be an example of the PAX approach, while Vertica has some features that appear to implement the fine-grained approach.

I would argue that the future belongs more to pure columnar or fractured mirror than to row-oriented or the other two flavors of hybrid. Here is my reasoning: data stores continue to scale in size by 50% a year; the proportion of storage devoted to main memory and solid-state devices (SSDs) is likewise going up. The disadvantage of columnar in “random reads” is therefore decreasing, as databases are increasingly effective at ensuring that the data accessed is already in fast-response storage. In other words, it’s not just the number of I/Os, it’s what you are inputting from.

There is another factor: as database size goes up, the disadvantages of row-oriented queries increase. As database size increases, the number of “random reads” does not necessarily increase, but the amount of data that must be accessed in the average query does necessarily increase. Compression applied across all columns and indexes increases its advantage over selective compression and no compression at all in this case, because there is less data to upload. And the “query from hell” that scans all of a petabyte data warehouse is not only the extreme case of this increasing advantage; scaling the system’s ability to handle such queries is often a prime concern of IT.

I would also argue that the same trends make the pure or fractured-mirror columnar database of increasing interest for operational data stores that combine online transaction processing with lots of updates and continuous querying and reporting. For updates, the competition between columnar and row-oriented is getting closer, as many of these updates involve only 1-3 fields/columns of a row, while updates are most likely to affect the most recent data and therefore the data increasingly likely to be in main-memory cache. For inserts/deletes, updating in-memory indexes immediately along with “write to disk later” means that the need of columnar for multiple I/Os need not be a major disadvantage in many cases. And for online backup, the compressed data of the columnar approach wins hands down.

My takeaway with regard to Oracle Hybrid Columnar Compression, therefore, is that over time your mileage may vary. I would not be surprised if, someday, Oracle moved beyond “disk-block hybrid” to a fractured mirror approach, and that such an approach took over many of the high-end tasks for which vanilla row-oriented Oracle Database 11g r2 is now the database of choice.

Lean and Agile: Partners or Enemies?

An increasing number of commentators on both lean New Product Development (NPD) and agile software development have noted efforts to combine the two. A key reason for the interest in the combination is that lean has long been entrenched in the manufacturing process (e.g., via Kanban and ERP), while agile development has recently established itself as an effective software development strategy, and a major segment of NPD now involves software-only products, software-infused products, or software-plus-hardware solutions. A recent Aberdeen Group study notes that 66% of today’s NPD uses software to drive innovation, and the number is rising rapidly. Meanwhile, several efforts are being made to apply lean-only to software development (cf. Hibbs, Jewett, Sullivan, “The Art of Lean Software Development”).

At first glance, lean is a very good complement for agile. Users have concerns about agile’s ability to scale; lean makes sure that developer “resources” are available for each iterative step in a flexible and cost-effective way. Lean allows a different process time-line each time depending on what resources are available when; agile “spirals” in towards a solution, ensuring a different process timeline for each iteration of a solution “guesstimate”. Both aim to increase value to the customer; both emphasize incremental improvements.

However, a reading of the blogs on lean and agile betrays a fundamental misunderstanding of the agile mindset. Here’s a quote from Martin Heller (http://www.infoworld.com/d/developer-world/should-software-development-be-lean-or-agile-284): “They both strive to improve software quality, reduce waste, increase developer productivity, accept changes to requirements, and prize meeting the customer's real needs.” Agile doesn’t strive to improve software quality; it strives to improve software usefulness. It doesn’t aim to reduce waste; it aims to increase “wasteful” experimentation, which turns out actually to reduce the time-to-value, as a side-effect. Its way of increasing developer productivity – ensuring that development is more in sync with customer needs – is the opposite of lean’s attempt to improve developer productivity by eliminating bottlenecks (and customer interference is potentially a bottleneck). Lean accepts changes to requirements as a necessary part of incremental manufacturing process improvement; agile welcomes and emphasizes changes to requirements as part of improving product design. Lean views customer needs as consisting primarily of quality; agile views customer needs as consisting primarily of rapidly-changing functionality.

Moreover, we have already seen how an emphasis on Deming-like quality can retard product development to the detriment of a firm, as when Motorola lost its race to dominate processors to Intel by being late and customer-insensitive with the design of a “quality”-manufactured product. I can’t quote figures from my recent study under Aberdeen auspices of agile software development vs. other processes; but I can certify that a focus on software quality improvement was far less effective in improving product TCO, ROI, customer value, and ongoing agility than an agile process; in fact, there was no clear long-term benefit to ROI of quality-improving processes at all.

I wish I could say that this is a side-issue, and that once agile is properly understood, lean and agile are indeed complements. After all, some of the confusion about agile stems from a lean mindset that views everything from the lens of the manufacturing process. In many companies, integrating NPD and the manufacturing process has indeed reaped rewards, and lean has been a part of that improvement. Agile, by contrast, is effective at NPD and R&D/innovation, and will probably never be applicable to manufacturing – imagine asking the NC machine to “spiral in on” the correct product each time! Surely, a compromise can be worked out in which lean resource allocation is applied to agile software development and lean manufacturing becomes much more nimble at producing prototypes. However, I regretfully conclude that in the real world, mixing lean and agile is likely to produce worse results than applying agile alone.

Consider the idea of applying just-in-time resource allocation to an agile process. In each iteration, the agile process is going to change the specs during the “sprint.” Thus, half the time, allocation of resources will be inadequate, causing more bottlenecks than a “wasteful” strategy.

More fundamentally, lean increases the need for tight control over and visibility into costs and resources. This kind of oversight, as developers well know, slows things down. As per my study, it also yields nothing in improved productivity, TCO, ROI, or customer satisfaction.

Above all, the mindset of lean and agile are antithetical. Lean is reactive, prescriptive, and cost-oriented, despite its attempts to connect to the customer. It works where requirements change little from iteration to iteration, as in a manufacturing process. Agile is proactive, creative, and customer-value-oriented. It works where customer needs are constantly changing – which is a much better description of the real world of customers, as opposed to the artificial “turn off the spigot until we’ve finished” world of much of today’s NPD.

In the end, however, the conflict of lean and agile is only one instance of the difficulties companies have with processes like agile when the command-and-control approach has become entrenched. As I noted in a recent post, it may well be caused by budgeting processes that constrain business flexibility. But whatever the reason, there is strong cultural resistance to increasing corporate results long-term by adopting processes that increase business agility, such as agile development.

In an old story whose author I cannot unfortunately remember, a manager is called in to turn the efforts of R&D types into profit. The first thing he notices was that they seem to spend a lot of time with their feet on their desks, thinking – a highly wasteful practice. How can he change that? He puts his legs up on his desk and leans back, thinking … In this story, as in the real world, lean is more the enemy than the partner of agile; and trying to constrain agile via lean will cost the company much more in the long run than it seems to gain according to short-term cost measures.

About The Genius of Tolkien

A great deal of ink has been spilled (and storage space wasted) in analyzing the writing of Tolkien as comparable to Great Authors, as strictly a writer in the fantasy genre, or as an odd specimen with no relation to other literary streams. Commentators have focused on the sources of his work in myth, on his role as the precursor of the quest type of fantasy, and on the relation between his life experiences and the Lord of the Rings, as well as the enormous “fantasy world” of which it is presented as a part.

I find that I came to Tolkien originally from a somewhat different background. My reading in childhood was as much English “adventure as escape” – John Buchan, H. Rider Haggard, Sanders of the River, Swallows and Amazons, G.A. Henty – and English novel/play – Shakespeare, Hardy – as standard American kid’s stuff – all the Oz books, John Tunis’ sports books. The folk songs I and my Dad sang were as much English – There Was a Tailor Had a Mouse – as Appalachian – John Langstaff’s version of Hi Falero. I am not sure why English commentators have not brought up some of the things that strike me about Tolkien’s writing; but I am pretty sure why American ones without the same background tend not to see them.

So, off the top of my head, here is a brief discussion of points I have not seen covered in the most visible analyses of Tolkien.

1. Poetry as folk song

It is quite striking how certain sets of poetry match up to the rhythm of English folk songs. Take that short one from Bilbo arriving at Rivendell in The Hobbit: “tra la la la lally, here down in the valley, with Bilbo and …” Now try singing it to the tune of “The Flowers that Bloom in the Spring”, itself a folk song imitation, in Gilbert and Sullivan’s Mikado. In both cases, the music itself suddenly gives a more sober, more realistic “tone” to what on the printed page seems silly nonsense. Singing the poetry this way gives an insight into the Elves as not the mischievous sprites of Shakespeare but rather superficially vacuously cheerful beings with hidden depths and age.

The part of Tolkien that most find hard to swallow is Tom Bombadil. I feel that the commentators find him mannered, irrelevant, and wasting page on page when the action should be advancing. However, if one more or less matches the words to the tune of, say, Among the Leaves So Green Oh, then it becomes a “proto-folk-song”. Tom becomes the underlying source of English folk songs and their spillover into the Shakespearean plays – “with a hey ho, the wind and the rain”. Tom’s every phrase is a blank verse with a folk-song-like meter, and his appearance transitions the hobbits from “modern rustic” speech and colloquial writing to “heroic” speech and “pre-modern” writing.

That, in turn, leads to:

2. Tolkien’s “pre-modern” English

One commentator has neatly noted Tolkien’s deliberate contrast between “modern” and “ancient” speech, pointing out the advantages of the latter: “What a lot of things you use good day for, said Gandalf”. No one seems to have noted the large expansion of the English language to accommodate older word meanings with wonderful resonance. “The words were not the doom of Minas Tirith. But doom and great deeds are indeed at hand.” “Into most places some haggard of spring would creep.” “Ithilien in spring still kept its disheveled dryad loveliness.” “Ent moot”.

At the same time, Tolkien deliberately strips his prose of both fancy modern-poetry-like flourishes and “just the facts” Hemingway style. Rather, the style is of simple metaphors “with a different eye”. “High above, an eagle might have seen the hobbits, small figures …” “Tom stood on the threshold, and a golden glow was behind him”

There is a transition from “modern” to “heroic”, and it happens so gradually and carefully that commentators seem to have missed it. The introduction is almost comically academic; the first few chapters, in the Shire, are almost entirely “modern”, except for Gandalf, who presents information to Frodo in “adult modern” terms with subtle shadings of “pre-modern”. “Who is he? Frodo said. I don’t think I’ve heard of him. He is high among our order [Gandalf], … subtle …”. Then a brief meeting with the Elves, with “pre-modern” presented as a foreign language that Frodo speaks with difficulty. Then Tom Bombadil as a kind of half-way language that seems rustic but is stripped of colloquialisms: “Tom he is the Master; his songs are stronger songs, and his boots are faster”. In Bree, Aragorn gives the first touch of extended heroic speech, beginning by presenting himself as almost hobbit-like: “Though I say it as shouldn’t, you might say …” but ending in “The day is coming when it shall be forged anew”. From there until the end of Book I, hobbit speech is balanced by Aragorn’s “pre-modern speech”. Then in the beginning of Book 2, even Bilbo gives a “pre-modern” poem, and “modern” – represented by the hobbits – is always the exception rather than the rule, all the way to the very end, when Frodo (and we) find ourselves viewing hobbit speech as more strange than the companions’ new heroic speech.

3. Viewpoint

Tolkien’s “reader’s viewpoint” is actually quite subtle, and much more varied than one realizes. However, there are overall “main viewers”, and their viewpoint carefully balances moral issues and a necessary distance that will allow the reader to think for him/herself.

Book I is, for the most part, told from Frodo’s viewpoint. There are interesting exceptions: “hmm, the fox thought;” “Sam ran back along the path, shouting help!” The dreams in Tom’s house are from people who will all have their own “main viewpoint” later. In Book II, again, Frodo is the “main viewpoint”, with Pippin and Sam occasionally allowed a quick “snapshot”. Then in Book III, we start with Aragorn, but Gimli is occasionally given the viewpoint; and Pippin and Merry are effectively the main viewpoint in their scenes.

Book IV begins a transition of great importance. At the start, the viewpoint is ambiguous between Frodo and Sam, but with Frodo very much more frequent. By the end of Book IV, Sam predominates. In effect, the transition reflects the transition from Frodo as the prime actor and decider, to Sam as the prime actor when the burden becomes too great for Frodo.

Book V, again, starts with Pippin the main viewpoint. Interestingly, when the action shifts to Gimli, Legolas, and Aragorn, Gimli is the prime viewpoint, keeping “heroic thinking” from getting too far out of hand. Only at the end does some of the viewpoint return to Aragorn, as he directs the attack on Mordor.

Then there’s Book VI. Now, the viewpoint is almost entirely Sam. The main exception is Faramir and Eowyn; almost a “heroic” counterpoint to the relentlessly “hobbit/half-modern” viewpoint. And by the end, the viewpoint is deliberately, consciously, entirely Sam.

The point is that the viewpoint is almost never that of the author, nor solely that of one main character driving the action forward. Look where you will, to Hemingway, Hardy, Bellow, Updike, or Flaubert, and the viewpoint is almost always that of the main character or of the author; not so with Tolkien. And the result is to gain perspective on the main character and to draw the reader more deeply into the world.

4. Weather and fauna

Commentators from fantasy writers to critics have confessed their feeling that they would like to live in or experience Tolkien’s locations. But few have noted the way in which the weather – and the places the characters go – is strongly related to the characters’ (and our) moods. Tolkien himself noted that, unusual among authors, he took great care to reflect an actual sequence of weather, just about day by day, from September to March. Moreover, the action takes place just about day by day, with only breaks for Rivendell, for part of the march south, for a short space after the knife in the dark, and for the time within the Mines of Moria. In just about every other day until the Ring is cast into the fire, there is a description of the weather for that day – and the character with the main viewpoint spends a lot of that day outdoors. The weather is always varied, always in some way notable, always in some ways memorable. And it both reflects and causes the characters’ moods; the tenseness of the descent of the Emyn Muil reflected by the thunderstorm, which itself almost causes a disastrous fall; the “heavier air” as the Riders go to war, that both reflects and increases the characters’ tension. In Tolkien’s world, unlike our own, characters feel the weather – “the air seemed to throb”.

As far as I can tell, there is no comparable achievement in all of literature. I simply cannot think of another novel in which there is such a clear and integral description of the weather.

Likewise, there is a clear link between the vegetation and the character of the people inhabiting it. I don’t mean just the fact that pastoral Riders need rolling hills and grasslands; I mean that the Shire is very regulated, with gardens and forest boundaries, while Lothlorien is regulated in a more subtle way, with long-term planning, and Ithilien was once a kind of hothouse of a wide variety of plants, shrubs, and trees, but has clearly been unregulated for some time.

Again, Tolkien is, as far as I can see, unprecedented in his knowledge of botany, and his ability to use it to set a scene and indicate a kind of people. The initial description of Ithilien is the outstanding example; the description of the desolation before the Gates of Mordor is a miniature gem of the opposite kind. But it is interlaced all the way through the novel: from the start, with the description of the Party Tree and the path to Crickhollow, to the swaying leaves above Sam’s head as he awakes in Ithilien, every tree and fern is a part of the main viewpoint.

5. The Ending

I have seen some commentators touch on this, but it should be emphasized: Tolkien crafts neither a tragic (in the classic sense) nor a happy (for the main character, Frodo) ending. This appears to spring out of a constant wrestling on Tolkien’s part with how God justifies death. Through twists and turns in his background notes published by Christopher, it appears that he conceives of Elves as the first answer to Death; they accept it because they are effectively “burned out”, passive shells of their passionate former selves. But Man is the second and better answer to Death, because he is forced into Death prematurely, with no assurance of what happens after, but by conquering his fears he achieves preparation for the best “next” existence, an existence in which may he become, as it were, a “child” of God to be taught by God.

Frodo’s realization that he cannot escape the effects of the Ring and that he has outgrown the Shire for which he thought he was killing himself becomes a tragedy that can’t be solved in Middle Earth; but it allows him to conquer his fears after he leaves for the Blessed Lands, by seeing there is hope ahead. Sam’s happy ending is tainted by the realization that the person for whom he sacrificed himself did not benefit by the sacrifice; but it allows him to live a life that will enable him also to conquer his fear of death. In either case, we are far from “happy ever after”, and just as far from “bad things happen to bad/flawed people, good things happen to good people” tragedy or even “bad things happen for no reason” tragedy.

6. Names

While commentators have noted Tolkien’s creation of a “new language”, his background as a philologist, and the origins in European history of the names of people, they do not seem to have noted the importance of his ability, to some extent, to avoid having the names seem ridiculous. Before Tolkien, fantasists tried to make up alien names as a baroque variation of English ones (e.g., Eddison), or as pagan-sounding (Dejah Thoris in Edgar Rice Burroughs’ Mars series). In either case, the names had no “depth” of source in a complete language. One of the reasons many readers recoiled from fantasy as serious adult literature was that, well, the people’s names sounded silly.

Again, the notes published by Christopher Tolkien show that in his original conception, J.R.R.’s names would have struck us as pretty silly, too – Kor, Gnomes for Elves. The process of developing a full-fledged language with clear associations to “normal” names, to our ear, apparently involved a fair bit of subtle changes to make the names less likely to cause laughter – Strider for Trotter, for example. The image of Aragorn as a pig would have been hard to expunge.

Not that, to some ears, the effort was successful. Reading between the lines, one of the main reasons that Edmund Wilson wrote “Ooh, Those Awful Orcs” was that to his ears, names like Shagrat sounded just as silly and juvenile as Burroughs’. But I think it’s fair to say that there’s a dividing line between fantasy before Tolkien, in which Jhirel of Joiry joined Fritz Leibner’s Fafnr, and after Tolkien, with the far more carefully crafted Welsh names of Morgon and Astrin in McKillip’s works, or Pyanfar and Arafel in Cherryh. Fantasy and science fiction not only learned world-building and history-building from Tolkien; to some extent, they learned language-building.

7. The Sad Lack of Successors

I can’t say that I find authors who have some sense of most of the preceding Tolkien traits, much less the ability to use them in other contexts, in most non-fantasy/sf writing. Fantasy and science fiction authors, in particular, have been quite successful at building much more detailed worlds that give a much greater sense of “depth” to the story, from Weber to Modesitt. Likewise, authors pay much more attention to the “tone” of the story, and do much better (by and large) at avoiding silliness and making heroic speech seem less alien. But I can find none, in fantasy or out, that has a comparable use of viewpoint, of weather and fauna, of simple metaphor for description, of endings that are neither comedy nor tragedy, or even (in most cases) of poetry as folk song.

Consider three who seem to have made a conscious effort to think deeply about Tolkien: Donaldson, Jordan, and Le Guin. Donaldson is of interest because he took many of the half-sketched ideas of Tolkien – land with a physical sense of health or disease, telepathic horses as equals to humans, immortal evil, and inevitable failure that can only be redeemed by God/chance – and fleshed them out in his own way. But the sense of language and viewpoint is sometimes really off – it is hard to imagine a juxtaposition of three more disparate eras of the English language than describing a character as “fey, anile, and gutless”, or to swallow a sentence that states “her breasts traced opaque circles of invitation beneath her shirt”. I know, these are early efforts; but the names established there (Lord Foul) still recur in his work thirty years later. Some of his “modern” writing (a graduation speech) is far more powerful. Viewpoint is far less subtle, poetry is less informed by folk song or history, and the weather appears to exist often so as to put as much physical stress as possible on the main character.

Jordan, by contrast, seems to have focused on the “quest” and world-building aspects of Tolkien. His gift for detail, however, is more about character and less about weather and fauna. Unfortunately, in his later books, this causes an over-proliferation of characters that make the story harder to follow. His poetry as folk song is actually quite good, but he does not really give the kind of day-to-day sense of weather and fauna as characters in the story, or at least as indicators of mood, that Tolkien can achieve. His approach to viewpoint is less subtle, and his endings are a bit more traditional.

Of these three, le Guin is perhaps the most successful at building on Tolkien’s additions to the writing craft. Her Earthsea books, apparently beginning with a light attempt to imagine a hobbit as a proto-dragon, have a strong sense of the weather, if not of fauna, and a simple style that, while light on metaphor, nevertheless fully understands the power of pre-modern word meanings. Her viewpoints are mostly traditional but show interesting variation, especially inserting women’s viewpoints into what is traditionally a man’s narrative. Names show great “depth”, even if the world-building is a bit “retrofitted” as her concepts expand from story to story. Ultimately, however, more than Jordan or Donaldson, her stories are less day-by-day and more episode-by-episode, which makes the feel of her stories more “abstract” overall.

A French critic is reported to have said “Victor Hugo is our greatest author – alas”, meaning that even Hugo did not compare to the English or American great writers. In the same vein, I feel that authors of whatever stripe have yet to fully appreciate or build on the aspects of Tolkien’s stylistic genius that I have described above; and it’s too bad.