Thoughts From a Software IT Analyst

The Other BI: Orange EDA and Statistical Analytics

2012-01-30T11:50:00.000-08:00

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about BI today.

The Importance of Orange-Type Statistical Analysis to Analytics

BI has taken a major step forward in maturity over the last few years, as statistical packages have become more associated with analytics. Granted, SAS has for years distinguished itself by its statistics-focused BI solution; but when IBM recently acquired SPSS, the grand-daddy of statistical packages, the importance of more rigorous analysis of company and customer data seemed both confirmed and more obvious. Moreover, over the years, data miners have begun to draw on the insights of university researchers about things like “data mining bias” and Bayesian statistics – and the most in-depth, competitive-advantage-determining analyses have benefited as a result. And so, it would seem that we are on a nice technology glide path, as statistics completes the flexibility of analytics by covering one extreme of certainty and analytical complexity, while traditional analytics tools cover the rest of the spectrum up from situations where shallow and imprecise analysis is appropriate, and as statistical techniques filter down by technology evolution to the “unwashed masses” of end users. Or are we?

You see, there is a glaring gap in this picture of increasing knowledge of what’s going on – or at least a gap that should be glaring. This gap might be summed up as Alice in Wonderland’s “verdict first, then the trial”, or business’ “when you have a hammer, everything looks like a nail.” Both the business and the researcher start with their own narrow picture of what the customer or research subject should look like, and the analytics and statistics that start with such hypotheses are designed to narrow in on a solution rather than expand due to unexpected data, and so the business/researcher is very likely to miss key customer insights, psychological and otherwise. Pile on top of this the “not invented here” syndrome characteristic of most enterprises, and the “confirmation bias” that recent research has shown to be prevalent among individuals and organizations, and you have a real analytical problem on your hands.

This is not a purely theoretical problem, if you will excuse the bad joke. In the psychological statistics area, the recent popularity of “qualitative methods” has exposed, to those who are willing to see, the enormous amount of insights that traditional statistics fails to capture about customer psychology, sociology, and behavior. Both approaches, of course, would seem to suffer from the deficit that Richard Feynman pointed out – the lack of control groups that renders any conclusion suspect because a “placebo” or “Hawthorne” effect may be involved – but it should be noted that even when (as seems to be happening) this problem is compensated for, the “verdict first” problem remains, because the world of people is far less easy to pre-define than that of nuclear physics.

In the world of business, as I can personally attest, the same type of problem exists in data-gathering. For more than a decade, I have run TCO studies, particularly on SMB use of databases. I discovered early on that open-ended interviews of relatively few sysadmins were far more effective in capturing the real costs of databases than far wider-spread on-a-scale-from-1-to-5 inflexible surveys of CIOs. Moreover, if I just included the ability of the interviewee to tell a story from his or her point of view, the respondent would consistently come up with an insight of extraordinary value, such as the idea that SMBs didn’t care so much about technology that saved operational costs as much as technology that saved a local-office head time by requiring him or her to just press a button as he or she shut off the lights on Saturday night. The key to success for my “surveys” was that they were designed to be open-ended (able to go in a new direction during the interview, and leaving space for whatever the interviewer might have left out), interviewee-driven (they started by letting the interviewee tell a story as he or she saw it), and flexible in the kind of data collected (typically, an IT organization did not know the overall costs of database administration for their organization [and in a survey, they would have guessed – badly], but they almost invariably knew how many database instances per administrator).

As it turns out, there is a comparable statistical approach for the data-analysis side of things. It’s called Exploratory Data Analysis, or EDA.

As it has evolved in the decades since John Tukey first popularized it, EDA is about analyzing smaller amounts of data to generate as many plausible hypotheses (or “patterns in the data”) as possible, before winnowing them down with further data. To further clear the statistical researcher’s mind of bias, the technique creates abstract unlabeled visualizations (“data visualization”) of the patterns, such as the strangely-named box-and-whisker plot. The analysis is not deep – but it identifies far more hypotheses, and therefore quite a few more areas where in-depth analysis may reveal key insights. The automation of these techniques has made the application of EDA a minor blip in the average analyst’s process, and so effective use of EDA should yield a major improvement in analytics effectiveness “at the margin” (in the resulting in-depth analyses) for a very small time “overhead cost.” In fact, EDA has reached the point, as in the Orange open-source solution, where it is merged with a full-fledged data-mining tool.

And yet, I find that most in university research and in industry are barely aware that EDA exists, much less that it might have some significant use. For a while, SAS’ JMP product stood bravely and alone as a tool that could at least potentially be used by businesses – but I note that according to Wikipedia they have recently discontinued support for its use on Linux.

So let’s summarize: EDA is out there. It’s easy to use. Now that statistical analysis in general is creeping into greater use in analytics, users are ready for it. I fully anticipate that it would have major positive effects on in-depth analytics for enterprises from the very largest down at least to the larger medium-sized ones. IT shops will have to do some customization and integration themselves, because most if not all vendors have not yet fully integrated it as part of the analytics process in their BI suites; but with open-source and other “standard” EDA tools, that’s not inordinately difficult. The only thing lacking is for somebody, anybody, to wake up and pay attention.

The Relevance of Orange EDA to Statistical-Analysis-Type BI

Orange’s relevance may already be apparent from the above, but I’ll say it again anyway. Orange’s EDA solution includes integration with enterprise-type data-mining analytics, and supports a wide range of data visualization techniques, making it a leadership supplier in “fit to your enterprise’s analytics.” Orange is open source, which means it’s as cheap as you can get for quick-and-dirty, and also means it’s not going to go away. Most importantly, Orange lays down a solid, relatively standardized foundation that should be easy to incorporate or upgrade from, when someday the major vendors finally move into the area and provide fancier techniques and better integration with a full-fledged BI suite. That’s all; and that’s plenty.

Potential Uses of Orange-Type EDA in Analytics for IT

Since IT will need to do some of the initial legwork here, without the usual help from one’s BI supplier, the most effective initial use of Orange-type EDA is in support of the longer-term efforts of today’s business analysts, and not in IT-driven agile BI. However, IT should find these business analysts to be surprisingly receptive – or, at the least, as recent surveys suggest, amazed that IT isn’t being a “boat anchor” yet again. You see, EDA has a sheen of “innovation” about it, and so folks who are in some way associated with the business’ “innovation” efforts should like it a lot. The rest is simply a matter of its becoming part of these business analysts' steadily accumulating toolkit of rapid-query-generation and statistical-in-depth-insight-at-the-margin tools. EDA may not in the normal course of usage get the glory of notice as the source of a new competition-killer; but with a little assiduous use-case monitoring by IT, the business case can be made.

It is equally important for IT to note that EDA is twice as effective if it is joined at the front end by a data-gathering process that is to a much greater extent (to recap) open-ended, customer-driven, and flexible (in fact, agile) in the type of data gathered. Remember, there are ways of doing this – such as parallel in-depth customer interviews or Internet surveys that don’t just parrot SurveyMonkey – that add very little “overhead” to data-gathering. IT should seriously consider doing this as well, and preferably design the data-gathering process so as to feed the gathered data to Orange-type EDA tools where in-depth statistical analysis of that data will probably be appropriate as the next step. The overall effect will be like replacing a steadily narrowing view of the data with one that expands the potential analyses until the right balance between “data blindness” and “paralysis by analysis” risks is reached.

The Bottom Line for IT Buyers

To view Orange-type EDA as comparable to the other BI technologies/solutions I have discussed so far is to miss the point. EDA is much more like agile development – its main value lies in changing our analytics methodology, not in improving analytics itself. It helps the organization itself to think not “outside the box”, but “outside the organization” – to be able to combine the viewpoint of the vendor with the viewpoint and reality of the customer, rather than trying to force customer interactions into corporate fantasies of the way customers should think and act for maximum vendor profit. We have all seen the major public-relations disaster of Bank of America charges for debit cards – one that, if we were honest, we would admit most other enterprises find it all too easy to stumble into. If EDA (or, better still, EDA plus open-ended, customer-driven, flexible data-gathering) prevents only one such misstep, it will have paid for itself ten times over, no matter what the numbers say. In a nutshell: EDA seems like it’s about competitive advantage; that’s true as far as it goes, but EDA is actually much more about business risk.

The Orange value proposition for such uses of EDA has been noted twice already; no need to repeat it a third time. For IT buyers, it simply means that any time you decide to do EDA, Orange is there as part of a rather short short list. So that leaves the IT buyer’s final question: what’s the hurry?

And, of course, since EDA is about competitive advantage (sarcasm), there is no hurry. Unless you consider the possibility that each non-EDA enterprise is a bit like a drunk staggering along a sidewalk who has just knocked over the fence bordering an abyss, and who if he then happens to stagger over the edge is busy blaming the owner of the fence (the CEO?) all the way to the bottom. That abyss is the risk of offending the customer. That inebriation is business as usual. EDA helps you sober up, fast.

I can’t say that you have to implement EDA now or you’ll fall. But do you really want to risk doing nothing?

The Other Agile Development: Thoughtworks and Continuous Delivery

2012-01-26T13:27:00.000-08:00

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in agile development, agile new product development, and business agility over the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about agile development today.

The Importance of Continuous Delivery to Agile Development

One of the most enjoyable parts of writing about “the other” in general and “the other agile development” in particular is that it allows me to revisit and go more in-depth on cool new technologies. And this is a cool new technology if ever there was one.

Continuous Delivery, as Thoughtworks presents it, aims to develop, upgrade, and evolve software by constant, incremental bug fixes, changes, and addition of features (note: CD is not to be confused with Continuous Integration, which I hope to cover in a future blog post). The example cited is that of Flickr, the photo sharing site, which is using Continuous Delivery to change its production web site at the rate of ten or more changes per day. Continuous Delivery achieves this rate not only by overlapping development of these changes, but also by modularizing them in small chunks that still “add value” to the end user, as well as by shortening the process from idea to deployment to less than a day in many cases.

Continuous Delivery, therefore, is a logical end point of the whole idea of agile development – and, indeed, agile development processes are the way that Thoughtworks and Flickr choose to achieve this end point. Close, constant interaction with customers/end users is in there; so is the idea of changing directions rapidly, either within each feature’s development process or by a follow-on short process that modifies the original. Operations and development, as well as testing and development, are far more intertwined. The shortness of the process allows such efficiencies as “trunk-based development”, in which the process specifically forbids multi-person parallel development “branches” and thus avoids their inevitable communication and collaboration time, which in a short process turns out to be greater than the time saved by parallelization.

Now, let’s take a really broad view of Continuous Delivery. Unfortunately, blog posts are not great at handling graphs, so I’d like you to visualize in your head a graph with two axes, Features and Time. Over time, in each graph, the user’s need for features in a product and solution tends to go up at a fairly steady rate over time, as do consumer needs in general. What varies is how well the vendor(s) supply those needs.

The old model – as old as markets and technology – of what happened was this: somewhere between each version and the next, the disconnect between what the consumer wants and what the product delivers becomes too great, and at that point the vendor starts developing a new version, based on where the consumer is right now (plus a very minor projection into the future which we’ll ignore). For the most part, during this 6 month-2 year development process, the original spec does not change; so for 6 months to 2 years before another version comes out, no or few new features are added – but meanwhile, consumers start looking for new features on top of what they already wanted. The result is a stair-step progression, in which each new version takes the product only partway to meeting the consumer’s needs at that time, and the space between the user line and the vendor line represents lost sales and consumer frustration. However, since every other vendor is doing the same thing, no harm, no foul.

Now consider agile development. Agile development, remember, is about rapid delivery of incremental time-to-value, plus frequent changes to the spec based on end-user input. What that looks like in our graph, more or less, is a stairstep in which each step takes a shorter amount of time, and each “rise” is much closer to the level of user need at that time – but we’re still a little bit behind.

Conceptually, Continuous Delivery takes that idea almost as far as it can be taken. Now, our graph looks more like a squiggly product line overlapping the user need line. And here’s the key point: it actually goes above the user need line, just by a little, frequently.

How can that be, you say? Well, despite the way we disparage technology-driven products compared to need-driven ones, the fact remains that sometimes techies anticipate consumer needs. The typical way is that implicit in the design of the product are future features that the end user, with his or her tunnel vision on immediate frustrations, will never think of. It is the developer who suggests these to the user, not the other way around, or the developer who puts these in the product “for free”, understanding that since they are a logical technical evolution of the design, the user will see them as less strange and simpler to use. This may sound risky to the development manager, but in point of fact this is a minimal-risk kind of customer anticipation, with minimal impact on customer frustration even if it doesn’t pan out, and maximal impact on the consumer’s image of “the brand that anticipates my needs”.

One more variant on the graph: suppose we are talking about New Product Development (NPD) in general. Well, one thing about agile software development is that software is becoming an increasing part of competitive advantage in “hardware” and “services” across most industries. In other words, the development of a new “hardware” or “services” product now typically includes a fair amount of software, whose development is in-house, outsourced, or assigned to packaged-software vendors. In each of these cases, application of agile development processes produces a “mix” between the traditional-graph vendor line and the agile-development one. Visually, the “steps” between “rises” are no longer flat, but broken into little “mini-rises” and “ministeps” that take you a little closer to the user needs line. Continuous Delivery on software that is half of NPD effectively eliminates about half of the lost sales and customer frustration from the traditional approach.

Do you remember that I said: “CD takes the idea almost as far as it can be taken?” Well, the one thing agile development via CD doesn’t handle is a big jump in consumer needs because the consumer wants one of the features that is being developed over in the next county – “disruptive” technology. For instance, Apple really put a hurt on other user-interface vendors when it tweaked touch-screen technology for the iPhone. However, the software in other cell-phone and computer products at least allowed a more rapid partial response to the threat. So CD isn’t a cure-all – it just comes amazingly close to it. How to “go the final mile” is a discussion for a future blog post.

Let’s summarize: CD is an incredibly cool and incredibly useful technology, both to the vendor and to the consumer, because it results in a major increase in sales for the vendor and a major increase in satisfaction for the consumer. Moreover, because it’s also cheaper than traditional software development, both vendors and consumers see their costs decrease as the vendor’s use of CD in NPD rises (and for the typical business, that’s in addition to the cost savings and competitive advantage from more rapid development of better business-process software). Finally, because their needs are both satisfied and anticipated, customers become far more loyal, reducing business risk drastically.

Really minor nit: I note that CD is often applied strictly to the delivery stage of development. To my mind, extension to the entire process is appropriate, because delivery is the only major development stage (if we exclude operations after delivery) where the agile development methodology today is often applied minimally. In other words, “delivery” can mean one stage or the whole process, and in the real world if you make that one stage agile you are usually making the whole process agile – so it’s a good idea to emphasize the point of the whole exercise by equating continuous completion and continuous delivery.

The Relevance of Thoughtworks to Continuous Delivery

Thoughtworks is one of those smaller consultancies that took the Agile Manifesto’s ideas and ran with them while large development-tool vendors focused on other, ultimately far less effective, “best practices.” I have had my differences with Thoughtworks in the past, but always from a position of respect for an organization that clearly has agile development in its DNA to a greater extent than most. In the case of CD, a quick scan of the first page of a Google search on Continuous Delivery reveals no one else visibly applying it to the extent that Thoughtworks claims to be doing.

Does that mean that Thoughtworks is a stable vendor for the long term? One of the fascinating things about the agile-development market is that the question matters to a much lesser extent than in all previous markets. Look, in the early years some tried SCRUM and some tried extreme programming and some tried half-way solutions like slightly modified spiral programming, but it didn’t matter in the long run: the Wipros of the world have still done almost universally better than the startups focusing on Java or even folks like Cambridge Technology Partners. And that’s because agile development firms are, well, agile. It doesn’t just rub off on developers; some of it rubs off on managers, and strategists, and even, Ghu help us, on CEOs. They focus on changes; so, on average, they evolve more effectively. That’s as true of Thoughtworks as anyone else.

What isn’t true of most other agile development vendors, right now, is that Thoughtworks appears to have a significant edge in experience in the CD extension of agile development. That matters because, as I’ve said, something like Thoughtworks-type CD is the logical endpoint of agile development. So, if you want to get to maximal agile-development benefits sooner rather than later, it certainly seems as if Thoughtworks should be on your short list.

One point here requires elaboration: there is sometimes a misconception that outside providers are selling you agile-development services. That’s at least partially wrong. They are – or should be – fundamentally selling you agile-development training. They will make their money from being ahead of you in experience, and constantly selling you the improvements in agile development that their experience teaches them. Think of them as more like business-strategy consultants, always looking ahead to the new strategic idea and delivering that to you. Believe me, that’s just as valuable as running your data center – and is often more valuable than that.

Thus, Thoughtworks’ advantage in experience is not to be sneezed at. How well it will hold up over time, we will see. However, considering that the bulk of software development, even if it has adopted a modified version of SCRUM en masse, is still typically not very successful in embedding frequent user feedback into the typical project, I would say that Thoughtworks’ edge should last for at least a couple of years – an eon in the timeframe of the agile business.

Potential Uses of Thoughtworks-Type Agile Development for IT

An IT organization must crawl before it can walk; but it should also learn about walking before it tries to do so. That means that if IT is still at the early stages of adopting agile development, it should still apply CD to a “skunkworks” project, and if it doesn’t have such a project, it should create one.

Otherwise, this is not a “targets of opportunity” situation, but rather a “learn and merge” one. IT should bring CD on board as each project is ready for it, no faster, no slower. In my opinion, “ready” means that a project’s development process has (a) adequate process management tools specifically tuned to support agile development, thus allowing it to scale, (b) an adequate “store” of reusable infrastructure software to build on, so that moving to the next incremental feature is not too great a leap, and (c) an attitude from everyone involved that the first thing you do when you get something new is that you make it agile. That’s all. Print and ship.

Well, but today’s CD isn’t adapted to the peculiar needs of my development. Excuse me? You did say your process was agile, didn’t you? Believe me, CD is not only flexible but agile, and if you don’t know the difference, then you need far more help in achieving agility than you realize. What you’re really saying is that you don’t have an agile development process at all, because otherwise it would be straightforward to adapt your methodology to move steadily towards CD – using a vendor just speeds up the process change.

The Bottom Line for IT Buyers

I am always wary of being too enthusiastic about new technologies, because I remember a story in Gore Vidal’s Lincoln. A rich man has a tendency to exaggerate, and hires someone to nudge him at table when he does so. “How was your trip to Egypt?” “Amazing! Why, they have things called pyramids made of pure gold!” Nudge. “And they go a mile high!” Hard stamp on his foot, just as someone asks, “How wide?” He answers, in agony, “About a foot.” I worry that while any given technology’s value may seem a mile high, its real-world application will be about a foot wide.

That said, I really do see Continuous Delivery as a cool new technology that will have an impact, eventually, that will be a mile high and globe-wide. Here’s what I said in a previous post:

[[The real-world success of Continuous Delivery, I assert, signals a Third Age, in which software development is not only fast in aggregate, but also fast in unitary terms – so fast as to make the process of upgrade of a unitary application by feature additions and changes seem “continuous”. Because of the Second Age, software is now pervasive in products and services. Add the new capabilities, and all software-infused products/services -- all products/services – start changing constantly, to the point where we start viewing continuous product change as natural. Products and services that are fundamentally dynamic, not successions of static versions, are a fundamental, massive change to the global economy.

But it goes even further. These Continuous-Delivery product changes also more closely track changes in end user needs. They also increase the chances of success of introductions of the “new, new thing” in technology that are vital to a thriving, growing global economy, because those introductions are based on an understanding of end user needs at this precise moment in time, not two years ago. According to my definition of agility – rapid, effective reactive and proactive changes – they make products and services truly agile. The new world of Continuous Delivery is not just an almost completely dynamic world. It is an almost Agile World. The only un-agile parts are the rest of the company processes besides software development that continue, behind the scenes of rapidly changing products, to patch up fundamentally un-agile approaches in the same old ways.]]

But you don’t need to know about that. What you need to know is that for every IT organization that appreciates agile development, kicking the tires or adopting CD is a good idea, right now. I don’t even have to say it’s necessary, because that’s not the way an agile organization operates.

As for Thoughtworks, here’s what I think IT’s attitude should be. I have often trashed Sun for an ad that said “We built the Internet. Let us build your Internet.” I knew, based on personal experience, that this claim was, to say the least, exaggerated. Well, if Thoughtworks came to your door with a pitch that said “We built Continuous Delivery. Let us build your Continuous Delivery,” I would not only not trash them, I would encourage you to believe them, and consider doing as they request. The pyramid is that high. The materials with which it is built are that valuable. And my foot is still intact.

The Other BI: Oracle TimesTen and In-Memory-Database Streaming BI

2012-01-25T17:00:00.000-08:00

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about BI today.

The Importance of TimesTen-Type In-Memory Database Technology to BI

All right, now I’m really stretching the definition of “other”. Let’s face it, Oracle is “top of the mind” when we talk about BI, and they recently announced a TimesTen appliance, so TimesTen is not an invisible product, either. And finally, the hoopla about SAP HANA means that in-memory database technology itself is probably presently pretty close to the center of IT’s radar screen.

So why do I think Oracle’s TimesTen is in some sense not “top of the mind”? Answer: because there are potential applications of in-memory databases in BI for which the technology itself, much less any vendor’s in-memory database solution, is not a visible presence. In particular, I am talking about in-memory streaming databases.

To understand the relevance of in-memory databases to complex event processing and BI, let’s review the present use cases of in-memory databases. Originally, in-memory technology was just the thing for analyzing medium-scale amounts of financial-market information in real time, information such as constantly changing stock prices. Lately, in-memory databases have added two more BI duties: (a) serving as a “cache” database for enterprise databases, to speed up massive BI where smaller chunks of data could be localized, and (b) serving as a really-high-performance platform for mission-critical small-to-medium-scale BI applications that require less scaling year-to-year, such as some SMB reporting. These new tasks have arrived because rapid growth in main-memory storage has inevitably allowed in-memory databases to tackle a greater share of existing IT data-processing needs. To put it another way, when you have an application that is always going to require 100 GB of storage, sooner or later it makes sense to use an in-memory database and drop the old disk-based one, because in-memory database performance will typically be up to 10-100 times faster.

Now let’s consider event-processing or “streaming” databases. Their main constraint today in many cases is how much historical context they can access in real-time in order to deepen their analysis of incoming data before they have to make a routing or alerting decision. If that data can be accessed in main memory instead of disk, effectively up to 10-100 times the amount of “context” information can be brought to bear in the analysis in the same amount of time.

In other words, for streaming BI, IT potentially has two choices – a traditional event-processing database that is often entirely separate from a back-end disk-based database, or (2) a traditional main-memory database already pre-optimized for in-depth main-memory analysis and usually pre-integrated with a disk-based database (as TimesTen is with Oracle Database) as a “cache database” in cases where disk must be accessed. How to choose between the two? Well, if you don’t need much historical context for analysis, the event-processing database probably has the edge – but if you’re looking to upgrade your streaming BI, that’s not likely to be the case. In other cases, such as those where the processing is “routing-light” and “analysis-heavy”, an in-memory database not yet optimized for routing but far more optimized for in-depth analytics performance would seem to make more sense.

Thus, one way of looking at the use case of in-memory database event processing is to distinguish between in-enterprise and extra-enterprise data streams (more or less). Big Data is an example of an extra-enterprise stream, and can involve a fire hose of “sensor-driven Web” (GPS) and social media data that needs routing and alerting as much as it needs analytics. Business-critical-application-destined and embedded-analytics data streams are an example of in-enterprise data, even if admixed with a little extra-enterprise data; they require heavier-duty cross-analysis of smaller data streams. For these, the in-memory database’s deeper analysis before a split-second decision is made is probably worth its weight in gold, as it is in the traditional financial in-memory-database use case.

Won’t having two databases carrying out the general task of handling streaming data complicate the enterprise architecture? Not really. Past experience shows us that using multiple databases for finer-grained performance optimization actually decreases administrative costs, since the second database, at least, is typically much more “near-lights-out,” while switching between databases doesn’t affect users at all, because a database is infrastructure software that presents the same standard SQL-derivative interfaces no matter what the variant. And, of course, the boundary between event-processing database use cases and in-memory ones is flexible, allowing new ways of evolving performance optimization as user needs change.

The Relevance of Oracle TimesTen to Streaming BI

In many ways, TimesTen is the granddaddy of in-memory databases, a solution that I have been following for fifteen years. It therefore has leadership status in in-memory database use-case experience, and especially in the financial-industry stock-market-data applications that resemble my streaming-BI use case as described above. What Oracle has added since the acquisition is database-cache implementation and experience, especially integrated with Oracle Database. At the same time, TimesTen remains separable at need from other Oracle database products, as in the new TimesTen Appliance.

These characteristics make TimesTen a prime contender for the potential in-memory streaming BI market. Where SAP HANA is a work in progress, and approaches like Volt are perhaps less well integrated with enterprise databases, TimeTen and IBM’s solidDB stand out as combining both in-memory original design and database-cache experience – and of these two, TimesTen has the longer in-memory-database pedigree.

It may seem odd of me to say nice things about Oracle TimesTen, after recent events have raised questions in my mind about Oracle BI pricing, long-term hardware growth path, and possible over-reliance on appliances. However, inherently an in-memory database is much less expensive than an enterprise database. Thus, users appear to have full flexibility to use TimesTen separately from other Oracle solutions, free from worries about possible long-term effects of vendor lock-in.

Potential Uses of TimesTen-Type In-Memory Streaming BI for IT

As noted above, the obvious IT use cases for TimesTen-type streaming BI lie in driving deeper analysis in in-enterprise streaming applications. In particular, in the embedded-analytics area, in-memory performance speedups can allow consideration of a wider array of systems-management data in fine-tuning downtime-threat and performance-slowdown detection. In the real-time analytics area, an in-memory database might be of particular use in avoiding “over-steering”, as when predictable variations in inventories cause overstocking because of lack of historical context. In the Big Data area, an in-memory database might apply where the data has been pre-winnowed to certain customers, and a deeper analysis of those customers fine-tunes an ad campaign. For example, within a half-hour of the end of the game, Dick’s Sporting Goods had sent me an offer of a Patriots’ AFC Championship T-shirt, complete with visualization of the actual T-shirt – a reasonably well-targeted email. That’s something that’s far easier to do with an in-memory database.

IT should also consider the likely evolution of both event-processing and in-memory databases over the next few years, as their capabilities will likely become more similar. Here, the point is that event-processing databases often started out not with data-management tools, but with file-management ones – making them significantly less optimized “from the ground up” for analysis of data in main memory. Still, event-processing databases such as Progress Apama may retain their event-handling, routing, and alerting advantages, and thus the situation in which in-memory is better for in-enterprise and event-processing is better for extra-enterprise is likely to continue. In the meanwhile, increasing use of in-memory databases for the older use cases cited above means that in-memory streaming-BI databases offer an excellent way of gaining experience in their use, before they become ubiquitous. That, in turn, means that narrow initial “targets of opportunity” in one of the situations cited in the previous paragraph are a good idea, whatever the scope of one’s overall in-memory database commitment right now.

The Bottom Line for IT Buyers

In some ways, this is the least urgent and most speculative of the “other BI” solutions I have discussed so far. We are, after all, discussing additional performance and deeper analytics in a particular subset of IT’s needs, and in an area where the technology of in-memory databases and their event-processor alternatives is moving ahead rapidly. In a sense, this is really an opportunity for those IT shops that specialize in applying a little extra effort and “designing smarter” across multiple new technologies to provide a nice ongoing competitive advantage. For the rest, if the shoe can easily be made to fit, why not wear it?

My suggestion for most IT buyers, therefore, is therefore to have a “back-pocket” in-memory-database-for-streaming-BI short list that can be whipped out at the appropriate time. Imho, Oracle TimesTen right now should be on that list.

I hate to close without noting the overall long-term BI potential of in-memory databases. The future of in-memory databases is not, in my firm opinion, to supersede the IBM DB2s, Oracle Databases, and Microsoft SQL Servers of the world, at any time in the next four years. The hardware technologies to enable such a thing are not yet clear, much less competitive. Rather, the value of in-memory databases is to allow us to optimize our querying for both main-memory and disk storage – which are two very different things, and which will both apply appropriately to many key customer needs over the next few years. Overall, the effect will be another major ongoing jump in data-processing performance. As we enter this new database-technology era, those who initially kick the tires in a wider variety of BI projects will find themselves with a significant “experience” advantage over the rest, especially because the key to outstanding success will be determining the appropriate boundary between disk-based and in-memory database usage. Don’t force in-memory streaming BI into the organization. Do keep checking to see if it will fit your immediate needs. Sooner or later, it probably will.

The Other Agile Development: AccuRev SCM and Code Refactoring

2012-01-20T17:20:00.000-08:00

Methane Update: Less Worried, Still Very Worried

2012-01-19T13:12:00.000-08:00

I just saw an interview by Skeptical Science with Ms. Sharapova, a Russian scientist, on the Russian findings that sparked my recent methane worries. Her responses clarified the answers to the main questions I had about Arctic sea methane clathrates. Her key information, imho, was the following:

1. The Russians recently discovered that methane clathrates could form not just in the 200-1000 m depth range, but also in the 20-200 m range.
2. They also found that clathrates there did not just melt from the top down; they also melted in pockets below the surface melt.
3. The 2011 survey, for the first time, looked at the 20-200 m Siberian coastal shelf, rather than the 200-2000 m deeper waters.

Let’s look at the implications for my methane analysis. In the first place, this explains why methane was able to bubble to the surface, instead of popping or being eaten by methane-munching bacteria: it was too close to the surface, and especially if, as appears to have been the case, it was being released in larger chunks/bubbles.

In the second place, this appears to indicate that the ramp-up in methane emissions at any particular point is less than I feared. There are several possible reasons to anticipate that at greater depths, methane release from the sediment would ramp up more slowly than a 100-fold increase in one year. Likewise, there are several possible reasons to anticipate that initial methane releases from the shallow continental shelves would be greater than that from deeper areas, if there were methane clathrates there in the first place.

However, in the third place, this newly discovered source of methane clathrates appears to be a much bigger source of emissions, both in terms of melting more rapidly and of having more methane stored to begin with. Because sea shelves slope more rapidly the deeper they get (to a point beyond 1000 m), the sea-surface area of 20-200 m deep shelves is comparable to the sea-surface area of 200-1000 m ones.

Under the surface, the methane clathrates can be stored much deeper before earth heating and pressure melt them. Take these two things together, and the amount of methane in Arctic clathrates may be 2-4 times the amount previously estimated. Meanwhile, this 20-200 m range lies almost entirely in the “shallow ocean” range where warming currents from the south plus warming of newly exposed surface waters by the summer sun create hotter water next to the sediment – and thus melt things faster. These points are confirmed by the observations of rapid bubble generation and much larger funnels from which methane flows in the 20-200 m range.

In the fourth place, the ability of 20-200 m methane bubbles to rise to the surface means that we probably grossly underestimate the percentage of emitted methane that will rise into the atmosphere as methane rather than carbon dioxide. Frankly, this is probably good news, since that means less of it will eventually stay in the atmosphere as carbon dioxide – but there’s still a chance it may stay as methane for a long time – and be far worse for global warming. This would happen if there’s too much methane up there and the OH in the atmosphere that removes a lot of that methane runs out, a possibility some scientists have raised.

In the fifth place, the existence of pockets indicates that methane emissions may be bursty, as surface melt “burns through” to those pockets that are themselves melted, but are still trapped by frozen clathrates above. Those bursts should be frequent enough to keep methane emissions at a higher yearly rate.

Implications

Overall, this makes me a little more hopeful about overall methane emissions and their effect on global warming. While there is perhaps 2-4 times the amount of methane clathrates to emit than I thought, there may be 30-70% lower emission rates than I anticipated, and that’s the key to methane’s overall effect in the next 160 years, when it matters. To put it another way, the net methane emission rates per year should be lower than I expected, and the amount in the atmosphere as methane in the next 160 years should be lower than my worst-case all-methane scenario (assuming there’s enough OH). Hopefully, the amount of carbon dioxide should be lower as well, because of the decreased yearly emissions amount and increased percentage arriving as methane (only half of that turns into carbon dioxide). However, this isn’t sure, because if the OH runs out, the effect of yet more “steady state” methane over, say, 600 years on global warming will be worse than I had anticipated.

All in all, I would now tend to put the likely overall new natural-source methane emission effects (also including permafrost and wetlands) in the 3-6 degree C range over the next over the next 200 years, and in the 2-5 degree Celsius range in the 400 years after that – overall, perhaps a 25% boost to global warming rather than a 50% one, protracted over more years. High water may not be delayed, but hell may be a little less hellish in temperature, and the end of life on earth ever so slightly less likely, than I feared.

Unless, of course, the OH in the atmosphere runs out … the worries never end, do they?

A Few Books That "Rocked My World"

2012-01-14T15:48:00.000-08:00

I recently saw a list of “100 books that rocked my world” from a blogger. It turned out to be a list of “books that are really cool” and not “books that made me think differently in fundamental ways.” So I thought I’d look back and ask myself, years later, what books changed me fundamentally?

1. Godel’s Proof, Nagel and Freeman. As Shaw says in “Man and Superman”, it made me want to “think more, so that I would be more.” The idea of there being some things that I will never know or prove, is something that I am still wrestling with.

2. Spark, Ratey. The idea that we fluctuate chemically between addiction to pessimism and addiction to optimism based on whether we are channeling our hunter ancestors seeking prey by exercising in company, between learning by moderate physical stress and forgetting based on sedentary habits, between inoculating against disorders by moderately poisoning ourselves with food and moderately stressing ourselves with exercise and opening ourselves to disease and death by eating unchallenging foods and avoiding challenging exercise, seems to apply to and alter every aspect of my life.

3. The Age of Diminished Expectations, Krugman. It began to give me an ability not only to understand my sense of disconnect between conventional economics and what was happening to me in the real world, but to apply new tools to understand and improve the broad scope of my life in terms of money and generational cycles – something I’d been looking for over 20 years of college and work. Of course, I needed several additional books and articles to understand the full scope of Krugman’s approach.

4. The Fellowship of the Ring, Tolkien. I wasn’t expecting what I got when I scrounged my Dad’s library for yet more books at age 12. Suddenly, I was able to see the non-human world around me as a separate, interconnected, wonderful, alive thing. And the idea that you could frame a book or part of a life as the necessary preface leading to the beginning of a journey – like Frodo’s, stripped of his teachers, into Mordor – made me see that my life could be seen that way – and that has always given me hope.

5. Falconer, Cheever. This one is painful. I had to ask myself, after it was over, am I, like the protagonist, seeing my relationships with women too much in terms of my own needs, or can I finally grow up? I don’t know how much I’ve changed in my behavior after reading it; but I know I can never think the way I used to about relationships without far greater discomfort and dissatisfaction with myself.

6. How to Win Friends and Influence People, Dale Carnegie. There are many, many things wrong with this book, as I have come to realize. But it gave me the humility of understanding that my artsy and intellectual achievements were of very little value to others, and showed me that I really liked people, if I just listened to them. It also gave me a basis for understanding people’s social viewpoint that has allowed me to connect, slowly, over 30 years.

7. A Connecticut Yankee at King Arthur’s Court, Twain. I don’t think I recognized this at the time, but it was my introduction to what I might call the “science fiction viewpoint”: the idea that by facing scientific facts and leveraging technology, you could make a fundamental change for good in the world – the true meaning of “progress”. I can never quite shake that idea, and it has led me in quite a different direction from the rest of my family, into math and computers, and away from Great Literature that had few solutions to offer. Of course, I never quite accepted Twain’s other idea: that this progress could vanish like the mist from history, resisted by willfully ignorant humans frightened of change, unless you were lucky.

The Other BI: HP Vertica and Columnar Databases

2012-01-13T12:22:00.000-08:00

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about BI today.

The Importance of Vertica-Type Columnar Database Technology to BI

Last year, I wrote a blog post saying that it was likely that HP would underestimate the columnar database technology in Vertica, and if so they were missing a major opportunity. In the last year, HP has been pretty quiet about Vertica, but I have partially changed my mind, to the point where I want to call attention to Vertica as a less visible candidate for IT buyers to get the full benefits of columnar database technology over the next 2-3 years.

Let’s start with columnar technology. Here, I want to go more in-depth into Vertica’s core technology than usual, because it’s an excellent way to begin to see the benefits of columnar beyond traditional row-oriented databases.

The original idea of Vertica was to recast the relational database to focus on the (data warehousing) case where there are few if any updates. The redesign started with the idea that the data should be stored in "columns" rather than rows; the details of this are that the columns themselves (because they don't have to follow relational dogma) can be stored in a highly compressed format, with lots of compression techniques like inverted list, bit-mapped indexing, and hashing, as appropriate. Thus, (a) the database can use the column format to zero in faster on the data that the query is gathering, (b) because the data is compressed an average of 10 times (according to Vertica), more data can be crammed into main memory for faster processing. Result: a claimed 10-100 times speedup in performance, comparable to in-memory databases but far more scalable. It also means the database can handle at least 10 times more data (say, 100 terabytes instead of 5) with the same performance for a given query; or that the data center can use an order of magnitude less storage.

Now, all this does not come without a cost, and the typical cost would at first seem to be speed of updating. That is, the column storage format requires more revision of the data stored on disk when an update arrives, so update is slower. But this is counteracted by the ability to load more of localized data at once into main memory in a compressed form, for faster in-memory updating. Only at update frequencies typical of old-style operational online transaction processing (OLTP) does the row-oriented relational database have a clear edge.

The elaboration of the design in Vertica is that the basic data is also stored as "projections" (aka materialized views). That is, a set of columns in a tuple is stored one (relational) way; each column also shows up in a projection, but the projection is cross-tuple (one from tuple A, one from tuple B, etc.). This accomplishes two things: one, it gives an alternative way of querying which may be faster than basic storage, and two, it gives redundancy and therefore robustness, in a similar way to RAID 5 (projections can be "striped" across disks).

Now, here's where things get really interesting. Practically speaking, today, in data-warehousing-type databases, updates via "load windows" are becoming more and more frequent, to the point where data is pretty up-to-date and updates are a bigger part of data warehousing. To keep "write locks" from gumming up performance (especially with column update being slower), Vertica splits the storage into a write-optimized column store (WOS; effectively, a cache) and a Read-optimized Column Store (ROS). Periodically, the WOS becomes the ROS. So the write locks for the updates only interfere with reads when there’s a mass update. At the same time, such a mass update can re-store whole chunks of the ROS for optimum storage efficiency. Moreover, to gain currency, the query can be carried out across the ROS and WOS. And, because there is all this redundancy, there is no need for logs—another performance improvement. Note that because of its redundancy, Vertica doesn't need to do roll-back/roll-forward nor backup/restore.

The net of all this for IT buyers is that columnar databases in general, and Vertica in particular, should be able to deliver on average much better performance than traditional relational databases in the majority of not-highly-update-intensive cases, due mostly to its compression abilities, and that addition of other technologies like in-memory technology to both alternatives will not alter this superiority.

The Relevance of HP Vertica to BI

This kind of approach cries out for integration with or development of sophisticated admin tools, expansion beyond data warehousing and analytics to “mixed” transactions in competition with the noSQL fad, better programming tools to build up a war chest of business/industry customized solutions, and using a relational database as an OLTP complement. The resulting data-management platform would be a solid alternative for all sizes of enterprise to the “relational fits all” or “let the thousand flowers bloom” strategies of most organizations.

Once this platform is in place, it needs to become the keystone of enterprise architectures, not just an analytics or business intelligence “super-scaling” engine. That means adding integration with semi-structured and unstructured data. It also means adding major functionality for handling content, and integration with storage software for additional performance optimization. And so, anticipating that HP would not do this, I criticized the HP acquisition of Vertica last year.

Well, two things happened: HP did more than I thought it would, and competitors did less. HP bought a company called Autonomy, which added semi-structured/unstructured data support. Necessarily, this takes Vertica beyond pure data-warehousing-style analytics into a more update-intensive world, and HP’s redirection of Mercury Interactive towards agile ALM (application lifecycle management) associated Vertica with better programming tools. Meanwhile, SAP took its eye off Sybase IQ with its focus on HANA, IBM at least temporarily walked away from its Netezza semi-columnar database technology, and Oracle’s columnar-optional appliance ran into questions about its long-term hardware growth path. In other words, the result of half a loaf from HP and less than half a loaf from everyone else is that Vertica is moving towards leadership status in delivering columnar database technology to all scales of BI and analytics.

Meanwhile, of course, only the deluded think that HP will suddenly vanish, while database technology and the rest of the new software embed themselves ever deeper in HP’s DNA. HP Vertica is going to be around for quite a while; and it will be an attractive option for quite a while.

Potential Uses of Vertica-Type Columnar-Based BI for IT

The use cases of a columnar database IT is straightforward. IT should use a columnar database in new projects as an alternative or complement to a traditional relational database, unless the operations are update-intensive, in which case row-oriented relational is preferred. As a complement, columnar databases operate on a “switching” basis, in which an overall engine decides which queries should be allocated to row-oriented, which to columnar, usually on the basis of whether two or more of the “fields” involved in an operation can be compressed highly by using a columnar format. Oracle (and, until recently, IBM Netezza) takes this approach; but IT can also do its own switching mechanism.

And that’s it. Over the next 2-3 years, if not already, columnar can scale as high as querying, can integrate with as many data types and upper-level tools and applications, and can evolve to greater performance/scalability just as rapidly as the traditional row-oriented database. In the long run, in a lot of use cases, and sometimes in the short run, that favors Vertica-type columnar.

However, right now, columnar requires in some cases to “grow into” its assigned role in a new project, by adding administrative tools for particular cases. Therefore, in most applications where 24x7 operation and an adequate level of customer response time is business-critical, relational row-oriented should still be preferred. That should leave plenty of analytical and other BI uses for which Vertica-type columnar database software will deliver an important performance advantage.

The Bottom Line for IT Buyers

Over the next few years, IT buyers can take one of two views: the author of this blog post is prescient, columnar will replace row-oriented in the majority of new applications in BI and other areas, and we should include columnar in all our short lists from now on; or, the author of this blog post is wrong about the future, but columnar is useful for some things right now, and trying to standardize on one database is a fool’s game that we no longer bother to try to play. If IT buyers hold the second view, then they should be focused on applying columnar to analysis of huge amounts of structured data with “sparse” fields where high compression is achievable – like five-field customer names (Mr. John Taylor Jakes, Jr.) and product codes. Spend the resulting improvements on increased performance, lowered storage costs, or both.

Again, this is not a matter of a pre-short list, unless you have a “gray area” BI project involving somewhat update-intensive or somewhat business-critical little-downtime apps, in which case you want to wait for columnar to evolve a little. In all other cases, HP Vertica should go on the short list along with the obvious others, like Sybase IQ. Right now, Vertica appears to be ahead both in some of the needed features to adapt to new analytics needs and in speed of evolution. One never knows – but over the next year, that leadership role may continue.

Above all, IT buyers should not listen to any FUD from traditional relational vendors suggesting that this is yet another new technology, like object databases, that will eventually fall to earth with a thud. Columnar database technology proved its superiority in many situations long ago in the non-relational world, with CCA’s Model 204, and has found uses continuously since then, like bit-mapped indexing. Most times there’s a fair BI matchup, as with some of the TPC benchmarks of the last seven years, columnar comes out well ahead. Under whatever name, columnar database technology is not going away. Therefore, its markets will continue to grow relative to row-oriented relational. For IT buyers, acquiring columnar BI solutions like HP’s Vertica is simply being smart and getting a little ahead of the curve.

The Other Agile Development: CollabNet and Collaborative Agile Development

2012-01-12T13:29:00.000-08:00

Ooh, Those Hilarious Computer Industry Marketers!

2012-01-11T13:21:00.000-08:00

Look, I know that picking on marketers is unfair. They have a really tough job, trying to explain the technology they can’t possibly fully understand to business customers and depending on techies to cover their rear flanks. But sometimes, sometimes it’s just impossible not to make fun of them. Because what they say is just so – just so … funny.

I just got an email from IBM that starts out like this: “IBM has announced the closing of its acquisition of Green Hat, which expands IBM's ability to help customers improve the quality of software applications by enabling developers to conduct testing on a software application prior to its delivery.” The italics are mine, all mine. Look, I know what they were trying to say. I know that Green Hat has valuable technology, and it can indeed improve development quality in certain cases. I know that bad grammar played a role. But still …

Gosh, IBM, this is marvelous. I never thought of testing an application before delivery before. What an incredible insight! Maybe that explains why every software application in the world before now has crashed before you can even use it! If it weren’t for the computer hardware that just sits there and blinks its lights, I bet we would never have been able to have such a vibrant and humongous computer industry. Please, IBM, don’t just stop there. Submit this for a Nobel Prize. Turn Green Hat into open source, for the good of all humankind. I beg you – or I will, when I can stop laughing.

Or maybe you’re restricting this to IBM customers. Maybe it’s only IBM customers who have never looked up, never considered the possibility of testing their applications before delivering them. 50 years in the salt mines, never able to get an application to work, always being teased by the developer down the hall who has written a Windows app to add two and two, with the dust and cobwebs accumulating all over you, and now, at last, a ray of sunlight, the mummy arises, the worm turns, and you can write your very first workable application for zEnterprise using Java and JDBC, Web-enabled out of the box, infinitely scalable, but baby steps … to let the entire corporation add three plus three. Oh, the ecstasy!

And the best part is that now I’m remembering all the other screamingly funny computer-industry marketing gems. Perhaps my favorite is a billboard I saw in Boston in the late 1990s, from Sun: “We built the Internet. Let us help you build your Internet.”

Oh, gosh, Mr. McNeally. So that was you in the government all along, pretending to be working for DARPA, physically going out to all the computers in all the telcos all over the world and secretly installing all the Internet software. All those employees at BB&N and CCA that thought up TCP/IP and newsgroups, routing and ftp, they were Sun employees, even though nominally Sun had about five people working for it at the time, supposedly on CAD/CAM! All those techies who did Berkeley UNIX, Mosaic, hypertext, HTTP, and all those administrators who installed them on existing enterprise servers and clients, they were all little Sun robots! It’s a shadow government! It’s the Illuminati! Of course I want you to come into my house and link my two PCs and publish their addresses! I wouldn’t dare not to, because you clearly control everything else in the world … Just give me a moment to stop laughing.

I remember that I actually pointed out that one out to a Sun marketer, back in the day. He was quite surprised, and insisted that it made perfect sense, because Sun had acquired some former BB&N employees. It was very hard to talk to him, because he kept making me want to crack up.

However, aside from their humor value, these marketing claims extraordinaire really do have value to me, and possibly to you, too. No, really. That is a straight face. Let me tell you a story …

Once there was a boy taking biology, faced with a test on which there was a question: When does the female rabbit ovulate? He couldn’t remember the answer. He knew it had to be something dramatic, because he knew that rabbits multiply like crazy – his family had had two in the backyard, until within a very short time of arrival they had gone from two to about 20, and his parents had moved very quickly to get rid of them all before they ate all the vegetation and multiplied again. And so he panicked, and his answer was: upon sight of the male rabbit.

The next class, the teacher handed out the graded papers. But before he did so, he made an announcement. I was grading these last night, he said, and I was getting too tired to finish. And then I ran across this answer – and, without naming the source, he related the boy’s answer to the question.

And so, he concluded, I was so amused by that answer, that it woke me up and I had enough energy to finish grading the papers. By the time the teacher said that, of course, the class was gone – gone to laughter. ROTFL.

And so, thank you, IBM. Thank you, Sun. Thank you, all the hard-working marketers who have ever made a mistake like this. Thanks to you, I can finish this blog post. Thanks to you, I will always remember that there are many amazing new technologies yet to be discovered, that will continue to enthrall me and improve the lot of humankind in the years to come. Like software testing. Like an entire global cloud infrastructure built by one person working for the Illuminati. Like female robots that ovulate on sight of the male. Like marketers that make it all happen without writing one single line of code.

It’s a wonderful world to write about. I’ll start writing immediately. Just as soon as I can manage to stop laughing.

The Other BI: Birst and the Mobile Ad-Hoc Query

2012-01-10T13:56:00.000-08:00

Oh, Those Statistical Nits!

2012-01-09T09:05:00.000-08:00

Recently, I read a Paul Krugman NY Times column on income inequality, which referenced a NY Times article on income inequality, which referenced a Pew Research Center study on income mobility over generations. The NY Times article stated flatly that the Pew study found that 81% of Americans have more income than their parents. I read the first part of the study carefully, and it did indeed state that most sons of parents in the study, whether white or African-American, bottom or middle or top third of parental income, earned more than their parents did. And then I read even more carefully, and realized that the data absolutely did not support a conclusion that today’s American sons earn more than their parents did.

What went wrong? Well the study took a longitudinal study of families whose sons were between 0-18 years of age in 1967-1971, and compared the family income of the parents in 1967-1971 to the family income of the sons in 1995-2002 (omitting a couple of years). There were three basic problems with the analysis. First, government data shows that for the bottom third of family incomes, family income grew from 1967 to 1979, decreased slightly until 1994, grew again until 2001, and decreased to below the 1979 level by 2010. For the middle third, there was a similar trend, except that family incomes are now about the same or slightly below the 1979 level. For the upper third (and especially for the upper 1%), family income has grown consistently and, by 2010, substantially over 1967-1971. So the choice of the two “snapshot” time periods maximized the growth of income between parents and sons in all three income strata. Thus, it is likely that had the period been, say, 1980-1984 vs. 2006-2010, far fewer sons would have increased their family income compared to their parents.

The second problem with the study was the interval of the two “snapshots”. We know from demographic data that people were having kids, on average, earlier, in the 1950s and 1960s, and so it is reasonable to suppose that those sons who were 0-18 in 1967-1971 typically had parents that were 21–49, and most frequently about 35, while the sons in turn during 1995-2002 would be 24-55, and most frequently around 40. The reason this matters is that government data shows that families’ earnings trend steadily upwards from 20 onwards, and reach their peak from 45-55. In other words, the time period chosen exaggerated the income earned by the sons by pushing more of them into a peak earnings period.

The third problem with the study’s statistical approach is that it took “family income” as equivalent to “personal income.” Back in 1967-1971, across all three strata, less than one-third of women worked. According to the latest Census data, perhaps 80% as many women as men work, and that was pretty much true in the 1995-2002 period. These, in turn, have been earning perhaps 80% as much as men. So, especially in the bottom and middle thirds, women contributed less than 10% of average “family income” in the 1967-1971 period, and about 40% of “family income” in the 1995-2002 period. If we are really comparing apples to apples, we have to say that if we compare fathers to sons, it is clear that any upward trend in income is far less frequent. Now, the study notes that the “family income” is converted to personal income by being “family-size adjusted in all analyses”; but all this does is exaggerate things even further, because family sizes were slightly smaller in 1995-2002 (and now) than in 1967-1971.

One caveat: I was unable to access the Appendix to the study, which explained Pew’s methodology in greater detail. It is always possible that they dealt with these problems to some extent by further statistical tweaks. However, I view that as pretty unlikely, since these considerations are so important that they should have been noted in some way in the main paper.

Now, it is important to keep in mind that I am not a statistics “rocket scientist.” All it took me to figure this one out was a little ongoing digging on the topic of income inequality, and a careful lay-person reading of the methodology section of the Pew paper. The problem here is not that Pew was “lying with statistics”, because the facts were right there in the front of their study report. The real problem is that the so-called journalist of the NY Times apparently didn’t even bother to read that section carefully, much less do a little additional research which would have called the Times’ “81% of Americans” into further question.

So, as that fellow in the insurance commercials would say, what have we learned here? Well, first of all, statistical nits matter. I suspect that when the dust settles, we will find that less than half of all American males are presently earning as much, in real terms, as their fathers did (and the women aren’t a slam dunk either, since much of the surge in their employment and wages happened by the late 1980s). Even if that isn’t true, there’s no way the figure is anywhere near 81%. You need to consider statistical nits like the ones I have cited to convert a statistical study into a realistic picture of what’s going on in the real world.

Second, and equally important, you can’t trust any old reporter to do it for you, no matter how prestigious the name of their institution. You at least have to make a stab at the statistical nits yourself – or you’ll wind up believing what just isn’t so. Thank heavens for the Web, so that we can begin to check those statistical nits. Thank heavens for the Web, which gives us pointers to data that put those statistical nits in context. If we fail to do so, then by all means blame the knucklehead at the NY Times – but also blame ourselves.

The Other BI: Composite CIS/Studio and Agile BI

2012-01-06T16:48:00.000-08:00

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about BI today.

The Importance of the Composite Software Technology to BI

Today’s enthusiasm for “agile BI” that often consists of slapdash applications of SCRUM to rapid roll-out of new analytics applets obscures the fact that business agility, not to mention New Product Development (NPD) agility, involves applying more of agile methodologies than a sprint, and using agile methodologies in many more areas than software development. However, this is where the market is right now. The issue at hand is not so much the better application of business agility in the future, as applying today’s “agile BI” more effectively in the next 2-3 years. This is best done by recognizing one key fact: infusing the organization’s BI with agility is not primarily about development agility – it is about information agility.

Speeding up an inflexible business process via targeted analytics simply doubles your investment in and commitment to an information-handling process that needs instead the ability to fundamentally change. As I have noted in past posts, surveys show that in the typical process of receiving data and using analytics as part of the process of turning it into the right decision by the right decision-maker, or the right change in the business, the biggest problem is not that information is delivered too slowly. The fundamental problem is (a) that “leaks” at every step in the process result in more than 2/3 of the actionable information being lost along the way, and (b) the inflexibility of the process – specifically, its lack of openness to new types of data as they arrive – keep making the “leaks” worse.

Today’s best technology for dealing with (a) and (b) is something now called “data federation” or “data virtualization”. I won’t repeat the long litany of benefits to be expected from data virtualization; here, I will simply note that data virtualization, like master data management, provides one view of related data across the enterprise to the developer and the end user, and, unlike master data management, it can actively reach out, within or beyond the enterprise (as in Composite Discovery), to discover new data sources that can likewise be part of that global view of potential information. The one view of data tackles problem (a) at several critical points by ensuring that everyone can potentially see all data, and that analytics tools can be potentially applied to all useful data. The discovery feature tackles problem (b), if used effectively, by constantly refreshing the actionable data from, let’s say, new social-media data sources not presently covered by BI – as some Composite solutions do for Big Data.

Solving problem (b), of course, improves the organization’s information agility, not just its development agility. So the quick and effective method of tackling information agility in the here and now is to make a data virtualization development tool part and parcel of agile BI projects. Tools from folks like Composite Software, Tibco, Denodo, IBM, and SAP/Sybase are among those that would seem to fit the bill.

The Relevance of Composite Software to BI

Surprisingly enough, for a while around five years ago data virtualization’s predecessor, Enterprise Information Integration, was a hot topic in BI, as a way of extending data warehousing to operational data stores in order to query certain key new data in “near real time.” The primary results of this “fad” were to finally and definitively establish a major market for data virtualization, as well as to cement BI as Composite Software’s core market. In other words, unlike most or all other data-virtualization vendors, Composite Software has ten years of experience with real-world data virtualization at medium to large scales, five years of experience with BI at the same scales, a demonstrated ability to combine with other flexible BI tools such as MicroStrategy, and a focus on handling more “complex” needs that translates naturally to the more complex in-depth analytics typical of social-media and related Big Data analytics.

It may reasonably be objected that Composite, as well as other data virtualization vendors, has shown little or no interest in agile software development methodologies up to now, and therefore use of data virtualization solutions such as Composite’s may tend to slow down agile BI development efforts. Again, I would argue, as someone who has been touting agile development nearly since its inception twelve years ago, that this misses the point on two counts. In the first place, the rule of thumb with agile tools is “lead, follow, or get out of the way.” Where a SCRUM-supporting tool leads by urging developers towards agile best practices, and a really good refactoring tool follows in its wake by adding flexibility to the code thus produced, there is a third category of tools that simply automates and simplifies the necessary tasks of software development, while not getting in the way of the developer – and that is what something like Composite Information Server (CIS) with Composite Studio does for data-accessing functions in agile-BI programs.

In the second place, data virtualization such as Composite Software’s comes with a built-in gain in any agile-BI program’s information agility, which is often just as valuable as increased analytic-program implementation speed. Out of the box, the developer is assured of not missing a key in-enterprise data source. More subtly, the developer is made aware of other data sources than the one he or she may initially have targeted, which means that, as should be expected in agile development projects, new and unexpected features with value-add surface in the middle of the project.

In other words, while Composite Software may be in the middle of the pack when it comes to agility applied to agile-BI software development, it has long taken a leadership role in the features of data virtualization that really matter to agile-BI users, including BI services expertise, discovery, and complex data virtualization.

One final caveat appears to be going away. Since their inception, data virtualization tools have been oriented towards users above a certain size of medium-sized organization. Obviously, much of the action and excitement around agile BI has centered around its use in the public cloud by SMBs, who traditionally don’t deal with lots of various data sources that could use virtualization. However, what these SMBs are finding is that some public-cloud data is inherently “multi-organization”, and therefore requires federation and virtualization – especially since many of these same SMBs are now using multiple public clouds for their analytics. Sooner or later, Big Data has to be combined with ERP data on the cloud, and then with customer data on salesforce.com or wherever, and so on. As a result, even these SMBs are probably going to find data virtualization and Composite-type technology useful in their agile-BI efforts sooner rather than later.

Potential Uses of CIS-Type Agile BI for IT

Until recently, data virtualization in the enterprise has typically been done on a per-project basis, with the result that spreading it across all in-enterprise data is typically a work in progress. Obviously, this limits a little the usefulness of data virtualization in agile-BI development in the near future; but it is also an opportunity, in those enterprises that qualify, to kick-start the “global metadata repository” effort into high gear. Slap a data virtualization engine and development tool like CIS with Composite Studio into the agile-BI development team’s toolset, and have them scream to you about needing it to cover more data. That will get corporate’s attention, fast.

For those organizations that haven’t used data virtualization before, especially the SMBs, using data virtualization on things like Big Data is an excellent opportunity to avoid the arrows that the large-enterprise data-virtualization pioneer users faced. More specifically, if you start out by federating key customer Big Data with ERP on the cloud, private or public, you have already created a repository that covers most of your likely business-critical data in the next 2-3 years, and so you will not have to retrofit it later on.

A third, often underestimated, use of data virtualization is as a tool to support the new “business analysts” on the corporate side who are also doing agile analytics. A recent survey cited by Sloan Management Review suggests that these now view IT as slower and lower than dirt. Give them and support a tool that allows them to go out and discover new data sources, and I would not be surprised if they responded, like the woman conducting a choir in the movie Love Story, with “That is absolutely, stupendously, incredibly – OK.” That is, they won’t kiss you, but they will make a big shift towards tolerating you.

However, the use of CIS-type data virtualization that is dearest to my heart, and should be dearest to yours if you ever want to accomplish business agility rather than just BI “agility,” is applying its discovery features constantly, out on the Web, embedded in your analytics applications, to find and bring in-house new data and new data types. One key finding of business surveys is that only half of respondents find out about new key data heralding new market trends on the Web in less than half a year. Data virtualization can cut that to a day. You see, business agility isn't about reacting quickly and effectively to changes in your environment that you happen to find out about because they show up eventually in your customer complaints; the critical success factor is finding out all the changes that you can, as they happen out there, and then reacting quickly and effectively.

The Bottom Line for IT Buyers

Over the next few years, IT buyers should be viewing the acquisition and use of data virtualization tools in agile BI, among other things, as a no-brainer – but they don’t. Those smart IT buyers who do, however, will find that a pre-short-list is not necessary: on closer examination, the use cases almost present themselves. At any rate, that has been users’ experience in data virtualization’s original core markets such as the armed forces in the government – another Composite installed base.
However, for those constrained by budgets even in their strategic analytics initiatives, Composite and others may well have to go on a “pre-short list.” This should be of a peculiar type, however. The IT buyer should not passively wait for corporate to give IT a mandate that requires data virtualization. Instead, the IT buyer should be actively looking for the opportunity to introduce or extend agile-BI data virtualization, like the politician looking in every budget process or special spending initiative for a way of cutting taxes.

Composite Software is one among several who should be on that short list, as well as other data-virtualization-focused firms like Denodo and Tibco, not to mention the data federation solutions of IBM, Oracle via its BEA WebLogic Liquid Data acquisition, Red Hat/Metamatrix, and SAP/Sybase. Composite Software’s edge over most of these is simply that, over the last 10 years, Composite has established its experience and a leadership role in features and services useful to agile BI, and is among the most vocal about their plans to support Big Data and agile BI. Since that leadership role hasn’t diminished over the last three years, Composite Software will probably maintain its position at or near the top of agile-BI data virtualization short lists over the next 2-3 years, as well.

The bottom line for IT buyers of Other agile-BI data virtualization in general, and Composite Software’s CIS and Composite Studio in particular, is this: buy it right now or buy it a year from now, use it for BI agility or use it for business agility, but buy it and use it for some agile-BI initiative sometime in the next 2-3 years. Like an agile development methodology, it looks like it might cost more and cut into the bottom line, but it repeatedly costs less and delivers more revenue – because, used right, it improves your information agility.

The Other BI: EMC Greenplum and Embedded Analytics

2012-01-05T16:43:00.000-08:00

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about BI today.

The Importance of the Greenplum Software Technology to BI

I am stretching a point when I say that EMC’s Greenplum is not “top of the mind” today. EMC has done an extensive and effective job of marketing Greenplum’s virtues in dealing with Big Data. However, what I am talking about here is embedded analytics – and there, neither Greenplum nor any other vendor solution is “top of the mind” with IT today.

More specifically, I am talking about middle-tier analytics, the area that most embedded analytics will aim for in the next three years. This is not massive-data-store, in-depth-analytics BI like the data warehouse; nor is it the “smart sensor,” small-form-factor analytics that will increasingly come to the fore with the arrival of the sensor-driven Web (e.g., analytics on your iPhone). No, I am talking about medium-sized data stores, moderately in-depth analytics, and in-enterprise BI applied at the level of the department, local office, loosely-coupled storage array, or server network. This analytics does best when it is embedded in other software or in firmware, and operates semi-automatically to pick up business-process flows and alert the business before they get out of whack, or offloads load balancing from a central server. Unlike systems management software, embedded analytics not only monitors and “fixes” but also analyzes what is going on, and reports this analysis either to the top-tier data warehouse or a specific set of software, end users, and/or administrators.

Up to now, the fledgling beginnings of embedded analytics have begun to show up in the systems management software of folks like CA; but they are not separable pieces usable by other distributed software. Increasingly, the major vendors like IBM are now talking about taking analytic software from BI and analytics software suites and applying it to organization operations across the board.

However, these often involve databases retrofitted to BI in general and decision support in particular. What Greenplum represents is the obvious next step: applying a database designed from the ground up and optimized for querying and analytics. The point is that these will inevitably be better suited than data management approaches intended to handle updates as well as queries and result massaging.

This is not to say that an embedded analytics database is the end point of embedded-analytics evolution. Because most if not all available analytics databases were designed for the top tier, they are too “heavyweight” for their intended purpose: they perform more slowly, because they are tuned for much higher data-store sizes. However, whether the next turn of the market crowns a slimmed-down top-tier database or a new ground-up-designed middle-tier analytics database as the winner, either one will really do.

Over the next 2-3 years, it is reasonable for IT buyers to expect some of this technology to arrive on their doorsteps embedded in upgrades of existing solutions – but far from all of it. At some point in this period, separable analytics solutions will show up that will allow the user to go far beyond what a particular vendor is offering – if, of course, IT wants to.

Why would IT want to do this? Answer: to handle areas in which one-size-fits-all vendors are simply not moving fast enough. Take, for example, carbon accounting. Vendors have been very proactive in this area, but some of the market is moving faster still, towards monitoring that picks up on and alerts to excess emissions as they happen, and connects with the carbon accounting software when necessary. Likewise, as health care providers grapple with government mandates and Electronic Health Records, they can see coming a day in which they will need to perform damage control on breaches of privacy; but today’s tools are much slower than they could be to detect such a problem. In either case, customizable middle-tier embedded analytics that goes beyond most likely vendor offerings is needed.

The primary organization benefit of this technology, therefore, is deeper real-time understanding of in-enterprise problems that leads to better decision-making –a very cost-effective application of analytics’ general ability to improve gross margins. Embedded analytics via an analytics-adapted database may take longer to arrive than most of the Other BI that I talk about, but its advent and benefits are just as sure.

The Relevance of EMC to BI

While EMC has continued its tradition of “hands off the technology, add our markets” in the Greenplum acquisition, it has also continued another tradition: adding the technology where appropriate to its core storage software/firmware. That is, according to EMC (and I see no reason to doubt them), Greenplum technology is being put in storage controllers to offload querying from the server to the storage array. Obviously, that has a major positive implication for storage and large-BI performance. Less appreciated is the fact that this embedding of Greenplum requires that it “slim down” into a form that can operate not only on storage but also, in a middle-tier fashion, on loosely-coupled LANs serving local offices, departments, and so on. In other words, embedding on storage should mean that embedding on all other middle-tier form factors is within reach. And the acquisition of Greenplum also should mean that EMC is finally beginning to add database and BI smarts to its DNA, ensuring reasonable long-term service and support for its embedded-analytics solutions.

EMC’s market strength and apparent relative freedom from threat in the scale-out market mean that in the 2-3 year time frame I am talking about, and probably in the medium term as well, Greenplum is in no danger of going away. No, the real question for IT buyers of embedded analytics is whether EMC will have Greenplum take the next step, abstracting its slimmed-down form for embedded analytics on all vendor platforms. I can offer no guarantees of this, since it is not apparent that EMC has done such a thing before. All I can say is, if they do so, at least some sort of market will be there.

Potential Uses of Greenplum-Type Analytics for IT

It is time to point out that embedded-analytics technology is unusual in that vendors have relative freedom to delay delivering, say, multivendor or open-source middle-tier analytical databases, since it’s not high on IT wish lists. It could happen next week, or it could happen 3 years from now. So any IT acquisition of, and use of, this kind of embedded analytics will just have to wait until the vendors get around to it.

At that point, the obvious application is per-project – improving a specific business process or case-management implementation. More than other technologies, embedded analytics does not require full, integrated organizational implementation to be maximally effective. Rather, it does just fine applied to a task, a process, a function, a locality, or a local or strategic initiative. IT simply looks down the list of mission-critical projects and picks the one that benefits most from risk management or analytical automation.

The critical success factor in such projects is rapid implementation and upgrade, caused by automation of the implementation/upgrade process, allowing strategic projects a head start. Right now, while most vendors do well at this, high-end vendors like EMC seem to be setting the pace. And so, choosing EMC Greenplum (assuming it fits) in all likelihood means a better chance of rapid implementation and a database better fitted to a broad range of embedded-analytics tasks – not to mention better ongoing support for tricky cases.

The Bottom Line for IT Buyers

The IT buyer should view embedded analytics as a technology that may take a while to materialize. However, when it does, Greenplum-type embedded analytics will deliver analytics-type benefits at least equal to the whiz-bang high-end analytics now being sold – although those benefits will arrive in smaller per-project chunks. And that, in turn, means that this technology is definitely worth the IT buyer’s ongoing attention.

More specifically, the IT buyer might consider a “pre-pre-short-list” type of approach. That would involve identifying solutions such as EMC Greenplum that may wind up as part of the embedded-analytics short list in the next 2 years, and steadily moving those products in the pre-pre list over to the “pre-short list” as their technology reaches the point of usefulness (that is, it can be applied by IT rather than being embedded in another vendor solution, and it’s optimized for middle-tier analytics). Today, I would say that it appears Greenplum is probably among the closest to that take-off point. So, put it on the pre-pre short list, and get ready to put it on the short list. If everything goes right, and your CEO hits you with an urgent requirement that really demands embedded analytics, you will definitely be glad you had EMC’s Greenplum embedded analytics solution in your back pocket.

The Other BI: Progress Apama and Event Processing

2012-01-03T14:00:00.000-08:00

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about BI today.

The Importance of the Apama Software Technology to BI

The value-add of Apama to BI, in my opinion, is the value-add of applying analytics to “data in motion” on a very broad range of data. Apama carries out “event processing”: conceptually, I think of event processing as a “processing head” monitoring an Enterprise Service Bus (ESB). Most data entering the organization, as well as data moving between data stores and between users within the organization, is wrapped up as messages and sent across the ESB to its destination. In the process, the “processing head” monitors the whole stream of data-in-motion and performs analytics, alerting, and other processing based on the type of data being reviewed (or, in the aggregate, the “pattern” of a stream of related data). What is unprecedented about this kind of data processing is that (a) it focuses on data across organizational units, unlike the typical data warehouse or operational database; (b) it intercepts some data the moment it arrives in the organization, which is the ultimate in real-time business-critical data processing; and (c) it can draw a direct line between that data and a decision-maker by alerting, so that business-critical decisions can be made as quickly as possible.

Practically, of course, an “event processor” can do (c) only for a certain small subset of the information in the organization, because by itself an event-processing database does not scale nearly as much as a data warehouse. The event processor has much less historical “context” as it processes each datum, because it simply does not have the time to perform a “query from hell” on terabytes of historical data before the next datum must be processed. In-depth analytics will simply have to wait, often for an hour or more. Nevertheless, this kind of instantaneous response is, in the real world, enormously valuable when fast response to the type of events that the event processor detects from the data is indeed mission-critical and/or business-critical.

At the same time, (a) – the ability to correlate data across organizational units – is an often-underestimated value of the event processor. As the discipline of systems analysis understands, a collection of business units is as much a set of process flows between units as a set of stand-alone companies. The job of corporate is often to ensure that these process flows work well, and the value-add of the event processor in this case is to provide enterprise performance management (EPM) that reads the tea leaves of particular process flows and ties them back to glitches in the performance of the units and their coordination. In plain English, a good event processor goes beyond what you could do before because it lets you respond immediately to some new threats and opportunities in your environment, and because it tells you some of the things that are really happening to muck up your business’ overall performance as it coordinates business units. If you combine the two, you get the famous “360-degree view” inside and outside the organization.

Apama, like other event processors, never operates in a vacuum. All organizations already have operational and decision-support databases supporting key applications, and event processing must adapt itself to handle what these do not. Therefore, Apama’s value-add within the limits of (a)-(c) above can vary quite widely. Always, however, if the user does a careful analysis of the most important decisions that need to be speeded up and the gaps in business performance information, the BI done by an event processor like Apama has a major impact on the organization – not on its bottom line, necessarily, but always on its business risk. The out-of-the-blue event that businesses always face becomes much less risky when an event processor manages to detect it in a timely fashion.

Right now, businesses of all sizes are still in the early stages of use of event processing – you can tell, because case studies typically trumpet particular per-project uses. Therefore, the field is wide open for approaches such as the “event-driven architecture”, in which an event processor on top of an ESB becomes the focal point at which corporate can not only monitor but also direct information flows; the “complex event processor”, in which on-the-fly analytics becomes far deeper; and “data streaming”, in which the whole notion of a data-warehouse database is upended to be a “processing head” handling querying on multiple parallel “streams” of XML-type data. These, however, will often not achieve full implementation until 2-4 years from now, at the least. The key value-add of event processing in real-world BI over the next 2-3 years, I believe, will be the ongoing identification and implementation of the most important alerts, decisions, and business-unit correlations that it can handle, and their integration with the existing BI architecture.

The Relevance of Progress Software to BI
The relevance of Progress Software itself to BI, and to data processing in general, is less clear than in its “glory days,” when (imho) it was a pioneer in near-lights-out database administration, rapid application development, Software as a Service (SaaS), and the ESB. In those days, it had a gift for identifying the innovative infrastructure-software simplification that led its SMB/departmental customers to rapid implementation of the latest large-enterprise functionality, and beyond. Apama arrived at the end of that period, as part of a successful Progress effort to provide services and software to allow its departmental customers to scale their SMB-type technology to the division, the line of business, and even the “edge” in the data center. Apama first found its niche in rapid analysis of massive streaming financial-market data; Progress Software appears, with its Event Manager, Event Modeler process-design end-user tool, SmartBlocks, and Dashboard Studio, to have added Progress’ own strengths in SMB-driven simplicity of use. To put it another way: Apama was born large-enterprise-ready; Progress added the veneer and tools that makes it fully in sync with open-source or agile BI as applied, say, to Big Data.

Thus, five years ago Progress Software would have been seen as an operational and decision-support database for an SMB, and a fast-moving local-level operational adjunct to enterprise BI in the Global 1000. Today, Progress Software has much less visibility in BI, and its connection to the latest BI technology is less visible; but if you look at the actual technology, Apama is indeed innovative and can deliver value-add across a wide range of enterprises. It only remains to ask, what’s the future of Progress Software as a supplier of event processing technology, and in general?

There are two parts to my answer. First, let me note the characteristic that Progress Software shares with just about every database company: it has a core loyal base of customers whose size may shrink, but whose tendency not to migrate away from the platform ensures that Progress Software will be extraordinarily long-lived. As in the past few years, database revenues may ebb over time; but few if any database companies in my 30 years of acquaintanceship with the industry see a massive collapse of their installed base. In the next 2-3 years, as sure as the sun rises, reasonable management plus this revenue flow will see Progress Software still standing (acquired or not) and still supporting Apama plus the infrastructure software like the Progress ESB that complements Apama.

However, there is little in the past three years, where revenues have been essentially flat, to suggest that Progress Software’s glory days will return, and that it will identify another new infrastructure technology to get strong growth started again. Moreover, Progress’ strength has never been that its solutions were a key part of the mainstream of Web innovation, and so there is no obvious reason to expect that Apama can, at a minimum, transition to form the core of cutting-edge open-source BI solutions. And that brings me to the second part of my answer.

I assert that these caveats almost certainly do not matter, in the next 2-3 years and probably further out. The reason is that Progress Software’s DNA may not be Web or open-source, but it is very definitely SMB-simple and flexible, with no vendor lock-in. Apama pre-acquisition would probably be a niche financial-market BI product. Apama plus Progress Software is a uniquely flexible and easy-to-use event processor that integrates with the rest of your architecture just fine, and it will stay that way. Progress Software’s new services prowess just ensures that the simplicity scales up to the largest of enterprises.

Potential Uses of DataRush for IT
The net of the contributions of both Apama technology and Progress Software’s “approach” to the Progress Apama solution is that, whether you are an SMB tackling BI for the first time or a large enterprise trying to become more agile in your querying of Big Data, Apama offers differentiated value-add, either stand-alone or as the complement to a large-vendor event processor and database architecture like IBM Streams and IBM Information Server. In the case of an SMB, the use case is straightforward: you can fit Apama with agile-BI efforts as a front end to catch urgent alerts implicit in Big Data, raw operational data, or ongoing analytics; and you can begin to develop EPM. Large enterprises typically have bottom-up Windows/desktop computing in parallel with the massive datacenter data warehouse: they can grow Apama with the evolution of that side of the enterprise architecture, while if appropriate also driving forward their top-down, datacenter-driven event processing projects using other vendors’ event-processing solutions, and easily integrating the two.

In either use case, the key to the most rapid possible success (I think) will be to identify on an ongoing basis the key targets for alerting and rapid decision-making, and to tie Apama as much as possible to historical data in order to allow the greatest “depth” of analytics at the point of data intercept. One good detection of a major customer about to dump you and immediate, effective reaction to prevent it will make the whole exercise more than worthwhile. And, remember, we are talking low-touch, very-low-TCO event processing here.

It is unusually hard to think of things that can go wrong with Progress Apama implementation. Services? Not as much needed, and Progress Software services that may be needed are already real-world-proven from more than 20 years of rave reviews from SMB and departmental clients, plus 5 years of driving similar technology into the division and LOB. Training? Again, the Progress Software track record is that even an untrained local-office manager can handle database maintenance – which is usually the biggest concern – and any developer can handle the drag-and-drop development tools. Integration? Progress Software has what I view as the standard set of adapters and gateways. Limits to scalability? Not if the Progress ESB is any guide. Let’s face it, IT implementation of Apama is very unlikely to be rocket science – and neither is gaining BI insights with it.

The Bottom Line for IT Buyers
At this point, an IT buyer should view Progress Software’s Apama as roughly equivalent to a “diamond in the rough.” It is the center of attention neither of BI, nor streaming technology, nor even sometimes of Progress Software itself. It suffers undeservedly from questions about Progress Software’s future, the future of event processing technology, whether Apama will continue to track BI technology, and a reputation as a high-end or specialized event processor. All it has going for it is that it is a more simple, more flexible, powerful event processing tool for a wide variety of use cases and scales, and should continue to deliver for the next 3-5 years, and almost certainly longer. And that should be plenty.

I have noted that Apama’s (and event processing’s) main value-add in BI is more in the risk area than in the top or bottom line (although some implementations, like EPM, do indeed impact revenues and costs). However, this is one technology that, when it succeeds, is really, really visible. Sell it to corporate as the latest technology fad if you like; the odds are that when the IT buyer acquires and then IT implements Apama, a big success story happens in the next year. And then you can concentrate on what’s really important: integration with the rest of your BI so that it all works optimally, in harmony.

The net-net for IT buyers, therefore, is to do a “reality check” on present-day event processing in BI, and then prepare a short list of event-processing software vendors to take the next step. I see no reason why, in most if not all cases, Progress Software’s Apama should not be on that Other BI “pre-short list.”

The Other BI: Pervasive DataRush and Parallel Data Streaming

2012-01-02T13:17:00.000-08:00

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not yet typically “top of the mind” when we talk about BI today.

The Importance of the DataRush Software Technology to BI
The basic idea of DataRush, as I understand it, is to superimpose a “parallel dataflow” model on top of typical data management code, in order to improve the performance (and therefore scalability) of the data-processing operations used by typical large-scale applications. Right now, your processing in general and your BI querying in particular are typically done either by “query optimization” within a “database engine” that takes one stream of “basic” instructions and parallelizes it by figuring out (more or less) how to run each step in parallel on separate chunks of data, or by programmer code that attempts a wide array of strategies for speeding things up further, ranging from “delayed consistency” (in cases where lots of updates are also happening) to optimization for the special case of unstructured data (e.g., files consisting of videos or pictures). “Parallel dataflow” instead requires that particular types of querying/updates be separated into multiple streams depending on the type of operation. This is done up front, as a specification by a programmer of a dataflow “model” that applies across all applications with the same types of operation.

There is good reason to believe, as I do, that this approach can yield major, ongoing performance improvements in a wide variety of BI areas. In the first place, the approach should deliver performance improvements over and beyond existing engines and special-case solutions, and not force you into supporting yet another alternate technology path. The idea of dataflow is not new, but for various historical reasons this variant has not been the primary focus of today’s database engines, and so the job of retrofitting to support “parallel dataflow” is nowhere near completion in most database engines. That means that, potentially, using “parallel dataflow” on top of these engines can squeeze out additional parallelism, due to the increased number and sophistication of the streams, especially on massively parallel architectures such as today’s multicore-chip server farms.

At the same time, the increasing importance of unstructured and semi-structured data has created something of a “green field” in processing this data, especially in areas such as health care’s handling of CAT scans, vendors streaming video over the Web, and everyone querying social-media Big Data. Where existing data-processing techniques are not set in concrete, “parallel dataflow” is very likely to yield outsized performance gains when applied, because it operates at a greater level of abstraction than most database engines and special-case file handlers like Hadoop/MapReduce, and so can be customized more effectively to new data transaction mixes and data types.

There is always a caveat in dealing with “new” software technologies that are really an evolution of techniques whose time has come. In this case, the caveat concerns the fact that, as noted, programmers or system designers need to specify the dataflows, rather than the database engine, and this dataflow “model” is not a general case for all data processing. That, in turn, means that at least some programmers need to understand dataflows on an ongoing basis.

It is my guess that this is a task that users of “parallel dataflow” and DataRush should embrace. There is a direct analogy here between agile development and DataRush-based development. The usefulness of agile development lies not only in the immediate speedup of application development, but also in the way that agile development methodologies embed end-user knowledge in the development organization, with all sorts of positive follow-on effects on the organization as a whole. In the same way, setting up dataflows for a particular application leads typically to a new way of thinking about applications as dataflows, and that improves the quality and often the performance of every application that the organization handles, whether it is optimizable by “parallel dataflow” or not.

In other words, in my opinion, developers’ knowledge of data-driven programming is increasingly inadequate in many cases. Automating this programming in the database engine and user interface can only do so much to make up for the lack. It is more than worth the pain of additional ongoing dataflow programming to reintroduce the skill of programming based on a data “model” to today’s generation of developers.

The Relevance of Pervasive Software to BI
Let me state my conclusion up front: I view investment in Pervasive Software’s DataRush technology as every bit as safe as investment in an IBM or Oracle product. Why do I say this?

Let’s start with Pervasive Software’s “DNA.” Originally, more than 15 years ago, I ran across Pervasive Software as a spin-off of Novell’s Windows database of the 1980s. Over time, as databases almost always do, the solution that has become Pervasive PSQL has provided a stable source of ongoing revenue. More importantly, it has centered Pervasive Software from the very start in Windows, PC-server, and distributed database technologies servicing the SMB/large-enterprise-department market. In other words, Pervasive has demonstrated over 15 years of ups and downs that it is nowhere near failure, and that it knows the world even of the Windows/PC-server side of the Global 10,000 quite well.

At the same time, having followed the SMB/departmental market (and especially the database side) for more than 15 years, I am struck by the degree to which, now, software technologies move “bottom-up” from that market to the large enterprise market. Software as a Service, the cloud, and now some of the latest capabilities in self-service and agile BI are all taking their cue from SMB-style operations and technologies. Thus, in the Big Data market in particular and in data management in general, Pervasive is one leading-edge vendor well in tune with an overall movement of SMB-style open-source and other solutions centered around the cloud and Web data.

I therefore see the risks of Pervasive Software DataRush vendor lock-in and technology irrelevance over the next few years as minimal. And, of course, participation in the cloud open-source “movement” means crowd-sourced support as effective as IT’s existing open-source software product support.

Aren’t there any risks? Well, yes, in my opinion, there are the product risks of any technology, i.e., that technology will evolve to the point where “parallel dataflow” or its equivalent is better integrated into another company’s product. However, if that happens, dollars to doughnuts there will be a straightforward path from a DataRush dataflow model to that product’s data-processing engine – because the open-source market, at the very least, will provide it.

Potential Uses of DataRush for IT

The obvious immediate uses of DataRush in IT are, as Pervasive Software has pointed out, in Big Data querying and pharmaceutical-company grid searches. In the case of Big Data, DataRush front-ending Hadoop for both public and hybrid clouds is an interesting way to both reduce the number of instances of “eventual consistency” turning into “never consistent” and to increase the depth of analytics by allowing a greater amount of Big Data to be processed in a given length of time, either on-site at the social-media sites or in-house as part of handling the “fire hose” of just-arrived Big Data from the public cloud.

However, I don’t view these as the most important use cases for IT to keep an eye on. Ideally, IT could infuse the entire Windows/PC-server part of its enterprise architecture with “parallel dataflow” smarts, for a semi-automatic ongoing data-processing performance boost. Failing that, IT should target the Windows/small-server information handling in which increased depth of analytics of near-real-time data is of most importance – e.g., agile BI in general.

These suggestions come with the usual caveats. This technology is more likely than most to require initial experimentation by internal R&D types, and some programmer training, as well. Finding the initial project with the best immediate value-add is probably not going to be as straightforward as in some other cases, as the exact performance benefit of this technology for any kind of database architecture is apparently not yet fully predictable. Effectively, these caveats say: if you don’t have the IT depth or spare cash to experiment, just point the technology at a nagging BI problem and odds are very good that it’ll pay off – but it may not be a home run the first time out.

The Bottom Line for IT Buyers
Really, Pervasive DataRush is one among several performance-enhancing approaches that offer potential additional analytical power in the next few years, and so if IT passes this one up and opts for another, they may well keep pace with the majority of their peers. However, in an environment that most CEOs seem to agree is unusually uncertain, out-performing the majority, and extreme IT smarts in order to do so, are more frequently becoming necessary. At the least, therefore, IT buyers in medium-sized and large organizations should keep Pervasive DataRush ready to insert in appropriate short lists over the next two years. Preferably, they should also start the due diligence now.

The key to getting the maximum out of DataRush, I think, will be to do some hard thinking about how one’s BI and data-processing applications “group” into dataflow types. Pervasive Software, I am sure, can help, but you also need to customize for the particular characteristics of your industry and business. Doing that near the beginning will make extension of DataRush’s performance benefits to all kinds of existing applications far quicker, and thus will deliver far wider-spread analytical depth to your BI.

How will a solution like DataRush impact the organization’s bottom line? The same as any increase in the depth of real-time analysis – and right now that means that, over time, it will improve the bottom line substantially. For that reason, at the very least, Pervasive Software’s DataRush is an Other BI solution that is worth the IT buyer’s attention.

Methane Talk-Down: Partial

2011-12-28T17:18:00.000-08:00

One of the true joys of learning about science – as opposed to, say, economics – is that eventually you can usually get to a scientific summary that clears up many of the distortions that popular reports create. In the midst of wading through yet another cherry-picked-evidence blog post (this one on methane) by Andrew Revkin of the NY Times, it suddenly occurred to me that I should check out Justin Gillis of the Times, whose posts have been praised iirc by Joe Romm of climateprogress fame. Gillis’ reporting still seemed a little superficial to me, but he had a link to a 2006 scientific summary of the research about methane and climate change, an oldie but goodie where I found the answers to many of my questions. My recent blog post on methane laid out the doomsday scenario that I fear; Chapter 6 of this summary, as Rachel Maddow would say, talked me down – but only partially.

Because the broad scenario that I laid out is not drastically affected by the information in the summary, it is easier to lay out the summary’s picture of methane and then, at the end, note how this may affect my scenario. I will focus on methane clathrates, since the changes to everything else are less substantial. And, of course, I am sure that more misconceptions remain – because a summary article of ongoing research can’t be expected to answer everything. Anyway, let’s begin.

Methane Clathrates, Water Methane

Last time, I presented a very summarized picture of natural-source methane as coming from three sources: methane clathrates under the sea, permafrost on land at high latitudes, and peat bogs next to the permafrost or in the tropics. It turns out that the picture is a bit more complicated, and the complications matter.

To start with, methane clathrates are formed and remain stable in sea-floor sediment in particular combinations of sea temperature and pressure from the sea above that limit them to the sea floor somewhere between 200 meters and 1000 meters below sea level. In other words, the water has to be near zero F, and the clathrate has to be lower than 200 meters below sea level and higher than 1000 meters below sea level. Between those two limits, the deeper the sea floor, the wider the zone in the sediment where it can exist. Guesstimates for a typical clathrate “stability zone depth” might be 250-300 meters. Btw, a confusing part of the scientific lingo apparently refers to Arctic clathrates as “subsea permafrost.”

What happens to melt the clathrates? The water next to the sea floor warms up, or warmer temps further up the sea slope cause the equivalent of a mudslide on the sea floor that basically slices through the clathrate, stirs up everything above the slice as a cloud of sediment, and melts all the clathrate above the new sea floor. That is what they think happened at Storegg, a place near Norway where there is a “crater” 30 km across that may have released a gigaton of carbon, all at once (methane is CH4).

Now here’s an odd part. We are used to thinking of gas coming up to the surface in bubbles and releasing itself into the atmosphere when the bubble pops. Not so with clathrate methane – most bubbles pop long before they rise the 200 meters or more to the surface, according to the models. Instead, one of several things happens: the methane rises to the surface but not as bubbles (it is “buoyant”) and then releases into the atmosphere, or it is eaten by methane-eating bacteria, or it converts (typically to carbon dioxide) en route. Initial indications are that a small percentage of melted clathrate should rise to the surface combined with water and is released into the atmosphere as methane, which happens effectively immediately; a large percentage should be eaten by bacteria, who convert it into carbon dioxide on the surface of the sea, and the carbon dioxide is released into the atmosphere in order to equalize atmospheric and oceanic CO2; and a medium-sized percentage should convert to carbon dioxide without going through the bacteria, to be released into the atmosphere as carbon dioxide in the same way.

The methane clathrates in the Arctic seas contain perhaps 50%-80% of all clathrates. They are also by far the most likely to be affected by global warming, since water temperature variation due to increased sunlight on the water and increased temps of sun-warmed currents from the south are widest there.

Other Methane Sources

The picture of land-based methane sources also needs amendment. It appears that much of the methane stored in permafrost is stored in peat within the permafrost – which can extend as far down as 200 meters or so. Meanwhile, wetlands at whatever latitude are generators of methane, the Amazon as much as Ireland. When the permafrost melts, the water plus peat turns into a bog that (under global warming) is maintained by increased precipitation: that’s what often drives increased methane production.

Here, the translation to the atmosphere is more clear. Melting of permafrost releases any methane locked in the ice (but not in clathrates), and also creates new constantly-emitting sources of methane. Likewise, wetlands inject methane directly into the atmosphere.

Now we come to the tricky part. We are accustomed to thinking of methane in the atmosphere as separate from carbon dioxide. Not so. What often happens to methane in the atmosphere is that it "oxidizes”, which typically means that one of the hydrogen atoms is broken off to help form H2O (water), while the rest forms a methyl group (CH3) which eventually breaks down to carbon dioxide. In other words, much of the methane tossed into the atmosphere actually winds up as the major greenhouse gas, and stays up there for 150-250 years.

What’s the Effect? Um …

OK, so now the scientist wants to figure out what the global-warming effect of unlocking all that methane is going to be. The problem is that we have two sources of comparison, and neither of them is great.

The first is to use what happens over 10-20,000 years immediately after a Milankovitch-cycle minimum (a “glaciation”) as a model. Using that model, scientists have pretty well determined that in such times of rising global temperatures, the amount of methane in the atmosphere probably doesn’t vary by a heck of a lot, and the effects on global temps compared to atmospheric carbon are pretty minimal. Methane melt in general might have a role in things like sea-ice melting near Greenland, which has been shown to have surprisingly wide effects on global climate, but most of the good candidates for that type of melt (subsea, permafrost, wetlands) just don’t make a strong case for themselves.

The problem with this type of analysis is that it looks only at periods when most of the ice remains – because that’s what happens at the peak temps of a Milankovitch cycle. We have almost certainly moved above those peak temps in the last couple of decades, and so we are in much less charted waters. For a period much more comparable, you have to go back to the PETM – 55 million years ago.

OK, in the PETM, temps were 5-10 degrees C warmer than now. Increases in carbon in the atmosphere just don’t seem to be enough to justify those warmer temps. So for a while, there were theories floating around that methane was the complete reason for that kind of warming – no carbon needed. That would have been nice, since figuring out why carbon suddenly spiked in the first place, not to mention why the time period of this rapid warming was around 20,000 years as the latest research suggests, has been a headache. Bad news: there simply doesn’t seem to be a natural source of methane that comes near to explaining the whole temperature rise, not to mention keeping going for 20,000 years. So it looks like we have a choice between carbon emissions plus “unknown”, and carbon plus methane. Tentatively, the scientists are voting for carbon plus methane.

But the PETM isn’t great as a model, either. The problem there is that things happened slowly compared to today. If we say that the carbon atmospheric-concentration rise then happened over the course of 20,000 years, well, our carbon rise appears to be happening over 350 years – and it may very well double the rise of the PETM over the course of those 350 years. In other words, this is happening at least a hundred times faster. And, as we’ve seen in the case of carbon, that can mean that the positive follow-on effects happen well before the negative “stabilizer” effects. So, for example, don’t necessarily expect the magical munching methane sea bacteria to appear in the Arctic and save the day.

OK, so the models we have aren’t great. Can we at least use them for some guesstimates?

Preliminary Guesses

Well, the scientists have done the guessing for me. The key sentences I find in Chapter 6 say, more or less (with the usual caveats about my understanding), that the amount of atmospheric methane from natural sources pre-Industrial Revolution equals the amount of methane added from human sources since then, which equals the likely amount of methane to be added at some point due to all natural sources except subsea methane, which equals the potential amount of methane from subsea methane. In other words, in a worst-case scenario with 2006 models, at some point in the next 300 years, we might expect atmospheric methane four times what it was in 1850.

How much added heating would that translate to? Again, reading between the lines, perhaps 1 degree C from the methane alone. However, if we take the PETM as a model, it might be more like 2 degrees C. And that’s the maximum, so we can all semi-relax, right?

Well, no. You see, there are two problems. First of all there’s the fact that much of that methane is going to convert to carbon dioxide when it’s up there. Second, there’s the fact that the more methane gets into the atmosphere from now on, the longer it sits there. The 2006 estimate was that methane hangs up in the atmosphere an average of 9 years. But at twice the concentration, I think we can count on it sitting up there for 12-18 years on average. So those two things should add another ½-1 degrees C to the “additive effect” of methane in the atmosphere.

And then, of course, there’s the question of methane that converts to carbon dioxide before it gets into the atmosphere. Here, the summary didn’t really have much to add in the long term. Even by their time-frame estimates, all that methane-to-carbon-dioxide, even if it doesn’t get there in the next 100 years, will almost certainly show up in the next 1000 years. So it’s a more extreme version of my “pay me now or pay me later” scenario – except that we can at least hope that by the time the methane-turned-carbon-dioxide shows up, we will have managed to cut down on our human-caused carbon emissions and the amount in the atmosphere will have begun to go downhill significantly.

All in all, not great, but not as bad as my full doomsday scenario. Instead of 6 degrees C from methane-turned-CO2, perhaps 2-3, although that increase will stick around for maybe twice as long; instead of 7-9 degrees C from methane-stayed-methane over the next 160 years, perhaps 1-2 degrees over the next 300-500 years. And it will happen more gradually, so it won’t be really noticeable, probably, for the next 30-40 years. Except …

I’m Not All the Way Down

Read carefully the interview with the head of the survey of methane releases in the Siberian Sea. He states, effectively, that the diameter of the “craters” I referred to earlier had increased by up to 100 times this year, and this methane was “bubbling to the surface.” If you look at the 2006 summary, neither is supposed to happen. Very little methane should arise to the surface in a bubble, as noted above, and the methane hydrates should not suddenly do a big jump in melting: and 5 degrees C increase in water temps (since 1984, a jump of 2.1 degrees C has been observed) should cause perhaps 1 meter’s worth or less of methane hydrates to melt over the next 40-80 years – and it can’t be explained as mudslides, since it has happened in quite a few places.

So why would scientists’ models be wrong? Well, in the first place, they assume that relative sea-water warming will only occur in a short space in the summer, when the ice is melted and the sun warms its top. However, the depth of the surface ice in winter is also less than before, and the water carried by currents from the south is warmer. Clearly, it’s very possible that scientists are underestimating the amount of melting going on the rest of the year. Add this to the known problems with the original model developed in 1995, and you have some, but maybe not all, of the increase in clathrate atmospheric methane release explained.

The second flaw may be the modeled prediction that very little methane melt will rise to the surface as bubbles. Why might this model be wrong? I don’t have a clear answer from the summary – it could be that the turbulence of the water keeps the bubble from popping, although that seems unlikely. One thing seems clear: the magical munching methane bacteria are nowhere to be seen.

And the third flaw, which also affects the land methane emissions rate, is a major underestimate in the models of the rate of global warming. The models implicitly assume that the Arctic sea ice won’t melt entirely in summer before somewhere between 2030 and 2100, and year-round perhaps never – that one seems clearly wrong. Therefore, they underestimate the speed of the follow-on effects, including much faster warming of water within 100 meters of the surface, which would inevitably mean much faster warming at the 200-500 meter level – sorry, that’s not “deep ocean.”

In other words, what the latest information is telling us is that the semi-comforting story I just gave you is almost certainly an underestimate. The “true” effect of methane is somewhere between my doomsday estimate and the one above – except that the roles of methane-stayed-methane and methane-turned-carbon-dioxide have switched, because we now know that much of that atmospheric methane is going to change to carbon dioxide while it’s up there.

I find the logic of the summary convincing as well as semi-comforting; so if I had to guess, I would say that the net effect is somewhere between 3-5 degrees C, mainly in carbon dioxide, and spiking over the next 40-150 years before leveling off. But that’s a complete guess. Until I understand just how the models went wrong, I’m only partially talked down from my panic. So here’s to the New Year: It will be a season of hope, it will be a season of despair, it will be a season of enormous impatience until the first scientific explanations come out.

Methane: The Final Shoe

2011-12-21T16:00:00.000-08:00

Recently, Neven’s blog on Arctic sea ice (neven1.typepad.com) featured a new post on recent scientific observations of methane – observations that Neven said made him “sick to my stomach.” I am not as easily panicked – I reserved my stomach sickness for a recent British report about how most life in the oceans, except jellyfish, will be dead within the century unless we do far more than we are doing. However, I do understand his reaction. Effectively, these reports indicate that the dreaded “final shoe” of global warming, the one reinforcing side-effect of global warming that we hoped against hope would not happen, appears to be partially beginning to drop. Moreover, it seems clear to me that most if not all folks, even those who are aware of methane’s role in climate, are underestimating its potential impact in causing additional and more rapid murder, disaster, and then catastrophe.

So here is my understanding of methane’s role in our tragedy – for yes, some small tragedy is unavoidable now, even without methane’s impact and even if we do everything we can from now on. I am sure that as an amateur I am missing or misrepresenting some points. I am also pretty sure that most amateur commentators are doing far worse. If I were you, I would not take comfort from any of this post’s stumbles or missteps.

How It Works

Methane is CH4, or a carbon atom with four hydrogen atoms. It is, imho, the second most important “greenhouse gas”, and to understand its effects it is best to compare it with carbon tossed into the atmosphere and combined with oxygen to form CO2, or carbon dioxide.

Here’s how carbon emissions that form carbon dioxide work (excess detail stripped away). As they ascend in the air they combine with the oxygen to form carbon dioxide. Most of that carbon dioxide sits in the atmosphere for perhaps 150 years, and most of the carbon has fallen to earth again within 250 years. While it is up there, doubling the amount of carbon (in CO2 form) in the atmosphere adds about 3 degrees C to global temperatures, and double that in the far north and south, especially in the winter. The “normal” rate of carbon in the atmosphere is 250-280 ppm (parts per million), and we are presently somewhere around 395 ppm.

Methane emissions work in a similar fashion, but with some important differences. In the first place, methane is typically stored in the earth and emitted as a gas – i.e., not as carbon but as CH4. Once it gets into the air, it can either split the carbon atom to form carbon dioxide – hence increasing that greenhouse gas – or remain as methane. If it stays methane, most of it stays in the atmosphere for 10 years, and most is gone after 15 years. So what’s the problem?

Well, the problem is that while methane is in the atmosphere it has up to 70 times the impact on global warming of comparable amounts of carbon dioxide. I find it useful here to imagine the old image of keeping a ping-pong ball in the air with jets blowing from beneath. If I toss a ball of carbon dioxide in the air, it stays up there for 200 years. With methane, I have to keep blowing like crazy – or, if you like, adding the same number of new balls of methane every 12 years. But if I keep an amount of methane in the atmosphere comparable to doubling carbon dioxide, then I drive up temps not by 3 degrees C, but by 100 degrees C. No, we haven’t gotten to the worrisome part yet.

In other words, the effects of increased amounts of atmospheric methane, piled on top of increasing carbon dioxide from other sources, fall somewhere in between two extremes. At one extreme, all the methane turns into carbon dioxide, and hangs there for 150-250 years. As we will see, that means that carbon dioxide may double or quadruple compared to global warming without intervention by methane, for an additional 3-6 degree C global warming. At the other extreme, all the methane stays methane. As we will see, a reasonable guess for its effects then is a 12-18 degree C additional increase starting somewhere around 20 years from now and going for 120 years, and then fading out. To put it bluntly: we roast more now (stays methane) or we fry more later (changes to carbon dioxide).

The Real Worry

So where are these new methane emissions coming from? Mainly, there are two potential types of source. The first is human-caused activity: just as we emit more carbon by burning fossil fuels as our population and industry grows, so emissions of methane for industry and personal tasks and by increasing populations of farm animals like cows increases accordingly. The second is methane frozen in the earth between periods of unusual global warming. That methane lies in three main places:

i. The shallow Arctic seas, especially the shallow Siberian sea north of Russia, where it is frozen not in ice but in a comparable substance known as a clathrate;

ii. The permafrost of Siberia and northern Canada/Alaska, where it is locked in frozen ground tens of meters deep on top of unfrozen earth;

iii. The peat bogs further south, where it is often mixed with water.

The methane emissions from our first type of source have caused methane in the atmosphere to shoot up fairly steadily over the last 100-odd years, so that methane is now a significant contributor to today’s global warming. It would be a really excellent idea to cut down on it. However, to some extent that increase has apparently leveled off. No, what really scares us over the next 200 years is the second set of sources.

The last time the globe apparently became (not when it was, when it became) this warm or warmer – maybe 55 million years ago, in what is called the Paleocene-Eocene Thermal Maximum, or PETM for short – it seems very likely that methane from the second source type was indeed emitted in quantity as methane, as that is a very good explanation for why temps actually went a little higher than the amount of carbon dioxide in the air would seem to dictate. However, that should not give us comfort. Ken Caldeira in Nature notes that methane of this type was stored in much smaller quantities then. That would mean that methane from sources i-iii emitted now would either (a) have similar effects over a much longer period of time or (b) would have much greater effects over the same period of time. So which is it, (a) or (b)?

Well, one obvious factor in deciding between (a) and (b) is how fast our global temps are going up already, before we start emitting i-iii. Once we start that faster rate of emissions, of course, that will speed up global warming even further, so we can bet that a faster initial rate of global-temp increase will keep methane emissions higher right throughout the process. And every available bit of evidence points to the fact that we are warming already much faster than in the PETM – because we humans are emitting carbon stored in the earth as “fossil fuels” (really, mostly decayed vegetable matter) much faster.

All right, so it’s faster. Is it fast enough to worry about? Here we have to consider sources i, ii, and iii separately. Methane clathrates are apparently a big honking source of methane, according to scientific estimates. No one is entirely sure about how fast these clathrates will “melt” and methane bubbles will rise to the surface, once they start melting. However, they are sitting in shallow seas and they start right at the surface of the sea-floor. We know what it takes: warming of the water above the clathrates. And that has been happening, as the Arctic sea ice in that area at the top of the water melts in the summer where it hasn’t before, the sun beats down on the newly-exposed water to heat it, and warmer water from the south moves in.

Now let’s consider ii. Joe Romm at www.thinkprogress.com has an extended post focusing on this source. The net of what he has to say is: Methane stored in permafrost is comparable in amount to methane stored in clathrates – big and honking. Permafrost melting is already underway at a brisk pace. Projections that are unrealistically conservative about how fast global warming will occur project that methane/carbon dioxide emissions from that permafrost will reach a high level about 20 years from now and continue at that level or somewhat higher for 120 years, at which point most of the permafrost will be gone. Make your own adjustments – however you adjust, it’s going to reach a higher level than that sooner, and stay there for a shorter period of time. Is that enough, by itself, to worry about? You betcha.

Then there’s iii – peat bogs and wetlands, even in the tropics. It’s not clear that there is as much methane there, or that it will be released as quickly. Remember, the further south (north, for the Southern Hemisphere) you go, the slower the rate of global warming. But it’s very clear that it’s happening. That was what the Russian summer fires were all about: global warming led to warmer temperatures that dried up the peat bogs and they went up in smoke, releasing methane. My totally random guesstimate is that peat bog methane emissions will follow much the same trajectory as those in i and ii, and will therefore have ½ to ¼ the impact at any one time or overall of either i or ii.

Now let’s reach ahead and note that things get drastically worse if all of these emissions increases happen over the same period of time – somewhere around 2-2.5 times worse. Luckily, so far emissions source i has not yet kicked in. Scientists report that as of 2010, there were no atmospheric signs of unusual methane or CO2 from Arctic sea sources. Be careful. There’s a trick in that statement.

Doing the Math

Before we hurry on to our conclusion, let’s pause and see if we can nail down a little better what those effects are likely to be if all three sources of the second type fire off fast at the same time. In particular, let’s assume a scenario like that predicted for source ii, only this time with all three sources emitting like crazy.

One estimate has the amount of stored methane, converted to carbon dioxide, at about 7 times the amount in the atmosphere right now. Let’s assume that, starting 20 years from now, this emits about 1/3 of itself at a fairly steady rate over the course of the following 160 years. So, 2.3 times 400 ppm or 920 ppm is the amount added to the atmosphere by 2170, on top of the existing amount (400 ppm) and the amount estimated to be added to the atmosphere by 2100 under “business as usual” (900 ppm). We’re up to about 7-9 degrees C global warming somewhere between 2100 and 2150. Even if we cut our emissions to zero today (totally unrealistic), we’re up to 5-7 degrees Celsius.

If you want to be gloomy, you can assume almost all the methane, turned to carbon dioxide, vents in the same time period. Add on another 1800 ppm, and then add on another 500 ppm for continuing “business as usual” between 2100 and 2170. Now we’re talking 12 degrees C.

OK, same thing, but it all stays methane. If you remember, this is over 160 years, but methane falls from the sky after about 12 years, so we’re talking about 12/160 = 1/12.33 of the equivalent amount of carbon dioxide on an ongoing basis over that 160 years. However, that methane has perhaps 33 times the effect on temps while it’s up there. So starting 20 years from now, there is an overall jump of about 8 degrees C on top of the effect from carbon dioxide noted above – and that effect lasts for 160-odd years. That baked-in non-methane carbon dioxide effect is going to be around 3 degrees C under the most optimistic of assumptions, and could go as high as 9 degrees C. And remember, if you want to be gloomy, tack on an additional 6 degrees C from emitting almost all the methane.

So here’s your two extremes. If you’re lucky, it’ll all go up as methane, and fall right down again. Now we’re talking 11-23 degrees C global warming between 2030 and 2190, and we fall down to a nice comfortable 6 degrees C after that. If you’re unlucky, it’ll all go up as carbon dioxide, in which case we’ll see maybe 8 degrees C of global warming from 2050 to about 2330. By the way, initial estimates are that more of it will rise as carbon dioxide.

The best part of this analysis is that I left out other “positive forcings.” In particular, I left out the fact that all this warming is going to turn part of the ocean (The Arctic and Antarctic) and part of the land (all those nasty glaciers) darker, from white (snow) and off-white (ice) to dark brown and green (land) and dark blue (ocean). Darker colors store heat. I’m not sure what how much warming effect that will have, but scientific estimates suggest indirectly (it’s included in some scientific estimates suggesting 3-6 degrees additional warming beyond that due to carbon dioxide in the atmosphere) that it’s likely to be at least an additional 1 degree C. Icing on the cake. Or not icing.

The Best Part

Now back to that trick. You noticed, didn’t you, that I said “as of 2010.” Well, the recent article cited by Neven said that a Russian scientist reported that in their annual sample of Siberian Sea methane emissions, which they had been doing for 20 years, for the first time ever they were seeing, not “funnels” of tens of meters across from which methane was bubbling up, but lots of funnels “more than a thousand meters across”. Do the math: that’s between 1000 and 10,000 times the rate they had ever seen before. He was very confident that results across much of the Siberian Sea would be similar. Is that enough to signal the start of Arctic sea methane emissions on the scale we’ve been talking about? How can it not be enough?

Now let’s add the usual caveats: wide variance inherent in the estimates, lack of confirming evidence in some areas, uncertainties in data collection, blah, blah. The scientist’s reaction to these is to minimize the impact by stating the most likely impact of which he or she can be certain. The realistic reaction is to ask what is the impact of median likelihood, with equal likelihood of a lesser or greater impact – and, as far as I can tell, that’s what I’ve given you.

And, by the way, don’t bother to object that present projections don’t show this. Guess what – most models don’t consider the impact of even one natural methane source behaving this way, and the rest (only recently) of just one (permafrost).
Like I said, I don’t get sick to my stomach about this – because I did my own guesstimates more than a year ago and got my puking done then. I’m still hoping that the methane shoe will drop more slowly; and also that I’ll win the lottery. Right now, the latter seems more likely. Happy holidays, all.

The Psychological Failure of Libertarianism

2011-12-20T12:19:00.000-08:00

One of the theories of government that I have been considering lately is this: the long-term success or failure of a system of government is determined in many cases by how they treat their insane. Or, more precisely, how they avoid discarding people with psychological problems unnecessarily while dealing with people with psychological problems who potentially can gain power over other people.
Until the early 20th century, we did not even have the vocabulary to face this question. I remember one student of history who was quoted as saying that our reading of history before about the 12th century should be tempered by the fact that everyone, from Aristotle to Abelard, was absolutely nuts – that they led lives of trauma and were genetically predisposed to serious psychological illnesses. Yet no history before about 1920 that I can recall ever raised such a question.

Nevertheless, we can guess at some of the ways certain systems of government approached the matter. On the one hand, Charles of France was treated as intermittently mad while Henry V of England invaded, and so was George III of England (although we now think it may have been an inherited medical condition); while the unclassifiable were confined to madhouses and Bedlam in unbelievable squalor that certainly did not improve their plight. On the other hand, people like Napoleon (one theory posits he had extreme ADHD or a Type A personality, since he was always hyperactive and slept little) and Alexander (at the very least, megalomania exacerbated by alcoholism) were rewarded with excessive power despite the destructive effects on society of over-accommodation of their insanity – so that not only military dictatorship but also hereditary kingship tended to choke on its own psychological excesses.

Democracy (or, more properly, until the early 1900s, republicanism) as practiced in Britain and the United States did offer some “checks and balances” on power which served to limit the power of the insane whose insanity was indetectible: a General Sherman gave every indication of occasionally going off the deep end, but was never in a position of political power where it could really ruin things in the long run – except for his power over Native Americans.

However, with the popularization of Freud’s theories, the veil was ripped away from many of the clearly problematic psychologies of power, and it has never been possible again to accept with equanimity addiction, sadism, psychopathy, narcissism, and other conditions that are deeply serious when allied with power to govern others. In this light, while non-democratic societies that hide and deny psychological problems are clearly inferior, just as clearly our democracies have far to go in facing up to the psychological problems of those in power, and in treating the psychologically disabled in ways that ensure we don’t waste those who can contribute, rather than typecasting them or stuffing them in jails to be forgotten.

Rejecting the Deeper World
The effect of our new understanding of psychology has been subtle but pervasive. I see it clearest in a sharp divide occurring about 1920 in literature. I remember picking up a stray “woman’s novel” once from around that time, and marveling at how its differences cast any novel from 1910 and before as flat, bland, and somehow shallow – the stick figure of the madwoman was revealed to have a past of tragedy that not only shaped itself into depression and alcoholism, but led with painful examination of all the “unmentionables” repressed for the sake of survival to a very nuanced hope. Contrast that with a Samuel Butler, a G.A. Henty, or even a Mark Twain!

At the same time, however, it has led Western society, at least, into an uncomfortable and apparently unending examination of the psychological characteristics of our leaders. We have learned, for example, that our habit of demanding fidelity to our wishes from our male legislators while subjecting them to incredible amounts of campaign contempt has led to amazing numbers of them seeking unconditional adoration from groupies, and frequenting whorehouses where they alternately dominate and are spanked. We have in some cases learned to accommodate leaders who have been treated for depression or who smoked pot when they were young, perhaps because we have begun to realize just how understandable and relatively unimportant such traits are in a leader.

The psychological viewpoint has also affected how we view major “new” theories of the government from the past. Communism – to be clearly distinguished from socialism, which seeks to balance a greater “insurance” role for government with continuing democracy – has fallen short in practice, we now assert, not only for its other sins, but also because its secrecy and naïve view of human psychology (“from each according to his abilities, to each according to his needs”, to misparaphrase Marx) leads more often to pathologically paranoid leaders and the depression and alcoholism of the command-and-control economy. So does Fascism. A capitalism-driven government – or, if you prefer, an oligarchy, not of the landed but of the moneyed rich – falters as a theory when we consider the narcissism and Type A obsessive-compulsive behavior of its leaders, the so-called “captains of industry and finance,” and its failure to face the highly irrational behavior of its participants that leads to “bubbles”, “imperfect markets”, “Ponzi schemes”, and above all to “doubling down on failure.”

And then there is libertarianism – the only theory, in my opinion, that can be said to have gained currency after the arrival of psychology on the scene. What is striking about libertarianism is that, superficially at least, it takes no notice of psychology’s insights; in fact, it acts as if they don’t exist. When previous theories were developed, ignoring the deeper world of psychology’s insights was understandable, since those insights had not yet been developed; in libertarianism, it amounts to a – unconscious! – rejection of those insights. What’s going on? What does it mean for libertarianism in practice?

Libertarianism in Theoretical Practice
I have been hearing about libertarianism since the early 1980s, when it was all the rage in the political discussion groups on the nascent Internet – in fact, the incessant posts about how “libertarianism solves this!” were early markers of today’s trolls who crowd out all other viewpoints. My question, then and now, was, OK, how does it really work in practice? Ayn Rand’s The Fountainhead, which I had picked up 15 years before, was of very little help. Even at 15, I could see that the climactic trial scene was utterly unrealistic (the prosecutor in any sane trial would, instead, be talking about law and order and how the protagonist should be locked up to prevent future acts that would kill people); and this was the case of an isolated libertarian in a non-libertarian society, not a fully-fledged functioning libertarian culture. Likewise, Robert Heinlein’s Moon is a Harsh Mistress seems to show a complete (prison) libertarian society; but proponents seem to keep forgetting that at the end, when supposedly the truly independent libertarian society is launched, it immediately breaks down in “political infighting”, and the protagonist leaves for hopefully more libertarian pastures.

In fact, the best portrayal of a libertarian society I have seen is in a gem of a science-fiction novel by a long-underappreciated author, Eric Frank Russell. It is really three essays in political theory, in which a benign military invasion seeks to restore three planetary societies to “normal” government, and fails miserably at each point. The first is a Mafia government, where the inhabitants reject return to “normalcy” by treating every military move as a mobster power play. The second is a government of narcissists, which simply regards everyone else as incredibly ugly and therefore to be exterminated or at least removed from sight immediately. The third is the most appealing to the protagonist – and to us.

In it, the whole society functions by a form of barter known as “obs” – short for obligations. If you want something from someone, you do something for that person (or someone else, in trade) in return – there is a very clear relationship between work and reward. Attempts at domination are met, confusingly, with “myob!” – short for “mind your own business.” The person who tries to get things for free is progressively blocked until, if he does not stop, he cannot get anything at all, and starves to death.

At this point, readers of the Laura and Mary books should begin to suspect what makes this society so appealing – it’s very like the frontier society described in those books. Pa and Ma are utterly self-reliant and function well in a society of other self-reliant types, where everyone feels an obligation to help others and expects others to help them in the same measure, where payment is as much by that kind of trade as by exchange of goods for money, where the children are brought up to behave in the same way, with fewer stereotypes about “helpless women”, and where outsiders are few.

And yet, the Laura and Mary books, written as they are from the viewpoint of a pre-psychology frontier society, cannot help but note things that call into question the long-term stability of such a society, and form a sharp contrast with Mr. Russell’s society. The violent, disturbed young student in Laura’s class is handled without regulations, yes, but by application of a whip that will in all likelihood lead to an adulthood in which the student will view the whip as the answer to everything – “might makes right.” The government that fails to get the trains through in the Long, Hard Winter is rightly viewed as incompetent by Pa; but, as technically dexterous as people are, there is no sign that the frontier society is anywhere near providing the same level of technology as the train and distribution system that keep folk alive and keep people from going stir-crazy. When Almanzo proves incompetent at farming and goes back to his expertise in running a country store, while Laura wrecks her health trying to pick up the slack at the farm while pregnant, Pa comes to the rescue, yes, but the result is that Laura moves back to a non-frontier, non-libertarian society, where there is a better social safety net. The drunks, the rowdy miners, the rancher-farmer friction of much of the rest of the American West and the attendant psychological problems, are mostly offstage, but we can still perceive that they are out there. The problems of class (the Olsens), of isolation (the Long, Hard Winter), of excess disease (Mary’s blindness) are all psychologically-related strains that this society can only handle up to a point, and whose solutions are far from ideal.

Libertarian Power Vulnerability
At this point, the libertarian will typically answer with a variant of “Here’s how libertarianism solves the problem!”, usually an assumption that today’s or tomorrow’s technology (computer or otherwise) will allow a very loosely coupled massive collection of frontier societies to function without the psychological stresses caused by hunger, disease, and so on. But that is not my point. Because we now come to the psychologically disturbed who seek excess power in such a society.

Note Mr. Russell’s way of solving the problem of the self-centered, amoral individual. Leave aside the fact that piling more “obs” on a person may face that person with reality, but may also create a burden greater than an individual who is partly but not wholly self-centered can bear. No, my objection is rather that such a solution still leaves the door wide open to exploitation by people with known and dangerous psychological conditions – in many ways, wider open than other forms of government.

Consider, for example, the psychopath. The psychopath has no objection to working hard – although for the life of him he cannot understand why other people care. Rather, the psychopath is often superb at counterfeiting the well-adjusted, hard-working individual, while behind the scenes he seeks control and immediate gratification of whims. Want to see how he behaves in his private life? Myob! He will happily undertake all sorts of obs for what he views as useless things in return for more and more power; and he is exceptionally good at convincing you that what he is doing is natural and normal and desirable.

In most societies, not the psychopath but at least the existence of abnormal power is recognized, and so at least psychopaths are kept under more control via laws, competition from other psychopaths, and, more recently, recognition of and monitoring for the condition. In libertarianism, by definition, psychopaths are supposed to be kept in check “automagically”, by the reality of the “ob” plus the perception of their saner fellows. But, as we have seen, the psychopath is not impeded by work to satisfy “obs” nor by inability to deceive those linked in the “ob” chain. On the contrary, the loosely-coupled nature of this society means that if the technology cannot pick it up (and why would a libertarian trust a computer Big Brother?) then the victims simply don’t compare notes – myob! And the psychopath has every incentive to reach out to other psychopaths in other communities to increase power without threat to himself. The result is a libertarian veneer over the worst of totalitarian societies – a kind of libertarian Animal Farm.

A closely allied problem is that of the narcissist. The narcissist in fact has a great incentive to work hard, and a superficial self-confidence that counterfeits sanity, but, like the psychopath, she has a total lack of empathy, and she also has a strong and violent reaction to criticism (btw, in case you wondered, both conditions affect both sexes). The object of the narcissist, therefore, is to avoid such criticism at all costs, by surrounding herself with sycophants controlled by massive obs. Narcissists also tend to reach armed truces with other narcissists in other communities leading to mutual admiration societies, with Mutually Assured Criticism replacing Mutually Assured Destruction as a deterrent to competition between them.

In most societies, the narcissist is more easy to detect than the psychopath, and criticism that cannot be held at bay is the ultimate control – which we can sometimes achieve in both democracies and autocracies by occasional but nonetheless somewhat effective random acts of criticism/punishment fueled by laws. But while Nelly Olsen may have been easy to counter in a frontier society, a massive collection of such societies allows the Nelly Olsens of the world a surprising power to combine to hold up “fashions” as a tool for power. Subtly, the freedom of the libertarian to think different thoughts is corrupted into the freedom to buy into the narcissist’s idea of fashion. Again, the libertarian idea that such things can be handled “automagically” simply enables a libertarian veneer over a world in which we respond as the narcissist wants, with useless obs traded for narcissist praise and control.

I pass over addiction here because, in fact, it can be argued that libertarianism has an equally effective method of dealing with it – which is, I believe, the method outlined by Mr. Russell, of refusal to be blinded by promises not backed up by fulfillment of obs, and of effective withdrawal of the addiction: kill or cure. However, I would note that this is adequately effective only if you believe that approaches that seek to end the addiction before starvation arrives are useless. I happen to believe that the carrot of psychological encouragement and the medical fix as well as the stick does better, and so libertarianism, which by its nature tends to reject the first two, would in the end prove less effective. In any case, the addict has a destructive effect on any scheme of government that is limited by his or her excesses to the relatively short run, hence the short reign of some of Rome’s Augustan emperors – so the point is moot.

And now we can revert to the problem of how a government handles the psychologically powerless who are disturbed. I would repeat, again, that the reasons I view libertarianism as not as good a societal solution as many of the others cited above – the unusual power they give to certain types of psychological disturbance by their inability to recognize those types – do not apply to the case of the psychologically ill who are powerless. Here, I rely only on my own experience of having a hand in the care of the autistic. The help of others that was like that of a frontier community helping with each others’ burdens; it was well-meant but in almost all cases – whether it involved the autistic person taking responsibility for his or her entirely automatic destructive actions, or volunteering the helper’s own creative thought based on irrelevant experience about what it was and what to do – was almost invariably counterproductive. The help of businesses was entirely focused on what made money for the purveyor, with as little regard as possible for the needs of a particular as opposed to the “average” autistic person. The help of government was bumbling, bureaucratic, intrusive, and in most cases the most effective of all, because it bore some vague relation to both the common good and solid science – particularly as regards psychology. We can argue endlessly about whether “libertarianism can solve that problem!” The fact is, the libertarian communities I hear about out there don’t have psychological smarts, aren’t set up to inhale or accept scientific knowledge about Tourette’s or ADHD, and tend to be misogynistic and indiscriminate in their enthusiasms; so even if the nature of libertarianism does not preclude decent handling of the psychological illness of the powerless, I see absolutely no sign of real-world libertarianism coming up with such a solution in the foreseeable future.

A Better Way
And yet, as I have noted, we perceive some merit in the ideas that libertarianism promotes. This merit, I would say, lies entirely in two aspects of their criticism of present-day societies and governments. Firstly, yes, there should be a much better line between actions and consequences. We respond positively, rightly, to the notion that we have to earn as much as, but not more than, we get in return, and we have to pay the right price for doing wrong things – because it tells us that our irrepressible self-esteem is truly earned. Secondly, it is indeed reasonable to be concerned about increasing powerlessness and threat of blackmail in one’s life as governments intrude, even to try to make things better. It is this kind of merit, I believe, that leads a scientist like James Hansen to protest that he is a bit of a libertarian himself, even as he is being viciously attacked by climate deniers in the guise of libertarians.

And yet, libertarianism, like anarchism, is effectively defined by its negatives. Rather than accommodating a theory of human psychology that requires consideration of structures to contain the harmful societal effects of mental illness, while preserving the potential of those temporarily ill for later contributions, libertarianism “solves” the problems of skewed action/reward pairs and powerlessness by abolishing such structures entirely. The result, by libertarianism’s own definition, is a much greater inability to cope with society-destabilizing and destructive psychological conditions such as psychopathy and narcissism.

And so, sadly, we must return to trying to improve the systems we already have in these areas. These, at least, have made some effort to accommodate psychology’s insights; and therefore their structures are to some extent pre-adapted to the task. However, our development of these structures has up to now been almost entirely reactive, involving after-the-fact regulations and sanctions, leaving the powerful but disturbed always one step ahead in adapting to the new rules of the game.

A better answer to the action/reward problem, I believe, is to embed a deliberate integration of psychology and action/reward fine-tuning into the workings of the government. Yes, like everything else, it can be diverted by the powerful. But introducing the notion of psychopathology into the fabric of normally-functioning society and power structures makes the notion a fundamental part of life that the powerful cannot quite argue away. Too blatant a violation, and only sweeping aside other powerful people to alter the government will do; and that’s a solution that even a psychopath will usually find difficult. The action/reward fine-tuning can then be undertaken as limited by psychology concerns, rather than used to justify deregulation leading to libertarian dystopias.

In the powerlessness area, we are faced with the dilemma that societal success and growth increases distorting power bases that affect us. Our technological might and numbers allow us to create better monitoring technologies for threats like al Qaeda, technologies which in turn are very hard to point away from us, much less prevent the powerful in government and outside it from using these technologies to keep us in line, whatever that means. As our numbers and tasks increase, the power of our vote decreases. The answer here, I believe, is less in the checks and balances and regulations that are harder and harder to preserve and extend, and more in the extension of fundamental limits on any powerful person to throw sand in the gears, by the constitutional elevation of scientific fact as a limit on all government. It is now time to refuse to permit laws that define pi as 3.00, as the Indiana legislature is once reported to have done. And that job should not be the sole duty of the courts, but also of scientists whose independence and peer-review process is preserved constitutionally as a necessary part of a free society. The best way to prevent burgeoning power from ever being used against us unjustly is to require that its users never distort the truth beyond recognition in their efforts to do so; because the powerful will view changing these rules are not worth the bother, while this kind of rule places a better limit on the powerful, no matter what their tactics.

And finally, the better way would ensure that we continue to improve our handling of the powerless and psychologically ill, as we have been doing, free of interference from those who would wipe out such efforts in the name of libertarianism and the like. I am no automatic fan of today’s efforts in this area. I am, I think, fully aware of their misuse, not least by some psychologists. I simply assert that we are much, much better off than in the days when Bedlam was the proper place for the autistic – and further improvements lie along much the same path.

Viewed in terms of human psychological excesses, it seems clear to me that libertarianism is by its nature a cure that is worse than the disease. Within our present frustrating systems, there lie the seeds of a way at least marginally better than today’s governmental mess. One way to grow those seeds is to do a better job of facing the realities of human psychology when allied with power, leading to alterations within the general framework of those systems. Libertarianism offers good criticisms but also a poor solution that rejects psychology. We should incorporate those criticisms into a government that handles human psychology better and therefore rejects libertarianism.

Energy Usage and IT: Is the Name Not a Clue?

2011-12-08T15:41:00.000-08:00

At the IBM Systems and Technology Group analyst briefing two days ago, IBM displayed three notable statistics:

1. The global amount of information stored has been growing at 70-100% per year the last 5 years, with the result that the amount of storage has been growing by 20-40% per year;

2. The amount of enterprise expenditures for datacenter power/cooling has grown by more than 10-fold over the last 15 years, with the result that these expenditures are now around 16% of system TCO – equal to the cost of the hardware, although well below the also-rising costs of administration;

3. Datacenter energy usage has doubled over the last five years.

These statistics almost certainly underestimate the growth in computing’s energy usage, inside and outside IT. They focus on infrastructure in place 5 years ago, ignoring a highly likely shift to new or existing data centers in developing countries that are highly likely to be more energy-inefficient. Also, they ignore the tendency to shift computing usage outside of the data center and into the small-form-factor devices ranging from the PC to the iPhone that are proliferating in the rest of the enterprise and outside its virtual walls. Even without those increases, it is clear that computing has moved from an estimated 2 % of global energy usage 5 years ago to somewhere between 3 and 4%. Nor has more energy usage in computing led to a decrease in other energy usage – if anything, it has had minimal or no effect at all. In other words, computing has not been effectively used to increase energy efficiency or decrease energy use by more than marginal amounts – not because the tools are not beginning to arrive, but rather because they are not yet being used by enterprises and governments to monitor and improve energy usage in an effective enough way.

And yet, there have been voices – mine among them – pointing out that this was a significant problem, and that there were ways to move much more aggressively, since the very beginning. I remember giving a speech in 2008 to IT folks, in the teeth of the recession, stressing that the problem would only get worse if ignored, that doing something about it would in fact have a short payback period, and that tools for making a major impact were already there. Here we are, and the reaction of the presenters and audience at the STG conference is that the rise in energy usage is no big deal, that datacenters are handling it just fine with a few tweaks, and that IT should focus almost exclusively on cutting administrative costs.

All this reminds me of a Robin Williams comedy routine after the Wall Street implosion. Noting the number of people blindly investing with Bernard Madoff, pronounced “made off” as in “made off with your money”, Robin simply asked, “Was the name not a clue?” So, I have to ask, “energy usage”: is the name not a clue? What does it take to realize that this is a serious and escalating problem?

The Real Danger

Right now, it is all too easy to play the game of “out of sight, out of time, out of mind.” Datacenter energy usage seems as if it is easily handled over the next few years. Related energy usage is out of the sight of corporate. Costs in a volatile global economy that stubbornly refuses to lift off (except in “developing markets” with lower costs to begin with), not to mention innovations to attract increasingly assertive consumers, seem far more urgent than energy issues.

However, the metrics we use to determine this are out of whack. Not only do they, as noted above, ignore the movement of energy usage to areas of lower efficiency, but they also ignore the impact of the Global 10,000 moving in lockstep to build on instead of replacing existing solutions.

Let’s see how it has worked up to now. Corporate demands that IT increase capabilities while not increasing costs. The tightness of the constraints and the existence of less-efficient infrastructure causes IT to increase wasteful scale-out computing almost as much as fast-improving scale-up computing, and also to move some computing outside the data center – e.g., Bring Your Own Device – or overseas – e.g., to an available facility in Manila that is cheaper to provision if it is not comparably energy-optimized at the outset. Next year, the same scenario plays out, only with even greater costs from rebuilding from scratch a larger amount of existing inefficient physical and hardware infrastructure. And on it goes.

But all this would mean little – just another little cost passed on to the consumer, since everyone’s doing it – were it not for two things; two Real Dangers. First, the same process impelling too-slow dealing with energy inefficiency is also impelling a decreasing ability of the enterprise to monitor and control energy usage in an effective way, once it gets around to it. More of the energy usage that should be under the company’s eye is moving to developing countries and to employees/consumers using their own private energy sources inside the walls, so that the barriers to monitoring are greater and the costs of implementing monitoring are higher.

Second – and this is more long-term but far more serious – shifts to carbon-neutral economies are taking far too long, so that every government and economy faces an indefinite future of increasing expenditures to cope with natural disasters, decreasing food availability, steadily increasing human and therefore plant/office/market migration, and increasing energy inefficiency as heating/cooling systems designed for one balance of winter and summer are increasingly inappropriate for a new balance. While all estimates are speculative, the ones I think most realistic indicate that over the next ten years, assuming nothing effective is done, the global economy will reach underperformance by up to 1% per year due to these things, and up to double that by 2035. That, in turn, translates into narrower profit margins due primarily both to consumer demand underperformance and rising energy and infrastructure maintenance costs, hitting the least efficient first, but hitting everyone eventually.

The Blame and the Task

While it’s easy to blame the vendors or corporate blindness for this likely outcome, in this case I believe that IT should take its share of the blame – and of the responsibility for turning things around. IT was told that this was a problem, five years ago. Even had corporate been unwilling to worry about the future that far ahead, IT should at least have considered the likely effects of five years of inattention and pointed them out to corporate.

That, in turn, means that IT bears an outsized responsibility for doing so now. As I noted, I see no signs that the vendors are unwilling to provide solutions for those willing to be proactive. In the last five years, carbon accounting, monitoring within and outside the data center, and “smart buildings” have taken giant leaps, while solar technologies at whatever cost are far more easily implemented and accessed if one doesn’t double down on the existing utility grid. Even within the datacenter, new technologies were introduced 4 years ago by IBM among others that should have reduced energy usage by around 80% out of the box – more than enough to deliver a decrease instead of a doubling of energy usage. The solutions are there. They should be implemented comprehensively and immediately, as, by and large, has not been done.

Alternate IT Futures

I am usually very reluctant to criticize IT. In fact, I can’t remember the last time I laid the weight of the blame on them. In this case, there are many traditional reasons to lay the primary blame elsewhere, and simply suggest that IT look to neat new vendor solutions to handle urgent but misdirected corporate demands. But that begs the question: who will change the dysfunctional process? Who will change a dynamic in which IT claims cost constraints prevent it from “nice to have” energy tools, while corporate’s efforts to respond to consumer “green” preferences only brush the surface of a sea of energy-usage embedded practices in the organization?

Suppose IT does not take the extra time to note the problem, identify solutions, and push for moderate-cost efforts even when strict short-term cost considerations seem to indicate otherwise. The history of the past five years suggests that, fundamentally, nothing will change in the next five years, just as in the past five, and the enterprise will be deeper in the soup than ever.

Now suppose IT is indeed proactive. Maybe nothing will happen; or maybe the foundation will be laid for a much quicker response when corporate does indeed see the problem. In which case, in five years, the enterprise as a whole is likely to be on a “virtuous cycle” of increasing margin advantages over the passive-IT laggards.

Energy usage. Is the name not a clue? What will IT do? Get the clue or sing the blues?

Rethinking Printing

2011-11-03T17:38:00.000-07:00

A recent piece about HP’s plan to assert the key role of its printer division in modern computing, and move it beyond the “cash cow” status it presently seems to have by redefining printers’ use cases, left me underwhelmed. As a printer/copier/scanner/fax machine user, I could not see major benefits to me (or for that matter, to the typical SMB) from the proposed “paper is still relevant, cut printing costs, use the printer as a mailbox” strategy.

Still, it made me think. If I Ruled the World, how would I like to redesign things? How could printing technology play a key role in the Web 3.0 of the future? What follows is, I realize, an exercise in fantasy – it will probably never happen. Still, I think it might form a useful framework with which to think about what printing will really do in the future – and what it won’t.

Printing a Hamburger

One of my favorite old ads was one for A-1 steak sauce that began by saying, “My friends, what is a hamburger?” The point being, of course, that it was chopped steak, and therefore people should consider A-1 instead of ketchup. I would suggest that the whole printer area would benefit from asking, “My friends, what is a printer?”

My answer would be a “virtual” one: printing/scanning/copying is about creating a computer representation of a physical image – call it a graphic – that can be displayed on a wide variety of form factors. Thus, a scanner can take these physical graphics from the outside world; a “snapshot” can take such a graphic from a video stored inside the computer; software can intercept physical representations being sent to output devices from applications, such as internal representations of photos, internal representations of Web pages, internal representations of reports, check images, signed legal documents. These standardized representations of graphics then can be customized for, and streamed to, a wide variety of form factors: computer screens, cell phone and laptop displays, printers, email messages (attachments), or fax machines (although I tend to think that these are fading away, replaced by email PDFs).

Is this Enterprise Content Management? No. The point of such a “gateway”, that represents many graphic formats in a few ways and then customizes for a wide variety of physical displays, is that it is aimed at physical display – not at managing multiple users’ workflow. Its unit is the pixel, and its strength is the ability to utterly simplify the task of, say, taking a smartphone photo and simultaneously printing, emailing, faxing, and displaying on another worker’s screen.

One of those output devices – and probably the most useful – is the printer/scanner/copier. However, the core of the solution is distributed broker software like the Web’s email servers that pass the common representations from physical store to physical store, and route them to “displays” on demand. Rather, today’s printer is simply the best starting point for creating such a solution, because it does the best at capturing the full richness of a graphic.

We are surprisingly close to being able to do such a thing. Documents or documents plus signatures can be converted into “PDF” graphics; photos into JPGs and the like; emails, instant messages, Twitter, and Facebook into printable form; screen and Web page printing is mature; check image scanning has finally become somewhat functional; and we are not far from a general ability to do a freeze-frame JPG from a video, just by pressing a button. But, somehow, nobody has put it all together in such a “gateway.”

Extreme Fantasy

As a side note, I’d like to ask if the following is at all feasible. Visualize in your mind your smartphone for a second. Suppose you had optional add-ons in back. They would contain 4-5 pieces of phone-sized “image paper” in one slim add-on, and a laser-type “print head” that would place the appropriate color on each pixel on the paper A button or screen icon on the front would allow printing of whatever was displaying on the screen – including, as I suggested, a “snapshot” of a video.
Can it be done? I don’t know. I know that I would love to have such a thing for my mobile devices, even crippled as it probably would be. Remember the old Polaroid OneShot? The joy of that was the immediacy, the ability to share with someone physically present what was not a really high-quality photo, but was still a great topic of conversation.

Why haven’t printer vendors moved more in this direction? Yes, I know that attempts to provide cut-down attachable printers for laptops haven’t sold. But I think that’s because the vendors have failed to distinguish two use cases. One is the highly mobile case, where you’re running around a local office or factory and need to print on whatever form factor: small laptop, tablet, or even cell phone. That’s the case where all you need is the ability to print out 4-5 pages worth of info, bad quality and all. For that, you need a very light printer and preferably one that attaches to the form factor so that you can carry both – like strapping two books together.

The second is the “mobile office” case, where you go somewhere and then sit and work. In that case, you need full power, including a multi-sheet scanner/copier feeder – why don’t printer vendors realize how useful that is? It should be light enough so it can be carried like a briefcase or laptop, but it doesn’t have to be attachable; and it should be wireless and work with a MiFi laptop. Above all, it should be foldable so that it’s compact and its parts don’t fall off when you carry it.

Sad Conclusions

I realize that some of the details of the above fantasy may be unrealistic. he real point of the fantasy is that paper and the printer are not dead, no matter what; because, as HP pointed out people keep using them more. And, I believe, that’s because it is just so useful to have a “static” representation of lots of information to carry around with you. Nothing on the horizon replaces that need, not the cell phone or laptop with its small display, nor the immobile PC or TV, nor even the tablet with its book support but inability to “detach” valued excerpts for file-cabinet storage and real-world sharing.

But not being dead doesn’t mean wild success. I believe that HP’s printer “vision” is still too parochial, because it fails to detach the core value proposition from a particular physical device. It may be safe to extrapolate present markets; but I believe that such a marketing approach falls far short of print technology’s potential.

Still, I don’t believe that printer vendors will do what I fantasize. The risks of failure are too great, and the present revenues too comforting. No, it seems likely to me that five years from now, the printer business will still be seen as a cash cow – because nobody asked what a hamburger is.

Pity.

Update to Libya War Post

2011-10-21T09:25:00.000-07:00

Well, things have gotten better -- but not much.

A quick look at the archives of Britain's Guardian -- which did better than any US paper -- reveals far more coverage of Qadaffi's death than of the fact that the war was ongoing, or even that the rebels had finally eliminated the last prominent resistance in Sirte. And as late as a month ago, it was talking as if the military part of the war was over, while discussing an assessment of NATO "military success" as if that was the key element in military victory.

Meanwhile, Michelle Bachmann, a Presidential candidate, was quoted as saying 2 days ago (more or less) "Obama got us into Libya; now he's getting us into Africa" -- aside from the obvious but probably accidental geographical mistake, it seems clear she believed that the US was deeply involved in the Libya war and would continue to be involved in the near future.

This is just the commentary on the war; one might also cite the persistent failure to note Qadaffi's role in horrendous wars in Liberia, or the complete nonsense of any references to al Qaeda.

Still, the press did manage to finally understand that the war is really effectively over as of about the date of Qadaffi's death, that the rebels are indeed functioning as a government in all occupied territories, and that despite overclaims by the rebels, their statements about what was going on were far more credible than those by either regime spokesmen or remote reporters. Best of all, some news organizations finally got their reporters' butts out of their hotels and were able to confirm specific rebel successes.

Finally, let me recall this approximate quote from my previous post: "the war is not over ... at least a month, and in the worst case, two months more." It's a little less than two months, and here we are. In the meantime, I can't recall one news organization making that obvious projection. Meanwhile, Wikipedia still managed to provide me with clear, accurate news about the war ahead of all major news organizations about 1/3 of the time, by looking carefully at published NATO briefings.

I can't wait to see what our news organizations and politicians do for an encore. Probably forget all about Libya except to use it as a convenient stick figure for fear, uncertainty, and doubt. At this point, I am tempted to echo the advice of Professor Higgins' mother in My Fair Lady:

Stick to the weather and health.

Steve Jobs

2011-10-21T09:19:00.000-07:00

I believe that most folks were expecting an announcement of Steve’s death sometime soon. And yet, it is clear from most computing industry reactions to the announcement that there was something different and valuable about Steve, and that it is not clear who will succeed him in providing that unique “something.” But what is it?

I believe that it’s a much more subtle thing than most folks realize – because they don’t understand the mistakes he made along the way to becoming what he was just before he died, and how he managed to change himself just enough to begin to deliver user-friendly innovation after user-friendly innovation to meet the real needs of the consumer.

My Own Jobs History: The Early Apple Years
In his tenure as leader of Apple from its founding until his ouster, Steve Jobs displayed many of the traits that were later associated with his recent success. He was relentlessly focused on a “vision” for each release of an Apple product, and the vision was his vision, and everything in the product was highly integrated in the service of that vision. Thus, a product like the Apple II C had its own hardware, a different processor (i.e., not Intel), its own operating system (i.e., not DOS), and, above all, strict controls over how applications could be built on top of that operating system.

The results of that style were significant success and much more significant failure. As part of the overall PC revolution, Apple shared in the rapid growth of the consumer market. Steve Job’s vision, evidenced in “cool” products fitted for students and graphics departments, gave Apple a big share of those submarkets. But these submarkets never led to other markets – because application developers flocked to the more open free-for-all of Microsoft and its then competitors. At the computer software sections of stores in those days, you could find plenty of game, spreadsheet, and “do-it-yourself business” offerings on the PC side, and very few on the Apple side. And that, in turn, led to low market share for Apple in the consumer market, and in the business market as well.

There were other side-effects of Steve Jobs’ “vision” in those days. For a couple of years, I worked in a business workgroup environment using Macs, and I can tell you that the reality was not all it was cracked up to be. The print server software – a key part of our operation – frequently crashed, at bad times. The storage on Macs was prone to total failure, and there was a big risk of having all our work and records for a couple of years destroyed – as happened to me. I used PCs for years before and after; they were less user-friendly, and more prone to the “blue screen of death” in the abstract, but in the real world I could do a lot more and have a lot less risk of serious failure. And I suspect that the real reason for the difference was that things like print servers and storage just weren’t the focus of Steve’s “vision”.

At Apple, reportedly, Steve Jobs’ new-product-development style was a mixed bag. On the one hand, his insistence on user-interface quality and consistency clearly produced solutions that consumers intuitively liked, and his desire for innovation satisfied his programmers’ own urge to innovate. On the other hand, it seemed clear that his relentless upsets and negativity when the product wasn’t what he wanted were wearing after a while, gave little scope to the Apple programmers’ own creativity, and showed some blindness to what consumers really wanted. It is significant that Apple was able to spin off or foster few if any now-large software companies. FileMaker had and has excellent and useful database-design software, but is far smaller today than a Quicken.

Had Steve Jobs not been eased out at Apple back then, I am not at all convinced that he would have delivered the successful innovations of today. If I had been a board member tasked with deciding what to do, I would have weighed the short-term innovations like the Mac (which was a major step in graphical user interface [GUI] design, and had come about because Jobs stuck with the idea after the fizzle of the Lisa) against the long-term isolation to niche markets, and the waste of resources in never-win markets like Apple servers, and I might have agreed that it was time to try someone else.

A Decade in Exile
Initially, after he stepped down as head of Apple, Steve Jobs appeared to be positioning himself to make a return to Apple, but not necessarily to succeed any better once he did so. Specifically, Steve founded a company called NeXT, which aimed at producing the “next generation of operating system” using an object-oriented microkernel. Applied to a PC like the Apple Macintosh, this theoretically should have produced all sorts of benefits: better application performance, easier development of applications on top of the operating system (because object-oriented programming was a bit faster than the existing “structured” programming) and especially GUI-oriented ones, and faster turn-around on new versions of the operating system supporting all sorts of new capabilities. And, to a small extent, that was what happened when Steve returned from exile to Apple and made the NeXT operating system into Mac OS X Lion (the latest version).

However (again looking ahead), the effect was limited. Of course, at that point, Apple had nowhere to go but up, and in its traditional educational and graphics markets, the effect of faster delivery of Jobs’ “vision” products was to attract a whole new generation to Apple; but there was no clear indication that Jobs would be able to capture more than about 10% of the PC market, which was his high point during the early Apple years. In other words, the results show us that the effects of Steve Jobs’ NeXT “vision” were limited, and might have been short-term in and of themselves. The Steve Jobs of NeXT would probably have saved Apple; he probably would not have achieved more than a fraction of Apple’s present success.

It may seem odd to point to Pixar as the source of most of the computer-industry success that we have come to associate with Steve; but I believe that that is where he made a significant alteration in style that permitted that success. In very crude terms, I think he learned to ally with others. More specifically, he learned how to make deals with other companies that compounded his company’s revenues, how to allow companies (and later developers and consumers) to take his ideas and run with them, and how to accept ideas from others in areas he knew less about, and incorporate them into products in areas he did know very well.

The story of Pixar up to now is briefly told: five years of plowing money into a startup before things like Toy Story took off, an unequal alliance with Disney to allow survival that led to a major share in Disney once Toy Story took off, and the ability of Pixar to deliver innovation in computer-generated animation movie after movie, combined with solid success. But the key element in that success story was Jobs’ ability to strike and maintain a deal with Disney that infused not only Jobs’ “vision” but also Disney’s sense of commercial entertainment and third-party animators’ “vision” into Pixar, allowing it to repeat its initial successes without wandering off course.

The Real Triumph

When Steve Jobs returned to Apple, remembering his past history, I was wary of becoming too enthusiastic. And, indeed, until the arrival of the iPhone, it wasn’t completely clear that things had changed. But there were two things about the iPhone that signaled a significantly different Jobs. First, the touch screen was done right. Not only was it innovative (somewhat), and user-friendly in its simplicity, but it was entertaining. It seemed to me that Steve had taken what he had learned at Pixar, and applied it to giving a broad set of consumers a new “cool” tool.

The second key difference was that iPhone did development right. Yes, there was reluctance and there were limits initially on iPhone app creation, but the old Steve Jobs might very well have kept tight control on app development forever. The new Steve Jobs allowed outside developers to get a little compensation for each app, and, as they say, the rest is history – a history that shows that the result is not short-term but long-term evolutionary innovation of iPhone and the smart phone.

Things like iPad and iCloud are very much a repeat of the same story. The tablet has a long history of failure, and indeed the old Apple took full part in the belief that what mattered was making it easy for people to scribble on a computer screen. But iPad – and its competitors – were and are mainly about touch-screen commands and downloading books and movies at lower costs for use on the go. That’s a smart phone and media industry insight as much as a computer industry one. And, of course, Steve Jobs’ “vision” embodied in the user interface persuaded most consumers to accept the innovation and take the next big step forward. While iCloud will probably wind up making less of a smash, its development and introduction appear like iPhone and iPad in miniature.

Envoi
And so, I salute the Steve Jobs of the last few years as having indeed delivered major innovation that is unique and valuable. But I assert that the real question should be, not who if anyone in future will succeed him in his “vision”-type style, but who will be able to deliver that kind of results, whatever their style?

You see, there are two types of failure inherent in Steve’s story. One is the failure of the “visionary” who may be right about the general direction of innovation, but undercuts that innovation by “I only see my way” implementation of the vision. The other, which we are much more apt to perceive, is the incessant failure of others in the computer industry to pay the price initially to implement the vision, or even to have the innovative chops at all.

That the later Steve Jobs avoided both failures is, I think, an indication that at any time he would be a pretty rare bird. But I would argue that slowly, reluctantly, corporations of all stripes are coming to realize that innovation that both connects to the customer constantly and taps into company or third-party creativity is a Good Thing. So I am hoping that we will see more successors to Steve Jobs, not in his style, but in his result of successful innovation – as long as these same companies ensure that one creative voice integrates all these strands of innovation together, version after version, product after product.

Therefore, Steve Jobs, rest in peace. I always liked the Vachel Lindsay poem that went “Sleep softly, Eagle Forgotten, under the stone … To live in mankind is far more than to live in a name.” In this case, I can hope that Steve, warts and all, will live in innovators of the future. And that will be far more important to the computer industry and us than his legacy, however great, to Apple.

The Real Effects of Computing Brilliance

2011-09-26T17:06:00.000-07:00

Recently, L.E. Modesitt, a noted and highly perceptive (imho) science fiction author with a long background in business, government, education, and the like, wrote a blog post in which he decried the contributions of “brilliant minds, especially at organizations like Google and Facebook” to “society and civilization.” He noted Eric Schmidt’s remarks in a recent interview that young people from colleges were brighter than their predecessors, and argued that they had applied their brilliance in “pursuit of the trivial … [and] of mediocrity.” More specifically, he claimed that their brilliance has helped lead to the undermining of literary copyright and therefore of an important source of ideas, the creation of bubble wealth, the destabilization of the financial system, and potentially the creation of the worst US political deadlock since the Civil War.

Here, however, Mr. Modesitt steps onto my turf as well as his own. I can claim to have been involved with computers and their results since I was in the same freshman class in college as Bill Gates (Junior, thank you). I have had a unique vantage point on those results that combines the mathematical, the technological, the business, and even the political, and have enjoyed an intimate acquaintance with, and given long-running attention to, the field. I am aware of the areas in which this generation (as well as the previous two) has contributed, and their effects. And I see a very different picture.

Let’s start with Mr. Schmidt, who is a very interesting man in his own right. Having been at Sun Microsystems (but not heading it) in its glory days of growth, struggle, and growth (1983-1997), he became CEO of Novell as it continued its slide into oblivion, and then in 2001, a key part of the necessary transition of Google from its founders to successors who can move the company beyond one person’s vision. Only in the last role was he clearly involved with successful technology leadership, as Sun was a master at marketing average technology as visionary, while Novell’s influence on industry innovation by that point was small.

All that being said, Mr. Schmidt is speaking from the viewpoint of having seen all three of computing’s great creative “generations,” waves of college-bred innovators who were influenced by comparable generations of their societies, and who combined that social influence with their own innovations to create the stew that is the Web of today. I would divide those generations as follows:

1. Graduating from college in 1971-1981, the “baby boomer/socially conscious”.
2. Graduating from college in 1982-1995, the “individual empowerment Gen Xer”.
3. Graduating from college in 1996-2007, the “Web is the air we breathe” generation.

Where Mr. Modesitt sees only some of the effects of all three generations from the outside, I believe that Mr. Schmidt sees from within the way that generation 3 incorporates all the technological advances of the previous generations and builds on them. Therefore, where Mr. Modesitt is too negative, I believe that Mr. Schmidt is too positive. Generations 1 and 2 have created a vast substructure – all the technologies of computing and the Internet that allow Generation 3 to move rapidly to respond to, or anticipate, the needs of each segment of a global society. Because the easiest needs to tackle are the most “trivial”, those will be most visible to Mr. Modesitt. Because Generation 3 appears to move from insight to insight faster by building on that infrastructure, Mr. Schmidt sees them as more brilliant.

Let’s get down to cases: the Web, and Google. The years of Generation 3 were the years when computing had a profound effect on the global economy: it gave productivity more than a decade of 2.5-3% growth, a kind of “sunset re-appearance” of the productivity gains of the Industrial Revolution. This effect, however, was not due to Generation 3. Rather, Generation 2’s creation of desktop productivity software, unleashed in the enterprise in the early 1990s, drove personal business productivity higher, and the advent of the business-usable Web browser (if anything, a product of Generation 1’s social consciousness) allowed online sales that cut out “middleman” businesses, cutting product and service costs sharply.

But where Generation 3 has had a profound effect is in our understanding of social networks. Because of its theoretical work, we now understand in practical terms how ideas propagate, where are the key facilitators and bottlenecks, and how to optimize a network. These are emphatically not what we think of when we talk about the value of computing; but they are the necessary foundation for open source product development, new viral and community markets, and the real prospect of products and services that evolve constantly with our needs instead of becoming increasingly irrelevant (see my blog post on Continuous Delivery).

Has that made life better? I would say, that’s like asking if Isaac Newton’s calculus was a good idea, given that it has mainly been used to calculate trajectories for artillery. Ideas can be used for good or ill; and often the inventor or creator has the least control over their long-term use by others.

But let’s play the game, anyway. Undermining copyright? I would blame that more on Generation 2, which used the Web to further its libertarian ideologies that now form a key part of the “Web ethos”. Financial algorithms and bubble wealth? Sorry, that’s a product of the broader society’s reaction to the perceived (but not necessarily real) lawlessness and government “do-gooding” of the Vietnam era, so that law and order and “leave me alone” permitted business deregulation and lack of outside-the-box thinking about the risks of new financial technologies. Political deadlock? Same thing. “Leave me alone” created a split between Vietnam-era social consciousness and previous generations, while removing later generations from the political process except as an amoral or totally self-interested game. In both those cases, no computing Generation could have done very much about it except to enable it.

What are the most profound ideas and impacts of all three Generations’ “brilliant minds”? I would say, first, the idea of a “meta-product” or “meta-thing” (today’s jargon probably would call it a “virtual reality”), an abstraction of a physical good or process or person that allows much more rapid analysis and change of what we need. To a far greater degree than we realize, our products are now infused with computer software that not only creates difficulties in carrying out tasks for poor Mr. Modesitt, but also gives an array of capabilities that can be adjusted, adapted, evolved, and elaborated much more rapidly than ever before. Mr. Modesitt, I know, rightly resents the stupidities of his word processing software; but there is simply no way that his typewriter hardware by now would have the spelling and grammar checking capabilities, much less the rapid publishing capabilities, of that same word processor. It may be more work to achieve the same result; but I can guarantee that, good or bad, many of the results achieved could not have been attained forty years ago. Our computing Generations did that.

The second, I believe, is that of global information. The Web, as imperfect as it is, and as security-ridden as it has become, has meant a sea-change in the amount of important information available to most if not all of us. When I was in college, the information readily available was mostly in textbooks and bookstores, which did not begin to capture the richness of information in the world. The key innovation here was not so much the search engine, which gave a way to sort this information, but the global nature of the Web user interface, the embedding of links between bits of information, and the invention of “crowdsourcing” communities, of which Wikipedia is by far the most successful as a pointer from information area to information area. Say all you like about the lies, omissions, and lack of history behind much of this global information pool; it is still better than a world in which such information appeared not to exist. Our computing Generations did that, too.

The third idea, the one I mentioned above, is understanding of, and ability to affect the design of, social networks. Facebook is only the latest variant on this theme, which has shown up in Internet groups, bulletin boards, open source and special-interest communities, smartphone and text-message interfaces, file sharing services, blogs, job sites, and other “social networks.” The Arab Spring may be an overhyped manifestation of its power; but I think it is fair to say that some of the underlying ideas about mass peaceful regime change could never have been global or effective without computing’s “brilliant minds.” I give Generation 3 most of the credit for that.

So here’s an example of why I disagree with Mr. Modesitt: a couple of years ago, I sat down to understand climate change. At first, the best information came from old-style books, mostly from the local public library. But soon, I was using www.climateprogress.com and neven1.typepad.com to dig deeper, to see the connections between what the books were saying, and then Wikipedia to point me to research articles and concept explanations that allowed me to put it into a coherent whole, adding my own technology, economic, business, political, and government experience and knowledge to fill in the gaps. No set of books, then or now, could possibly have given me enough information to do this. No phone network could possibly have shown me the people and institutions involved. No abacus could have put the statistics together, and no typewriter or TV could have given me such a flexible window to the information. The three great ideas of computing’s three Generations made such a thing possible.

So, no, I don’t believe that much of what this Generation of brilliant minds has been working on is at all trivial or mediocre, despite appearances – not to mention the contributions of the other two Generations – nor has its effect been predominantly bad. Rather, it reminds me of the remark in Dilbert that it is the work of a few smart, unprepossessing drones that moves the world forward, hindered rather than helped by their business or the greater society. Despite society’s – and sometimes their own – best efforts, these Generations’ blind pursuit of “the new new thing”, I say, has produced something that overall is good. In every sense.

Human Math and Arctic Sea Ice

2011-09-12T08:41:00.000-07:00

Over the last year and a half, instead of following baseball and football, I have been following the measurements of Arctic sea ice. Will it entirely melt away at minimum? If so, when? Will it continue to melt past that point? Will the Arctic reach the point of being effectively ice-free year-round? If so, when? Will Arctic sea ice set a new record low in extent this year? In area? In volume? By the way, the answers to the last three questions in 2011 are already, by some measures, yes, yes, and yes.

In the process, I have become entirely addicted to the Arctic Sea Ice web site (neven1.typepad.com), the Arctic sea ice equivalent of a fantasy football league. There, I and other scientist and math wannabees can monitor and pit themselves against near-experts, and plunder much of the same data available to scientists to try our own hand at understanding the mechanics of the ebb and flow of this ice, or to do our own predictions. Periodically, scientific research pops up that deepens our understanding in a particular area, or challenges some of our assumptions. Strange weather patterns occur as we near minimum, making the final outcome unpredictable. In fact, this year we still don’t know whether the game is over for the year or not – whether we are going into overtime.

But what disturbs me about the glimpse I am getting into science, this late in life, is that I keep seeing half-veiled glimpses of what I might call “data snow blindness,” not just from us newbies, but from some of the scientific research noted in the site. By that I mean that those who analyze Arctic sea ice data tend to get so wrapped up in elaborating the mathematical details of, say, a decrease in extent at minimum and how short-term and long-term changes in sea-ice albedo, the North Atlantic Dipole Anomaly, cloud cover, and increased Arctic surface temperatures can predict this minimum, that they seem to forget what extent data really conveys and does not convey, and how other factors not yet clearly affecting extent data should play an increasing role in future extent predictions. They keep talking about “tipping points” and “equilibria” and “fundamental changes in the system” that do not appear to be there. And so, what surfaces in the media, with few if any exceptions, gives an entirely misleading picture of what is to come.

So let me lay out my understanding of what the data is really saying, and how people seem to be going wrong. Bearing in mind, of course, that I’m not the greatest mathematician in the world, either.

The Model
I find that the easiest way to visualize what’s going on with Arctic sea ice is to think of a giant glass full of water, with an ice cube floating on top that typically almost but not quite fills the top of the glass. In the summer, the sun shines on the ice and top of the water, adding heat on top; in winter, that heat turns off. In summer and winter, the water below the ice is being heated, although a little less so in winter.

This does not quite complete the picture. You see, water is constantly flowing in from the south on one side and out to the south on another. As it hits the ice, it freezes, and moves throughout the cube until it exits on the other side – an average of about five years, apparently. These currents, too, act to break the ice apart as it is melting in summer, so (especially on the edge) there are lots of little “cubelets”. Where the ice is solidly packed together in the glass, then “area” – the amount of ice we see from above – is the same as “extent” – the region in which there is enough ice to be easily detectable. But where there are cubelets, then extent is much greater than area – as much as 70% more, according to recent figures.

One more important point: the depth of Arctic sea ice is not at all the same everywhere. This was true when the first nuclear sub approached the Pole underwater decades ago, and it is true as the Polarstern measures depth from above this year. The age of the ice, the point in the year in which it first froze, and where it is in relation to the Pole all factor in; but even in small regions, “thickness” varies. Generally, less than half the ice is almost exactly the same thickness as the average, with lots above and lots below. I find it useful to think of a normal curve of sea-ice thickness more or less peaking at the average.

In a state of “equilibrium” such as apparently has existed for perhaps 5-10 million years, the Arctic cube fills at least 70% of the Arctic in summer and just about all of the Arctic in winter. In summer, the sun melts the top and sides of the cube. However, since we only see the top, which only peeks out a little bit from the top of the water, we don’t see the way that the bottom of the cube rises in the water – the “balance point” between “salt” water below and “no-salt” ice above goes up. In the same way, in winter, we don’t see the way that the balance point goes down deeper.

Now suppose the globe starts warming. As it happens, it warms faster in the Arctic than in the equator, and it warms in both the near-surface air and in the water that is flowing through the Arctic underneath the ice, year-round. Estimates are that the air temperatures in summer in or near the Arctic have reached more than 10 degrees higher, and the rate of rise is accelerating; likewise, the ocean temperature is up more than a degree, and the rate of rise is accelerating. What we would expect to happen is that there is less and less ice during the summer – and also less and less ice during the winter, because the water underneath that cube is warmer, and the “balance point” is higher. And the rate we would be losing ice would be accelerating.

The Measures
Now how do we measure what’s going on with the ice? We point our satellites at the Arctic, and we measure in three ways: extent, area, and volume. Extent and area we have explained above; volume is area times average thickness.

To get extent, we divide the Arctic into regions, and then into subregions. If, say, more than 15% of the subregions in a region show ice, then that’s part of the ice extent. The area is the extent times the percent of subregions that show all ice (this is a very simplified description). There are difficulties with this, such as the fact that satellites tend to find it difficult to distinguish between melt ponds at the top of the ice during melting and melting of the ice “all the way down”; but it appears that those cause only minor misestimations of “actual” extent and area. Storms that wash over “cubelet” ice can cause temporary fairly large downward jumps in what the satellite perceives as ice, and hence in area; but again, this goes away as the storm subsides. All in all, the measuring system gives a pretty good picture (by now) of what’s going on with area and extent.

Volume is much more difficult to get at – because it’s very hard to get an accurate picture of thickness from a satellite. Instead, we create a model of how thick each part of the ice should be at any time of the year, and then check it out “on foot”: walking around on the ice (or icebreaking) and testing.

Now here’s what most people either don’t know or don’t think about: that model has been tested against samples almost constantly since it was first developed decades ago, and it is pretty darn accurate. When it has said, this year, that ice is as little as a meter thick near the North Pole, vessels went out and, lo and behold, found a large patch of ice 0.9 meters thick near the North Pole. That’s clearly not something that many scientists were anticipating six years ago as likely – but the model was predicting it.

It’s the Volume, Stupid
All right, let’s get back to the model. Suppose we have equilibrium, and we start an accelerating rise in air and ocean temperatures. What changes would we expect to see in real extent, area, and volume year to year?

Well, think about what’s happening. The water heating is nibbling away at the “balance point” from beneath, and at the edges. The air heating is nibbling away at the top of the ice during the summer, and expanding the length of the summer a bit, and heating the surface water at the edges a bit. So it all depends on the importance of water heating vs. air heating. If water heating has little effect compared to air heating, what happens to volume is not too far from what happens to area and extent: They start plummeting earlier and go farther down, but during the winter, when just about the whole basin gets frozen again, they go back to about where they were – a little less, because the air is more often above freezing at the very edge of the ice even in the depth of winter.

Now suppose water heating has a major role to play. This role shows up mostly as “bottom melt”, it shows up year-round in about the same amounts, and it shows up almost completely, until nearly all ice is melted, in volume. So what you’ll see, when you look at extent, area, and volume, is that volume goes down for quite a while before it’s really clear that area and extent are changing, and then area and extent start going down at minimum but not much if at all at maximum, and then the melt moves the balance point up and up and some of that normal curve starts reaching “negative thickness” as some of that melt reaches the surface of the ice and that means area starts moving down fast, and extent with it, and then in a year or two half of the ice reaches “negative thickness” and it seems like the area is cut in half and the extent by one-third, and the same thing happens the next year, while the rate of volume decrease actually starts slowing, and then in two more years there is effectively less than 1% of the ice area at maximum showing up at minimum, and less than 15% of the extent.

This is where lots of folks seem to fail to understand the difference between a measure and what it’s measuring. If you look at the volume curve, until the last few years before it reaches zero it seems to be accelerating downward – then it starts flattening out. But what’s actually happening is that some parts of the ice have reached the point where they’re melting to zero – others aren’t, because there’s a great variation in ice thickness. If we represented those parts of the ice that have melted as “negative thickness”, then we would continue to see a “volume” curve accelerating downward. Instead, we represent them as “zero thickness”, and the volume delays going to zero for four or five years.

So what are our measures telling us about the relative importance of water and air heating? First of all, the volume curve is going down, and going down at an accelerating rate. From 1980 (start of the model) to 2005, average thickness at minimum area (a good proxy for thickness at ice minimum), went from more than 3 meters to a little more than 2 meters. From 2006 to 2010, it has gone from there to a little more than 1 meter. If it were to follow the same path until 2013 or 2014, then “volume” would go to zero then at minimum, with the model’s measures of volume going to zero perhaps 3 years later, as explained in the last paragraph.

But there’s another key fact about volume: volume at maximum went down by about the same amount (15 million cubic meters) from 1980 to 2010 as volume at minimum (14 million cubic meters). In other words, instead of volume springing back during the winter almost to what it was 30 years ago, it’s almost exactly tracking the loss of volume the rest of the year. The only way this happens is if water melting from the bottom is a major part of the loss of volume from year to year.

Let’s sum up. The samples show that volume is going down at an accelerating rate throughout the year. The accelerating drop in volume shows that bottom melt is driving accelerating loss of Arctic sea ice year-round. Thus, we can predict that area and extent will soon take dramatic drops not obvious from the area and extent data, and that these will approach zero in the next 7 years or so. In other words, the key to understanding what’s going on, and what should happen next, is as far more the measure of volume – rightly understood – than area or extent.

Where’s The Awareness?
And yet, consistently, scientists and wannabes alike seem to show a surprising degree, not just of disagreement with these projections, but also of a seeming lack of awareness that there should be any issue at all. Take some of the models that participants in Arctic Sea Ice are using to predict yearly minimum extent and area. To quote one of them, “volume appears to have no effect.” The resulting model kept being revised down, and down, and down, as previous years that had shorter summers, slower rates of melt for a given weather pattern, and less thin ice, proved bad predictors.

Or, take one of the recent scientific articles that showed – a very interesting result – that if, for example, the Arctic suddenly became ice free right now, it would snap back to “equilibrium” – the scientist’s phrase – within a few years. But why was the scientist talking about “equilibrium” in the first place? Why not “long-term trend”? It is as if the scientist was staring at the volume figures and saying, well, they’re in a model, but they’re not really nailed down, are they, so I’ll just focus on area and extent – where the last IPCC report talked about zero ice in 2100, maybe. So suppose we substituted the volume’s “long-term trend” for “equilibrium”, what would we expect? Why, that a sudden “outlier” dip in volume, area, or extent would return, not to the same level as before, but to where the long-term accelerating downward curve of projected volume would say it should return. And, sure enough, the “outlier” volume dip in 2006-2007 and extent dip in 2007 returned only partway to the 2005 level – in 2009 -- before resuming their decrease at an accelerating rate.

And that brings us to another bizarre tendency: the assumption that any declines in area and extent – and, in the PIOMAS graph, in volume – should be linear. Air and ocean temperatures are rising at an accelerating rate; so an accelerating downward curve of all three measures is more likely than a linear decline. And, sure enough, since 2005, volume has been consistently farther and farther below PIOMAS’ linear-decline curve.

Then there’s “tipping point”: the idea that somehow if Arctic sea ice falls below a certain level, it will inevitably seek some other equilibrium than the one it used to have. Any look at the volume data tells you that there is no such thing as a tipping point there. Moreover, a little more thought tells you that as long as air and ocean temperatures continue their acceleration upwards, there is no such thing as a new equilibrium either, short of no ice year-round – which is a kind of equilibrium, if you don’t consider the additional heat being added to the Arctic afterwards. Sooner or later, the air and ocean temperature at all times in winter reaches above minus 4 degrees C, so that water can’t expel its salt and freeze. And that’s not hypothetical; some projections have that happening by around 2060, if not before.

And, of course, there’s “fundamental change in the system”, which implies that somehow, before or after now, the Arctic climate will act in a new way, which will cause a new equilibrium. No; the long-term trend will cause basic changes in climate, but will not be affected by them in a major way.

Follow-On Failures
The result of this data snow blindness is an inability to see the possibility – and the likelihood -- of far faster additional effects that spread well beyond the Arctic. For instance, the Canadian Prime Minister recently welcomed the idea that the Northwest Passage is opening in late summer, because Canada can skim major revenues off the new summer shipping from now on. By the data I have cited above, it is likely that all the Arctic will be ice-free in part of July, August, and September by 2020, and most of the rest of the year by 2045, and very possibly by 2035. Then shippers simply go in one end of the Arctic via the US or Russia, and the other end via the Danes (Greenland) or Russia, and bypass Canada completely.

And then there’s the Greenland land ice. Recent scientific assessments confirm that net land ice loss has doubled each decade for the last three decades. What the removal of Arctic sea ice means is the loss of a plug that was preventing many of the glaciers from sliding faster into the sea. Add a good couple of degrees of increased global warming from the fact the Arctic sea ice isn’t reflecting light back into space any more, and you have a scenario of doubling net Greenland land ice loss for the next three decades, as well. This, in turn, leads to a global sea ice rise much faster than the 16 feet (or, around the US, maybe 20 feet) rise that today’s most advanced models project by assuming that from now on, Greenland land ice loss will be rising linearly (again, why linearly? Because they aren’t thinking about the reality underlying the data). And, of course, the increased global warming will advance the day when West Antarctica starts contributing too.

The Net-Net: The Burden of Proof

As a result of these instances of data snow blindness, I continually hear an attitude that runs something like this: I’m being scientifically conservative. Show me why I shouldn’t be assuming a possibility of no long-run change, or a new equilibrium short of year-round lack of Arctic ice, or a slow, linear descent such as shows up in the area and extent figures.

What I am saying -- what the data, properly understood, are saying to me – is that, on the contrary, the rapid effects I am projecting are the most likely future outcomes – much more rapid declines in area and extent at minimum in the very near future, and an ice-free Arctic and accelerating Greenland land ice loss over the 20-30 years after. Prima facie, the “conservative” projections are less likely. The burden of proof is not on me; it’s on you, to disprove the null hypothesis.
So what about it, folks? Am I going to hear the same, tired arguments of “the data doesn’t show this yet” and “you have to prove it”, or will you finally start panicking, like those of us that understand what the data seem to be saying? Will you really understand which mathematics applies to the situation, or will you entrance yourselves with cookbook statistics and a fantasy world of “if this goes on”? It may be lonely out here; but it’s real.

The Third Age of Software Development: Andreessen, Thoughtworks, and the Agile World

2011-09-07T18:25:00.000-07:00

Recently there passed in front of me two sets of thoughts: one, an article by Marc Andreessen (once a Web software developer, now a VC) in the Wall Street Journal about how software is “eating the world”; one by a small development tools/services company called Thoughtworks about a new technique called Continuous Delivery.

Guess which one I think is more important.

Still, both make important points about changes in computing that have fundamental effects on the global economy. So let’s take them one at a time, and then see why I think Continuous Delivery is more important – indeed, IMHO, far more important. A brief preview: I think that Continuous Delivery is the technique that takes us into the Third Age of Software Development – with effects that could be even more profound than the advent of the Second Age.

To Eat the World, First Add Spices

A very brief and cursory summary of Andreessen’s argument is that companies selling software are not only dominant in the computer industry, but have disrupted (Amazon for books, Netflix for movies) or are “poised to disrupt” (retail marketing, telecom, recruiting, financial services, health care, education, national defense) major additional industries. The result will be a large proportion of the world’s industries and economy in which software companies will be dominant – thus, software will “eat the world”.

My quarrel with this argument is not that it overstates the case, but rather that it understates it. What Andreessen omits, I assume in the interest of conciseness, is that in far more industries and companies, software now is the competitive differentiator between companies, even though the companies’ typical pedigree is primarily “hardware” or “services”, and the spending on software is a small percentage of overall costs.

My favorite examples, of course, are the company providing audio and video for stadiums, which relies on software to synchronize the feeds to the attendees better than the competition, and Boeing, which would not be able to provide the differentiating cost-saving use of composites were it not for the massive amounts of software code coordinating and monitoring the components of the Dreamliner. In health care, handling the medical health record and support for the new concept of the medical home requires differentiating software to glue together existing tools and health care specialists, and allow them to offer new types of services. It isn’t just the solo software company that is eating the world; it’s the spice of software differentiation in the mostly non-software businesses that is eating these companies from within.

So Andreessen’s article signals a growing awareness in the global economy of the importance not just of computing, but of software specifically, and not just as a separate element of the economy, but as a pervasive and thoroughly embedded aspect of the economy whose use is critical to global economic success or failure. How far we have come, indeed, from the point 30-odd years ago when I suggested to Prof. Thurow of MIT that software competitive advantage would allow the US to triumph over the Japanese economic onslaught, and he quite reasonably laughed the idea to scorn.

I do have one nit to pick with Andreessen. He trots out the assertion, which I have heard many, many times in the past 35 years, that companies just can’t find the programming skills they need – and this time, he throws in marketing. Every time people have said this, even at the height of the Internet boom, I have observed that a large body of programmers trained in a previous generation of the technology is being overlooked. That sounds as if it makes sense; but, as anyone should know by now, it is part of the DNA of any decent programmer to constantly acquire new knowledge within and beyond a skill set – good programmers are typically good programmers in any arena, from Web-site design to cloud “program virtualization”.
Meanwhile, I was hearing as late as a year ago about the college pipeline of new programmers being choked off because students saw no future jobs (except in mainframes!), and I can attest that available, technically savvy marketers in the Boston area – a well-known breeding ground – are still thick on the ground, while prospective employers in the Boston area continue to insist that program managers linking marketing and programming have at least five years of experience in that particular job. It just won’t wash, Mr. Andreessen. The fault, as is often the case, is probably not mainly in the lack of labor, but in companies’ ideological blinders.

The Idea of Continuous Delivery

Continuous Delivery, as Thoughtworks presents it, aims to develop, upgrade, and evolve software by constant, incremental bug fixes, changes, and addition of features. The example cited is that of Flickr, the photo sharing site, which is using Continuous Delivery to change its production web site at the rate of ten or more changes per day. Continuous Delivery achieves this rate not only by overlapping development of these changes, but also by modularizing them in small chunks that still “add value” to the end user and by shortening the process from idea to deployment to less than a day in many cases.

Continuous Delivery, therefore, is a logical end point of the whole idea of agile development – and, indeed, agile development processes are the way that Thoughtworks and Flickr choose to achieve this end point. Close, constant interaction with customers/end users is in there; so is the idea of changing directions rapidly, either within each feature’s development process or by a follow-on short process that modifies the original. Operations and development, as well as testing and development, are far more intertwined. The shortness of the process allows such efficiencies as “trunk-based development”, in which the process specifically forbids multi-person parallel development “branches” and thus avoids their inevitable communication and collaboration time, which in a short process turns out to be greater than the time saved by parallelization.

I am scanting the many fascinating details of Continuous Delivery in the real world in order to focus on my main point: it works. In fact, as agile development theory predicts, it appears to work better than even other agile approaches. Specifically, Thoughtworks and Flickr report:

• A much closer fit between user needs and the product over time, because it is being constantly refreshed, with end-user input an integral part of the process;
• Less risk, because the small changes, introduced into production through a process that automatically minimizes risk and involves operations, decrease both the number of operational bugs and the time to identify and fix each; and
• Lower development costs per value added, as loosely measured by the percentage of developed features used frequently as opposed to rarely or never.

The Ages of Software Development

I did my programming time during the First Age of software development, although I didn’t recognize it as such at the time. The dominant idea then was that software development was essentially a bottleneck. While hardware and network performance and price-performance moved briskly ahead according to Moore’s Law, software developer productivity on average barely budged.

That mattered because in order to get actual solutions, end users had to stand in line and wait. They stood in line in front of the data center, while IT took as long as two years to generate requested new reports. They stood in line in front of the relatively few packaged application and infrastructure software vendors, while new features took 9 months to arrive and user wish lists took 1 ½ years to satisfy. They stood in line in front of those places because there was nowhere else to turn, or so it seemed.

I noted the death of the First Age and the arrival of the Second Age in the late ‘90s, when a major Internet vendor (Amazon, iirc) came in and told me that more than half of their software was less than six months old. Almost overnight, it seemed, the old bottlenecks vanished. It wasn’t that IT and vendor schedules were superseded; it was that companies began to recognize that software could be “disposable.” Lots of smaller-scale software could be developed independent of existing software, and its aggregate, made available across the Internet by communities, as well as mechanisms such as open source, meant that bunches of outside software could be added quickly to fill a user need. It was messy, and unpredictable, and much of the new software was “disposable” wasted effort; but it worked. I haven’t heard a serious end user complaint about 2-year IT bottlenecks for almost a decade.

Andreessen’s “software eating the world”, I believe, is a straightforward result of that Second Age. It isn’t just that faster arrival of needed software from the new software development approach allows software to start handling some new complex tasks faster than physical products by themselves – say, tuning fuel mixtures constantly for a car via software rather than waiting for the equivalent set of valves to be created and tested. It is also that the exponential leap in the amount of the resulting software means that for any given product or feature, software to do it is becoming easier to find than people or machines.

However, it appears clear that even in the Second Age, software as well as physical products retain their essential characteristics. Software development fits neatly into new product development. New products and services still operate primarily through more or less lengthy periods of development of successive versions. Software may take over greater and greater chunks of the world, and add spice to it; but it’s still the same world.

The real-world success of Continuous Delivery, I assert, signals a Third Age, in which software development is not only fast in aggregate, but also fast in unitary terms – so fast as to make the process of upgrade of a unitary application by feature additions and changes seem “continuous”. Because of the Second Age, software is now pervasive in products and services. Add the new capabilities, and all software-infused products/services -- all products/services – start changing constantly, to the point where we start viewing continuous product change as natural. Products and services that are fundamentally dynamic, not successions of static versions, are a fundamental, massive change to the global economy.

But it goes even further. These Continuous-Delivery product changes also more closely track changes in end user needs. They also increase the chances of success of introductions of the “new, new thing” in technology that are vital to a thriving, growing global economy, because those introductions are based on an understanding of end user needs at this precise moment in time, not two years ago. According to my definition of agility – rapid, effective reactive and proactive changes – they make products and services truly agile. The new world of Continuous Delivery is not just an almost completely dynamic world. It is an almost Agile World. The only un-agile parts are the rest of the company processes besides software development that continue, behind the scenes of rapidly changing products, to patch up fundamentally un-agile approaches in the same old ways.

And so, I admit, I think Thoughtworks’ news is more important than Andreessen’s. I think that the Third Age of Software Development is more important than the Second Age. I think that changing to an Agile World of products and services is far, far more important than the profound but more superficial changes that the software infused in products and services via the Second Age has caused and will cause.

Final Thoughts and Caveats

Having heralded the advent of a Third Age of Software Development and an Agile World, I must caution that I don’t believe that it will fully arrive any time in the next 2-3 years, and perhaps not for a decade, just as it took the Second Age more than a decade to reach the point of Andreessen’s pronouncements. There is an enormous amount of momentum in the existing system. It took agile development almost a decade, by my count, to reach the critical mass and experience-driven “best practices” it has achieved that made Continuous Delivery even possible to try out. It seems logical that a similar time period will be required to “crowd out” other agile new-product development processes and supersede yet more non-agile ones.

I should also add that, as it stands, it seems to me that Continuous Delivery has a flaw that needs to be worked on, although it does not detract from its superiority as a process. Continuous Delivery encourages breaking down features and changes into smaller chunks that reflect shorter-term thinking. This causes two sub-optimal tendencies: features that “look less far ahead”, and features that are less well integrated. To encapsulate this argument: the virtue of a Steve Jobs is that he has been able to see further ahead of where his customers were, and that he has been able to integrate all the features of a new product together exceptionally well, in the service of one, focused vision rather than many, diffused visions.

Continuous Delivery, as it stands, pushes the bar more towards more present-focused features that are integrated more as a politician aggregates the present views of his or her constituents. Somehow, Continuous Delivery needs to re-infuse the new-product development process with the ability to be guided at appropriate times by the strong views of an individual “design star” like Steve Jobs – else the Agile World will lose some of its ability to deliver the “new, new thing.” And it would be the ultimate irony if agile development, which aims to release programmers from the stultifying, counter-productive constraints of corporate development, wound up drowning their voices in a sea of other (end-user) views.

But who cares? In this latest of bad-news seasons, it’s really nice to look at something that is fundamentally, unreservedly good. And the Agile World brought about by Thoughtworks’ Continuous Delivery’s Third Age of software development, I wholeheartedly believe, is Good News.

Analysis of the War in Libya

2011-08-26T11:11:00.000-07:00

I find myself extremely frustrated by the coverage and analysis of the war in Libya. The narrative is of an initial surge of revolt along the coast, with counterattacks by the regime eliminating all western revolt except in Misrata, which barely survived, followed by months of stalemate between east and west, and then suddenly an attack on Tripoli from the interior which has now effectively succeeded. I admit, I am an armchair general, but this seems so clearly wrongheaded to me that even I can do better.

Here is my analysis of the war: the initial revolt was, in the east, so comprehensive as to give the rebels a strong base to west of Benghazi. That was rapidly complemented by control of the borders with Egypt and Sudan. However, to the west and in the rest of the south, the regime was able to counterattack because risings were less strong and uncoordinated. As a result, effectively, the regime was able to control not only that stretch of the seacoast, but, more importantly, the southern borders – which meant that it could buy mercenaries from the starving men of Chad in whatever numbers to supplement existing troops. The counterbalance for the rebels along the coast was NATO bombing, which limited the ability of regime troops to attack within the cities of the coast.

This balance was precarious, for the regime. They needed a guaranteed flow of oil for their tanks. They needed recruits to replace the steady flow of defectors. The rebels had some money worries, but as long as Misrata was in play there would be no effective attacks towards Benghazi, and even if Misrata had fallen it is doubtful that the regime could have pressed home attacks towards Benghazi with the extra men in the face of NATO bombings. No, it was a “stalemate” that in all likelihood could be broken by the rebels but not by the regime.

At this point, a second point of insurrection emerged in the west, on a line from deep in the interior on the Tunisia border towards Gharyan midway along the key Tripoli-to-southern-border road. It was by no means a neat, clean, fully controlled territory; but the playing field was effectively tilted against the regime in that area, since the rebels from Tunisia with local support could muster more people to attack more areas than the regime could send troops to counter them, through a long stretch of tough terrain, in order to drive them back into Tunisia. However, the longer the line from the Tunisian border, the shorter the line to Tripoli, and so there was a reasonable prospect that the line between the two would stabilize before it threatened north-south shipments of oil and mercenaries.

The point when it was clear that this push was breaking the stalemate was well over two months ago, when the rebels managed to start attacking the towns that were the last ones before Gharyan. The moment they could do that, they could threaten the north-south flow, and unless the regime somehow managed to counterattack to send the Nafusa push beyond the range of Gharyan, that threat would remain.

But there was another, equally significant development at about the same time, even further south. The rebels, attacking across a wide swath of desert near the southern border, managed to take two of the towns relatively near that border. As a result, the flow of mercenaries was bound to be badly disrupted.

At this point – well over two months ago – the situation along the coast becomes relevant again. Because the rebels attacking via Nafusa were often only hundreds strong. It may seem unbelievable that regime could not send enough troops to the interior to counterattack effectively; but the fact on the ground is, they did not. Why?

The easy answer is, they were tied down by NATO bombings. However, it is clear that NATO bombings played very little role in the fighting in the interior, neither in the Nafusa region nor near the southern border. Rather, the NATO bombings served only to hinder the regime’s ability to attack, and only along the seacoast.

The correct answer, I believe, is that the regime had neither the personnel to spare nor a way of sending enough of them. The choking off of mercenaries meant that the rebels could replenish their troops after battlefield losses in the interior, while the regime could not. Moreover, the regime could not transfer troops from various areas along the coast, because they were pretty much all tied up facing the rebels along the coast.

It is at this point that the “stalemate” along the coast begins to matter. If the regime had drawn its troops from the east, around Sirte, they would have a long way to travel; they would have to skirt Misrata, with the danger of NATO bombing; and they were fully engaged with Misrata and with Brega, the more or less fluid border between the regime and the eastern rebels in Benghazi. That left the troops in the west, and what the current attacks show is that there were enough regime troops short of Tripoli to repel any attack along the coast, but not enough to counterattack into the interior.

That left Tripoli. However, again, it seems clear that Tripoli troops were fully engaged, in the east toward Misrata, and to some extent in the south to protect the nearby airport and the initial stages of the road and oil pipeline south. In effect, the line of the coast between Tripoli and the Tunisian border could be attacked at any point from the interior, and the regime would be slow to respond and would have little to spare.

Nevertheless, according to reports, the actual attack from the interior was so feeble as to invite disbelief. The number of attackers was about 300; they drove to their target in the equivalent of Jeeps, having little military ordnance; and they were a combination of previously apparently separate smaller groups. It is as if those members of the Harvard Class of 2001 who attended this year’s class reunion had decided to attack Stamford, Connecticut, and had all just grabbed a gun and driven down in their BMWs – and succeeded.

Still, to anyone looking at the situation in the middle of June with a critical eye, it should have been obvious that the rebels had a very good chance of succeeding in whatever they tried. This is because the regime at that point had failed in its counterattack – if it even tried. Reporters didn’t notice it; analysts, preoccupied with the NATO bombings, didn’t notice it; but it was vitally important that the regime not only kick the Nafusa rebels out of the towns they were attacking, but attack the towns beyond them, as well; and the regime didn’t do that.

After mid-June, therefore, the stalemate was definitively over and the end of the regime was only a matter of time. Still, it could have been a matter of a fairly long time. The rebel push could have run out of gas around Gharyan, and it might have been months to a year before the full effects of the loss of mercenaries and oil were felt.

It did not take a year because the rebels at this point, as noted above, took a chance. They basically picked a point on the coast between Tripoli and Tunisia, and the moment they fully took the town before Gharyan, instead of sending their full weight against Gharyan, they immediately sent the bulk of their troops in a headlong dash at Al Zawia, the biggest town on the coast west of Tripoli, and closer to Tripoli than the Tunisian border.

This move reminds one of the remark by Steven Brust, the fantasy author, that battles are won when one side fails to make a mistake at a critical moment. There were certainly endless mistakes by the strategists on either side, and especially the tendency of the rebels to send their troops in to attack positions carefully prepared for maximum casualties by the regime, and the tendency of the regime to expose its troops to NATO bombings unnecessarily, instead of just tying up the rebels inside the towns, where NATO would find it hard to distinguish friend from foe. I believe that once the Nafusa offensive reached a certain point, and the southern border was contested, the rebels were pretty much bound to win; but attacking immediately after the coast was in range was what meant that the war would be won now.

The reason the attack on Al Zawia was successful, and that it was then followed by success on all coastal fronts, was that the attack started a cascade of successes in which regime troops surrendered piecemeal, decreasing the regime’s manpower, and the rebels began to decrease the number of fronts on which they had to engage, allowing that manpower to be funneled into the remaining areas. Thus, the eventual success at Al Zawia meant that troops shipped in from the eastern front, where pressure had eased, could supplement troops from the interior in the attack on Tripoli, and cut off any troops between Al Zawia and the Tunisian border. Before, an attack on the coast over that border had failed. Now, troops have swept along the coast to that border and control almost all of it.

However – and here again commentators seem to me to have entirely missed the mark – the war is by no means over. It is not only that Tripoli is by no means secure; it is that there remain pockets of resistance along the coast (the remaining pockets of resistance in the interior, while huge, represent very little regime military manpower, and certainly not enough to threaten the rebels on the coast in the foreseeable future).

The two key points, right now, are Zuwara and Ben Jawad. Zuwara is pretty much the only holdout along the coast between Tripoli and Tunisia [update: It has now been taken, but attack from the interior is still holding troops in play]. Once that is secure, the rebels will add their manpower there to the troops in Tripoli, tilting the balance in favor of the rebels. Ben Jawad is the only town still contested on the coast to the east of Tripoli. Once that is gone, the rebels from Misrata and those from Benghazi will join hands around Sirte. But based on past experience, both of those events can happen tomorrow, or a month from now; and therefore, the final fall of Tripoli and Sirte could be as much as 2 months down the road, in a worst case scenario.

Nevertheless, I believe that there is just about nothing the regime can do, now, to restore any kind of stalemate. Their troops will keep becoming fewer, the rebels, greater, because of inevitable regime surrenders and lack of any way to add more regime troops. The fact that the regime will run out of supplies is only relevant in that commanders will see that and surrender sooner rather than later.

So let’s list the ways that reporters and analysts would have you think, and the ways I disagree. “The rebels were innocents led by inexperienced and flawed commanders that only won because NATO bombed the hell out of the regime” – flatly wrong. The rebels for the last 6 months have had a steady, smart long-term strategy that was just as important as NATO bombings, and that won the war in the interior, where NATO had very little effect.

“The war was fought and won mostly on the coast.” Wrong again; the stalemate was ended decisively in the interior. “There was a stalemate from March until early August”; nope, there was an increasingly tilted semi-stalemate until mid-June, when the rebels clearly established that they would win. “The war is over”. No; as Yogi Berra once said, “it ain’t over until it’s over”; and the news reports clearly establish that there is still plenty of fighting going on.

This experience has taught me one thing, very clearly; most of the analysts I see and hear are pretty much useless in certain cases, reporters or not. There were analysts and statesmen still talking about the importance of NATO bombing civilians and of the stalemate in the region as little as three weeks ago, while members of Congress questioned the President’s power to bomb Libya as if it mattered any more. Meanwhile, Wikipedia was pretty much the only source I needed to see that things were different. Folks, what’s wrong with this picture? How could it be that, almost universally, the authorities we depend on screwed this up?

I’m only an armchair general, and not a very good one, so I don’t know the answer to that question. But I can tell you, the next time there’s a conflict out there, I’m going to Wikipedia first.

Giving the Devil His Due -- Then Naming Him Evil

2011-08-03T17:03:00.000-07:00

Careful readers of one of my previous blog posts will note that I said nothing about the validity of the complaint of scientific misconduct against the “Arctic scientist muzzled.” I had assumed that better commentators would cover that subject quite well. However, thinking about it, it occurs to me that I have a unique perspective on the specifics of the case. So I’d like to lay out what I think happened – and what I conclude it means.

Setting the Stage

Monnett, the scientist, is an employee of the Mining and Materials Service (recently renamed) in the Interior Dept. As it happened, in the summer of 1969 I was an intern at the Office of the Under Secretary of Interior in Washington, so I got a birds’-eye view of Interior, later supplemented by a senior thesis on its workings.
Back then, Interior had two types of Services: “foxes” (the Army Corps of Engineers and MMS) and “chickens” (Fish and Wildlife Service). In very broad terms, foxes would tend to favor the interest of business, while chickens would tend to favor the interest of environmentalists. At the time, new environmental departments were being formed, and the thought was that placing these in Interior would tilt the department more towards environmentalism, both in the department and in its representation in Congress, so that the overall result was a stronger US environmental policy.

As it happened, Nixon chose to place EPA and NOAA in a separate agency to cut down Interior’s power, because he felt Secretary Hickel was being politically disloyal (Hickel himself confirmed this to me). It appears, therefore, that Interior remains today much the same as it was then – with fox and chicken departments, and MMS being one of the foxes. Monnett, his investigators, and his bosses and co-workers all seem to be employees of the MMS.

Monnett had an odd function at the MMS. As part of an earlier compromise between Alaskan native communities and oil companies seeking permits to drill offshore, he was to monitor animal activities vital to these natives – specifically, the bowhead whale migration – to see if oil drilling had any effect on them. It was important to the oil companies and to MMS that drilling not be interrupted for no reason; it was important to the natives that their means of livelihood not be disrupted, and therefore Monnett’s primary responsibility was supposed to be to determine if it was. At the same time, Monnett as a scientist was supposed to provide and analyze “just the facts”.

The Complaint

What I am saying here is pure conjecture, but is my most positive take as to what the person(s) who generated the complaint against Monnett was thinking.

In the years shortly before Monnett’s paper, oil company engineers and/or geologists were probably making complaints about Monnett. He appeared to be changing the statistics for determining problems with bowhead whale migration in the middle of the game, in favor of finding problems. There was friction with his superiors, who were questioning his conclusions. The engineers and/or geologists were credible complainers, who could be expected to understand his math and his experiments. And then the paper arrived.

To the complainer, looking at the paper, it appeared odd. For one thing, it appeared that most of the data in the paper was not collected as part of a designed survey, but rather as the “other” category in a survey on whale migration. Over the last 10 years Monnett himself collected the majority of the data, and the categories in which the data was typically recorded were unclear. In order to get records from 20-10 years ago, Monnett appeared to simply ask his predecessor what he recalled.

Then there were the “statistics” of the paper. In the current year’s data (the only one that seemed to have independent verification) there were only 4 swimming polar bears in the first day’s survey, and 3 dead polar bears in the second day’s survey. Traditionally, this is usually far too small a sample to draw any statistical conclusions at all.

“Scientific misconduct” to me typically means misrepresenting the real data. Switching to legal terms for a moment, Monnett apparently had the means, the motive, and the opportunity to do so. But to show scientific misconduct, Monnett had to actually misrepresent the data. Was there something in the paper that seemed to show data misrepresentation? Well, yes. It might have seemed to the complainer (as an example) that Monnett was claiming that “death of polar bears caused by their attempts to swim to receding ice” was conclusively proven. In other words, Monnett may or may not have falsified the data; but at least he seemed to be claiming that the data proved something that the clearly valid data he had could not possibly support.

Given this, the complainer may have felt he had the equivalent of a “prima facie” case: a case in which the evidence tended to show misconduct, and the burden was on Monnett to show it was untrue.

The Game

In reading the transcript of the interview, it is helpful to think of it as partially a legal “game.” This is the way it apparently worked. The investigators arrived and were surprised to find that Monnett was represented by legal counsel, and they were being recorded. This meant that all of the actors in the interview had specific aims and tasks. The easiest role was that of the lawyer. As a good lawyer, he could be counted on to be quite aggressive in making claims about his client’s innocence, and noting instances where he could claim the investigators themselves were breaking the law.

Monnett himself was walking a careful line. On the one hand, the best way to put himself in a strong legal position was to overwhelm the investigators with details of the things that he had done right. On the other hand, the more he talked, the more possible mistakes he might reveal. Thus, throughout the interview, Monnett is providing massive amounts of detail, trying on the fly to shore up things that might be used to convict him, and trying to avoid being drawn into conclusions beyond what he had already said in the paper.

Meanwhile, the investigators split their role. Gary went through the details of the complainer’s allegation; thus, he focused on data collection, whether the data or conclusions in the paper had been reviewed adequately, and whether the data justified the conclusions. He was, in effect, trying to build the bricks to solidify and support the prima facie case. Lynn was his protector; when he seemed to be saying things that might expose the investigators or the department to legal attack, she would cut him off. Therefore, the investigators as a whole spent great effort in avoiding laying out what the details of the complaint were: these details might alert Monnett to areas of vulnerability and expose the department to counter-actions.

My Verdict – Not Just Not Proven, Innocent

Having heard the prima facie case, let’s analyze the data and the paper based on the information in the interview. What did it show?

First, let’s assume that the data is valid. Can we draw any statistical conclusions at all from it? Actually, yes. 4 and 3 may seem like small deviations from zero, but they follow a string of 20 zeros. The only plausible model I can see in the real world for a null hypothesis, that more than 3 swimming or dead polar bears is “the luck of the draw,” is a lambda probability function, in which the most probable occurrence is 0, the next most probable 1, and so on. Given 20 years of zeros, the only lambda function that would fit would have a probability of less than 1% of seeing 1 swimming/dead polar bear in year 21, and less than .1% of seeing 3 or more. It seems very clear to me that, if the data are valid, it is overwhelmingly likely that something new has happened to cause the swimming and dead polar bears.

All right, how about the validity of the data? Well, the three dead polar bears are clearly valid: there were attested by several other people besides Monnett. The four swimming polar bears are very likely to be valid, both because no one else in the helicopter, hearing the unusual sighting, found anything wrong, and because the death of the polar bears had to be caused by something, and that something was in the context of things highly likely to have been swimming. That leaves the question of whether Monnett omitted previous sightings of swimming and/or dead polar bears.

Actually, that one’s pretty easy to dispose of without recourse to statistics or the likelihood of personal actions. As Monnett makes clear, this was about the first year in which ice had clearly drawn off from the land. In most if not all previous years, Monnett or his predecessor could not possibly have seen swimming polar bears, since there was no water to swim in, just ice. And as for dead polar bears, again, it is overwhelmingly likely that 2 or 3 would have been noted in previous years in any case, since the researchers were trying to note any anomaly and the natives would care about this one, and no one has advanced a clear alternative cause (in previous years) for three to die.

Did Monnett overclaim based on this data? It appears that he said: (1) The numbers of swimming and dead polar bears was unusual, (2) It appears that the deaths were caused by the polar bears’ need to swim to the ice, (3) The 4 and 3 bears in 11% of his total area can be “extrapolated” to 36 and 27 across his entire area and a 25% survival rate (of bears that swam). We’ve already shown that statistically (whether or not he did the statistics), conclusion (1) is completely sound.

As for (2), this is not an overclaim. Something caused the deaths, and the most obvious explanation is the most obvious new factor that should cause an increase in deaths: withdrawal of ice from the land. That’s the equivalent of what Monnett said.
And then there’s (3). As I noted in my previous blog post, “extrapolation” is not an indication that something is highly likely. It’s an indication that if your sample from a population turns up this result, it’s the most likely frequency in the entire population, but not very likely in and of itself – and the frequency in the population is just as likely to be greater than that number as to be less than that number.

Were I a reviewer of such a paper, I would understand its import to be just as Monnett indicated in the interview: an establishment of a result that cannot be explained by the hypothesis of “just random chance”, a proposed explanation for the result that is a logically and physically appropriate model, and an invitation to other scientists to do work to replicate or propose alternative new explanations for the results. I would regard this as an entirely appropriate use of necessarily limited data.

My verdict: the prima facie case is entirely destroyed. Yes, in an ideal case the experiment could have been done better and better statistical tests might have been applied; but they were not needed. Monnett’s paper shows no scientific misconduct and is entirely appropriate to the data.

Why the Devil is in the Follow-on

Having, as I have said, given the “devil” (the complainer) his due, I maintain that it seems very likely that whoever has suspended Monnett has, in fact, been guilty of legal and scientific misconduct himself (or herself). Why? Because of what happened after the interview.

I have shown, I hope, that Monnett established, at the least, a strong prima facie case during the interview that he did not misrepresent the data. From then until now, there have been no public communications, nor communications with Monnett, about the case.

However, a week ago, Monnett was placed on “administrative leave”, and the stated cause was an investigation by the same investigator of allegations that Monnett had mishandled administration of a contract with U. of Alberta as subcontracted researchers into polar bear migration. Moreover, the same paper trail of permissions for his work is apparently available for the contract administration. That means that (1) investigators failed to investigate Monnett’s allegations of research distortion; (2) Monnett was suspended without even double-checking a “prima facie case”, if there was one – since investigators already knew that he typically had a paper trail; (3) Monnett was never cleared of the original charge by the investigators, although that was supposedly their only concern. What seems very likely is that this suspension is the result of proceedings covered by legal requirements that deliberately failed to follow them, both with regard to the paper and the administration. Whatever the relevant law, that suggests to me misconduct according to the law.

It also suggests scientific misconduct. The lack of contact with Monnett about his paper makes it clear that they failed to follow up with him his allegation that his research report had been distorted. Misrepresenting the data in a research report should also be considered scientific misconduct, whether the report is distributed beyond the department or not. And failure to correct a report that you should have known is distorted means that you share in the scientific misconduct, whether you are a scientist or not.

In the abstract, this type of misconduct seems isolated and unimportant. But it is also clear that these issues matter a great deal to everyone. Had the oil companies had a legitimate complaint, and if additional oil had mattered a great deal to the world’s economy and well-being, a Monnett prevention of drilling would have had major negative effects. If a valid Monnett scientific finding is not weighed and responded to appropriately and fossil fuels are indeed causing a global warming disaster, then this suspension and similar actions are equally if not more damaging to the world.
It appears from the only available evidence that those who suspended Monnett have thereby put themselves beyond excuse. I have tried to give the devil his due – is there any way I can avoid now naming him evil?

The Final Question -- and Cupcakes

2011-08-01T12:30:00.000-07:00

Once upon a time, there was a math professor that gave an exam. When the students entered the testing room, they were given a mathematical framework and a list of problems to solve, each of which built on the solution to the previous one. The professor told them that they must solve all the problems in the time allotted, and that whoever did so, would pass; whoever did not, would fail.

The professor also told them that they should take each proposed problem solution to a machine in the corner. If the solution was correct, the machine would give them a cupcake.

Well, on the first problem some of the students who found the solution first got a bit rowdy and started waving their cupcakes in the faces of those who were slower and teasing them. The professor told them to settle down and stop it, or the professor would kick them out and they would fail.

At that that point, one of the students said to several others: “Look, this test is hard, and I think it’s so hard that none of us is going to pass unless the professor lets us. So the best way to do that is to show the professor that we appreciate all that he has done to teach us, and that we’re trying hard to please him, and we’re good people – not like those rowdy ones. So let’s all work together on each problem, and when we get our cupcakes we’ll keep giving some of them to the professor. And if we give him enough cupcakes, even though we will never answer all the questions, we’ll still pass.”

Many of the students, looking at the nasty problems coming up, were convinced that this was the only way they were going to pass, and so they agreed. But the student did not convince everybody. Some students rejected the idea that the test was impossible, and that passing the exam depended in any way on the professor.

Then there ensued a strange scene. On each problem, the students working together in a group solved the problem faster, and so should have given the machine the solution ahead of the few who were working solo. But many of them were concerned that they didn’t have enough cupcakes to give, and so they were over trying to convince the ones who were doing it alone that they should try to join the group. And sometimes they would succeed, as one of the loners felt that the test was becoming too hard; and sometimes they would lose a member, as some members of the group felt their point of view wasn’t being listened to. And then sometimes one of the group would get hungry and eat a cupcake instead of contributing it; so they had to kick that member out of the group until the member said he or she was sorry. And then they had to spend some time gathering the cupcakes and giving them to the professor. So as it worked out, the loners were solving the problems just about as fast as the group.

And then they got to the problem right before the last one. And it was a really nasty problem. Moreover, because they had been working in a group, the members of the group hadn’t been paying close attention to all the details of the previous solution, so none of them could put it all together and come up with the last step in this solution. And they were really worried that they hadn’t gotten enough cupcakes yet.

So finally one of the loners came up with the solution. And this caused an even bigger argument among the group, but finally one part of the group went over and looked at the loner’s answer, and got the solution – although they didn’t really understand that last step. And now there were two groups. The first group said, we haven’t got enough cupcakes, but if everyone comes back and helps us solve the next-to-last problem, and gives us their cupcakes, we’ll have enough. And the second group said, yes, we solved the next-to-last problem, but if we don’t get the cupcakes for the next-to-last problem from everybody, the first group and the loners, and give them to the professor, the professor will decide that we can’t work together and just aren’t nice enough. And the loners just kept being stubbornly convinced that it was all about solving the final problem.

And now there was frantic activity all over the room. The first group was still trying to solve the next-to-last problem, and badgering the second group and the loners for cupcakes. The second group had given up on the final problem (partly because they still thought it was impossible, and partly because they couldn’t solve it anyway, since they didn’t understand the solution to the next-to-last problem) and was badgering the first group and the loners for cupcakes. And the loners were trying to work on the final problem, and telling both groups that if they weren’t going to help solve the final problem, at least they should go away and stop bothering them. And as they did this, the clock kept ticking; and the end of the exam drew closer; and closer; and closer.

The moral of the story is – no, I’m not going to tell you the moral of the story. Consider this the final question of your exam: what is the moral of this story? If you get it right, you pass; if you get it wrong, you fail.

You can send me some cupcakes, if you think that will help.

Trashing JRR Tolkien

2011-07-30T13:23:00.000-07:00

I note in the news that here in the US, a Republican Senator has criticized Republican House members by calling them naïve “hobbits”, and a Republican Representative has riposted by saying he’d rather be a hobbit than a troll. This is just the latest instance of Tolkien’s Lord of the Rings being referenced by US politicians, especially Republicans.

Enough, enough of this. I may be alone in saying this, but I suspect that Tolkien would be sick at heart at what his message has become.

It is clear from his published Letters that Tolkien distrusted Americans – which, at that time (1950s and early 1960s), meant American men. He was, for example, very wary of having his books made into movies by Americans – his stated reason was that the books’ words were meant to be mythical, and didn’t sound right when spoken. As it turns out, the makers of the movie found that in many cases the words of the book sounded better than a modern rewrite, so in this case he was unnecessarily modest. Also, a Ballantine version, while popularizing the book in America, infringed his copyright, so that at first he was deprived of badly needed royalties. But there was more than that at work; what else caused him to distrust American men, is not clear.

However, I find increasing evidence since that time that he was right to do so. I first read the books in the early 1960s, so I have watched the phenomenon unfold since the beginning. The divergence began then, I think.

One of the notable features about The Lord of the Rings that I have noted is that it has appealed to an unusual number of women – not only men. This was unusual for its time: fantasy then in America was focused on Robert Howard’s over-muscled superheroes, with the occasional female jock who somehow lacked the basic protective armor except in strategic but minimal places – not the kind of thing that many women then or now find interesting, except as an obvious example of men’s incomprehensible, irritating taste. Moreover, as female norms have changed over here, Tolkien’s focus on men characters would normally seem dated.

The key, I think, lies in a surprising amount of “rethinking” underlying the text. Not that Tolkien was a proto-feminist. However, there is a very interesting story called “Aldarion and Eldaris” set in the same world that he wrote at about that time, that clearly enters into his thinking. In it, he sets out at great length and with some sympathy the objections of the partner whom the hero-king on his quest has left. Moreover, in her own way Eldaris sets up a counter-kingdom excluding men, showing clear leadership qualities. We see traces of this not in Arwen, as the movie would have it, but in Eowyn, who is able to make the switch from killing things to being an equal ruler who grows things.

This, in turn, allows women who read it for the first time to feel connected to the story, and then to appreciate Tolkien’s focus on relationships and beauty. It was noteworthy that when the movie came out, the one scholar who focused on Tolkien’s vision of a nature that was so alive it glowed with an inner light, was the one woman scholar quoted.

However, right from the start, many American men have read these same sections with impatience, even scorn at Tolkien’s style. Early on, Gary Gygax, creator of the Dungeons and Dragons game that was in many ways a straightforward elaboration of Tolkien’s cast of creatures, was quoted as saying that he could write better, because he would go straight to the action. Peter Jackson (yes, I know he’s from New Zealand, but he was clearly tailoring the movie to American tastes) did his very best to keep the sense of dread and action constant, and dropped the much of the first “book” in consequence. Moreover, his experience in horror movies tailored to American tastes, I think, led him to the understandable decision to give a “horror” or “action” tinge to all the scenes, especially the fight scenes. This movie, in turn, reinforced a new generation of American men who may never have read the book in the belief that it’s about plucky little and big superheroes who go out and save the world from evil by winning wars.

This is so far from Tolkien’s apparent thinking that it’s hard to know where to begin. His experience in the trenches of WW I left him with what today might be described as Post Traumatic Stress Disorder, and he was reluctant to have his son serve in WW II, because he felt that it was being used by the government to demonize the enemy. Lord of the Rings reflects that thinking: war is a horrible experience (“it was Sam’s first experience of a battle, and he didn’t like it much”), and a last resort, and those who wage it must be very clear-eyed about why it is necessary and how to minimize its effects. The Lord of Gondor, who believes the struggle is all about Gondor and only he can save it, is just as destructive as the counselor in Rohan who betrays the kingdom in order to “get” the woman he lusts after.

The second key point is that the story is about people changing – or not. Above all, it’s a story about Frodo changing. Perhaps the sentence that is most charged with meaning in the entire story is this: “I wanted to save the world, and it was saved – but not for me.” The entire first book is filled with meetings in which Frodo’s (and our) vision of the world is stretched, and stretched again, until he understands much of what is valuable in the world from its beginning until now. Then – and this point cannot be emphasized enough – he sets out to help save that world, and he fails. The result of that failure – the failure to resist the Ring enough to cast it in the Fire – and of his wounds, the equivalent of PTSD, is that he can’t stop missing the Ring, and he can’t enjoy the world that is saved, and especially because many of the things he wanted to save are vanishing anyway. I repeat: this is not a story about superheroes winning wars; it is about people enlarging their vision so they understand what’s at stake, instead of thinking it’s all about them and only they know the right answer. “Even Sauron was not evil in the beginning”; and that was the key mistake Sauron made.

The third key point is that in Tolkien’s world, not only the people are connected parts of an integral whole, but so is nature. The weather, the trees, all affect people’s moods and thinking, and are in turn fostered and destroyed by people. This is no abstract “the hero and his buddies go on a quest” folktale. It is a story in which your success is connected to the success of Brand and Dain halfway around the world, and your failure at Weathertop to the north is connected to the success of the Haradrim at Pelargir in the uttermost south. It is a story in which you would not succeed unless you helped save the trees of Fangorn Forest, and unless the wind from the South dissipated the storm of Mordor. If you lose that connection, the world is “blasted beyond repair, unless the healing hand of the Sea should cover the land in merciful oblivion.”

OK, so back to my quotes. It’s about plucky hobbits and evil trolls and fighting the enemy – yes, that sounds like an American man; and is the opposite of Tolkien. It’s time to cut down the evil government, we know how to do it, and we don’t believe the alarmists who say the cure is worse than the disease – yes, that sounds like an American man; and is the opposite of Tolkien. Let’s cut back on environmental regulation to save money, and drill more oil for our energy needs, as the rich suggest; well, let’s see what Tolkien says: “He started shipping food down south. People didn’t like it, what with winter coming on … But lately, he’s been pouring filth out of that mill for no reason, fouling the water …” When there’s a clear connection between present increases in fossil fuels and future injury of most people’s habitat, Tolkien would very clearly vote for better environmental protection and less drilling. Whether you think he’s right or not, he very likely would regard the US House efforts in this regard as clearly evil.

There’s a wonderful quote I have never been able to track down, that runs something like this: “When a man first commits murder, it is to be expected that he will then pass to assault and battery; he may indeed go on to wanton destruction of personal property, and in some cases from thence to defacement of public property; indeed, as hard as it is to believe, even violation of the Sabbath may not be beyond his reach.” I realize that in the scale of things, compared to risking the full faith and credit of the US government and threatening the global economy, trashing JRR Tolkien’s deep personal beliefs may not be of the same measure. But still, you American men, can you not at least spare those few of us who appreciate the full measure of his work the act of trampling on his grave? At long last, sirs, have you no shame? No shame at all?

Hello?

OMG This Math Is Depressing

2011-07-29T13:15:00.000-07:00

I just had the dubious pleasure of reading a transcript of an interview related to the “Arctic scientist muzzled for paper about polar bears.” It is incredibly depressing, because of the lack of knowledge of absolutely basic math it reveals – and not by the scientist, who does just fine.

Here is my summary of the interview. Let’s see if you can do better than the interviewers. I have even simplified the numbers ever so slightly.

Two Interviewers: Hi. We’re here from the investigative branch of the department to investigate allegations of scientific misconduct in a paper you wrote about a sudden apparent increase in deaths of polar bears.

Scientist: Do you have the scientific background to understand the paper?

Interviewers: No.

Scientist: OK, I’ll do my best.

We had been doing surveys of whales up here off the Alaska coast for 20 years, noting all other creatures out there as well. Each sweep covers (randomly) 10% of the total area we watch over. One year, for the first time, the ice moved well away from the land. On our next sweep that year, we saw four polar bears swimming. On the sweep after that, we saw three dead polar bears. Now, we couldn’t ever remember seeing such a thing, so I went and checked the notes and checked the memory and notes of the guy who had been doing this before me, since the beginning, and we’d never seen such a thing. So we wrote up a paper about it, passed it by everyone at the agency, had it anonymously peer reviewed by three people, and it was published by Polar Biology.

Interviewers: OK, so what you’re saying is, you saw 7 polar bears. How can you say there were 30 dead polar bears out there?

Scientist: What?! I didn’t say there were 7 dead polar bears – I said there were three.

Interviewers: No, in your paper, you say 4 polar bears on one sweep, and 3 in the next. Four plus three equals seven.

Scientist: But …

Interviewers: Also, why didn’t you say 7 dead polar bears instead of 30, since that’s all you saw?

Scientist: Look, in the first place I only swept 10% of the area, so I multiplied the number I saw by 10 …

Interviewers: Why would you multiply by 10?

Scientist: Excuse me, but have you ever taken any fifth grade math?

Interviewer: And even if it was OK to multiply by 10, that would mean you were claiming you saw 70 dead polar bears.

Scientist: No, I’m claiming I saw 3 dead polar bears, and that the best guess for the total in my area was 30 dead polar bears.

Interviewers: Ah, so let me read back to you what you have said. ‘I am claiming in my paper that it is likely that there are 30 dead polar bears out there.’ Is that correct?

Scientist: No. Have you ever taken any statistics? Even just a little? It is not “likely” that there are 30 dead polar bears out there. It’s just the most likely number, and there is an almost 50% chance of a number less than that, and an almost 50% chance of a number grea - …

Interviewers: Well, I think we have all we need.

Scientist: In that case, on the record, let me tell you what’s really going on. First, the purpose of the paper was not to establish a final determination of what was going on, but to say that something odd was going on. Second, my hypothesis – that there are increased polar bear deaths because of ice withdrawal from land – has been amply proven since by scientific research, anonymously peer reviewed. Third, this department has persistently attempted to prevent me and others from publishing any research that might support global warming, even though this research is a clear part of my job as a scientist and a clear part of my task in this department. Fourth, I am supposed to be checking out anything that might affect the natives here, not just whales. They want to know about my research, whales and otherwise. The only people who don’t are the oil companies whose permits might be affected and the political appointees in this department who seem to be doing their bidding. Instead of investigating me, why don’t you investigate them for “scientific misconduct”? I can certainly document attempts to distort my research, to the point where I took my name off the product …

Interviewers: Goodbye.

Don't Cloud the Computing Carbon Emissions Reduction Issue

2011-07-26T17:18:00.000-07:00

I recently read a post by Jon Koomey, Consulting Professor at Stanford, at www.climateprogress.org, called “4 reasons why cloud computing is efficient”. He argues (along with some other folks) that cloud computing – by which he apparently means almost entirely public clouds – is much more beneficial for reducing computing’s carbon emissions than the real-world alternatives. As a computer industry analyst greatly concerned by carbon emissions, I'd like to agree with Jon; I really would. However, I feel that his analysis omits several factors of great importance that lead to a different conclusion.

The study he cites compares the public cloud -- not a private or hybrid cloud -- to "the equivalent". It is clear from context that it is talking about a "scale-out" solution of hundreds and thousands of small servers, each with a few processors. This is, indeed, typical of most public clouds, and other studies have shown that in isolation, these servers do indeed have a utilization rate of perhaps 10-20%. However, the scale-up hundreds-of-processors servers that are a clear alternative, and which are typically not used in public clouds (but are often used in private clouds), have a far better record. The most recent mainframe implementations, which support up to a thousand "virtual machines", achieve utilization rates of better than 90% -- a three times better carbon efficiency than the public cloud, right up front.

The second factor Jon omits is the location of the public cloud. According to Carol Baroudi, author of "Green IT For Dummies", only one public cloud site that she studied is located in an area that has a strong record of electricity that is carbon-emission-light (Oregon). The others are in areas where the energy is "cheaper" because of fossil fuel use. That may change; but you don't move a public cloud data center easily, because the petabytes of data stored there to deliver high performance to nearby customers doesn't move easily, even over short distances. Corporate data centers are more movable, because the data storage sizes are smaller and they have extensive experience with "consolidation". While until recently most organizations were not conscious of the carbon-emission effects of their location, it appears that companies like IBM are indeed more conscious of this concern than most public cloud providers.

The third factor that Jon omits is what I call "flight to the dirty". High up-front costs of more efficient scale-up servers leads unconsciously to use of less energy-efficient scale-out servers. Controls over access to public and private clouds and data centers, and visibility of their costs, moves consumer and local computing onto PCs and smartphones. Apparent cheapness of labor and office space in developing nations leads companies to rapidly implement data centers and computing there using existing energy-inefficient and carbon-wasting electrical supplies. All of these "carbon inefficiencies" are not captured in typical analyses.

Personally, I come to three different conclusions:

1. The most carbon-efficient computing providers use scale-up computing and integrated energy management, and so far most if not all of those are private clouds.

2. The IT shops that are most effective at improving carbon efficiency in computing monitor energy efficiency and carbon emissions use not only inside but outside the data center, and those inevitably are not public clouds.

3. Public clouds, up to now, appear to be "throwing good money after bad" in investing in locations that will be slower to provide carbon-emission-light electricity -- so that public clouds may indeed slow the movement towards more carbon-efficient IT.

A better way of moving computing as a whole towards carbon-emission reductions is by embedding carbon monitoring and costing throughout the financials and computers of companies. Already, a few visionary companies are doing just that. Public cloud companies should get on this bandwagon, by making their share of carbon emissions transparent to these companies (and by doing such monitoring and costing themselves). This should lead both parties to the conclusion that they should either relocate their data centers or develop their own solar/wind energy sources, that they should move towards scale-up servers and integrated energy management, and that they should not move to less costly countries without achieving energy efficiency and carbon-emission reduction for their sites up front.

Will We All Speak IT?

2011-07-19T11:06:00.000-07:00

At a recent teleconference, I heard the speaker first refer to “provisioning” a solution and then to people who would “on-board” that solution. It suddenly struck me that I was witnessing a new stage in the intrusion of language derived from computing into our daily lives.

Here’s how it used to go: we grew up with the rules of grammar and vocabulary as taught us in school, and as computer technology evolved, its new ideas and products used the words of, and fit neatly into, the English we were taught. A machine made a computation of a number, it computed, it was a computer. A piece of information in a computer, from the Latin, was a datum, plural data, stored in a data base, managed by a database management system.

In the same way, the jargon of computer techies, even when it spilled over into the population at large, was ultimately derived from ideas already in English. Bogosity – the quality of being bogus; bogon – a unit of bogosity. Misfeature – a combination of mistake and feature, a mistake that was touted by marketdroids (mindless marketers) as a feature.

ITSpeak 2.0
I first noticed things beginning to change in the late 1990s. In the 1980s, I had been frustrated as a purist by the universal tendency of my fellow programmers to refer to an example problem, an example case, an example screen, instead of a sample screen, as I had always been taught. Still, until the late 1990s, I never saw anyone else use “example” as an adjective; then marketing, and sometimes business blogs for a general audience, started to use “example” that way. However, even in the computing industry, there was strong purist resistance. I well remember the difficulties I had at Aberdeen Group persuading the editors that in computing, it was now “lifecycle”, not “life cycle”. Today, I can’t remember having seen “life cycle” in years.

In some ways, these tinkerings with basic English had a positive effect, I believe. Using “example” for “sample” is a good case in point: the meaning is clear from context, and it’s easier to use one word for the concept than learn two.

But the changes were not all for the good. I still remember some annoying marketer at Sybase, iirc, deciding in the late 1990s that from now on it was to be “database”, not “database management system”. The result was that users ever since are constantly confused as to whether they are talking about the software, or the data stored for use by that software – which I now have to always call the “data store” to make myself clear. In the same way, Enterprise Information Integration is now “data virtualization”, which captures only half the qualities of the software.

And, of course, with the advent of the Web IT words became far more ubiquitous, from blog to tweet. Sadly, these words have now become a measure of age, as each successive fad embeds its IT words into popular language, and we now divide generations into those that know what “to friend” means, and those who don’t.

ITSpeak Takes Over?
Even so, I didn’t see until now any clear indication that computer jargon was crowding out basic English words. But consider “provision”. Until very recently, a male was a “provider” who made enough money to put “provisions” on the table for the family. Now, IT has taken the word and abstracted it, to describe a general process of populating the empty shell of any new solution, and turned it from a noun into a verb. This major change in meaning is coming from IT, but it isn’t stopping there. Pretty soon, I expect to hear supermarkets start talking about “provisioning” their new stores, and then home builders and buyers start to talk about “provisioning” the new house with furniture.

The same goes for “on-board”. Like “friend” and “provision”, it’s a straightforward conversion of another part of speech to a verb. Like “provision”, it is a major switch in meaning that carries with it the notion of a process rather than an individual act. In the teleconference, it appeared to mean users carrying out the tasks of becoming part of a new IT solution themselves. But, again, I expect that soon employees will be expected to “on-board” themselves via “self-service portals”, and then students starting at college, and then what? Will we create new automated birthing centers where newborns will be expected to “on-board” themselves by responding to automated nipples? Will end-of-life hospices be referred to as “off-boarding centers?”

What If ITSpeak Does Take Over?
If we all starting talking ITSpeak – a language many of whose concepts originated in computing – is that good or bad? I believe that it’s way too early to tell. On the positive side, many of these words come from trying to distinguish more clearly between similar things, when the differences matter. The idea that a misfeature is not the same as a feature is important, and useful to us.

On the negative side, some historical richness of meaning may be lost. Always employing “utilize” instead of “use” (not really ITSpeak, but analogous) is not only unnecessarily lengthy, it also misses the importance in history of the distinction between “applying an object for a use for which it is designed” and “applying an object whether it helps in a task or not”. You utilize a Phillips screwdriver in following the directions for assembling a kid’s toy; you use a user’s manual for Microsoft Word even though it often doesn’t give you the answer you need.

No, my point here is that I think this represents a fundamental shift in our thinking, as we begin to see the world as IT folks do. At the least, this might mean that we think more of software-type abstractions and less of “legacy” physical objects, see life more in terms of processes and less in terms of interactions, and view others less in terms of irrationality and psychology and more in terms of categories and connections. So to maximize the chances of something good coming out of this, I think we ought to at least recognize that it is going on.

Will we all speak IT, all the time? Someday, quite possibly. Right now, it’s time to prepare to provision, so that we may on-board effectively.

The IBM Acquisition Game

2011-07-08T14:03:00.000-07:00

Recently, I was contacted by a firm called Software Advice, which has a very interesting business model: pay-for-results advice on short lists for IT buying. They just posted a blog on “IBM M&A: Who’s Next”, and were interested in my thoughts. I took a look, and found it quite impressive; and therefore, in accordance with my philosophy of comforting the afflicted and afflicting the comfortable, I decided to pick nits about their conclusions. I believe that both their and my thoughts offer some potentially useful insights to IT buyers, not just about IBM, but about how quickly vendors are likely to deliver what users need in the next 1-2 years.

The reason it’s not only fun but instructive to play the IBM acquisition game is that it implicitly asks, given user needs over the next 1-2 years, what are the holes in IBM’s lineup to meet those needs that it should fill immediately? And that also allows us to ask, if they don’t fill those needs, will it come back to bite them, because someone else is likely to beat them to the punch? And then we can ask, will folks really want to use someone else besides IBM if IBM doesn’t supply this need – or is this something for which the IT buyer will have to “roll his or her own” at greater expense?

So let the game begin!

Historical Nits
Software Advice begins with a graphic nicely capturing the extent of IBM’s acquisitions over the last decade or so. The problem lies in the headings that split the acquisitions into “applications”, “infrastructure”, and “services”. You see, IBM has been firm in disclaiming any intention of getting into “applications”, and so most if not all of the acquisitions classified as “applications” are in fact what is usually called “infrastructure software.” That also means that almost identical infrastructure software is in one case classified as an application and in another as infrastructure – for example, the Rational software development toolset is counted as infrastructure software, but the Telelogic requirements management toolset, which is almost always used as the first step in the development process as part of a “lifecycle” software development toolset, is classified as an “application”. Hence it’s very easy to assume that IBM doesn’t need any applications acquisitions.

The interesting thing about this nit is that it raises the question: should IBM, at long last, go into the “apps business”, either on the business or consumer side? Yes, they’ve never needed to before, since until recently both Oracle and SAP (the dominant players in enterprise apps) have shown themselves willing to support all hardware vendors, but now that Oracle owns Sun and has shown it can play hardball with respect to HP Itanium, should IBM rethink that posture? Does the market now need a platform that it can be sure its enterprise or other business-critical applications will support?

The answer to that, I believe, depends on SAP. In other words, whatever the merits of other app vendors like Salesforce.com, the run-the-business applications of SAP are presently the main alternatives to Oracle Apps. If SAP remains a strong alternative, then IBM is entirely correct in continuing to keep its hands off enterprise application companies, reinforcing its image as less prone to vendor lock-in than Microsoft or Oracle.

And yet, I have to say, whether SAP will be a strong alternative remains an open question. SAP has made some major acquisitions of its own, like Business Objects and Sybase, which have taken it down the software stack with some quality infrastructure software. However, it is not yet clear that SAP can drive rapidly-changing database technology ahead fast enough to provide a long-run all-in-one enterprise-app or analytics alternative to Oracle Apps. The signs are very good: SAP appears to understand the importance of Sybase, and the potential of integrating its technologies with SAP’s present stack. Still, SAP has to execute that strategy.

I think it would make most sense for IBM to beef up its SAP application support with a smaller acquisition or two, this time of cross-database administrative tools that specialize in Sybase. Later, of course, if things get bad, IBM could always acquire SAP. In the meanwhile, the IT buyer should note that IBM and Oracle SAP support is a space to watch.

Strategic Investment Nits
Software Advice then goes on to identify general areas of future customer need where IBM may need to acquire companies. Their main focus – certainly a good one – is cloud administration. They also note – although with a much shorter analysis – IBM’s need to expand its analytics and BI offerings even further – and that makes sense too. Everyone, not just IBM, is scrambling to fill in the blanks and achieve fully automated hybrid-cloud deployment and administration.

However, I would disagree with their analysis of virtualization as a key area of acquisition. While VMWare continues to be an outstanding success, the pace of virtualization to a public cloud – the main lock-in for VMWare – remains quite slow. Private clouds in larger enterprises tend to be top-down, which means that IBM is doing quite well at driving its own virtualization software across the data center. I would argue that IBM has no need to acquire either VMWare or EMC either now or in the next two years – and a good reason to wait to see what happens as Oracle continues to compete with EMC more strongly in storage.

What might make sense, on the other hand, is for IBM to consider acquiring Red Hat. The two have been working together pretty effectively, and IBM needs to build up its open-source brand as a new market of tech-savvy open-source-oriented firms opens up. As long as it leaves the open-source culture of Red Hat in place, IBM can use Red Hat as an “early warning system” for changes in the new market – because that market cares less about VMWare vs. KVM and more about open-source-based services for cloud deployment.

My second nit regards mobile technology. It appears likely that the movement of mobile business workers towards having a laptop for some situations and a small-form-factor smartphone or tablet for others has reached flood stage, and needs to be addressed better. Sybase would have been a great entry point, but it’s not available now. Buying Apple would be fun to imagine, but seems impossible to achieve. Perhaps IBM might consider RIM. The value-add of Blackberry cell phones has always been in their business software, and while they are under threat in the consumer market, business users still find them appropriate. Here is an area of great user need where all vendors – not just IBM – fall short; so if IBM doesn’t do a good acquisition soon, IT buyers should anticipate a lot of “roll your own”.

My third nit concerns the whole area of BI/analytics. There seems to be a pervasive confusion of BI, analytics, and Big Data, as if they are the same thing. My short take on the differences is: BI is basic repeated reporting and querying plus ad-hoc or goal-oriented querying, both for corporate; analytics is ad-hoc or goal-oriented querying, not only for corporate but also embedded in other software across the organization (e.g., security and administrative analytics); Big Data is a wide range of new large-footprint data types, more usually on the Web, that provides insights into such new marketing topics as social media, and therefore typically complements BI with extra-organizational data. The result is that any good push to meet user needs is going to need to tackle all three areas.

As I noted in a previous blog post, what’s users need in all three areas is some combination of scaling and user friendliness, especially for the burgeoning SMB BI market. It’s hard to buy or create user friendliness – the BI market still has a ways to go in this area. However, there are ways that IBM could improve its scalability. For one thing, Netezza and the new IBM z appliance have columnar database technology that’s too tied to a particular appliance. It’s not clear just how fast IBM will move into this area, but Amazon’s investment in ParAccel reminds us that there are still interesting columnar database suppliers out there.

On the Big Data side, users must also consider integrating BI with file-system-stored Web data such as that accessed via Hadoop. There are quite a few NoSQL open-source efforts that may be worth productizing and integrating with DB2 or a columnar database. Again, this is an area where all vendors – not just IBM – need to do more to make the path to combined BI/analytics/Big Data clear. In the meanwhile, IT buyers should think carefully about buying from only one database vendor, because until one of them shows they have the full Big Data story there is no guarantee that any of them will not fall short of what users need – and past experience suggests that database lock-in is about as locked in as you can get.

Endgame
Having played Software Advice’s IBM Acquisition Game, I draw three conclusions from it. First, IBM is in a surprisingly strong position going forward. There is no obvious hole in its solution lineup that immediately threatens the company, and that it cannot fix by careful re-tuning of the same strategies it has had up to now. And that’s good news for IT buyers.

Which leads me to conclusion two: there’s still enough choice in the market. We have seen a lot of acquisitions, not just from IBM but from other major vendors, in the last decade; but the fact that there are still smaller companies out there to plug holes for IBM and others means that IT buyers can still find a way to stitch together a solution where one’s favored vendor doesn’t quite cover all needs.

And that leads to conclusion three: despite the hype, users are still a long way from taking full advantage of mobile, cloud, or analytics/Big Data. This may well be a transition as slow and incomplete as the one in the early 2000s to service-oriented architectures – and don’t get me started about Business Process Integration. In fact, it might be a good idea to play the Acquisition Game with other vendors on your short lists – and then see what those vendors do in the real world to cover the holes you find, before committing irrevocably and totally to one of them. That’s not to say you shouldn’t press ahead with all deliberate speed, as your competitors will be doing -- but cover your bets.

Big Data, MapReduce, Hadoop, NoSQL: Pay No Attention to the Relational Technology Behind the Curtain

2011-06-29T10:37:00.000-07:00

One of the more interesting features of vendors’ recent marketing push to sell BI and analytics is the emphasis on the notion of Big Data, often associated with NoSQL, Google MapReduce, and Apache Hadoop – without a clear explanation of what these are, and where they are useful. It is as if we were back in the days of “checklist marketing”, where the aim of a vendor like IBM or Oracle was to convince you that if competitors’ products didn’t support a long list of features, that those competitors would not provide you with the cradle-to-grave support you needed to survive computing’s fast-moving technology. As it turned out, many of those features were unnecessary in the short run, and a waste of money in the long run; remember rules-based AI? Or so-called standard UNIX? The technology in those features was later to be used quite effectively in other, more valuable pieces of software, but the value-add of the feature itself turned out to be illusory.

As it turns out, we are not back in those days, and Big Data via Hadoop and NoSQL does indeed have a part to play in scaling Web data. However, I find that IT buyer misunderstandings of these concepts may indeed lead to much wasted money, not to mention serious downtime. These misunderstandings stem from a common source: marketing’s failure to explain how Big Data relates to the relational databases that have fueled almost all data analysis and data-management scaling for the last 25 years. It resembles the scene in Wizard of Oz where a small man, trying to sell himself as a powerful wizard by manipulating stage machines from behind a curtain, becomes so wrapped up in the production that when someone notes “There’s a man behind the curtain” the man shouts “Pay no attention to the man behind the curtain!” In this case, marketers are shouting about the virtues of Big Data related to new data management tools and “NoSQL” that they fail to note the extent to which relational technology is complementary to, necessary to, or simply the basis of, the new features.

So here is my understanding of the present state of the art in Big Data, and the ways in which IT buyers should and should not seek to use it as an extension of their present (relational) BI and information management capabilities. As it turns out, when we understand both the relational technology behind the curtain and the ways it has been extended, we can do a much better job of applying Big Data to long-term IT tasks.

NoSQL or NoREL?
The best way to understand the place of Hadoop in the computing universe is to view the history of data processing as a constant battle between parallelism and concurrency. Think of the database as a data store plus a protective layer of software that is constantly being bombarded by transactions – and often, another transaction on a piece of data arrives before the first is finished. To handle all the transactions, databases have two choices at each stage in computation: parallelism, in which two transactions are literally being processed at the same time, and concurrency, in which a processor switches between the two rapidly in the middle of the transaction. Pure parallelism is obviously faster; but to avoid inconsistencies in the results of the transaction, you often need coordinating software, and that coordinating software is hard to operate in parallel, because it involves frequent communication between the parallel “threads” of the two transactions.

At a global level (like that of the Internet) the choice now translates into a choice between “distributed” and “scale-up” single-system processing. As it happens, back in graduate school I did a calculation of the relative performance merits of tree networks of microcomputers versus machines with a fixed number of parallel processors, which provides some general rules. There are two key factors that are relevant here: “data locality” and “number of connections used” – which means that you can get away with parallelism if, say, you can operate on a small chunk of the overall data store on each node, and if you don’t have to coordinate too many nodes at one time.

Enter the problems of cost and scalability. The server farms that grew like Topsy during Web 1.0 had hundreds and thousands of PC-like servers that were set up to handle transactions in parallel. This had obvious cost advantages, since PCs were far cheaper; but data locality was a problem in trying to scale, since even when data was partitioned correctly in the beginning between clusters of PCs, over time data copies and data links proliferated, requiring more and more coordination. Meanwhile, in the High Performance Computing (HPC) area, grids of PC-type small machines operating in parallel found that scaling required all sorts of caching and coordination “tricks”, even when, by choosing the transaction type carefully, the user could minimize the need for coordination.

For certain problems, however, relational databases designed for “scale-up” systems and structured data did even less well. For indexing and serving massive amounts of “rich-text” (text plus graphics, audio, and video) data like Facebook pages, for streaming media, and of course for HPC, a relational database would insist on careful consistency between data copies in a distributed configuration, and so could not squeeze the last ounce of parallelism out of these transaction streams. And so, to squeeze costs to a minimum, and to maximize the parallelism of these types of transactions, Google, the open source movement, and various others turned to MapReduce, Hadoop, and various other non-relational approaches.

These efforts combined open-source software, typically related to Apache, large amounts of small or PC-type servers, and a loosening of consistency constraints on the distributed transactions – an approach called eventual consistency. The basic idea was to minimize coordination by identifying types of transactions where it didn’t matter if some users got “old” rather than the latest data, or it didn’t matter if some users got an answer but others didn’t. As a communication from Pervasive Software about an upcoming conference shows, a study of one implementation finds 60 instances of unexpected unavailability “interruptions” in 500 days – certainly not up to the standards of the typical business-critical operational database, but also not an overriding concern to users.

The eventual consistency part of this overall effort has sometimes been called NoSQL. However, Wikipedia notes that in fact it might correctly be called NoREL, meaning “for situations where relational is not appropriate.” In other words, Hadoop and the like by no means exclude all relational technology, and many of them concede that relational “scale-up” databases are more appropriate in some cases even within the broad overall category of Big Data (i.e., rich-text Web data and HPC data). And, indeed, some implementations provide extended-SQL or SQL-like interfaces to these non-relational databases.

Where Are the Boundaries?
The most popular “spearhead” of Big Data, right now, appears to be Hadoop. As noted, it provides a distributed file system “veneer” to MapReduce for data-intensive applications (including Hadoop Common that divides nodes into a master coordinator and slave task executors for file-data access, and Hadoop Distributed File System [HDFS] for clustering multiple machines), and therefore allows parallel scaling of transactions against rich-text data such as some social-media data. It operates by dividing a “task” into “sub-tasks” that it hands out redundantly to back-end servers, which all operate in parallel (conceptually, at least) on a common data store.

As it turns out, there are also limits even on Hadoop’s eventual-consistency type of parallelism. In particular, it now appears that the metadata that supports recombination of the results of “sub-tasks” must itself be “federated” across multiple nodes, for both availability and scalability purposes. And Pervasive Software notes that its own investigations show that using multiple-core “scale-up” nodes for the sub-tasks improves performance compared to proliferating yet more distributed single-processor PC servers. In other words, the most scalable system, even in Big Data territory, is one that combines strict and eventual consistency, parallelism and concurrency, distributed and scale-up single-system architectures, and NoSQL and relational technology.

Solutions like Hadoop are effectively out there “in the cloud” and therefore outside the enterprise’s data centers. Thus, there are fixed and probably permanent physical and organizational boundaries between IT’s data stores and those serviced by Hadoop. Moreover, it should be apparent from the above that existing BI and analytics systems will not suddenly convert to Hadoop files and access mechanisms, nor will “mini-Hadoops” suddenly spring up inside the corporate firewall and create havoc with enterprise data governance. The use cases are too different.

The remaining boundaries – the ones that should matter to IT buyers – are those between existing relational BI and analytics databases and data stores and Hadoop’s file system and files. And here is where “eventual consistency” really matters. The enterprise cannot treat this data as just another BI data source. It differs fundamentally in that the enterprise can be far less sure that the data is up to date – or even available at all times. So scheduled reporting or business-critical computing based on this data is much more difficult to pull off.

On the other hand, this is data that would otherwise be unavailable – and because of the low-cost approach to building the solution, should be exceptionally low-cost to access. However, pointing the raw data at existing BI tools is like pointing a fire hose at your mouth. The savvy IT organization needs to have plans in place to filter the data before it begins to access it.

The Long-Run Bottom LineThe impression given by marketers is that Hadoop and its ilk are required for Big Data, where Big Data is more broadly defined as most Web-based semi-structured and unstructured data. If that is your impression, I believe it to be untrue. Instead, handling Big Data is likely to require a careful mix of relational and non-relational, data-center and extra-enterprise BI, with relational in-enterprise BI taking the lead role. And as the limits to parallel scalability of Hadoop and the like become more evident, the use of SQL-like interfaces and relational databases within Big Data use cases will become more frequent, not less.

Therefore, I believe that Hadoop and its brand of Big Data will always remain a useful but not business-critical adjunct to an overall BI and information management strategy. Instead, users should anticipate that it will take its place alongside relational access to other types of Big Data, and that the key to IT success in Big Data BI will be in intermixing the two in the proper proportions, and with the proper security mechanisms. Hadoop, MapReduce, NoSQL, and Big Data, they’re all useful – but only if you pay attention to the relational technology behind the curtain.

Pentaho and Open Source BI: The New SMB

2011-06-29T10:35:00.001-07:00

On Monday, Pentaho, an open source BI vendor, announced Pentaho BI 4.0, its new release of its “agile BI” tool. To understand the power and usefulness of Pentaho, you should understand the fundamental ways in which the markets that we loosely call SMB have changed over the last 10 years.

First, a review. Until the early 1990s, it was a truism that computer companies in the long run would need to sell to central IT at large enterprises, eventually – else the urge of CIOs to standardize on one software and hardware vendor would favor larger players with existing toeholds in central IT. This was particularly true in databases, where Oracle sought to recreate the “nobody ever got fired for buying IBM” hardware mentality of the 1970s in software stacks. It was not until the mid-1990s that companies such as Progress Software and Sybase (with its iAnywhere line) showed that databases delivering near-lights-out administration could survive the Oracle onslaught. Moreover, companies like Microsoft showed that software aimed at the SMB could over time accumulate and force its way into central IT – not only Windows, Word, and Excel, but also SQL Server.

As companies such as IBM discovered with the bursting of the Internet bubble, this “SMB” market was surprisingly large. Even better, it was counter-cyclical: when large enterprises whose IT was a major part of corporate spend cut IT budgets dramatically, SMBs kept right on paying the yearly license fees for the apps on which they ran, which in turn hid the brand on the database or app server. Above all, it was not driven by brand or standards-based spending, nor even solely by economies of scale in cost.

In fact, the SMB buyer was and is distinctly and permanently different from the large-enterprise IT buyer. Concern for costs may be heightened, yes; but also the need for simplified user interfaces and administration that a non-techie can handle. A database like Pervasive could be run by the executive at a car dealership, who would simply press a button to run backup on his or her way out on the weekend, or not even that. The ability to fine-tune for maximum performance is far less important than the avoidance of constant parameter tuning. The ability to cut hardware costs by placing apps in a central location matters much less than having desktop storage to work on when the server goes down.

But in the early 2000s, just as larger vendors were beginning to wake up to the potential of this SMB market, a new breed of SMB emerged. This Web-focused SMB was and is tech-savvy, because using the Web more effectively is how it makes its money. Therefore, the old approach of Microsoft and Sybase when they were wannabes – provide crude APIs and let the customer do the rest – was exactly what this SMB wanted. And, again, this SMB was not just the smaller-sized firm, but also the skunk works and innovation center of the larger enterprise.

It is this new type of SMB that is the sweet spot of open source software in general, and open source BI in particular. Open source has created a massive “movement” of external programmers that have moved steadily up the software stack from Linux to BI, and in the process created new kludges that turn out to be surprisingly scalable: MapReduce, Hadoop, noSQL, and Pentaho being only the latest examples. The new SMB is a heavy user of open source software in general, because the new open source software costs nothing, fits the skills and Web needs of the SMB, and allows immediate implementation of crude solutions plus scalability supplied by the evolution of the software itself. Within a very few years, many users, rightly or wrongly, were swearing that MySQL was outscaling Oracle.

Translating Pentaho BI 4.0
The new features in Pentaho BI can be simply put, because the details simply show that they deliver what they promise:

· Simple, powerful interactive reporting – which apparently tends to be used more for ad-hoc reporting that the traditional enterprise reporting, but can do either;
· A more “usable” and customizable user interface with the usual Web “sizzle”;
· Data discovery “exploration” enhancements such as new charts for better data visualization.

These sit atop a BI tool that distinguishes itself by “data integration” that handles an exceptional number of input data warehouses and data stores for inhaling to a temporary “data mart” for each use case.

With these features, Pentaho BI, I believe, is valuable especially to the new type of SMB. For the content-free buzz word “agile BI”, read “it lets your techies attach quickly to your existing databases as well as Big Data out there on the Web, and then makes it easy for you to figure out how to dig deeper as a technically-minded user who is not a data-mining expert.” Above all, Pentaho has the usual open source model, so it’s making its money by services and support – allowing the new SMB to decide exactly how much to spend. Note also Pentaho’s alliance not merely with the usual cloud open source suspects like Red Hat but also with database vendors with strong BI-performance technology such as Vertica.

The BI Bottom Line
No BI vendor is guaranteed a leadership position in cloud BI these days – the field is moving that fast. However, Pentaho is clearly well suited to the new SMB, and also understands the importance of user interfaces, simplicity for the administrator, ad hoc querying and reporting, and rapid implementation to both new and old SMBs.

Pentaho therefore deserves a closer look by new-SMB IT buyers, either as a cloud supplement to existing BI or as the core of low-cost, fast-growing Web-focused BI. And, remember, these have their counterparts in large enterprises – so those should take a look as well. Sooner than I expected, open source BI is proving its worth.

This Made Me Sick To My Stomach

2011-06-22T08:42:00.000-07:00

I just finished reading the report summary of the International Earth system expert workshop on ocean stresses and impacts, released on Monday. Here is what I think the headline of that report should have been:

The Oceans Are Mostly DEAD Unless We Reduce Carbon Emissions Drastically AND Set Up a Global Police Over ALL Ocean Uses NOW

I won’t bother with the evidence of cascading destruction, with more to come, and the explanation of how much of what we do now that affects the oceans reinforces that cascade. That has been covered somewhat in Joe Romm’s blog, www.climateprogress.com. I will simply note that the end point will be a mass ocean species destruction comparable to any in the past, plus massive ocean acid “dead zones” where nothing can live and a time to recover in the thousands of years. One species that might survive is jellyfish – and it has “low nutritional value,” i.e., you can’t live on jellyfish.

The one thing that no one seems to be covering is what they say we should do to avoid this. They say that everyone on Earth must stop all ocean misuses now, and to do this the UN should set up a global enforcement body. The burden of proof will be on all ocean users – yes, that includes ocean liners and drillers for oil, gas, and minerals – to show that their next use will not be harmful, else they can’t do it. Contributions to the body would be mandatory, and it would have jurisdiction over the “High Seas” that aren’t the property of particular nations, but obviously it will affect waters that are now said to be the property of particular nations, as well as fisheries.

If you want to go fish, get permission from the global enforcement body. If you want to ship components from abroad, get permission from the global enforcement body. If you want to drill in the Arctic now that it’s getting warmer and less icy, get permission. If you dump fertilizer and waste into rivers and it’s washed out to sea, the commission will be after you. And the commission’s key criteria will be: Does this add to the carbon footprint? Does this make a dead ocean more likely? Is this a sustainable use?

As far as I can see, the only reason the workshop would recommend such a thing is that the situation is that serious. And it’s serious not only because we lose seafood, but because the ocean will reach its capacity for absorbing the excess carbon we’re dumping in the atmosphere, and then global warming on land will get worse, faster than we expect even now – leading to faster sea rise and more massive storms that “salt” estuaries that product a significant proportion of the world’s food, more droughts over much of the world that desertify another major proportion of the world’s food, and possibly to “toxic blooms” in the waters next to the land that periodically release toxic gases that kill those living on the shore.

Just thinking about the ocean, near which I have lived for most of my life, being dead makes me sick to my stomach.

In the End, Godel Has Won

2011-06-12T17:45:00.000-07:00

This post was originally written last fall, and set aside as being too speculative. I felt that there was too little evidence to back up my idea that “accepting limits” would pay off in business.

Since then, however, the Spring 2011 edition of MIT Sloan Management Review has landed on my desk. In it, a new “sustainability” study shows that “embracers” are delivering exceptional comparative advantage, and that a key characteristic of “embracers” is that they view “sustainability” as a culture to be “wired into the business” – “it’s the mindset”, says Bowman of Duke Energy. According to Wikipedia, the term “sustainability” itself is fundamentally about accepting limits, including environmental “carrying capacity” limits, energy limits, and limits in which use rates don’t exceed regeneration rates.

This attitude is in stark contrast to the attitude pervading much of human history. I myself have grown up in a world in which one of the fundamental assumptions, one of the fundamental guides to behavior, is that it is possible to do anything. The motto of the Seabees in World War II, I believe, was “The difficult we do immediately; the impossible takes a little longer.” Over and over, we have believed adjustments in the market, inventions and advances, daring to try something else, an all-out effort, something, anything, can fix any problem.

In mathematics, they, too, believed at the turn of the century that any problem was solvable: that any truth of any consistent, infinite mathematical system could be proved. And then Kurt Godel came along and showed that in every such system, either you could not prove all truths or you could also prove false things, one or the other. And over the next thirty years, mathematics applied to computing showed that some problems were unsolvable, and others had a fundamental lower limit on the time taken to solve the problem that meant that they could not be solved before the universe ended. By accepting these limits, mathematics and programming have flourished.

This mindset is fundamentally different from the “anything is possible” mindset. It says to work smarter, not harder, by not wasting your time on the unachievable. It says to identify the highly improbable up front and spend most of your time on solutions that don’t involve that improbability. It says, as agile programming does, that we should focus on changing our solutions as we find out these improbabilities and impossibilities, rather than piling on patch after patch. It also says, as agile programming does, that while by any short-run calculation the results of this mindset might seem worse than the results of the “anything is possible” mindset, over the long run – and frequently over the medium term – it will produce better results.

It seems more and more apparent to me that we have finally reached the point where the “anything is possible” approach is costing us dearly. I am speaking specifically about climate change – one key driver for the sustainability movement. The more I become familiar with the overwhelming scientific evidence for massive human-caused climate change and the increasing inevitability of at least some major costs of that change in every locality and country of the globe, the more I realize that an “anything is possible” mentality is a fundamental cause of most people’s failure to respond adequately so far, and a clear predictor of future failure.

Let me be more specific: as noted in the UN scientific conferences and recent additional data, “business as usual” is leading us to a carbon dioxide concentration of 1000 ppm in the atmosphere, of which about 450 ppm or 150-200 ppm over the natural amount is already “baked in”. This will result, at minimum, in global increases in temperature of 5-10 degrees Fahrenheit, which will result, among other things, in order-of-magnitude increases in the damage caused by extreme weather events, the extinction of many ecosystems supporting existing urban and rural populations – because many of these ecosystems are blocked from moving north or south by paved human habitations – so that food and shelter production must both change their location and find new ways to deliver to new locations, movement of all populations from locations on seacoasts up to 20 feet above existing sea level, and adjustment of a large proportion of heating and cooling systems to a new mix of the two – not to mention drought, famine, and economic stress. And these are just the effects over the next 60 or so years.

Adjusting to this will place additional costs on everyone, very possibly similar to a 10% tax yearly on every individual and business in every country for the next 50 years, no matter how wealthy or adept. Continuing “business as usual” for another 30 years would result in a similar, almost equally costly additional adjustment.

Our response to this so far has been in the finest tradition of “anything is possible”. We search for technological fixes under the belief that they will solve the problem, since they appear to have done so before. Most of us – except the embracers – assume that existing business incentives, focused on cutting costs – but these costs have not yet occurred – will somehow respond years before the impact begins to be felt. (Embracers, by the way, actively seek out new metrics to capture things like carbon emissions’ negative effects) We are skeptical and suspicious, since those who have predicted doom before, for whatever reason, have generally seemed to have turned out to be wrong. We hide our heads in the sand, because we have too much else to do and concerns that seem more immediate. We are distracted by possible fixes, and by their flaws.

The “embrace limits” mindset for climate changes makes one simple change: accept steady absolute reductions in carbon emissions as a limit. For example, every business, every country, every region, every county accepts that every year, its emissions are to be reduced by 1% in that year. If a business, that business also accepts that its products’ emissions are to be reduced by 1% in that year, no matter how successful the year has been. If a locality does better one year, it still is expected not to increase emissions the next year. If a country rejects this idea, investments from conforming countries are reduced by 1% each year, and products accepted from that country are expected to comply.

But this is a crude, blunt-force suggested application of “embrace limits”. There are all sorts of other applications. Investors will no longer invest in equities that seem to promise 2% long-term returns above historical norms, and will limit the amount of their capital invested in “bets,” because those investments are overwhelmingly likely to be con jobs. Project managers will no longer use metrics like time to deployment, but rather “time to value” and “agility”, because there is a strong possibility that during the project, the team will discover a limit and need to change its objective.

Because, fundamentally, climate change is a final, clear signal that Godel has won. Whether we accept limits or not, they are there; and the less we accept them and use them to work smarter, the more it costs us.

HP, IBM, and So On: It's Not the Personalities, People!

2011-05-17T17:43:00.000-07:00

I note from various sources that some VSPs (Very Serious People, to borrow an acronym from Paul Krugman) are now raising questions about HP’s financials in the wake of Mark Hurd’s departure for Oracle. To cherrypick some quotes: “They need to .. regain investor confidence”; “HP is in a difficult situation”; “It sounds like … Hurd took too many costs out of [the services] business”; “HP … now … are known for inconsistency … It could become a value trap.” And, of course, there are comparisons with IBM, Dell, software vendors like Oracle, and so on.

I am certainly not an unalloyed HP booster. In fact, I have made many unflattering comparisons of HP with IBM myself over the years. However, I disagree with the apocalyptic tone of these pronouncements. In fact, I will stick out my neck and predict that HP will not implode over the next 3 years, and it will not fall behind IBM in revenues either, barring a truly epochal acquisition by IBM. I believe that these VSPs are placing too much emphasis on upcoming strategies bearing the imprint of personalities like HP’s Leo Aptheker and IBM’s Sam Palmisano, and not enough emphasis on the existing positioning of IBM, Dell, HP, Microsoft, Oracle, and Apple.

Let’s Start With the Negative!

So what are these problems that I have criticized HP for? Well, let’s start with its solution portfolio. Of the major computer vendors, HP may be the closest to a conglomerate – and that’s not a good thing. Let’s see, it has a printer/all-in-one company, a PC company, one or two server companies (including Tandem), a business/IT services/outsourcing company, and even, if you want to stretch a point, an administrative software utility company (the old SystemView) with some more recent software (Mercury Interactive testing) attached. Moreover, because HP has until very recently not tried very hard to stitch these together either as solutions or as a software/hardware stack, they are not as integrated as others – strikingly, not as integrated as IBM, which was once known for announcing global solutions whose components turned out to be in the early stages of learning to talk to each other. At first glance, HP’s endowments seem impressive; closer up, these seem, as someone once said in another context, like cats fighting in a burlap bag.

Moreover, HP, unlike any of the other companies I have mentioned except Dell, simply does not have software in its DNA. Back in the early 1990s, a Harvard Business Review article asserted that hardware companies at the time would suffer unless they became primarily services companies; I asserted then, and I assert now, that they also should become software companies.

I believe that this lack of software solutions and development personnel has several bad effects that have decreased HP’s revenues and profits by at least 20% over the last 20 years. Software development connects you with the open source community, the consumer market, and the latest technologies that impinge on computer vendors quite effectively. It allows your services arm to offer more leading-edge services, rather than trying to customize others’ software in a quick and dirty fashion for one particular services engagement. And, in the end, it moves hardware development ahead faster, as it focuses chip development on major real-world workloads that your software supports. Moreover, as IBM itself has proven, even if investment in software doesn’t pay off immediately, eventually you get it right.

A third, more recent problem, does relate to Mark Hurd’s cost focus – although the same might be said for IBM. A truism of business strategy proved by the problems created by CFO dominance at US car companies in the 1980s and 1990s is that too long a focus on the financials rather than product innovation costs a company dearly. It is quite possible that HP has eaten its innovation seed corn in the process of turning into a “consistent” money maker.

Finally, HP has in the past had a tendency in its hardware products to be “the nice alternative”: not locking you in or like Sun or Microsoft, willing to provide a platform for Oracle and Microsoft databases, open to anyone’s middleware. Whatever the merits of that approach, it creates a perception among customers that HP is not leading-edge in the sense that Apple, or even Microsoft and Oracle, are. Twenty-one years ago, in my first HP briefing, famous analyst Nina Lytton showed up in a brilliant pink outfit and immediately announced that HP’s strategy reminded her of a “great pink cloud.” That sort of rosy but not clear-cut presentation of one’s strategy and future plans does not create the sort of excitement among customers that Steve Jobs’ iPhone and iPad announcements, or even IBM’s display of Watson, do.

It Doesn’t Matter

And yet, when we look at HP vs. IBM in the longer run – from 1990, when I started as an analyst, to now – the ongoing success of HP is striking. At the start, IBM’s yearly revenues were in the $80s billion, and HP’s perhaps a quarter as much. Today, HP’s revenues are perhaps 1/3 greater, IBM at around a $100B run rate and HP perhaps at $135B. Some of that HP growth can be attributed to acquisitions; but a lot of it comes from growth of its core business and its acquisitions. To put it another way, IBM has been very successful at growing its profit margin; HP has been very successful at growing.

And growth at that scale is not easy. Companies have been trying to knock IBM off its Number 1 perch in revenues since the 1960s, and only HP has succeeded. Nobody else is in striking distance yet – Microsoft is at a $70B run rate, apparently, with seemingly no prospects of exceeding $100B in the next couple of years.

The reason, I think, is HP’s acquisition of Compaq back in the 1990s. Since then, having beaten back Dell’s challenge, HP is in a very strong position in the PC-form-factor scale-out markets. Despite recent apparent gains by System x, IBM focuses on the business market, and all of the other vendors mentioned above do not compete in PC hardware. Moreover, the PC market aligns HP with Intel and Microsoft, and thereby is relatively well protected from IBM’s POWER chips or even Oracle/Sun’s SPARC chipset, whatever life that has left in it (there is still no sign that AMD threatens Intel’s chip dominance significantly).

So let HP’s scale-up servers and storage falter in technology (e.g., the Itanium bet) relative to IBM and EMC, if they do; with the steady decrease in market share by Sun, HP is, and will in the short term remain, the IBM alternative in this market. Let Dell and IBM’s System x tout their business Linux scale-out prowess; the prevalence of existing scale-out PCs in public clouds and Microsoft LOBs means that HP is well positioned to handle competition in those areas over the next couple of years.
And who else but IBM can attack HP? Oracle may talk big, but Sun’s market share appears to be shrinking, and 15 years of Larry Ellison talking about the virtual desktop and Oracle Database handling your filesystem have failed to make a dent in Windows, much less Wintel. Microsoft has no need to move into hardware, and apparently no desire. Apple appears to be playing a different sport altogether.

In fact, the only serious threat to HP over the short term is any major movement of consumers off PCs and laptops as they move to smartphones and tablets. Here again, I think, analysts are too apocalyptic. Yes, iPhones can handle an astonishing range of consumer tasks, but not as easily or in as sophisticated a fashion as PCs, and users still continue to want to create and organize personal stores of photos etc. as well as share them – something the smartphone does not yet do. Meanwhile, the tablet offers the small form factor and attractive user interface that today’s laptop does not; but it is more likely that the tablet will acquire PC features, than that it will morph into an iPhone.

And Whither IBM?

In fact, an interesting question, given IBM’s status as the most direct competitor of HP, is whether IBM can begin to speed up its revenue growth. IBM has been delivering strong financials for almost 20 years, while talking a good game about innovation. In fact, I would say that they have indeed been innovative in some areas – but not enough yet to grow their revenues fast. Will the big innovation be green technology? The cloud? Analytics? Because, let’s face it, the only two things that recently have delivered big revenue gains are cell phones and Web 2.0/social media – and Apple, Google, and Facebook are the ones reaping the most revenues from these, not IBM.

In fact, as I have argued, IBM can do quite well with its present strong position in scale-up, but it cannot dominate the business side of computer markets when HP, Microsoft, and Intel have such a strong position in scale-out, nor can it match HP in consumer markets – and these affect business sales.

User Bottom Line: Don’t Panic, Do Buy Both

It would be nice, wouldn’t it, to be back in the old days when no one ever got fired for buying IBM systems, or Oracle databases? Well, those days are gone forever, and no blunder or inspired move, by Aptheker, Palmisano, Hurd, Ellison, Ballmer, Dell, or Jobs, will bring them back.

Given that, the smart IT buyer will acquire a little of each, in the areas in which each is best. It is true, for example, that IBM has exceptional services scope that allows effective integration – including integration of scale-out technology from Microsoft, Intel, and HP, or for that matter (System x) from IBM itself. This “mixed” enterprise architecture is the New Normal; vendor lock-in or a tide of Web innovation fueled by an Oracle and a Sun is so 1990s.

It is said that when Mary Queen of Scots wed the King of France, she was saluted with: “Let others wage war; let you, happy Scotland, bear children” (it’s better in Latin). Let the VSPs and apocalyptic analysts assert that vendor personalities waging war should affect your buying decision; you, happy CIO, should buy products from any of the vendors mentioned above, without worrying that a vendor is about to go belly-up in two seconds. And the vendors that have the greatest ability to integrate, like IBM and HP, will do quite well if you do.

Don't Blame The Developer!

2011-03-31T09:02:00.000-07:00

Recently, I read yet another blog post in which a user ranted against an annoying feature of the latest word processing consumer software that wiped out work back to an autosave. The problem, he seemed to think, lay with those annoying developers who kept adding features to products that weren’t really needed, making it harder to understand and use, and increasing the chance of accidental mistakes causing a meltdown. Sorry, but I disagree. And having helped develop a word processor back in the late 70s and a file system back in the late 80s, and followed the field as an analyst since, I’ve seen the world from both sides now, as a computer scientist and as a marketer; so I think I have a good perspective on the problem.

There are two related problems with most software used by consumers: what I call orthogonality (in math, elegance) and metaphor. Orthogonality says that your basic operations on which everything else is built are "on the same level" and together they cover everything -- power plus intuitive sense to the user. Metaphor says that the idea of how the user operates with this software is a comparison to a model -- and that model should be as powerful as possible.

In the case of word processing (and most other consumer software) all products are not as orthogonal as they should be. One of the reasons the original Word succeeded was that it was more orthogonal than its competitors in its commands: file, edit, etc. are a pretty good take. That means that necessary additions and elaborations are also more orthogonal; the rich get richer.

Where everyone (including Apple and Google) really falls down is in metaphor. To take one example we are still haunted by: the original metaphor for word processors and other desktop software was, indeed, a physical desktop, with a one-level filing system underneath. It took a while for people to accept a wholly unfamiliar metaphor, the folder within folder within folder -- even though it was far more powerful, easier to program and upgrade, and, on average, made things easier for the user who learned the new metaphor. For the last 25 years, all consumer software vendors have consistently rejected an even better metaphor: what is called in math the directed acyclic graph. This would allow multiple folders to access the same folder or file: essentially, incredibly easy cross-filing. I know from design experience that using this approach in a word processor or other consumer software would be almost as intuitive as the present "tree" (folder) metaphor. Instead, software vendors have adopted kludges such as "aliases" that only make the product far more complicated. The same is true of supporting both dynamic and static file storage on the desktop (too long a discussion).

The reason orthogonality and good metaphor rarely get done or last is that almost never do a good developer and a good marketer (one who understands not only what consumers say they want but what they could want) connect in software development. Sorry, I have watched Steve Jobs for 30 years now, and while he is superb at the marketing end, he does very badly at understanding metaphor plus orthogonality from the mathematical/technical point of view. And the rest are probably worse.
The net result for Word, and, sorry Mr. User, for all those "better" previous word processors, is that time makes all these problems worse, and it results in either failure to incorporate valuable new metaphors (and I do think that spell- and grammar-checking are overall better than the old days, and worth the frustrations of poor orthogonality and awkward usage in isolated cases) or retrofitting of a more orthogonal approach. Specifically, I suspect (because I think I've seen it) that supporting the old WordPerfect ctrl-a approach for both the old Word command interface and the new toolbar style plus new features added just one too many dangerous key combinations next to the ones traditionally used. You miss, you pay -- and yes, the same thing will happen with touch screen gestures.

Whether this business game is worth the candle I leave to anyone who is a user. I know, however, from long experience, who to blame -- and it's not primarily the latest developer. Fundamentally, I blame a long series of marketers who at least are told about the problem -- I've told many of them myself -- and when push comes to shove, keep chickening out. The reasons for not doing orthogonality with a better metaphor always seem better to them at the time, the development time longer, the risks of the new higher, the credibility of the trouble-maker suspect, and they won't be around to deal with the problems of playing it safe. These are all good superficial reasons; but they're wrong. And we all suffer, developers not least – because they have to try to clean up the mess.

So give a little blame, if it makes you feel better, to the latest developer or product upgrade designer, who didn’t understand how the typical consumer would use the latest version; give a little blame, if you can figure out how, to the previous developers and designers, who didn’t anticipate these problems. But the marketer, be he the CEO or a lowly product marketer, who makes the fundamental decision about where to go next is really the only person who can hear both the voice of the consumer and the voice that understands the technical/mathematical usefulness of orthogonality and a good metaphor. The marketer is the one who has a real opportunity to make things better; to him, the user should assign the primary blame.

The writer Peter Beagle, commenting favorably on JRR Tolkien’s anti-heroes, once wrote “We worship all the wrong heroes.” I won’t go that far. But I will say that we need to hold our present consumer software marketing heroes to higher standards. And stop reflexively making the developer the villain.

Yet Another Top Ten Classical Composers List

2011-02-13T16:55:00.000-08:00

I see that Anthony Tommasini of the NY Times has joined the crowd who attempt to rank the top classical composers of all time. Since, as Josephine Tey once wrote, there are far too many books on such subjects written as is, I was strongly tempted not to respond to his article. However, it does give me another reason to remember some of the cherished experiences of my life – and I would like to try, yet again, to lift a lone voice against the “loud orchestra bigots” who dominate today’s classical music scene.

Aside from the differences over whether loudness should be valued for its own sake and whether works with large numbers of instruments or voice involved should be valued over others, I differ with Mr. Tommasini in one more subtle but important way: he appears to value the ability of the composer to stretch the compositional palette by being innovative; I value the composer’s ability to do so extremely well. Yes, Stravinsky and Bartok did radical things, and did them pretty well; but, to my mind, only pretty well. By contrast, when Mozart and Sibelius stretched the boundaries, they did so with exceptional integration into an overall “sound personality.”

Enough of that. Here’s my list.

(1) Brahms. I really wrestled with this, between Brahms and Mozart. I don’t think that Brahms’ symphonies are the best, nor what I have heard of his vocal works. But, over and over again, his chamber works, his solo works, his harmonies are at the very highest level, and all in the service of a feeling that I see as so profound as to be indescribable. I confess that I see the Brahms Horn Trio as among the two or three greatest pieces of music of all time.

(2) Mozart. Yes, it was a little tough between him and Beethoven. Beethoven is more of our era; his melodies appeal to all, and his harmonies connect to ours. But the real problem is that we don’t know how to perform Mozart. There is wonderful feeling there, and – if you understand conventions – a constant play with the ear’s expectation for how to finish that the best performers bring out, to give an air of endless, blossoming astonishment. And, yes, his operas show his knack for the single, piercing phrase that makes the whole piece memorable. The main problem is that his later works never use the solo strings to their full potential. Other than that, every inner instrument has its song, and every harmony is fresh and new. We just have to learn how to play him – strings, especially.

(3) Beethoven. I got there eventually. Mostly, I have been turned off for years by his brute-force repetition of octaves and the way he forces you to use the maximum number of instruments as loudly as possible. But in pieces such as the Violin Concerto or Romanzen, not to mention some parts of the chamber music, the performer who really knows how to bring out the soft voice will be extraordinarily rewarded. And, of course, his melodies range from memorable to great.

(4) Bach. In some ways, I feel that Bach should be rated lower than this. He is, above all, obsessed by the beat, and too much of an organ/small orchestra composer. His voice compositions are great but don’t truly wed the words to the music in the way Wagner and Mozart do. However, there are so many superb melodies with such superb harmony, if you spend extraordinary effort as a performer – the Double Violin Concerto, the Violin and Oboe concerto, some of the interludes from the Cantatas, the Passacaglia and Fugue. Now there was a composer that truly valued the bass voice.

(5) Schubert. Yes, the songs are extraordinary; but the chamber music is truly great. There is a story that a musician put one of the melodies from one of his trios on his tombstone; and having heard it, I understand why. It’s odd that the Trout Quintet and Death and the Maiden quartet, his best known chamber music, are not really his best chamber music. Again, his chamber music must be played just right to understand its greatness.

(6) Sibelius. No, this isn’t really about chamber pieces. Sibelius is an exception to that rule; he achieves his great effects in orchestral works, with the possible exception of the Violin Concerto. But in those works, the ability to find the right harmony, and to challenge the ear, puts him at the top of the second tier.

(7) Ravel. Yes, I know that everyone likes Debussy, and thinks he’s the best of the French. But I consistently find that Debussy is a one-trick pony as far as harmony goes (and it’s a very good trick!). Ravel, on the other hand, uses instruments like the English horn to achieve harmonies that are superb, and works like Gaspard de la Nuit and the Quartet show his other, incredibly varied gifts.

(8) Wagner. Yes, he forces singers to shout endlessly, and yes, he does go on and on. But he uses consonance, assonance, and alliteration superbly in his lyrics, forming an amazingly integrated, powerful whole. Tristan and Isolde’s leitmotif is enough for any composer.

(9) Stravinsky. The Classical Symphony shows both his strengths and limitations: superb orchestration and the ability to sing, set against an inability to plumb the full depths of a phrase. That’s good enough for the third tier.

(10) Barber. I confess that I don’t know enough about Barber to make this more than tentative. But the quartet that everyone plays now plus the first two movements of the Violin Concerto make him stand out for me.

Ones with occasional gifts that make me regret leaving them out: Monteverdi, of course, because he is the best of the best in madrigals, imho. Schumann, a little for his symphonies but more for his occasional chamber genius, and especially for the Quintet. Copland – although I think his “Western” pieces are overrated, Fanfare and the Shaker hymn are not. Gershwin – untutored as he is, both Rhapsody in Blue and Porgy and Bess deserve better ratings. Puccini – I still feel I am learning to plumb the depths of Turandot, whatever the lack of depth of orchestration of his other works. Mendelssohn – for his Piano Trio, one of the truly great works.

Overrated: Mahler – sorry, not a good enough ratio of quality to length. Bartok – what his left hand giveth with feeling, his right hand taketh away with atonal lack of point; and yes, I do know his violin concerto. Debussy – I might have put him at 11, but see above. Shostakovitch – actually, Symphony Number 11 sounds best, despite its being forced to be the most tonal of all; which is damning with faint praise. Dvorak – his gifts are superb, and then he forces the strings into unnatural high octaves and harmonies.

Don’t know enough – Verdi.

Wonder about; maybe there’s something there I don’t know about – Faure, Poulenc, Grieg, Elgar, Vivaldi, Gade.

Never great, but good enough – Haydn, Berlioz, sometimes Bizet, Ward, some Rossini, Handel (all right, sometimes he’s great; but not in long enough bursts).

And so many more, that I will never discover.

IBM's Watson: Less Than It Seems, More Maybe Than It Will Seem

2011-02-11T14:05:00.000-08:00

As we move towards yet another revival of Jeopardy-type game popularity, IBM’s Watson – a computer competing in Jeopardy with Americans -- has garnered an enormous amount of attention. The excitement seems to center around the following beliefs:

• Watson is an enormous step forward in computer capabilities for understanding and interacting with humans;
• Straightforward evolution of these capabilities via “learning” will lead to human-level “artificial intelligence” (AI) sooner rather than later;
• In the meanwhile, Watson can be applied to a wide variety of BI/analytic functions in the typical business to deliver amazing new insights;
• These insights will have their primary value in enhancing existing customer-facing business processes such as help desk and decreasing fraud.

I think that all of these beliefs are wrong. I see Watson as a minor step forward for existing AI, natural-language processing, and computer learning software. I see a huge gulf remaining between Watson and most useful “intelligence,” and no indication of software on the horizon to “evolve” or “learn” to bridge that gap. I think that Watson’s advances will only be applicable to a small subset of business data, and will yield only incremental “deep insight.” And finally, I see the immediate applications of Watson as cited by IBM – help-desk improvement and fraud detection, for example – as far less valuable than, say, improving eDiscovery or restaurant finding by less-dumb response to text-oriented searches.

And yet, the potential is there for the smart business to use a successor to Watson effectively. The real potential of Watson is cross-domain correlation: the ability to identify connections between two diverse, well-defined areas of knowledge. The same process that draws the relationship between customer and business ever tighter serves to exclude many potential customers ever more strongly, often because the language becomes different: how easy is it for the makeup expert to understand the computer geek, and vice versa? Customizing interactions by balancing the potential customer’s and the company’s domain of expertise makes acquiring new customers more cost-effective. That’s not computer evolution or a New New Thing; but once the excitement dies down and the disappointment sets in, it’s nothing to sneeze at.

Deep Analysis of Deep Analysis
First, let’s pierce through the hype to understand what, from my viewpoint, Watson is doing. It appears that Watson is building on top of a huge amount of “domain knowledge” amassed in the past at such research centers as GTE Labs, plus the enormous amount of text that the Internet has placed in the public domain – that’s its data. On top of these, it places well-established natural-language processing, AI (rules-based and computer-learning-based), querying, and analytics capabilities, with its own “special sauce” being to fine-tune these for a Jeopardy-type answer-question interaction (since Watson is an analytical successor to IBM’s Deep Thought chess-playing machine, should we call it Deep Analysis?). Note that sometimes Watson must combine two or more different knowledge domains in order to provide its question: “We call the first version of this an abacus (history). What is a calculator (electronics)?”

Nothing in this design suggests that Watson has made a giant leap in AI (or natural-language processing, or analytics). For 40 years and more, researchers have been building up AI rules, domains, natural-language translators, and learning algorithms – but progress towards meeting a true Turing test, in which the human side of the interaction can never tell that a computer is the other side of the interaction, has been achingly slow. All that the Jeopardy challenge shows is that the computer can now provide one-word answers to a particular type of tricky question – using beyond-human amounts of data and of processing parallelism.

Nor should we expect this situation to change soon. The key and fundamental insight of AI is that when faced with a shallow layer of knowledge above a vast sea of ignorance, the most effective learning strategy is to make mistakes and adjust your model accordingly. As a result, brute-force computations without good models don’t get you to intelligence, models that attempt to approximate human learning fall far short of reality, and models that try to invent a new way of learning have turned out to be very inefficient. To get as far as it does, Watson uses 40 years of mistake-driven improvements in all three approaches, showing that it’s going to require many years of further improvements – not just letting the present approach “learn” more – before we can seriously talk about human and computer intelligence as apples and apples.

The next point is that Jeopardy is all about text data: not numbers, yes, but not video, audio, or graphics (so-called “unstructured” data), either. The amount of text on Web sites is enormous, but it’s dwarfed by the amount of other data from our senses inside and outside the business, and in our heads. In fact, even in the “semi-structured data” category to which Watson’s Jeopardy data belongs, other types of information such as e-mails, text messages, and perhaps spreadsheets are now comparable in amount – although Watson could to some extent extend to these without much effort. In any case, the name of the game in BI/analytics these days is to tap into not only the text on Facebook and Twitter, but also the information inherent in the videos and pictures provided via Facebook, GPS locators, and cell phones. As a result, Watson is still a ways away from providing good unstructured “context” to analytics – rendering it far less useful to BI/analytics. And bear in mind that analysis of visual information in AI, as evidenced in such areas as robotics, is still in its infancy, used primarily in small doses to direct an individual robot.

As noted above, I see the immediate value of Watson’s capabilities to the large enterprise (although I suppose the cloud can make it available to the SMB as well) to be more in the area of cross-domain correlation in existing text databases, including archived emails. There, Watson could be used in historical and legal querying to do preliminary context analysis, to avoid having eDiscovery take every reference to nuking one’s competitors as a terrorist threat. Ex post facto analysis of help desk interactions (one example that IBM cites) may improve understanding of what the caller wants, but Watson will likely do nothing for user irritation at language or dialect barriers from offshoring, not to mention encouraging “interaction speedup” that the most recent Sloan Management Review suggests actually loses customers.

The Bottom Line: Try, Get What You Need
Still, I do believe that a straightforward evolution of Watson’s capabilities can be highly valuable to the really smart, really agile enterprise. Or, as the Rolling Stones put it, “You can’t always get what you want, but if you try, somehow, you get what you need.”

In this case, trying Watson-type analytics will involve applying it primarily to cross-domain correlation, primarily to Internet data outside the enterprise, and primarily in ways that are well integrated with better unstructured-data analytics. Such an engine would sieve the sensor-driven Web in real time, detecting trends not just in immediate “customer prospects” but also in the larger global markets that have been “included out” of enterprises’ present market definitions. The enterprise can then use these trends to identify niches whose entry would require least “translation” between a niche prospect’s knowledge domain and company jargon, and then provide such translation for the initial sale. Gee, maybe even real estate contracts might become more understandable … nah.

More importantly, such an application would reinforce a company’s efforts to become more outward-facing and (a complementary concept) more truly agile. It abets a restructuring of company processes and culture to become more focused on change and on reactive and proactive evolution of solutions in interaction with customers and markets.

What I am suggesting to IT buyers is that they do not rush to acquire Watson as the latest fad. Rather, they should determine at what point IBM or similar analytics products intersect with a long-term company strategy of Web-focused unstructured-data analytics, and then integrate cross-domain correlation into the overall analytics architecture. Try to implement such a strategy. And find, somehow, that you get from Watson-type technology not what you wanted, but what you need.

Another Requiem: Attachmate Acquires Novell

2010-11-29T15:32:00.000-08:00

And so, another founder of the computing industry as we know it today officially bites the dust. A few days ago, Attachmate announced that it was acquiring Novell – and the biggest of the PC LAN companies will be no more.

I have more fond memories of Novell than I do of Progress, Sun, or any of the other companies that have seen their luster fade over the last decade. Maybe it was the original facility in Provo, with its Star Trek curving corridors and Moby Jack as haute cuisine, just down the mountain from Robert Redford’s Sundance. Maybe it was the way that when they sent us a copy of NetWare, they wrapped it in peanuts instead of bubble wrap, giving us a great wiring party. Or maybe it was Ray Noorda himself, with his nicknames (Pearly Gates and Ballmer the Embalmer) and his insights (I give him credit for the notion of coopetition).

But if Novell were just quirky memories, it wouldn’t be worth the reminiscence. I firmly believe that Novell, more than any other company, ushered out the era of IBM and the Seven Dwarves, and ushered in the world of the PC and the Internet.

Everyone has his or her version of those days. I was at Prime at the time, and there was a hot competition going on between IBM at the high end and DEC, Wang, Data General, and Prime at the “low end”. Even with the advent of the PC, it looked as if IBM or DEC might dominate the new form factor; Compaq was not a real competitor until the late 1980s.

And then along came the PC LAN companies: 3Com, Banyan, Novell. While IBM and the rest focused on high end sales, and Sun and Apollo locked up workstations, the minicomputer makers’ low ends were being stealthily undercut by PC LANs, and especially from the likes of Novell. The logic was simple: the local dentist, realtor, or retailer bought a PC for personal use, brought it to the business, and then realized that it was child’s play – and less than $1K – to buy LAN software to hook the PCs in the office together. It meant incredibly cheap scalability, and when I was at Prime it gutted the low end of our business, squeezing the mini makers from above (IBM) and below (Novell).

There was never a time when Novell could breathe easily. At first, there were Banyan and 3Com; later, the mini makers tried their hand at PC LANs; then came the Microsoft partnership with IBM to push OS/2 LAN Manager; and finally, in the early 1990s, Microsoft took dead aim at Novell, and finally managed to knock them off their perch. However, until the end, NetWare had two simple ideas to differentiate it, well executed by the “Magic 8” (the programmers doing fundamental NetWare design, including above all Drew Major): the idea that to every client PC, the NetWare file system should look like just another drive, and the idea that frequently accessed files should be stored in main memory on the server PC, so that, as Novell boasted, you could get a file faster from NetWare than you could from your own PC’s hard drive.

Until the mid 1990s, analysts embarrassed themselves by predicting rapid loss of market share to the latest competitor. Every year, surveys showed that purchasing managers were planning to replace their NetWare with LAN Server, with LAN Manager, with VINES; and at the end of the year, the surveys would show that NetWare had increased its hold, with market share in the high 70s. Why? Because what drove the market was purchases below the purchasing manager’s radar screen (less than the $10K that departments were supposed to report upstairs). One DEC employee told me an illustrative story: while DEC was trying to mandate in-house purchase of its PC LAN software, the techies at DEC were expanding their use of NetWare by leaps and bounds, avoiding official notice by “tunneling” NetWare communications as part of the regular DEC network. The powers that be finally noticed what was going on because the tunneled communications became the bulk of all communications across the DEC network.

In the early 1990s, Microsoft finally figured out what to do about this. Shortly after casting off OS/2 and LAN Manager, Microsoft developed its own, even more basic, PC LAN software that at first simply allowed sharing across a couple of “peer” PCs. Using this as a beachhead, Microsoft steadily developed Windows’ LAN capabilities, entirely wrapped in the Windows PC OS, so that it cost practically nothing to buy both the PC and the LAN. This placed Novell in an untenable position, because what was now driving the market was applications developed on top of the PC and LAN OS, and NetWare had never paid sufficient attention to LAN application development; it was easy for Microsoft to turn Windows apps into Windows plus LAN apps, while it was very hard for Novell to do so.

Nevertheless, Novell’s core market made do with third-party Windows apps that could also run on NetWare, until the final phase of the tragedy: Windows 2000. You see, PC LANs always had the limitation that they were local. The only way that PC LAN OSs could overcome the limitations of geography was to provide real-time updates to resource and user data stored in multiple, geographically separate “directories”: in effect, to carry out scalable multi-copy updates on data. Banyan had a pretty good solution for this, but Microsoft created an even better one in Windows 2000, well before Novell’s solution; and after that, as the world shifted its attention to the Internet, Novell was not even near anyone’s short list for distributed computing.

Over the last decade, Novell has not lacked good solutions; its own directory product, administrative and security software, virtualization software, and most recently what I view as a very nice approach to porting Windows apps to Linux and mainframes. Still, a succession of CEOs failed to turn around the company, and, in the ultimate irony, Attachmate, with strengths and a long history itself in remote PC software, has decided to take on Novell’s assets.

I think that the best summing up of Novell’s ultimate strategic mistake was the remark of one of its CEOs shortly after assuming command: “Everyone thinks about Microsoft as the biggest fish in the ocean. It is the ocean.” In other words, Novell would have done better by treating Microsoft as the vendor of the environment that Novell had to support, and aiming to service that market, rather than trying to out-feature Microsoft. But everyone else made that mistake; why should Novell have been any different?

We are left not only with Novell’s past contributions to computing, but also with the contributions of its alumni. Some fostered the SMB market with products like the Pervasive database; some were drivers of the UNIX standards push and later the TP monitors that led to today’s app servers. One created the Burton Group, a more technically-oriented analyst firm that permanently improved the quality of the analyst industry.

And we are also left with an enthusiasm that could not be contained by traditional firms, and that moved on to UNIX, to the Web, to open source. The one time, in the late 1980s, I went to Novell’s user group meeting, it was clearly a bit different. After one of the presentations, a LAN servicer rose to ask a question. “So-and-so, LANs Are My Life”, he identified himself. That was the name of his firm: LANs Are My Life, Inc. It’s not a bad epitaph for a computer company: we made a product so good that for some people – not just Novell employees – it was our life. Rest in peace, Novell.

BI For The Masses: 3 Solutions That Will Never Happen

2010-10-10T12:12:00.000-07:00

I was reading a Business Intelligence (BI) white paper feed – I can’t remember which – when I happened across one whose title was, more or less, “Data Visualization: Is This the Way To Attract the Common End User?” And I thought, boy, here we go again.

You see, the idea that just a little better user interface will finally get Joe and Jane (no, not you, Mr. Clabby) to use databases dates back at least 26 years. I know, because I had an argument with my boss at CCA, Dan Ries iirc (a very smart fellow), about it. He was sure that with a fill-out-the-form approach, any line employee could do his or her own ad-hoc queries and reporting. Based on my own experiences as a naïve end user, I felt we were very far from being able to give the average end user an interface that he or she would be able or motivated to use. Here we are, 26-plus years later, and all through those years, someone would pipe up and say, in the immortal words of Bullwinkle, “This time for sure!” And every time, it hasn’t happened.

I divide the blame for this equally between vendor marketing and IT buying. Database and BI vendors, first and foremost, look to extend the ability of specific targets within the business to gain insights. That requires ever more sophisticated statistical and relationship-identifying tools. The vendor looking to design a “common-person” user interface retrofits the interface to these tools. In other words, the vendor acts like it is selling to a business-expert, not a consumer, market.

Meanwhile, IT buyers looking to justify the expense of BI try to extend its use to upper-level executives and business processes, not demand that it extend the interface approach of popular consumer apps to using data, or that it give the line supervisor who uses it at home a leg up at work. And yet, that is precisely how Word, Excel, maybe PowerPoint, and Google search wound up being far more frequently used than SQL or OLAP.

I have been saying things like this for the last 26 years, and somehow, the problem never gets solved. At this point, I am convinced that no one is really listening. So, for my own amusement, I give you three ideas – ideas proven in the real world, but never implemented in a vendor product – that if I were a user I would really like, and that I think would come as close as anything can to achieving “BI for the masses.”

Idea Number 1: Google Exploratory Data Analysis

I’m reading through someone’s blog when they mention “graphical analysis.” What the hey? There’s a pointer to another blog, where they make a lot of unproven assertions about graphical analysis. Time for Google: a search on graphical analysis results in a lot of extraneous stuff, some of it very interesting, plus Wikipedia and a vendor who is selling this stuff. Wikipedia is off-topic, but carefully reading the article shows that there are a couple of articles that might be on point. One of them gives me some of the social-networking theory behind graphical analysis, but not the products or the market. Back to Google, forward to a couple of analyst market figures. They sound iffy, so I go to a vendor site and get their financials to cross-check. Not much in there, but enough that I can guesstimate. Back to Google, change the search to “graphical BI.” Bingo, another vendor with much more market information and ways to cross-check the first vendor’s claims. Which products have been left out? An analyst report lists the two vendors, but in a different market, and also lists their competitors. Let’s take a sample competitor: what’s their response to “graphical analysis” or graphical BI? Nothing, but they seem to feel that statistical analysis is their best competitive weapon. Does statistical analysis cover graphical analysis? The names SAS and SPSS keep coming up in my Google searches. It doesn’t seem as if their user manuals even mention the word “graph”. What are the potential use cases? Computation of shortest path. Well, only if you’re driving somewhere. Still, if it’s made easy for me … Is this really easier than Mapquest? Let’s try a multi-step trip. Oog. It definitely could be easier than Mapquest. Can I try out this product? All right, I’ve got the free trial version loaded, let’s try the multi-step trip. You know, this could do better for a sales trip than my company’s path optimization stuff, because I can tweak it for my personal needs. Combine with Google Maps, stir … wouldn’t it be nice if there was a Wikimaps, so that people could warn us about all these little construction obstructions and missing signs? Anyway, I’ve just given myself an extra half-hour on the trip to spend on one more call, without having to clear it.

Two points about this. First, Google is superb at free-association exploratory analysis of documents. You search for something, you alter the search because of facts you’ve found, you use the results to find other useful facts about it, you change the topic of the search to cross-check, you dig down into specific examples to verify, you even go completely off-topic and then come back. The result is far richer, far more useful to the “common end user” and his or her organization, and far more fun than just doing a query on graphical data in the company data warehouse.

Second, Google is lousy at exploratory data analysis, because it is “data dumb”: It can find metadata and individual pieces of data, but it can’t detect patterns in the data, so you have to do it yourself. If you are searching for “graphical analysis” across vendor web sites, Google can’t figure out that it would be nice to know that 9 of 10 vendors in the market don’t mention “graph” on their web sites, or that no vendors offer free trial downloads.

The answer to this seems straightforward enough: add “guess-type” data analysis capabilities to Google. And, by the way, if you’re at work, make the first port of call your company’s data-warehouse data store, full of data you can’t get anywhere else. You’re looking for the low-priced product for graphical analysis? Hmm, your company offers three types through a deal with the vendor, but none is the low-cost one. I wonder what effect that has had on sales? Your company did a recent price cut; sure enough, it hasn’t had a big effect. Except in China: does that have to do with the recent exchange rate manipulations, and the fact that you sell via a Chinese firm instead of on your own? It might indeed, since Google tells you the manipulations started 3 weeks ago, just when the price cut happened.

You get the idea? Note that the search/analysis engine guessed that you wanted your company’s data called out, and that you wanted sales broken down by geography and in a monthly time series. Moreover, this is exploratory data analysis, which means that you get to see both the summary report/statistics and individual pieces of raw data – to see if your theories about what’s going on make sense.

In Google exploratory data analysis, the search engine and your exploration drive the data analysis; the tools available don’t. It’s a fundamental mind shift, and one that explains why Excel became popular and in-house on-demand reporting schemes didn’t, or why Google search was accepted and SQL wasn’t. One’s about the features; the other’s about the consumer’s needs.

Oh, by the way, once this takes off, you can start using information about user searches to drive adding really useful data to the data warehouse.

Idea Number 2: The Do The Right Thing Key

Back in 1986, I loved the idea behind the Spike Lee movie title so much that I designed an email system around it. Here’s how it works:

You know how when you are doing a “replace all” in Word, you have to specify an exact character string, and then Word mindlessly replaces all occurrences, even if some should be capitalized and some not, or even if you just want whole words to be changed and not character strings within words? Well, think about it. If you type a whole word, 90% of the time you want only words to be replaced, and capitals to be added at the start of sentences. If you type a string that is only part of a word, 90% of the time you want all occurrences of that string replaced, and capitals when and only when that string occurs at the start of a sentence. So take that Word “replace” window, and add a Do the Right Thing key (really, a point and click option) at the end. If it’s not right, the user can just Undo and take the long route.

The Do The Right Thing key is a macro; but it’s a smart macro. You don’t need to create it, and it makes some reasonable guesses about what you want to do, rather than you having to specify what it should do exactly. I found when I designed my email system that every menu, and every submenu or screen, would benefit from having a Do The Right Thing key. It’s that powerful an idea.

How does that apply to BI? Suppose you are trying to track down a sudden drop in sales one week in North America. You could dive down, layer by layer, until you found that stores in Manitoba all saw a big drop that week. Or, you could press the Break in the Pattern key, which would round up all breaks in patterns of sales, and dig down not only to Manitoba but also to big offsetting changes in sales in Vancouver and Toronto, with appropriate highlighting. 9 times out of ten, that will be the right information, and the other time, you’ll find out some other information that may prove to be just as valuable. Now do the same type of thing for every querying or reporting screen …

The idea behind the Do The Right Thing key is actually very similar to that behind Google Exploratory Data Analysis. In both cases, you are really considering what the end user would probably want to do first, and only then finding a BI tool that will do that. The Do The Right Thing key is a bit more buttoned-up: you’re probably carrying out a task that the business wants you to do. Still, it’s way better than “do it this way or else.”

Idea Number 3: Build Your Own Data Store

Back in the days before Microsoft Access, there was a funny little database company called FileMaker. It had the odd idea that people who wanted to create their own contact lists, their own lists of the stocks they owned and their values, their own grades or assets and expenses, should be able to do so, in just the format they wanted. As Oracle steadily cut away at other competitors in the top end of the database market, FileMaker kept gaining individual customers who would bring FileMaker into their local offices and use it for little projects. To this day, it is still pretty much unique in its ability to let users quickly whip up small-sized, custom data stores to drive, say, class registrations at a college.

To my mind, FileMaker never quite took the idea far enough. You see, FileMaker was competing against folks like Borland in the days when the cutting edge was allowing two-way links between, let’s say, students and teachers (a student has multiple teachers, and teachers have multiple students). But what people really want, often, is “serial hierarchy”. You start out with a list of all your teachers; the student is the top level, the teachers and class location/time/topic the next level. But you next want to see if there’s an alternate class; now the topic is the top level, the time at the next level, the students (you, and if the class is full) at a third level. If the number of data items is too small to require aggregation, statistics, etc.; you can eyeball the raw data to get your answers. And you don’t need to learn a new application (Outlook, Microsoft Money, Excel) for each new personal database need.

The reason this fits BI is that, often, the next step after getting your personal answers is to merge them with company data. You’ve figured out your budget, now do “what if”: does this fit with the company budget? You’ve identified your own sales targets, so how do these match up against those supplied by the company? You download company data into your own personal workspace, and use your own simple analysis tools to see how your plans mesh with the company’s. You only get as complex a user interface as you need.

Conclusions

I hope you enjoyed these ideas, because, dollars to doughnuts, they’ll never happen. It’s been 25 years, and the crippled desktop/folder metaphor and its slightly less crippled cousin, the document/link browser metaphor, still dominate user interfaces. It’s been fifteen years, and only now is Composite Software’s Robert Eve getting marketing traction by pointing out that trying to put all the company’s data in a data warehouse is a fool’s errand. It’s been almost 35 years, and still no one seems to have noticed that seeing a full page of a document you are composing on a screen makes your writing better. At least, after 20 years, Google Gmail finally showed that it was a good idea to group a message and its replies. What a revelation!

No, what users should really be wary of is vendors who claim they do indeed do any of the ideas listed above. This is a bit like vendors claiming that requirements management software is an agile development tool. No; it’s a retrofitted, slightly less sclerotic tool instead of something designed from the ground up to serve the developer, not the process.

But if you dig down, and the vendor really does walk the walk, grab the BI tool. And then let me know the millennium has finally arrived. Preferably not after another 26 years.