Lean and Agile: Partners or Enemies?

An increasing number of commentators on both lean New Product Development (NPD) and agile software development have noted efforts to combine the two. A key reason for the interest in the combination is that lean has long been entrenched in the manufacturing process (e.g., via Kanban and ERP), while agile development has recently established itself as an effective software development strategy, and a major segment of NPD now involves software-only products, software-infused products, or software-plus-hardware solutions. A recent Aberdeen Group study notes that 66% of today’s NPD uses software to drive innovation, and the number is rising rapidly. Meanwhile, several efforts are being made to apply lean-only to software development (cf. Hibbs, Jewett, Sullivan, “The Art of Lean Software Development”).

At first glance, lean is a very good complement for agile. Users have concerns about agile’s ability to scale; lean makes sure that developer “resources” are available for each iterative step in a flexible and cost-effective way. Lean allows a different process time-line each time depending on what resources are available when; agile “spirals” in towards a solution, ensuring a different process timeline for each iteration of a solution “guesstimate”. Both aim to increase value to the customer; both emphasize incremental improvements.

However, a reading of the blogs on lean and agile betrays a fundamental misunderstanding of the agile mindset. Here’s a quote from Martin Heller (http://www.infoworld.com/d/developer-world/should-software-development-be-lean-or-agile-284): “They both strive to improve software quality, reduce waste, increase developer productivity, accept changes to requirements, and prize meeting the customer's real needs.” Agile doesn’t strive to improve software quality; it strives to improve software usefulness. It doesn’t aim to reduce waste; it aims to increase “wasteful” experimentation, which turns out actually to reduce the time-to-value, as a side-effect. Its way of increasing developer productivity – ensuring that development is more in sync with customer needs – is the opposite of lean’s attempt to improve developer productivity by eliminating bottlenecks (and customer interference is potentially a bottleneck). Lean accepts changes to requirements as a necessary part of incremental manufacturing process improvement; agile welcomes and emphasizes changes to requirements as part of improving product design. Lean views customer needs as consisting primarily of quality; agile views customer needs as consisting primarily of rapidly-changing functionality.

Moreover, we have already seen how an emphasis on Deming-like quality can retard product development to the detriment of a firm, as when Motorola lost its race to dominate processors to Intel by being late and customer-insensitive with the design of a “quality”-manufactured product. I can’t quote figures from my recent study under Aberdeen auspices of agile software development vs. other processes; but I can certify that a focus on software quality improvement was far less effective in improving product TCO, ROI, customer value, and ongoing agility than an agile process; in fact, there was no clear long-term benefit to ROI of quality-improving processes at all.

I wish I could say that this is a side-issue, and that once agile is properly understood, lean and agile are indeed complements. After all, some of the confusion about agile stems from a lean mindset that views everything from the lens of the manufacturing process. In many companies, integrating NPD and the manufacturing process has indeed reaped rewards, and lean has been a part of that improvement. Agile, by contrast, is effective at NPD and R&D/innovation, and will probably never be applicable to manufacturing – imagine asking the NC machine to “spiral in on” the correct product each time! Surely, a compromise can be worked out in which lean resource allocation is applied to agile software development and lean manufacturing becomes much more nimble at producing prototypes. However, I regretfully conclude that in the real world, mixing lean and agile is likely to produce worse results than applying agile alone.

Consider the idea of applying just-in-time resource allocation to an agile process. In each iteration, the agile process is going to change the specs during the “sprint.” Thus, half the time, allocation of resources will be inadequate, causing more bottlenecks than a “wasteful” strategy.

More fundamentally, lean increases the need for tight control over and visibility into costs and resources. This kind of oversight, as developers well know, slows things down. As per my study, it also yields nothing in improved productivity, TCO, ROI, or customer satisfaction.

Above all, the mindset of lean and agile are antithetical. Lean is reactive, prescriptive, and cost-oriented, despite its attempts to connect to the customer. It works where requirements change little from iteration to iteration, as in a manufacturing process. Agile is proactive, creative, and customer-value-oriented. It works where customer needs are constantly changing – which is a much better description of the real world of customers, as opposed to the artificial “turn off the spigot until we’ve finished” world of much of today’s NPD.

In the end, however, the conflict of lean and agile is only one instance of the difficulties companies have with processes like agile when the command-and-control approach has become entrenched. As I noted in a recent post, it may well be caused by budgeting processes that constrain business flexibility. But whatever the reason, there is strong cultural resistance to increasing corporate results long-term by adopting processes that increase business agility, such as agile development.

In an old story whose author I cannot unfortunately remember, a manager is called in to turn the efforts of R&D types into profit. The first thing he notices was that they seem to spend a lot of time with their feet on their desks, thinking – a highly wasteful practice. How can he change that? He puts his legs up on his desk and leans back, thinking … In this story, as in the real world, lean is more the enemy than the partner of agile; and trying to constrain agile via lean will cost the company much more in the long run than it seems to gain according to short-term cost measures.

About The Genius of Tolkien

A great deal of ink has been spilled (and storage space wasted) in analyzing the writing of Tolkien as comparable to Great Authors, as strictly a writer in the fantasy genre, or as an odd specimen with no relation to other literary streams. Commentators have focused on the sources of his work in myth, on his role as the precursor of the quest type of fantasy, and on the relation between his life experiences and the Lord of the Rings, as well as the enormous “fantasy world” of which it is presented as a part.

I find that I came to Tolkien originally from a somewhat different background. My reading in childhood was as much English “adventure as escape” – John Buchan, H. Rider Haggard, Sanders of the River, Swallows and Amazons, G.A. Henty – and English novel/play – Shakespeare, Hardy – as standard American kid’s stuff – all the Oz books, John Tunis’ sports books. The folk songs I and my Dad sang were as much English – There Was a Tailor Had a Mouse – as Appalachian – John Langstaff’s version of Hi Falero. I am not sure why English commentators have not brought up some of the things that strike me about Tolkien’s writing; but I am pretty sure why American ones without the same background tend not to see them.

So, off the top of my head, here is a brief discussion of points I have not seen covered in the most visible analyses of Tolkien.

1. Poetry as folk song

It is quite striking how certain sets of poetry match up to the rhythm of English folk songs. Take that short one from Bilbo arriving at Rivendell in The Hobbit: “tra la la la lally, here down in the valley, with Bilbo and …” Now try singing it to the tune of “The Flowers that Bloom in the Spring”, itself a folk song imitation, in Gilbert and Sullivan’s Mikado. In both cases, the music itself suddenly gives a more sober, more realistic “tone” to what on the printed page seems silly nonsense. Singing the poetry this way gives an insight into the Elves as not the mischievous sprites of Shakespeare but rather superficially vacuously cheerful beings with hidden depths and age.

The part of Tolkien that most find hard to swallow is Tom Bombadil. I feel that the commentators find him mannered, irrelevant, and wasting page on page when the action should be advancing. However, if one more or less matches the words to the tune of, say, Among the Leaves So Green Oh, then it becomes a “proto-folk-song”. Tom becomes the underlying source of English folk songs and their spillover into the Shakespearean plays – “with a hey ho, the wind and the rain”. Tom’s every phrase is a blank verse with a folk-song-like meter, and his appearance transitions the hobbits from “modern rustic” speech and colloquial writing to “heroic” speech and “pre-modern” writing.

That, in turn, leads to:

2. Tolkien’s “pre-modern” English

One commentator has neatly noted Tolkien’s deliberate contrast between “modern” and “ancient” speech, pointing out the advantages of the latter: “What a lot of things you use good day for, said Gandalf”. No one seems to have noted the large expansion of the English language to accommodate older word meanings with wonderful resonance. “The words were not the doom of Minas Tirith. But doom and great deeds are indeed at hand.” “Into most places some haggard of spring would creep.” “Ithilien in spring still kept its disheveled dryad loveliness.” “Ent moot”.

At the same time, Tolkien deliberately strips his prose of both fancy modern-poetry-like flourishes and “just the facts” Hemingway style. Rather, the style is of simple metaphors “with a different eye”. “High above, an eagle might have seen the hobbits, small figures …” “Tom stood on the threshold, and a golden glow was behind him”

There is a transition from “modern” to “heroic”, and it happens so gradually and carefully that commentators seem to have missed it. The introduction is almost comically academic; the first few chapters, in the Shire, are almost entirely “modern”, except for Gandalf, who presents information to Frodo in “adult modern” terms with subtle shadings of “pre-modern”. “Who is he? Frodo said. I don’t think I’ve heard of him. He is high among our order [Gandalf], … subtle …”. Then a brief meeting with the Elves, with “pre-modern” presented as a foreign language that Frodo speaks with difficulty. Then Tom Bombadil as a kind of half-way language that seems rustic but is stripped of colloquialisms: “Tom he is the Master; his songs are stronger songs, and his boots are faster”. In Bree, Aragorn gives the first touch of extended heroic speech, beginning by presenting himself as almost hobbit-like: “Though I say it as shouldn’t, you might say …” but ending in “The day is coming when it shall be forged anew”. From there until the end of Book I, hobbit speech is balanced by Aragorn’s “pre-modern speech”. Then in the beginning of Book 2, even Bilbo gives a “pre-modern” poem, and “modern” – represented by the hobbits – is always the exception rather than the rule, all the way to the very end, when Frodo (and we) find ourselves viewing hobbit speech as more strange than the companions’ new heroic speech.

3. Viewpoint

Tolkien’s “reader’s viewpoint” is actually quite subtle, and much more varied than one realizes. However, there are overall “main viewers”, and their viewpoint carefully balances moral issues and a necessary distance that will allow the reader to think for him/herself.

Book I is, for the most part, told from Frodo’s viewpoint. There are interesting exceptions: “hmm, the fox thought;” “Sam ran back along the path, shouting help!” The dreams in Tom’s house are from people who will all have their own “main viewpoint” later. In Book II, again, Frodo is the “main viewpoint”, with Pippin and Sam occasionally allowed a quick “snapshot”. Then in Book III, we start with Aragorn, but Gimli is occasionally given the viewpoint; and Pippin and Merry are effectively the main viewpoint in their scenes.

Book IV begins a transition of great importance. At the start, the viewpoint is ambiguous between Frodo and Sam, but with Frodo very much more frequent. By the end of Book IV, Sam predominates. In effect, the transition reflects the transition from Frodo as the prime actor and decider, to Sam as the prime actor when the burden becomes too great for Frodo.

Book V, again, starts with Pippin the main viewpoint. Interestingly, when the action shifts to Gimli, Legolas, and Aragorn, Gimli is the prime viewpoint, keeping “heroic thinking” from getting too far out of hand. Only at the end does some of the viewpoint return to Aragorn, as he directs the attack on Mordor.

Then there’s Book VI. Now, the viewpoint is almost entirely Sam. The main exception is Faramir and Eowyn; almost a “heroic” counterpoint to the relentlessly “hobbit/half-modern” viewpoint. And by the end, the viewpoint is deliberately, consciously, entirely Sam.

The point is that the viewpoint is almost never that of the author, nor solely that of one main character driving the action forward. Look where you will, to Hemingway, Hardy, Bellow, Updike, or Flaubert, and the viewpoint is almost always that of the main character or of the author; not so with Tolkien. And the result is to gain perspective on the main character and to draw the reader more deeply into the world.

4. Weather and fauna

Commentators from fantasy writers to critics have confessed their feeling that they would like to live in or experience Tolkien’s locations. But few have noted the way in which the weather – and the places the characters go – is strongly related to the characters’ (and our) moods. Tolkien himself noted that, unusual among authors, he took great care to reflect an actual sequence of weather, just about day by day, from September to March. Moreover, the action takes place just about day by day, with only breaks for Rivendell, for part of the march south, for a short space after the knife in the dark, and for the time within the Mines of Moria. In just about every other day until the Ring is cast into the fire, there is a description of the weather for that day – and the character with the main viewpoint spends a lot of that day outdoors. The weather is always varied, always in some way notable, always in some ways memorable. And it both reflects and causes the characters’ moods; the tenseness of the descent of the Emyn Muil reflected by the thunderstorm, which itself almost causes a disastrous fall; the “heavier air” as the Riders go to war, that both reflects and increases the characters’ tension. In Tolkien’s world, unlike our own, characters feel the weather – “the air seemed to throb”.

As far as I can tell, there is no comparable achievement in all of literature. I simply cannot think of another novel in which there is such a clear and integral description of the weather.

Likewise, there is a clear link between the vegetation and the character of the people inhabiting it. I don’t mean just the fact that pastoral Riders need rolling hills and grasslands; I mean that the Shire is very regulated, with gardens and forest boundaries, while Lothlorien is regulated in a more subtle way, with long-term planning, and Ithilien was once a kind of hothouse of a wide variety of plants, shrubs, and trees, but has clearly been unregulated for some time.

Again, Tolkien is, as far as I can see, unprecedented in his knowledge of botany, and his ability to use it to set a scene and indicate a kind of people. The initial description of Ithilien is the outstanding example; the description of the desolation before the Gates of Mordor is a miniature gem of the opposite kind. But it is interlaced all the way through the novel: from the start, with the description of the Party Tree and the path to Crickhollow, to the swaying leaves above Sam’s head as he awakes in Ithilien, every tree and fern is a part of the main viewpoint.

5. The Ending

I have seen some commentators touch on this, but it should be emphasized: Tolkien crafts neither a tragic (in the classic sense) nor a happy (for the main character, Frodo) ending. This appears to spring out of a constant wrestling on Tolkien’s part with how God justifies death. Through twists and turns in his background notes published by Christopher, it appears that he conceives of Elves as the first answer to Death; they accept it because they are effectively “burned out”, passive shells of their passionate former selves. But Man is the second and better answer to Death, because he is forced into Death prematurely, with no assurance of what happens after, but by conquering his fears he achieves preparation for the best “next” existence, an existence in which may he become, as it were, a “child” of God to be taught by God.

Frodo’s realization that he cannot escape the effects of the Ring and that he has outgrown the Shire for which he thought he was killing himself becomes a tragedy that can’t be solved in Middle Earth; but it allows him to conquer his fears after he leaves for the Blessed Lands, by seeing there is hope ahead. Sam’s happy ending is tainted by the realization that the person for whom he sacrificed himself did not benefit by the sacrifice; but it allows him to live a life that will enable him also to conquer his fear of death. In either case, we are far from “happy ever after”, and just as far from “bad things happen to bad/flawed people, good things happen to good people” tragedy or even “bad things happen for no reason” tragedy.

6. Names

While commentators have noted Tolkien’s creation of a “new language”, his background as a philologist, and the origins in European history of the names of people, they do not seem to have noted the importance of his ability, to some extent, to avoid having the names seem ridiculous. Before Tolkien, fantasists tried to make up alien names as a baroque variation of English ones (e.g., Eddison), or as pagan-sounding (Dejah Thoris in Edgar Rice Burroughs’ Mars series). In either case, the names had no “depth” of source in a complete language. One of the reasons many readers recoiled from fantasy as serious adult literature was that, well, the people’s names sounded silly.

Again, the notes published by Christopher Tolkien show that in his original conception, J.R.R.’s names would have struck us as pretty silly, too – Kor, Gnomes for Elves. The process of developing a full-fledged language with clear associations to “normal” names, to our ear, apparently involved a fair bit of subtle changes to make the names less likely to cause laughter – Strider for Trotter, for example. The image of Aragorn as a pig would have been hard to expunge.

Not that, to some ears, the effort was successful. Reading between the lines, one of the main reasons that Edmund Wilson wrote “Ooh, Those Awful Orcs” was that to his ears, names like Shagrat sounded just as silly and juvenile as Burroughs’. But I think it’s fair to say that there’s a dividing line between fantasy before Tolkien, in which Jhirel of Joiry joined Fritz Leibner’s Fafnr, and after Tolkien, with the far more carefully crafted Welsh names of Morgon and Astrin in McKillip’s works, or Pyanfar and Arafel in Cherryh. Fantasy and science fiction not only learned world-building and history-building from Tolkien; to some extent, they learned language-building.

7. The Sad Lack of Successors

I can’t say that I find authors who have some sense of most of the preceding Tolkien traits, much less the ability to use them in other contexts, in most non-fantasy/sf writing. Fantasy and science fiction authors, in particular, have been quite successful at building much more detailed worlds that give a much greater sense of “depth” to the story, from Weber to Modesitt. Likewise, authors pay much more attention to the “tone” of the story, and do much better (by and large) at avoiding silliness and making heroic speech seem less alien. But I can find none, in fantasy or out, that has a comparable use of viewpoint, of weather and fauna, of simple metaphor for description, of endings that are neither comedy nor tragedy, or even (in most cases) of poetry as folk song.

Consider three who seem to have made a conscious effort to think deeply about Tolkien: Donaldson, Jordan, and Le Guin. Donaldson is of interest because he took many of the half-sketched ideas of Tolkien – land with a physical sense of health or disease, telepathic horses as equals to humans, immortal evil, and inevitable failure that can only be redeemed by God/chance – and fleshed them out in his own way. But the sense of language and viewpoint is sometimes really off – it is hard to imagine a juxtaposition of three more disparate eras of the English language than describing a character as “fey, anile, and gutless”, or to swallow a sentence that states “her breasts traced opaque circles of invitation beneath her shirt”. I know, these are early efforts; but the names established there (Lord Foul) still recur in his work thirty years later. Some of his “modern” writing (a graduation speech) is far more powerful. Viewpoint is far less subtle, poetry is less informed by folk song or history, and the weather appears to exist often so as to put as much physical stress as possible on the main character.

Jordan, by contrast, seems to have focused on the “quest” and world-building aspects of Tolkien. His gift for detail, however, is more about character and less about weather and fauna. Unfortunately, in his later books, this causes an over-proliferation of characters that make the story harder to follow. His poetry as folk song is actually quite good, but he does not really give the kind of day-to-day sense of weather and fauna as characters in the story, or at least as indicators of mood, that Tolkien can achieve. His approach to viewpoint is less subtle, and his endings are a bit more traditional.

Of these three, le Guin is perhaps the most successful at building on Tolkien’s additions to the writing craft. Her Earthsea books, apparently beginning with a light attempt to imagine a hobbit as a proto-dragon, have a strong sense of the weather, if not of fauna, and a simple style that, while light on metaphor, nevertheless fully understands the power of pre-modern word meanings. Her viewpoints are mostly traditional but show interesting variation, especially inserting women’s viewpoints into what is traditionally a man’s narrative. Names show great “depth”, even if the world-building is a bit “retrofitted” as her concepts expand from story to story. Ultimately, however, more than Jordan or Donaldson, her stories are less day-by-day and more episode-by-episode, which makes the feel of her stories more “abstract” overall.

A French critic is reported to have said “Victor Hugo is our greatest author – alas”, meaning that even Hugo did not compare to the English or American great writers. In the same vein, I feel that authors of whatever stripe have yet to fully appreciate or build on the aspects of Tolkien’s stylistic genius that I have described above; and it’s too bad.

Prediction and the Good Analyst: IBM Rational

I have been unable to find a clear source for this quote, but some author in the 1970s or before defined science fiction as exploring any of three premises about the future – “ (1) What if? (2) If only … and (3) If this goes on …” Rearranging the premises, slightly, what we have is a good description of the role of prediction in analysis, especially for a good analyst:

1. “If this goes on …” is extrapolation. It uses a model based on historical data, and predicts by assuming that things will continue to operate according to the model. For the analyst, this means looking at a vendor’s product release and figuring out how people will react to it, based on how they have reacted in the past.

2. “What if?” is stress testing. It introduces new events or players not seen before, but which have some reasonable chance of happening, and sees what happens within the model. For the analyst, this means asking whether new products from other vendors, or new trends, will positively or negatively affect the product.

3. “If only” is normation[1]. That is, it seeks to identify changes that will make for a better outcome for some or all people affected by this future. For the analyst, this means identifying ways that the product could be made better in the future, or ways in which another combination of products can provide better results.

I am reminded of this way of looking at the analyst’s job by recent news regarding IBM Rational. At IBM’s Software Connect, the group announced initiatives including “completing the software lifecycle” by linking development and operational information about software, improving software delivery predictability by applying risk management and analytic techniques, and new ways of integrating IBM’s rapidly proliferating array of software. This was complemented by a Dec. 16 press release that noted change management, collaboration, embedded-system development, SOA-based development, and requirements management improvements to existing Rational products. All this, as Steve Mills noted in his Software Connect keynote speech, is in service to IBM’s commitment to creation of “smarter” products – with instrumentation, automation, and reporting.

To understand my reaction to this news, you should walk back with me over a decade and a half of applying extrapolation, stress testing, and normation to Rational, and then IBM Rational, software development offerings. If you remember, Rational’s original claim to fame was as the most effective purveyor of object-oriented design tools. This focus on development of sophisticated and formalized control tools for large-scale and complex development projects continued until as recently as three years ago.

However, during all that time, there was an alternative – first embodied in the 4GL offerings of the mid 1990s, and then in the agile development community of the early 2000s – that stressed flexible, iterative, solo or loosely collaborating, open-ended development. My recent research suggests that agile development has a paradoxically positive effect on project risks, development times and costs, and organization ROI and agility – and except in very complex projects such as design of the latest Boeing plane, the Rational approach (as it was then) does not. So extrapolation suggested that Rational tools would serve an ever smaller percent of the development market; stress testing suggested in the mid-2000s that adoption of agile development techniques was beginning in offshore development firms as well as small shops in the US; and normation suggested that IBM could leverage the efforts of agile development methodologies such as SCRUM as well as the rapid prototyping features of non-Rational software such as WebSphere Portal to quickly implement a strong agile-development product and focus. Still, as late as 2007, there was no clear indication that IBM, via Rational, would change its focus from “formal” to “agile”.

That was then; this is now. Over the last three years, IBM Rational has clearly made a major transition to focusing on agile rather than just “high-end complex” development, and its announcements represent the culmination of that effort. IBM Jazz, the in-house-developed product for agile mashups, receives top billing; older Rational products have been retrofitted to integrate with agile development tools and support a “spiral”/iterative rather than a “waterfall” (i.e., no going back to a previous development stage) model; and there is a welcome focus on constant incremental upgrades to software via infusion of runtime “operational” data on a piece of software in the development process. What’s not to like?

Ah, but that was then, this is now: it’s time to apply extrapolation, stress testing, and normation to the new Rational. Extrapolation says that the agile focus will make IBM of much more use to a wider array of developers. Stress testing says that while IBM’s competitors at all levels may have some advantages in having “built from the ground up” agile development tools, IBM’s ability to scale development to the high end offers advantages to larger customers. However, stress testing and normation suggest that the new Rational has one glaring hole: lack of tools to handle the “instrumented world”, as IBM calls it, or the “sensor-driven Web”, as I call it.

New types of data – including RFID, GPS, and cell phone photos and videos – are being passed over the Web, and within the enterprise, in massive amounts. The architecture to handle these appears to be evolving as the “event-driven architecture”, a filter on top of an Enterprise Service Bus, in which a “one-size-fits-all” program filters events passing over the bus, aggregates them and places them in context, and then either takes action itself according to pre-defined business rules, or sends notifications to a human that action may be needed. The software for these programs is fundamentally different from most past programming models, because it requires “near-real-time” response, and because the structure is trigger/response rather than input-process-output (old-style programs) or loosely-coupled communication (today’s object-oriented programs). If a development vendor fails to provide competitive tools to handle this new model over the next three years, and if other vendors do, the laggard will miss out on a major part of tomorrow’s development market – just as IBM ceded a large segment of the of the medium-sized, medium-complexity “sweet spot” in the development market to others from 1995-2006.

The good news for IBM is that, its competitors Oracle, Microsoft, and HP also appear to be quite some distance from creating a full-fledged sensor-driven Web development tool – and at least IBM recognizes the opportunity. It also appears that moving in the direction of such a tool will not be such a stretch for IBM, given their existing “stream processing” tools. However, because of IBM’s recent history of “missing an opportunity and then buying a third party” in areas such as global repositories, I won’t count IBM Rational as a leading-edge sensor-driven Web development tool until I see some quality productized code.

So, two cheers for the agile IBM Rational heading into 2010; and an echoing silence where the third cheer ought to be. To figure out where that third cheer went, they might apply a little extrapolation, stress testing, and normation; or they could ask a good analyst …

---

[1] A made-up word. It means the process of developing normative recommendations from data.

Nagging Questions About Investment Theory

At the recent IBM STG (Systems and Technologies) Group analyst briefing, I heard Helene Armitage assert that a primary cause of the failure of financial-services models to detect the problems with the mortgage-backed securities that were the proximate cause of our economic-system seize-up was “firms’ failure to examine the models’ assumptions adequately.” I think this is flat wrong.

I think that the root cause of analysis failure was the mind-set behind the models themselves. I believe that analysts and firms failed to challenge the models because of a mindset that said – and continues to say – “a security is a security is a security.” More specifically, I think that firms look at investments with the notion that all investments are fundamentally alike, and can be represented in terms of return and risk, strictly defined. But as, over the last thirty years, this notion has been steadily extended from stocks to bonds, commodities, private equity, derivatives, and mortgages, no one has exhaustively examined the ways in which, beneath the surface of alpha and beta, risk and return for these investment types differ from those of stocks.

What follows is a couple of things that have bothered me about this mindset as I look at capital market pricing theory (CAPM) and the like. This is not an argument about whether CAPM is accurate – my impression is that, accurate or not, it shares these “bothersome” assumptions with other alternative proposed theories. It simply provides a well-defined way to examine these assumptions.

The first thing that bothers me is the notion of diversification. Let’s suppose I have two equally weighted stocks with exact -1 correlation (or an index and costless shorting of that index). If I understand the theory, this is the ultimate in diversification: no matter what, I earn zero. There is no unsystematic risk; in fact (conceptually), there is no systematic risk. I never earn money; the only way I lose money is if the United States collapses.

Now suppose I invest the same amount of money in the “risk-free investment” (U.S. bond, let’s say). This contains the systematic risk I’ve eliminated; also, I always (conceptually) earn at least as much money as in index-and-short, and (practically) always earn more money than index-and-short. In other words, despite the fact that I am more fully diversified, there is no justification for me ever to do index-and-short. I have not reduced the “real” risk of losing money; in fact, after inflation, I have increased it. I have not increased my return, compared to the risk-free asset; I have reduced it. So diversification, for certain types of investments, appears to be worse than useless.

Now apply this same notion to commodities – that is, not ownership of companies producing commodities, but ownership of the commodities themselves. Economics suggests that over the long run, due to technology applied to the commodity or its substitutes, prices fall. Either the commodity is seen as a store of value (gold as money) or not. If it is a store of value, over the long run it will track inflation (like the risk-free investment). If it is not, over the long run it will do worse than inflation. So, by this logic, commodities behave much like index-and-short: in the long run, they always do as badly as, or worse than, the risk-free investment.

So why, in a recent study of the best investments to preserve an annuity after retirement, is there a recommendation to invest in commodities for “diversification”? Well, it turns out that there was a particular point at which commodities performed better than the risk-free asset or anything else, conceptually allowing 5% yearly withdrawals rather than 4%. Translation: 1% more if I assume that I have to keep my “principal” at 100% or more of its initial value. But if I know that on average I will earn 1% more from the risk-free asset than from commodities, then it really doesn’t matter if my principal (when I substitute the risk-free asset for commodities) goes down in one of those years, and stays down; I can still withdraw the same dollar amount as if commodities were in the mix – in effect, I will get a 1% additional return on lower principal, and I can start withdrawing a greater percentage of the entire investment. I realize that this is very crude math, but you get the point.

So, to summarize, my first problem with “a security is a security” is that for certain types of investments, both risk and return are almost always negative compared to the risk-free investment, and therefore diversification does not seem to work.

My second problem is with the notion that risk – defined as variance – is effectively stable over time, for a particular asset type. Take the example of mortgages. It is very easy to think that mortgage failure rates are a creature of the business cycle alone, just like company or bond default rates. In that case, mortgage-backed securities will behave just like other securities, with a large body of historical data allowing a reasonable approximation of failure rates, and well-established procedures for handling them. Because actuaries have long experience with establishing the “risks” of individual mortgages, it seems a natural assumption that the actuarial “risk of failure” of an individual mortgage translates straightforwardly into the variance-type “security is a security” risk of a mortgage-backed security.

I would suggest that a mortgage differs from the usual investment in two key ways: it is more “sticky”, and there is a power imbalance between borrower and lender. It is more “sticky” because, even when you have adjustable rates, changing the terms of the contract is a matter of changing millions of individual policies, so that the outcome is more typically complete failure rather than part payment. There is a greater imbalance in favor of the lender than in the typical investment, because the bank or other lender has economies of scale compared to the individual borrower. The result of both differences, I would argue, is that there are incentives in the system to increase the likelihood of failure over time, across multiple business cycles, and therefore past data tends to underestimate the rates and effects of failures and the increased failure-type risk of the next business-cycle downturn or excess investment.

I think of it this way: suppose, to finance a house, an individual sold a $250,000, 5% coupon, sinking-fund bond. The security would be the bond itself. If interest rates/inflation changed, the price of the bond would change. If credit risk increased, the price of the bond would decrease. The investor who bought the bond would assume this risk, and the bond issuer – the borrower – would have a market to handle increases in credit risk normal to the business cycle. Does anyone really think that this describes the way mortgages work? And if not, why would you think that devices appropriate for bonds, like stripping, matching, and hedging, would be appropriate for mortgages? But because people did, risks were underestimated in formulating derivatives, demand for securities drove demand for mortgages, and unnecessary failures with little yield to the investor drove banks and hedge funds to their knees, with horrendous spill-over effects.

To summarize: sorry, Helene, I think that the real cause of financial-firm model failure is not a failure to examine assumptions, but a failure in the mindset that produced the models: the assumption that “a security is a security.” I question the mindset that thinks diversification handles risk for every investment type, and I question the mindset that thinks that variance-type risk is stable across business cycles for every investment type. I am not the right person to suggest a better mindset. But I sure hope someone does, instead of accepting the half-solutions that seem to me to be implicit not just in Helene’s critique, but those of most commentators.

Parenthetical note: it seems to me that we are seeing two types of “extreme” risks in today’s world, and only two. There is the risk of country collapse, in which case it doesn’t matter if you invested your money in your mattress or in gold or in a risk-free investment or whatever; it’s all worthless anyway. Then there’s the bank-seize-up/Great Depression type of risk, which we have seen at least twice in the last 100 years. While the jury is still out, it appears that this doesn’t distort the relative risk/return of stocks, bonds, and the risk-free asset over periods of 15-20 years. So why, as investors and investment theorists, are we living in fear that it will?

Dave Hill, Data Protection, and the Future of IT

Full disclosure: Dave is a friend, and a long-time colleague. He has just written an excellent book on Data Protection; hence the following musings.

As I was reading (a rapid first scan), I tried to pin down why I liked the book so much. It certainly wasn’t the editing, since I helped with that. The topic is reasonably well covered, albeit piecemeal, by vendors, industry associations, and bloggers. And while I have always enjoyed Dave’s stories and jokes, the topic does not lend itself to elaborate stylistic flourishes.

After thinking about it some more, I came to the conclusion that it’s Dave’s methodology that I value. Imho, Dave in each chapter will lay out a comprehensive and innovative classification of the topic at hand – data governance, information lifecycle management, data security – and then use that classification to bring new insight into a well-covered topic. The reason I like this approach is that it allows you to use the classification as a springboard, to come to your own conclusions, to extend the classification and apply it in other areas. In short, I found myself continually translating classifications from the narrow world of storage to the broad world of “information”, and being enlightened thereby.

One area in particular that called forth this type of analysis was the topic of cloud computing and storage. If data protection, more or less, involves considerations of compliance, operational/disaster recovery, and security, how do these translate to a cloud external to the enterprise? And what is the role of IT in data protection when both physical and logical information are now outside of IT’s direct control?

But this is merely a small part of the overall question of the future of IT, if external clouds take over large chunks of enterprise software/hardware. If the cloud can do it all cheaper, because of economies of scale, what justification is there for IT to exist any longer? Or will IT become “meta-IT”, applying enterprise-specific risk management, data protection, compliance, and security to their own logical part of a remote, multi-tenant physical infrastructure?

I would suggest another way of slicing things. It is reasonable to think of a business, and hence underlying IT, as cost centers, which benefit from commodity solutions provided externally, and competitive-advantage or profit centers, for which being like everything else is actually counter-productive. In an ideal world, where the cloud can always underprice commodity hardware and software, IT’s value-add lies where things are not yet commodities. In other words, in the long run, IT should be the “cache”, the leading edge, the driver of the computing side of competitive advantage.

What does that mean, practically? It means that the weight of IT should shift much more towards software and product development and initial use. IT’s product-related and innovative-process-related software and the systems to test and deploy them are IT’s purview; the rest should be in the cloud. But this does not make IT less important; on the contrary, it makes IT more important, because not only does IT focus on competitive advantage when things are going well, it also focuses on agile solutions that pay off in cost savings by more rapid adaptation when things are going poorly. JIT inventory management is a competitive advantage when orders are rising; but also a cost saver when orders are falling.

I realize that this future is not likely to arrive any time soon. The problem is that in today’s IT, maintenance costs crowd out new-software spending, so that the CEO is convinced that IT is not competent to handle software development. But let’s face it, no one else is, either. Anyone following NPD (new product development) over the last few years realizes that software is an increasing component in an increasing number of industries. Outsourcing competitive-advantage software development is therefore increasingly like outsourcing R&D – it simply doesn’t work unless the key overall direction is in-house. Whether or not IT does infrastructure governance in the long run, it is necessarily the best candidate to do NPD software-development governance.

So I do believe that IT has a future; but quite a different one from its present. As you can see, I have wandered far afield from Data Protection, thanks to Dave Hill’s thought-provoking book. The savvy reader of this tome will, I have no doubt, be able to come up with other, equally fascinating thoughts.

Speed vs. Agility

A recent product announcement by IBM and a series of excellent (or at least interesting) articles in Sloan Management Review have set me to musing on one unexamined assumption in most assessments: that increased process speed equals increased business agility. My initial take: this is true in most cases, but not in all, and can be misleading as a cookie-cutter strategy.

The IBM announcement centered around integration of their business-process management (BPM) capabilities, in order to achieve agility by speeding up business processes. What was notably missing was integration with IBM’s capabilities for New Product Development (NPD) – Rational and the like. However, my initial definition and application of KAIs (key agility indicators) at a business level suggests that speeding up NPD, including development of new business processes, has far more of an impact on long-term business agility than speeding up existing processes. To put it another way, increasing the Titanic’s ability to turn sharply is far more likely to avert disaster than increasing its top speed charging straight ahead – in fact, increasing its speed makes it more likely to crash into an iceberg.

A similar assumption seems to have been made in SMR’s latest issue, in the article entitled “Which Innovation Efforts Will Pay?” The message of this article appears to be that improving innovation efforts is primarily a matter of focusing more on the “healthy innovation” middle region of internally-developed modest “base hits”, with little or no effect from speeding up internal innovation processes or expanding them to include outside innovation. By contrast, the article “Does IP Strategy Have to Cripple Open Innovation?” suggests that collaborative strategies across organizational lines focused on NPD make users far more agile and businesses far better off, despite requiring as much (or more) time to implement as in-house efforts. And finally, we might cite a study in SMR suggesting that users estimating inventory-refill needs were more likely to make sub-optimal decisions when fed data daily than when fed a weekly summary, or the recent book on system dynamics by Donella Meadows that argued that increasing the speed of a process was often accomplished by increasing its rigidity (constraining the process in order to optimize the typical case right now), which made future disasters, as the system inevitably grows, less avoidable and more life-threatening.

All of this suggests that (a) people are assuming that increased process speed automatically translates to increased business agility, and (b) on the contrary, in many cases it translates to insignificant improvements or significant decreases in agility. But how do we tell when speed equals agility, and when not? What rules of thumb are there to tell us when increased speed does not positively impact business agility, and, in those cases, what should we do?

I don’t pretend to have the final answers for these questions. But I do have some initial thoughts on typical situations that lead to increased speed but decreased agility, and on how to assess and improve business investment strategies in those cases.

If it isn’t sustainable it isn’t agile. Let us suppose that we improve a business process by applying technology that speeds the process by decreasing the need for human resources. Further, suppose that the technology involves increased carbon or energy use – a 1/1 replacement of people-hours by computing power, say. Over the long run, this increased energy use will need to be dealt with, adding costs for future business-process redesign and decreasing the money available for future innovation. The obvious rejoinder is that cost savings will fund those future costs; except that today, most organizations are still digging themselves deeper into an energy hole, while operational IT costs, driven by an increased need for storage, continue to exert upward pressure and crowd out new-app development.

As the most recent SMR issue notes, the way to handle this problem is to build sustainability into every business process. If lack of sustainability decreases agility, then the converse is also true: building sustainability into the company, including both ongoing processes and NPD, increases revenues, decreases costs – and increases agility.

If it’s less open it’s less agile. In some ways, this is a tautology: if an organization changes a business process so as to preclude some key inputs in the name of speed, it will be less successful in identifying problems that call for adaptation. However, it does get at one of the subtler causes of organizational rigidity: the need to do something, anything, quickly in order to survive. A new online banking feature may make check processing much more rapid, but if customers are not listened to adequately, it may be rejected in the marketplace, or cost the business market share.

Detecting and correcting this type of problem is hard, because organizational politics points everyone to the lesson only after the process has already been implemented and has gained a toehold – and because businesses may draw the wrong conclusion (e.g., it’s about better design, not about ensuring open collaboration). The best fix is probably a strong, consistent, from-the-top emphasis on collaboration, agile processes, and not shooting the messenger.

Those are the main ways I have seen in which increased speed can actually make things worse. I want to add one more suggestion, which affects not so much situations where speed has negative consequences, but rather cases in which speed and agility can be improved more cost-effectively: upgrading the development process is better. That is, even if you are redesigning an existing business process like disaster recovery for better speed, you get a better long-term bang for your buck by also improving the agility of the process by which you create and implement the speedier solution. Not only does this have an immediate impact in making the solution itself more agile; it also bleeds over into the next project to improve a business process, or a product or service. And the best way I’ve found so far to improve development-process speed, quality, and effectiveness is an agile process.

Open Source, Windows, Portability, and Borges

There is a wonderful short story by Jorge Luis Borges ("Pierre Menard, Author of the Quixote") that, I believe, captures the open source effort to come to terms with Windows – which in some quarters is viewed as the antithesis of the philosophy of open source. In this short story, a critic analyzes Don Quixote as written by someone four hundred years later – someone who has attempted to live his life so as to be able to write the exact same words as in the original Don Quixote. The critic’s point is that even though the author is using the same words, today they mean something completely different.

In much the same way, open source has attempted to mimic Windows on “Unix-like” environments (various flavors of Unix and Linux) without triggering Microsoft’s protection of its prize operating system. To do this, they have set up efforts such as Wine and ReactOS (to provide the APIs of Windows from Win2K onwards) and Mono (to provide the .NET APIs). These efforts attempt to support the same APIs as Microsoft’s, but with no knowledge of how Microsoft created them. This is not really reverse engineering, as the aim of reverse engineering is usually to figure out how functionality was achieved. These efforts don’t care how the functionality was achieved – they just want to provide the same collection of words (the APIs and functionality).

But while the APIs are the same, the meaning of the effort has changed in the twenty-odd years since people began asking how to make moving programs from Wintel to another platform (and vice versa) as easy as possible. Then, every platform had difficulties with porting, migration, and source or binary compatibility. Now, Wintel and the mainframe, among the primary installed bases, are the platforms that are most difficult to move to or from. Moreover, the Web, or any network, as a distinct platform did not exist; today, the Web is increasingly a place in which every app and most middleware must find a way to run. So imitating Windows is no longer so much about moving Windows applications to cheaper or better platforms; it is about reducing the main remaining barrier to being able to move any app or software from any platform to any other, and into “clouds” that may hide the underlying hardware, but will still suffer when apps are platform-specific.

Now, “moving” apps and “easy” are very vague terms. My own hierarchy of ease of movement from place to place begins with real-time portability. That is, a “virtual machine” on any platform can run the app, without significant effects on app performance, robustness, and usability (i.e., the user interface allows you to do the same things). Real-time portability means the best performance for the app via load balancing and dynamic repartitioning. Java apps are pretty much there today. However, apps in other programming languages are not so lucky, nor are legacy apps.

The next step down from real-time portability is binary compatibility. The app may not work very well when moved in real time from one platform to another, but it will work, without needing changes or recompilation. That’s why forward and backward compatibility matter: they allow the same app to work on earlier or later versions of a platform. As time goes on, binary compatibility gets closer and closer to real-time portability, as platforms adapt to be able to handle similar workloads. Windows Server may not scale as well as the mainframe, but they both can handle the large majority of Unix-like workloads. It is surprising how few platforms have full binary compatibility with all the other platforms; it isn’t just Windows to the mainframe but also compatibility between different versions of Unix and Linux. So we are a ways away from binary compatibility, as well.

The next step down is source-code compatibility. This means that in order to run on another platform, you can use the same source code, but it must be recompiled. In other words, source-code but not binary compatibility seems to rule out real-time movement of apps between platforms. However, it does allow applications to generate a version for each platform, and then interoperate/load balance between those versions; so we can crudely approximate real-time portability in the real world. Now we are talking about a large proportion of apps on Unix-like environments (although not all), but Windows and mainframe apps are typically not source-code compatible with the other two environments. Still, this explains why users can move Linux apps onto the mainframe with relative ease.

There’s yet another step down: partial compatibility. This seems to come in two flavors: higher-level compatibility (that is, source-code compatibility if the app is written to a higher-level middleware interface such as .NET) and “80-20” compatibility (that is, 80% of apps are source-code incompatible in only a few, easily modified places; the other 20% are the nasty problems). Together, these two cases comprise a large proportion of all apps; and it may be comforting to think that legacy apps will sunset themselves so that eventually higher-level compatibility will become de facto source-code compatibility. However, the remaining cases include many important Windows apps and most mission- and business-critical mainframe apps. To most large enterprises, partial compatibility is not an answer. And so we come to the final step down: pure incompatibility, only cured by a massive portation/rewrite effort that has become much easier but is still not feasible for most such legacy apps.

Why does all this matter? Because we are closer to Nirvana than we realize. If we can imitate enough of Windows on Linux, we can move most Windows apps to scale-up servers when needed (Unix/Linux or mainframe). So we will have achieved source-code compatibility from Windows to Linux, Java real-time portability from Linux to Windows, source-code compatibility for most Windows apps from Windows to Linux on the mainframe, and Linux source-code compatibility and Java real-time portability from Linux to the mainframe and back. It would be nice to have portability from z/OS apps to Linux and Windows platforms; but neither large enterprises nor cloud vendors really need this – the mainframe has that strong a TCO/ROI and energy-savings story for large-scale and numerous (say, more than 20 apps) situations.

So, in an irony that Borges might appreciate, open-source efforts may indeed allow lower costs and greater openness for Windows apps; but not because open source free software will crowd out Windows. Rather, a decent approximation of cross-platform portability with lower per-app costs will be achieved because these efforts allow users to leverage Windows apps on other platforms, where the old proprietary vendors could never figure out how to do it. The meaning of the effort may be different than it would have been 15 years ago; but the result will be far more valuable. Or, as Borges’ critic might say, the new meaning speaks far more to people today than the old. Sometimes, Don Quixote tilting at windmills is a useful thing.