Tuesday, January 19, 2010

Lean and Agile: Partners or Enemies?

An increasing number of commentators on both lean New Product Development (NPD) and agile software development have noted efforts to combine the two. A key reason for the interest in the combination is that lean has long been entrenched in the manufacturing process (e.g., via Kanban and ERP), while agile development has recently established itself as an effective software development strategy, and a major segment of NPD now involves software-only products, software-infused products, or software-plus-hardware solutions. A recent Aberdeen Group study notes that 66% of today’s NPD uses software to drive innovation, and the number is rising rapidly. Meanwhile, several efforts are being made to apply lean-only to software development (cf. Hibbs, Jewett, Sullivan, “The Art of Lean Software Development”).

At first glance, lean is a very good complement for agile. Users have concerns about agile’s ability to scale; lean makes sure that developer “resources” are available for each iterative step in a flexible and cost-effective way. Lean allows a different process time-line each time depending on what resources are available when; agile “spirals” in towards a solution, ensuring a different process timeline for each iteration of a solution “guesstimate”. Both aim to increase value to the customer; both emphasize incremental improvements.

However, a reading of the blogs on lean and agile betrays a fundamental misunderstanding of the agile mindset. Here’s a quote from Martin Heller (http://www.infoworld.com/d/developer-world/should-software-development-be-lean-or-agile-284): “They both strive to improve software quality, reduce waste, increase developer productivity, accept changes to requirements, and prize meeting the customer's real needs.” Agile doesn’t strive to improve software quality; it strives to improve software usefulness. It doesn’t aim to reduce waste; it aims to increase “wasteful” experimentation, which turns out actually to reduce the time-to-value, as a side-effect. Its way of increasing developer productivity – ensuring that development is more in sync with customer needs – is the opposite of lean’s attempt to improve developer productivity by eliminating bottlenecks (and customer interference is potentially a bottleneck). Lean accepts changes to requirements as a necessary part of incremental manufacturing process improvement; agile welcomes and emphasizes changes to requirements as part of improving product design. Lean views customer needs as consisting primarily of quality; agile views customer needs as consisting primarily of rapidly-changing functionality.

Moreover, we have already seen how an emphasis on Deming-like quality can retard product development to the detriment of a firm, as when Motorola lost its race to dominate processors to Intel by being late and customer-insensitive with the design of a “quality”-manufactured product. I can’t quote figures from my recent study under Aberdeen auspices of agile software development vs. other processes; but I can certify that a focus on software quality improvement was far less effective in improving product TCO, ROI, customer value, and ongoing agility than an agile process; in fact, there was no clear long-term benefit to ROI of quality-improving processes at all.

I wish I could say that this is a side-issue, and that once agile is properly understood, lean and agile are indeed complements. After all, some of the confusion about agile stems from a lean mindset that views everything from the lens of the manufacturing process. In many companies, integrating NPD and the manufacturing process has indeed reaped rewards, and lean has been a part of that improvement. Agile, by contrast, is effective at NPD and R&D/innovation, and will probably never be applicable to manufacturing – imagine asking the NC machine to “spiral in on” the correct product each time! Surely, a compromise can be worked out in which lean resource allocation is applied to agile software development and lean manufacturing becomes much more nimble at producing prototypes. However, I regretfully conclude that in the real world, mixing lean and agile is likely to produce worse results than applying agile alone.

Consider the idea of applying just-in-time resource allocation to an agile process. In each iteration, the agile process is going to change the specs during the “sprint.” Thus, half the time, allocation of resources will be inadequate, causing more bottlenecks than a “wasteful” strategy.

More fundamentally, lean increases the need for tight control over and visibility into costs and resources. This kind of oversight, as developers well know, slows things down. As per my study, it also yields nothing in improved productivity, TCO, ROI, or customer satisfaction.

Above all, the mindset of lean and agile are antithetical. Lean is reactive, prescriptive, and cost-oriented, despite its attempts to connect to the customer. It works where requirements change little from iteration to iteration, as in a manufacturing process. Agile is proactive, creative, and customer-value-oriented. It works where customer needs are constantly changing – which is a much better description of the real world of customers, as opposed to the artificial “turn off the spigot until we’ve finished” world of much of today’s NPD.

In the end, however, the conflict of lean and agile is only one instance of the difficulties companies have with processes like agile when the command-and-control approach has become entrenched. As I noted in a recent post, it may well be caused by budgeting processes that constrain business flexibility. But whatever the reason, there is strong cultural resistance to increasing corporate results long-term by adopting processes that increase business agility, such as agile development.

In an old story whose author I cannot unfortunately remember, a manager is called in to turn the efforts of R&D types into profit. The first thing he notices was that they seem to spend a lot of time with their feet on their desks, thinking – a highly wasteful practice. How can he change that? He puts his legs up on his desk and leans back, thinking … In this story, as in the real world, lean is more the enemy than the partner of agile; and trying to constrain agile via lean will cost the company much more in the long run than it seems to gain according to short-term cost measures.

Sunday, January 17, 2010

About The Genius of Tolkien

A great deal of ink has been spilled (and storage space wasted) in analyzing the writing of Tolkien as comparable to Great Authors, as strictly a writer in the fantasy genre, or as an odd specimen with no relation to other literary streams. Commentators have focused on the sources of his work in myth, on his role as the precursor of the quest type of fantasy, and on the relation between his life experiences and the Lord of the Rings, as well as the enormous “fantasy world” of which it is presented as a part.

I find that I came to Tolkien originally from a somewhat different background. My reading in childhood was as much English “adventure as escape” – John Buchan, H. Rider Haggard, Sanders of the River, Swallows and Amazons, G.A. Henty – and English novel/play – Shakespeare, Hardy – as standard American kid’s stuff – all the Oz books, John Tunis’ sports books. The folk songs I and my Dad sang were as much English – There Was a Tailor Had a Mouse – as Appalachian – John Langstaff’s version of Hi Falero. I am not sure why English commentators have not brought up some of the things that strike me about Tolkien’s writing; but I am pretty sure why American ones without the same background tend not to see them.

So, off the top of my head, here is a brief discussion of points I have not seen covered in the most visible analyses of Tolkien.

  1. Poetry as folk song

It is quite striking how certain sets of poetry match up to the rhythm of English folk songs. Take that short one from Bilbo arriving at Rivendell in The Hobbit: “tra la la la lally, here down in the valley, with Bilbo and …” Now try singing it to the tune of “The Flowers that Bloom in the Spring”, itself a folk song imitation, in Gilbert and Sullivan’s Mikado. In both cases, the music itself suddenly gives a more sober, more realistic “tone” to what on the printed page seems silly nonsense. Singing the poetry this way gives an insight into the Elves as not the mischievous sprites of Shakespeare but rather superficially vacuously cheerful beings with hidden depths and age.

The part of Tolkien that most find hard to swallow is Tom Bombadil. I feel that the commentators find him mannered, irrelevant, and wasting page on page when the action should be advancing. However, if one more or less matches the words to the tune of, say, Among the Leaves So Green Oh, then it becomes a “proto-folk-song”. Tom becomes the underlying source of English folk songs and their spillover into the Shakespearean plays – “with a hey ho, the wind and the rain”. Tom’s every phrase is a blank verse with a folk-song-like meter, and his appearance transitions the hobbits from “modern rustic” speech and colloquial writing to “heroic” speech and “pre-modern” writing.

That, in turn, leads to:

  1. Tolkien’s “pre-modern” English

One commentator has neatly noted Tolkien’s deliberate contrast between “modern” and “ancient” speech, pointing out the advantages of the latter: “What a lot of things you use good day for, said Gandalf”. No one seems to have noted the large expansion of the English language to accommodate older word meanings with wonderful resonance. “The words were not the doom of Minas Tirith. But doom and great deeds are indeed at hand.” “Into most places some haggard of spring would creep.” “Ithilien in spring still kept its disheveled dryad loveliness.” “Ent moot”.

At the same time, Tolkien deliberately strips his prose of both fancy modern-poetry-like flourishes and “just the facts” Hemingway style. Rather, the style is of simple metaphors “with a different eye”. “High above, an eagle might have seen the hobbits, small figures …” “Tom stood on the threshold, and a golden glow was behind him”

There is a transition from “modern” to “heroic”, and it happens so gradually and carefully that commentators seem to have missed it. The introduction is almost comically academic; the first few chapters, in the Shire, are almost entirely “modern”, except for Gandalf, who presents information to Frodo in “adult modern” terms with subtle shadings of “pre-modern”. “Who is he? Frodo said. I don’t think I’ve heard of him. He is high among our order [Gandalf], … subtle …”. Then a brief meeting with the Elves, with “pre-modern” presented as a foreign language that Frodo speaks with difficulty. Then Tom Bombadil as a kind of half-way language that seems rustic but is stripped of colloquialisms: “Tom he is the Master; his songs are stronger songs, and his boots are faster”. In Bree, Aragorn gives the first touch of extended heroic speech, beginning by presenting himself as almost hobbit-like: “Though I say it as shouldn’t, you might say …” but ending in “The day is coming when it shall be forged anew”. From there until the end of Book I, hobbit speech is balanced by Aragorn’s “pre-modern speech”. Then in the beginning of Book 2, even Bilbo gives a “pre-modern” poem, and “modern” – represented by the hobbits – is always the exception rather than the rule, all the way to the very end, when Frodo (and we) find ourselves viewing hobbit speech as more strange than the companions’ new heroic speech.

  1. Viewpoint

Tolkien’s “reader’s viewpoint” is actually quite subtle, and much more varied than one realizes. However, there are overall “main viewers”, and their viewpoint carefully balances moral issues and a necessary distance that will allow the reader to think for him/herself.

Book I is, for the most part, told from Frodo’s viewpoint. There are interesting exceptions: “hmm, the fox thought;” “Sam ran back along the path, shouting help!” The dreams in Tom’s house are from people who will all have their own “main viewpoint” later. In Book II, again, Frodo is the “main viewpoint”, with Pippin and Sam occasionally allowed a quick “snapshot”. Then in Book III, we start with Aragorn, but Gimli is occasionally given the viewpoint; and Pippin and Merry are effectively the main viewpoint in their scenes.

Book IV begins a transition of great importance. At the start, the viewpoint is ambiguous between Frodo and Sam, but with Frodo very much more frequent. By the end of Book IV, Sam predominates. In effect, the transition reflects the transition from Frodo as the prime actor and decider, to Sam as the prime actor when the burden becomes too great for Frodo.

Book V, again, starts with Pippin the main viewpoint. Interestingly, when the action shifts to Gimli, Legolas, and Aragorn, Gimli is the prime viewpoint, keeping “heroic thinking” from getting too far out of hand. Only at the end does some of the viewpoint return to Aragorn, as he directs the attack on Mordor.

Then there’s Book VI. Now, the viewpoint is almost entirely Sam. The main exception is Faramir and Eowyn; almost a “heroic” counterpoint to the relentlessly “hobbit/half-modern” viewpoint. And by the end, the viewpoint is deliberately, consciously, entirely Sam.

The point is that the viewpoint is almost never that of the author, nor solely that of one main character driving the action forward. Look where you will, to Hemingway, Hardy, Bellow, Updike, or Flaubert, and the viewpoint is almost always that of the main character or of the author; not so with Tolkien. And the result is to gain perspective on the main character and to draw the reader more deeply into the world.

  1. Weather and fauna

Commentators from fantasy writers to critics have confessed their feeling that they would like to live in or experience Tolkien’s locations. But few have noted the way in which the weather – and the places the characters go – is strongly related to the characters’ (and our) moods. Tolkien himself noted that, unusual among authors, he took great care to reflect an actual sequence of weather, just about day by day, from September to March. Moreover, the action takes place just about day by day, with only breaks for Rivendell, for part of the march south, for a short space after the knife in the dark, and for the time within the Mines of Moria. In just about every other day until the Ring is cast into the fire, there is a description of the weather for that day – and the character with the main viewpoint spends a lot of that day outdoors. The weather is always varied, always in some way notable, always in some ways memorable. And it both reflects and causes the characters’ moods; the tenseness of the descent of the Emyn Muil reflected by the thunderstorm, which itself almost causes a disastrous fall; the “heavier air” as the Riders go to war, that both reflects and increases the characters’ tension. In Tolkien’s world, unlike our own, characters feel the weather – “the air seemed to throb”.

As far as I can tell, there is no comparable achievement in all of literature. I simply cannot think of another novel in which there is such a clear and integral description of the weather.

Likewise, there is a clear link between the vegetation and the character of the people inhabiting it. I don’t mean just the fact that pastoral Riders need rolling hills and grasslands; I mean that the Shire is very regulated, with gardens and forest boundaries, while Lothlorien is regulated in a more subtle way, with long-term planning, and Ithilien was once a kind of hothouse of a wide variety of plants, shrubs, and trees, but has clearly been unregulated for some time.

Again, Tolkien is, as far as I can see, unprecedented in his knowledge of botany, and his ability to use it to set a scene and indicate a kind of people. The initial description of Ithilien is the outstanding example; the description of the desolation before the Gates of Mordor is a miniature gem of the opposite kind. But it is interlaced all the way through the novel: from the start, with the description of the Party Tree and the path to Crickhollow, to the swaying leaves above Sam’s head as he awakes in Ithilien, every tree and fern is a part of the main viewpoint.

5. The Ending

I have seen some commentators touch on this, but it should be emphasized: Tolkien crafts neither a tragic (in the classic sense) nor a happy (for the main character, Frodo) ending. This appears to spring out of a constant wrestling on Tolkien’s part with how God justifies death. Through twists and turns in his background notes published by Christopher, it appears that he conceives of Elves as the first answer to Death; they accept it because they are effectively “burned out”, passive shells of their passionate former selves. But Man is the second and better answer to Death, because he is forced into Death prematurely, with no assurance of what happens after, but by conquering his fears he achieves preparation for the best “next” existence, an existence in which may he become, as it were, a “child” of God to be taught by God.

Frodo’s realization that he cannot escape the effects of the Ring and that he has outgrown the Shire for which he thought he was killing himself becomes a tragedy that can’t be solved in Middle Earth; but it allows him to conquer his fears after he leaves for the Blessed Lands, by seeing there is hope ahead. Sam’s happy ending is tainted by the realization that the person for whom he sacrificed himself did not benefit by the sacrifice; but it allows him to live a life that will enable him also to conquer his fear of death. In either case, we are far from “happy ever after”, and just as far from “bad things happen to bad/flawed people, good things happen to good people” tragedy or even “bad things happen for no reason” tragedy.

6. Names

While commentators have noted Tolkien’s creation of a “new language”, his background as a philologist, and the origins in European history of the names of people, they do not seem to have noted the importance of his ability, to some extent, to avoid having the names seem ridiculous. Before Tolkien, fantasists tried to make up alien names as a baroque variation of English ones (e.g., Eddison), or as pagan-sounding (Dejah Thoris in Edgar Rice Burroughs’ Mars series). In either case, the names had no “depth” of source in a complete language. One of the reasons many readers recoiled from fantasy as serious adult literature was that, well, the people’s names sounded silly.

Again, the notes published by Christopher Tolkien show that in his original conception, J.R.R.’s names would have struck us as pretty silly, too – Kor, Gnomes for Elves. The process of developing a full-fledged language with clear associations to “normal” names, to our ear, apparently involved a fair bit of subtle changes to make the names less likely to cause laughter – Strider for Trotter, for example. The image of Aragorn as a pig would have been hard to expunge.

Not that, to some ears, the effort was successful. Reading between the lines, one of the main reasons that Edmund Wilson wrote “Ooh, Those Awful Orcs” was that to his ears, names like Shagrat sounded just as silly and juvenile as Burroughs’. But I think it’s fair to say that there’s a dividing line between fantasy before Tolkien, in which Jhirel of Joiry joined Fritz Leibner’s Fafnr, and after Tolkien, with the far more carefully crafted Welsh names of Morgon and Astrin in McKillip’s works, or Pyanfar and Arafel in Cherryh. Fantasy and science fiction not only learned world-building and history-building from Tolkien; to some extent, they learned language-building.

  1. The Sad Lack of Successors

I can’t say that I find authors who have some sense of most of the preceding Tolkien traits, much less the ability to use them in other contexts, in most non-fantasy/sf writing. Fantasy and science fiction authors, in particular, have been quite successful at building much more detailed worlds that give a much greater sense of “depth” to the story, from Weber to Modesitt. Likewise, authors pay much more attention to the “tone” of the story, and do much better (by and large) at avoiding silliness and making heroic speech seem less alien. But I can find none, in fantasy or out, that has a comparable use of viewpoint, of weather and fauna, of simple metaphor for description, of endings that are neither comedy nor tragedy, or even (in most cases) of poetry as folk song.

Consider three who seem to have made a conscious effort to think deeply about Tolkien: Donaldson, Jordan, and Le Guin. Donaldson is of interest because he took many of the half-sketched ideas of Tolkien – land with a physical sense of health or disease, telepathic horses as equals to humans, immortal evil, and inevitable failure that can only be redeemed by God/chance – and fleshed them out in his own way. But the sense of language and viewpoint is sometimes really off – it is hard to imagine a juxtaposition of three more disparate eras of the English language than describing a character as “fey, anile, and gutless”, or to swallow a sentence that states “her breasts traced opaque circles of invitation beneath her shirt”. I know, these are early efforts; but the names established there (Lord Foul) still recur in his work thirty years later. Some of his “modern” writing (a graduation speech) is far more powerful. Viewpoint is far less subtle, poetry is less informed by folk song or history, and the weather appears to exist often so as to put as much physical stress as possible on the main character.

Jordan, by contrast, seems to have focused on the “quest” and world-building aspects of Tolkien. His gift for detail, however, is more about character and less about weather and fauna. Unfortunately, in his later books, this causes an over-proliferation of characters that make the story harder to follow. His poetry as folk song is actually quite good, but he does not really give the kind of day-to-day sense of weather and fauna as characters in the story, or at least as indicators of mood, that Tolkien can achieve. His approach to viewpoint is less subtle, and his endings are a bit more traditional.

Of these three, le Guin is perhaps the most successful at building on Tolkien’s additions to the writing craft. Her Earthsea books, apparently beginning with a light attempt to imagine a hobbit as a proto-dragon, have a strong sense of the weather, if not of fauna, and a simple style that, while light on metaphor, nevertheless fully understands the power of pre-modern word meanings. Her viewpoints are mostly traditional but show interesting variation, especially inserting women’s viewpoints into what is traditionally a man’s narrative. Names show great “depth”, even if the world-building is a bit “retrofitted” as her concepts expand from story to story. Ultimately, however, more than Jordan or Donaldson, her stories are less day-by-day and more episode-by-episode, which makes the feel of her stories more “abstract” overall.

A French critic is reported to have said “Victor Hugo is our greatest author – alas”, meaning that even Hugo did not compare to the English or American great writers. In the same vein, I feel that authors of whatever stripe have yet to fully appreciate or build on the aspects of Tolkien’s stylistic genius that I have described above; and it’s too bad.

Prediction and the Good Analyst: IBM Rational

I have been unable to find a clear source for this quote, but some author in the 1970s or before defined science fiction as exploring any of three premises about the future – “ (1) What if? (2) If only … and (3) If this goes on …” Rearranging the premises, slightly, what we have is a good description of the role of prediction in analysis, especially for a good analyst:

1. “If this goes on …” is extrapolation. It uses a model based on historical data, and predicts by assuming that things will continue to operate according to the model. For the analyst, this means looking at a vendor’s product release and figuring out how people will react to it, based on how they have reacted in the past.

  1. “What if?” is stress testing. It introduces new events or players not seen before, but which have some reasonable chance of happening, and sees what happens within the model. For the analyst, this means asking whether new products from other vendors, or new trends, will positively or negatively affect the product.
  1. “If only” is normation[1]. That is, it seeks to identify changes that will make for a better outcome for some or all people affected by this future. For the analyst, this means identifying ways that the product could be made better in the future, or ways in which another combination of products can provide better results.

I am reminded of this way of looking at the analyst’s job by recent news regarding IBM Rational. At IBM’s Software Connect, the group announced initiatives including “completing the software lifecycle” by linking development and operational information about software, improving software delivery predictability by applying risk management and analytic techniques, and new ways of integrating IBM’s rapidly proliferating array of software. This was complemented by a Dec. 16 press release that noted change management, collaboration, embedded-system development, SOA-based development, and requirements management improvements to existing Rational products. All this, as Steve Mills noted in his Software Connect keynote speech, is in service to IBM’s commitment to creation of “smarter” products – with instrumentation, automation, and reporting.

To understand my reaction to this news, you should walk back with me over a decade and a half of applying extrapolation, stress testing, and normation to Rational, and then IBM Rational, software development offerings. If you remember, Rational’s original claim to fame was as the most effective purveyor of object-oriented design tools. This focus on development of sophisticated and formalized control tools for large-scale and complex development projects continued until as recently as three years ago.

However, during all that time, there was an alternative – first embodied in the 4GL offerings of the mid 1990s, and then in the agile development community of the early 2000s – that stressed flexible, iterative, solo or loosely collaborating, open-ended development. My recent research suggests that agile development has a paradoxically positive effect on project risks, development times and costs, and organization ROI and agility – and except in very complex projects such as design of the latest Boeing plane, the Rational approach (as it was then) does not. So extrapolation suggested that Rational tools would serve an ever smaller percent of the development market; stress testing suggested in the mid-2000s that adoption of agile development techniques was beginning in offshore development firms as well as small shops in the US; and normation suggested that IBM could leverage the efforts of agile development methodologies such as SCRUM as well as the rapid prototyping features of non-Rational software such as WebSphere Portal to quickly implement a strong agile-development product and focus. Still, as late as 2007, there was no clear indication that IBM, via Rational, would change its focus from “formal” to “agile”.

That was then; this is now. Over the last three years, IBM Rational has clearly made a major transition to focusing on agile rather than just “high-end complex” development, and its announcements represent the culmination of that effort. IBM Jazz, the in-house-developed product for agile mashups, receives top billing; older Rational products have been retrofitted to integrate with agile development tools and support a “spiral”/iterative rather than a “waterfall” (i.e., no going back to a previous development stage) model; and there is a welcome focus on constant incremental upgrades to software via infusion of runtime “operational” data on a piece of software in the development process. What’s not to like?

Ah, but that was then, this is now: it’s time to apply extrapolation, stress testing, and normation to the new Rational. Extrapolation says that the agile focus will make IBM of much more use to a wider array of developers. Stress testing says that while IBM’s competitors at all levels may have some advantages in having “built from the ground up” agile development tools, IBM’s ability to scale development to the high end offers advantages to larger customers. However, stress testing and normation suggest that the new Rational has one glaring hole: lack of tools to handle the “instrumented world”, as IBM calls it, or the “sensor-driven Web”, as I call it.

New types of data – including RFID, GPS, and cell phone photos and videos – are being passed over the Web, and within the enterprise, in massive amounts. The architecture to handle these appears to be evolving as the “event-driven architecture”, a filter on top of an Enterprise Service Bus, in which a “one-size-fits-all” program filters events passing over the bus, aggregates them and places them in context, and then either takes action itself according to pre-defined business rules, or sends notifications to a human that action may be needed. The software for these programs is fundamentally different from most past programming models, because it requires “near-real-time” response, and because the structure is trigger/response rather than input-process-output (old-style programs) or loosely-coupled communication (today’s object-oriented programs). If a development vendor fails to provide competitive tools to handle this new model over the next three years, and if other vendors do, the laggard will miss out on a major part of tomorrow’s development market – just as IBM ceded a large segment of the of the medium-sized, medium-complexity “sweet spot” in the development market to others from 1995-2006.

The good news for IBM is that, its competitors Oracle, Microsoft, and HP also appear to be quite some distance from creating a full-fledged sensor-driven Web development tool – and at least IBM recognizes the opportunity. It also appears that moving in the direction of such a tool will not be such a stretch for IBM, given their existing “stream processing” tools. However, because of IBM’s recent history of “missing an opportunity and then buying a third party” in areas such as global repositories, I won’t count IBM Rational as a leading-edge sensor-driven Web development tool until I see some quality productized code.

So, two cheers for the agile IBM Rational heading into 2010; and an echoing silence where the third cheer ought to be. To figure out where that third cheer went, they might apply a little extrapolation, stress testing, and normation; or they could ask a good analyst …

[1] A made-up word. It means the process of developing normative recommendations from data.