Michelle Accardi-Peterson's Agile Marketing: The Flaws of So Many Virtues

I am partway through reading one of the best marketing tomes I have ever read, skimmed, read the blurb of, or seen summarized: Agile Marketing, by Michelle Accardi-Petersen (CA Press, 2011) – and it ranks pretty high among books in general. As someone who follows agile theory and practice carefully, and who has been a fellow-traveler with Michelle in the computer industry for lo these many years, as a programmer, B-school graduate, manager, analyst, and consultant to marketers, I found the first paragraph to be one of the most pitch-perfect I have ever seen, and the rest of it I have read so far has gone directly to the meat, presented it clearly and attractively, and grasped the overall implications of agile methods and marketing for each other well and with a strong, accurate sense of the importance of the topic.

And so, of course, as is my custom in these cases, I am now going to criticize this book viciously for two key ideas that I view as dangerous and in a profound way headed in the wrong un-agile direction (actually, there’s a third, but clearly Michelle is discussing that one just to humor a fad, even if she doesn’t realize it, so it doesn’t really taint her message. I’ll touch on that one at the end, briefly).
Briefly stated, those ideas are:

1. For the customer (but, of course, he said sarcastically, not for the marketer), perception, not product, is what's important; in a sense, perception is reality.
2. Agile marketing, like other agile methodologies, is about adapting rapidly and effectively to change.

Before I begin eviscerating these fascinating ideas, I should pause and invite those readers with “confirmation bias” to stop reading. By that I mean, if you feel that I am an arrogant, ignorant upstart in criticizing ideas that have been seen as important parts of marketing for many years, and have the sense that the rest of this piece will turn out to be wrong, I am sure you’re right. Please stop reading right now.

Still here? Boy, that reverse psychology worked. Because, of course, I’m just asking you to perceive the world differently, not accept that the world is different; and I’m only asking you to adapt to a customer’s suggested change in your own ideas, not develop additional new ideas before being prompted. Right?

All right, then, let’s dive in. Sharpen your knives for Idea 1.

Marketing Into the Delicate Dance of Perception and Reality

I have encountered this one many times over the years, in unexpressed assumptions, and expressed as “products never fail because of product characteristics, but because of flawed differentiation/positioning/advertising/understanding of the market we’re in.” Let me start with two stories that I believe directly contradict this.

The first one is the story of a software product back in the 1980s called LAN Manager. It was sold by Microsoft and IBM, who were dominant at the time in the PC market. It was sold to the correct market – businesses – and sold with conviction, imho, by both Microsoft and IBM. I can vouch for the fact that not only among businesses, but also among techies, the general view was that LAN Manager was the future. I have heard many rationalizations for the failure of LAN Manager over the years, and they all strike me as bad descriptions of what I saw at the time.

At that time, I was tasked with comparing LAN Manager and Novell NetWare technically, and as I examined LAN Manager I noted an odd fact: it required twice as much main memory on the client PC as any PC afforded that would be on the market for the next 1 ½-2 years. In other words, in the real world, LAN Manager would be effectively unusable except if an organization waited to buy a new round of PCs that wouldn’t exist for a couple of years, and even then might not make sense as replacements for existing PCs. And I should also add that the key, and very effective, selling proposition of the new network operating systems was that you could use a spare PC as the server.

At this point, I can hear the objections: probe a little deeper, and you will find that marketing failed to coordinate new product development effectively. Sorry, but you’re still missing the point: This was the typical act of organizations for whom the customer’s perception of reality was everything, and all you had to do was to manipulate that perception. Microsoft and IBM never saw the problem because they assumed that giving programmers general directions was enough and they could handle the rest by marketing. Because they were so focused on the customer’s perception, they were blind to the customer’s reaction to reality.

Hey, that’s one case, right? All right, here’s another. When I was at Yankee Group in the early 1990s, it was a game among publications to get projections from analysts at different firms as to network operating system share of market in the coming year. Three times I did this. Three times, unlike every other analyst, I picked Novell NetWare to increase market share – it was already at a hefty 70% as I recall, but the heavyweights of the overall computing industry, from DEC to IBM, now had viable, well-marketed NOSs of their own, and every business surveyed indicated that they were going to increase the share of these competitors in the coming year. Year after year, this was occurring, and year after year, Novell NetWare’s share increased, just as I predicted. Monopoly power? Don’t make me laugh. So why was I right, and everyone else wrong?

Well, here’s a gem of a case I collected from DEC itself. I was talking to one of their engineers, and he told me that in order to get their work done, he and others had set up a NetWare network. But because they weren’t supposed to be doing so, they “tunneled” as a small part of the overall DEC network, invisible to both their managers and the administrators keeping watch over expenditures and usage of competitors’ products. Until the volume on the NetWare network grew so large that it began crowding out all the other traffic on the DEC network. And that, typically, was how NetWare won out – because no matter what the corporate strategy was, individual employees acting on behalf of their own needs crowded out corporate plans. Marketing, meet reality.

I hear another bleat from the marketers – not proven. Again, you’re missing the point. In this case, DEC marketing was so focused on manipulating the perceptions of its traditional customers, the CIOs and CEOs who held the purse strings – which, obviously, it did quite well – it failed to realize how the reality of greater NetWare usefulness right now to reality-based programmers and techies trumped corporate plans. And, again, that was because marketing was so focused on manipulating the perceptions of customers, as had been highly successful in the past, that it overlooked the degree to which users’ clear view of the real comparative usefulness of the products at that point in time would create a situation in which, eventually, CIOs bowed to reality and started making NetWare a corporate standard; and then the game was well and truly over, and the mini makers had lost control of their low-end customers. Everyone else bet on perception, as a reflex; I bet on reality (I had used NetWare in the past), because I felt strongly that in this particular case it would trump well-manipulated perception.

I am not just saying that “perception is reality” is wrong in minor ways. I am saying that another idea is a much better model for what is going on, and that, in the long run, and sometimes in the short run, using my idea as a basis for agile marketing is likely to do much better than Michelle’s Idea 1. But before I go on to that idea, let me summarize the ways I believe Idea 1 is wrong-headed:


• If you analyze particular failures and successes carefully, the design of the product, independent of how marketing thinks it should be portrayed (i.e., its “reality”), often plays a key role in long-term success, and in a few cases in short-term success. I emphasize long-run success, because initial success that emphasizes perception too much takes the organization’s eye off the technology-development ball, and causes a greater and greater gap between a pleasant perception and a not-so-pleasant reality. That may have worked in the stable markets of fifty years ago, but not today.
• The idea that perception is reality reinforces in corporate marketing and strategic planning a fundamentally flawed – because biased – view of the customer. It focuses the marketer on manipulating a malleable and necessarily partially unreal perception, rather than engaging in a delicate dance with the customer to get him/her to play with your offered perception rather than another’s, always keeping in mind that a key segment of your market will be spending the majority of their time doing something else and dealing with the necessary realities of real life, and therefore there is only so far you can go in bending and misdirecting around reality. Bluntly: just because you’re a successful marketer, you think the customer is what you want him/her to be in order to sell your product. As an antidote, try to imagine (preferably, checking against reality) a day in the life of several customers and several who are not customers, and then see how much your “buy, buy, buy” caricatures his/her overall behavior.
• Above all, the idea that perception is reality leads to un-agile marketing behavior. It does so because it causes you to misperceive changes in the customer – misperceive them as greater and greater acceptance of the perception you are selling them, rather than the tendency of a sub-segment of the market to become addicts. Except when something is forbidden, addicts are not a good long-term market. If McDonald’s was only delivering more fat and sugar, it would not be growing; instead, its customers would be busy dealing with their excesses via ill-health and reduced buying. What else is it delivering? Not just convenience; it’s also delivering enough food to live on when cost is very important. If, in that delicate dance, you misperceive the customer, your rapid changes do not fundamentally get at what is croaking your markets; they are not, in the long term, effective responses. Therefore, they are fundamentally un-agile – they just give the perception of agility. But hey, perception is reality for the marketer too, right?
• So you can guess my final bullet point. If the marketer thinks perception is reality for the customer, perception will become reality for the marketer, and the rest of the organization: manipulate perception of the marketer by the rest of the organization, manipulate perception of the goals of the organization by employees, manipulate perception of the financials by the stock market – if we do it right, perception will become reality, because perception is all that really matters. And that’s why we see, in the latest surveys of CMOs, a new appreciation for the importance of company culture in sales, and appreciation of the difficulty of changing it to what the marketer would desire in order to sell perception better. Because there’s the marketer’s perception of what the company is; and then there’s the reality of employees’ daily lives and all the other messages their organization keeps sending them.

My Modest Alternative

Having gotten thus far in my musing, I found that Michelle’s wonderful discussion of marketing agility put me in an agile frame of mind, suggesting to me a new way of thinking about things.

So let me present it in its most provocative form: What companies are selling is not product so much as a partial worldview, and what customers do is not so much buy – that is the final or intermediate act – as take part in that worldview, fitting it with the rest of their worldview, for a brief time, hopefully repeatedly. Moreover, just as much as companies compete with other companies’ products for a share of the customers’ money, they compete with other customer worldviews for a slice of the customer’s time. That time, the company should accept, will and should always be a relatively small fraction of the customer’s overall time. If your employees are always buying your products, when do they work for you?

That customer worldview, or parts of it, are always evolving, not just as the customer moves through life but also as the customer always seeks better worldviews. Better means new, not the reverse. Better is, in some sense, reality, and if you are a lucky marketer, your new product will be part of that customer’s better reality. Yes, you can get away with perception that is well away from product reality in the customer’s life for a while, and the customer will not reject it, but, because it is not delivering any benefits but fantasy, after a while, the customer will increasingly prefer another worldview more based in reality. You can sell a Dungeons & Dragons game, but if you don’t think up ways to apply such fantasy usefully to some of the rest of the user’s life, the market of non-addicts will slowly drift away to fantasies that are more useful, like fantasy vacations. And then, the final refuge of the trapped marketer – monopolies that cling desperately to high prices in one area, until they fail to cross the chasm they no longer can recognize.

So marketing strategy no longer becomes, perception is reality, I must create the right perception. Rather, it becomes, how far can I go in creating perception before reality bites me in the donkey? How do I manage both perception and reality so that both are in tune with the customer? And how do I reality-check with customers that my worldview is a viable, differentiated customer worldview, worth spending time in the long run on?

Good time to discuss Idea 2.

Be Proactive!

Michelle is only the latest in a long string of commentators to imbibe received, unexamined wisdom about what agile software development is, and assume it always means reacting to the customer’s change, or the organization’s environment, or at best doing one’s utmost to predict future customer behavior based on the past, and then prepare for a few alternatives. No, no, no. Agile is equally about being proactive. Agile is about both sides of the conversation, the developer and the customer, contributing equally, each held in check by – and also using as a source of new ideas – the other. Agile is about product development that is as informed by the developer’s sense of the possible in the future as it is by the customer’s sense of what is needed right now.

Let me give you an example of a fundamental marketing failure because, almost universally, organizations have failed to grasp this idea. Back in the late 1970s, I worked on one of the first word processors. The problem then was how to make accessible to a consumer – a secretary, an administrative assistant, even, heaven forbid, a fast-track youngster doing spreadsheets – how they could store and view “files” of information on a word processor or computer. And there were two obvious answers: A desktop, on which you store individual “files”, and a file cabinet, in which you can store lots of folders containing files. A desktop was everyone’s experience; a file cabinet, in those days, was managed mostly by the secretary.

You are probably beginning to guess where this is going. The first word processors tried to implement file cabinets. Then they went for desktops with folders scattered around the tops. Meanwhile, the programmers were screaming at the marketers that this was crippling the full power of the word processor and PC to store information, and the marketers were screaming back that people would never understand folders within folders, until finally someone tried it, and, lo and behold, people just used the first level until they got the idea and then were perfectly happy to use several levels of folders; and the product that got there first was among the winners until the next chasm came along. Except that, sadly, the story doesn’t end there.

You see, there was another thing that some programmers were screaming about. With a simple extension of existing computer file storage organization, with the same look and feel as “folders within folders”, you could store the same file in multiple folders. In essence, you could cross-reference with incredible ease. But, gee, said the marketers, people will never understand this. And clearly, they don’t care; no one’s demanding it, are they? And so it has been, from that day to this, with one extremely minor exception:

GOOGLE SEARCH!!!

Yes, I’m being very rude, here. Has no marketer yet realized that one major value of search engines to the average user is that they give a temporary cross-reference of multiple files on the same keyword, different from the static “it’s on this Web site” address? And if you do realize it, how could you possibly, possibly not think that users would really, really find that kind of cross-reference useful on their basic PC or browser desktop, in addition to today’s way of storing information? But hey, developers couldn’t possibly have anything to with what users would find useful, would they? Technology-push. Dirty, dirty word. Oh, and by the way, funny how Steve Jobs dared to try drag-and-swipe, fifteen years after programmers first started talking about it. Yeah, he was very good at fitting it to customer uses; but customers weren’t asking for it, were they?

And the most ironic part of this is that Michelle, with her right brain, is saying, agile is reactive, while with her left brain she is talking about keeping marketing simple. Because that’s typically when programmers or designers can anticipate user needs well – something that mathematicians (and, believe it or not, good programmers are good mathematicians) call orthogonality. Basically, orthogonality is a user interface in which (at the top level) a minimum number of approximately equally powerful actions cover most or all of what the user wants to do. You want an example? Take a look at the original Microsoft Word GUI. File. Edit. A couple more. Help. Did you realize that was one of the key reasons people started preferring Word to WordPerfect? Yeah, there were some verbs and some nouns, but basically people could look at it and say, yeah, now I do something involving the overall file, like Print it. Now I do something that involves editing. Now I … That was a developer’s idea – orthogonality. WordPerfect was more a marketer’s idea. When it’s orthogonal it’s simple – see, Michelle? Only the developer can do orthogonal, because the customer doesn’t spend the time thinking about things from the developer’s point of view, just thinks about the next change for the next immediate need.

And that’s the funny thing about agile software development. It encourages orthogonal software development, because it turns out it’s quicker to develop. It encourages the developer to suggest ideas to the customer, not just the other way around. You know, in looking at implementing this human resources app, it occurred to me that it would be straightforward to link your profile to your Facebook page. You can do that??? Oh, yeah. In fact, to make it orthogonal, I can just leave open what you want to link to – like Google Groups. Wow. Let’s see, what could I do with that …

I don’t think we need to belabor what my fix is: it ought to be obvious. Marketers should start thinking like Saki’s master politician, who knows just how far to go (what the customer is demanding) and then goes just a little bit farther (extends it to simplify the design, or gets a programmer to do that during product development, or preferably both). Crowdsourcing? Crowdsourcing is still reactive. You need a single voice that simplifies and goes a little bit farther. And you do it in a rapid, spiral fashion. Customer leads a little bit; you lead a little bit; customer leads; you lead. Reactive agility goes to where the customer has been. Proactive agility goes where the customer doesn’t know he/she would like to go – and then reality-checks.

Just one last sneaky point, and then my sermon is over. Why do marketers ignore the very real and valuable orthogonality contributions of developers? Could it be because the reality of the product doesn’t matter, only its perception by customers?

Oh, Yeah, Third Criticism, But Even I Don’t Care in This Case

Somewhere in the first Chapter, Michelle drops a casual remark about “lean, agile” as if this were a good thing. Yeah, there are plenty of folks in product development who believe sincerely in that one. I have a simple test: When you are creating the product time-line, what’s more important? The agility of the participants, or just-in-time provision of resources to the project? If you rush one programmer towards the next task or delay him/her, are you doing it so no resource goes un-utilized and none are unnecessarily utilized, or because the programmer needs that to be done in order to spend extra time to improve the agility of his/her process? Who is the master, agile or lean?

I have written on this extensively elsewhere. I understand that Thoughtworks, these days, is generally doing “lean in the service of agile” right. Here’s the bottom line: If lean wins, you inevitably become less and less agile. And that, in turn, will mean that despite a lean veneer, you will have greater costs, less profit, and less customer satisfaction. Do you want the perception of cost savings in that project? Or do you want the reality of better company financials, short-term and long-term, rain or shine? Don’t tell me “lean, agile” is necessarily a good thing.

Keep On Changing, Michelle! Pay Attention, Everybody!

So, after many pages of attempted disemboweling, I anticipate that most readers who have hung in this far (ah, so my reverse psychology didn’t work after all, did it?) will have the nagging thought in the back of their heads: if the writer is this critical of Michelle, how can he possibly mean those nice things he said about her and the book at the top?

Again the answer is simple, and, I hope, orthogonal. There’s a story that Mrs. Justice Oliver Wendell Holmes once met President Theodore Roosevelt at a party, and, as was his habit, he asked her what she thought of his Presidency. “Oh, Mr. President,” she replied, batting her eyes at him, “I believe that everyone should be forgiven the vices of his virtues, and you, Mr. President, have so many virtues!” He beamed happily at that one – until he suddenly got the point.

I am, I hope, less catty than Mrs. Holmes in saying the same thing. There would be no point in pointing out the points I disagree with if I did not feel that this book was so close to getting it right (there’s that arrogance again) that it is vitally important that Michelle be read, and her readers consider these points. And consider them agilely. Pay attention to Michelle, all you marketers! Keep on changing, as she says! And, please, keep on changing, Michelle! Changing my agile marketing blues away …

How To Have Fun Following the Stock Market

Over the course of 40 years, I have been following the stock market. Obviously, I have a bit of a personal interest, but it’s also fun to make up my own mind about what it all means. And, of course, it’s a bit like rooting for a team – with the added fun that, over the long term, the stock market has gone up by about 10% per year, so you tend to win at the end of the day a bit more frequently than you lose – definitely not the case with some of my sports teams.

So let’s talk about how I might do it (bearing in mind, of course, that these are brief glimpses via a TV stock market ticker in places like restaurants or on the radio while travelling on occasional days, plus checking the results at night as part of looking for good news to go to sleep with). The first thing I do is, concentrate on the S&P 500. The Dow Jones, I have found, isn’t always representative of the overall stock market, because its selection of stocks is a bit quirky; and, of course, the NASDAQ is too heavily high-tech-weighted. The S&P contains stocks from both, and so, if the NASDAQ goes one way and the Dow goes another, as happens surprisingly often, you can pretty much count on the S&P being somewhere in the middle.

The next thing I do is remind myself that the S&P 500 does not include income from dividends. According to studies I have seen, those dividends add about 2.15% to the S&P’s return each year. So, if the S&P is down a little on a given day, I can comfort myself with the reflection that so far in the year, it has actually returned, say, 1% more than the number suggests. If you want a further comforter, S&P publishes “total return” (S&P 500 plus dividends for the year so far). And that doesn’t include reinvestment of dividends, which, if the market has gone up by 15% so far, will yield you another 0.15% by the end of the year (p.s., due to the magic of compounding, that will be nothing to sneeze at in 5 years or so). See, I’m winning already!

If I happen to be able to look at the stock market before the opening bell, the only thing I look for is “S&P futures”. That happens to be a fairly reliable indicator of where the stock market will hover around until about 2-2:30 pm … more on that later. Be careful of the name, because there’s another S&P futures market whose abbreviation is very similar, but which tends to be negative when S&P futures is positive, and vice versa.

And then there’s one more thing I can do, and pretty much do every day. That is, look to see if it’s sunny out. You see, I live near Boston, reasonably near NYC – near enough to guess what the weather is like around 8 or 8:30 in the morning, when the traders go into NYSE to prepare for the day’s session. A fascinating study a while back showed that when those traders see sunny weather outside, they tend to be more optimistic in their outlook, and so the S&P 500 tends to go up more often. And so, when it’s sunny out at that time, no matter whether it’s pouring the rest of the day, I am more likely to anticipate a good day. It’s sunny this morning! I’m going to win, yay!

Unfortunately, I find that the news rarely gives a good indication on what will move the market today, no matter when in the day you check. After many years of listening to them, I tend to distrust ex post explanations. So I just check, if I’m lucky enough to see, what is happening around 10:30-11, when the spurt of the S&P 500 to its futures level has been pretty much over for a good half hour. On the rare occasions when the futures projection is countered by sudden events, that’s when the big drops or big gains will show up. Yay! The Fed is going to give guidance today, and traders usually overestimate the effect on the positive side before they announce around 2!

If by some amazing chance I get a chance around lunch to watch the S&P 500 update on an every-half-minute basis, I can watch the dance of the buyers and sellers. The way it plays out is, these days, a surge up of less than a point as buyers buy the prices up, followed by a drop of less than a point as sellers sell the price down. It’s only after 2-3 minutes that you can see whether the overall trend is up or down – unless a thrilling jump up suddenly moves the market up by a point or more. Or an annoying big drop down interrupts the festivities. To heck with the business news – where’s the sports?

But the real surprises tend to occur around 2-2:30, when suddenly the market can go completely away from the futures prediction. It almost always is due to breaking news – the Fed, late-in-the-day government reports, political news, late news from Europe. If no break occurs then, the futures prediction is probably good for the rest of the day. Yay if it was sunny this morning!

And then the final bell rings at 4, and about 4:04 to 4:06 you get the final S&P 500 figure of the day. But, oddly enough, if the S&P is up more than 1% (these days, about 13 points), I don’t get too happy, and if it’s down more than 1%, I don’t get too sad. Because if traders really understood what was happening, there’s no way there would be jumps like that. I’m a cynical guy; as far as I’m concerned, traders can be pretty unsophisticated in their fears and enthusiasms, so what this tells me is that traders are running around like chickens because it’s going to take them forever to see that the latest news, ultimately, is no big deal. And since that kind of thing can last for months, I just sit and wait until sanity arrives, with my own idea of where the S&P 500 should be. On the other hand, if it reaches the point where Intel is at 60 times earnings (as it was in 2000), or 5 times, that’s real craziness.

Next, I factor in yearly cycles. The one I like right now (or maybe “like” is too strong) is the end-of-the-month cycle in which, I suspect, traders start trading the price back towards the value at the beginning of the month, to show gains in their monthly performance report if there are any, or because they have oversold through panic and now are just waiting until the price comes back up through lack of sellers in order to cut their losses. That tends to mean that I pay less attention starting around the 21st. Ho-hum.

The next thing I like to anticipate is the quarterly period from about Month 1 Day 15 to about early in Month 2 when companies make their earnings reports. Those companies, I really suspect, manage their numbers to analysts’ estimates. And that means, most times, many more times where the companies beat the numbers by a little, and the stock market keeps bouncing up, bit by bit. Or, there’s negative news, but the earnings reports match it until the panic goes away (or outlast the reports, in a few cases). Fun times. Less than 1% S&P 500 daily increases. Yay!

And then there is the full yearly cycle in which, many times, the “Christmas rally” tends to take place right after New Years’, followed by continued upward movement to the later part of April, followed by slow and at the end more rapid decline to mid-October, followed by comeback until the end of the year. This varies a lot, but it typically means that about 2/3 of the year is an up-tick. Yay! And if it isn’t, one of my sports teams won again. Yay!

But the best part of it all is those times during the year when the S&P 500 is really on a hot streak, or at the end of the year. If the S&P 500 is on a tear, I can sit down and fantasize what my teeny S&P 500 index fund has grown to. If I’m at the end of a down year, I can sit down and remind myself that the underlying S&P 500 is growing by 10% per year, blithely ignoring inflation and eventual capital gains taxes (don’t rain on my parade!) And at the end of the year I can raise a toast, to my teeny S&P 500 index fund that over the last 25 years of 10% average returns has grown 10-fold, to a slightly less teeny amount, and fantasize that if life were just that index fund, I would be rich, rich beyond the dreams of avarice, and that my stock-market team had therefore won the Super Bowl.

Happy New Year! Yay!

The Other Agile Development: XtremeEDA and Product Development

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in agile development, agile new product development, and business agility over the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about agile development today.

The Importance of Continuous Delivery to Agile Development

I have taken a hiatus from this series over the last two weeks, but I knew one thing that I wanted to tackle: product development that included hardware, or New Product Development (NPD) solutions. And I have to say that, in general, the situation is distressing.

It’s been more than ten years since the Agile Manifesto, and in that time there have certainly been vocal proponents of agile NPD. The lean movement, for one, keeps claiming that “lean” has incorporated the tenets of agile software development, that “lean” and “agile” are complementary, and that this is a methodology that is being applied to NPD in general.

Looking at their and other evidence, I am not at all convinced. There is little or no evidence that their proponents are intensively applying agile software development techniques and methodologies to actual hardware development, whether you’re talking about engineering a plane or designing and fabricating a chip. There is a persistent tendency in what I have seen in the agile/lean movement to treat just-in-time leanness and fast-design-change-from-user-feedback agility as equally important (or, treat lean as more important than agile), rather than carefully considering what it would mean to have super-lean NPD give way when agility demands (as, in agile software development, it often does) temporary use of extra resources plus Gantt charts that are not time-optimized. Finally, I cannot find clear evidence of forward thinking on how, if we are going to maintain that agile hardware development is not appropriate in many hardware-design cases, agile and non-agile will integrate effectively. It is clear from some of the comments I have seen that many hardware engineers are bewildered by the seeming Wild West of agile software development, and in no mood to face an overall schedule in which they depend on the changing specs and indefinite deadlines of the software side.

And yet, I was able to find one small but amazingly perceptive example of another kind of thinking about NPD. XtremeEDA Corporation, and in particular Neil Johnson there, is just what I’m talking about.

If you want to see an example, try submit2011.agilealliance.org/files/session_pdfs/applying%20agile%20to%20ic%20development.pdf. There it all is: How to turn ASIC development into agile; how to coordinate with agile software developers; how to use more customer feedback; and the message “to heck with minimal just-in-time resources and up with ‘one person owns a task.’” He even – be still, my heart – quotes Ken Schwaber, whom I knew, before his days as Mr. Scrum and as one of the original Manifesto signers, as the vendor of a superb rapid application development tool that basically “got” agile (alas, it was only for the AS/400).

Even I don’t want to overstate the relevance of agile development approaches to hardware NPD. There are still constraints, not the least of which is the fact that producing a hardware prototype or even a functioning sub-product prototype is in many cases a costly and time-consuming process that we cannot yet completely accomplish as swiftly as compile-load-go.

And yet, what Mr. Johnson makes clear is that, in the real world, agile hardware development and integrated, totally agile joint hardware/software NPD still do better than the traditional counterparts, in time, in money, and in customer satisfaction. And that’s just one product iteration. You tell me what rapid, incremental, customer-pleasing hardware improvements would mean for your bottom line.

Isn’t it time that not only IT but also the CAD and manufacturing side gave it some thought?

The Relevance of XtremeEDA to Agile NPD

I don’t want to misrepresent the imminence of agile NPD, so these next two sections will be short. Neil Johnson is one person in XtremeEDA, and clearly the firm doesn’t make a big point of his efforts, I am guessing in order not to spook its traditional customers for consulting and ASICs. I am sure that XtremeEDA does a fine job with these tasks, and that if you want to use them you need never run the risk (?????) of agile hardware development, or even hear about it, if you don’t want.

That said, I am also sure that if anybody does want to see how it’s really done, XtremeEDA would be very pleased to have someone like Neil show them, and any fees involved have got to be minimal considering the advantage to be gained. So what, bottom line, could XtremeEDA potentially offer if you go that route?

First, hands-on experience – as in the url to which I just sent you. Second, creative thinking about how to apply this experience and the resulting insights to a wide variety of (at least in the chip area) hardware development projects, fab and otherwise. Third, both experience and creative thinking in how to mesh embedded-software and hardware agile development. Fourth, consulting and support for your efforts. Above all, fifth, suggestions on scaling agile hardware and hardware/software NPD.

The point of this exercise in imagining how xTremeEDA could help you is to lead you to the next question: How will this information make my NPD – specifically, NPD that includes hardware, like a cell phone or a child’s toy – agile? And the answer, in case I haven’t said it sufficiently, is that agile is, more than anything else, a process and a culture. You process NPD that way, you think NPD that way. If you are really doing agile NPD right, NPD agility comes with the package. If you use information from someone like XtremeEDA to implement agile hardware development and integration between your agile software development and agile hardware development, your NPD will be agile.

Potential Uses of XtremeEDA-Type Agile NPD for IT and NPD

In this case, IT and NPD are two separate things, and often, in the case of hardware development, two separate organizations. On the IT side, if there is a separate NPD organization, this means evangelism. It may not work – but with someone who has walked the hardware walk at your side, like XtremeEDA, you may actually get listened to. Of course, if the NPD organization is folded under a joint hardware-plus-software IT organizatio, it’s more a matter of top-down commitment from the CIO (and, hopefully, the CEO), and then using someone like XtremeEDA to hand-hold and calm the rightful fears of experienced hardware engineers.

If we’re talking about an overall NPD organization to whom software development is both important and an annoying pest, on the other hand, you may want to consider initially leaving IT and/or agile software development organizations out of the conversation completely. That will avoid the usual antagonisms that surface between hardware and software teams, and it will also make the organization squarely face the fact that pure agile hardware development works – and there’s a hardware geek like someone at XtremeEDA to tell them so. Once the hardware-only implementation has shown the doubters, and worriers begin to turn into enthusiasts, you can let loose the software and hardware sides on each other, and they will automagically start practicing their getting-user-feedback skills on each other. An integrating process tool is great, but don’t forget that these folks may well do some of the integration on their own.

The Bottom Line for IT Buyers

I freely admit that an XtremeEDA approach to NPD may turn out to be of little value to most companies in the next 2-3 years. That, however, is not because it intrinsically has no value. Rather, it is because I am fully confident that most companies with significant hardware in their products will not figure out how to walk the walk before 2015. They will probably not be in a hurry, because they will rightly figure that most of their competitors will take the same attitude. Into the pool? You first. I live in hope; but I’ve seen too much.

However, when – not if – agile NPD really begins to take over, I urge you to consider carefully the further implications of truly agile NPD. My survey results, as well as experiences when software is the only product of the company, suggest that NPD is the absolute biggest bang for the buck for the business in implementing business agility, leading to permanent, 10-35% improvements in revenues, ROI, and customer satisfaction (not to mention quality), and similar reductions in costs, compared to what they would have been had you not implemented agile. Today, in many companies, agile software development is really about competitive advantage. In NPD, it is about the top and bottom lines.

Moreover, because NPD is always at the core of company success, agile NPD leaks into other areas of the company, far more than agile software development. Marketers cannot help being more agile. CEO strategies, given new power to change product directions fast, become more nimble, and the CEO begins to think more in terms of strategy change than enforcing execution. Even finance – well, maybe someday.
If, however, you really are a visionary, and would like to do it now, then I beg you not to talk about it publicly. What? XtremeEDA? Oh, they’re just here to help us out with some ASICs. Nothing to see here. These aren’t the droids you’re looking for. And your company’s success? Oh, just lucky, I guess. Fun, isn’t it?

Yes, it is. And XtremeEDA-type agile product development just happens to be quite profitable, too. Just a coincidence, I’m sure. Yeah, right.

Levels of the Game

I’m sorry, I can’t help it. I want to talk about the greatest football game I have ever seen. It was almost perfect.

You see, I’m a Patriots fan. I’m a Giants fan. I had some basic understanding of their DNA. And what I hoped for, and longed for, for this game, was a perfect display of Bill Parcells football. In it, there would be move, countermove, every coach’s decision, every quarterback’s decision close to the very best one in the situation given the players’ capabilities, every player playing up to the maximum of their capabilities given their nervousness and sometimes their unfamiliarity with the Super Bowl, all the way to the end of the game, and at the end, at the very last play, the Patriots would win. But that wouldn’t matter. Because what mattered was the perfection.

And for the first three quarters, and into the fourth, it happened just as I hoped. Move. Countermove. Never really taking too much of a chance. Never really taking too little. Every player beginning to play in the moment. Did you see the way the Patriots’ defense tried to strip Ahmad Bradshaw of the ball, just as they should? Did you see the way the very minor parts of the Giants’ team, like Henry Hinoski, began to trail Bradshaw and Nicks, so that twice they were in the perfect position to recover a fumble? You never see that in a football game. Never. Move. Countermove. Perfect timing of timeouts (with a couple of unimportant, to-be-expected exceptions from Manning in the second half). Perfect move and countermove on the special teams. And then, early in the fourth quarter, the referees screwed up.

You have to understand that until then, I have never seen referees perform at such a consistently high level. Every spot far more perfect. Every call that mattered, right the first time, no matter how difficult. And then, on a third down with the Giants marching, the referees failed to pick up what I think was an extremely subtle foul by a defensive back. And that put everything out of whack. Tom Coughlin was furious. I don’t blame him. Because if everything had been perfect, and the referees had performed at the high levels of the rest of the game, the referees would have picked it up, and the game would probably have gone just as I hoped.

And then Eli Manning saved the day. Both teams adjusted to the new reality, but we were still probably going to see a Giants’ victory due to that referee’s mistake – and then, as the Giants were marching down the field towards the almost inevitable touchdown and two-point try that would succeed, Eli managed to hit Nicks for a 14-yard gain on second down, when he should have succeeded on third down, and that would have meant the Giants’ field goal would have come too late, and Brady could never have gotten up the field in time for the Patriots to win. Now, things were back on track for perfection.

And then, on the very next play, someone or someones in the Patriots’ defense screwed up when they shouldn’t have. And the perfection was irretrievably wrecked. It was almost inevitable that Jacobs or Bradshaw would run. The Patriots defense should have been ready enough. Belicheck had called time outs and challenged an obvious call and used the two-minute warning to give them time to gather enough energy to make this play. Bradshaw should have gotten 2 or 3 yards. Then, some way or another, the Giants should have gotten to fourth down in the 10-20 area, kicked a field goal with about a minute to play, and Brady would have marched the Patriots down to between the 20 and 35 yard lines on the last play of the game and Coughlin would have called his last time out and Gostkowski would have nailed the field goal anyway and the Patriots would have won. And that wouldn’t have mattered. Because what mattered was that I was seeing thinking, every-effort football that was yielding extraordinary play after extraordinary play, from Jason Pierre Paul blocking Tom Brady’s throw just right to Rob Gronkowski coming in on a bum leg and Tom Brady picking just the right spot to use him.

And now, the Patriots’ loss was just about inevitable. Now, Manning could use the 4-yard pass to Nicks that gave him a first down. Now, the Patriots had to use their second timeout and then let Bradshaw score. Now, the time was just too short, and Brady’s best effort could not get the Patriots to the 15-yard line or within that gave them a probable touchdown on the last play. Against the Giants’ defense, with both teams playing at the peak of their game as they were, Tom’s perfect throw into the end zone on the last play of the game had very little chance of succeeding. It would be only fitting that it fail; and it did.

But still, in the short interval before the almost inevitable happened, there were some extraordinary things that I have never seen. Did you see Tom Coughlin choosing to run for the extra points? There was minimal chance that a fumble would cause a runback. If it succeeded, then there was still some small chance if Brady somehow got a touchdown that the extra point could be blocked. If it failed, and Brady somehow screwed up on his timing and got the touchdown too early, there was a small chance that Manning could take the ball down to field goal range and Tynes could nail a field goal and we would be in overtime. It was the absolutely perfect Bill Parcells coach’s move.

But the thing that really opened my eyes was the way, in the middle of the play where he scored, Ahmad Bradshaw realized what the Patriots were up to and suddenly, just as he was about to cross the goal line, stopped. In the way that absolutely minimized the chance that he would not score a touchdown at the end of the play. Just to take 2 seconds off the clock. I have never seen a player do that pitch-perfect thinking at that point in the game. Never.

Now think ahead to the end of the game. Tom Brady is around his own 45. There are 5 seconds left in the game. The only thing he can safely do is throw it in the end zone, from his own 45. And that’s a very low-percentage play.

But suppose he had six seconds, or seven – if Bradshaw had not done that. He could have passed for another 11 yards, out of bounds or not, and the Giants, playing perfectly, would have let him. Now he would be on the Giants’ 45 yard line, with about one second to play. Now, just as before, he looks and looks downfield, is flushed out of the pocket, moves towards the sidelines, and a lineman is heading towards him and a contain man is just downfield, in the perfect defense, and then suddenly he runs past the scrimmage line. He darts past the contain man. Now the Giants defense in the end zone is beginning to react just as they should, and they move over to get him and handle possible nearby laterals in just the way they should, and he reaches about the 25 yard line just before they get to him and he whirls and throws a perfect strike across the field to some tight end who has drifted downfield. Before, that tight end would have been on the 35 yard line, so Brady would never have tried that play -- too low-percentage. And now we see that the Patriots’ defensive line has begun to block the Giants’ offensive line upfield, just as they should, and all those receivers in the end zone are coming to block the Giants’ defensive backs away from the other side of the field, just as they should, and still the Giants’ defense, playing perfectly, will get over there and tackle the tight end somewhere between the 5 and 10 yard line. Except that maybe they won’t. Maybe somehow that tight end will find a way to make it into the end zone. It’s a better chance than trying to throw the football into the end zone from one’s own 45. And it wouldn’t have mattered which of the two happened. I would have seen one more extraordinary play.

Yes, all this is very improbable. But that’s the point. What Ahmad did, that’s real perfection. What Coughlin and the rest of the team did to bring Ahmad to the point of making that play, that’s real perfection. What the Patriots did to bring the game to that point, that’s real perfection, marred only by that one defensive play. The whole game brought Ahmad to that point of perfection. And, as it turned out, it mattered a little bit. That, to me, is real enjoyment. That is football as I first saw Bill Parcells develop it, way back in 1986 as he marched to his first Super Bowl win, and this time played by both teams. That is an absolute symphony of football, every player playing close to the best of his capabilities within a great plan and completely integrated with their teammates, so that each side keeps forcing the other to make an extraordinary play at the end – or just miss it, which is almost as good.

Upset that Welker missed that big catch near the end? Don’t be. The Giants’ lineman were pressuring Brady just enough and the defenders were just close enough to Welker that about the best throw he could have made was that throw, and Welker would have had to have made an insanely great catch. He didn’t make it? That’s not the point. The point is that he had reached such a level at that point in the game that he almost made that catch. It wouldn’t have made the game as good if he had made that catch. Whether he had made it or not, wow.

You know, the best line I ever saw written about football was done by Dan Jenkins in his book Semi-Tough. At the end of a fictional Super Bowl, even though the rest of the game has been far from perfect, for one brief moment at the end of the game one of the great players on one of the teams has knocked over the best player on the other team with everyone else playing perfectly to score a touchdown that wins the game. And the winning player goes over to the losing player’s locker room to tell him that with all the unlucky breaks the losing team has had, that losing team should have won. And the losing player looks the winning player in the eye, and says, simply, “One thing I’ve learned, son: Them as should have won -- did.”

That is what we almost had. Not only that one team deserves to win, but that it is all proved out when both are playing and acting at an extraordinary pitch of that perfect amount of extra stretch in a completely integrated way, and even the lucky breaks even up, so that all doubt as to who deserves to win is removed, and what we are left with are the plays, and the plays, and the plays. I know a lot of people are happy or unhappy for other reasons, but I wish they would see it my way. I wish all the players would realize that no matter how many times they go to the Super Bowl, they will probably never take part in a game as great as this one. Please, be happy. I sure am.

The Other BI: Orange EDA and Statistical Analytics

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about BI today.

The Importance of Orange-Type Statistical Analysis to Analytics

BI has taken a major step forward in maturity over the last few years, as statistical packages have become more associated with analytics. Granted, SAS has for years distinguished itself by its statistics-focused BI solution; but when IBM recently acquired SPSS, the grand-daddy of statistical packages, the importance of more rigorous analysis of company and customer data seemed both confirmed and more obvious. Moreover, over the years, data miners have begun to draw on the insights of university researchers about things like “data mining bias” and Bayesian statistics – and the most in-depth, competitive-advantage-determining analyses have benefited as a result. And so, it would seem that we are on a nice technology glide path, as statistics completes the flexibility of analytics by covering one extreme of certainty and analytical complexity, while traditional analytics tools cover the rest of the spectrum up from situations where shallow and imprecise analysis is appropriate, and as statistical techniques filter down by technology evolution to the “unwashed masses” of end users. Or are we?

You see, there is a glaring gap in this picture of increasing knowledge of what’s going on – or at least a gap that should be glaring. This gap might be summed up as Alice in Wonderland’s “verdict first, then the trial”, or business’ “when you have a hammer, everything looks like a nail.” Both the business and the researcher start with their own narrow picture of what the customer or research subject should look like, and the analytics and statistics that start with such hypotheses are designed to narrow in on a solution rather than expand due to unexpected data, and so the business/researcher is very likely to miss key customer insights, psychological and otherwise. Pile on top of this the “not invented here” syndrome characteristic of most enterprises, and the “confirmation bias” that recent research has shown to be prevalent among individuals and organizations, and you have a real analytical problem on your hands.

This is not a purely theoretical problem, if you will excuse the bad joke. In the psychological statistics area, the recent popularity of “qualitative methods” has exposed, to those who are willing to see, the enormous amount of insights that traditional statistics fails to capture about customer psychology, sociology, and behavior. Both approaches, of course, would seem to suffer from the deficit that Richard Feynman pointed out – the lack of control groups that renders any conclusion suspect because a “placebo” or “Hawthorne” effect may be involved – but it should be noted that even when (as seems to be happening) this problem is compensated for, the “verdict first” problem remains, because the world of people is far less easy to pre-define than that of nuclear physics.

In the world of business, as I can personally attest, the same type of problem exists in data-gathering. For more than a decade, I have run TCO studies, particularly on SMB use of databases. I discovered early on that open-ended interviews of relatively few sysadmins were far more effective in capturing the real costs of databases than far wider-spread on-a-scale-from-1-to-5 inflexible surveys of CIOs. Moreover, if I just included the ability of the interviewee to tell a story from his or her point of view, the respondent would consistently come up with an insight of extraordinary value, such as the idea that SMBs didn’t care so much about technology that saved operational costs as much as technology that saved a local-office head time by requiring him or her to just press a button as he or she shut off the lights on Saturday night. The key to success for my “surveys” was that they were designed to be open-ended (able to go in a new direction during the interview, and leaving space for whatever the interviewer might have left out), interviewee-driven (they started by letting the interviewee tell a story as he or she saw it), and flexible in the kind of data collected (typically, an IT organization did not know the overall costs of database administration for their organization [and in a survey, they would have guessed – badly], but they almost invariably knew how many database instances per administrator).

As it turns out, there is a comparable statistical approach for the data-analysis side of things. It’s called Exploratory Data Analysis, or EDA.

As it has evolved in the decades since John Tukey first popularized it, EDA is about analyzing smaller amounts of data to generate as many plausible hypotheses (or “patterns in the data”) as possible, before winnowing them down with further data. To further clear the statistical researcher’s mind of bias, the technique creates abstract unlabeled visualizations (“data visualization”) of the patterns, such as the strangely-named box-and-whisker plot. The analysis is not deep – but it identifies far more hypotheses, and therefore quite a few more areas where in-depth analysis may reveal key insights. The automation of these techniques has made the application of EDA a minor blip in the average analyst’s process, and so effective use of EDA should yield a major improvement in analytics effectiveness “at the margin” (in the resulting in-depth analyses) for a very small time “overhead cost.” In fact, EDA has reached the point, as in the Orange open-source solution, where it is merged with a full-fledged data-mining tool.

And yet, I find that most in university research and in industry are barely aware that EDA exists, much less that it might have some significant use. For a while, SAS’ JMP product stood bravely and alone as a tool that could at least potentially be used by businesses – but I note that according to Wikipedia they have recently discontinued support for its use on Linux.

So let’s summarize: EDA is out there. It’s easy to use. Now that statistical analysis in general is creeping into greater use in analytics, users are ready for it. I fully anticipate that it would have major positive effects on in-depth analytics for enterprises from the very largest down at least to the larger medium-sized ones. IT shops will have to do some customization and integration themselves, because most if not all vendors have not yet fully integrated it as part of the analytics process in their BI suites; but with open-source and other “standard” EDA tools, that’s not inordinately difficult. The only thing lacking is for somebody, anybody, to wake up and pay attention.

The Relevance of Orange EDA to Statistical-Analysis-Type BI

Orange’s relevance may already be apparent from the above, but I’ll say it again anyway. Orange’s EDA solution includes integration with enterprise-type data-mining analytics, and supports a wide range of data visualization techniques, making it a leadership supplier in “fit to your enterprise’s analytics.” Orange is open source, which means it’s as cheap as you can get for quick-and-dirty, and also means it’s not going to go away. Most importantly, Orange lays down a solid, relatively standardized foundation that should be easy to incorporate or upgrade from, when someday the major vendors finally move into the area and provide fancier techniques and better integration with a full-fledged BI suite. That’s all; and that’s plenty.

Potential Uses of Orange-Type EDA in Analytics for IT

Since IT will need to do some of the initial legwork here, without the usual help from one’s BI supplier, the most effective initial use of Orange-type EDA is in support of the longer-term efforts of today’s business analysts, and not in IT-driven agile BI. However, IT should find these business analysts to be surprisingly receptive – or, at the least, as recent surveys suggest, amazed that IT isn’t being a “boat anchor” yet again. You see, EDA has a sheen of “innovation” about it, and so folks who are in some way associated with the business’ “innovation” efforts should like it a lot. The rest is simply a matter of its becoming part of these business analysts' steadily accumulating toolkit of rapid-query-generation and statistical-in-depth-insight-at-the-margin tools. EDA may not in the normal course of usage get the glory of notice as the source of a new competition-killer; but with a little assiduous use-case monitoring by IT, the business case can be made.

It is equally important for IT to note that EDA is twice as effective if it is joined at the front end by a data-gathering process that is to a much greater extent (to recap) open-ended, customer-driven, and flexible (in fact, agile) in the type of data gathered. Remember, there are ways of doing this – such as parallel in-depth customer interviews or Internet surveys that don’t just parrot SurveyMonkey – that add very little “overhead” to data-gathering. IT should seriously consider doing this as well, and preferably design the data-gathering process so as to feed the gathered data to Orange-type EDA tools where in-depth statistical analysis of that data will probably be appropriate as the next step. The overall effect will be like replacing a steadily narrowing view of the data with one that expands the potential analyses until the right balance between “data blindness” and “paralysis by analysis” risks is reached.

The Bottom Line for IT Buyers

To view Orange-type EDA as comparable to the other BI technologies/solutions I have discussed so far is to miss the point. EDA is much more like agile development – its main value lies in changing our analytics methodology, not in improving analytics itself. It helps the organization itself to think not “outside the box”, but “outside the organization” – to be able to combine the viewpoint of the vendor with the viewpoint and reality of the customer, rather than trying to force customer interactions into corporate fantasies of the way customers should think and act for maximum vendor profit. We have all seen the major public-relations disaster of Bank of America charges for debit cards – one that, if we were honest, we would admit most other enterprises find it all too easy to stumble into. If EDA (or, better still, EDA plus open-ended, customer-driven, flexible data-gathering) prevents only one such misstep, it will have paid for itself ten times over, no matter what the numbers say. In a nutshell: EDA seems like it’s about competitive advantage; that’s true as far as it goes, but EDA is actually much more about business risk.

The Orange value proposition for such uses of EDA has been noted twice already; no need to repeat it a third time. For IT buyers, it simply means that any time you decide to do EDA, Orange is there as part of a rather short short list. So that leaves the IT buyer’s final question: what’s the hurry?

And, of course, since EDA is about competitive advantage (sarcasm), there is no hurry. Unless you consider the possibility that each non-EDA enterprise is a bit like a drunk staggering along a sidewalk who has just knocked over the fence bordering an abyss, and who if he then happens to stagger over the edge is busy blaming the owner of the fence (the CEO?) all the way to the bottom. That abyss is the risk of offending the customer. That inebriation is business as usual. EDA helps you sober up, fast.

I can’t say that you have to implement EDA now or you’ll fall. But do you really want to risk doing nothing?

The Other Agile Development: Thoughtworks and Continuous Delivery

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in agile development, agile new product development, and business agility over the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about agile development today.

The Importance of Continuous Delivery to Agile Development

One of the most enjoyable parts of writing about “the other” in general and “the other agile development” in particular is that it allows me to revisit and go more in-depth on cool new technologies. And this is a cool new technology if ever there was one.

Continuous Delivery, as Thoughtworks presents it, aims to develop, upgrade, and evolve software by constant, incremental bug fixes, changes, and addition of features (note: CD is not to be confused with Continuous Integration, which I hope to cover in a future blog post). The example cited is that of Flickr, the photo sharing site, which is using Continuous Delivery to change its production web site at the rate of ten or more changes per day. Continuous Delivery achieves this rate not only by overlapping development of these changes, but also by modularizing them in small chunks that still “add value” to the end user, as well as by shortening the process from idea to deployment to less than a day in many cases.

Continuous Delivery, therefore, is a logical end point of the whole idea of agile development – and, indeed, agile development processes are the way that Thoughtworks and Flickr choose to achieve this end point. Close, constant interaction with customers/end users is in there; so is the idea of changing directions rapidly, either within each feature’s development process or by a follow-on short process that modifies the original. Operations and development, as well as testing and development, are far more intertwined. The shortness of the process allows such efficiencies as “trunk-based development”, in which the process specifically forbids multi-person parallel development “branches” and thus avoids their inevitable communication and collaboration time, which in a short process turns out to be greater than the time saved by parallelization.

Now, let’s take a really broad view of Continuous Delivery. Unfortunately, blog posts are not great at handling graphs, so I’d like you to visualize in your head a graph with two axes, Features and Time. Over time, in each graph, the user’s need for features in a product and solution tends to go up at a fairly steady rate over time, as do consumer needs in general. What varies is how well the vendor(s) supply those needs.

The old model – as old as markets and technology – of what happened was this: somewhere between each version and the next, the disconnect between what the consumer wants and what the product delivers becomes too great, and at that point the vendor starts developing a new version, based on where the consumer is right now (plus a very minor projection into the future which we’ll ignore). For the most part, during this 6 month-2 year development process, the original spec does not change; so for 6 months to 2 years before another version comes out, no or few new features are added – but meanwhile, consumers start looking for new features on top of what they already wanted. The result is a stair-step progression, in which each new version takes the product only partway to meeting the consumer’s needs at that time, and the space between the user line and the vendor line represents lost sales and consumer frustration. However, since every other vendor is doing the same thing, no harm, no foul.

Now consider agile development. Agile development, remember, is about rapid delivery of incremental time-to-value, plus frequent changes to the spec based on end-user input. What that looks like in our graph, more or less, is a stairstep in which each step takes a shorter amount of time, and each “rise” is much closer to the level of user need at that time – but we’re still a little bit behind.

Conceptually, Continuous Delivery takes that idea almost as far as it can be taken. Now, our graph looks more like a squiggly product line overlapping the user need line. And here’s the key point: it actually goes above the user need line, just by a little, frequently.

How can that be, you say? Well, despite the way we disparage technology-driven products compared to need-driven ones, the fact remains that sometimes techies anticipate consumer needs. The typical way is that implicit in the design of the product are future features that the end user, with his or her tunnel vision on immediate frustrations, will never think of. It is the developer who suggests these to the user, not the other way around, or the developer who puts these in the product “for free”, understanding that since they are a logical technical evolution of the design, the user will see them as less strange and simpler to use. This may sound risky to the development manager, but in point of fact this is a minimal-risk kind of customer anticipation, with minimal impact on customer frustration even if it doesn’t pan out, and maximal impact on the consumer’s image of “the brand that anticipates my needs”.

One more variant on the graph: suppose we are talking about New Product Development (NPD) in general. Well, one thing about agile software development is that software is becoming an increasing part of competitive advantage in “hardware” and “services” across most industries. In other words, the development of a new “hardware” or “services” product now typically includes a fair amount of software, whose development is in-house, outsourced, or assigned to packaged-software vendors. In each of these cases, application of agile development processes produces a “mix” between the traditional-graph vendor line and the agile-development one. Visually, the “steps” between “rises” are no longer flat, but broken into little “mini-rises” and “ministeps” that take you a little closer to the user needs line. Continuous Delivery on software that is half of NPD effectively eliminates about half of the lost sales and customer frustration from the traditional approach.

Do you remember that I said: “CD takes the idea almost as far as it can be taken?” Well, the one thing agile development via CD doesn’t handle is a big jump in consumer needs because the consumer wants one of the features that is being developed over in the next county – “disruptive” technology. For instance, Apple really put a hurt on other user-interface vendors when it tweaked touch-screen technology for the iPhone. However, the software in other cell-phone and computer products at least allowed a more rapid partial response to the threat. So CD isn’t a cure-all – it just comes amazingly close to it. How to “go the final mile” is a discussion for a future blog post.

Let’s summarize: CD is an incredibly cool and incredibly useful technology, both to the vendor and to the consumer, because it results in a major increase in sales for the vendor and a major increase in satisfaction for the consumer. Moreover, because it’s also cheaper than traditional software development, both vendors and consumers see their costs decrease as the vendor’s use of CD in NPD rises (and for the typical business, that’s in addition to the cost savings and competitive advantage from more rapid development of better business-process software). Finally, because their needs are both satisfied and anticipated, customers become far more loyal, reducing business risk drastically.

Really minor nit: I note that CD is often applied strictly to the delivery stage of development. To my mind, extension to the entire process is appropriate, because delivery is the only major development stage (if we exclude operations after delivery) where the agile development methodology today is often applied minimally. In other words, “delivery” can mean one stage or the whole process, and in the real world if you make that one stage agile you are usually making the whole process agile – so it’s a good idea to emphasize the point of the whole exercise by equating continuous completion and continuous delivery.

The Relevance of Thoughtworks to Continuous Delivery

Thoughtworks is one of those smaller consultancies that took the Agile Manifesto’s ideas and ran with them while large development-tool vendors focused on other, ultimately far less effective, “best practices.” I have had my differences with Thoughtworks in the past, but always from a position of respect for an organization that clearly has agile development in its DNA to a greater extent than most. In the case of CD, a quick scan of the first page of a Google search on Continuous Delivery reveals no one else visibly applying it to the extent that Thoughtworks claims to be doing.

Does that mean that Thoughtworks is a stable vendor for the long term? One of the fascinating things about the agile-development market is that the question matters to a much lesser extent than in all previous markets. Look, in the early years some tried SCRUM and some tried extreme programming and some tried half-way solutions like slightly modified spiral programming, but it didn’t matter in the long run: the Wipros of the world have still done almost universally better than the startups focusing on Java or even folks like Cambridge Technology Partners. And that’s because agile development firms are, well, agile. It doesn’t just rub off on developers; some of it rubs off on managers, and strategists, and even, Ghu help us, on CEOs. They focus on changes; so, on average, they evolve more effectively. That’s as true of Thoughtworks as anyone else.

What isn’t true of most other agile development vendors, right now, is that Thoughtworks appears to have a significant edge in experience in the CD extension of agile development. That matters because, as I’ve said, something like Thoughtworks-type CD is the logical endpoint of agile development. So, if you want to get to maximal agile-development benefits sooner rather than later, it certainly seems as if Thoughtworks should be on your short list.

One point here requires elaboration: there is sometimes a misconception that outside providers are selling you agile-development services. That’s at least partially wrong. They are – or should be – fundamentally selling you agile-development training. They will make their money from being ahead of you in experience, and constantly selling you the improvements in agile development that their experience teaches them. Think of them as more like business-strategy consultants, always looking ahead to the new strategic idea and delivering that to you. Believe me, that’s just as valuable as running your data center – and is often more valuable than that.

Thus, Thoughtworks’ advantage in experience is not to be sneezed at. How well it will hold up over time, we will see. However, considering that the bulk of software development, even if it has adopted a modified version of SCRUM en masse, is still typically not very successful in embedding frequent user feedback into the typical project, I would say that Thoughtworks’ edge should last for at least a couple of years – an eon in the timeframe of the agile business.

Potential Uses of Thoughtworks-Type Agile Development for IT

An IT organization must crawl before it can walk; but it should also learn about walking before it tries to do so. That means that if IT is still at the early stages of adopting agile development, it should still apply CD to a “skunkworks” project, and if it doesn’t have such a project, it should create one.

Otherwise, this is not a “targets of opportunity” situation, but rather a “learn and merge” one. IT should bring CD on board as each project is ready for it, no faster, no slower. In my opinion, “ready” means that a project’s development process has (a) adequate process management tools specifically tuned to support agile development, thus allowing it to scale, (b) an adequate “store” of reusable infrastructure software to build on, so that moving to the next incremental feature is not too great a leap, and (c) an attitude from everyone involved that the first thing you do when you get something new is that you make it agile. That’s all. Print and ship.

Well, but today’s CD isn’t adapted to the peculiar needs of my development. Excuse me? You did say your process was agile, didn’t you? Believe me, CD is not only flexible but agile, and if you don’t know the difference, then you need far more help in achieving agility than you realize. What you’re really saying is that you don’t have an agile development process at all, because otherwise it would be straightforward to adapt your methodology to move steadily towards CD – using a vendor just speeds up the process change.

The Bottom Line for IT Buyers

I am always wary of being too enthusiastic about new technologies, because I remember a story in Gore Vidal’s Lincoln. A rich man has a tendency to exaggerate, and hires someone to nudge him at table when he does so. “How was your trip to Egypt?” “Amazing! Why, they have things called pyramids made of pure gold!” Nudge. “And they go a mile high!” Hard stamp on his foot, just as someone asks, “How wide?” He answers, in agony, “About a foot.” I worry that while any given technology’s value may seem a mile high, its real-world application will be about a foot wide.

That said, I really do see Continuous Delivery as a cool new technology that will have an impact, eventually, that will be a mile high and globe-wide. Here’s what I said in a previous post:

[[The real-world success of Continuous Delivery, I assert, signals a Third Age, in which software development is not only fast in aggregate, but also fast in unitary terms – so fast as to make the process of upgrade of a unitary application by feature additions and changes seem “continuous”. Because of the Second Age, software is now pervasive in products and services. Add the new capabilities, and all software-infused products/services -- all products/services – start changing constantly, to the point where we start viewing continuous product change as natural. Products and services that are fundamentally dynamic, not successions of static versions, are a fundamental, massive change to the global economy.

But it goes even further. These Continuous-Delivery product changes also more closely track changes in end user needs. They also increase the chances of success of introductions of the “new, new thing” in technology that are vital to a thriving, growing global economy, because those introductions are based on an understanding of end user needs at this precise moment in time, not two years ago. According to my definition of agility – rapid, effective reactive and proactive changes – they make products and services truly agile. The new world of Continuous Delivery is not just an almost completely dynamic world. It is an almost Agile World. The only un-agile parts are the rest of the company processes besides software development that continue, behind the scenes of rapidly changing products, to patch up fundamentally un-agile approaches in the same old ways.]]

But you don’t need to know about that. What you need to know is that for every IT organization that appreciates agile development, kicking the tires or adopting CD is a good idea, right now. I don’t even have to say it’s necessary, because that’s not the way an agile organization operates.

As for Thoughtworks, here’s what I think IT’s attitude should be. I have often trashed Sun for an ad that said “We built the Internet. Let us build your Internet.” I knew, based on personal experience, that this claim was, to say the least, exaggerated. Well, if Thoughtworks came to your door with a pitch that said “We built Continuous Delivery. Let us build your Continuous Delivery,” I would not only not trash them, I would encourage you to believe them, and consider doing as they request. The pyramid is that high. The materials with which it is built are that valuable. And my foot is still intact.

The Other BI: Oracle TimesTen and In-Memory-Database Streaming BI

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about BI today.

The Importance of TimesTen-Type In-Memory Database Technology to BI

All right, now I’m really stretching the definition of “other”. Let’s face it, Oracle is “top of the mind” when we talk about BI, and they recently announced a TimesTen appliance, so TimesTen is not an invisible product, either. And finally, the hoopla about SAP HANA means that in-memory database technology itself is probably presently pretty close to the center of IT’s radar screen.

So why do I think Oracle’s TimesTen is in some sense not “top of the mind”? Answer: because there are potential applications of in-memory databases in BI for which the technology itself, much less any vendor’s in-memory database solution, is not a visible presence. In particular, I am talking about in-memory streaming databases.

To understand the relevance of in-memory databases to complex event processing and BI, let’s review the present use cases of in-memory databases. Originally, in-memory technology was just the thing for analyzing medium-scale amounts of financial-market information in real time, information such as constantly changing stock prices. Lately, in-memory databases have added two more BI duties: (a) serving as a “cache” database for enterprise databases, to speed up massive BI where smaller chunks of data could be localized, and (b) serving as a really-high-performance platform for mission-critical small-to-medium-scale BI applications that require less scaling year-to-year, such as some SMB reporting. These new tasks have arrived because rapid growth in main-memory storage has inevitably allowed in-memory databases to tackle a greater share of existing IT data-processing needs. To put it another way, when you have an application that is always going to require 100 GB of storage, sooner or later it makes sense to use an in-memory database and drop the old disk-based one, because in-memory database performance will typically be up to 10-100 times faster.

Now let’s consider event-processing or “streaming” databases. Their main constraint today in many cases is how much historical context they can access in real-time in order to deepen their analysis of incoming data before they have to make a routing or alerting decision. If that data can be accessed in main memory instead of disk, effectively up to 10-100 times the amount of “context” information can be brought to bear in the analysis in the same amount of time.

In other words, for streaming BI, IT potentially has two choices – a traditional event-processing database that is often entirely separate from a back-end disk-based database, or (2) a traditional main-memory database already pre-optimized for in-depth main-memory analysis and usually pre-integrated with a disk-based database (as TimesTen is with Oracle Database) as a “cache database” in cases where disk must be accessed. How to choose between the two? Well, if you don’t need much historical context for analysis, the event-processing database probably has the edge – but if you’re looking to upgrade your streaming BI, that’s not likely to be the case. In other cases, such as those where the processing is “routing-light” and “analysis-heavy”, an in-memory database not yet optimized for routing but far more optimized for in-depth analytics performance would seem to make more sense.

Thus, one way of looking at the use case of in-memory database event processing is to distinguish between in-enterprise and extra-enterprise data streams (more or less). Big Data is an example of an extra-enterprise stream, and can involve a fire hose of “sensor-driven Web” (GPS) and social media data that needs routing and alerting as much as it needs analytics. Business-critical-application-destined and embedded-analytics data streams are an example of in-enterprise data, even if admixed with a little extra-enterprise data; they require heavier-duty cross-analysis of smaller data streams. For these, the in-memory database’s deeper analysis before a split-second decision is made is probably worth its weight in gold, as it is in the traditional financial in-memory-database use case.

Won’t having two databases carrying out the general task of handling streaming data complicate the enterprise architecture? Not really. Past experience shows us that using multiple databases for finer-grained performance optimization actually decreases administrative costs, since the second database, at least, is typically much more “near-lights-out,” while switching between databases doesn’t affect users at all, because a database is infrastructure software that presents the same standard SQL-derivative interfaces no matter what the variant. And, of course, the boundary between event-processing database use cases and in-memory ones is flexible, allowing new ways of evolving performance optimization as user needs change.

The Relevance of Oracle TimesTen to Streaming BI

In many ways, TimesTen is the granddaddy of in-memory databases, a solution that I have been following for fifteen years. It therefore has leadership status in in-memory database use-case experience, and especially in the financial-industry stock-market-data applications that resemble my streaming-BI use case as described above. What Oracle has added since the acquisition is database-cache implementation and experience, especially integrated with Oracle Database. At the same time, TimesTen remains separable at need from other Oracle database products, as in the new TimesTen Appliance.

These characteristics make TimesTen a prime contender for the potential in-memory streaming BI market. Where SAP HANA is a work in progress, and approaches like Volt are perhaps less well integrated with enterprise databases, TimeTen and IBM’s solidDB stand out as combining both in-memory original design and database-cache experience – and of these two, TimesTen has the longer in-memory-database pedigree.

It may seem odd of me to say nice things about Oracle TimesTen, after recent events have raised questions in my mind about Oracle BI pricing, long-term hardware growth path, and possible over-reliance on appliances. However, inherently an in-memory database is much less expensive than an enterprise database. Thus, users appear to have full flexibility to use TimesTen separately from other Oracle solutions, free from worries about possible long-term effects of vendor lock-in.

Potential Uses of TimesTen-Type In-Memory Streaming BI for IT

As noted above, the obvious IT use cases for TimesTen-type streaming BI lie in driving deeper analysis in in-enterprise streaming applications. In particular, in the embedded-analytics area, in-memory performance speedups can allow consideration of a wider array of systems-management data in fine-tuning downtime-threat and performance-slowdown detection. In the real-time analytics area, an in-memory database might be of particular use in avoiding “over-steering”, as when predictable variations in inventories cause overstocking because of lack of historical context. In the Big Data area, an in-memory database might apply where the data has been pre-winnowed to certain customers, and a deeper analysis of those customers fine-tunes an ad campaign. For example, within a half-hour of the end of the game, Dick’s Sporting Goods had sent me an offer of a Patriots’ AFC Championship T-shirt, complete with visualization of the actual T-shirt – a reasonably well-targeted email. That’s something that’s far easier to do with an in-memory database.

IT should also consider the likely evolution of both event-processing and in-memory databases over the next few years, as their capabilities will likely become more similar. Here, the point is that event-processing databases often started out not with data-management tools, but with file-management ones – making them significantly less optimized “from the ground up” for analysis of data in main memory. Still, event-processing databases such as Progress Apama may retain their event-handling, routing, and alerting advantages, and thus the situation in which in-memory is better for in-enterprise and event-processing is better for extra-enterprise is likely to continue. In the meanwhile, increasing use of in-memory databases for the older use cases cited above means that in-memory streaming-BI databases offer an excellent way of gaining experience in their use, before they become ubiquitous. That, in turn, means that narrow initial “targets of opportunity” in one of the situations cited in the previous paragraph are a good idea, whatever the scope of one’s overall in-memory database commitment right now.

The Bottom Line for IT Buyers

In some ways, this is the least urgent and most speculative of the “other BI” solutions I have discussed so far. We are, after all, discussing additional performance and deeper analytics in a particular subset of IT’s needs, and in an area where the technology of in-memory databases and their event-processor alternatives is moving ahead rapidly. In a sense, this is really an opportunity for those IT shops that specialize in applying a little extra effort and “designing smarter” across multiple new technologies to provide a nice ongoing competitive advantage. For the rest, if the shoe can easily be made to fit, why not wear it?

My suggestion for most IT buyers, therefore, is therefore to have a “back-pocket” in-memory-database-for-streaming-BI short list that can be whipped out at the appropriate time. Imho, Oracle TimesTen right now should be on that list.

I hate to close without noting the overall long-term BI potential of in-memory databases. The future of in-memory databases is not, in my firm opinion, to supersede the IBM DB2s, Oracle Databases, and Microsoft SQL Servers of the world, at any time in the next four years. The hardware technologies to enable such a thing are not yet clear, much less competitive. Rather, the value of in-memory databases is to allow us to optimize our querying for both main-memory and disk storage – which are two very different things, and which will both apply appropriately to many key customer needs over the next few years. Overall, the effect will be another major ongoing jump in data-processing performance. As we enter this new database-technology era, those who initially kick the tires in a wider variety of BI projects will find themselves with a significant “experience” advantage over the rest, especially because the key to outstanding success will be determining the appropriate boundary between disk-based and in-memory database usage. Don’t force in-memory streaming BI into the organization. Do keep checking to see if it will fit your immediate needs. Sooner or later, it probably will.