Recently, I saw a blog post by an author with training in economics, L.E. Modesitt, noting that many female students who sought to make a career out of being an opera singer – something that is increasingly very unlikely for anyone – did not take actions necessary to maximize their chances to do so, and expressing bewilderment at their irrationality. I have seen this kind of analysis frequently in the past – it is, afaik, an accepted part of our beliefs. And yet, there appears to me to be a strong case that certain ways of taking long-shot chances – in one way or another, playing the lottery – are rational, are beneficial for the person, and are beneficial for the economy.
At this point, I have to stop and emphasize that, as I note in the title, this is no way, shape, or form and encouragement of anybody to play the lottery in any way. Not until it is very clear how people in general can avoid gambling addiction as a result of playing the lottery. No, this piece is aimed specifically and only at the non-addicts analyzing this behavior, hopefully so that they will see the rationality of certain types of playing the lottery and shape our policy approach to it accordingly.
At the same time, I have to note that this type of rationality, it seems clear, can be achieved. Once I realized the rationality of certain lottery-playing, I practiced it myself, over 15 years. Once the conditions no longer applied, I stopped immediately and completely, and haven’t played the lottery since, over the last 4 years. Although I didn’t win, the net negative effect on my net worth today was approximately 0.003% -- and if you think that matters, I have a bridge in Brooklyn I’d like to sell you. So it is doable.
Setting the Stage: The Players
First, let me give you a very crude picture of those who could potentially play the lottery, as I see it, in our society. There are four levels: the rich, the about-to-be-rich, the middle class, and the zero/negative net worth. Approximately, these break out as follows: the rich 0.3% of us; the about-to-be-rich another 0.3%; the zero/negative net worth, according to recent figures, 40% (50% of blacks and, I think, probably Hispanics); the middle class the remaining 59.4%. I am, I believe, probably underestimating the proportion of zero/negative net worth and overestimating the proportion of middle class; but those are the best figures I have.
Now, understand the nature of these four levels. The rich I define as those with about $2.5 million or more in investment wealth, stocks and bonds. According to my analysis, it is extremely hard for them to avoid becoming richer over time. Therefore, they have no need to play the lottery; fundamentally, they are secure and able to buy almost all of what they want immediately, forever. The about-to-be-rich are mostly CEOs; they have effectively guaranteed the income that will land them in the rich within approximately 1-6 years. Therefore, they have no need to play the lottery, either. Playing the lottery, up to extreme limits (such as thinking you can be a poker champion in Vegas), for these people, is not rational or irrational; it’s just irrelevant to their ultimate well-being.
Now we come to the middle class. By this, I don’t mean the traditional definition of middle class. I mean those who have every incentive to save rather than spend immediately in order to reach partial income security once they reach retirement, enough under most circumstances to allow them to live out their lives with prudent money management, but not enough to make them rich and without worries. These would play the lottery, rationally, for one fundamental reason: it is a choice between never getting rich and getting rich right now. It seems to me, based on the numbers, that there is a vanishingly small chance that any of these will get rich without taking a long-shot chance; and a lot better chance than that, if you take that long-shot chance. If, therefore, the accumulation of long-shot chance taking does not cost you a significant proportion of your ultimate net worth, then it seems to me to make a good deal of sense.
Finally, there are those with zero or negative net worth. I feel – I don’t know if it is justified by the facts – that their rationality is vastly underestimated. It seems to me likely that if there were reasonable chances of getting into the middle class without placing yourself under such a crushing psychological burden (e.g., working two full-time jobs as a maid plus bringing up children) that few can be expected to bear it, that these zero/negative net worth folks would take it. Instead, they tend to spend the little money they get immediately, often on relieving psychological pressure: smoking, drinking, betting, etc. Unfortunately, those things tend to lead to addiction – and the lottery is very much of the same ilk. Still, it is on the face of it no better or worse than any other addiction; and the potential payoff, it appears to me from a very cursory study of winners, is to place at least half of those winners firmly in the middle class. A major percentage of lottery winners do start to show saving-instead-of-spending-immediately behavior and do keep positive net worth until retirement. In other words, for this level, playing the lottery in ways that maximize your chances of getting into the middle class forever is entirely rational.
At this point, I have to pause to make sure that people understand fully what I mean by playing the lottery. Seeking out a job in a start-up company is playing the lottery, typically done by middle-class types. The number of those who get rich out of doing so is, sorry, far less than 1% -- and the hit to one’s ultimate net worth is likely to be significant. However, in most cases, it will not kick you down to the next level – you just wind up at a lower pay scale in a “safe” job.
Likewise, for the top of the middle class seeking to become rich, sucking up to the rich or the about-to-be-rich or taking a flyer on company stock options is also a form of playing the lottery. In most cases, the lion’s share of the stock/stock-option profits will be taken by the about-to-be-rich; but if an IPO is a wild success, you may get rich, or very close. It’s a lottery. Likewise, everyone is trying to suck up to the rich and vacuum up their money in order to get rich themselves, so in all likelihood you won’t get enough of his or her time to succeed – not to mention having the rich person fire you if you fail. Still, it’s worth a try. It’s a lottery.
In effect, the real lottery (or trying to break into show business, or win a TV show) is the only realistic lottery available to the lower middle class and zero/negative net worth folks. You simply aren’t likely to get in sniffing distance of a rich person. Your little stock flyer won’t make that much. Nope, it’s the real lottery or nothing.
Addiction and Rationality
Hopefully, the above analysis will allow you the reader to guess most of what I’m going to say here. The object of the rational lottery play, at whatever level except rich, is to move up one level, or move up far sooner than you otherwise would. To maximize the chances of that happening, you have to maximize the odds of success and establish a reasonable lower limit on negative consequences. And one more thing: you have to minimize the chances of addiction.
It may seem that simply setting a maximum amount to be spent per year on a lottery is sufficient to avoid addiction. In fact, it is not. Any time you play the lottery, in my experience, you wind up with an incredible swing in emotions, hovering near a high between the moment you place your bet and the time when the results are down to an incredible low immediately after – all of which lead to a strong desire to experience the high again, immediately.
That’s an addiction. The best way to avoid it, I believe, is to minimize the number of times you bet, and to avoid reminders that the bet is on until the results are announced. In effect, you make a bet once a year, and then get on with your life. And if you turn that approach into a habit, the habit itself will help prevent addiction, as any deviation takes undue effort. As I said, I know it can be done; and that was pretty much the way I did it. Again, I emphasize: that doesn’t mean I’m right. Let’s wait to do more analysis, and explore more ways. Don’t try this on your own.
First additional consideration: the amount for first prize in the lottery has to be enough to effectively move you up a level. Even $100,000 won’t cut it. My own rule-of-thumb cutoff back then was somewhere between $1 and 2 million dollars, since, after taxes and net present value reduction from installment payments, that would wind up being $600,000 to $1.2 million.
Second, one of the most effective ways to reduce the odds is to choose a lottery in which the maximum prize is the only prize. Other prizes simply reduce your odds of getting the maximum prize.
My wild guesstimate of the chance of success for lower middle class types and below over a lifetime is somewhere around 1 in 10,000. For upper middle class types who have more access to rich people and more reserve cash to allow playing lower-odds lotteries, I’d guess it’s 1 in 100. Either way, if you do it right, the alternative to winning is simply no real change in your net worth, so it’s always rational to do this.
From an Economic Point of View
It also seems to me that this kind of lottery-playing is beneficial from an economic point of view, if it does not increase gambling addiction. Thus, if it does not increase the net debt in the bottom level, and does not drive middle class participants into the bottom level, what we have is a significant increase in the assurance that middle class types will not need assistance in retirement, while preserving in most cases their incentive to defer spending partially in order to save – effectively balancing investment and spending. We also have an increase in the spending and investment of some previously zero/negative net worth individuals – granted, only 0.01% of that 40%, but still a nice chunk of change to add to the economy.
One concern is the constraints on the entity running the lottery. Government lotteries, as have often been pointed out, have an unfortunate incentive to create addiction in zero/negative net worth individuals, which, will it does not necessarily make their lives dramatically worse off – it’s just one more straw on the camel’s back – do mean that these individuals get too far in debt, and that decreases their spending on new stuff rather than paying off debt, one way or another – so it hurts the overall economy until you can dig these folks out of their debt that you created.
Large-business lotteries, by contrast, by their nature tend to be less frequent and focused more on the middle class, which can more easily afford to spend on their stuff. So these tend to be relatively harmless – which is not at all to say that a gambling casino is harmless, since it short-sightedly maximizes profit by maximizing addiction, only that a McDonald’s or AmEx lottery game isn’t too bad.
And finally, we may note that lotteries would do themselves and their clientele a favor by sticking to multi-million-dollar prizes – except gambling casinos, which are more often selling entertainment rather than getting rich, anyway.
Summary and Conclusions
Hopefully, this will be short and sweet. I believe I may have found the beginnings of a way to encourage the bottom two levels, acting rationally in their own self-interest, to use lotteries to possibly better themselves and the economy. Thoughts, anyone?
Tuesday, May 8, 2012
Tuesday, May 1, 2012
A Little Tolkien Music
Over the years, among other things, I have jotted down some notes about interesting writing techniques that Tolkien uses – techniques that, imho, make him a Great Writer worth learning from. I thought it would be fun to share an example of one of these with you here.
One of these of these techniques is the way he approaches his descriptions. He specifically avoids similes and metaphors. Instead, he tries to put us inside the mind (or see from the outside in a similar way) the thing described, as if it were a living, feeling, thinking being like ourselves. Then he suits the language to that – no “clinical” modern words, and the colors are primary ones.
Recently, I ran across an example of another author trying to write the same sort of idea in a fantasy novel (David Weber, War Maid’s Choice – and he’s a pretty good author as authors go): “It was as if her nerves were connected directly to the trunks of the apple trees, as if she could feel them yearning towards fruit, tossing their branches like widespread fingers to the caress of the wind.”
Let’s break down where that goes wrong according to my idea of Tolkien. “It was as if her nerves were connected to the trunks” – no. For such a person, it would be “Her nerves felt the life of the apple tree.” Now we can see things from the point of view of that tree.
“[As] if she could feel them yearning toward fruit.” Excuse me? Yearning means horny. A fruit is the tree’s baby. Does anyone really feel that the physical process of bearing a baby makes a woman more horny? You may feel horny towards someone, apparently unrelated to having a baby, and underlying that is the desire to start the process of having a baby; but the physical process? No way, Jose. Make it about having a baby, and don’t say “as if”. “It swelled towards birth of tiny fruits with pain and love.” If you want to really be the tree, make it “It swelled towards birth of tiny fruits with loving, careful inward regard.” Now add the Tolkien touch: “It grew towards opening to the world small, deep green buds of golden morning fruit.” Morning in this case is standing for the fresh eyes of a newborn or young child. Golden is in the golden sunlight.
“tossing their branches like widespread fingers to the caress of the wind.” Tossing one’s hair is the way a person outside perceives it. Shaking one’s hair is the way the person doing it sees it. These aren’t branches or fingers to the tree, they’re hair; say so. Suddenly we switch our viewpoint to another, entirely different alive personification: the wind. Why? Stick with the tree. “Caress” – that’s the view of an outsider. What does the tree feel (or the wind)? A hug. Except that that isn’t the best description; it’s more like being fanned by someone inattentive. OK, this translates to “It felt the wind disarranging its branches with a cool breeze, a playfully fanning servant.” Tolkien touch: “It felt the keen wind ruffling its gray-green crown of branches, combing them to playful disarray.”
Now feel it. I am a tree. I lie at ease on my bed, and look at the tiny life growing in me, and imagine it bursting forth into the world, and being young and innocent and beautiful. And beside me, my silly husband is fanning me so that I feel cool in the warmth of early summer, and the breeze is making my hair become a nice casual mess. And it feels profoundly wonderful, as if I should save this moment in memory forever.
“It was as if her nerves were connected directly to the trunks of the apple trees, as if she could feel them yearning towards fruit, tossing their branches like widespread fingers to the caress of the wind."
"She felt the life of the apple tree. It grew towards opening to the world small, bright green buds of golden morning fruit; and it felt the keen wind ruffling its gray-green crown of branches, combing them to playful disarray, as it lay at ease in its bower of deep blue lilacs."
Up to you. Me, I prefer the Tolkien.
One of these of these techniques is the way he approaches his descriptions. He specifically avoids similes and metaphors. Instead, he tries to put us inside the mind (or see from the outside in a similar way) the thing described, as if it were a living, feeling, thinking being like ourselves. Then he suits the language to that – no “clinical” modern words, and the colors are primary ones.
Recently, I ran across an example of another author trying to write the same sort of idea in a fantasy novel (David Weber, War Maid’s Choice – and he’s a pretty good author as authors go): “It was as if her nerves were connected directly to the trunks of the apple trees, as if she could feel them yearning towards fruit, tossing their branches like widespread fingers to the caress of the wind.”
Let’s break down where that goes wrong according to my idea of Tolkien. “It was as if her nerves were connected to the trunks” – no. For such a person, it would be “Her nerves felt the life of the apple tree.” Now we can see things from the point of view of that tree.
“[As] if she could feel them yearning toward fruit.” Excuse me? Yearning means horny. A fruit is the tree’s baby. Does anyone really feel that the physical process of bearing a baby makes a woman more horny? You may feel horny towards someone, apparently unrelated to having a baby, and underlying that is the desire to start the process of having a baby; but the physical process? No way, Jose. Make it about having a baby, and don’t say “as if”. “It swelled towards birth of tiny fruits with pain and love.” If you want to really be the tree, make it “It swelled towards birth of tiny fruits with loving, careful inward regard.” Now add the Tolkien touch: “It grew towards opening to the world small, deep green buds of golden morning fruit.” Morning in this case is standing for the fresh eyes of a newborn or young child. Golden is in the golden sunlight.
“tossing their branches like widespread fingers to the caress of the wind.” Tossing one’s hair is the way a person outside perceives it. Shaking one’s hair is the way the person doing it sees it. These aren’t branches or fingers to the tree, they’re hair; say so. Suddenly we switch our viewpoint to another, entirely different alive personification: the wind. Why? Stick with the tree. “Caress” – that’s the view of an outsider. What does the tree feel (or the wind)? A hug. Except that that isn’t the best description; it’s more like being fanned by someone inattentive. OK, this translates to “It felt the wind disarranging its branches with a cool breeze, a playfully fanning servant.” Tolkien touch: “It felt the keen wind ruffling its gray-green crown of branches, combing them to playful disarray.”
Now feel it. I am a tree. I lie at ease on my bed, and look at the tiny life growing in me, and imagine it bursting forth into the world, and being young and innocent and beautiful. And beside me, my silly husband is fanning me so that I feel cool in the warmth of early summer, and the breeze is making my hair become a nice casual mess. And it feels profoundly wonderful, as if I should save this moment in memory forever.
“It was as if her nerves were connected directly to the trunks of the apple trees, as if she could feel them yearning towards fruit, tossing their branches like widespread fingers to the caress of the wind."
"She felt the life of the apple tree. It grew towards opening to the world small, bright green buds of golden morning fruit; and it felt the keen wind ruffling its gray-green crown of branches, combing them to playful disarray, as it lay at ease in its bower of deep blue lilacs."
Up to you. Me, I prefer the Tolkien.
Thursday, April 12, 2012
A Draft Agile Business Manifesto (& Agile Marketing Manifesto)
More than a decade ago, the experience of several far-seeing developers drove an Agile Development Manifesto that has more than stood the test of time. Now, finally, other areas of the business are considering equivalent follow-on manifestoes of their own – in particular, an Agile Marketing Manifesto. In order to insure that the fundamental insights of the original Agile Manifesto are not lost, I would like to propose a similar set of principles for the agile business in general – principles within which efforts such as an agile marketing manifesto should be expected to fit.
Because “the old ways” are much more entrenched in the business as a whole than in the software development arm of an organization, I set these principles up in opposition to what I view as some of the key principles of business as we have known it – or at least the ones that the truly agile business contradicts. And, of course, I have a bit of fun in doing so.
I also believe that these principles will also serve for a draft agile marketing manifesto. The reasons why will become apparent later.
Agile Organization Goals: Fast, Effective Change
The old principle 1 (monetary goal): In the long run, the most successful are most likely to survive.
Agile principle 1 (fast, effective change goal): The successful thrive; the agile survive. In the long run, survive beats thrive.
Remember that in an agile organization, the focus is on maximizing the speed and effectiveness of change, not on maximizing short-run or long-run success as defined in monetary terms. Experience shows that in many development organizations, agile teams focused on speed of customer-coordinated change rather than cutting costs, improving quality, or speeding up development do better in the short and long run in monetary terms (costs, revenues, risk profile). This principle asserts that even if an organization is more successful in the short run by focusing on maximizing profit, it is less likely to survive in the long run, and less successful if it does survive, than the truly agile organization.
Agile Coopetition
The old principle 2: What does not kill you makes you stronger.
Agile principle 2 (coopetition): What tries to kill us and fails makes us weaker and more agile – and that’s good.
Recent research suggests that humans benefit most from eating foods that give us “moderate poisoning.” That is, they do not kill us, but they do weaken some existing systems. In order to adapt to these weaknesses, the body grows new capabilities – physical and mental – that let us handle a wider range of situations – that make us “survive by fitting” a wider range of environments. By contrast, foods with no poisoning capabilities offer much less benefit, and foods that cripple us offer no benefit at all.
There is a reasonable analogy here with competition vs. what is now called coopetition. The non-agile business assumes that one “wins” in the long term by crushing the competition, and that therefore the company emerges from the process stronger. This is doubtful; in the long-run, over-success in competition leads to monopoly behavior that decreases innovation and makes “crossing the chasm” much more difficult.
The agile business seeks to create “safe” competition by combining competition with cooperation – the same business that competes with you in one area cooperates with you in another area, and neither side expects to win overall in the long run. Rather, the relative weakness in one area of one of the two vendors will lead either to new capabilities or to redefinition of its focus on that side, using the increasing weakness of a particular area as an indicator of the need to move to something different. In the short run, that competition makes the agile vendor weaker. In the long run, the safety of that “competition” leads to increased agility, which leads to long-term survival and success. Meanwhile, cooperation in another area expands the solutions that the firm can offer and provides additional customer feedback.
Agile Culture: Happiness is Customer-Coordinated Change
The old principle 3: Greed is good.
Agile principle 3: Greed is bad. Customer-coordinated change is good.
Immortalized as the “invisible hand”, the pure form of the old principle 3 asserted that focusing on maximizing profit was good for the individual employee, good for the company, good for the consumer, and ultimately good for the economy and society. The agile principle asserts that focusing on speed and effectiveness of “safe” change is better for the individual employee, better for the consumer, and ultimately better for the economy and society in both short-term and long-term monetary and “social welfare” terms.
This is not a simple matter of “doing well by doing good.” Rather, the agile business recognizes that effective change must be constantly and firmly grounded in consumer reality, and that treating the consumer as an infinitely malleable toy to be squeezed of every last drop of money has bad long-term effects not only on survivability but also on corporate culture and thence on corporate success and “social welfare.”
Agile Strategy: Focus on Who
The old principle 4: If you don’t know where you’re going, any road will take you there.
Agile principle 4: If you don’t know who you will meet, you don’t know where you’re going.
As shown in such works as “Marketing Myopia”, the focus of a traditional firm is to accept the given of a global mass “market” as it is now, and try to determine just what market within that global one fits the firm best, and how the firm can steadily increase its profits within and by extensions of that market. The agile strategist focuses on a global market full of individual consumers with common lifecycles and fine-grained differences, and seeks to capture as much as possible of the lifecycle and time allotment of the largest group of such consumers as their needs evolve over time. When the agile firm talks about “continuous delivery” of what the customer wants, that is what it means. The traditional firm focuses on the visible, static market of an undifferentiated mass of needs; the agile firm focuses on understanding and meeting the inevitably changing individual needs of each of a class of individual consumers.
The agile firm believes that such a strategy gives it two key advantages over the traditional firm: it grounds the agile firm in reality rather than its own wishes about what the customer “should” be like in order to maximize firm profit, and it focuses the agile firm on striving toward the most effective kind of long-run strategy: cradle-to-grave, comprehensive, open-ended customer support.
Agile Target Market: Learners
The old principle 5: People want solutions
The new principle 5: People buy emotions
Agile principle 5: People buy learning
This agile principle can be seen as a corrective to the extremes of the old and new principles. It is true that in the short run, people will go out and buy a solution for an immediate problem, and that given extra money with which to buy things, people will then choose a solution that makes them feel better, or even an “emotional trigger” without any usefulness at all. However, in the long run, people usually continue buying in a particular area because it continues to relate to the ongoing reality of their lifecycle. Moreover, the best kind of customer buys because he or she enjoys the open-ended pleasure of constantly learning more in a “safe” (usually successful) way.
The alternative customer is the addict: one that develops such a need for an offering that the addict is willing to continue to spend on it forever, as it becomes less and less useful. As industries like tobacco and alcohol have shown, the addict can be a successful short-term market. However, firms in such industries tend to be unusually un-agile – reflecting their customers. Moreover, these industries react to the inevitable challenges of evolving customer needs by attempting to “game the system” and prevent these changes from happening – increasing the costs to society as a whole, as well as the impact to the firm of the inevitable failure to prevent change. The tobacco and alcohol industries are among the least successful over the last 60 years. By contrast, the computer software industry, which ranks among the most agile, is close to the most successful, and software companies that focus on the consumer rather than the business more successful still (see agile principle 4).
There is another implication of agile principle 5: the agile business should never simply react to the customer, but rather constantly alternate getting ahead of the customer and reacting to the customer. Helping a person to learn forever means constantly generating new, follow-on, useful things for the customer to learn, as the customer moves through the stages of a life. The customer, in return, tells you which of these innovations is desired and what his or her own innovations are.
This Is Also A Draft Agile Marketing Manifesto
The reason that these principles apply verbatim to agile marketing, I believe, is because it is probable that in the agile business, marketing will become a “loosely-coupled CEO”. That is, agile marketing’s primary job will be to integrate the constant iteration of new products into a long-term, effective business strategy.
This is not to say that this is marketing’s present job, nor that those trying to achieve agile marketing do integration very well right now. Rather, it says that marketing is the only existing business function that can carry out this necessary task effectively in the agile organization. The other functions are too focused on particular parts of the overall business’ agile process. The CEO simply does not have the capability of handling all the integrating tasks that must be done.
Rather, individual marketers, empowered by their unique focus on long-term customer need evolution, must connect and coordinate the enthusiastically wandering dogs of the other agile business functions. Meanwhile, the CEO imposes a single long-term personality on the agile marketers’ work, giving the results a touch of individuality that appeals to the individual consumer – much the same role that a great soloist plays in giving an old war-horse of a classic musical piece new life, or the kind of role that Steve Jobs occasionally played toward the end.
Thus, agile marketing in the long run will be effectively carrying out the agile organization’s principles – but in an agile way, in which the customers are equally the other agile business functions and the firm’s actual customers.
The Long Road Ahead
I say this is a draft manifesto; but, to me, it is also a platform with which to challenge other upcoming manifestos. I believe that if another manifesto contradicts or ignores these principles, there may very well be fundamental problems with that manifesto. If so, these principles give me (and, I hope, others) a basis for criticizing and, if necessary, rejecting such a manifesto. We are trying to create manifestoes at a much earlier – dangerously early – stage of our understanding of business agility. It may be OK not to get a manifesto right; but it is very important not to get it irretrievably wrong.
This is why I am so dogmatic about what goes into an agile manifesto. If the experience of agile software development and its alternatives teaches us anything, it is that if you think agile, your process becomes more and more agile and more and more successful; while if you don’t, it doesn’t. There doesn’t seem to be much of a middle ground. And it can be hard to wrap your mind around such a different way of thinking, and therefore easy to backslide subtly but permanently.
I will try in the next few weeks to flesh out the agile marketing “thought experiment” on which these suggested principles are based. In the meanwhile, enjoy – or not.
Agile principle 1 (fast, effective change goal): The successful thrive; the agile survive. In the long run, survive beats thrive.
Agile principle 2 (coopetition): What tries to kill us and fails makes us weaker and more agile – and that’s good.
Agile principle 3: Greed is bad. Customer-coordinated change is good.
Agile principle 4: If you don’t know who you will meet, you don’t know where you’re going.
Agile principle 5: People buy learning
Because “the old ways” are much more entrenched in the business as a whole than in the software development arm of an organization, I set these principles up in opposition to what I view as some of the key principles of business as we have known it – or at least the ones that the truly agile business contradicts. And, of course, I have a bit of fun in doing so.
I also believe that these principles will also serve for a draft agile marketing manifesto. The reasons why will become apparent later.
Agile Organization Goals: Fast, Effective Change
The old principle 1 (monetary goal): In the long run, the most successful are most likely to survive.
Agile principle 1 (fast, effective change goal): The successful thrive; the agile survive. In the long run, survive beats thrive.
Remember that in an agile organization, the focus is on maximizing the speed and effectiveness of change, not on maximizing short-run or long-run success as defined in monetary terms. Experience shows that in many development organizations, agile teams focused on speed of customer-coordinated change rather than cutting costs, improving quality, or speeding up development do better in the short and long run in monetary terms (costs, revenues, risk profile). This principle asserts that even if an organization is more successful in the short run by focusing on maximizing profit, it is less likely to survive in the long run, and less successful if it does survive, than the truly agile organization.
Agile Coopetition
The old principle 2: What does not kill you makes you stronger.
Agile principle 2 (coopetition): What tries to kill us and fails makes us weaker and more agile – and that’s good.
Recent research suggests that humans benefit most from eating foods that give us “moderate poisoning.” That is, they do not kill us, but they do weaken some existing systems. In order to adapt to these weaknesses, the body grows new capabilities – physical and mental – that let us handle a wider range of situations – that make us “survive by fitting” a wider range of environments. By contrast, foods with no poisoning capabilities offer much less benefit, and foods that cripple us offer no benefit at all.
There is a reasonable analogy here with competition vs. what is now called coopetition. The non-agile business assumes that one “wins” in the long term by crushing the competition, and that therefore the company emerges from the process stronger. This is doubtful; in the long-run, over-success in competition leads to monopoly behavior that decreases innovation and makes “crossing the chasm” much more difficult.
The agile business seeks to create “safe” competition by combining competition with cooperation – the same business that competes with you in one area cooperates with you in another area, and neither side expects to win overall in the long run. Rather, the relative weakness in one area of one of the two vendors will lead either to new capabilities or to redefinition of its focus on that side, using the increasing weakness of a particular area as an indicator of the need to move to something different. In the short run, that competition makes the agile vendor weaker. In the long run, the safety of that “competition” leads to increased agility, which leads to long-term survival and success. Meanwhile, cooperation in another area expands the solutions that the firm can offer and provides additional customer feedback.
Agile Culture: Happiness is Customer-Coordinated Change
The old principle 3: Greed is good.
Agile principle 3: Greed is bad. Customer-coordinated change is good.
Immortalized as the “invisible hand”, the pure form of the old principle 3 asserted that focusing on maximizing profit was good for the individual employee, good for the company, good for the consumer, and ultimately good for the economy and society. The agile principle asserts that focusing on speed and effectiveness of “safe” change is better for the individual employee, better for the consumer, and ultimately better for the economy and society in both short-term and long-term monetary and “social welfare” terms.
This is not a simple matter of “doing well by doing good.” Rather, the agile business recognizes that effective change must be constantly and firmly grounded in consumer reality, and that treating the consumer as an infinitely malleable toy to be squeezed of every last drop of money has bad long-term effects not only on survivability but also on corporate culture and thence on corporate success and “social welfare.”
Agile Strategy: Focus on Who
The old principle 4: If you don’t know where you’re going, any road will take you there.
Agile principle 4: If you don’t know who you will meet, you don’t know where you’re going.
As shown in such works as “Marketing Myopia”, the focus of a traditional firm is to accept the given of a global mass “market” as it is now, and try to determine just what market within that global one fits the firm best, and how the firm can steadily increase its profits within and by extensions of that market. The agile strategist focuses on a global market full of individual consumers with common lifecycles and fine-grained differences, and seeks to capture as much as possible of the lifecycle and time allotment of the largest group of such consumers as their needs evolve over time. When the agile firm talks about “continuous delivery” of what the customer wants, that is what it means. The traditional firm focuses on the visible, static market of an undifferentiated mass of needs; the agile firm focuses on understanding and meeting the inevitably changing individual needs of each of a class of individual consumers.
The agile firm believes that such a strategy gives it two key advantages over the traditional firm: it grounds the agile firm in reality rather than its own wishes about what the customer “should” be like in order to maximize firm profit, and it focuses the agile firm on striving toward the most effective kind of long-run strategy: cradle-to-grave, comprehensive, open-ended customer support.
Agile Target Market: Learners
The old principle 5: People want solutions
The new principle 5: People buy emotions
Agile principle 5: People buy learning
This agile principle can be seen as a corrective to the extremes of the old and new principles. It is true that in the short run, people will go out and buy a solution for an immediate problem, and that given extra money with which to buy things, people will then choose a solution that makes them feel better, or even an “emotional trigger” without any usefulness at all. However, in the long run, people usually continue buying in a particular area because it continues to relate to the ongoing reality of their lifecycle. Moreover, the best kind of customer buys because he or she enjoys the open-ended pleasure of constantly learning more in a “safe” (usually successful) way.
The alternative customer is the addict: one that develops such a need for an offering that the addict is willing to continue to spend on it forever, as it becomes less and less useful. As industries like tobacco and alcohol have shown, the addict can be a successful short-term market. However, firms in such industries tend to be unusually un-agile – reflecting their customers. Moreover, these industries react to the inevitable challenges of evolving customer needs by attempting to “game the system” and prevent these changes from happening – increasing the costs to society as a whole, as well as the impact to the firm of the inevitable failure to prevent change. The tobacco and alcohol industries are among the least successful over the last 60 years. By contrast, the computer software industry, which ranks among the most agile, is close to the most successful, and software companies that focus on the consumer rather than the business more successful still (see agile principle 4).
There is another implication of agile principle 5: the agile business should never simply react to the customer, but rather constantly alternate getting ahead of the customer and reacting to the customer. Helping a person to learn forever means constantly generating new, follow-on, useful things for the customer to learn, as the customer moves through the stages of a life. The customer, in return, tells you which of these innovations is desired and what his or her own innovations are.
This Is Also A Draft Agile Marketing Manifesto
The reason that these principles apply verbatim to agile marketing, I believe, is because it is probable that in the agile business, marketing will become a “loosely-coupled CEO”. That is, agile marketing’s primary job will be to integrate the constant iteration of new products into a long-term, effective business strategy.
This is not to say that this is marketing’s present job, nor that those trying to achieve agile marketing do integration very well right now. Rather, it says that marketing is the only existing business function that can carry out this necessary task effectively in the agile organization. The other functions are too focused on particular parts of the overall business’ agile process. The CEO simply does not have the capability of handling all the integrating tasks that must be done.
Rather, individual marketers, empowered by their unique focus on long-term customer need evolution, must connect and coordinate the enthusiastically wandering dogs of the other agile business functions. Meanwhile, the CEO imposes a single long-term personality on the agile marketers’ work, giving the results a touch of individuality that appeals to the individual consumer – much the same role that a great soloist plays in giving an old war-horse of a classic musical piece new life, or the kind of role that Steve Jobs occasionally played toward the end.
Thus, agile marketing in the long run will be effectively carrying out the agile organization’s principles – but in an agile way, in which the customers are equally the other agile business functions and the firm’s actual customers.
The Long Road Ahead
I say this is a draft manifesto; but, to me, it is also a platform with which to challenge other upcoming manifestos. I believe that if another manifesto contradicts or ignores these principles, there may very well be fundamental problems with that manifesto. If so, these principles give me (and, I hope, others) a basis for criticizing and, if necessary, rejecting such a manifesto. We are trying to create manifestoes at a much earlier – dangerously early – stage of our understanding of business agility. It may be OK not to get a manifesto right; but it is very important not to get it irretrievably wrong.
This is why I am so dogmatic about what goes into an agile manifesto. If the experience of agile software development and its alternatives teaches us anything, it is that if you think agile, your process becomes more and more agile and more and more successful; while if you don’t, it doesn’t. There doesn’t seem to be much of a middle ground. And it can be hard to wrap your mind around such a different way of thinking, and therefore easy to backslide subtly but permanently.
I will try in the next few weeks to flesh out the agile marketing “thought experiment” on which these suggested principles are based. In the meanwhile, enjoy – or not.
Agile principle 1 (fast, effective change goal): The successful thrive; the agile survive. In the long run, survive beats thrive.
Agile principle 2 (coopetition): What tries to kill us and fails makes us weaker and more agile – and that’s good.
Agile principle 3: Greed is bad. Customer-coordinated change is good.
Agile principle 4: If you don’t know who you will meet, you don’t know where you’re going.
Agile principle 5: People buy learning
Tuesday, April 3, 2012
Go, Go, Go, Said the Old Programmer
The other day, a query crossed my desk about Google’s public release of its new Go programming language – a fascinating and probably often applicable new take on the genre. As sometimes happens in these cases, what remains of my memories of computer science resurfaced as I read Google’s description of Go. And so, what follows are some possibly useful thoughts about its features.
Before I start, let me sum up, for those who dislike tech jargon (and especially jargon from someone who has not wielded a debugger in anger for a couple of decades): Google Go is worth playing with, for a wide variety of programming needs. Check it out.
Error Handling Without Exceptions
I have been waiting for many years for some recognition that error handling is actually on a level with writing correct code. To put it another way, each recognition of the possibility of an error is like the old case statement, in which good programming considers the “error” cases as just as important to code as the “correct” case.
Google Go has two back-to-the-future ways of approaching this: first, rule out “exceptions”, and second, allow multiple return variables/values. I agree with their critique entirely: an exception was always treated exceptionally, making programming more complex, and the ease (and decrease in bugs) from having one return value was more than counterbalanced by the complexity of trying to report errors in odd ways.
I do have one caveat: I am not yet convinced that programmers don’t need some form of coercion to make sure that errors are adequately handled. I don’t care whether it’s a required error value in the multi-value return or what, I just feel that error handling is still not as good as it should be.
No Assertions?
I hear Google’s point that these have been misused in error handling. However, I would suggest that the proper place to handle this type of misuse might be in the compiler, if possible. Assertions should still be used to check assumptions between two key lines of code, because simply forcing proper error handling does not account for major blind spots in the programmer’s view of his/her own code or his/her knowledge of the language. In other words, there still needs to be a final fallback check.
High-Level Concurrency and Goroutines
Now here’s a refreshing approach to concurrency and parallelism! It passes belief that we have not been able to categorize types of concurrency well enough to take the mechanisms up a level and hide things like mutexes. Likewise, goroutines seem a small-overhead way of adding some additional “multithreading” parallelism – and I suspect that this will compensate partially for the need for scaling software multithreading now that multicore chips are muddying the waters.
At the same time, I note a lack of consideration of the equivalent problem in data sharing. Let’s face it, not all data is isolatable in objects, and so concurrency of unitary data-processing functions (e.g., add, delete, update, read) needs a database-type “commit” consideration of concurrency.
No Type Hierarchy, No Type Inheritance, Interfaces
I don’t think I have a real feel for what “interfaces” are, but the idea of jettisoning object-oriented programming’s type hierarchy is a fascinating experiment. The reusability and careful thought about relationships between object classes have always been to the good, but at scale (something that we have been butting up against since the mid-1990s), the sheer effort of keeping track of that hierarchy placed severe limits on the ability to reuse code, and slowed down programming. I hope this one works; if it does, it will give a major boost to programming speed for Java types.
Focusing on string and map support
I don’t have a good feel for this approach to maps, as yet, but thank heavens someone realized that strong string operators were a very good idea. Anyone who ever had the privilege of using SNOBOL soon realized just how useful these can be.
Garbage collection and compiled code
I regard this as a sign that finally compilation speed within a virtual machine has come of age. It was always true that compilation is preferable to interpretation if compilation is infrequent, as long as the recompilation “hiccup” is not significant. I hope that Google Go really can make this work; there is, imho, a fair amount of legacy Java code out there that’s really sub-optimal in performance.
And, by the way, embedding decent online garbage collection in the machine is a major step forward, as well. After witnessing the birth pangs of JVMs in the late 1990s, I am well aware of the perils of memory leaks and other related ailments. I really think that, this time, we can do garbage collection and object-oriented long-running programs without major overhead. If so, it’s going to be another improvement in software quality from Go.
Assumed Architecture
If I understand what Google are saying, there’s a complex interplay between clients and servers in the typical Go process that should provide some better scalability in the typical Web situation. I’m not entirely convinced that the architecture is flexible enough, but it seems to me pretty easy to improve on the present client/Web server/app server architecture. I’m all for trying it.
No pointer arithmetic
Oh, yes. Finally, after all these years, that mess left over from the original C design back in the 1970s can be eliminated. It was a cute idea when it arrived; but it was soon apparent that its problems from increased nasty bugs outweighed its small benefits in performance in certain cases. I hope this puts a stake through its heart.
Minor caveats
Google has noted that performance out-of-the-box is in some cases not in striking distance of C. They make a reasonable case that that will go away; but frankly, as long as it’s within shouting distance of C, Go should be preferable. It’s much better at preventing nasty little programmer errors, which means that programming speed will lead to faster improvements in performance which will more than make up for C’s “coding on the bare metal” virtues.
Meanwhile, I’m still not convinced that Go (like all the other object-oriented or semi-object-oriented languages) really does an adequate job of handling data. The emphasis on structure orthogonality is good, but from my viewpoint, minor. We are still back in the days of the object-relational mismatch.
Simplified typing? Simplified dependency management? Simplified multicore concurrency? I’m not convinced; but I’m definitely hoping it’s true in the general case, not just within Google. Anyway, it’s worth it just to see another experiment shake up the programming world.
Repeating the Message
The quote in the title, by the way, is from T.S. Eliot’s first Quartet, “Go, go, go, said the thrush …”. Like the thrush, an old programmer thinks you should go and play with Google’s Go. After lo these many years of dealing with Java and JavaScript development, at the least, it will be a mind-opening change of pace.
Before I start, let me sum up, for those who dislike tech jargon (and especially jargon from someone who has not wielded a debugger in anger for a couple of decades): Google Go is worth playing with, for a wide variety of programming needs. Check it out.
Error Handling Without Exceptions
I have been waiting for many years for some recognition that error handling is actually on a level with writing correct code. To put it another way, each recognition of the possibility of an error is like the old case statement, in which good programming considers the “error” cases as just as important to code as the “correct” case.
Google Go has two back-to-the-future ways of approaching this: first, rule out “exceptions”, and second, allow multiple return variables/values. I agree with their critique entirely: an exception was always treated exceptionally, making programming more complex, and the ease (and decrease in bugs) from having one return value was more than counterbalanced by the complexity of trying to report errors in odd ways.
I do have one caveat: I am not yet convinced that programmers don’t need some form of coercion to make sure that errors are adequately handled. I don’t care whether it’s a required error value in the multi-value return or what, I just feel that error handling is still not as good as it should be.
No Assertions?
I hear Google’s point that these have been misused in error handling. However, I would suggest that the proper place to handle this type of misuse might be in the compiler, if possible. Assertions should still be used to check assumptions between two key lines of code, because simply forcing proper error handling does not account for major blind spots in the programmer’s view of his/her own code or his/her knowledge of the language. In other words, there still needs to be a final fallback check.
High-Level Concurrency and Goroutines
Now here’s a refreshing approach to concurrency and parallelism! It passes belief that we have not been able to categorize types of concurrency well enough to take the mechanisms up a level and hide things like mutexes. Likewise, goroutines seem a small-overhead way of adding some additional “multithreading” parallelism – and I suspect that this will compensate partially for the need for scaling software multithreading now that multicore chips are muddying the waters.
At the same time, I note a lack of consideration of the equivalent problem in data sharing. Let’s face it, not all data is isolatable in objects, and so concurrency of unitary data-processing functions (e.g., add, delete, update, read) needs a database-type “commit” consideration of concurrency.
No Type Hierarchy, No Type Inheritance, Interfaces
I don’t think I have a real feel for what “interfaces” are, but the idea of jettisoning object-oriented programming’s type hierarchy is a fascinating experiment. The reusability and careful thought about relationships between object classes have always been to the good, but at scale (something that we have been butting up against since the mid-1990s), the sheer effort of keeping track of that hierarchy placed severe limits on the ability to reuse code, and slowed down programming. I hope this one works; if it does, it will give a major boost to programming speed for Java types.
Focusing on string and map support
I don’t have a good feel for this approach to maps, as yet, but thank heavens someone realized that strong string operators were a very good idea. Anyone who ever had the privilege of using SNOBOL soon realized just how useful these can be.
Garbage collection and compiled code
I regard this as a sign that finally compilation speed within a virtual machine has come of age. It was always true that compilation is preferable to interpretation if compilation is infrequent, as long as the recompilation “hiccup” is not significant. I hope that Google Go really can make this work; there is, imho, a fair amount of legacy Java code out there that’s really sub-optimal in performance.
And, by the way, embedding decent online garbage collection in the machine is a major step forward, as well. After witnessing the birth pangs of JVMs in the late 1990s, I am well aware of the perils of memory leaks and other related ailments. I really think that, this time, we can do garbage collection and object-oriented long-running programs without major overhead. If so, it’s going to be another improvement in software quality from Go.
Assumed Architecture
If I understand what Google are saying, there’s a complex interplay between clients and servers in the typical Go process that should provide some better scalability in the typical Web situation. I’m not entirely convinced that the architecture is flexible enough, but it seems to me pretty easy to improve on the present client/Web server/app server architecture. I’m all for trying it.
No pointer arithmetic
Oh, yes. Finally, after all these years, that mess left over from the original C design back in the 1970s can be eliminated. It was a cute idea when it arrived; but it was soon apparent that its problems from increased nasty bugs outweighed its small benefits in performance in certain cases. I hope this puts a stake through its heart.
Minor caveats
Google has noted that performance out-of-the-box is in some cases not in striking distance of C. They make a reasonable case that that will go away; but frankly, as long as it’s within shouting distance of C, Go should be preferable. It’s much better at preventing nasty little programmer errors, which means that programming speed will lead to faster improvements in performance which will more than make up for C’s “coding on the bare metal” virtues.
Meanwhile, I’m still not convinced that Go (like all the other object-oriented or semi-object-oriented languages) really does an adequate job of handling data. The emphasis on structure orthogonality is good, but from my viewpoint, minor. We are still back in the days of the object-relational mismatch.
Simplified typing? Simplified dependency management? Simplified multicore concurrency? I’m not convinced; but I’m definitely hoping it’s true in the general case, not just within Google. Anyway, it’s worth it just to see another experiment shake up the programming world.
Repeating the Message
The quote in the title, by the way, is from T.S. Eliot’s first Quartet, “Go, go, go, said the thrush …”. Like the thrush, an old programmer thinks you should go and play with Google’s Go. After lo these many years of dealing with Java and JavaScript development, at the least, it will be a mind-opening change of pace.
Wednesday, March 28, 2012
Oracle, EMC, IBM, and Big Data: Avoiding The One-Legged Marathon
Note: this is a post of an article published in Nov. 2011 in another venue, and the vendors profiled here have all upgraded their Big Data stories significantly since then. Imho, it remains useful as a starting point for assessing their Big Data strategies, and deciding how to implement a long-term IT Big Data strategy oneself.
In recent weeks, Oracle, EMC, and IBM have issued announcements that begin to flesh out in solutions their vision of Big Data, its opportunities, and its best practices. Each of these vendor solutions has significant advantages for particular users, and all three are works in progress.
However, as presented so far, Oracle’s and EMC’s approaches appear to have significant limitations compared to IBM’s. If the pair continues to follow the same Big Data strategies, I believe that many of their customers will find themselves significantly hampered in dealing with certain types of Big Data analysis over the long run – an experience somewhat like voluntarily restricting yourself to one leg while running a marathon.
Let’s start by reviewing the promise and architectural details of Big Data, then take a look at each vendor’s strategy in turn.
Big Data, Big Challenges
As I noted in an earlier piece, some of Big Data is a relabeling of the incessant scaling of existing corporate queries and their extension to internal semi-structured (e.g., corporate documents) and unstructured (e.g., video, audio, graphics) data. The part that matters the most to today’s enterprise, however, is the typically unstructured data that is an integral part of customers’ social-media channel – including Facebook, instant messaging, Twitter, and blogging. This is global, enormous, very fast-growing, and increasingly integral (according to IBM’s recent CMO survey) to the vital corporate task of engaging with a "customer of one" throughout a long-term relationship.
However, technically, handling this kind of Big Data is very different from handling a traditional data warehouse. Access mechanisms such as Hadoop/MapReduce combine open-source software, large amounts of small or PC-type servers, and a loosening of consistency constraints on the distributed transactions (an approach called eventual consistency). The basic idea is to apply Big Data analytics to queries where it doesn’t matter if some users get "old" rather than the latest data, or if some users get an answer while others don’t. As a practical matter, this type of analytics is also prone to unexpected unavailability of data sources.
The enterprise cannot treat this data as just another BI data source. It differs fundamentally in that the enterprise can be far less sure that the data is current – or even available at all times. So, scheduled reporting or business-critical computing based on Big Data is much more difficult to pull off. On the other hand, this is data that would otherwise be unavailable for BI or analytics processes – and because of the approach to building solutions, should be exceptionally low-cost to access.
However, pointing the raw data at existing BI tools would be like pointing a fire hose at your mouth, with similarly painful results. Instead, the savvy IT organization will have plans in place to filter Big Data before it begins to access it.
Filtering is not the only difficulty. For many or most organizations, Big Data is of such size that simply moving it from its place in the cloud into an internal data store can take far longer than the mass downloads that traditionally lock up a data warehouse for hours. In many cases, it makes more sense to query on site and then pass the much smaller result set back to the end user. And as the world of cloud computing keeps evolving, the boundary between "query on site" and "download and query" keeps changing.
So how are Oracle, EMC, and IBM dealing with these challenges?
Oracle: We Control Your Vertical
Reading the press releases from Oracle OpenWorld about Oracle’s approach to Big Data reminds me a bit of the old TV show Outer Limits, which typically began with a paranoia-inducing voice intoning "We control your horizontal, we control your vertical …" as the screen began flickering.
Oracle’s announcements included specific mechanisms for mass downloads to Oracle Database data stores in Oracle appliances (Oracle Loader for Hadoop), so that Oracle Database could query the data side-by-side with existing enterprise data, complete with data-warehouse data-cleansing mechanisms.
The focus is on Oracle Exalytics BI Machine, which combines Oracle Database 11g and Oracle’s TimesTen in-memory database for additional BI scalability. In addition, there is a "NoSQL" database that claims to provide "bounded latency" (i.e., it limits the "eventual" in "eventual consistency"), although how it should combine with Oracle’s appliances was not clearly stated.
The obvious advantage of this approach is integration, which should deliver additional scalability on top of Oracle Database’s already-strong scalability. Whether that will be enough to make the huge leap to handling hundred-petabyte data stores that change frequently remains to be seen.
At the same time, these announcements implicitly suggest that Big Data should be downloaded to Oracle databases in the enterprise, or users should access Big Data via Oracle databases running in the cloud, but provide no apparent way to link cloud and enterprise data stores or BI. To put it another way, Oracle is presenting a vision of Big Data used by Oracle apps, accessed by Oracle databases using Oracle infrastructure software and running on Oracle hardware with no third party needed. We control your vertical, indeed.
What also concerns me about the company’s approach is that there is no obvious mecha-nism either for dealing with the lateness/unavailability/excessive lack of quality of Big Data, or for choosing the optimal mix of cloud and in-enterprise data location. There are only hints: the bounded latency of Oracle NoSQL Database, or the claim that Oracle Data Integrator with Application Adapter for Hadoop can combine Big Data with Oracle Database data – in Oracle Database format and in Oracle Database data stores. We control your horizontal, too. But how well are we controlling it? We’ll get back to you on that.
EMC: Competing with the Big Data Boys
The recent EMC Forum in Boston in many ways delivered what I regard as very good news for the company and its customers. In the case of Big Data, its acquisition of Greenplum with its BI capabilities led the way. And Greenplum finally appears to have provided EMC with the data management smarts it has always needed to be a credible global information management solutions vendor. In particular, Greenplum is apparently placing analytical intelligence in EMC hardware and software, giving EMC a great boost in areas such as being able to handle querying within the storage device and monitoring distributed systems (such as VCE’s VBlocks) for administrative purposes. These are clearly leading-edge, valuable features.
EMC’s Greenplum showed itself to be a savvy supporter of Big Data. It supports the usual third-party suspects for BI: SQL, MapReduce, and SAS among others. Querying is "software shared-nothing", running in virtual machines on commodity VBlocks and other scale-out/grid x86 hardware. Greenplum has focused on the fast-deploy necessities of the cloud, claiming a 25-minute data-model change – something that has certainly proved difficult in large-scale data warehouses in the past.
Like Oracle, Greenplum offers "mixed" columnar and row-based relational technology; un-like Oracle, it is tuned automatically, rather than leaving it up to the customer how to mix and match the two. However, its answer for combining Big Data and enterprise data is also tactically similar to Oracle’s: download into the Greenplum data store.
Of our three vendors, EMC via Greenplum has been the most concrete about what one can do with Big Data, offering specific support for combining social graphing, customer-of-one tracking of Twitter/Facebook posts, and mining of enterprise customer data. The actual demo had an unfortunate "1984" flavor, however, with a customer’s casual chat about fast cars being used to help justify doubling his car insurance rate.
The bottom line with Greenplum appears to be that its ability to scale is impressive, even with Big Data included, and it is highly likely to provide benefits out of the box to the savvy social-media analytics implementer. Still, it avoids, rather than solves, the problems of massive querying across Big Data and relational technology – it assumes massive downloads are possible and data is "low latency" and "clean", where in many cases it appears that this will not be so. EMC Greenplum is not as "one-vendor" a solution as Oracle’s, but it does not have the scalability and robustness track record of Oracle, either.
IBM: Avoiding Architectural Lock-In
At first glance, IBM appears to have a range of solutions for Big Data similar to Oracle and EMC – but more of them. Thus, it has the Netezza appliance; it has InfoSphere BigInsights for querying against Hadoop; it has the ability to download data into both its Informix/in-memory technology and DB2 databases for in-enterprise data warehousing; and it offers various Smart Analytics System solutions as central BI facilities.
Along with these, it provides Master Data Management (InfoSphere MDM), data-quality features (InfoSphere DataStage), InfoSphere Streams for querying against streaming Web sensor data (like mobile GPS), and the usual quick-deployment models and packaged hardware/software solutions on its scale-out and scale-up platforms. And, of course, everyone has heard of Watson – although, intriguingly, its use cases are not yet clearly pervasive in Big Data implementations.
To my mind, however, the most significant difference in IBM’s approach to Big Data is that it offers explicit support for a wide range of ways to combine Big-Data-in-place and enterprise data in queries. For example, IBM’s MDM solution allows multiple locations for customer data and supports synchronization and replication of linked data. Effectively used, this allows users to run alerts against Big Data in remote clouds or dynamically shift customer querying between private and public clouds to maximize performance. And, of course, the remote facility need not involve an IBM database, because of the MDM solution’s cross-vendor "data virtualization" capabilities.
Even IBM’s traditional data warehousing solutions are joining the fun. The IBM IOD conference introduced the idea of a "logical warehouse", which departs from the idea of a single system or cluster that contains the enterprise’s "one version of the truth", and towards the idea of a "truth veneer" that looks like a data warehouse from the point of view of the analytics engine but is actually multiple operational, data-warehouse, and cloud data stores. And, of course, IBM’s Smart Analytics Systems run on System x (x86), Power (RISC) and System z (mainframe) hardware.
On the other hand, there are no clear IBM guidelines for optimizing an architecture that combines traditional enterprise BI with Big Data. It gives one the strong impression that IBM is providing customers with a wide range of solutions, but little guidance as to how to use them. That IBM does not move the customer towards an architecture that may prevent effective handling of certain types of Big-Data queries is good news; that IBM does not yet convey clearly how these queries should be handled, not so much.
Composite Software and the Missing Big-Data Link
One complementary technology that users might consider is data virtualization (DV), as provided by vendors such as Composite Software and Denodo. In these solutions, subqueries on data of disparate types (such as Big Data and traditional) are optimized flexibly and dynamically, with due attention to "dirty" or unavailable data. DV solution vendors’ accrued wisdom can be simply summed up: usually, querying on-site instead of doing a mass download is better.
How to deal with the "temporal gap" between "eventually consistent" Big Data and "hot off the press" enterprise sales data is a matter for customer experimentation and fine-tuning, but that customer can always decide what to do with full assurance that subqueries have been optimized and data cleansed in the right way.
The Big Data Bottom Line
To me, the bottom line of all of this Big Data hype is that there is indeed immediate business value in there, and specifically in being able to go beyond the immediate customer interaction to understand the customer as a whole and over time – and thereby to establish truly win-win long-term customer relationships. Simply by looking at the social habits of key consumer "ultimate customers," as the Oracle, EMC, and IBM Big Data tools already allow you to do, enterprises of all sizes can fine-tune their interactions with the immediate customer (B2B or B2C) to be far more cost-effective.
However, with such powerful analytical insights, it is exceptionally easy to shoot oneself in the foot. Even skipping quickly over recent reports, I can see anecdotes of "data overload" that paralyze employees and projects, trigger-happy real-time inventory management that actually increases costs, unintentional breaches of privacy regulations, punitive use of newly public consumer behavior that damages the enterprise’s brand or perceived "character", and "information overload" on the part of the corporate strategist.
The common theme running through these user stories is a lack of vendor-supplied context that would allow the enterprise to understand how to use the masses of new Big Data properly, and especially an understanding of the limitations of the new data.
Thus, in the long run, the best performers will seek a Big-Data analytics architecture that is flexible, handles the limitations of Big Data as the enterprise needs them handled, and allows a highly scalable combination of Big Data and traditional data-warehouse data. So far, Oracle and EMC seem to be urging customers somewhat in the wrong direction, while IBM is providing a "have it your way" solution; but all of their solutions could benefit strongly from giving better optimization and analysis guidance to IT.
In the short run, users will do fine with a Big-Data architecture that does not provide an infrastructure support "leg" for a use case that they do not need to consider. In the long run, the lack of that "leg" may be a crippling handicap in the analytics marathon. IT buyers should carefully consider the long-run architectural plans of each vendor as they develop.
In recent weeks, Oracle, EMC, and IBM have issued announcements that begin to flesh out in solutions their vision of Big Data, its opportunities, and its best practices. Each of these vendor solutions has significant advantages for particular users, and all three are works in progress.
However, as presented so far, Oracle’s and EMC’s approaches appear to have significant limitations compared to IBM’s. If the pair continues to follow the same Big Data strategies, I believe that many of their customers will find themselves significantly hampered in dealing with certain types of Big Data analysis over the long run – an experience somewhat like voluntarily restricting yourself to one leg while running a marathon.
Let’s start by reviewing the promise and architectural details of Big Data, then take a look at each vendor’s strategy in turn.
Big Data, Big Challenges
As I noted in an earlier piece, some of Big Data is a relabeling of the incessant scaling of existing corporate queries and their extension to internal semi-structured (e.g., corporate documents) and unstructured (e.g., video, audio, graphics) data. The part that matters the most to today’s enterprise, however, is the typically unstructured data that is an integral part of customers’ social-media channel – including Facebook, instant messaging, Twitter, and blogging. This is global, enormous, very fast-growing, and increasingly integral (according to IBM’s recent CMO survey) to the vital corporate task of engaging with a "customer of one" throughout a long-term relationship.
However, technically, handling this kind of Big Data is very different from handling a traditional data warehouse. Access mechanisms such as Hadoop/MapReduce combine open-source software, large amounts of small or PC-type servers, and a loosening of consistency constraints on the distributed transactions (an approach called eventual consistency). The basic idea is to apply Big Data analytics to queries where it doesn’t matter if some users get "old" rather than the latest data, or if some users get an answer while others don’t. As a practical matter, this type of analytics is also prone to unexpected unavailability of data sources.
The enterprise cannot treat this data as just another BI data source. It differs fundamentally in that the enterprise can be far less sure that the data is current – or even available at all times. So, scheduled reporting or business-critical computing based on Big Data is much more difficult to pull off. On the other hand, this is data that would otherwise be unavailable for BI or analytics processes – and because of the approach to building solutions, should be exceptionally low-cost to access.
However, pointing the raw data at existing BI tools would be like pointing a fire hose at your mouth, with similarly painful results. Instead, the savvy IT organization will have plans in place to filter Big Data before it begins to access it.
Filtering is not the only difficulty. For many or most organizations, Big Data is of such size that simply moving it from its place in the cloud into an internal data store can take far longer than the mass downloads that traditionally lock up a data warehouse for hours. In many cases, it makes more sense to query on site and then pass the much smaller result set back to the end user. And as the world of cloud computing keeps evolving, the boundary between "query on site" and "download and query" keeps changing.
So how are Oracle, EMC, and IBM dealing with these challenges?
Oracle: We Control Your Vertical
Reading the press releases from Oracle OpenWorld about Oracle’s approach to Big Data reminds me a bit of the old TV show Outer Limits, which typically began with a paranoia-inducing voice intoning "We control your horizontal, we control your vertical …" as the screen began flickering.
Oracle’s announcements included specific mechanisms for mass downloads to Oracle Database data stores in Oracle appliances (Oracle Loader for Hadoop), so that Oracle Database could query the data side-by-side with existing enterprise data, complete with data-warehouse data-cleansing mechanisms.
The focus is on Oracle Exalytics BI Machine, which combines Oracle Database 11g and Oracle’s TimesTen in-memory database for additional BI scalability. In addition, there is a "NoSQL" database that claims to provide "bounded latency" (i.e., it limits the "eventual" in "eventual consistency"), although how it should combine with Oracle’s appliances was not clearly stated.
The obvious advantage of this approach is integration, which should deliver additional scalability on top of Oracle Database’s already-strong scalability. Whether that will be enough to make the huge leap to handling hundred-petabyte data stores that change frequently remains to be seen.
At the same time, these announcements implicitly suggest that Big Data should be downloaded to Oracle databases in the enterprise, or users should access Big Data via Oracle databases running in the cloud, but provide no apparent way to link cloud and enterprise data stores or BI. To put it another way, Oracle is presenting a vision of Big Data used by Oracle apps, accessed by Oracle databases using Oracle infrastructure software and running on Oracle hardware with no third party needed. We control your vertical, indeed.
What also concerns me about the company’s approach is that there is no obvious mecha-nism either for dealing with the lateness/unavailability/excessive lack of quality of Big Data, or for choosing the optimal mix of cloud and in-enterprise data location. There are only hints: the bounded latency of Oracle NoSQL Database, or the claim that Oracle Data Integrator with Application Adapter for Hadoop can combine Big Data with Oracle Database data – in Oracle Database format and in Oracle Database data stores. We control your horizontal, too. But how well are we controlling it? We’ll get back to you on that.
EMC: Competing with the Big Data Boys
The recent EMC Forum in Boston in many ways delivered what I regard as very good news for the company and its customers. In the case of Big Data, its acquisition of Greenplum with its BI capabilities led the way. And Greenplum finally appears to have provided EMC with the data management smarts it has always needed to be a credible global information management solutions vendor. In particular, Greenplum is apparently placing analytical intelligence in EMC hardware and software, giving EMC a great boost in areas such as being able to handle querying within the storage device and monitoring distributed systems (such as VCE’s VBlocks) for administrative purposes. These are clearly leading-edge, valuable features.
EMC’s Greenplum showed itself to be a savvy supporter of Big Data. It supports the usual third-party suspects for BI: SQL, MapReduce, and SAS among others. Querying is "software shared-nothing", running in virtual machines on commodity VBlocks and other scale-out/grid x86 hardware. Greenplum has focused on the fast-deploy necessities of the cloud, claiming a 25-minute data-model change – something that has certainly proved difficult in large-scale data warehouses in the past.
Like Oracle, Greenplum offers "mixed" columnar and row-based relational technology; un-like Oracle, it is tuned automatically, rather than leaving it up to the customer how to mix and match the two. However, its answer for combining Big Data and enterprise data is also tactically similar to Oracle’s: download into the Greenplum data store.
Of our three vendors, EMC via Greenplum has been the most concrete about what one can do with Big Data, offering specific support for combining social graphing, customer-of-one tracking of Twitter/Facebook posts, and mining of enterprise customer data. The actual demo had an unfortunate "1984" flavor, however, with a customer’s casual chat about fast cars being used to help justify doubling his car insurance rate.
The bottom line with Greenplum appears to be that its ability to scale is impressive, even with Big Data included, and it is highly likely to provide benefits out of the box to the savvy social-media analytics implementer. Still, it avoids, rather than solves, the problems of massive querying across Big Data and relational technology – it assumes massive downloads are possible and data is "low latency" and "clean", where in many cases it appears that this will not be so. EMC Greenplum is not as "one-vendor" a solution as Oracle’s, but it does not have the scalability and robustness track record of Oracle, either.
IBM: Avoiding Architectural Lock-In
At first glance, IBM appears to have a range of solutions for Big Data similar to Oracle and EMC – but more of them. Thus, it has the Netezza appliance; it has InfoSphere BigInsights for querying against Hadoop; it has the ability to download data into both its Informix/in-memory technology and DB2 databases for in-enterprise data warehousing; and it offers various Smart Analytics System solutions as central BI facilities.
Along with these, it provides Master Data Management (InfoSphere MDM), data-quality features (InfoSphere DataStage), InfoSphere Streams for querying against streaming Web sensor data (like mobile GPS), and the usual quick-deployment models and packaged hardware/software solutions on its scale-out and scale-up platforms. And, of course, everyone has heard of Watson – although, intriguingly, its use cases are not yet clearly pervasive in Big Data implementations.
To my mind, however, the most significant difference in IBM’s approach to Big Data is that it offers explicit support for a wide range of ways to combine Big-Data-in-place and enterprise data in queries. For example, IBM’s MDM solution allows multiple locations for customer data and supports synchronization and replication of linked data. Effectively used, this allows users to run alerts against Big Data in remote clouds or dynamically shift customer querying between private and public clouds to maximize performance. And, of course, the remote facility need not involve an IBM database, because of the MDM solution’s cross-vendor "data virtualization" capabilities.
Even IBM’s traditional data warehousing solutions are joining the fun. The IBM IOD conference introduced the idea of a "logical warehouse", which departs from the idea of a single system or cluster that contains the enterprise’s "one version of the truth", and towards the idea of a "truth veneer" that looks like a data warehouse from the point of view of the analytics engine but is actually multiple operational, data-warehouse, and cloud data stores. And, of course, IBM’s Smart Analytics Systems run on System x (x86), Power (RISC) and System z (mainframe) hardware.
On the other hand, there are no clear IBM guidelines for optimizing an architecture that combines traditional enterprise BI with Big Data. It gives one the strong impression that IBM is providing customers with a wide range of solutions, but little guidance as to how to use them. That IBM does not move the customer towards an architecture that may prevent effective handling of certain types of Big-Data queries is good news; that IBM does not yet convey clearly how these queries should be handled, not so much.
Composite Software and the Missing Big-Data Link
One complementary technology that users might consider is data virtualization (DV), as provided by vendors such as Composite Software and Denodo. In these solutions, subqueries on data of disparate types (such as Big Data and traditional) are optimized flexibly and dynamically, with due attention to "dirty" or unavailable data. DV solution vendors’ accrued wisdom can be simply summed up: usually, querying on-site instead of doing a mass download is better.
How to deal with the "temporal gap" between "eventually consistent" Big Data and "hot off the press" enterprise sales data is a matter for customer experimentation and fine-tuning, but that customer can always decide what to do with full assurance that subqueries have been optimized and data cleansed in the right way.
The Big Data Bottom Line
To me, the bottom line of all of this Big Data hype is that there is indeed immediate business value in there, and specifically in being able to go beyond the immediate customer interaction to understand the customer as a whole and over time – and thereby to establish truly win-win long-term customer relationships. Simply by looking at the social habits of key consumer "ultimate customers," as the Oracle, EMC, and IBM Big Data tools already allow you to do, enterprises of all sizes can fine-tune their interactions with the immediate customer (B2B or B2C) to be far more cost-effective.
However, with such powerful analytical insights, it is exceptionally easy to shoot oneself in the foot. Even skipping quickly over recent reports, I can see anecdotes of "data overload" that paralyze employees and projects, trigger-happy real-time inventory management that actually increases costs, unintentional breaches of privacy regulations, punitive use of newly public consumer behavior that damages the enterprise’s brand or perceived "character", and "information overload" on the part of the corporate strategist.
The common theme running through these user stories is a lack of vendor-supplied context that would allow the enterprise to understand how to use the masses of new Big Data properly, and especially an understanding of the limitations of the new data.
Thus, in the long run, the best performers will seek a Big-Data analytics architecture that is flexible, handles the limitations of Big Data as the enterprise needs them handled, and allows a highly scalable combination of Big Data and traditional data-warehouse data. So far, Oracle and EMC seem to be urging customers somewhat in the wrong direction, while IBM is providing a "have it your way" solution; but all of their solutions could benefit strongly from giving better optimization and analysis guidance to IT.
In the short run, users will do fine with a Big-Data architecture that does not provide an infrastructure support "leg" for a use case that they do not need to consider. In the long run, the lack of that "leg" may be a crippling handicap in the analytics marathon. IT buyers should carefully consider the long-run architectural plans of each vendor as they develop.
Tuesday, March 20, 2012
Agility and Sustainability: Complement or Clash?
It seems to me that both the concept of “agility” and the concept of “sustainability” have reached the point of real-world use where they will begin to intersect, especially in businesses, and where we will need to think about whether one should be subordinated to the other – because sustainability projects may detract from efforts towards agility, or agility efforts might detract from moves toward sustainability. Or, it may be that, fundamentally, the two complement each other, and can act in synergy. Since agility, at least, has proven its long-term worth in bottom-line results, and sustainability may turn out to be as necessary as the air we wish to breathe, the answer matters.
What follows is very far removed from daily reality. If business agility is fully achieved in a business – and we are so far from such a business that it is hard to say if it will ever exist – and if sustainability really takes over as a global business strategy and economic system – and we still do not know how such a system can function in the long term – then the relationship between the two will be a major issue. Still, some sort of answer would be helpful right now, so that as these two business and societal cultures lurch slowly into possible ubiquity, we can smooth the way for either/both. All we can do is paint with a broad brush; well, let’s get to it.
Setting the Stage: Similarities and Differences
Let’s start with business agility in its extreme form. Such a business (or society) is not focused on success as measured in money or power, but rather on changing rapidly and effectively. The reason this works is that all our institutions and thoughts bring us back to considering success – what we need from agility is a reward for constantly moving on from success. What grounds the agile business is constant negotiation with its customers and the environment as a whole, in which first one, then the other takes the lead in saying what to do next.
The mind shift described here is not as great as it sounds. In any given business, people want to learn, as long as it’s “safe”: it’s accepted by the people around you, people value you for being able to change as well as for specific accomplishments, you have some idea what to change to (i.e., the customer or end user gives you lots of suggestions), and the success rate of this type of thinking is high.
Such an approach has not only enormous positive side-effects on traditional but real measures of success – revenue, cost reduction, profit, customer satisfaction, power – but also great long-term psychic rewards – constant learning with constant positive feedback that what you are doing matters to other people as well as you, which can in some cases only end with the end of your life. In other words, once introduced, business agility can feed on itself – as long as it can get past the initial major mind and culture shift, which involves new metrics, new daily work routines, and baffling encounters with other parts of the company who are doing the same agility thing in a different way. But it all works out, because the work is more fun and because the new agile you starts treating those other parts of the company as customers.
Now let’s switch to the extreme form of sustainability. Wikipedia’s definition continually circles back to: preservation of a system’s ability to endure. More practically, this means: don’t put a system into such a state that it will break down. Usually, breakdown is done by scaling beyond the system’s ability to be patched, or destroying enough of the system’s infrastructure that it can no longer function. So, in my own definition, sustainability means creating a system that will not scale beyond a certain limit, forever, and will not destroy necessary infrastructure. All that we do, all the ways that we grow and change, are forever bounded by those limits – so a sustainable system is one in which not only are limits enforced, but the system itself encourages no violation of those limits. And yet, how do you do that without eventually banning all important change?
Well, strictly speaking, you can’t. And yet, you can indeed get so close to that, that we might well defer the problem until the end of the universe or thereabouts. The answer is equal parts “flight to the huge” and “flight to the virtual.” By that I mean, there are enough physical resources of certain types in the universe (like sunlight) that are beyond our capacity to exhaust for a very long time, and there is enough computer capacity even now that we can shift the things we do from physical to virtual for a very long time, and when we combine the two we see that most other limits that we are approaching can be handled for a huge amount of time by shifting them into the huge and virtual buckets as fast as we ramp up our systems.
The obvious example is putting carbon in the atmosphere. In a fairy tale world, we figure out how to capture and store sunlight with maximum effectiveness and level out its use, shifting from fossil fuels to handle our expanding use of energy as we increase in numbers and prosper. However, that’s not enough; we also need to wipe out that increase of energy usage. And so, we shift from physical to virtual, changing the content of our physical machines to include greater and greater amounts of software, avoiding the need for physical machines by accomplishing much of their tasks by software. This is not about making the physical worker useless before an omnipotent machine; this is about 6 billion people with 6 billion “virtual machines” doing more than what 5 billion people with 1 billion “virtual machines” were doing before, but with the same number of real machines and the same energy usage, ad infinitum. That’s “close enough” sustainability: you are not trapped in a pond, circling the same shores forever, but always flowing down a river that appears to be forever widening and changing, with enormous shores on either side and a riverbed that never deepens.
And now we can see the similarities and the differences between this type of sustainability and this type of agility. Here are the similarities:
• Both are all-encompassing mind sets, cultures, and processes that are clearly quite different from the ones we have now.
• Both are likely to be more effective – help us to survive more, help us to be better off physically and psychologically – than what we are doing today, in the long term.
• Both allow for growth and change – the growth and change that has brought us this far.
But, here are the differences:
• Agility is a more positive message: there’s always something to learn, something better to do; while sustainability is a more negative message: there are limits, and life is about staying within those limits.
• Agility is open and more uncertain, you assume that there will always be new things to do, but you have no idea what they are; sustainability is closed and more certain, as the things that matter are avoiding breakdowns, which are typically bounded problems that are physical, definable, and measurable.
• Fundamentally, agility focuses on changing and ignores survivability, and so improves your chance to survive; sustainability limits change in the name of survival, and who knows what that does to your ability to grow and change?
In other words, agility and sustainability can be seen as rivals for the better business, culture, and society of the future, which can all too easily either undercut the other in their competition for being the One True Way, or fight so fiercely that neither is achieved. And, as I noted in the introduction, either kind of “clash” outcome will eventually matter a lot.
Initial Thoughts on Synergy
It seems to me that there are two approaches that could ensure that, despite this, both agility and sustainability can thrive, and perhaps even reinforce each other. I call them the tactical and the strategic.
The tactical approach tries to manage combining agility and sustainability on a case-by-case basis. This means, for example, setting the boundaries of sustainability very clearly, so that they are intuitive for all, for the business as well as the society, and enforcing them strongly when clearly violated, but allowing endless agility within those boundaries. Carbon pricing is an example of the tactical approach: if effectively administered, it constantly sets clear monetary boundaries on carbon use, within a sustainable carbon yearly limit. Do that, says the tactical approach, and then let the business be as agile as it wishes.
The strategic approach, by contrast, seeks to create an integrated mindset combining agility and sustainability. One way to do this that I view as feasible is to get people used to the idea that trying to do the impossible is not the most effective way of doing things. As I have previously written, “working smarter, not harder” comes down to “instead of trying the impossible, be smart and focus your (in this case, change ideas) on things that are possible.” Knowing short-term and long-term limits becomes an essential part of the really agile business, because it allows people to increase the number of new ideas that are really good ones. But then, the strategic approach involves yet a third mind shift, to trying to work smarter instead of harder and seeing limits as positive idea-producers; how many mind shifts can we do, anyway?
Summing Up
My overall conclusion is brief: Tentatively, I lean towards taking a strategic approach to melding agility and sustainability, and applying it wherever frictions between the two crop up; but my main conclusion is that we had better start thinking more about tactical and strategic approaches, and how to do them, right now. It may be that the successes of agility will lead to a mindset that treats sustainability as yet another source of new ideas, and so we muddle through to survival; but I wouldn’t count on it. Much better to be proactive even when agility doesn’t call for it, and think about how to combine carbon metrics and agility metrics. In the meanwhile, I’ll continue preaching agility – because it’s fun – and keeping a wary eye on sustainability – because we may need it sooner than we think.
That’s as far as I can get. Ideas, anyone?
What follows is very far removed from daily reality. If business agility is fully achieved in a business – and we are so far from such a business that it is hard to say if it will ever exist – and if sustainability really takes over as a global business strategy and economic system – and we still do not know how such a system can function in the long term – then the relationship between the two will be a major issue. Still, some sort of answer would be helpful right now, so that as these two business and societal cultures lurch slowly into possible ubiquity, we can smooth the way for either/both. All we can do is paint with a broad brush; well, let’s get to it.
Setting the Stage: Similarities and Differences
Let’s start with business agility in its extreme form. Such a business (or society) is not focused on success as measured in money or power, but rather on changing rapidly and effectively. The reason this works is that all our institutions and thoughts bring us back to considering success – what we need from agility is a reward for constantly moving on from success. What grounds the agile business is constant negotiation with its customers and the environment as a whole, in which first one, then the other takes the lead in saying what to do next.
The mind shift described here is not as great as it sounds. In any given business, people want to learn, as long as it’s “safe”: it’s accepted by the people around you, people value you for being able to change as well as for specific accomplishments, you have some idea what to change to (i.e., the customer or end user gives you lots of suggestions), and the success rate of this type of thinking is high.
Such an approach has not only enormous positive side-effects on traditional but real measures of success – revenue, cost reduction, profit, customer satisfaction, power – but also great long-term psychic rewards – constant learning with constant positive feedback that what you are doing matters to other people as well as you, which can in some cases only end with the end of your life. In other words, once introduced, business agility can feed on itself – as long as it can get past the initial major mind and culture shift, which involves new metrics, new daily work routines, and baffling encounters with other parts of the company who are doing the same agility thing in a different way. But it all works out, because the work is more fun and because the new agile you starts treating those other parts of the company as customers.
Now let’s switch to the extreme form of sustainability. Wikipedia’s definition continually circles back to: preservation of a system’s ability to endure. More practically, this means: don’t put a system into such a state that it will break down. Usually, breakdown is done by scaling beyond the system’s ability to be patched, or destroying enough of the system’s infrastructure that it can no longer function. So, in my own definition, sustainability means creating a system that will not scale beyond a certain limit, forever, and will not destroy necessary infrastructure. All that we do, all the ways that we grow and change, are forever bounded by those limits – so a sustainable system is one in which not only are limits enforced, but the system itself encourages no violation of those limits. And yet, how do you do that without eventually banning all important change?
Well, strictly speaking, you can’t. And yet, you can indeed get so close to that, that we might well defer the problem until the end of the universe or thereabouts. The answer is equal parts “flight to the huge” and “flight to the virtual.” By that I mean, there are enough physical resources of certain types in the universe (like sunlight) that are beyond our capacity to exhaust for a very long time, and there is enough computer capacity even now that we can shift the things we do from physical to virtual for a very long time, and when we combine the two we see that most other limits that we are approaching can be handled for a huge amount of time by shifting them into the huge and virtual buckets as fast as we ramp up our systems.
The obvious example is putting carbon in the atmosphere. In a fairy tale world, we figure out how to capture and store sunlight with maximum effectiveness and level out its use, shifting from fossil fuels to handle our expanding use of energy as we increase in numbers and prosper. However, that’s not enough; we also need to wipe out that increase of energy usage. And so, we shift from physical to virtual, changing the content of our physical machines to include greater and greater amounts of software, avoiding the need for physical machines by accomplishing much of their tasks by software. This is not about making the physical worker useless before an omnipotent machine; this is about 6 billion people with 6 billion “virtual machines” doing more than what 5 billion people with 1 billion “virtual machines” were doing before, but with the same number of real machines and the same energy usage, ad infinitum. That’s “close enough” sustainability: you are not trapped in a pond, circling the same shores forever, but always flowing down a river that appears to be forever widening and changing, with enormous shores on either side and a riverbed that never deepens.
And now we can see the similarities and the differences between this type of sustainability and this type of agility. Here are the similarities:
• Both are all-encompassing mind sets, cultures, and processes that are clearly quite different from the ones we have now.
• Both are likely to be more effective – help us to survive more, help us to be better off physically and psychologically – than what we are doing today, in the long term.
• Both allow for growth and change – the growth and change that has brought us this far.
But, here are the differences:
• Agility is a more positive message: there’s always something to learn, something better to do; while sustainability is a more negative message: there are limits, and life is about staying within those limits.
• Agility is open and more uncertain, you assume that there will always be new things to do, but you have no idea what they are; sustainability is closed and more certain, as the things that matter are avoiding breakdowns, which are typically bounded problems that are physical, definable, and measurable.
• Fundamentally, agility focuses on changing and ignores survivability, and so improves your chance to survive; sustainability limits change in the name of survival, and who knows what that does to your ability to grow and change?
In other words, agility and sustainability can be seen as rivals for the better business, culture, and society of the future, which can all too easily either undercut the other in their competition for being the One True Way, or fight so fiercely that neither is achieved. And, as I noted in the introduction, either kind of “clash” outcome will eventually matter a lot.
Initial Thoughts on Synergy
It seems to me that there are two approaches that could ensure that, despite this, both agility and sustainability can thrive, and perhaps even reinforce each other. I call them the tactical and the strategic.
The tactical approach tries to manage combining agility and sustainability on a case-by-case basis. This means, for example, setting the boundaries of sustainability very clearly, so that they are intuitive for all, for the business as well as the society, and enforcing them strongly when clearly violated, but allowing endless agility within those boundaries. Carbon pricing is an example of the tactical approach: if effectively administered, it constantly sets clear monetary boundaries on carbon use, within a sustainable carbon yearly limit. Do that, says the tactical approach, and then let the business be as agile as it wishes.
The strategic approach, by contrast, seeks to create an integrated mindset combining agility and sustainability. One way to do this that I view as feasible is to get people used to the idea that trying to do the impossible is not the most effective way of doing things. As I have previously written, “working smarter, not harder” comes down to “instead of trying the impossible, be smart and focus your (in this case, change ideas) on things that are possible.” Knowing short-term and long-term limits becomes an essential part of the really agile business, because it allows people to increase the number of new ideas that are really good ones. But then, the strategic approach involves yet a third mind shift, to trying to work smarter instead of harder and seeing limits as positive idea-producers; how many mind shifts can we do, anyway?
Summing Up
My overall conclusion is brief: Tentatively, I lean towards taking a strategic approach to melding agility and sustainability, and applying it wherever frictions between the two crop up; but my main conclusion is that we had better start thinking more about tactical and strategic approaches, and how to do them, right now. It may be that the successes of agility will lead to a mindset that treats sustainability as yet another source of new ideas, and so we muddle through to survival; but I wouldn’t count on it. Much better to be proactive even when agility doesn’t call for it, and think about how to combine carbon metrics and agility metrics. In the meanwhile, I’ll continue preaching agility – because it’s fun – and keeping a wary eye on sustainability – because we may need it sooner than we think.
That’s as far as I can get. Ideas, anyone?
Friday, March 16, 2012
Information Agility and Data Virtualization: A Thought Experiment
In an agile world, what is the value of data virtualization?
For a decade, I have been touting the real-world value-add of what is now called data virtualization – the ability to see disparate data stores as one, and to perform real-time data processing, reporting, and analytics on the “global” data store – in the traditional world of IT and un-agile enterprises. However, in recent studies of agile methodologies within software development and in other areas of the enterprise, I have found that technologies and solutions well-adapted to deliver risk management and new product development in the traditional short-run-cost/revenue enterprises of the past can actually inhibit the agility and therefore the performance of an enterprise seeking the added advantage that true agility brings. In other words: your six-sigma quality-above-all approach to new product development may seem as if it’s doing good things to margins and the bottom line, but it’s preventing you from implementing an agile process that delivers far more.
In a fully agile firm – and most organizations today are far from that – just about everything seems mostly the same in name and function, but the CEO or CIO is seeing it through different eyes. The relative values of people, processes, and functions change, depending no longer on their value in delivering added profitability (that’s just the result), but rather on the speed and effectiveness with which they evolve (“change”) in a delicate dance with the evolution of customer needs.
What follows is a conceptualization of how one example of such a technology – data virtualization – might play a role in the agile firm’s IT. It is not that theoretical – it is based, among other things, on findings (that I have been allowed by my employer at the time, Aberdeen Group, to cite) about the agility of the information-handling of the typical large-scale organization. I find that, indeed, if anything, data virtualization – and here I must emphasize, if it is properly employed – plays a more vital role in the agile organization.
Information Agility Matters
One of the subtle ways that the ascendance of software in the typical organization has played out is that, wittingly or unwittingly, the typical firm is selling not just applications (solutions) but also information. By this, I don’t just mean selling data about customers, the revenues from which the Facebooks of the world fatten themselves on. I also mean using information to attract, sell, and follow-on sell customers, as well as selling information itself (product comparisons) to customers.
My usual example of this is Amazon’s ability to note customer preferences for certain books/music/authors/performers, and to use that information to pre-inform customers of upcoming releases, adding sales and increasing customer loyalty. The key characteristic of this type of information-based sale is knowledge of the customer that no one else can have, because it involves their sales interactions with you. And that, in turn, means that the information-led sell is on the increase, because the shelf life of application comparative advantage is getting shorter and shorter – but comparative advantage via proprietary information, properly nurtured, is practically unassailable. In the long run, applications keep you level with your competitors; information lets you carve out niches in which you are permanently ahead.
This, however, is the traditional world. What about the agile world? Well, to start with, the studies I cited earlier show that in almost any organization, information handling is not only almost comically un-agile but also almost comically ineffective. Think of information handling as an assembly line (yes, I know, but here it fits). More or less, the steps are:
1. Inhale the data into the organization (input)
2. Relate it to other data, including historical data, so the information in it is better understood (contextualize)
3. Send it into data stores so it is potentially available across the organization if needed (globalize)
4. Identify the appropriate people to whom this pre-contextualized information is potentially valuable, now and in the future, and send it as appropriate (target)
5. Display the information to the user in the proper context, including both corporate imperatives and his/her particular needs (customize)
6. Support ad-hoc additional end-user analysis, including non-information-system considerations (plan)
The agile information-handling process, at the least, needs to add one more task:
7. Constantly seek out new types of data outside the organization and use those new data types to drive changes in the information-handling process – as well, of course, as in the information products and the agile new-information-product-development processes (evolve).
The understandable but sad fact is that in traditional IT and business information-handling, information “leaks” and is lost at every stage of the process – not seriously if we only consider any one of the stages, but (by self-reports that seem reasonable) yielding loss of more than 2/3 of actionable information by the end of steps one through six. And “fixing” any one stage completely will only improve matters by about 11% -- after which, as new types of data arrive, that stage typically begins leaking again.
Now add agile task 7, and the situation is far worse. By self-reports three years ago (my sense is that things have only marginally improved), users inside an organization typically see important new types of data on average ½ year or more after they surface on the Web. Considering the demonstrated importance of things like social-media data to the average organization, as far as I’m concerned, that’s about the same as saying that one-half the information remaining after steps one to six lacks the up-to-date context to be truly useful. And so, the information leading to effective action that is supplied to the user is now about 17% of the information that should have been given to that user.
In other words, task 7 is the most important task of all. Stay agilely on top of the Web information about your customers or key changes in your regulatory, economic, and market environment, and it will be almost inevitable that you will find ways to improve your ability to get that information to the right users in the right way – as today’s agile ad-hoc analytics users (so-called business analysts) are suggesting is at least possible. Fail to deliver on task 7, and the “information gap” between you and your customers will remain, while your information-handling process falls further and further out of sync with the new mix of data, only periodically catching up again, and meanwhile doubling down on the wrong infrastructure. Ever wonder why it’s so hard to move your data to a public cloud? It’s not just the security concerns; it’s all that sunk cost in the corporate data warehouse.
In other words, in information agility, the truly agile get richer, and the not-quite-agile get poorer. Yes, information agility matters.
And So, Data Virtualization Matters
It is one of the oddities of the agile world that “evolution tools” such as more agile software development tools are not less valuable in a methodology that stresses “people over processes”, but more valuable. The reason why is encapsulated in an old saying: “lead, follow, or get out of the way.” In a world where tools should never lead, and tools that get out of the way are useless, evolution tools that follow the lead of the person doing the agile developing are priceless. That is, short of waving a magic wand, they are the fastest, easiest way for the developer to get from point A to defined point B to unexpected point C to newly discovered point D, etc. So one should never regard a tool such as Data Virtualization as “agile”, strictly speaking, but rather as the best way to support an agile process. Of course, if most of the time a tool is used that way, I’m fine with calling it “agile.”
Now let’s consider data virtualization as part of an agile “new information product development” process. In this process, the corporation’s traditional six-step information-handling process is the substructure (at least for now) of a business process aimed at fostering not just better reaction to information but also better creation and fine-tuning of combined app/information “products.”
Data virtualization has three key time-tested capabilities that can help in this process. First is auto-discovery. From the beginning, one of the nice side benefits of data virtualization has been that it will crawl your existing data stores – or, if instructed, the Web – and find new data sources and data types, fit them into an overall spectrum of data types (from unstructured to structured), and represent them abstractly in a metadata repository. In other words, data virtualization is a good place to start proactively searching out key new information as it sprouts outside “organizational boundaries”, because they have the longest experience and the best “best practices” at doing just that.
Data virtualization’s second key agile-tool capability is global contextualization. That is, data virtualization is not a cure-all for understanding all the relationships between an arriving piece of data and the data already in those data stores; but it does provide the most feasible way of pre-contextualizing today’s data. A metadata repository has been proven time and again to be the best real-world compromise between over-abstraction and zero context. A global metadata repository can handle any data type thrown at it. For global contextualization, a global metadata repository that has in it an abstraction of all your present-day key information is your most agile tool. It does have some remaining problems with evolving existing data types; but that’s a discussion for another day, not important until our information agility reaches a certain stage that is still far ahead.
Data virtualization’s third key agile-tool capability is the “database veneer.” This means that it allows end users, applications, and (although this part has not been used much) administrators to act as if all enterprise data stores were one gigantic data store, with near-real-time data access to any piece of data. It is a truism in agile development that the more high-level the covers-all-cases agile software on which you build, the more rapidly and effectively you can deliver added value. The database veneer that covers all data types, including ones that are added over time, means more agile development of both information and application products on top of the veneer. Again, as with the other two characteristics, data virtualization is not the only tool out there to do this; it’s just the one with a lot of experience and best practices to start with.
And, for some readers, that will bring up the subject of master data management, or MDM. In some agile areas, MDM is the logical extension of data virtualization, because it begins to tackle the problems of evolving old data types and integrating new ones in a particular use case (typically, customer data) – which is why it offers effectively identical global contextualization and database-veneer capabilities. However, because it does not typically have auto-discovery and is usually focused on a particular use case, it isn’t as good an agile-tool starting point as data virtualization. In agile information-product evolution, the less agile tool should follow the lead of the more agile one – or, to put it another way, a simple agile tool that covers all cases trumps an equivalently simple agile tool that only covers some.
So, let’s wrap it up: You can use data virtualization, today, in a more agile fashion than other available tools, as the basis of handling task 7, i.e., use it to auto-discover new data types out on the Web and integrate them continuously with the traditional information-handling process via global contextualization and the database veneer. This, in turn, will create pressure on the traditional process to become more agile, as IT tries to figure out how to “get ahead of the game” rather than drowning in a continuously-piling-up backlog of new-data-type requests. Of course, this approach doesn’t handle steps 4, 5, and 6 (target, customize, and plan); for those, you need to look to other types of tools – such as (if it ever grows up!) so-called agile BI. But data virtualization and its successors operate directly on task 7, directly following the lead of an agile information-product-evolution process. And so, in the long run, since information agility matters a lot, so does data virtualization.
Some IT-Buyer Considerations About Data Virtualization Vendors
To bring this discussion briefly down to a more practical level, if firms are really serious about business agility, then they should consider the present merits of various data virtualization vendors. I won’t favor one against another, except to say that I think that today’s smaller vendors – Composite Software, Denodo, and possibly Informatica – should be considered first, for one fundamental reason: they have survived over the last nine years in the face of competition from IBM among others.
The only way they could have done that, as far as I can see, is to constantly stay one step ahead in the kinds of data types that customers wanted them to support effectively. In other words, they were more agile – not necessarily very agile, but more agile than an IBM or an Oracle or a Sybase in this particular area. Maybe large vendors’ hype about improved agility will prove to be more than hype; but, as far as I’m concerned, they have to prove it to IT first, and not just in the features of their products, but also in the agility of their data-virtualization product development.
The Agile Buyer’s Not So Bottom Line
Sorry for the confusing heading, but it’s another opportunity to remind readers that business agility applies to everything, not just software development. Wherever agility goes, it is likely that speed and effectiveness of customer-coordinated change are the goals, and improving top and bottom lines the happy side-effect. And so with agile buyers of agile data-virtualization tools for information agility – look at the tool’s agility first.
To recap: the average organization badly needs information agility. Information agility, as in other areas, needs tools that “follow” the lead of the person(s) driving change. Data virtualization tools are best suited for that purpose as of now, both because they are best able to form the core of a lead-following information-product-evolution toolset, and because they integrate effectively with traditional information-handling and therefore speed its transformation into part of the agile process. Today’s smaller firms, such as Composite Software and Denodo, appear to be farther along the path to lead-following than other alternatives.
Above all: just because we have been focused on software development and related organizational functions, that doesn’t mean that information agility isn’t equally important. Isn’t it about time that you did some thought experiments of your own about information agility? And about data virtualization’s role in information agility?
For a decade, I have been touting the real-world value-add of what is now called data virtualization – the ability to see disparate data stores as one, and to perform real-time data processing, reporting, and analytics on the “global” data store – in the traditional world of IT and un-agile enterprises. However, in recent studies of agile methodologies within software development and in other areas of the enterprise, I have found that technologies and solutions well-adapted to deliver risk management and new product development in the traditional short-run-cost/revenue enterprises of the past can actually inhibit the agility and therefore the performance of an enterprise seeking the added advantage that true agility brings. In other words: your six-sigma quality-above-all approach to new product development may seem as if it’s doing good things to margins and the bottom line, but it’s preventing you from implementing an agile process that delivers far more.
In a fully agile firm – and most organizations today are far from that – just about everything seems mostly the same in name and function, but the CEO or CIO is seeing it through different eyes. The relative values of people, processes, and functions change, depending no longer on their value in delivering added profitability (that’s just the result), but rather on the speed and effectiveness with which they evolve (“change”) in a delicate dance with the evolution of customer needs.
What follows is a conceptualization of how one example of such a technology – data virtualization – might play a role in the agile firm’s IT. It is not that theoretical – it is based, among other things, on findings (that I have been allowed by my employer at the time, Aberdeen Group, to cite) about the agility of the information-handling of the typical large-scale organization. I find that, indeed, if anything, data virtualization – and here I must emphasize, if it is properly employed – plays a more vital role in the agile organization.
Information Agility Matters
One of the subtle ways that the ascendance of software in the typical organization has played out is that, wittingly or unwittingly, the typical firm is selling not just applications (solutions) but also information. By this, I don’t just mean selling data about customers, the revenues from which the Facebooks of the world fatten themselves on. I also mean using information to attract, sell, and follow-on sell customers, as well as selling information itself (product comparisons) to customers.
My usual example of this is Amazon’s ability to note customer preferences for certain books/music/authors/performers, and to use that information to pre-inform customers of upcoming releases, adding sales and increasing customer loyalty. The key characteristic of this type of information-based sale is knowledge of the customer that no one else can have, because it involves their sales interactions with you. And that, in turn, means that the information-led sell is on the increase, because the shelf life of application comparative advantage is getting shorter and shorter – but comparative advantage via proprietary information, properly nurtured, is practically unassailable. In the long run, applications keep you level with your competitors; information lets you carve out niches in which you are permanently ahead.
This, however, is the traditional world. What about the agile world? Well, to start with, the studies I cited earlier show that in almost any organization, information handling is not only almost comically un-agile but also almost comically ineffective. Think of information handling as an assembly line (yes, I know, but here it fits). More or less, the steps are:
1. Inhale the data into the organization (input)
2. Relate it to other data, including historical data, so the information in it is better understood (contextualize)
3. Send it into data stores so it is potentially available across the organization if needed (globalize)
4. Identify the appropriate people to whom this pre-contextualized information is potentially valuable, now and in the future, and send it as appropriate (target)
5. Display the information to the user in the proper context, including both corporate imperatives and his/her particular needs (customize)
6. Support ad-hoc additional end-user analysis, including non-information-system considerations (plan)
The agile information-handling process, at the least, needs to add one more task:
7. Constantly seek out new types of data outside the organization and use those new data types to drive changes in the information-handling process – as well, of course, as in the information products and the agile new-information-product-development processes (evolve).
The understandable but sad fact is that in traditional IT and business information-handling, information “leaks” and is lost at every stage of the process – not seriously if we only consider any one of the stages, but (by self-reports that seem reasonable) yielding loss of more than 2/3 of actionable information by the end of steps one through six. And “fixing” any one stage completely will only improve matters by about 11% -- after which, as new types of data arrive, that stage typically begins leaking again.
Now add agile task 7, and the situation is far worse. By self-reports three years ago (my sense is that things have only marginally improved), users inside an organization typically see important new types of data on average ½ year or more after they surface on the Web. Considering the demonstrated importance of things like social-media data to the average organization, as far as I’m concerned, that’s about the same as saying that one-half the information remaining after steps one to six lacks the up-to-date context to be truly useful. And so, the information leading to effective action that is supplied to the user is now about 17% of the information that should have been given to that user.
In other words, task 7 is the most important task of all. Stay agilely on top of the Web information about your customers or key changes in your regulatory, economic, and market environment, and it will be almost inevitable that you will find ways to improve your ability to get that information to the right users in the right way – as today’s agile ad-hoc analytics users (so-called business analysts) are suggesting is at least possible. Fail to deliver on task 7, and the “information gap” between you and your customers will remain, while your information-handling process falls further and further out of sync with the new mix of data, only periodically catching up again, and meanwhile doubling down on the wrong infrastructure. Ever wonder why it’s so hard to move your data to a public cloud? It’s not just the security concerns; it’s all that sunk cost in the corporate data warehouse.
In other words, in information agility, the truly agile get richer, and the not-quite-agile get poorer. Yes, information agility matters.
And So, Data Virtualization Matters
It is one of the oddities of the agile world that “evolution tools” such as more agile software development tools are not less valuable in a methodology that stresses “people over processes”, but more valuable. The reason why is encapsulated in an old saying: “lead, follow, or get out of the way.” In a world where tools should never lead, and tools that get out of the way are useless, evolution tools that follow the lead of the person doing the agile developing are priceless. That is, short of waving a magic wand, they are the fastest, easiest way for the developer to get from point A to defined point B to unexpected point C to newly discovered point D, etc. So one should never regard a tool such as Data Virtualization as “agile”, strictly speaking, but rather as the best way to support an agile process. Of course, if most of the time a tool is used that way, I’m fine with calling it “agile.”
Now let’s consider data virtualization as part of an agile “new information product development” process. In this process, the corporation’s traditional six-step information-handling process is the substructure (at least for now) of a business process aimed at fostering not just better reaction to information but also better creation and fine-tuning of combined app/information “products.”
Data virtualization has three key time-tested capabilities that can help in this process. First is auto-discovery. From the beginning, one of the nice side benefits of data virtualization has been that it will crawl your existing data stores – or, if instructed, the Web – and find new data sources and data types, fit them into an overall spectrum of data types (from unstructured to structured), and represent them abstractly in a metadata repository. In other words, data virtualization is a good place to start proactively searching out key new information as it sprouts outside “organizational boundaries”, because they have the longest experience and the best “best practices” at doing just that.
Data virtualization’s second key agile-tool capability is global contextualization. That is, data virtualization is not a cure-all for understanding all the relationships between an arriving piece of data and the data already in those data stores; but it does provide the most feasible way of pre-contextualizing today’s data. A metadata repository has been proven time and again to be the best real-world compromise between over-abstraction and zero context. A global metadata repository can handle any data type thrown at it. For global contextualization, a global metadata repository that has in it an abstraction of all your present-day key information is your most agile tool. It does have some remaining problems with evolving existing data types; but that’s a discussion for another day, not important until our information agility reaches a certain stage that is still far ahead.
Data virtualization’s third key agile-tool capability is the “database veneer.” This means that it allows end users, applications, and (although this part has not been used much) administrators to act as if all enterprise data stores were one gigantic data store, with near-real-time data access to any piece of data. It is a truism in agile development that the more high-level the covers-all-cases agile software on which you build, the more rapidly and effectively you can deliver added value. The database veneer that covers all data types, including ones that are added over time, means more agile development of both information and application products on top of the veneer. Again, as with the other two characteristics, data virtualization is not the only tool out there to do this; it’s just the one with a lot of experience and best practices to start with.
And, for some readers, that will bring up the subject of master data management, or MDM. In some agile areas, MDM is the logical extension of data virtualization, because it begins to tackle the problems of evolving old data types and integrating new ones in a particular use case (typically, customer data) – which is why it offers effectively identical global contextualization and database-veneer capabilities. However, because it does not typically have auto-discovery and is usually focused on a particular use case, it isn’t as good an agile-tool starting point as data virtualization. In agile information-product evolution, the less agile tool should follow the lead of the more agile one – or, to put it another way, a simple agile tool that covers all cases trumps an equivalently simple agile tool that only covers some.
So, let’s wrap it up: You can use data virtualization, today, in a more agile fashion than other available tools, as the basis of handling task 7, i.e., use it to auto-discover new data types out on the Web and integrate them continuously with the traditional information-handling process via global contextualization and the database veneer. This, in turn, will create pressure on the traditional process to become more agile, as IT tries to figure out how to “get ahead of the game” rather than drowning in a continuously-piling-up backlog of new-data-type requests. Of course, this approach doesn’t handle steps 4, 5, and 6 (target, customize, and plan); for those, you need to look to other types of tools – such as (if it ever grows up!) so-called agile BI. But data virtualization and its successors operate directly on task 7, directly following the lead of an agile information-product-evolution process. And so, in the long run, since information agility matters a lot, so does data virtualization.
Some IT-Buyer Considerations About Data Virtualization Vendors
To bring this discussion briefly down to a more practical level, if firms are really serious about business agility, then they should consider the present merits of various data virtualization vendors. I won’t favor one against another, except to say that I think that today’s smaller vendors – Composite Software, Denodo, and possibly Informatica – should be considered first, for one fundamental reason: they have survived over the last nine years in the face of competition from IBM among others.
The only way they could have done that, as far as I can see, is to constantly stay one step ahead in the kinds of data types that customers wanted them to support effectively. In other words, they were more agile – not necessarily very agile, but more agile than an IBM or an Oracle or a Sybase in this particular area. Maybe large vendors’ hype about improved agility will prove to be more than hype; but, as far as I’m concerned, they have to prove it to IT first, and not just in the features of their products, but also in the agility of their data-virtualization product development.
The Agile Buyer’s Not So Bottom Line
Sorry for the confusing heading, but it’s another opportunity to remind readers that business agility applies to everything, not just software development. Wherever agility goes, it is likely that speed and effectiveness of customer-coordinated change are the goals, and improving top and bottom lines the happy side-effect. And so with agile buyers of agile data-virtualization tools for information agility – look at the tool’s agility first.
To recap: the average organization badly needs information agility. Information agility, as in other areas, needs tools that “follow” the lead of the person(s) driving change. Data virtualization tools are best suited for that purpose as of now, both because they are best able to form the core of a lead-following information-product-evolution toolset, and because they integrate effectively with traditional information-handling and therefore speed its transformation into part of the agile process. Today’s smaller firms, such as Composite Software and Denodo, appear to be farther along the path to lead-following than other alternatives.
Above all: just because we have been focused on software development and related organizational functions, that doesn’t mean that information agility isn’t equally important. Isn’t it about time that you did some thought experiments of your own about information agility? And about data virtualization’s role in information agility?
Subscribe to:
Posts (Atom)