In an agile world, what is the value of data virtualization?
For a decade, I have been touting the real-world value-add of what is now called data virtualization – the ability to see disparate data stores as one, and to perform real-time data processing, reporting, and analytics on the “global” data store – in the traditional world of IT and un-agile enterprises. However, in recent studies of agile methodologies within software development and in other areas of the enterprise, I have found that technologies and solutions well-adapted to deliver risk management and new product development in the traditional short-run-cost/revenue enterprises of the past can actually inhibit the agility and therefore the performance of an enterprise seeking the added advantage that true agility brings. In other words: your six-sigma quality-above-all approach to new product development may seem as if it’s doing good things to margins and the bottom line, but it’s preventing you from implementing an agile process that delivers far more.
In a fully agile firm – and most organizations today are far from that – just about everything seems mostly the same in name and function, but the CEO or CIO is seeing it through different eyes. The relative values of people, processes, and functions change, depending no longer on their value in delivering added profitability (that’s just the result), but rather on the speed and effectiveness with which they evolve (“change”) in a delicate dance with the evolution of customer needs.
What follows is a conceptualization of how one example of such a technology – data virtualization – might play a role in the agile firm’s IT. It is not that theoretical – it is based, among other things, on findings (that I have been allowed by my employer at the time, Aberdeen Group, to cite) about the agility of the information-handling of the typical large-scale organization. I find that, indeed, if anything, data virtualization – and here I must emphasize, if it is properly employed – plays a more vital role in the agile organization.
Information Agility Matters
One of the subtle ways that the ascendance of software in the typical organization has played out is that, wittingly or unwittingly, the typical firm is selling not just applications (solutions) but also information. By this, I don’t just mean selling data about customers, the revenues from which the Facebooks of the world fatten themselves on. I also mean using information to attract, sell, and follow-on sell customers, as well as selling information itself (product comparisons) to customers.
My usual example of this is Amazon’s ability to note customer preferences for certain books/music/authors/performers, and to use that information to pre-inform customers of upcoming releases, adding sales and increasing customer loyalty. The key characteristic of this type of information-based sale is knowledge of the customer that no one else can have, because it involves their sales interactions with you. And that, in turn, means that the information-led sell is on the increase, because the shelf life of application comparative advantage is getting shorter and shorter – but comparative advantage via proprietary information, properly nurtured, is practically unassailable. In the long run, applications keep you level with your competitors; information lets you carve out niches in which you are permanently ahead.
This, however, is the traditional world. What about the agile world? Well, to start with, the studies I cited earlier show that in almost any organization, information handling is not only almost comically un-agile but also almost comically ineffective. Think of information handling as an assembly line (yes, I know, but here it fits). More or less, the steps are:
1. Inhale the data into the organization (input)
2. Relate it to other data, including historical data, so the information in it is better understood (contextualize)
3. Send it into data stores so it is potentially available across the organization if needed (globalize)
4. Identify the appropriate people to whom this pre-contextualized information is potentially valuable, now and in the future, and send it as appropriate (target)
5. Display the information to the user in the proper context, including both corporate imperatives and his/her particular needs (customize)
6. Support ad-hoc additional end-user analysis, including non-information-system considerations (plan)
The agile information-handling process, at the least, needs to add one more task:
7. Constantly seek out new types of data outside the organization and use those new data types to drive changes in the information-handling process – as well, of course, as in the information products and the agile new-information-product-development processes (evolve).
The understandable but sad fact is that in traditional IT and business information-handling, information “leaks” and is lost at every stage of the process – not seriously if we only consider any one of the stages, but (by self-reports that seem reasonable) yielding loss of more than 2/3 of actionable information by the end of steps one through six. And “fixing” any one stage completely will only improve matters by about 11% -- after which, as new types of data arrive, that stage typically begins leaking again.
Now add agile task 7, and the situation is far worse. By self-reports three years ago (my sense is that things have only marginally improved), users inside an organization typically see important new types of data on average ½ year or more after they surface on the Web. Considering the demonstrated importance of things like social-media data to the average organization, as far as I’m concerned, that’s about the same as saying that one-half the information remaining after steps one to six lacks the up-to-date context to be truly useful. And so, the information leading to effective action that is supplied to the user is now about 17% of the information that should have been given to that user.
In other words, task 7 is the most important task of all. Stay agilely on top of the Web information about your customers or key changes in your regulatory, economic, and market environment, and it will be almost inevitable that you will find ways to improve your ability to get that information to the right users in the right way – as today’s agile ad-hoc analytics users (so-called business analysts) are suggesting is at least possible. Fail to deliver on task 7, and the “information gap” between you and your customers will remain, while your information-handling process falls further and further out of sync with the new mix of data, only periodically catching up again, and meanwhile doubling down on the wrong infrastructure. Ever wonder why it’s so hard to move your data to a public cloud? It’s not just the security concerns; it’s all that sunk cost in the corporate data warehouse.
In other words, in information agility, the truly agile get richer, and the not-quite-agile get poorer. Yes, information agility matters.
And So, Data Virtualization Matters
It is one of the oddities of the agile world that “evolution tools” such as more agile software development tools are not less valuable in a methodology that stresses “people over processes”, but more valuable. The reason why is encapsulated in an old saying: “lead, follow, or get out of the way.” In a world where tools should never lead, and tools that get out of the way are useless, evolution tools that follow the lead of the person doing the agile developing are priceless. That is, short of waving a magic wand, they are the fastest, easiest way for the developer to get from point A to defined point B to unexpected point C to newly discovered point D, etc. So one should never regard a tool such as Data Virtualization as “agile”, strictly speaking, but rather as the best way to support an agile process. Of course, if most of the time a tool is used that way, I’m fine with calling it “agile.”
Now let’s consider data virtualization as part of an agile “new information product development” process. In this process, the corporation’s traditional six-step information-handling process is the substructure (at least for now) of a business process aimed at fostering not just better reaction to information but also better creation and fine-tuning of combined app/information “products.”
Data virtualization has three key time-tested capabilities that can help in this process. First is auto-discovery. From the beginning, one of the nice side benefits of data virtualization has been that it will crawl your existing data stores – or, if instructed, the Web – and find new data sources and data types, fit them into an overall spectrum of data types (from unstructured to structured), and represent them abstractly in a metadata repository. In other words, data virtualization is a good place to start proactively searching out key new information as it sprouts outside “organizational boundaries”, because they have the longest experience and the best “best practices” at doing just that.
Data virtualization’s second key agile-tool capability is global contextualization. That is, data virtualization is not a cure-all for understanding all the relationships between an arriving piece of data and the data already in those data stores; but it does provide the most feasible way of pre-contextualizing today’s data. A metadata repository has been proven time and again to be the best real-world compromise between over-abstraction and zero context. A global metadata repository can handle any data type thrown at it. For global contextualization, a global metadata repository that has in it an abstraction of all your present-day key information is your most agile tool. It does have some remaining problems with evolving existing data types; but that’s a discussion for another day, not important until our information agility reaches a certain stage that is still far ahead.
Data virtualization’s third key agile-tool capability is the “database veneer.” This means that it allows end users, applications, and (although this part has not been used much) administrators to act as if all enterprise data stores were one gigantic data store, with near-real-time data access to any piece of data. It is a truism in agile development that the more high-level the covers-all-cases agile software on which you build, the more rapidly and effectively you can deliver added value. The database veneer that covers all data types, including ones that are added over time, means more agile development of both information and application products on top of the veneer. Again, as with the other two characteristics, data virtualization is not the only tool out there to do this; it’s just the one with a lot of experience and best practices to start with.
And, for some readers, that will bring up the subject of master data management, or MDM. In some agile areas, MDM is the logical extension of data virtualization, because it begins to tackle the problems of evolving old data types and integrating new ones in a particular use case (typically, customer data) – which is why it offers effectively identical global contextualization and database-veneer capabilities. However, because it does not typically have auto-discovery and is usually focused on a particular use case, it isn’t as good an agile-tool starting point as data virtualization. In agile information-product evolution, the less agile tool should follow the lead of the more agile one – or, to put it another way, a simple agile tool that covers all cases trumps an equivalently simple agile tool that only covers some.
So, let’s wrap it up: You can use data virtualization, today, in a more agile fashion than other available tools, as the basis of handling task 7, i.e., use it to auto-discover new data types out on the Web and integrate them continuously with the traditional information-handling process via global contextualization and the database veneer. This, in turn, will create pressure on the traditional process to become more agile, as IT tries to figure out how to “get ahead of the game” rather than drowning in a continuously-piling-up backlog of new-data-type requests. Of course, this approach doesn’t handle steps 4, 5, and 6 (target, customize, and plan); for those, you need to look to other types of tools – such as (if it ever grows up!) so-called agile BI. But data virtualization and its successors operate directly on task 7, directly following the lead of an agile information-product-evolution process. And so, in the long run, since information agility matters a lot, so does data virtualization.
Some IT-Buyer Considerations About Data Virtualization Vendors
To bring this discussion briefly down to a more practical level, if firms are really serious about business agility, then they should consider the present merits of various data virtualization vendors. I won’t favor one against another, except to say that I think that today’s smaller vendors – Composite Software, Denodo, and possibly Informatica – should be considered first, for one fundamental reason: they have survived over the last nine years in the face of competition from IBM among others.
The only way they could have done that, as far as I can see, is to constantly stay one step ahead in the kinds of data types that customers wanted them to support effectively. In other words, they were more agile – not necessarily very agile, but more agile than an IBM or an Oracle or a Sybase in this particular area. Maybe large vendors’ hype about improved agility will prove to be more than hype; but, as far as I’m concerned, they have to prove it to IT first, and not just in the features of their products, but also in the agility of their data-virtualization product development.
The Agile Buyer’s Not So Bottom Line
Sorry for the confusing heading, but it’s another opportunity to remind readers that business agility applies to everything, not just software development. Wherever agility goes, it is likely that speed and effectiveness of customer-coordinated change are the goals, and improving top and bottom lines the happy side-effect. And so with agile buyers of agile data-virtualization tools for information agility – look at the tool’s agility first.
To recap: the average organization badly needs information agility. Information agility, as in other areas, needs tools that “follow” the lead of the person(s) driving change. Data virtualization tools are best suited for that purpose as of now, both because they are best able to form the core of a lead-following information-product-evolution toolset, and because they integrate effectively with traditional information-handling and therefore speed its transformation into part of the agile process. Today’s smaller firms, such as Composite Software and Denodo, appear to be farther along the path to lead-following than other alternatives.
Above all: just because we have been focused on software development and related organizational functions, that doesn’t mean that information agility isn’t equally important. Isn’t it about time that you did some thought experiments of your own about information agility? And about data virtualization’s role in information agility?
Showing posts with label agile BI. Show all posts
Showing posts with label agile BI. Show all posts
Friday, March 16, 2012
Monday, January 30, 2012
The Other BI: Orange EDA and Statistical Analytics
This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about BI today.
The Importance of Orange-Type Statistical Analysis to Analytics
BI has taken a major step forward in maturity over the last few years, as statistical packages have become more associated with analytics. Granted, SAS has for years distinguished itself by its statistics-focused BI solution; but when IBM recently acquired SPSS, the grand-daddy of statistical packages, the importance of more rigorous analysis of company and customer data seemed both confirmed and more obvious. Moreover, over the years, data miners have begun to draw on the insights of university researchers about things like “data mining bias” and Bayesian statistics – and the most in-depth, competitive-advantage-determining analyses have benefited as a result. And so, it would seem that we are on a nice technology glide path, as statistics completes the flexibility of analytics by covering one extreme of certainty and analytical complexity, while traditional analytics tools cover the rest of the spectrum up from situations where shallow and imprecise analysis is appropriate, and as statistical techniques filter down by technology evolution to the “unwashed masses” of end users. Or are we?
You see, there is a glaring gap in this picture of increasing knowledge of what’s going on – or at least a gap that should be glaring. This gap might be summed up as Alice in Wonderland’s “verdict first, then the trial”, or business’ “when you have a hammer, everything looks like a nail.” Both the business and the researcher start with their own narrow picture of what the customer or research subject should look like, and the analytics and statistics that start with such hypotheses are designed to narrow in on a solution rather than expand due to unexpected data, and so the business/researcher is very likely to miss key customer insights, psychological and otherwise. Pile on top of this the “not invented here” syndrome characteristic of most enterprises, and the “confirmation bias” that recent research has shown to be prevalent among individuals and organizations, and you have a real analytical problem on your hands.
This is not a purely theoretical problem, if you will excuse the bad joke. In the psychological statistics area, the recent popularity of “qualitative methods” has exposed, to those who are willing to see, the enormous amount of insights that traditional statistics fails to capture about customer psychology, sociology, and behavior. Both approaches, of course, would seem to suffer from the deficit that Richard Feynman pointed out – the lack of control groups that renders any conclusion suspect because a “placebo” or “Hawthorne” effect may be involved – but it should be noted that even when (as seems to be happening) this problem is compensated for, the “verdict first” problem remains, because the world of people is far less easy to pre-define than that of nuclear physics.
In the world of business, as I can personally attest, the same type of problem exists in data-gathering. For more than a decade, I have run TCO studies, particularly on SMB use of databases. I discovered early on that open-ended interviews of relatively few sysadmins were far more effective in capturing the real costs of databases than far wider-spread on-a-scale-from-1-to-5 inflexible surveys of CIOs. Moreover, if I just included the ability of the interviewee to tell a story from his or her point of view, the respondent would consistently come up with an insight of extraordinary value, such as the idea that SMBs didn’t care so much about technology that saved operational costs as much as technology that saved a local-office head time by requiring him or her to just press a button as he or she shut off the lights on Saturday night. The key to success for my “surveys” was that they were designed to be open-ended (able to go in a new direction during the interview, and leaving space for whatever the interviewer might have left out), interviewee-driven (they started by letting the interviewee tell a story as he or she saw it), and flexible in the kind of data collected (typically, an IT organization did not know the overall costs of database administration for their organization [and in a survey, they would have guessed – badly], but they almost invariably knew how many database instances per administrator).
As it turns out, there is a comparable statistical approach for the data-analysis side of things. It’s called Exploratory Data Analysis, or EDA.
As it has evolved in the decades since John Tukey first popularized it, EDA is about analyzing smaller amounts of data to generate as many plausible hypotheses (or “patterns in the data”) as possible, before winnowing them down with further data. To further clear the statistical researcher’s mind of bias, the technique creates abstract unlabeled visualizations (“data visualization”) of the patterns, such as the strangely-named box-and-whisker plot. The analysis is not deep – but it identifies far more hypotheses, and therefore quite a few more areas where in-depth analysis may reveal key insights. The automation of these techniques has made the application of EDA a minor blip in the average analyst’s process, and so effective use of EDA should yield a major improvement in analytics effectiveness “at the margin” (in the resulting in-depth analyses) for a very small time “overhead cost.” In fact, EDA has reached the point, as in the Orange open-source solution, where it is merged with a full-fledged data-mining tool.
And yet, I find that most in university research and in industry are barely aware that EDA exists, much less that it might have some significant use. For a while, SAS’ JMP product stood bravely and alone as a tool that could at least potentially be used by businesses – but I note that according to Wikipedia they have recently discontinued support for its use on Linux.
So let’s summarize: EDA is out there. It’s easy to use. Now that statistical analysis in general is creeping into greater use in analytics, users are ready for it. I fully anticipate that it would have major positive effects on in-depth analytics for enterprises from the very largest down at least to the larger medium-sized ones. IT shops will have to do some customization and integration themselves, because most if not all vendors have not yet fully integrated it as part of the analytics process in their BI suites; but with open-source and other “standard” EDA tools, that’s not inordinately difficult. The only thing lacking is for somebody, anybody, to wake up and pay attention.
The Relevance of Orange EDA to Statistical-Analysis-Type BI
Orange’s relevance may already be apparent from the above, but I’ll say it again anyway. Orange’s EDA solution includes integration with enterprise-type data-mining analytics, and supports a wide range of data visualization techniques, making it a leadership supplier in “fit to your enterprise’s analytics.” Orange is open source, which means it’s as cheap as you can get for quick-and-dirty, and also means it’s not going to go away. Most importantly, Orange lays down a solid, relatively standardized foundation that should be easy to incorporate or upgrade from, when someday the major vendors finally move into the area and provide fancier techniques and better integration with a full-fledged BI suite. That’s all; and that’s plenty.
Potential Uses of Orange-Type EDA in Analytics for IT
Since IT will need to do some of the initial legwork here, without the usual help from one’s BI supplier, the most effective initial use of Orange-type EDA is in support of the longer-term efforts of today’s business analysts, and not in IT-driven agile BI. However, IT should find these business analysts to be surprisingly receptive – or, at the least, as recent surveys suggest, amazed that IT isn’t being a “boat anchor” yet again. You see, EDA has a sheen of “innovation” about it, and so folks who are in some way associated with the business’ “innovation” efforts should like it a lot. The rest is simply a matter of its becoming part of these business analysts' steadily accumulating toolkit of rapid-query-generation and statistical-in-depth-insight-at-the-margin tools. EDA may not in the normal course of usage get the glory of notice as the source of a new competition-killer; but with a little assiduous use-case monitoring by IT, the business case can be made.
It is equally important for IT to note that EDA is twice as effective if it is joined at the front end by a data-gathering process that is to a much greater extent (to recap) open-ended, customer-driven, and flexible (in fact, agile) in the type of data gathered. Remember, there are ways of doing this – such as parallel in-depth customer interviews or Internet surveys that don’t just parrot SurveyMonkey – that add very little “overhead” to data-gathering. IT should seriously consider doing this as well, and preferably design the data-gathering process so as to feed the gathered data to Orange-type EDA tools where in-depth statistical analysis of that data will probably be appropriate as the next step. The overall effect will be like replacing a steadily narrowing view of the data with one that expands the potential analyses until the right balance between “data blindness” and “paralysis by analysis” risks is reached.
The Bottom Line for IT Buyers
To view Orange-type EDA as comparable to the other BI technologies/solutions I have discussed so far is to miss the point. EDA is much more like agile development – its main value lies in changing our analytics methodology, not in improving analytics itself. It helps the organization itself to think not “outside the box”, but “outside the organization” – to be able to combine the viewpoint of the vendor with the viewpoint and reality of the customer, rather than trying to force customer interactions into corporate fantasies of the way customers should think and act for maximum vendor profit. We have all seen the major public-relations disaster of Bank of America charges for debit cards – one that, if we were honest, we would admit most other enterprises find it all too easy to stumble into. If EDA (or, better still, EDA plus open-ended, customer-driven, flexible data-gathering) prevents only one such misstep, it will have paid for itself ten times over, no matter what the numbers say. In a nutshell: EDA seems like it’s about competitive advantage; that’s true as far as it goes, but EDA is actually much more about business risk.
The Orange value proposition for such uses of EDA has been noted twice already; no need to repeat it a third time. For IT buyers, it simply means that any time you decide to do EDA, Orange is there as part of a rather short short list. So that leaves the IT buyer’s final question: what’s the hurry?
And, of course, since EDA is about competitive advantage (sarcasm), there is no hurry. Unless you consider the possibility that each non-EDA enterprise is a bit like a drunk staggering along a sidewalk who has just knocked over the fence bordering an abyss, and who if he then happens to stagger over the edge is busy blaming the owner of the fence (the CEO?) all the way to the bottom. That abyss is the risk of offending the customer. That inebriation is business as usual. EDA helps you sober up, fast.
I can’t say that you have to implement EDA now or you’ll fall. But do you really want to risk doing nothing?
The Importance of Orange-Type Statistical Analysis to Analytics
BI has taken a major step forward in maturity over the last few years, as statistical packages have become more associated with analytics. Granted, SAS has for years distinguished itself by its statistics-focused BI solution; but when IBM recently acquired SPSS, the grand-daddy of statistical packages, the importance of more rigorous analysis of company and customer data seemed both confirmed and more obvious. Moreover, over the years, data miners have begun to draw on the insights of university researchers about things like “data mining bias” and Bayesian statistics – and the most in-depth, competitive-advantage-determining analyses have benefited as a result. And so, it would seem that we are on a nice technology glide path, as statistics completes the flexibility of analytics by covering one extreme of certainty and analytical complexity, while traditional analytics tools cover the rest of the spectrum up from situations where shallow and imprecise analysis is appropriate, and as statistical techniques filter down by technology evolution to the “unwashed masses” of end users. Or are we?
You see, there is a glaring gap in this picture of increasing knowledge of what’s going on – or at least a gap that should be glaring. This gap might be summed up as Alice in Wonderland’s “verdict first, then the trial”, or business’ “when you have a hammer, everything looks like a nail.” Both the business and the researcher start with their own narrow picture of what the customer or research subject should look like, and the analytics and statistics that start with such hypotheses are designed to narrow in on a solution rather than expand due to unexpected data, and so the business/researcher is very likely to miss key customer insights, psychological and otherwise. Pile on top of this the “not invented here” syndrome characteristic of most enterprises, and the “confirmation bias” that recent research has shown to be prevalent among individuals and organizations, and you have a real analytical problem on your hands.
This is not a purely theoretical problem, if you will excuse the bad joke. In the psychological statistics area, the recent popularity of “qualitative methods” has exposed, to those who are willing to see, the enormous amount of insights that traditional statistics fails to capture about customer psychology, sociology, and behavior. Both approaches, of course, would seem to suffer from the deficit that Richard Feynman pointed out – the lack of control groups that renders any conclusion suspect because a “placebo” or “Hawthorne” effect may be involved – but it should be noted that even when (as seems to be happening) this problem is compensated for, the “verdict first” problem remains, because the world of people is far less easy to pre-define than that of nuclear physics.
In the world of business, as I can personally attest, the same type of problem exists in data-gathering. For more than a decade, I have run TCO studies, particularly on SMB use of databases. I discovered early on that open-ended interviews of relatively few sysadmins were far more effective in capturing the real costs of databases than far wider-spread on-a-scale-from-1-to-5 inflexible surveys of CIOs. Moreover, if I just included the ability of the interviewee to tell a story from his or her point of view, the respondent would consistently come up with an insight of extraordinary value, such as the idea that SMBs didn’t care so much about technology that saved operational costs as much as technology that saved a local-office head time by requiring him or her to just press a button as he or she shut off the lights on Saturday night. The key to success for my “surveys” was that they were designed to be open-ended (able to go in a new direction during the interview, and leaving space for whatever the interviewer might have left out), interviewee-driven (they started by letting the interviewee tell a story as he or she saw it), and flexible in the kind of data collected (typically, an IT organization did not know the overall costs of database administration for their organization [and in a survey, they would have guessed – badly], but they almost invariably knew how many database instances per administrator).
As it turns out, there is a comparable statistical approach for the data-analysis side of things. It’s called Exploratory Data Analysis, or EDA.
As it has evolved in the decades since John Tukey first popularized it, EDA is about analyzing smaller amounts of data to generate as many plausible hypotheses (or “patterns in the data”) as possible, before winnowing them down with further data. To further clear the statistical researcher’s mind of bias, the technique creates abstract unlabeled visualizations (“data visualization”) of the patterns, such as the strangely-named box-and-whisker plot. The analysis is not deep – but it identifies far more hypotheses, and therefore quite a few more areas where in-depth analysis may reveal key insights. The automation of these techniques has made the application of EDA a minor blip in the average analyst’s process, and so effective use of EDA should yield a major improvement in analytics effectiveness “at the margin” (in the resulting in-depth analyses) for a very small time “overhead cost.” In fact, EDA has reached the point, as in the Orange open-source solution, where it is merged with a full-fledged data-mining tool.
And yet, I find that most in university research and in industry are barely aware that EDA exists, much less that it might have some significant use. For a while, SAS’ JMP product stood bravely and alone as a tool that could at least potentially be used by businesses – but I note that according to Wikipedia they have recently discontinued support for its use on Linux.
So let’s summarize: EDA is out there. It’s easy to use. Now that statistical analysis in general is creeping into greater use in analytics, users are ready for it. I fully anticipate that it would have major positive effects on in-depth analytics for enterprises from the very largest down at least to the larger medium-sized ones. IT shops will have to do some customization and integration themselves, because most if not all vendors have not yet fully integrated it as part of the analytics process in their BI suites; but with open-source and other “standard” EDA tools, that’s not inordinately difficult. The only thing lacking is for somebody, anybody, to wake up and pay attention.
The Relevance of Orange EDA to Statistical-Analysis-Type BI
Orange’s relevance may already be apparent from the above, but I’ll say it again anyway. Orange’s EDA solution includes integration with enterprise-type data-mining analytics, and supports a wide range of data visualization techniques, making it a leadership supplier in “fit to your enterprise’s analytics.” Orange is open source, which means it’s as cheap as you can get for quick-and-dirty, and also means it’s not going to go away. Most importantly, Orange lays down a solid, relatively standardized foundation that should be easy to incorporate or upgrade from, when someday the major vendors finally move into the area and provide fancier techniques and better integration with a full-fledged BI suite. That’s all; and that’s plenty.
Potential Uses of Orange-Type EDA in Analytics for IT
Since IT will need to do some of the initial legwork here, without the usual help from one’s BI supplier, the most effective initial use of Orange-type EDA is in support of the longer-term efforts of today’s business analysts, and not in IT-driven agile BI. However, IT should find these business analysts to be surprisingly receptive – or, at the least, as recent surveys suggest, amazed that IT isn’t being a “boat anchor” yet again. You see, EDA has a sheen of “innovation” about it, and so folks who are in some way associated with the business’ “innovation” efforts should like it a lot. The rest is simply a matter of its becoming part of these business analysts' steadily accumulating toolkit of rapid-query-generation and statistical-in-depth-insight-at-the-margin tools. EDA may not in the normal course of usage get the glory of notice as the source of a new competition-killer; but with a little assiduous use-case monitoring by IT, the business case can be made.
It is equally important for IT to note that EDA is twice as effective if it is joined at the front end by a data-gathering process that is to a much greater extent (to recap) open-ended, customer-driven, and flexible (in fact, agile) in the type of data gathered. Remember, there are ways of doing this – such as parallel in-depth customer interviews or Internet surveys that don’t just parrot SurveyMonkey – that add very little “overhead” to data-gathering. IT should seriously consider doing this as well, and preferably design the data-gathering process so as to feed the gathered data to Orange-type EDA tools where in-depth statistical analysis of that data will probably be appropriate as the next step. The overall effect will be like replacing a steadily narrowing view of the data with one that expands the potential analyses until the right balance between “data blindness” and “paralysis by analysis” risks is reached.
The Bottom Line for IT Buyers
To view Orange-type EDA as comparable to the other BI technologies/solutions I have discussed so far is to miss the point. EDA is much more like agile development – its main value lies in changing our analytics methodology, not in improving analytics itself. It helps the organization itself to think not “outside the box”, but “outside the organization” – to be able to combine the viewpoint of the vendor with the viewpoint and reality of the customer, rather than trying to force customer interactions into corporate fantasies of the way customers should think and act for maximum vendor profit. We have all seen the major public-relations disaster of Bank of America charges for debit cards – one that, if we were honest, we would admit most other enterprises find it all too easy to stumble into. If EDA (or, better still, EDA plus open-ended, customer-driven, flexible data-gathering) prevents only one such misstep, it will have paid for itself ten times over, no matter what the numbers say. In a nutshell: EDA seems like it’s about competitive advantage; that’s true as far as it goes, but EDA is actually much more about business risk.
The Orange value proposition for such uses of EDA has been noted twice already; no need to repeat it a third time. For IT buyers, it simply means that any time you decide to do EDA, Orange is there as part of a rather short short list. So that leaves the IT buyer’s final question: what’s the hurry?
And, of course, since EDA is about competitive advantage (sarcasm), there is no hurry. Unless you consider the possibility that each non-EDA enterprise is a bit like a drunk staggering along a sidewalk who has just knocked over the fence bordering an abyss, and who if he then happens to stagger over the edge is busy blaming the owner of the fence (the CEO?) all the way to the bottom. That abyss is the risk of offending the customer. That inebriation is business as usual. EDA helps you sober up, fast.
I can’t say that you have to implement EDA now or you’ll fall. But do you really want to risk doing nothing?
Friday, January 20, 2012
The Other Agile Development: AccuRev SCM and Code Refactoring
This blog post highlights a software company and technology that I view as potentially useful to organizations investing in agile development, agile new product development, and business agility over the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about agile development today.
The Importance of Code Refactoring to Agile Development
Ever since agile development first surfaced in the Agile Manifesto, code refactoring has been an under-appreciated and often-ignored part of its value-add. My own fascination with the subject dates back to my Cornell days, when research there in how to find minimal fixes to program compiler errors led inevitably to the question: what is the best code design to allow minimal-effort additions to a program’s features? In the 1980s, research into “structured design” showed that modularized subroutines at the same level were far easier to extend than subroutines in a complex tree structure. When I first became aware of the agile development movement, I immediately saw the benefit of rapid program extension to development agility, and my search for such tools immediately yielded the first code refactoring toolsets.
In the decade since the advent of agile development, however, major development-tool vendors, even those who tout their leadership role in agile development, often ignore or are completely ignorant of the potential usefulness of code refactoring. Until recently, most IBM Rational marketers and some techies had never heard of refactoring, and today a quick Google search on “code refactoring” will turn up not one major vendor on its front page. Even when vendors are taking a longer-term view of agile development projects under the heading of “agile ALM (Application Lifecycle Management),” code refactoring is typically a minor or non-existent part of their “vision.”
The key reason for this neglect, I believe, is a sea-change in development that happened in the late 1990s, in which the up-to-two-year IT development logjam was forever broken by “disposable” software by the likes of Amazon – half of whose code, at one point, was less than six months old. In other words, in parallel with but not related to the rise of agile development, IT and vendors both acquired a belief that long-term “software sclerosis” such as had been happening to mainframe mission-critical apps, in which the app had reached a point where each bug fixed introduced a new one, and lack of documentation and code clarity meant that no one knew any longer what each module really did, no longer mattered – over time, those apps would be superseded by newer ones. And, at the time, this was pretty much true.
However, acceptance of agile development changes that equation yet again. Agile development operates by many, many incremental changes to a base of code, and therefore harder-to-change software accumulates in a version of “software sclerosis” that compresses decades into months – a process sometimes called “going into technical debt.” That technical debt doesn’t disappear once a major rev arrives: new features don’t replace existing user-interface code, and they often depend on past infrastructure code. It’s not as bad as the old days; but technical debt is definitely a boat anchor holding back agile development.
In other words, in the typical use case, a refactoring tool is a major reason for the difference between ongoing average agile development and ongoing superior agile development. Increments are shorter, the code of higher quality, the development more scalable in app size, the testing shorter, and the code itself better able to “turn on a dime” – all at the price of spending a little extra time running a code refactoring tool on your code.
The greatest value of code refactoring, however, imho, is not what Wikipedia notes as improvements in code quality and improvements in coding speed. It is, rather, the implicit training that refactoring tools deliver in improved program design. Just as an agile development methodology improves the agility of the programmer as much as of the program, so code refactoring improves the design skills of the programmer as much as the design of the program. This designing improvement often translates into program designs that lead to apps that are simpler for the end user as well as the programmer, and more apt to deliver value-add that anticipates – rather than simply reflecting – the wishes of the end user.
The Relevance of AccuRev to Agile Development
Originally, I ran across AccuRev in connection with a fascinating white paper on best practices in agile software testing. As I probed deeper, I found that one of AccuRev’s main products is a Software Configuration Management (SCM) tool for agile development, including code refactoring tools. A Google search revealed, as I have noted, little competition from major vendors in this area. And as I thought about it, I realized that SCM is a logical place to apply code refactoring effectively.
This is because a key function of SCM is version management. In any agile development project, multi-person or not, version management is more important than in traditional programming, because the increments are very close together and often involve deployment of added-value software to end users. Establishing that an old version has reached a stopping point and a new version can begin is a logical place to remind one or many programmers to refactor code – either to make sure that the last version has been completely refactored, or to ensure that the upcoming version has refactoring designed in at the start. In a sense, it gives the agile programmer opt-out rather than opt-in code refactoring.
Moreover, as the history of “version control” over the past 25 years has demonstrated, SCM is a relatively small but very stable market. More than one programmer, and you need SCM; and the technology is mostly mature, so once you’ve used a vendor for one project, you’ll use that vendor for the next. PVCS and ClearCase, under different management, continue to be at least as widely used as they were in the early 1990s. To put it another way: AccuRev is very unlikely to collapse tomorrow – no way.
Thus, AccuRev is right in the “sweet spot” of use of code refactoring to give anyone’s agile development a major boost, is more visible than most in supporting code refactoring, and should remain useful over the next few years, both for agile SCM and for code refactoring.
Potential Uses of AccuRev-Type Code Refactoring for IT
If an IT organization has nothing going on in agile development right now, then it probably has major problems that require it to start to tackle agile development before it can even begin to think of code refactoring. Otherwise, it is hard to think of an agile-development situation in which code refactoring in general, and code refactoring via AccuRev-type SCM in particular, will not improve things. Still, there are use cases and use cases.
If you are looking for the biggest initial bang for the buck, agile BI springs to mind immediately – because, right now, it is “top of the mind” for most CEOs and CFOs. Some of that is hype, but enough of it is substance that IT should anticipate doing agile BI app development for at least the next 2-3 years while the fad lasts, and almost certainly beyond that, when it will become an integral part of another fad like “Extreme Analytics” (no, I’m not talking about anything real here – I devoutly hope). Because of this corporate commitment, the amount of technical debt we are going to see embedded in end users’ favorite analytics apps is likely to dwarf anything agile development has seen up to now, and especially because many SMBs are trying to do agile BI on the cheap – unless code refactoring steps in.
A second “bang for the buck” for code refactoring, if IT needs one, is in many-person-team (and especially distributed) agile-development projects. By spreading code refactoring across a team, IT puts them on the same design page in crucial respects, encouraging a more consistent application look and feel. Obviously, AccuRev-type SCM is a good way to introduce code refactoring with minimal disruption – if developers accept any management, it’s version control.
The Bottom Line for IT Buyers
I have been writing about code refactoring for a decade, and it still seems to be the low-hanging fruit that even agile development organizations rarely pluck. Well, let’s try again: code refactoring, like agile development, seems like it wastes time, costs money, and lowers revenues by delaying delivery, and it always does the opposite, over the short and long run. Failure to use it doesn’t spell disaster for the agile development organization; using it spells much greater success, in customer satisfaction, value-delivery speed, and app quality, that in turn (to a lesser degree than agile development itself) significantly affects the agile-development-using enterprise’s top and bottom lines. Buy it. Use it.
There are a fair amount of choices once IT buyers decide to acquire code refactoring capabilities. In my opinion, AccuRev’s approach makes it one of the most attractive vendors in the immediate future. If you want to do a short list, fine. If you don’t, and you don’t have any other ideas, skip the short list and get AccuRev. In any case, don’t waste time; just buy it. Then use it.
Maybe someday soon code refactoring will be fully automated as part of a standardized agile development toolkit, requiring no effort from the programmer, and no need to go out and buy the stuff. Maybe; but since today, after twelve years of availability, we are still nowhere near that day, I wouldn’t hold up improving your agile development because you think it might happen next year. If I were an IT buyer, I wouldn’t waste two seconds on hoping that technology advances will swoop down and save the day; I’d be taking the first steps to acquire a code refactoring solution such as the one included in AccuRev SCM. Just buy something like that. Just use it.
The Importance of Code Refactoring to Agile Development
Ever since agile development first surfaced in the Agile Manifesto, code refactoring has been an under-appreciated and often-ignored part of its value-add. My own fascination with the subject dates back to my Cornell days, when research there in how to find minimal fixes to program compiler errors led inevitably to the question: what is the best code design to allow minimal-effort additions to a program’s features? In the 1980s, research into “structured design” showed that modularized subroutines at the same level were far easier to extend than subroutines in a complex tree structure. When I first became aware of the agile development movement, I immediately saw the benefit of rapid program extension to development agility, and my search for such tools immediately yielded the first code refactoring toolsets.
In the decade since the advent of agile development, however, major development-tool vendors, even those who tout their leadership role in agile development, often ignore or are completely ignorant of the potential usefulness of code refactoring. Until recently, most IBM Rational marketers and some techies had never heard of refactoring, and today a quick Google search on “code refactoring” will turn up not one major vendor on its front page. Even when vendors are taking a longer-term view of agile development projects under the heading of “agile ALM (Application Lifecycle Management),” code refactoring is typically a minor or non-existent part of their “vision.”
The key reason for this neglect, I believe, is a sea-change in development that happened in the late 1990s, in which the up-to-two-year IT development logjam was forever broken by “disposable” software by the likes of Amazon – half of whose code, at one point, was less than six months old. In other words, in parallel with but not related to the rise of agile development, IT and vendors both acquired a belief that long-term “software sclerosis” such as had been happening to mainframe mission-critical apps, in which the app had reached a point where each bug fixed introduced a new one, and lack of documentation and code clarity meant that no one knew any longer what each module really did, no longer mattered – over time, those apps would be superseded by newer ones. And, at the time, this was pretty much true.
However, acceptance of agile development changes that equation yet again. Agile development operates by many, many incremental changes to a base of code, and therefore harder-to-change software accumulates in a version of “software sclerosis” that compresses decades into months – a process sometimes called “going into technical debt.” That technical debt doesn’t disappear once a major rev arrives: new features don’t replace existing user-interface code, and they often depend on past infrastructure code. It’s not as bad as the old days; but technical debt is definitely a boat anchor holding back agile development.
In other words, in the typical use case, a refactoring tool is a major reason for the difference between ongoing average agile development and ongoing superior agile development. Increments are shorter, the code of higher quality, the development more scalable in app size, the testing shorter, and the code itself better able to “turn on a dime” – all at the price of spending a little extra time running a code refactoring tool on your code.
The greatest value of code refactoring, however, imho, is not what Wikipedia notes as improvements in code quality and improvements in coding speed. It is, rather, the implicit training that refactoring tools deliver in improved program design. Just as an agile development methodology improves the agility of the programmer as much as of the program, so code refactoring improves the design skills of the programmer as much as the design of the program. This designing improvement often translates into program designs that lead to apps that are simpler for the end user as well as the programmer, and more apt to deliver value-add that anticipates – rather than simply reflecting – the wishes of the end user.
The Relevance of AccuRev to Agile Development
Originally, I ran across AccuRev in connection with a fascinating white paper on best practices in agile software testing. As I probed deeper, I found that one of AccuRev’s main products is a Software Configuration Management (SCM) tool for agile development, including code refactoring tools. A Google search revealed, as I have noted, little competition from major vendors in this area. And as I thought about it, I realized that SCM is a logical place to apply code refactoring effectively.
This is because a key function of SCM is version management. In any agile development project, multi-person or not, version management is more important than in traditional programming, because the increments are very close together and often involve deployment of added-value software to end users. Establishing that an old version has reached a stopping point and a new version can begin is a logical place to remind one or many programmers to refactor code – either to make sure that the last version has been completely refactored, or to ensure that the upcoming version has refactoring designed in at the start. In a sense, it gives the agile programmer opt-out rather than opt-in code refactoring.
Moreover, as the history of “version control” over the past 25 years has demonstrated, SCM is a relatively small but very stable market. More than one programmer, and you need SCM; and the technology is mostly mature, so once you’ve used a vendor for one project, you’ll use that vendor for the next. PVCS and ClearCase, under different management, continue to be at least as widely used as they were in the early 1990s. To put it another way: AccuRev is very unlikely to collapse tomorrow – no way.
Thus, AccuRev is right in the “sweet spot” of use of code refactoring to give anyone’s agile development a major boost, is more visible than most in supporting code refactoring, and should remain useful over the next few years, both for agile SCM and for code refactoring.
Potential Uses of AccuRev-Type Code Refactoring for IT
If an IT organization has nothing going on in agile development right now, then it probably has major problems that require it to start to tackle agile development before it can even begin to think of code refactoring. Otherwise, it is hard to think of an agile-development situation in which code refactoring in general, and code refactoring via AccuRev-type SCM in particular, will not improve things. Still, there are use cases and use cases.
If you are looking for the biggest initial bang for the buck, agile BI springs to mind immediately – because, right now, it is “top of the mind” for most CEOs and CFOs. Some of that is hype, but enough of it is substance that IT should anticipate doing agile BI app development for at least the next 2-3 years while the fad lasts, and almost certainly beyond that, when it will become an integral part of another fad like “Extreme Analytics” (no, I’m not talking about anything real here – I devoutly hope). Because of this corporate commitment, the amount of technical debt we are going to see embedded in end users’ favorite analytics apps is likely to dwarf anything agile development has seen up to now, and especially because many SMBs are trying to do agile BI on the cheap – unless code refactoring steps in.
A second “bang for the buck” for code refactoring, if IT needs one, is in many-person-team (and especially distributed) agile-development projects. By spreading code refactoring across a team, IT puts them on the same design page in crucial respects, encouraging a more consistent application look and feel. Obviously, AccuRev-type SCM is a good way to introduce code refactoring with minimal disruption – if developers accept any management, it’s version control.
The Bottom Line for IT Buyers
I have been writing about code refactoring for a decade, and it still seems to be the low-hanging fruit that even agile development organizations rarely pluck. Well, let’s try again: code refactoring, like agile development, seems like it wastes time, costs money, and lowers revenues by delaying delivery, and it always does the opposite, over the short and long run. Failure to use it doesn’t spell disaster for the agile development organization; using it spells much greater success, in customer satisfaction, value-delivery speed, and app quality, that in turn (to a lesser degree than agile development itself) significantly affects the agile-development-using enterprise’s top and bottom lines. Buy it. Use it.
There are a fair amount of choices once IT buyers decide to acquire code refactoring capabilities. In my opinion, AccuRev’s approach makes it one of the most attractive vendors in the immediate future. If you want to do a short list, fine. If you don’t, and you don’t have any other ideas, skip the short list and get AccuRev. In any case, don’t waste time; just buy it. Then use it.
Maybe someday soon code refactoring will be fully automated as part of a standardized agile development toolkit, requiring no effort from the programmer, and no need to go out and buy the stuff. Maybe; but since today, after twelve years of availability, we are still nowhere near that day, I wouldn’t hold up improving your agile development because you think it might happen next year. If I were an IT buyer, I wouldn’t waste two seconds on hoping that technology advances will swoop down and save the day; I’d be taking the first steps to acquire a code refactoring solution such as the one included in AccuRev SCM. Just buy something like that. Just use it.
Thursday, January 12, 2012
The Other Agile Development: CollabNet and Collaborative Agile Development
This blog post highlights a software company and technology that I view as potentially useful to organizations investing in agile development, agile new product development, and business agility over the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about agile development today.
The Importance of Collaborative Software Technology to Agile Development
Today, I received a presentation about IBM’s collaborative technology under the heading of Lotus that emphasized its positive effects on business agility. And yet, there was no mention of increased support for agile development tools and methodologies. Instead, the focus was on incorporating “social business” and “mobile” in collaborative technology. This approach, all too common among vendors of collaborative software solutions ranging from email to videoconferencing to Facebook-type information sharing, has already proven itself almost entirely ineffective in improving development or business agility, and sometimes has caused organizations to reinforce instead of moving away from un-agile practices.
A careful study of software development shows that only three software development approaches have had any positive impact at all recently on development TCO, ROI, percent completed on time/within budget, and customer satisfaction: agile development, collaborative development, and “hybrid” open-source/traditional development.
However, when we isolate the three, agile alone is clearly superior to hybrid alone, and far superior to collaborative alone. Add agile back to the mix, and the result is only slightly better than agile alone. The conclusion is clear: the main benefits from either collaborative or hybrid alone come from “unconscious agility”, such as the ability of hybrid to incorporate unexpected but valuable additions in the middle of the development process, or the ability of collaboration to improve a slightly agile development process such as a spiral methodology.
These results, however, do not consider the case where collaboration software is deliberately designed to focus on supporting agile development. In those cases, collaboration software’s support for distributed and multi-person development where necessary should remove much of the “drag” on agile development that such situations create, by wedding the maximum of “methodology fit”, loosely-coupled flexibility, and automation to the necessary coordination techniques like versioning and application lifecycle management. Beyond that, collaboration, with its ubiquitous presence in the organization can, once targeted to agile NPD or another agile business process, deliver the same scalability and distribution benefits. In other words, the collaborative framework of an agile, scalable business process is already there; all you need to do is infuse the collaboration technology with agile “smarts” and insert the agile methodology.
And yet, up to now, those solutions are thin on the ground. Look where you will, at IBM’s Jazz plus Lotus or HP’s Mercury Interactive extended to Application Lifecycle Management (ALM), and the major vendors are not delivering collaboration software clearly targeted to agile development methodologies. However, these solutions are indeed beginning to arrive – from the likes of CollabNet.
The Relevance of CollabNet to Agile Development
The leadership role of CollabNet in collaborative agile development is no surprise, given its original leadership role in collaborative development in the late 1990s and early 2000s. However, that did not guarantee that CollabNet would recognize the synergies between collaborative and agile development when collaborative was targeted to agile. For a while, rather, CollabNet aimed partly at the offshoring market, which in some cases decreased development agility by forcing design separated temporally and physically from the development phase – not to mention the end user. This lasted until the likes of Wipro climbed on the bandwagon of agile, often ahead of IT and the major development-tool vendors.
Still, it is surprising and heartening to see how far CollabNet has come since that time. Every other word on the product section of the web site is “agile,” and CollabNet backs that up with a SCRUM-support toolset called SCRUMworks with which its other solutions such as TeamForge and the ALM toolset are specifically integrated.
That, in turn, is why I conclude that CollabNet is settled in for the long haul, and is therefore a safe bet for IT buyers. As a Web-based collaboration and then an offshoring-focused solution provider, CollabNet was more dependent on speculative markets (not to mention smaller) than it is now. Having survived the transition from one strategic focus to another, it is centered around an industry – agile development tools and services – that has enormous “legs”, as agile development not only spreads throughout organizations but also matures and elaborates its methodology in technology to achieve far more – such as the “continous delivery” that I have mentioned in a previous blog post. Yes, the majors may roll over and crowd out CollabNet by using their clout in existing business collaboration software – but that would be to suddenly deep-six their existing high-end-development toolsets, which I believe require long periods of retrofitting before they can support agile development project management adequately. No, I think that CollabNet is well set up to be among the leaders in collaborative agile development over the next 2-3 years, and probably well beyond that.
Potential Uses of CollabNet-Type Agile Development for IT
Obviously, CollabNet is best suited to organizations of all sizes right now that require distributed team development and are adhering to a SCRUM-type agile technology. However, it is also appropriate to cloud agile development (especially public cloud), because that also emphasizes distributed development and deployment for multi-tenant SaaS in globe-serving multi-location server farms. And, of course, IT with legacy development offshoring and outsourcing will find that CollabNet, due to its experience in those areas, provides a transition to agile practices.
It may seem odd to also propose CollabNet for NPD, rather than the heavyweight lean-plus-agile or innovation-centric tools that have been used up to now. However, I believe that CollabNet should be considered as an alternative/complement to these, especially for the SMB, because its DNA lies in the Web-oriented movement that emphasizes crowdsourcing and global-consumer attention rather than in the existing “lean-focused” NPD tools that emphasize information flow within the business. For SMBs, especially those for whom software is increasingly a product differentiator, something like CollabNet offers an opportunity yet again to outmaneuver the clumsier large enterprises. This is particularly true when the SMB is already leading-edge in agile NPD.
Of course, CollabNet cannot incorporate existing and upcoming social-media input quite as well as a Lotus or perhaps a Google. That’s when the IT buyer should complement CollabNet with features from the appropriate social-media-rich-collaboration-software-plus-development-tool vendor. But combining the two, rather than tossing CollabNet, is probably the way to go.
The Bottom Line for IT Buyers
The bottom line is short and sweet: Over the next 2-3 years, IT buyers should not seek to implement collaboration and hope for increased agility, but rather buy collaboration tools focused on agile development, and be assured of bottom-line results even beyond those of agile development itself. Don’t bother with a pre-short list; get a short list, put CollabNet on it, make your choice, and get going. Hopefully, we’re past the time when anything about agile development had to prove itself by traditional cost and quality metrics – which never saw the cost/revenue/margin/customer-satisfaction forest for the “you’re wasting time and money right now” trees. The benefits are already proven. Get on with it, already.
Of course, you never know which smaller companies will fold, or be acquired and lose their technology edge as they scramble to integrate with less-agile large-vendor solutions. I never say never, but CollabNet appears to be one of the least likely to fold – and the larger vendor may also provide additional useful collaboration software and services.
If you’re still reading this, you’re not agile enough. Go look at CollabNet-type agile-aimed collaborative development solutions. Make CollabNet one of the solutions to be inspected. Go.
The Importance of Collaborative Software Technology to Agile Development
Today, I received a presentation about IBM’s collaborative technology under the heading of Lotus that emphasized its positive effects on business agility. And yet, there was no mention of increased support for agile development tools and methodologies. Instead, the focus was on incorporating “social business” and “mobile” in collaborative technology. This approach, all too common among vendors of collaborative software solutions ranging from email to videoconferencing to Facebook-type information sharing, has already proven itself almost entirely ineffective in improving development or business agility, and sometimes has caused organizations to reinforce instead of moving away from un-agile practices.
A careful study of software development shows that only three software development approaches have had any positive impact at all recently on development TCO, ROI, percent completed on time/within budget, and customer satisfaction: agile development, collaborative development, and “hybrid” open-source/traditional development.
However, when we isolate the three, agile alone is clearly superior to hybrid alone, and far superior to collaborative alone. Add agile back to the mix, and the result is only slightly better than agile alone. The conclusion is clear: the main benefits from either collaborative or hybrid alone come from “unconscious agility”, such as the ability of hybrid to incorporate unexpected but valuable additions in the middle of the development process, or the ability of collaboration to improve a slightly agile development process such as a spiral methodology.
These results, however, do not consider the case where collaboration software is deliberately designed to focus on supporting agile development. In those cases, collaboration software’s support for distributed and multi-person development where necessary should remove much of the “drag” on agile development that such situations create, by wedding the maximum of “methodology fit”, loosely-coupled flexibility, and automation to the necessary coordination techniques like versioning and application lifecycle management. Beyond that, collaboration, with its ubiquitous presence in the organization can, once targeted to agile NPD or another agile business process, deliver the same scalability and distribution benefits. In other words, the collaborative framework of an agile, scalable business process is already there; all you need to do is infuse the collaboration technology with agile “smarts” and insert the agile methodology.
And yet, up to now, those solutions are thin on the ground. Look where you will, at IBM’s Jazz plus Lotus or HP’s Mercury Interactive extended to Application Lifecycle Management (ALM), and the major vendors are not delivering collaboration software clearly targeted to agile development methodologies. However, these solutions are indeed beginning to arrive – from the likes of CollabNet.
The Relevance of CollabNet to Agile Development
The leadership role of CollabNet in collaborative agile development is no surprise, given its original leadership role in collaborative development in the late 1990s and early 2000s. However, that did not guarantee that CollabNet would recognize the synergies between collaborative and agile development when collaborative was targeted to agile. For a while, rather, CollabNet aimed partly at the offshoring market, which in some cases decreased development agility by forcing design separated temporally and physically from the development phase – not to mention the end user. This lasted until the likes of Wipro climbed on the bandwagon of agile, often ahead of IT and the major development-tool vendors.
Still, it is surprising and heartening to see how far CollabNet has come since that time. Every other word on the product section of the web site is “agile,” and CollabNet backs that up with a SCRUM-support toolset called SCRUMworks with which its other solutions such as TeamForge and the ALM toolset are specifically integrated.
That, in turn, is why I conclude that CollabNet is settled in for the long haul, and is therefore a safe bet for IT buyers. As a Web-based collaboration and then an offshoring-focused solution provider, CollabNet was more dependent on speculative markets (not to mention smaller) than it is now. Having survived the transition from one strategic focus to another, it is centered around an industry – agile development tools and services – that has enormous “legs”, as agile development not only spreads throughout organizations but also matures and elaborates its methodology in technology to achieve far more – such as the “continous delivery” that I have mentioned in a previous blog post. Yes, the majors may roll over and crowd out CollabNet by using their clout in existing business collaboration software – but that would be to suddenly deep-six their existing high-end-development toolsets, which I believe require long periods of retrofitting before they can support agile development project management adequately. No, I think that CollabNet is well set up to be among the leaders in collaborative agile development over the next 2-3 years, and probably well beyond that.
Potential Uses of CollabNet-Type Agile Development for IT
Obviously, CollabNet is best suited to organizations of all sizes right now that require distributed team development and are adhering to a SCRUM-type agile technology. However, it is also appropriate to cloud agile development (especially public cloud), because that also emphasizes distributed development and deployment for multi-tenant SaaS in globe-serving multi-location server farms. And, of course, IT with legacy development offshoring and outsourcing will find that CollabNet, due to its experience in those areas, provides a transition to agile practices.
It may seem odd to also propose CollabNet for NPD, rather than the heavyweight lean-plus-agile or innovation-centric tools that have been used up to now. However, I believe that CollabNet should be considered as an alternative/complement to these, especially for the SMB, because its DNA lies in the Web-oriented movement that emphasizes crowdsourcing and global-consumer attention rather than in the existing “lean-focused” NPD tools that emphasize information flow within the business. For SMBs, especially those for whom software is increasingly a product differentiator, something like CollabNet offers an opportunity yet again to outmaneuver the clumsier large enterprises. This is particularly true when the SMB is already leading-edge in agile NPD.
Of course, CollabNet cannot incorporate existing and upcoming social-media input quite as well as a Lotus or perhaps a Google. That’s when the IT buyer should complement CollabNet with features from the appropriate social-media-rich-collaboration-software-plus-development-tool vendor. But combining the two, rather than tossing CollabNet, is probably the way to go.
The Bottom Line for IT Buyers
The bottom line is short and sweet: Over the next 2-3 years, IT buyers should not seek to implement collaboration and hope for increased agility, but rather buy collaboration tools focused on agile development, and be assured of bottom-line results even beyond those of agile development itself. Don’t bother with a pre-short list; get a short list, put CollabNet on it, make your choice, and get going. Hopefully, we’re past the time when anything about agile development had to prove itself by traditional cost and quality metrics – which never saw the cost/revenue/margin/customer-satisfaction forest for the “you’re wasting time and money right now” trees. The benefits are already proven. Get on with it, already.
Of course, you never know which smaller companies will fold, or be acquired and lose their technology edge as they scramble to integrate with less-agile large-vendor solutions. I never say never, but CollabNet appears to be one of the least likely to fold – and the larger vendor may also provide additional useful collaboration software and services.
If you’re still reading this, you’re not agile enough. Go look at CollabNet-type agile-aimed collaborative development solutions. Make CollabNet one of the solutions to be inspected. Go.
Friday, January 6, 2012
The Other BI: Composite CIS/Studio and Agile BI
This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about BI today.
The Importance of the Composite Software Technology to BI
Today’s enthusiasm for “agile BI” that often consists of slapdash applications of SCRUM to rapid roll-out of new analytics applets obscures the fact that business agility, not to mention New Product Development (NPD) agility, involves applying more of agile methodologies than a sprint, and using agile methodologies in many more areas than software development. However, this is where the market is right now. The issue at hand is not so much the better application of business agility in the future, as applying today’s “agile BI” more effectively in the next 2-3 years. This is best done by recognizing one key fact: infusing the organization’s BI with agility is not primarily about development agility – it is about information agility.
Speeding up an inflexible business process via targeted analytics simply doubles your investment in and commitment to an information-handling process that needs instead the ability to fundamentally change. As I have noted in past posts, surveys show that in the typical process of receiving data and using analytics as part of the process of turning it into the right decision by the right decision-maker, or the right change in the business, the biggest problem is not that information is delivered too slowly. The fundamental problem is (a) that “leaks” at every step in the process result in more than 2/3 of the actionable information being lost along the way, and (b) the inflexibility of the process – specifically, its lack of openness to new types of data as they arrive – keep making the “leaks” worse.
Today’s best technology for dealing with (a) and (b) is something now called “data federation” or “data virtualization”. I won’t repeat the long litany of benefits to be expected from data virtualization; here, I will simply note that data virtualization, like master data management, provides one view of related data across the enterprise to the developer and the end user, and, unlike master data management, it can actively reach out, within or beyond the enterprise (as in Composite Discovery), to discover new data sources that can likewise be part of that global view of potential information. The one view of data tackles problem (a) at several critical points by ensuring that everyone can potentially see all data, and that analytics tools can be potentially applied to all useful data. The discovery feature tackles problem (b), if used effectively, by constantly refreshing the actionable data from, let’s say, new social-media data sources not presently covered by BI – as some Composite solutions do for Big Data.
Solving problem (b), of course, improves the organization’s information agility, not just its development agility. So the quick and effective method of tackling information agility in the here and now is to make a data virtualization development tool part and parcel of agile BI projects. Tools from folks like Composite Software, Tibco, Denodo, IBM, and SAP/Sybase are among those that would seem to fit the bill.
The Relevance of Composite Software to BI
Surprisingly enough, for a while around five years ago data virtualization’s predecessor, Enterprise Information Integration, was a hot topic in BI, as a way of extending data warehousing to operational data stores in order to query certain key new data in “near real time.” The primary results of this “fad” were to finally and definitively establish a major market for data virtualization, as well as to cement BI as Composite Software’s core market. In other words, unlike most or all other data-virtualization vendors, Composite Software has ten years of experience with real-world data virtualization at medium to large scales, five years of experience with BI at the same scales, a demonstrated ability to combine with other flexible BI tools such as MicroStrategy, and a focus on handling more “complex” needs that translates naturally to the more complex in-depth analytics typical of social-media and related Big Data analytics.
It may reasonably be objected that Composite, as well as other data virtualization vendors, has shown little or no interest in agile software development methodologies up to now, and therefore use of data virtualization solutions such as Composite’s may tend to slow down agile BI development efforts. Again, I would argue, as someone who has been touting agile development nearly since its inception twelve years ago, that this misses the point on two counts. In the first place, the rule of thumb with agile tools is “lead, follow, or get out of the way.” Where a SCRUM-supporting tool leads by urging developers towards agile best practices, and a really good refactoring tool follows in its wake by adding flexibility to the code thus produced, there is a third category of tools that simply automates and simplifies the necessary tasks of software development, while not getting in the way of the developer – and that is what something like Composite Information Server (CIS) with Composite Studio does for data-accessing functions in agile-BI programs.
In the second place, data virtualization such as Composite Software’s comes with a built-in gain in any agile-BI program’s information agility, which is often just as valuable as increased analytic-program implementation speed. Out of the box, the developer is assured of not missing a key in-enterprise data source. More subtly, the developer is made aware of other data sources than the one he or she may initially have targeted, which means that, as should be expected in agile development projects, new and unexpected features with value-add surface in the middle of the project.
In other words, while Composite Software may be in the middle of the pack when it comes to agility applied to agile-BI software development, it has long taken a leadership role in the features of data virtualization that really matter to agile-BI users, including BI services expertise, discovery, and complex data virtualization.
One final caveat appears to be going away. Since their inception, data virtualization tools have been oriented towards users above a certain size of medium-sized organization. Obviously, much of the action and excitement around agile BI has centered around its use in the public cloud by SMBs, who traditionally don’t deal with lots of various data sources that could use virtualization. However, what these SMBs are finding is that some public-cloud data is inherently “multi-organization”, and therefore requires federation and virtualization – especially since many of these same SMBs are now using multiple public clouds for their analytics. Sooner or later, Big Data has to be combined with ERP data on the cloud, and then with customer data on salesforce.com or wherever, and so on. As a result, even these SMBs are probably going to find data virtualization and Composite-type technology useful in their agile-BI efforts sooner rather than later.
Potential Uses of CIS-Type Agile BI for IT
Until recently, data virtualization in the enterprise has typically been done on a per-project basis, with the result that spreading it across all in-enterprise data is typically a work in progress. Obviously, this limits a little the usefulness of data virtualization in agile-BI development in the near future; but it is also an opportunity, in those enterprises that qualify, to kick-start the “global metadata repository” effort into high gear. Slap a data virtualization engine and development tool like CIS with Composite Studio into the agile-BI development team’s toolset, and have them scream to you about needing it to cover more data. That will get corporate’s attention, fast.
For those organizations that haven’t used data virtualization before, especially the SMBs, using data virtualization on things like Big Data is an excellent opportunity to avoid the arrows that the large-enterprise data-virtualization pioneer users faced. More specifically, if you start out by federating key customer Big Data with ERP on the cloud, private or public, you have already created a repository that covers most of your likely business-critical data in the next 2-3 years, and so you will not have to retrofit it later on.
A third, often underestimated, use of data virtualization is as a tool to support the new “business analysts” on the corporate side who are also doing agile analytics. A recent survey cited by Sloan Management Review suggests that these now view IT as slower and lower than dirt. Give them and support a tool that allows them to go out and discover new data sources, and I would not be surprised if they responded, like the woman conducting a choir in the movie Love Story, with “That is absolutely, stupendously, incredibly – OK.” That is, they won’t kiss you, but they will make a big shift towards tolerating you.
However, the use of CIS-type data virtualization that is dearest to my heart, and should be dearest to yours if you ever want to accomplish business agility rather than just BI “agility,” is applying its discovery features constantly, out on the Web, embedded in your analytics applications, to find and bring in-house new data and new data types. One key finding of business surveys is that only half of respondents find out about new key data heralding new market trends on the Web in less than half a year. Data virtualization can cut that to a day. You see, business agility isn't about reacting quickly and effectively to changes in your environment that you happen to find out about because they show up eventually in your customer complaints; the critical success factor is finding out all the changes that you can, as they happen out there, and then reacting quickly and effectively.
The Bottom Line for IT Buyers
Over the next few years, IT buyers should be viewing the acquisition and use of data virtualization tools in agile BI, among other things, as a no-brainer – but they don’t. Those smart IT buyers who do, however, will find that a pre-short-list is not necessary: on closer examination, the use cases almost present themselves. At any rate, that has been users’ experience in data virtualization’s original core markets such as the armed forces in the government – another Composite installed base.
However, for those constrained by budgets even in their strategic analytics initiatives, Composite and others may well have to go on a “pre-short list.” This should be of a peculiar type, however. The IT buyer should not passively wait for corporate to give IT a mandate that requires data virtualization. Instead, the IT buyer should be actively looking for the opportunity to introduce or extend agile-BI data virtualization, like the politician looking in every budget process or special spending initiative for a way of cutting taxes.
Composite Software is one among several who should be on that short list, as well as other data-virtualization-focused firms like Denodo and Tibco, not to mention the data federation solutions of IBM, Oracle via its BEA WebLogic Liquid Data acquisition, Red Hat/Metamatrix, and SAP/Sybase. Composite Software’s edge over most of these is simply that, over the last 10 years, Composite has established its experience and a leadership role in features and services useful to agile BI, and is among the most vocal about their plans to support Big Data and agile BI. Since that leadership role hasn’t diminished over the last three years, Composite Software will probably maintain its position at or near the top of agile-BI data virtualization short lists over the next 2-3 years, as well.
The bottom line for IT buyers of Other agile-BI data virtualization in general, and Composite Software’s CIS and Composite Studio in particular, is this: buy it right now or buy it a year from now, use it for BI agility or use it for business agility, but buy it and use it for some agile-BI initiative sometime in the next 2-3 years. Like an agile development methodology, it looks like it might cost more and cut into the bottom line, but it repeatedly costs less and delivers more revenue – because, used right, it improves your information agility.
The Importance of the Composite Software Technology to BI
Today’s enthusiasm for “agile BI” that often consists of slapdash applications of SCRUM to rapid roll-out of new analytics applets obscures the fact that business agility, not to mention New Product Development (NPD) agility, involves applying more of agile methodologies than a sprint, and using agile methodologies in many more areas than software development. However, this is where the market is right now. The issue at hand is not so much the better application of business agility in the future, as applying today’s “agile BI” more effectively in the next 2-3 years. This is best done by recognizing one key fact: infusing the organization’s BI with agility is not primarily about development agility – it is about information agility.
Speeding up an inflexible business process via targeted analytics simply doubles your investment in and commitment to an information-handling process that needs instead the ability to fundamentally change. As I have noted in past posts, surveys show that in the typical process of receiving data and using analytics as part of the process of turning it into the right decision by the right decision-maker, or the right change in the business, the biggest problem is not that information is delivered too slowly. The fundamental problem is (a) that “leaks” at every step in the process result in more than 2/3 of the actionable information being lost along the way, and (b) the inflexibility of the process – specifically, its lack of openness to new types of data as they arrive – keep making the “leaks” worse.
Today’s best technology for dealing with (a) and (b) is something now called “data federation” or “data virtualization”. I won’t repeat the long litany of benefits to be expected from data virtualization; here, I will simply note that data virtualization, like master data management, provides one view of related data across the enterprise to the developer and the end user, and, unlike master data management, it can actively reach out, within or beyond the enterprise (as in Composite Discovery), to discover new data sources that can likewise be part of that global view of potential information. The one view of data tackles problem (a) at several critical points by ensuring that everyone can potentially see all data, and that analytics tools can be potentially applied to all useful data. The discovery feature tackles problem (b), if used effectively, by constantly refreshing the actionable data from, let’s say, new social-media data sources not presently covered by BI – as some Composite solutions do for Big Data.
Solving problem (b), of course, improves the organization’s information agility, not just its development agility. So the quick and effective method of tackling information agility in the here and now is to make a data virtualization development tool part and parcel of agile BI projects. Tools from folks like Composite Software, Tibco, Denodo, IBM, and SAP/Sybase are among those that would seem to fit the bill.
The Relevance of Composite Software to BI
Surprisingly enough, for a while around five years ago data virtualization’s predecessor, Enterprise Information Integration, was a hot topic in BI, as a way of extending data warehousing to operational data stores in order to query certain key new data in “near real time.” The primary results of this “fad” were to finally and definitively establish a major market for data virtualization, as well as to cement BI as Composite Software’s core market. In other words, unlike most or all other data-virtualization vendors, Composite Software has ten years of experience with real-world data virtualization at medium to large scales, five years of experience with BI at the same scales, a demonstrated ability to combine with other flexible BI tools such as MicroStrategy, and a focus on handling more “complex” needs that translates naturally to the more complex in-depth analytics typical of social-media and related Big Data analytics.
It may reasonably be objected that Composite, as well as other data virtualization vendors, has shown little or no interest in agile software development methodologies up to now, and therefore use of data virtualization solutions such as Composite’s may tend to slow down agile BI development efforts. Again, I would argue, as someone who has been touting agile development nearly since its inception twelve years ago, that this misses the point on two counts. In the first place, the rule of thumb with agile tools is “lead, follow, or get out of the way.” Where a SCRUM-supporting tool leads by urging developers towards agile best practices, and a really good refactoring tool follows in its wake by adding flexibility to the code thus produced, there is a third category of tools that simply automates and simplifies the necessary tasks of software development, while not getting in the way of the developer – and that is what something like Composite Information Server (CIS) with Composite Studio does for data-accessing functions in agile-BI programs.
In the second place, data virtualization such as Composite Software’s comes with a built-in gain in any agile-BI program’s information agility, which is often just as valuable as increased analytic-program implementation speed. Out of the box, the developer is assured of not missing a key in-enterprise data source. More subtly, the developer is made aware of other data sources than the one he or she may initially have targeted, which means that, as should be expected in agile development projects, new and unexpected features with value-add surface in the middle of the project.
In other words, while Composite Software may be in the middle of the pack when it comes to agility applied to agile-BI software development, it has long taken a leadership role in the features of data virtualization that really matter to agile-BI users, including BI services expertise, discovery, and complex data virtualization.
One final caveat appears to be going away. Since their inception, data virtualization tools have been oriented towards users above a certain size of medium-sized organization. Obviously, much of the action and excitement around agile BI has centered around its use in the public cloud by SMBs, who traditionally don’t deal with lots of various data sources that could use virtualization. However, what these SMBs are finding is that some public-cloud data is inherently “multi-organization”, and therefore requires federation and virtualization – especially since many of these same SMBs are now using multiple public clouds for their analytics. Sooner or later, Big Data has to be combined with ERP data on the cloud, and then with customer data on salesforce.com or wherever, and so on. As a result, even these SMBs are probably going to find data virtualization and Composite-type technology useful in their agile-BI efforts sooner rather than later.
Potential Uses of CIS-Type Agile BI for IT
Until recently, data virtualization in the enterprise has typically been done on a per-project basis, with the result that spreading it across all in-enterprise data is typically a work in progress. Obviously, this limits a little the usefulness of data virtualization in agile-BI development in the near future; but it is also an opportunity, in those enterprises that qualify, to kick-start the “global metadata repository” effort into high gear. Slap a data virtualization engine and development tool like CIS with Composite Studio into the agile-BI development team’s toolset, and have them scream to you about needing it to cover more data. That will get corporate’s attention, fast.
For those organizations that haven’t used data virtualization before, especially the SMBs, using data virtualization on things like Big Data is an excellent opportunity to avoid the arrows that the large-enterprise data-virtualization pioneer users faced. More specifically, if you start out by federating key customer Big Data with ERP on the cloud, private or public, you have already created a repository that covers most of your likely business-critical data in the next 2-3 years, and so you will not have to retrofit it later on.
A third, often underestimated, use of data virtualization is as a tool to support the new “business analysts” on the corporate side who are also doing agile analytics. A recent survey cited by Sloan Management Review suggests that these now view IT as slower and lower than dirt. Give them and support a tool that allows them to go out and discover new data sources, and I would not be surprised if they responded, like the woman conducting a choir in the movie Love Story, with “That is absolutely, stupendously, incredibly – OK.” That is, they won’t kiss you, but they will make a big shift towards tolerating you.
However, the use of CIS-type data virtualization that is dearest to my heart, and should be dearest to yours if you ever want to accomplish business agility rather than just BI “agility,” is applying its discovery features constantly, out on the Web, embedded in your analytics applications, to find and bring in-house new data and new data types. One key finding of business surveys is that only half of respondents find out about new key data heralding new market trends on the Web in less than half a year. Data virtualization can cut that to a day. You see, business agility isn't about reacting quickly and effectively to changes in your environment that you happen to find out about because they show up eventually in your customer complaints; the critical success factor is finding out all the changes that you can, as they happen out there, and then reacting quickly and effectively.
The Bottom Line for IT Buyers
Over the next few years, IT buyers should be viewing the acquisition and use of data virtualization tools in agile BI, among other things, as a no-brainer – but they don’t. Those smart IT buyers who do, however, will find that a pre-short-list is not necessary: on closer examination, the use cases almost present themselves. At any rate, that has been users’ experience in data virtualization’s original core markets such as the armed forces in the government – another Composite installed base.
However, for those constrained by budgets even in their strategic analytics initiatives, Composite and others may well have to go on a “pre-short list.” This should be of a peculiar type, however. The IT buyer should not passively wait for corporate to give IT a mandate that requires data virtualization. Instead, the IT buyer should be actively looking for the opportunity to introduce or extend agile-BI data virtualization, like the politician looking in every budget process or special spending initiative for a way of cutting taxes.
Composite Software is one among several who should be on that short list, as well as other data-virtualization-focused firms like Denodo and Tibco, not to mention the data federation solutions of IBM, Oracle via its BEA WebLogic Liquid Data acquisition, Red Hat/Metamatrix, and SAP/Sybase. Composite Software’s edge over most of these is simply that, over the last 10 years, Composite has established its experience and a leadership role in features and services useful to agile BI, and is among the most vocal about their plans to support Big Data and agile BI. Since that leadership role hasn’t diminished over the last three years, Composite Software will probably maintain its position at or near the top of agile-BI data virtualization short lists over the next 2-3 years, as well.
The bottom line for IT buyers of Other agile-BI data virtualization in general, and Composite Software’s CIS and Composite Studio in particular, is this: buy it right now or buy it a year from now, use it for BI agility or use it for business agility, but buy it and use it for some agile-BI initiative sometime in the next 2-3 years. Like an agile development methodology, it looks like it might cost more and cut into the bottom line, but it repeatedly costs less and delivers more revenue – because, used right, it improves your information agility.
Subscribe to:
Posts (Atom)