So far, I’ve talked about my personal experiences as a programmer that have led me to anticipate – apparently correctly – that agile development would deliver consistently better results by cost, profit, and agility metrics. What about the down side? Or, to put it another way, what else could users do that agile development hasn’t done?
After I left Prime, I started as a computer industry analyst at The Yankee Group. I will always be grateful to Yankee for giving me the broadest possible canvas on which to paint my visions of what could be – as I used to put it, I covered “everything below the mainframe”. Of course, in the early 90s that was only ½ of the computing field … Anyway, one of the things I wrote was a comprehensive report called “Software Development: Think Again or Fail”. [yes, I know; I was a bit immodest in those days]
The reason I bring this up is that two things mentioned in that report seem to be missing in agile development:
1. High-level tools, languages, and components; and
2. Tasking programmers with keeping track of particular markets.
As far as I can see, agile theory and practice doesn’t give a hoot whether those programmers are using Java, Perl, Python, or Ruby on Rails. I use those examples because they all have been touted as ways to speed up programming in the Java/open-source world, and because only tunnel vision leads people to conclude that they’re anything but dolled-up 3GLs that do very well at handling function-driven programming and only adequately at rapidly generating data-access and user-interface code. Compare that to M204 UL, drag-and-drop VPEs (visual programming environments), and the like, and I am forced to conclude that in some respects, these tools are still less productive than what was available 13 years ago. The point is that, even if agile succeeds in improving the speed of the individual programmer, the other productivity shoe will never drop as long as the first instinct of managers and programmers is to reach for a 3GL.
The second point is that although agile does well with making sure that programmers talk to individual end users, that is different from following the entire software market. Following a market gives context to what the end user wants, and allows the designer to look at where the market appears to be going, rather than where end users have been.
So my caution about agile development is that my experience tells me that so much more can be done. The results are startling and consistent; but they could be more so. Agile development deserves praise; but the worst thing for software development would be to assume that no more fundamental changes in the paradigm need be done.
Monday, January 5, 2009
Some Personal Thoughts on Agile Development, Part 2
In Part 1, I talked about how I learned that the concepts of agile development could be effective, while I was at CCA as a programmer. But how does agile development achieve consistently improved results, as I found in the survey I mentioned? Well, I found some clues when I went to work for Prime Computer, right after CCA.
Prime was the first time I got to deal with project leadership in anything but a one-person project (except a 3-week stint at CCA). Prime was a very different company – thousands instead of 40-100 people, and therefore 40 software-development-related people in my own corner of the company out of more than a thousand; plus, Prime was really still a company where the software was aimed at selling the hardware. So I not only learned about development from a much broader and managerial perspective; I also learned about large organizations’ overall attitude towards development.
Insight # 1: Theoretically, doing a Gantt chart based on your experience and measuring yourself on being on time, within budget, and doing all the functionality is the way to go; in the real world, it has little relation to success. Here’s my story; I set out a schedule and sent it to my boss; in order to fund it, he figured out how to reschedule, and cut the time and costs by 25%; he sent it to his boss, who in order to sell it to the company head, did the same thing. So my best guess of how long it would take was sliced by almost ½. When I pointed out that this was probably not going to happen, my bosses reassured me that they understood that the numbers were optimistic – and then expressed concern during the project when things fell behind their schedule. I have seen this frequently enough since that I have cockily created Kernochan’s Law: Multiply the estimated time for a project by 4/3 ** (number of levels of management review) to get a realistic number.
Insight #2: There is no developer career path. In Prime’s case, after a certain level you could either be a marketer or a “Principal Software Engineer”. From that point on, if you stayed on the developer hierarchy, you would either be an advisor thinking grand theoretical thoughts or a manager; but as an advisor you would not expand your say in the non-technical aspects of the product, and as a manager you would move out of the developer career path. Again, this is a pattern I frequently see elsewhere: doctors doctor and also manage the hospital; physicists run biotech shops and also direct the research; software developers rarely are allowed to combine the technical and marketing aspects of product design as part of their career path, nor are they trained to do so.
What does this have to do with getting consistent results from agile development? Well, if you think about it, the number of levels of managerial review in agile development is effectively zero: it’s just you the developer and the end user for every change. Moreover, you’ve opened yourself to changes in the spec; so who knows what the realistic estimate for project end should be? And yet, when you do this, you look back at the end of the project, and by any realistic measure, you’ve actually done things faster – even though each change should mean you’re falling further and further behind your original budget. Agile will do consistently better in all realistic metrics, like development cost compared to what the same product should have cost by the old process, because it does not manage nor measure by flawed numbers and incentives.
Moreover, agile will also do consistently better because it implicitly gives good developers more of a professional career path. At the same time you think you are madly asking for intense programming spurts, you are actually asking for lots of programmer input into the non-technical design of the product, using the end user as an excuse. The reason the user is more satisfied is not just because he or she is listened to more; it is also that the programmer is combining technical and non-technical aspects in program design, making a better combined product. That doesn’t affect costs, except costs to retrofit; it does affect return on investment, customer satisfaction, and the “agility” of the resulting product (because some of the non-technical reasons for a new version have already been dealt with).
All right, so my experience tells me that agile development is a great idea and will deliver consistent results. Is it enough? As the British say, not by a long chalk. In my Finale, I explore what my personal story tells me is missing.
Prime was the first time I got to deal with project leadership in anything but a one-person project (except a 3-week stint at CCA). Prime was a very different company – thousands instead of 40-100 people, and therefore 40 software-development-related people in my own corner of the company out of more than a thousand; plus, Prime was really still a company where the software was aimed at selling the hardware. So I not only learned about development from a much broader and managerial perspective; I also learned about large organizations’ overall attitude towards development.
Insight # 1: Theoretically, doing a Gantt chart based on your experience and measuring yourself on being on time, within budget, and doing all the functionality is the way to go; in the real world, it has little relation to success. Here’s my story; I set out a schedule and sent it to my boss; in order to fund it, he figured out how to reschedule, and cut the time and costs by 25%; he sent it to his boss, who in order to sell it to the company head, did the same thing. So my best guess of how long it would take was sliced by almost ½. When I pointed out that this was probably not going to happen, my bosses reassured me that they understood that the numbers were optimistic – and then expressed concern during the project when things fell behind their schedule. I have seen this frequently enough since that I have cockily created Kernochan’s Law: Multiply the estimated time for a project by 4/3 ** (number of levels of management review) to get a realistic number.
Insight #2: There is no developer career path. In Prime’s case, after a certain level you could either be a marketer or a “Principal Software Engineer”. From that point on, if you stayed on the developer hierarchy, you would either be an advisor thinking grand theoretical thoughts or a manager; but as an advisor you would not expand your say in the non-technical aspects of the product, and as a manager you would move out of the developer career path. Again, this is a pattern I frequently see elsewhere: doctors doctor and also manage the hospital; physicists run biotech shops and also direct the research; software developers rarely are allowed to combine the technical and marketing aspects of product design as part of their career path, nor are they trained to do so.
What does this have to do with getting consistent results from agile development? Well, if you think about it, the number of levels of managerial review in agile development is effectively zero: it’s just you the developer and the end user for every change. Moreover, you’ve opened yourself to changes in the spec; so who knows what the realistic estimate for project end should be? And yet, when you do this, you look back at the end of the project, and by any realistic measure, you’ve actually done things faster – even though each change should mean you’re falling further and further behind your original budget. Agile will do consistently better in all realistic metrics, like development cost compared to what the same product should have cost by the old process, because it does not manage nor measure by flawed numbers and incentives.
Moreover, agile will also do consistently better because it implicitly gives good developers more of a professional career path. At the same time you think you are madly asking for intense programming spurts, you are actually asking for lots of programmer input into the non-technical design of the product, using the end user as an excuse. The reason the user is more satisfied is not just because he or she is listened to more; it is also that the programmer is combining technical and non-technical aspects in program design, making a better combined product. That doesn’t affect costs, except costs to retrofit; it does affect return on investment, customer satisfaction, and the “agility” of the resulting product (because some of the non-technical reasons for a new version have already been dealt with).
All right, so my experience tells me that agile development is a great idea and will deliver consistent results. Is it enough? As the British say, not by a long chalk. In my Finale, I explore what my personal story tells me is missing.
Some Personal Thoughts on Agile Development
Recently, I have been involved in a survey of development organizations, assessing their practices and how they are changing. It is not giving away too much, I hope, to say that the results were quite surprising to me. I never expected, for one thing, that the whole idea of “agile” development would have become so pervasive in so short a time – not only in the small innovation-seeking software companies but also in large non-software firms doing massive projects. Granted, it is more popular in the US than abroad, and granted, in a few cases “agile” seems more like a fuzzy marketing term slapped onto collaborative technologies. But time and again, users are getting specific – looking at variants of SCRUM and DSDM, using tools with Agile in the product name, mentioning extreme programming practices.
It is less surprising to me that agile programming is delivering results – consistently. You see, I’m a former programmer – although I haven’t done much if anything in the last 18 years. More specifically, I’m a graduate of Computer Corp. of America, whose Model 204 User Language taught me a lot about programming that neither schools nor other development shops came within a mile of suggesting.
At first glance, M204 UL seemed a bit like a third-generation language – say, PL/1. It had English-language variants of the usual C-like commands, plus string handling that was strongly reminiscent of SNOBOL. Ah, but then there were the data-access commands. These were superbly crafted to do what they said in an intuitive fashion, and to give the programmer full rein to optimize the code using simple, orthogonal statements. What do I mean by orthogonal? I mean that each different statement type covered a large, distinct set of operations, and together they covered everything “data” you wanted to do.
Now, what does this have to do with agile? Well, if you didn’t worry about little things like performance, you could rip off a program to do just about anything in nothing flat (remember, this is before GUIs). So some M204 UL experts did just that, traveling to user sites to customize like crazy. Iterative prototyping back at CCA was a way of life, not a whole new process.
The second thing that M204 UL taught me was that close ties with the user were good, not bad (up to a point; users couldn’t tell you where the market was going, just what they needed right then). I did several revs of an email product built using M204 UL, and what drove the specs for each rev was user input, because there just wasn’t enough bandwidth to do an elaborate, controlled development process. And guess what? The result was better, and was developed quicker, than just about anything else out there. As a one-person shop, I was turning out major revs every 6 months, if needed.
And finally, M204 UL and CCA taught me a great deal about the perils of over-controlling the development process. My last project was a development tool, done in parallel with a 9-person search-tool project. I was forced to sit in “design by committee” meetings all day, every day, for 5 weeks straight. The resulting file-search-tool design was thoroughly documented, bland, and highly complicated. I wrote out some hasty, vague specs for the development tool, because I wanted freedom to elaborate as I went. The result was that the search-tool team got kudos for their design, and I got put on probation – but they still needed me to do the development tool.
Well, to make a long story short, I finished that tool – a very original, easy-to-use one that sold just fine – in about 3 months, or shortly before I left. The search tool development was done in the same over-controlled manner that its design was, and after a year the project was cancelled – they just couldn’t get it to work. And that taught me that flexibility and room for change trumped control, at least for medium-scale projects – and, as we saw from the Internet, that goes for large-scale projects done iteratively and incrementally, as well. I’m not knocking control; but it has to be control in the service of flexibility, not preventing it.
So I learned, in my gut, that the kinds of ideas that agile is talking about now – fast “spiral” iterations, lots of user feedback, loosely-coupled programmers – could work. What about getting consistent results? Well, that’s another story – for the next blog.
It is less surprising to me that agile programming is delivering results – consistently. You see, I’m a former programmer – although I haven’t done much if anything in the last 18 years. More specifically, I’m a graduate of Computer Corp. of America, whose Model 204 User Language taught me a lot about programming that neither schools nor other development shops came within a mile of suggesting.
At first glance, M204 UL seemed a bit like a third-generation language – say, PL/1. It had English-language variants of the usual C-like commands, plus string handling that was strongly reminiscent of SNOBOL. Ah, but then there were the data-access commands. These were superbly crafted to do what they said in an intuitive fashion, and to give the programmer full rein to optimize the code using simple, orthogonal statements. What do I mean by orthogonal? I mean that each different statement type covered a large, distinct set of operations, and together they covered everything “data” you wanted to do.
Now, what does this have to do with agile? Well, if you didn’t worry about little things like performance, you could rip off a program to do just about anything in nothing flat (remember, this is before GUIs). So some M204 UL experts did just that, traveling to user sites to customize like crazy. Iterative prototyping back at CCA was a way of life, not a whole new process.
The second thing that M204 UL taught me was that close ties with the user were good, not bad (up to a point; users couldn’t tell you where the market was going, just what they needed right then). I did several revs of an email product built using M204 UL, and what drove the specs for each rev was user input, because there just wasn’t enough bandwidth to do an elaborate, controlled development process. And guess what? The result was better, and was developed quicker, than just about anything else out there. As a one-person shop, I was turning out major revs every 6 months, if needed.
And finally, M204 UL and CCA taught me a great deal about the perils of over-controlling the development process. My last project was a development tool, done in parallel with a 9-person search-tool project. I was forced to sit in “design by committee” meetings all day, every day, for 5 weeks straight. The resulting file-search-tool design was thoroughly documented, bland, and highly complicated. I wrote out some hasty, vague specs for the development tool, because I wanted freedom to elaborate as I went. The result was that the search-tool team got kudos for their design, and I got put on probation – but they still needed me to do the development tool.
Well, to make a long story short, I finished that tool – a very original, easy-to-use one that sold just fine – in about 3 months, or shortly before I left. The search tool development was done in the same over-controlled manner that its design was, and after a year the project was cancelled – they just couldn’t get it to work. And that taught me that flexibility and room for change trumped control, at least for medium-scale projects – and, as we saw from the Internet, that goes for large-scale projects done iteratively and incrementally, as well. I’m not knocking control; but it has to be control in the service of flexibility, not preventing it.
So I learned, in my gut, that the kinds of ideas that agile is talking about now – fast “spiral” iterations, lots of user feedback, loosely-coupled programmers – could work. What about getting consistent results? Well, that’s another story – for the next blog.
Friday, October 10, 2008
Some Thoughts on Real-Time Data and Innovation
First off, the news. Yes, I’m now working for Aberdeen Group. And a big shout out to my fellow analysts. J Now back to commentary.
The latest edition of MIT’s Sloan Management Review has a couple of interesting articles, one on the effects of real-time data and one on IBM’s Innovation Jam. The one on real-time data (“The Downside of Real-Time Data”, Alden Hayashi, reviewing recent research by Swaminathan and Lurie) suggests that getting new data in real time can actually be harmful – that is, decision-makers fed daily retail data instead of weekly averages tended to over-react to negative or positive news, resulting in inventory replacement orders that were actually less optimal than those of the “behind the times” managers (especially when day-to-day sales variance was high). The researchers do note that systems can compensate to some extent by ensuring that decision-makers are given some historical context on the same screen as the latest figures.
I think it’s reasonable to extrapolate to CEOs and CMOs (not necessarily CFOs; they are so tied to long-term data that it’s hard to imagine them over-reacting to the latest uptick in inventory in a rolling balance sheet). Yes, the latest craze for 360-degree view performance-management dashboards based on event processing could indeed enable CEOs who not only take a short-term view of the company’s stock price but also of the strategic levers he/she has to pull, resulting in erratic company shifts in tactics and/or strategy.
However, I think this point is entirely irrelevant to the main value-add of real-time presentation of event-type data (that is, data that represents a shift in the environment: a customer order is an event, a description of a product part is not). Each event-response loop is, or should be, part of a business process; the main idea is to speed up the response in order to satisfy customers, react to unanticipated events, or save money, not to improve decisions. The side-effect is that delaying a response often forecloses options, so that decisions are necessarily less optimal; these studies do not test that notion. In other words, in the real world, if you design speeded-up event-response loops by speeding real-time data presentation in the context of an existing business process, you will indeed include context or you won’t need it, and so 99% of the time what results is good for the organization.
Btw, the real danger, which we may run into someday, is “herd behavior” such as we have seen in the markets, in which multiple organizations “head for the exits” at the same time in immediate reaction to the same news, creating an avoidable vicious cycle. But the game is still worth the candle …
Anyway, the second article talks of how IBM did a massive “jam session” in 2006 to ask its employees to contribute ideas for turning IBM technology in its labs into products. The first phase simply collected the ideas; the second phase, after managers sifted and aggregated ideas, was to attempt to turn the ideas into clear product concepts with attractive markets. Particular examples of areas in which the Jam gave a boost that usually resulted in real-world products were green technology, health care (electronic health records), and integrated transportation management. The authors’ (Bjelland and Wood) conclusion was that this was a promising new way of doing an innovation process, with the usual tradeoffs (“powerful but ponderous”).
I think they’re missing a bet here. Has nobody noticed the similarity to open source? In other words, we’re back to the idea of world-wide but to some degree unrepresentative proxies for the customers helping to design the product. To put it another way, these designers (a) combine the technical skills and company knowledge to design products with the reactions of global customers; (b) are treated to some degree as an equal mass, not a hierarchy in which those lower down are ignored; and (c) insist on going their own way, not the way the company lays out for them.
Here I must make a confession: I participated in much the same sort of process at Prime, shortly before it melted down in 1989. The big advance in IBM’s approach, compared to that one, is that it (as far as possible) treated all ideas as equal – Prime by its DNA was unable to resist having “senior gurus” and desperate GMs take charge, with the result that the only ideas that might have saved Prime never surfaced or were ignored (my own idea was “let’s build a new kind of desktop that allows cross-filing”; I still think it was better than the senior-guru idea that was favored, “let’s build a framework and figure out what it’s a framework for as we go along”). The other major difference was that IBM in 2006 had pantloads of employees who could contribute real new ideas based on constant engagement with the world beyond IBM; Prime in 1989 had very few.
My net-net? For the favored few that are large and have that kind of employee, there should be more Innovation Jams, not less; anything to escape “not invented here.” For the rest of us, I think it just won’t work; not enough critical mass, not enough qualified participants. Instead, let’s go back to our communities. They’re a bit more reactive; but, boy, do they do a better job as proxies for the market.
The latest edition of MIT’s Sloan Management Review has a couple of interesting articles, one on the effects of real-time data and one on IBM’s Innovation Jam. The one on real-time data (“The Downside of Real-Time Data”, Alden Hayashi, reviewing recent research by Swaminathan and Lurie) suggests that getting new data in real time can actually be harmful – that is, decision-makers fed daily retail data instead of weekly averages tended to over-react to negative or positive news, resulting in inventory replacement orders that were actually less optimal than those of the “behind the times” managers (especially when day-to-day sales variance was high). The researchers do note that systems can compensate to some extent by ensuring that decision-makers are given some historical context on the same screen as the latest figures.
I think it’s reasonable to extrapolate to CEOs and CMOs (not necessarily CFOs; they are so tied to long-term data that it’s hard to imagine them over-reacting to the latest uptick in inventory in a rolling balance sheet). Yes, the latest craze for 360-degree view performance-management dashboards based on event processing could indeed enable CEOs who not only take a short-term view of the company’s stock price but also of the strategic levers he/she has to pull, resulting in erratic company shifts in tactics and/or strategy.
However, I think this point is entirely irrelevant to the main value-add of real-time presentation of event-type data (that is, data that represents a shift in the environment: a customer order is an event, a description of a product part is not). Each event-response loop is, or should be, part of a business process; the main idea is to speed up the response in order to satisfy customers, react to unanticipated events, or save money, not to improve decisions. The side-effect is that delaying a response often forecloses options, so that decisions are necessarily less optimal; these studies do not test that notion. In other words, in the real world, if you design speeded-up event-response loops by speeding real-time data presentation in the context of an existing business process, you will indeed include context or you won’t need it, and so 99% of the time what results is good for the organization.
Btw, the real danger, which we may run into someday, is “herd behavior” such as we have seen in the markets, in which multiple organizations “head for the exits” at the same time in immediate reaction to the same news, creating an avoidable vicious cycle. But the game is still worth the candle …
Anyway, the second article talks of how IBM did a massive “jam session” in 2006 to ask its employees to contribute ideas for turning IBM technology in its labs into products. The first phase simply collected the ideas; the second phase, after managers sifted and aggregated ideas, was to attempt to turn the ideas into clear product concepts with attractive markets. Particular examples of areas in which the Jam gave a boost that usually resulted in real-world products were green technology, health care (electronic health records), and integrated transportation management. The authors’ (Bjelland and Wood) conclusion was that this was a promising new way of doing an innovation process, with the usual tradeoffs (“powerful but ponderous”).
I think they’re missing a bet here. Has nobody noticed the similarity to open source? In other words, we’re back to the idea of world-wide but to some degree unrepresentative proxies for the customers helping to design the product. To put it another way, these designers (a) combine the technical skills and company knowledge to design products with the reactions of global customers; (b) are treated to some degree as an equal mass, not a hierarchy in which those lower down are ignored; and (c) insist on going their own way, not the way the company lays out for them.
Here I must make a confession: I participated in much the same sort of process at Prime, shortly before it melted down in 1989. The big advance in IBM’s approach, compared to that one, is that it (as far as possible) treated all ideas as equal – Prime by its DNA was unable to resist having “senior gurus” and desperate GMs take charge, with the result that the only ideas that might have saved Prime never surfaced or were ignored (my own idea was “let’s build a new kind of desktop that allows cross-filing”; I still think it was better than the senior-guru idea that was favored, “let’s build a framework and figure out what it’s a framework for as we go along”). The other major difference was that IBM in 2006 had pantloads of employees who could contribute real new ideas based on constant engagement with the world beyond IBM; Prime in 1989 had very few.
My net-net? For the favored few that are large and have that kind of employee, there should be more Innovation Jams, not less; anything to escape “not invented here.” For the rest of us, I think it just won’t work; not enough critical mass, not enough qualified participants. Instead, let’s go back to our communities. They’re a bit more reactive; but, boy, do they do a better job as proxies for the market.
Tuesday, July 22, 2008
Data Virtualization in the Real World
In a previous piece, I noted that data virtualization is (a) a logical extension of the idea of virtualization, (b) has definite benefits to IT and to the enterprise, and (c) is implementable but is not easy to implement. The basics, I noted, are an enterprise metadata repository and a “veneer” that makes multiple data sources look like one system. The end product is infrastructure software that makes data, presented as information (that is, in an understandable and usable non-physical format, such as in a database), appear as if it were in one cross-system data store. So what is out there to create a data virtualization solution today, and how does it need to be improved?
Today’s key data-virtualization technologies are Enterprise Information Integration (EII), the data warehouse, master data management (MDM), and metadata management. Each now has solutions creating value inside real-world proactive enterprises; each has its pros and cons. Let’s examine each in turn.
EII products provide a SQL/XQuery veneer for real-time database transactions crossing multiple data sources, including relational data, content, and the Web. In order to do this, they typically auto-discover data sources of these types, and store the information in a metadata repository, supplemented by the enterprise’s knowledge of data semantics, copies, and relationships. The pros for using EII as the basis for a data virtualization solution are its strengths in gathering metadata, its support for real-time transactions, and its flexibility in handling a wide range of data types. The cons are its typical use for specific projects (which means added work extending it to enterprise scope) and its lack of enforcement of corporate standards compared to MDM. Useful EII tools for data virtualization include IBM’s WebSphere Federation Server, Oracle/BEA WebLogic for Liquid Data, Sybase’s Avaki tool, Composite Software’s and Attunity’s products, Ipedo XIP (which also includes a database), and Red Hat/MetaMatrix.
Originally, data warehouses were receptacles for all corporate data replicated periodically (including data cleansing and merging) from multiple OLTP (online transaction processing) databases, with business-intelligence queries run against the days-old merged data. Recently, the lines have blurred, with data warehouses now able to query on the operational and content databases as well as the data warehouse, and to refresh in such a way that queries are on “near real-time” data. The pros of using today’s data warehouse for data virtualization are unmatched data quality and lots of available applications to use the virtualized data. The cons are that the data warehouse is getting farther and farther away from storing all of an organization’s data and therefore may not fit some data-virtualization needs, that the size of the data warehouse may slow both queries and update-type transactions against the virtualized data, and that a data warehouse typically does no auto-discovery of metadata. Useful data warehouses for data virtualization include the IBM, HP, NCR, Microsoft, Oracle, and Sybase databases.
The MDM field has gone from a gleam in vendors’ eyes to the focus of enterprise attention in about 2 years, but it’s still not a mature technology. The idea is to select key enterprise data and provide a “single view” via a transactional veneer and cross-database synchronization, typically involving a cross-database metadata repository. MDM accommodates a range of implementation architectures, from EII-like (no data moves around) to data-warehouse-like (all the data is replicated to a central repository). Moreover, MDM typically builds on top of EII, EAI (Enterprise Application Integration), and data-warehouse ETL (extract-transform-load) technologies. The pros of MDM are that it delivers virtualization of data that really matters to the organization, that the architecture is flexible and therefore can be fine-tuned for performance, and that it improves data quality in such a way that the enterprise really buys into IT’s data-virtualization effort. The cons are that it typically does not deal with some types of data virtualization (such as local data in different databases or content-heavy data), that it may not provide as extensive auto-discovery capabilities (although it is improving in that area), and that as a relatively immature technology it will typically require high service costs for vendor hand-holding. MDM tools now include IBM MDM Server, Sun’s MDM Suite, Microsoft’s Bulldog MDM project, Kalido MDM, Software AG’s webMethods Master Data Manager, and Oracle Master Data Management Suite.
Finally, enterprise metadata management (EMM) tools are the most immature of all of the data virtualization technologies. Here, the focus is on creating a repository of metadata for all (or most) enterprise data, with less focus on supplying tools for using that repository. The pros of EMM tools appear to be their strengths in auto-discovery and their savvy in representing metadata at the business level. The cons are the immaturity of the technology and the lack of tools for transactional use of the repository. While there are a few smaller-vendor products that have been out for a while from the likes of Adaptive Repository, the larger vendors’ products (e.g., IBM, CA, and Oracle/BEA’s AquaLogic Enterprise Repository) are pretty new to the market.
The point of this survey is that while it is hard to cite a product in the market that is specifically aimed at data virtualization, there is plenty of choice for users who want to repurpose an EII, MDM, data warehouse, or EMM product for data virtualization, and plenty of related real-world experience to draw on. Rather, the key barrier to data-virtualization success is rethinking the enterprise’s information architecture in terms of virtualization. Just as storage virtualization has allowed users to rethink their storage in terms of what can be grouped together, and thus has saved storage administration costs, data virtualization is best used to rethink how data stores can be grouped together. Then, by picking the product that is easiest and cheapest to repurpose for data virtualization, users can deliver its benefits immediately, and cost-effectively.
Today’s key data-virtualization technologies are Enterprise Information Integration (EII), the data warehouse, master data management (MDM), and metadata management. Each now has solutions creating value inside real-world proactive enterprises; each has its pros and cons. Let’s examine each in turn.
EII products provide a SQL/XQuery veneer for real-time database transactions crossing multiple data sources, including relational data, content, and the Web. In order to do this, they typically auto-discover data sources of these types, and store the information in a metadata repository, supplemented by the enterprise’s knowledge of data semantics, copies, and relationships. The pros for using EII as the basis for a data virtualization solution are its strengths in gathering metadata, its support for real-time transactions, and its flexibility in handling a wide range of data types. The cons are its typical use for specific projects (which means added work extending it to enterprise scope) and its lack of enforcement of corporate standards compared to MDM. Useful EII tools for data virtualization include IBM’s WebSphere Federation Server, Oracle/BEA WebLogic for Liquid Data, Sybase’s Avaki tool, Composite Software’s and Attunity’s products, Ipedo XIP (which also includes a database), and Red Hat/MetaMatrix.
Originally, data warehouses were receptacles for all corporate data replicated periodically (including data cleansing and merging) from multiple OLTP (online transaction processing) databases, with business-intelligence queries run against the days-old merged data. Recently, the lines have blurred, with data warehouses now able to query on the operational and content databases as well as the data warehouse, and to refresh in such a way that queries are on “near real-time” data. The pros of using today’s data warehouse for data virtualization are unmatched data quality and lots of available applications to use the virtualized data. The cons are that the data warehouse is getting farther and farther away from storing all of an organization’s data and therefore may not fit some data-virtualization needs, that the size of the data warehouse may slow both queries and update-type transactions against the virtualized data, and that a data warehouse typically does no auto-discovery of metadata. Useful data warehouses for data virtualization include the IBM, HP, NCR, Microsoft, Oracle, and Sybase databases.
The MDM field has gone from a gleam in vendors’ eyes to the focus of enterprise attention in about 2 years, but it’s still not a mature technology. The idea is to select key enterprise data and provide a “single view” via a transactional veneer and cross-database synchronization, typically involving a cross-database metadata repository. MDM accommodates a range of implementation architectures, from EII-like (no data moves around) to data-warehouse-like (all the data is replicated to a central repository). Moreover, MDM typically builds on top of EII, EAI (Enterprise Application Integration), and data-warehouse ETL (extract-transform-load) technologies. The pros of MDM are that it delivers virtualization of data that really matters to the organization, that the architecture is flexible and therefore can be fine-tuned for performance, and that it improves data quality in such a way that the enterprise really buys into IT’s data-virtualization effort. The cons are that it typically does not deal with some types of data virtualization (such as local data in different databases or content-heavy data), that it may not provide as extensive auto-discovery capabilities (although it is improving in that area), and that as a relatively immature technology it will typically require high service costs for vendor hand-holding. MDM tools now include IBM MDM Server, Sun’s MDM Suite, Microsoft’s Bulldog MDM project, Kalido MDM, Software AG’s webMethods Master Data Manager, and Oracle Master Data Management Suite.
Finally, enterprise metadata management (EMM) tools are the most immature of all of the data virtualization technologies. Here, the focus is on creating a repository of metadata for all (or most) enterprise data, with less focus on supplying tools for using that repository. The pros of EMM tools appear to be their strengths in auto-discovery and their savvy in representing metadata at the business level. The cons are the immaturity of the technology and the lack of tools for transactional use of the repository. While there are a few smaller-vendor products that have been out for a while from the likes of Adaptive Repository, the larger vendors’ products (e.g., IBM, CA, and Oracle/BEA’s AquaLogic Enterprise Repository) are pretty new to the market.
The point of this survey is that while it is hard to cite a product in the market that is specifically aimed at data virtualization, there is plenty of choice for users who want to repurpose an EII, MDM, data warehouse, or EMM product for data virtualization, and plenty of related real-world experience to draw on. Rather, the key barrier to data-virtualization success is rethinking the enterprise’s information architecture in terms of virtualization. Just as storage virtualization has allowed users to rethink their storage in terms of what can be grouped together, and thus has saved storage administration costs, data virtualization is best used to rethink how data stores can be grouped together. Then, by picking the product that is easiest and cheapest to repurpose for data virtualization, users can deliver its benefits immediately, and cost-effectively.
Monday, June 30, 2008
Data Virtualization: It Grows on You
When I first heard the term “data virtualization” from Composite Software, I admit, I was skeptical. Let’s face it, virtualization is one of the all-inclusive marketing terms of our time; it seems like everything is being labeled as virtualized these days. But when I sat down and tried to make sense of all of the “virtual” technologies I knew, data virtualization fit in neatly. In fact, it’s a logical extension of virtualization technology. And therefore, like other virtualization technologies, it has definite and distinct benefits to the user.
Let’s start with my definition. A “virtual” technology is one that detaches the logical scope of a system from its physical scope. More concretely, virtual memory, the virtual machine, virtual storage, etc. all make a subset or a superset of a physical system/resource like a computer, appear like one system/resource. Virtualization is the process of changing from the old physical way of seeing and handling a system to the “virtual” way.
Now note that all virtual technologies can either split up a single system or combine multiple systems, and that they can involve data or processes. For example, the virtual-machine meaning of virtualization (now its most common meaning) splits up the processes on a single machine into multiple virtual machines. The virtual-memory meaning of virtualization pretends that a small main memory (data) is really a large storage device. The storage virtualization meaning treats multiple systems (really data systems attached to multiple computer systems) as if they were one huge storage device. In fact, clustering and network file systems (really another form of virtualization) allow you to treat multiple systems (processes) as if they were one computer.
Here’s a neat little Table that captures how all the forms of virtualization that are being talked about these days fit into a nice neat framework (all right, it’s not really that neat in real life, but it’s a very good starting point).
,,,,,,,,Split Single System ,,,,,,,,,,,,,,,,,,, Combine Multiple Systems
Data: V. memory, V. disk ,,,,,,,,,,,,,,,,,, Storage virtualization, Data virtualization
Process: V. machine, app/desktop/profile v. ,,, Clustering, network file systems
Note that application, desktop, and profile virtualization may involve putting the virtual environment on another computer; but it’s still representing a part of a computer, not the whole thing.
Now all forms of virtualization share a common benefit of flexibility to users, but each category has specific benefits of its own. The benefit of single-system virtualization is a very good tradeoff of performance for cost (single-system); for much less cost and for a small decrease in performance, you can substitute disk for main memory (virtual memory, typically operating at 90% of the performance of an equivalent all-main-memory machine), or consolidate all your applications on one machine (virtual-machine virtualization).
The benefits of multiple-system virtualization still tend to be performance vs. cost, in a more complicated fashion. Storage virtualization makes administration easier, and administrators cost money. Clustering simplifies the programmer’s and system administrator’s life, as well as allowing failover that makes systems more robust; the tradeoff is less scalable performance (generally somewhere between 60% and 80% increase in single-system-equivalent performance per system added, vs. up to 100% per processor added in symmetric multiprocessing for up to 30 or more processors). Meanwhile, virtualization of processes, single-system or multiple-system, delivers flexibility to the system administrator to allocate resources and manage workloads; while virtualization of data makes the programmer’s job easier as well as easing the storage or database administrator’s burden.
Viewed in this way, the definition of data virtualization is straightforward: it is the implementation of a multiple-system data-source veneer that lets programmers, administrators, and users all view multiple data sources as one. And, likewise, the benefits of data virtualization flow from its being virtualization and its being multiple-system/data-oriented. That is, there may be a decrease in query performance (but maybe not; some IBM studies show that a tuned Enterprise Information Integration [EII] engine can actually deliver better performance on cross-database queries, by load balancing effectively). However, you will also give the administrator a simpler view of the “data archipelagoes” typical within an enterprise; you will give the programmer “one transactional interface to choke”, just as SOAs give the programmer “one application interface”; and you will give users, ultimately, a broader access to data. Ultimately, that adds up to more flexibility and less cost.
Like other forms of virtualization, data virtualization is not a pipe dream; but it’s not easy to get there, either. The basics are an enterprise metadata repository and a “veneer” that makes multiple data sources look like one system. In various forms, EII, the data warehouse, master data management (MDM), and repository products give you these basics; but it is extraordinarily hard to get the solution’s hands around all the data in the system and its copies and its relationships, much less semi-automatically adjusting to all the new types of data that are coming in and the important data outside the enterprise on the semantic Web.
However, just as the other forms of virtualization are useful even though they are not implemented company-wide, so is data virtualization. Within a data center, within a line of business, or just applied to the company’s most important data (as in master data management), data virtualization can deliver its benefits in a targeted fashion – as EII and MDM have already shown.
So yes, data virtualization may be marketing hype, but it also represents an important new body of products with important benefits to IT and to the business as a whole. Later, I hope to get down to the details.
Let’s start with my definition. A “virtual” technology is one that detaches the logical scope of a system from its physical scope. More concretely, virtual memory, the virtual machine, virtual storage, etc. all make a subset or a superset of a physical system/resource like a computer, appear like one system/resource. Virtualization is the process of changing from the old physical way of seeing and handling a system to the “virtual” way.
Now note that all virtual technologies can either split up a single system or combine multiple systems, and that they can involve data or processes. For example, the virtual-machine meaning of virtualization (now its most common meaning) splits up the processes on a single machine into multiple virtual machines. The virtual-memory meaning of virtualization pretends that a small main memory (data) is really a large storage device. The storage virtualization meaning treats multiple systems (really data systems attached to multiple computer systems) as if they were one huge storage device. In fact, clustering and network file systems (really another form of virtualization) allow you to treat multiple systems (processes) as if they were one computer.
Here’s a neat little Table that captures how all the forms of virtualization that are being talked about these days fit into a nice neat framework (all right, it’s not really that neat in real life, but it’s a very good starting point).
,,,,,,,,Split Single System ,,,,,,,,,,,,,,,,,,, Combine Multiple Systems
Data: V. memory, V. disk ,,,,,,,,,,,,,,,,,, Storage virtualization, Data virtualization
Process: V. machine, app/desktop/profile v. ,,, Clustering, network file systems
Note that application, desktop, and profile virtualization may involve putting the virtual environment on another computer; but it’s still representing a part of a computer, not the whole thing.
Now all forms of virtualization share a common benefit of flexibility to users, but each category has specific benefits of its own. The benefit of single-system virtualization is a very good tradeoff of performance for cost (single-system); for much less cost and for a small decrease in performance, you can substitute disk for main memory (virtual memory, typically operating at 90% of the performance of an equivalent all-main-memory machine), or consolidate all your applications on one machine (virtual-machine virtualization).
The benefits of multiple-system virtualization still tend to be performance vs. cost, in a more complicated fashion. Storage virtualization makes administration easier, and administrators cost money. Clustering simplifies the programmer’s and system administrator’s life, as well as allowing failover that makes systems more robust; the tradeoff is less scalable performance (generally somewhere between 60% and 80% increase in single-system-equivalent performance per system added, vs. up to 100% per processor added in symmetric multiprocessing for up to 30 or more processors). Meanwhile, virtualization of processes, single-system or multiple-system, delivers flexibility to the system administrator to allocate resources and manage workloads; while virtualization of data makes the programmer’s job easier as well as easing the storage or database administrator’s burden.
Viewed in this way, the definition of data virtualization is straightforward: it is the implementation of a multiple-system data-source veneer that lets programmers, administrators, and users all view multiple data sources as one. And, likewise, the benefits of data virtualization flow from its being virtualization and its being multiple-system/data-oriented. That is, there may be a decrease in query performance (but maybe not; some IBM studies show that a tuned Enterprise Information Integration [EII] engine can actually deliver better performance on cross-database queries, by load balancing effectively). However, you will also give the administrator a simpler view of the “data archipelagoes” typical within an enterprise; you will give the programmer “one transactional interface to choke”, just as SOAs give the programmer “one application interface”; and you will give users, ultimately, a broader access to data. Ultimately, that adds up to more flexibility and less cost.
Like other forms of virtualization, data virtualization is not a pipe dream; but it’s not easy to get there, either. The basics are an enterprise metadata repository and a “veneer” that makes multiple data sources look like one system. In various forms, EII, the data warehouse, master data management (MDM), and repository products give you these basics; but it is extraordinarily hard to get the solution’s hands around all the data in the system and its copies and its relationships, much less semi-automatically adjusting to all the new types of data that are coming in and the important data outside the enterprise on the semantic Web.
However, just as the other forms of virtualization are useful even though they are not implemented company-wide, so is data virtualization. Within a data center, within a line of business, or just applied to the company’s most important data (as in master data management), data virtualization can deliver its benefits in a targeted fashion – as EII and MDM have already shown.
So yes, data virtualization may be marketing hype, but it also represents an important new body of products with important benefits to IT and to the business as a whole. Later, I hope to get down to the details.
Friday, June 20, 2008
Where are the EII Tools of Yesteryear?
All in all, it seems bizarre to me to realize that the old “pure” EII (Enterprise Information Integration) vendors are no longer thick on the ground. It was only 6 years ago that I first discovered EII tools and issued my first report – an extremely short half-life for a technology. And yet, of the existing or nascent EII tools then, Metamatrix has gone to Red Hat, Avaki to Sybase, Venetica to IBM, and another whose name I no longer recall was folded into BEA as Liquid Data and is now being re-folded into Oracle. Meanwhile, IBM has renamed Information Integrator as Federation Server, and rarely mentions EII. Of the oldies, only Composite Software and Attunity remain proudly independent.
And yet, the picture is not as bleak as it appears at first glance. Composite Software and Attunity are still small (under $100M in revenues), but continue to survive and even thrive. Ipedo and Denodo, newer entrants with interesting differentiators (Ipedo with its XML database, Denodo with its Web-search capabilities), are likewise out there. In fact, Stylus Studio with its EII support for programmers and Silver Creek Systems with its “missing link for EII” appear to have entered the fray recently.
Just as important, many of the infrastructure-software companies that have acquired EII vendors have come to realize that EII is a positive thing to customers and should be kept as a distinct product, not treated as another component to fold into a product or hide under the umbrella of an overall strategy. Sybase, especially, has been unusually active in positioning their EII solution as a “key component” of their key data-integration product suite.
However, it is also fair to say that EII’s marketing is not all it should be. A Google search turned up very few of these tools, and in the advertising on the right-hand side, several of the companies were not in EII at all (SAS? I don’t think so) and Sybase’s webcast on the subject was no longer available.
What’s going on? I would suggest that one of the key reasons for EII’s mixed picture of success is that vendors have begun doing what they should have done three years ago: leverage EII tools’ metadata repositories as separate products/value-add. Metadata management is now In, to the point that even Microsoft and EMC are now talking about enterprise-wide metadata repositories (and realizing just how tough the job is). In other words, some of the revenue and visibility that are derived from EII products are now showing up under the heading of master data management and composite apps (along the lines of Denodo’s “data mashups”) instead of being credited to EII.
This is not to say that EII is the only source of an enterprise metadata repository; EAI (Enterprise Application Integration) tools like IBM’s Ascential, data-search tools like EMC’s Tablus, and “pure” repository plays like Syspedia are other tools that can be used as the foundation of an enterprise-metadata-repository product. Still, I would argue that EII tools have virtues that none of the others share, or at least to the same extent. Specifically, creating a good repository is key to the success of a good EII implementation; such repositories are often used for querying apps, so optimizing query performance should be considered; and EII tools do both automatic discovery of a wide variety of automated data sources and support for hand-definition of data relationships and data with different names but the same meaning (“customer ID” or “buyer name”). Contrast this with EAI, which is usually restricted to a narrower range of structured data, is often handed the metadata rather than discovering it, does not have to consider real-time performance in queries, and often doesn’t provide support for the kind of data reconciliation involved in master data management. I was reminded of this recently, when I talked to Composite Software about a new Composite Discovery product whose foundation is an innovative way of discovering “probable” relationships between data items/records across data sources.
So the EII market is doing quite well, thank you, even if many of its pioneers have been folded into larger companies. However, EII remains under-utilized by large infrastructure-software vendors, not because their users don’t appreciate their EII implementations, but because vendors aren’t giving visibility to their EII technology’s usefulness in larger projects such as master data management and composite-application development. Where are the EII tools of yesteryear? Doing well; but not visibly enough.
And yet, the picture is not as bleak as it appears at first glance. Composite Software and Attunity are still small (under $100M in revenues), but continue to survive and even thrive. Ipedo and Denodo, newer entrants with interesting differentiators (Ipedo with its XML database, Denodo with its Web-search capabilities), are likewise out there. In fact, Stylus Studio with its EII support for programmers and Silver Creek Systems with its “missing link for EII” appear to have entered the fray recently.
Just as important, many of the infrastructure-software companies that have acquired EII vendors have come to realize that EII is a positive thing to customers and should be kept as a distinct product, not treated as another component to fold into a product or hide under the umbrella of an overall strategy. Sybase, especially, has been unusually active in positioning their EII solution as a “key component” of their key data-integration product suite.
However, it is also fair to say that EII’s marketing is not all it should be. A Google search turned up very few of these tools, and in the advertising on the right-hand side, several of the companies were not in EII at all (SAS? I don’t think so) and Sybase’s webcast on the subject was no longer available.
What’s going on? I would suggest that one of the key reasons for EII’s mixed picture of success is that vendors have begun doing what they should have done three years ago: leverage EII tools’ metadata repositories as separate products/value-add. Metadata management is now In, to the point that even Microsoft and EMC are now talking about enterprise-wide metadata repositories (and realizing just how tough the job is). In other words, some of the revenue and visibility that are derived from EII products are now showing up under the heading of master data management and composite apps (along the lines of Denodo’s “data mashups”) instead of being credited to EII.
This is not to say that EII is the only source of an enterprise metadata repository; EAI (Enterprise Application Integration) tools like IBM’s Ascential, data-search tools like EMC’s Tablus, and “pure” repository plays like Syspedia are other tools that can be used as the foundation of an enterprise-metadata-repository product. Still, I would argue that EII tools have virtues that none of the others share, or at least to the same extent. Specifically, creating a good repository is key to the success of a good EII implementation; such repositories are often used for querying apps, so optimizing query performance should be considered; and EII tools do both automatic discovery of a wide variety of automated data sources and support for hand-definition of data relationships and data with different names but the same meaning (“customer ID” or “buyer name”). Contrast this with EAI, which is usually restricted to a narrower range of structured data, is often handed the metadata rather than discovering it, does not have to consider real-time performance in queries, and often doesn’t provide support for the kind of data reconciliation involved in master data management. I was reminded of this recently, when I talked to Composite Software about a new Composite Discovery product whose foundation is an innovative way of discovering “probable” relationships between data items/records across data sources.
So the EII market is doing quite well, thank you, even if many of its pioneers have been folded into larger companies. However, EII remains under-utilized by large infrastructure-software vendors, not because their users don’t appreciate their EII implementations, but because vendors aren’t giving visibility to their EII technology’s usefulness in larger projects such as master data management and composite-application development. Where are the EII tools of yesteryear? Doing well; but not visibly enough.
Subscribe to:
Posts (Atom)