When IBM announced that the next rev of DB2 would support Oracle PL/SQL stored procedures, native, it seemed like a good deal. As I understand it, suppose I want to move an Oracle database function onto DB2. Before, I would have had to rewrite the PL/SQL stored procedure for the business rule in DB2’s language; now, I just recompile the SP I had on the Oracle machine. Then I can redirect any apps that use the Oracle SP to DB2, which is less of a modification than also changing the SP call in the app. It’s native, so there’s little degradation in performance. Neat, yes?
Well, yes, except that (a) in many cases, I’ve just duplicated much of the functionality of a similar existing DB2 SP, and (b) I don’t care how native it is, it doesn’t run as well as the DB2-language SP – after all, that language was designed for DB2 25 years ago. If we do this a lot, we create a situation in which we have SP A on Oracle, plus SP A and A’ (the DB2-language one) on DB2, all of which have to be upgraded together, for lots and lots of As. We’ve already got SP proliferation, with no records of which apps are calling the SP and hence a difficult app modification when we change the SP; now you want to make it worse?
This kind of thinking is outdated. It assumes the old 1990s environments of one app per database and one database vendor per enterprise. Those days are gone; now, in most situations, you have multiple databases from multiple vendors, each servicing more than one app (and composite app!).
So what’s the answer? Well, in cases like this, I tend to support the idea of “one interface to choke”. The idea is that there should be one standard SP language, with adapters to convert to multiple database SP source and compiled code – a “many to one to many” model. I send a PL/SQL SP to the converter, it puts it in standard language, then converts it to all “target” languages. It then sends it down to all target databases, where each compiles it and has it available.
Now suppose I invoke an SP originally written in PL/SQL, but it is to be applied to a DB2 database. The converter intercepts the PL/SQL-type invocation and converts it to a DB2-type format, and then (because I have told it to) redirects the invocation to the DB2 database. The database runs the DB2-type SP version, and returns the result to the converter, which performs a reverse conversion to deliver the result back to the invoking app.
The cool thing about this is that when you migrate or copy an SP, no app needs to be changed – remember, you simply tell the converter to redirect the invocation. The usual objection is, yes, but the converted SP doesn’t run as well as one written for the target database. True (actually, in the real world sometimes not true!), but the difference is marginal, and, in case you hadn’t noticed, there’s less pressure these days to get the absolute maximum out of a SP, with so many other performance knobs to twiddle.
Where’s the best place to set up such a converter? Why, an EII (Enterprise Information Integration) tool! It has the architecture – it takes SQL/XQL and translates/adapts for various databases, then reverses the translation for the results. Yes, we have to add stored procedure conversion to its tasks, but at least the architecture is in place, and we can integrate SP invocation with general SQL-handling.
And, by the way, we will have a metadata repository of stored procedures, all served up and ready for use in event-driven architectures.
What do you think?
Friday, April 24, 2009
Tuesday, April 21, 2009
Sun Microsystems: In Memoriam
So many memories …
I first became really aware of Sun in the late ‘80s, when I was working for Prime. At the time, Sun was one of the two new competitors in the area of engineering workstations – itself a new market. The key area of competition was cross-machine file systems that made multiple workstations look like one system – in other words, you’d invoke a program on one machine, and if it didn’t reside there, the file system would do a remote procedure call (RPC) to the other. Sun’s system was called NFS.
Yes, Sun won that war – but the way it did it was a harbinger of things to come. With more than a little chutzpah, Sun insisted that Unix was the right way to do networked file systems. Now, at the time, there was nothing to indicate that Unix was better (or worse) than any other operating system for doing cross-machine operations. But Sun’s marketing tapped into a powerful Movement in computing. This Movement – geeks, first movers, technology enthusiasts, anti-establishment types – gave Sun a niche where established players like Prime and DEC could not crowd Sun off buyers’ short lists. The Movement was very pro-Unix, and that allowed Sun to establish itself as the engineering-workstation market leader.
Sun’s next marketing message appealed to the Movement very much: it said it was going down-market and attacking Microsoft. In fact, that became a feature of Sun for the next 15 years: Scott McNealy would get up at Sun sales, investor, and analyst events and make cracks about Bill Gates and Microsoft. Of course, when you looked closely at what was happening, that was pretty much hogwash: Sun wasn’t cutting into the PC market, because it couldn’t cut prices that much. Instead, Sun’s pricing was cutting into minicomputer low-end markets. Because PCs and Novell LANs were cutting into those markets far more, the demise of minicomputer vendors is rightly ascribed to PCs. But Sun’s real market growth came from moving up-market.
As everyone remembers, Scott McNealy as the public face of Sun had a real gift for pithy phrases criticizing competitors that really stuck in people’s minds. My favorite is the time in the early 1990s when IBM as Big Blue joined with Apple (corporate color: red) in a consortium to develop a common standard for some key market and crowd others out: Scott derided the result as “purple applesauce.”
But what really saved Sun in the early 90s was not the Movement nor Scott’s gift for credulity-straining claims. First among engineering-workstation vendors, Sun decided to move into servers. This took Sun from techie markets (although not consumer markets) to medium-scale to large-scale corporate IT – not the easiest market to crack. But at the time, lines of business were asserting their independence from central IT by creating their own corporate networks, and Sun was able to position itself against IBM, Unisys, NCR/AT&T, and HP in growing medium-scale businesses and lines of business. While Silicon Graphics (number 2 in workstations) waited too long to move into servers and spent too much time trying to move down-market to compete with Microsoft, Sun grew in servers as the workstation market matured.
I remember talking to the trade press at that time and saying that Sun’s days of 90%/year revenue growth were over. As a large company, you couldn’t grow as fast as a smaller one, and especially not in the server market. I wasn’t wrong; but I certainly didn’t anticipate Sun’s amazing growth rate in the late 90s. It was all due to the Internet boom in Silicon Valley. Every startup wanted “an Oracle on a Sun”. Sun marketing positioned Java as part of the Movement – an object-oriented way of cutting through proprietary barriers to porting applications from one machine to another – and all the anti-Microsoft users flocked to Sun. Never mind the flaws in the language or the dip in programmer productivity as Java encouraged lower-level programming for a highly complex architecture; the image was what mattered.
Sun’s marketing chutzpah reached its height in those years. I remember driving down the Southeast Expressway in Boston one day and seeing a Sun billboard that said “We created the Internet. Let us help you create your Internet.” Well, I was at Computer Corporation of America with full access to the Internet back in the early 1980s when the Internet was being “created”, and I can tell you that BBN was the primary creator of the Internet aside from the government and academia, and Sun was far less visible in Internet newsgroups than most other major computing vendors. Yet when I pointed this out to a Sun marketer, he was honestly surprised. Ah, the Koolaid was sweet in those days.
It is fashionable now to say that Sun’s downfall came because it was late to embrace Linux. It is certainly true that Sun’s refusal to move aggressively to Linux cost it badly, especially because it ran counter to the Movement, and my then colleague Bill Claybrook deserves lots of credit for pushing them early and hard to move to Linux. But I think the real mistake was in not moving from a company focused on hardware to one focused on software during the Internet-boom years. Oh, there were all sorts of excuses – B-school management theory said you should focus on services, and Sun did beef up its middleware – but it was always reacting, always behind, never really focused on differentiation via software.
The mood at analyst meetings during the Internet-bust years was highly defensive: You guys don’t get us, we’ve shown before that we see things no one else sees and we’ll do it again, all our competitors are suffering too. And yet, the signs were there: talking to an Oracle rep, it became clear that Sun was no longer one of their key partners.
I am less critical of Jonathan Schwartz than some other commentators I have read. I think that he was dealt a hand that would lose no matter how he played it. The Internet-boom users had gone forever, leaving a Sun with too high a cost structure to make money from the larger corporations and financial-services markets that remained. In fact, I think that however well he executed, he was right to focus on open source (thereby making peace with the Movement) and software. At my last Sun do in 2007 when I was with Illuminata, the sense of innovation in sync with the Movement was palpable – even if Sun was mostly catching up to what other open-source and Web efforts like Google’s were doing. But underlying the marketers’ bravado at that event was depression at the endless layoffs that were slowly paring back the company. The analysts’ dinner was populated as much by the ghosts of past Sun employees as by those that remained.
Even as a shadow of its former self, I am going to miss Sun. I am going to miss the techie enthusiasm that produced some really good if way over-hyped ideas that continue to help move the industry forward. I am going to miss many of the smart, effective marketers and technologists still there that will probably never again get such a bully pulpit. I am going to miss a competitor that still was part of the ongoing debate about the Next Big Thing, a debate which more often than not has produced the Next Big Thing. I’m not going to miss disentangling marketing claims that sound good but aren’t true, or competitor criticisms that are great sound bites but miss the point, while watching others swallow the Sun line, hook and sinker included; but the days when Sun’s misdeeds in those areas mattered are long past.
Rest in peace, Sun. All you employees and Sun alumni, take a bow.
I first became really aware of Sun in the late ‘80s, when I was working for Prime. At the time, Sun was one of the two new competitors in the area of engineering workstations – itself a new market. The key area of competition was cross-machine file systems that made multiple workstations look like one system – in other words, you’d invoke a program on one machine, and if it didn’t reside there, the file system would do a remote procedure call (RPC) to the other. Sun’s system was called NFS.
Yes, Sun won that war – but the way it did it was a harbinger of things to come. With more than a little chutzpah, Sun insisted that Unix was the right way to do networked file systems. Now, at the time, there was nothing to indicate that Unix was better (or worse) than any other operating system for doing cross-machine operations. But Sun’s marketing tapped into a powerful Movement in computing. This Movement – geeks, first movers, technology enthusiasts, anti-establishment types – gave Sun a niche where established players like Prime and DEC could not crowd Sun off buyers’ short lists. The Movement was very pro-Unix, and that allowed Sun to establish itself as the engineering-workstation market leader.
Sun’s next marketing message appealed to the Movement very much: it said it was going down-market and attacking Microsoft. In fact, that became a feature of Sun for the next 15 years: Scott McNealy would get up at Sun sales, investor, and analyst events and make cracks about Bill Gates and Microsoft. Of course, when you looked closely at what was happening, that was pretty much hogwash: Sun wasn’t cutting into the PC market, because it couldn’t cut prices that much. Instead, Sun’s pricing was cutting into minicomputer low-end markets. Because PCs and Novell LANs were cutting into those markets far more, the demise of minicomputer vendors is rightly ascribed to PCs. But Sun’s real market growth came from moving up-market.
As everyone remembers, Scott McNealy as the public face of Sun had a real gift for pithy phrases criticizing competitors that really stuck in people’s minds. My favorite is the time in the early 1990s when IBM as Big Blue joined with Apple (corporate color: red) in a consortium to develop a common standard for some key market and crowd others out: Scott derided the result as “purple applesauce.”
But what really saved Sun in the early 90s was not the Movement nor Scott’s gift for credulity-straining claims. First among engineering-workstation vendors, Sun decided to move into servers. This took Sun from techie markets (although not consumer markets) to medium-scale to large-scale corporate IT – not the easiest market to crack. But at the time, lines of business were asserting their independence from central IT by creating their own corporate networks, and Sun was able to position itself against IBM, Unisys, NCR/AT&T, and HP in growing medium-scale businesses and lines of business. While Silicon Graphics (number 2 in workstations) waited too long to move into servers and spent too much time trying to move down-market to compete with Microsoft, Sun grew in servers as the workstation market matured.
I remember talking to the trade press at that time and saying that Sun’s days of 90%/year revenue growth were over. As a large company, you couldn’t grow as fast as a smaller one, and especially not in the server market. I wasn’t wrong; but I certainly didn’t anticipate Sun’s amazing growth rate in the late 90s. It was all due to the Internet boom in Silicon Valley. Every startup wanted “an Oracle on a Sun”. Sun marketing positioned Java as part of the Movement – an object-oriented way of cutting through proprietary barriers to porting applications from one machine to another – and all the anti-Microsoft users flocked to Sun. Never mind the flaws in the language or the dip in programmer productivity as Java encouraged lower-level programming for a highly complex architecture; the image was what mattered.
Sun’s marketing chutzpah reached its height in those years. I remember driving down the Southeast Expressway in Boston one day and seeing a Sun billboard that said “We created the Internet. Let us help you create your Internet.” Well, I was at Computer Corporation of America with full access to the Internet back in the early 1980s when the Internet was being “created”, and I can tell you that BBN was the primary creator of the Internet aside from the government and academia, and Sun was far less visible in Internet newsgroups than most other major computing vendors. Yet when I pointed this out to a Sun marketer, he was honestly surprised. Ah, the Koolaid was sweet in those days.
It is fashionable now to say that Sun’s downfall came because it was late to embrace Linux. It is certainly true that Sun’s refusal to move aggressively to Linux cost it badly, especially because it ran counter to the Movement, and my then colleague Bill Claybrook deserves lots of credit for pushing them early and hard to move to Linux. But I think the real mistake was in not moving from a company focused on hardware to one focused on software during the Internet-boom years. Oh, there were all sorts of excuses – B-school management theory said you should focus on services, and Sun did beef up its middleware – but it was always reacting, always behind, never really focused on differentiation via software.
The mood at analyst meetings during the Internet-bust years was highly defensive: You guys don’t get us, we’ve shown before that we see things no one else sees and we’ll do it again, all our competitors are suffering too. And yet, the signs were there: talking to an Oracle rep, it became clear that Sun was no longer one of their key partners.
I am less critical of Jonathan Schwartz than some other commentators I have read. I think that he was dealt a hand that would lose no matter how he played it. The Internet-boom users had gone forever, leaving a Sun with too high a cost structure to make money from the larger corporations and financial-services markets that remained. In fact, I think that however well he executed, he was right to focus on open source (thereby making peace with the Movement) and software. At my last Sun do in 2007 when I was with Illuminata, the sense of innovation in sync with the Movement was palpable – even if Sun was mostly catching up to what other open-source and Web efforts like Google’s were doing. But underlying the marketers’ bravado at that event was depression at the endless layoffs that were slowly paring back the company. The analysts’ dinner was populated as much by the ghosts of past Sun employees as by those that remained.
Even as a shadow of its former self, I am going to miss Sun. I am going to miss the techie enthusiasm that produced some really good if way over-hyped ideas that continue to help move the industry forward. I am going to miss many of the smart, effective marketers and technologists still there that will probably never again get such a bully pulpit. I am going to miss a competitor that still was part of the ongoing debate about the Next Big Thing, a debate which more often than not has produced the Next Big Thing. I’m not going to miss disentangling marketing claims that sound good but aren’t true, or competitor criticisms that are great sound bites but miss the point, while watching others swallow the Sun line, hook and sinker included; but the days when Sun’s misdeeds in those areas mattered are long past.
Rest in peace, Sun. All you employees and Sun alumni, take a bow.
Thursday, April 16, 2009
Tuesday, April 7, 2009
System Dynamics and Business Agility
Never having had the chance to study system dynamics at Sloan School of Management (MIT), I was very happy recently to have the opportunity to read Donella Meadows’ “Thinking in Systems”, an excellent primer on the subject – I recommend it highly. Reading the book sparked some thoughts on how system dynamics and the concept of business and IT agility complement each other – and more importantly, how they challenge each other fundamentally.
Let’s start with the similarities. System dynamics says that most systems grow over time; my concept of business agility would argue that growth is a type of change, and agile businesses should do better at handling that type of change. System dynamics says that people have a lot to say about system functioning, and people resist change; I would argue that business agility includes building organizations in which people expect and know how to handle change, because they know what to do. System dynamics says that to change system behavior, it is better to change the system than replace components (including people); business agility says that business processes if changed can increase the agility of the company, even if the same people are involved.
What about the differences? System dynamics really doesn’t have a good analog for the proactive side of agility. They mention resilience, which is really the ability of a system to react well to a wider range of external changes, they mention “self-organization” as elaborating the complexity of systems, and they talk about a system having a certain amount of room to grow without reaching constraints or limits; but there is an implicit assumption that unexpected or planned change is the exception, not the norm. Likewise, according to system dynamics, tuning the system to handle changes better is in the long run simply delaying the inevitable; a more effective redesign changes the system itself, as profoundly as possible. Agility says that change is the norm, that redesign should be aimed at improving the ability to “proact” and the ability to react, and that increased agility has a value independent of what system is being used.
System dynamics poses challenges to the practice of business agility, as well. It says that how agility is to be improved matters: have we found the right “leverage point” for the improvement, have we understood well enough how people will “game the system”, have we anticipated future scenarios in which the “agilified” process generates new constraints and reaches new limits?
To my mind, the key question that system dynamics raises about business agility is, are we measuring it without incorporating the importance of the unmeasurable? Or, to put it in system-dynamics terms, in attempting to capture the business value of increased agility in terms of costs, revenues, and upside and downside risks, are we “emphasizing quantity over quality”?
I think, based on the data on agility improvements I’ve seen so far, that one of the most interesting ideas about business agility is that focusing on agility results in doing better in long-term costs, revenues, and upside/downside risks than a strategy focused on costs, revenues, or risks themselves. If this is true, and if organizations set out to improve agility “for agility’s sake”, I don’t think system dynamics and agility strategies are in disagreement: both want to create a process, an organization, a business built to do the right thing more often (“quality”), not one to improve a cost or revenue metric (“quantity”). Or, as Tom Lehrer the comedian once put it, we are “doing well by doing good”.
So my most important take-away from gaining an admittedly basic understanding of system dynamics is that metrics like AFI (agility from investment, which attempts to measure the long-term effects of a change in agility on costs, revenues, and risks) explain the relative agility effects of various strategies, but should not be used to justify strategies not focused on agility that may improve costs, revenues, and/or risks in the short term, but will actually have a negative effect in the long term. As Yoda in Star Wars might put it: “Build to change or be not agile; there is no accidental agility.”
Let’s start with the similarities. System dynamics says that most systems grow over time; my concept of business agility would argue that growth is a type of change, and agile businesses should do better at handling that type of change. System dynamics says that people have a lot to say about system functioning, and people resist change; I would argue that business agility includes building organizations in which people expect and know how to handle change, because they know what to do. System dynamics says that to change system behavior, it is better to change the system than replace components (including people); business agility says that business processes if changed can increase the agility of the company, even if the same people are involved.
What about the differences? System dynamics really doesn’t have a good analog for the proactive side of agility. They mention resilience, which is really the ability of a system to react well to a wider range of external changes, they mention “self-organization” as elaborating the complexity of systems, and they talk about a system having a certain amount of room to grow without reaching constraints or limits; but there is an implicit assumption that unexpected or planned change is the exception, not the norm. Likewise, according to system dynamics, tuning the system to handle changes better is in the long run simply delaying the inevitable; a more effective redesign changes the system itself, as profoundly as possible. Agility says that change is the norm, that redesign should be aimed at improving the ability to “proact” and the ability to react, and that increased agility has a value independent of what system is being used.
System dynamics poses challenges to the practice of business agility, as well. It says that how agility is to be improved matters: have we found the right “leverage point” for the improvement, have we understood well enough how people will “game the system”, have we anticipated future scenarios in which the “agilified” process generates new constraints and reaches new limits?
To my mind, the key question that system dynamics raises about business agility is, are we measuring it without incorporating the importance of the unmeasurable? Or, to put it in system-dynamics terms, in attempting to capture the business value of increased agility in terms of costs, revenues, and upside and downside risks, are we “emphasizing quantity over quality”?
I think, based on the data on agility improvements I’ve seen so far, that one of the most interesting ideas about business agility is that focusing on agility results in doing better in long-term costs, revenues, and upside/downside risks than a strategy focused on costs, revenues, or risks themselves. If this is true, and if organizations set out to improve agility “for agility’s sake”, I don’t think system dynamics and agility strategies are in disagreement: both want to create a process, an organization, a business built to do the right thing more often (“quality”), not one to improve a cost or revenue metric (“quantity”). Or, as Tom Lehrer the comedian once put it, we are “doing well by doing good”.
So my most important take-away from gaining an admittedly basic understanding of system dynamics is that metrics like AFI (agility from investment, which attempts to measure the long-term effects of a change in agility on costs, revenues, and risks) explain the relative agility effects of various strategies, but should not be used to justify strategies not focused on agility that may improve costs, revenues, and/or risks in the short term, but will actually have a negative effect in the long term. As Yoda in Star Wars might put it: “Build to change or be not agile; there is no accidental agility.”
Tuesday, March 24, 2009
Thoughts on Information-Handling Agility
Recently I’ve been writing down some thoughts about business and IT agility: What they are, how they evidence themselves in the bottom line and in risk (or its proxy, public-company beta), and how to measure them. At the same time, in my study of “data usefulness” (how much potentially useful data actually gets used effectively by the appropriate target), I included a factor called ‘data agility,’ or the ability of the organization to keep up to date with new useful data sources. What I want to do now is consider a larger set of questions: what does agility mean in the context of the organizational process that ideally gathers all potentially useful information in a timely fashion and leverages it effectively, how can we measure it, and what offers the biggest “bang for the agility-investment buck”?
My initial pass at “information-handling agility” is there are four sources of change that are key: Unexpected changes in the environment, planned changes in the non-informational organization/process (which also should cover expected changes in the environment), unplanned changes in the non-informational organization, and new data sources/types. Therefore, information-handling agility includes the ability to react rapidly and effectively in supplying information about unexpected changes in the environment, the proactively planned but timely supply of information about expected changes in the environment, the ability to react rapidly and effectively by supplying different types of information due to an unexpected internal change, and the ability to proactively seek and effectively use new data sources.
Note that, strictly speaking, this doesn’t cover all cases. For example, it doesn’t cover outside change during the information-handling process – but that’s reasonable, if in most cases that change either doesn’t change the ultimate information use or it’s so important that it’s already handled by special “alerts”, as seems to be the case in real-world businesses. Likewise, the definition of data agility doesn’t include all changes in the data, rather than just the new data-source ones; again, in the real world this seems to be much less of a problem.
To see how this can be measured and what offers the biggest “bang for the buck,” let’s create a “thought experiment”. Let’s take Firm A, a typical firm in my “data usefulness” research, and apply the Agility From Investment (AFI) metric, defined as AFI = (1 + % change [revenues] – % change [development and operational change in costs]) * (1 + %change [upside risk] - % change [downside risk]) - 1. Let’s assume that Firm A invests in decreasing the time it takes to deliver data to the average user from 7 days to 3 ½ days – and ensures that the data can be used as effectively as before. Btw, the different types of agility won’t show up again, but they underlie the analysis.
We can see that in the long term, assuming its competitors don’t match it, the “timeliness” strategy will improve revenues by increasing the agility of new-product development – but only if new-product development is agile itself. If we assume an “average” NPD out there of ½ the firms being “agile enough”, then we have 15% improvement in ROI x ½ = 7 ½ % (the maximum change in long-term revenues). Since we have only improved timeliness by ½, the maximum change becomes 3 ¾ %; the typical data usefulness is about 1/3, taking it down to 1 ¼ %; and information’s share of this takes it below 1%.
Costs are seemingly a different story. Reducing time to deliver information affects not only the per-transaction IT costs of delivery, but also every business process that depends on that information. So it is reasonable to assume a 1% decrease in NPD costs, but also a 5% decrease in operational costs, for an average of 3%. Meanwhile, the increase in upside risk goes through a similar computation as for revenues, yielding less that a 1% increase in that type of risk.
That leaves downside risk. Those risks appear to be primarily failure to get information in time to react effectively to a disaster, and failure to get the right information to act effectively. Because the effect on risk increases as the response time gets closer to zero, it is reasonable to estimate the effect on downside risk at perhaps a 5% decrease; and since only 1/3 of the data is useful, that takes it down below 2%. Putting it all together, AFI = (1 + 1% + 3%) * (1 + 1% + 2%) – 1 = a 7% overall improvement in the corporation’s bottom line and risk – and that’s being optimistic.
Now suppose we invested in doubling the percentage of potentially useful data that is effectively used – i.e., not timeliness but accuracy, consistency, scope, fit with the needs of the user/business, and analyzeability. Performing the same computations, I come out with AFI = (1 + 1% + 1.5%) * (1 + 7.5% + 1%) – 1 = an 11% long-term agility improvement.
One more experiment: suppose we invested in immediately identifying key new data sources and pouring them into our information-handling process, rather than waiting ½ year or more. Again, applying the same computations, but with one more assumption (a high degree of rapid change in the sources of key data), AFI = (1 + 2% + 2%) * (1 + 7.5% + 8%) – 1 = a 20% improvement in long-term contribution to the company’s bottom line.
Now, mind you, I have carefully shaped my assumptions, so please do not assume that this analysis is exactly what any firm will experience over the long term. There are, however, two take-aways that do not seem to be part of the general consensus today.
First, firms are probably typically underestimating the long-term effects of efforts aimed at improving data usefulness (including timeliness, effectiveness, and data agility). Reasonably enough, they are concerned with immediate decision-making and strategies that affect information-handling tangentially and piecemeal. However, the result, as I have noted, is a “whack-a-mole” game in which no sooner is one information-handling problem tackled than another pops up.
Second, firms are also clearly underestimating the long-term benefits of improving data usefulness compared to improving timeliness, and of improving data agility compared to improving both timeliness and data usefulness. The reason for that appears to be that firms don’t appreciate the value for new-product development of inserting better and new data in the new-product development process, compared to more timely delivery of the same old data.
I offer up one action item: IT organizations should seriously consider adding a Data Agility role. The job would be monitoring all organizational sources of data from the environment – especially the Web – and ensuring that they are appropriately added to the information-handling inputs and process as soon as possible.
My initial pass at “information-handling agility” is there are four sources of change that are key: Unexpected changes in the environment, planned changes in the non-informational organization/process (which also should cover expected changes in the environment), unplanned changes in the non-informational organization, and new data sources/types. Therefore, information-handling agility includes the ability to react rapidly and effectively in supplying information about unexpected changes in the environment, the proactively planned but timely supply of information about expected changes in the environment, the ability to react rapidly and effectively by supplying different types of information due to an unexpected internal change, and the ability to proactively seek and effectively use new data sources.
Note that, strictly speaking, this doesn’t cover all cases. For example, it doesn’t cover outside change during the information-handling process – but that’s reasonable, if in most cases that change either doesn’t change the ultimate information use or it’s so important that it’s already handled by special “alerts”, as seems to be the case in real-world businesses. Likewise, the definition of data agility doesn’t include all changes in the data, rather than just the new data-source ones; again, in the real world this seems to be much less of a problem.
To see how this can be measured and what offers the biggest “bang for the buck,” let’s create a “thought experiment”. Let’s take Firm A, a typical firm in my “data usefulness” research, and apply the Agility From Investment (AFI) metric, defined as AFI = (1 + % change [revenues] – % change [development and operational change in costs]) * (1 + %change [upside risk] - % change [downside risk]) - 1. Let’s assume that Firm A invests in decreasing the time it takes to deliver data to the average user from 7 days to 3 ½ days – and ensures that the data can be used as effectively as before. Btw, the different types of agility won’t show up again, but they underlie the analysis.
We can see that in the long term, assuming its competitors don’t match it, the “timeliness” strategy will improve revenues by increasing the agility of new-product development – but only if new-product development is agile itself. If we assume an “average” NPD out there of ½ the firms being “agile enough”, then we have 15% improvement in ROI x ½ = 7 ½ % (the maximum change in long-term revenues). Since we have only improved timeliness by ½, the maximum change becomes 3 ¾ %; the typical data usefulness is about 1/3, taking it down to 1 ¼ %; and information’s share of this takes it below 1%.
Costs are seemingly a different story. Reducing time to deliver information affects not only the per-transaction IT costs of delivery, but also every business process that depends on that information. So it is reasonable to assume a 1% decrease in NPD costs, but also a 5% decrease in operational costs, for an average of 3%. Meanwhile, the increase in upside risk goes through a similar computation as for revenues, yielding less that a 1% increase in that type of risk.
That leaves downside risk. Those risks appear to be primarily failure to get information in time to react effectively to a disaster, and failure to get the right information to act effectively. Because the effect on risk increases as the response time gets closer to zero, it is reasonable to estimate the effect on downside risk at perhaps a 5% decrease; and since only 1/3 of the data is useful, that takes it down below 2%. Putting it all together, AFI = (1 + 1% + 3%) * (1 + 1% + 2%) – 1 = a 7% overall improvement in the corporation’s bottom line and risk – and that’s being optimistic.
Now suppose we invested in doubling the percentage of potentially useful data that is effectively used – i.e., not timeliness but accuracy, consistency, scope, fit with the needs of the user/business, and analyzeability. Performing the same computations, I come out with AFI = (1 + 1% + 1.5%) * (1 + 7.5% + 1%) – 1 = an 11% long-term agility improvement.
One more experiment: suppose we invested in immediately identifying key new data sources and pouring them into our information-handling process, rather than waiting ½ year or more. Again, applying the same computations, but with one more assumption (a high degree of rapid change in the sources of key data), AFI = (1 + 2% + 2%) * (1 + 7.5% + 8%) – 1 = a 20% improvement in long-term contribution to the company’s bottom line.
Now, mind you, I have carefully shaped my assumptions, so please do not assume that this analysis is exactly what any firm will experience over the long term. There are, however, two take-aways that do not seem to be part of the general consensus today.
First, firms are probably typically underestimating the long-term effects of efforts aimed at improving data usefulness (including timeliness, effectiveness, and data agility). Reasonably enough, they are concerned with immediate decision-making and strategies that affect information-handling tangentially and piecemeal. However, the result, as I have noted, is a “whack-a-mole” game in which no sooner is one information-handling problem tackled than another pops up.
Second, firms are also clearly underestimating the long-term benefits of improving data usefulness compared to improving timeliness, and of improving data agility compared to improving both timeliness and data usefulness. The reason for that appears to be that firms don’t appreciate the value for new-product development of inserting better and new data in the new-product development process, compared to more timely delivery of the same old data.
I offer up one action item: IT organizations should seriously consider adding a Data Agility role. The job would be monitoring all organizational sources of data from the environment – especially the Web – and ensuring that they are appropriately added to the information-handling inputs and process as soon as possible.
Saturday, January 17, 2009
Random Thoughts on Agility
The more I write about agile software development, Key Agility Indicators, and users seeing an environment of rapid change as their most worrisome business pressure, the more I wonder why agility, or flexibility, is not a standard way of assessing how a business is doing. Here's my argument:
Agility is a different measure and target from costs or revenues or risks. It's really about the ability of the organization to respond to significant changes in its normal functioning or new demands from outside, rapidly and effectively. It's not just costs, because a more agile organization will reap added revenues by beating out its competitors for business and creating new markets. It's not just profits or revenues, because a more agile organization can also be more costly, just as an engine tuned for one speed can perform better at that speed than one tuned to minimize the cost of changing speeds; and bad changes, such as a downturn in the economy, may decrese your revenues no matter how agile you are. It's not just risk, because agility should involve responding well to positive risks and changes as well as negative ones, and often can involve generating changes in the organization without or before any pressures or risks.
That said, we should understand how increased or decreased agility impacts other business measures, just as we should understand how increased costs affect cash flow, profits, and business risks, or increased revenues affect costs (e.g., are we past the point where marginal revenue = marginal cost?), or the likelihood that computer failures will croak the business. My conjecture is that increased agility will always decrease downside risk, but should increase upside risk. Likewise, increased agility that exceeds the competition's rate of agility improvement will always have a positive effect on gross margin over time, whether through more rapid implementation of cost-decreasing measures or an effective competitive edge in implementing new products that increase revenues and margins. And, of course, decreased agility will operate in the opposite direction. However, the profit effects will in many cases be hard to detect, both because of stronger trends from the economy and from unavoidable disasters, and because the rate of change in the environment may vary.
How to measure agility? At the product-development level, the answer seems faily easy: lob a major change at the process and see how it reacts. Major changes happen all the time, so it's not as if we can't come up with some baseline and some way of telling whether our organization is doing better or worse than before.
At the financial-statement level, the answer isn't as obvious. Iirc, IBM suggested a measure like inventory turnover. Yes, if you speed up production, certainly you can react to an increase in sales better; but what I believe we're really talking about is a derivative effect: for example, a change in the level of sales OVER a change in cost of goods sold, or a percent change in product mix over the percent change in cost of goods sold, or change in financial leverage over change in revenues (a proxy for the ability to introduce better new products faster?).
So I wonder if financial analysts shouldn't take a crack at the problem of measuring a firm's agility. It would certainly be interesting to figure out if some earnings surprises could have been partially preducted by a company's relative agility, or lack of it.
At the level of the economy, I guess I don't see an obvious application so far. Measures of frictional unemployment over total employment, I would think, would serve as a interesting take both on how much economic change is going on and to what extent comparative advantage is shifting. But I'm not sure that they would also serve to get at how well and how quickly a nation's economy is responding to these changes. I suppose we could look at companies' gross margin changes over the short term in a particular industry compared to overall industry gross margin changes to guess at each company's relative agility in responding to changes in the industry. However, that's not a good cross-industry yardstick...
And finally, is this something where unforeseen factors make any measurement iffy? If what drives long-term success is innovation, which is driven by luck, then you can be very agile and still lose out to a competitor who is moderately agile and comes up with a couple of once-in-a-genertion market-defining good new products.
Everyone talks about the weather, Mark Twain said, but nobody does anything about it. Well, everyone's talking about agility, and lots of people are doing something about it; but I don't think anybody really knows how effective their efforts are. Ideas, anyone?
Agility is a different measure and target from costs or revenues or risks. It's really about the ability of the organization to respond to significant changes in its normal functioning or new demands from outside, rapidly and effectively. It's not just costs, because a more agile organization will reap added revenues by beating out its competitors for business and creating new markets. It's not just profits or revenues, because a more agile organization can also be more costly, just as an engine tuned for one speed can perform better at that speed than one tuned to minimize the cost of changing speeds; and bad changes, such as a downturn in the economy, may decrese your revenues no matter how agile you are. It's not just risk, because agility should involve responding well to positive risks and changes as well as negative ones, and often can involve generating changes in the organization without or before any pressures or risks.
That said, we should understand how increased or decreased agility impacts other business measures, just as we should understand how increased costs affect cash flow, profits, and business risks, or increased revenues affect costs (e.g., are we past the point where marginal revenue = marginal cost?), or the likelihood that computer failures will croak the business. My conjecture is that increased agility will always decrease downside risk, but should increase upside risk. Likewise, increased agility that exceeds the competition's rate of agility improvement will always have a positive effect on gross margin over time, whether through more rapid implementation of cost-decreasing measures or an effective competitive edge in implementing new products that increase revenues and margins. And, of course, decreased agility will operate in the opposite direction. However, the profit effects will in many cases be hard to detect, both because of stronger trends from the economy and from unavoidable disasters, and because the rate of change in the environment may vary.
How to measure agility? At the product-development level, the answer seems faily easy: lob a major change at the process and see how it reacts. Major changes happen all the time, so it's not as if we can't come up with some baseline and some way of telling whether our organization is doing better or worse than before.
At the financial-statement level, the answer isn't as obvious. Iirc, IBM suggested a measure like inventory turnover. Yes, if you speed up production, certainly you can react to an increase in sales better; but what I believe we're really talking about is a derivative effect: for example, a change in the level of sales OVER a change in cost of goods sold, or a percent change in product mix over the percent change in cost of goods sold, or change in financial leverage over change in revenues (a proxy for the ability to introduce better new products faster?).
So I wonder if financial analysts shouldn't take a crack at the problem of measuring a firm's agility. It would certainly be interesting to figure out if some earnings surprises could have been partially preducted by a company's relative agility, or lack of it.
At the level of the economy, I guess I don't see an obvious application so far. Measures of frictional unemployment over total employment, I would think, would serve as a interesting take both on how much economic change is going on and to what extent comparative advantage is shifting. But I'm not sure that they would also serve to get at how well and how quickly a nation's economy is responding to these changes. I suppose we could look at companies' gross margin changes over the short term in a particular industry compared to overall industry gross margin changes to guess at each company's relative agility in responding to changes in the industry. However, that's not a good cross-industry yardstick...
And finally, is this something where unforeseen factors make any measurement iffy? If what drives long-term success is innovation, which is driven by luck, then you can be very agile and still lose out to a competitor who is moderately agile and comes up with a couple of once-in-a-genertion market-defining good new products.
Everyone talks about the weather, Mark Twain said, but nobody does anything about it. Well, everyone's talking about agility, and lots of people are doing something about it; but I don't think anybody really knows how effective their efforts are. Ideas, anyone?
Monday, January 5, 2009
Some Personal Thoughts on Agile Development - Finale
So far, I’ve talked about my personal experiences as a programmer that have led me to anticipate – apparently correctly – that agile development would deliver consistently better results by cost, profit, and agility metrics. What about the down side? Or, to put it another way, what else could users do that agile development hasn’t done?
After I left Prime, I started as a computer industry analyst at The Yankee Group. I will always be grateful to Yankee for giving me the broadest possible canvas on which to paint my visions of what could be – as I used to put it, I covered “everything below the mainframe”. Of course, in the early 90s that was only ½ of the computing field … Anyway, one of the things I wrote was a comprehensive report called “Software Development: Think Again or Fail”. [yes, I know; I was a bit immodest in those days]
The reason I bring this up is that two things mentioned in that report seem to be missing in agile development:
1. High-level tools, languages, and components; and
2. Tasking programmers with keeping track of particular markets.
As far as I can see, agile theory and practice doesn’t give a hoot whether those programmers are using Java, Perl, Python, or Ruby on Rails. I use those examples because they all have been touted as ways to speed up programming in the Java/open-source world, and because only tunnel vision leads people to conclude that they’re anything but dolled-up 3GLs that do very well at handling function-driven programming and only adequately at rapidly generating data-access and user-interface code. Compare that to M204 UL, drag-and-drop VPEs (visual programming environments), and the like, and I am forced to conclude that in some respects, these tools are still less productive than what was available 13 years ago. The point is that, even if agile succeeds in improving the speed of the individual programmer, the other productivity shoe will never drop as long as the first instinct of managers and programmers is to reach for a 3GL.
The second point is that although agile does well with making sure that programmers talk to individual end users, that is different from following the entire software market. Following a market gives context to what the end user wants, and allows the designer to look at where the market appears to be going, rather than where end users have been.
So my caution about agile development is that my experience tells me that so much more can be done. The results are startling and consistent; but they could be more so. Agile development deserves praise; but the worst thing for software development would be to assume that no more fundamental changes in the paradigm need be done.
After I left Prime, I started as a computer industry analyst at The Yankee Group. I will always be grateful to Yankee for giving me the broadest possible canvas on which to paint my visions of what could be – as I used to put it, I covered “everything below the mainframe”. Of course, in the early 90s that was only ½ of the computing field … Anyway, one of the things I wrote was a comprehensive report called “Software Development: Think Again or Fail”. [yes, I know; I was a bit immodest in those days]
The reason I bring this up is that two things mentioned in that report seem to be missing in agile development:
1. High-level tools, languages, and components; and
2. Tasking programmers with keeping track of particular markets.
As far as I can see, agile theory and practice doesn’t give a hoot whether those programmers are using Java, Perl, Python, or Ruby on Rails. I use those examples because they all have been touted as ways to speed up programming in the Java/open-source world, and because only tunnel vision leads people to conclude that they’re anything but dolled-up 3GLs that do very well at handling function-driven programming and only adequately at rapidly generating data-access and user-interface code. Compare that to M204 UL, drag-and-drop VPEs (visual programming environments), and the like, and I am forced to conclude that in some respects, these tools are still less productive than what was available 13 years ago. The point is that, even if agile succeeds in improving the speed of the individual programmer, the other productivity shoe will never drop as long as the first instinct of managers and programmers is to reach for a 3GL.
The second point is that although agile does well with making sure that programmers talk to individual end users, that is different from following the entire software market. Following a market gives context to what the end user wants, and allows the designer to look at where the market appears to be going, rather than where end users have been.
So my caution about agile development is that my experience tells me that so much more can be done. The results are startling and consistent; but they could be more so. Agile development deserves praise; but the worst thing for software development would be to assume that no more fundamental changes in the paradigm need be done.
Subscribe to:
Posts (Atom)