This blog post highlights a software company and technology that I view as potentially useful to organizations investing in agile development, agile new product development, and business agility over the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about agile development today.
The Importance of Code Refactoring to Agile Development
Ever since agile development first surfaced in the Agile Manifesto, code refactoring has been an under-appreciated and often-ignored part of its value-add. My own fascination with the subject dates back to my Cornell days, when research there in how to find minimal fixes to program compiler errors led inevitably to the question: what is the best code design to allow minimal-effort additions to a program’s features? In the 1980s, research into “structured design” showed that modularized subroutines at the same level were far easier to extend than subroutines in a complex tree structure. When I first became aware of the agile development movement, I immediately saw the benefit of rapid program extension to development agility, and my search for such tools immediately yielded the first code refactoring toolsets.
In the decade since the advent of agile development, however, major development-tool vendors, even those who tout their leadership role in agile development, often ignore or are completely ignorant of the potential usefulness of code refactoring. Until recently, most IBM Rational marketers and some techies had never heard of refactoring, and today a quick Google search on “code refactoring” will turn up not one major vendor on its front page. Even when vendors are taking a longer-term view of agile development projects under the heading of “agile ALM (Application Lifecycle Management),” code refactoring is typically a minor or non-existent part of their “vision.”
The key reason for this neglect, I believe, is a sea-change in development that happened in the late 1990s, in which the up-to-two-year IT development logjam was forever broken by “disposable” software by the likes of Amazon – half of whose code, at one point, was less than six months old. In other words, in parallel with but not related to the rise of agile development, IT and vendors both acquired a belief that long-term “software sclerosis” such as had been happening to mainframe mission-critical apps, in which the app had reached a point where each bug fixed introduced a new one, and lack of documentation and code clarity meant that no one knew any longer what each module really did, no longer mattered – over time, those apps would be superseded by newer ones. And, at the time, this was pretty much true.
However, acceptance of agile development changes that equation yet again. Agile development operates by many, many incremental changes to a base of code, and therefore harder-to-change software accumulates in a version of “software sclerosis” that compresses decades into months – a process sometimes called “going into technical debt.” That technical debt doesn’t disappear once a major rev arrives: new features don’t replace existing user-interface code, and they often depend on past infrastructure code. It’s not as bad as the old days; but technical debt is definitely a boat anchor holding back agile development.
In other words, in the typical use case, a refactoring tool is a major reason for the difference between ongoing average agile development and ongoing superior agile development. Increments are shorter, the code of higher quality, the development more scalable in app size, the testing shorter, and the code itself better able to “turn on a dime” – all at the price of spending a little extra time running a code refactoring tool on your code.
The greatest value of code refactoring, however, imho, is not what Wikipedia notes as improvements in code quality and improvements in coding speed. It is, rather, the implicit training that refactoring tools deliver in improved program design. Just as an agile development methodology improves the agility of the programmer as much as of the program, so code refactoring improves the design skills of the programmer as much as the design of the program. This designing improvement often translates into program designs that lead to apps that are simpler for the end user as well as the programmer, and more apt to deliver value-add that anticipates – rather than simply reflecting – the wishes of the end user.
The Relevance of AccuRev to Agile Development
Originally, I ran across AccuRev in connection with a fascinating white paper on best practices in agile software testing. As I probed deeper, I found that one of AccuRev’s main products is a Software Configuration Management (SCM) tool for agile development, including code refactoring tools. A Google search revealed, as I have noted, little competition from major vendors in this area. And as I thought about it, I realized that SCM is a logical place to apply code refactoring effectively.
This is because a key function of SCM is version management. In any agile development project, multi-person or not, version management is more important than in traditional programming, because the increments are very close together and often involve deployment of added-value software to end users. Establishing that an old version has reached a stopping point and a new version can begin is a logical place to remind one or many programmers to refactor code – either to make sure that the last version has been completely refactored, or to ensure that the upcoming version has refactoring designed in at the start. In a sense, it gives the agile programmer opt-out rather than opt-in code refactoring.
Moreover, as the history of “version control” over the past 25 years has demonstrated, SCM is a relatively small but very stable market. More than one programmer, and you need SCM; and the technology is mostly mature, so once you’ve used a vendor for one project, you’ll use that vendor for the next. PVCS and ClearCase, under different management, continue to be at least as widely used as they were in the early 1990s. To put it another way: AccuRev is very unlikely to collapse tomorrow – no way.
Thus, AccuRev is right in the “sweet spot” of use of code refactoring to give anyone’s agile development a major boost, is more visible than most in supporting code refactoring, and should remain useful over the next few years, both for agile SCM and for code refactoring.
Potential Uses of AccuRev-Type Code Refactoring for IT
If an IT organization has nothing going on in agile development right now, then it probably has major problems that require it to start to tackle agile development before it can even begin to think of code refactoring. Otherwise, it is hard to think of an agile-development situation in which code refactoring in general, and code refactoring via AccuRev-type SCM in particular, will not improve things. Still, there are use cases and use cases.
If you are looking for the biggest initial bang for the buck, agile BI springs to mind immediately – because, right now, it is “top of the mind” for most CEOs and CFOs. Some of that is hype, but enough of it is substance that IT should anticipate doing agile BI app development for at least the next 2-3 years while the fad lasts, and almost certainly beyond that, when it will become an integral part of another fad like “Extreme Analytics” (no, I’m not talking about anything real here – I devoutly hope). Because of this corporate commitment, the amount of technical debt we are going to see embedded in end users’ favorite analytics apps is likely to dwarf anything agile development has seen up to now, and especially because many SMBs are trying to do agile BI on the cheap – unless code refactoring steps in.
A second “bang for the buck” for code refactoring, if IT needs one, is in many-person-team (and especially distributed) agile-development projects. By spreading code refactoring across a team, IT puts them on the same design page in crucial respects, encouraging a more consistent application look and feel. Obviously, AccuRev-type SCM is a good way to introduce code refactoring with minimal disruption – if developers accept any management, it’s version control.
The Bottom Line for IT Buyers
I have been writing about code refactoring for a decade, and it still seems to be the low-hanging fruit that even agile development organizations rarely pluck. Well, let’s try again: code refactoring, like agile development, seems like it wastes time, costs money, and lowers revenues by delaying delivery, and it always does the opposite, over the short and long run. Failure to use it doesn’t spell disaster for the agile development organization; using it spells much greater success, in customer satisfaction, value-delivery speed, and app quality, that in turn (to a lesser degree than agile development itself) significantly affects the agile-development-using enterprise’s top and bottom lines. Buy it. Use it.
There are a fair amount of choices once IT buyers decide to acquire code refactoring capabilities. In my opinion, AccuRev’s approach makes it one of the most attractive vendors in the immediate future. If you want to do a short list, fine. If you don’t, and you don’t have any other ideas, skip the short list and get AccuRev. In any case, don’t waste time; just buy it. Then use it.
Maybe someday soon code refactoring will be fully automated as part of a standardized agile development toolkit, requiring no effort from the programmer, and no need to go out and buy the stuff. Maybe; but since today, after twelve years of availability, we are still nowhere near that day, I wouldn’t hold up improving your agile development because you think it might happen next year. If I were an IT buyer, I wouldn’t waste two seconds on hoping that technology advances will swoop down and save the day; I’d be taking the first steps to acquire a code refactoring solution such as the one included in AccuRev SCM. Just buy something like that. Just use it.
Friday, January 20, 2012
Thursday, January 19, 2012
Methane Update: Less Worried, Still Very Worried
I just saw an interview by Skeptical Science with Ms. Sharapova, a Russian scientist, on the Russian findings that sparked my recent methane worries. Her responses clarified the answers to the main questions I had about Arctic sea methane clathrates. Her key information, imho, was the following:
1. The Russians recently discovered that methane clathrates could form not just in the 200-1000 m depth range, but also in the 20-200 m range.
2. They also found that clathrates there did not just melt from the top down; they also melted in pockets below the surface melt.
3. The 2011 survey, for the first time, looked at the 20-200 m Siberian coastal shelf, rather than the 200-2000 m deeper waters.
Let’s look at the implications for my methane analysis. In the first place, this explains why methane was able to bubble to the surface, instead of popping or being eaten by methane-munching bacteria: it was too close to the surface, and especially if, as appears to have been the case, it was being released in larger chunks/bubbles.
In the second place, this appears to indicate that the ramp-up in methane emissions at any particular point is less than I feared. There are several possible reasons to anticipate that at greater depths, methane release from the sediment would ramp up more slowly than a 100-fold increase in one year. Likewise, there are several possible reasons to anticipate that initial methane releases from the shallow continental shelves would be greater than that from deeper areas, if there were methane clathrates there in the first place.
However, in the third place, this newly discovered source of methane clathrates appears to be a much bigger source of emissions, both in terms of melting more rapidly and of having more methane stored to begin with. Because sea shelves slope more rapidly the deeper they get (to a point beyond 1000 m), the sea-surface area of 20-200 m deep shelves is comparable to the sea-surface area of 200-1000 m ones.
Under the surface, the methane clathrates can be stored much deeper before earth heating and pressure melt them. Take these two things together, and the amount of methane in Arctic clathrates may be 2-4 times the amount previously estimated. Meanwhile, this 20-200 m range lies almost entirely in the “shallow ocean” range where warming currents from the south plus warming of newly exposed surface waters by the summer sun create hotter water next to the sediment – and thus melt things faster. These points are confirmed by the observations of rapid bubble generation and much larger funnels from which methane flows in the 20-200 m range.
In the fourth place, the ability of 20-200 m methane bubbles to rise to the surface means that we probably grossly underestimate the percentage of emitted methane that will rise into the atmosphere as methane rather than carbon dioxide. Frankly, this is probably good news, since that means less of it will eventually stay in the atmosphere as carbon dioxide – but there’s still a chance it may stay as methane for a long time – and be far worse for global warming. This would happen if there’s too much methane up there and the OH in the atmosphere that removes a lot of that methane runs out, a possibility some scientists have raised.
In the fifth place, the existence of pockets indicates that methane emissions may be bursty, as surface melt “burns through” to those pockets that are themselves melted, but are still trapped by frozen clathrates above. Those bursts should be frequent enough to keep methane emissions at a higher yearly rate.
Implications
Overall, this makes me a little more hopeful about overall methane emissions and their effect on global warming. While there is perhaps 2-4 times the amount of methane clathrates to emit than I thought, there may be 30-70% lower emission rates than I anticipated, and that’s the key to methane’s overall effect in the next 160 years, when it matters. To put it another way, the net methane emission rates per year should be lower than I expected, and the amount in the atmosphere as methane in the next 160 years should be lower than my worst-case all-methane scenario (assuming there’s enough OH). Hopefully, the amount of carbon dioxide should be lower as well, because of the decreased yearly emissions amount and increased percentage arriving as methane (only half of that turns into carbon dioxide). However, this isn’t sure, because if the OH runs out, the effect of yet more “steady state” methane over, say, 600 years on global warming will be worse than I had anticipated.
All in all, I would now tend to put the likely overall new natural-source methane emission effects (also including permafrost and wetlands) in the 3-6 degree C range over the next over the next 200 years, and in the 2-5 degree Celsius range in the 400 years after that – overall, perhaps a 25% boost to global warming rather than a 50% one, protracted over more years. High water may not be delayed, but hell may be a little less hellish in temperature, and the end of life on earth ever so slightly less likely, than I feared.
Unless, of course, the OH in the atmosphere runs out … the worries never end, do they?
1. The Russians recently discovered that methane clathrates could form not just in the 200-1000 m depth range, but also in the 20-200 m range.
2. They also found that clathrates there did not just melt from the top down; they also melted in pockets below the surface melt.
3. The 2011 survey, for the first time, looked at the 20-200 m Siberian coastal shelf, rather than the 200-2000 m deeper waters.
Let’s look at the implications for my methane analysis. In the first place, this explains why methane was able to bubble to the surface, instead of popping or being eaten by methane-munching bacteria: it was too close to the surface, and especially if, as appears to have been the case, it was being released in larger chunks/bubbles.
In the second place, this appears to indicate that the ramp-up in methane emissions at any particular point is less than I feared. There are several possible reasons to anticipate that at greater depths, methane release from the sediment would ramp up more slowly than a 100-fold increase in one year. Likewise, there are several possible reasons to anticipate that initial methane releases from the shallow continental shelves would be greater than that from deeper areas, if there were methane clathrates there in the first place.
However, in the third place, this newly discovered source of methane clathrates appears to be a much bigger source of emissions, both in terms of melting more rapidly and of having more methane stored to begin with. Because sea shelves slope more rapidly the deeper they get (to a point beyond 1000 m), the sea-surface area of 20-200 m deep shelves is comparable to the sea-surface area of 200-1000 m ones.
Under the surface, the methane clathrates can be stored much deeper before earth heating and pressure melt them. Take these two things together, and the amount of methane in Arctic clathrates may be 2-4 times the amount previously estimated. Meanwhile, this 20-200 m range lies almost entirely in the “shallow ocean” range where warming currents from the south plus warming of newly exposed surface waters by the summer sun create hotter water next to the sediment – and thus melt things faster. These points are confirmed by the observations of rapid bubble generation and much larger funnels from which methane flows in the 20-200 m range.
In the fourth place, the ability of 20-200 m methane bubbles to rise to the surface means that we probably grossly underestimate the percentage of emitted methane that will rise into the atmosphere as methane rather than carbon dioxide. Frankly, this is probably good news, since that means less of it will eventually stay in the atmosphere as carbon dioxide – but there’s still a chance it may stay as methane for a long time – and be far worse for global warming. This would happen if there’s too much methane up there and the OH in the atmosphere that removes a lot of that methane runs out, a possibility some scientists have raised.
In the fifth place, the existence of pockets indicates that methane emissions may be bursty, as surface melt “burns through” to those pockets that are themselves melted, but are still trapped by frozen clathrates above. Those bursts should be frequent enough to keep methane emissions at a higher yearly rate.
Implications
Overall, this makes me a little more hopeful about overall methane emissions and their effect on global warming. While there is perhaps 2-4 times the amount of methane clathrates to emit than I thought, there may be 30-70% lower emission rates than I anticipated, and that’s the key to methane’s overall effect in the next 160 years, when it matters. To put it another way, the net methane emission rates per year should be lower than I expected, and the amount in the atmosphere as methane in the next 160 years should be lower than my worst-case all-methane scenario (assuming there’s enough OH). Hopefully, the amount of carbon dioxide should be lower as well, because of the decreased yearly emissions amount and increased percentage arriving as methane (only half of that turns into carbon dioxide). However, this isn’t sure, because if the OH runs out, the effect of yet more “steady state” methane over, say, 600 years on global warming will be worse than I had anticipated.
All in all, I would now tend to put the likely overall new natural-source methane emission effects (also including permafrost and wetlands) in the 3-6 degree C range over the next over the next 200 years, and in the 2-5 degree Celsius range in the 400 years after that – overall, perhaps a 25% boost to global warming rather than a 50% one, protracted over more years. High water may not be delayed, but hell may be a little less hellish in temperature, and the end of life on earth ever so slightly less likely, than I feared.
Unless, of course, the OH in the atmosphere runs out … the worries never end, do they?
Saturday, January 14, 2012
A Few Books That "Rocked My World"
I recently saw a list of “100 books that rocked my world” from a blogger. It turned out to be a list of “books that are really cool” and not “books that made me think differently in fundamental ways.” So I thought I’d look back and ask myself, years later, what books changed me fundamentally?
1. Godel’s Proof, Nagel and Freeman. As Shaw says in “Man and Superman”, it made me want to “think more, so that I would be more.” The idea of there being some things that I will never know or prove, is something that I am still wrestling with.
2. Spark, Ratey. The idea that we fluctuate chemically between addiction to pessimism and addiction to optimism based on whether we are channeling our hunter ancestors seeking prey by exercising in company, between learning by moderate physical stress and forgetting based on sedentary habits, between inoculating against disorders by moderately poisoning ourselves with food and moderately stressing ourselves with exercise and opening ourselves to disease and death by eating unchallenging foods and avoiding challenging exercise, seems to apply to and alter every aspect of my life.
3. The Age of Diminished Expectations, Krugman. It began to give me an ability not only to understand my sense of disconnect between conventional economics and what was happening to me in the real world, but to apply new tools to understand and improve the broad scope of my life in terms of money and generational cycles – something I’d been looking for over 20 years of college and work. Of course, I needed several additional books and articles to understand the full scope of Krugman’s approach.
4. The Fellowship of the Ring, Tolkien. I wasn’t expecting what I got when I scrounged my Dad’s library for yet more books at age 12. Suddenly, I was able to see the non-human world around me as a separate, interconnected, wonderful, alive thing. And the idea that you could frame a book or part of a life as the necessary preface leading to the beginning of a journey – like Frodo’s, stripped of his teachers, into Mordor – made me see that my life could be seen that way – and that has always given me hope.
5. Falconer, Cheever. This one is painful. I had to ask myself, after it was over, am I, like the protagonist, seeing my relationships with women too much in terms of my own needs, or can I finally grow up? I don’t know how much I’ve changed in my behavior after reading it; but I know I can never think the way I used to about relationships without far greater discomfort and dissatisfaction with myself.
6. How to Win Friends and Influence People, Dale Carnegie. There are many, many things wrong with this book, as I have come to realize. But it gave me the humility of understanding that my artsy and intellectual achievements were of very little value to others, and showed me that I really liked people, if I just listened to them. It also gave me a basis for understanding people’s social viewpoint that has allowed me to connect, slowly, over 30 years.
7. A Connecticut Yankee at King Arthur’s Court, Twain. I don’t think I recognized this at the time, but it was my introduction to what I might call the “science fiction viewpoint”: the idea that by facing scientific facts and leveraging technology, you could make a fundamental change for good in the world – the true meaning of “progress”. I can never quite shake that idea, and it has led me in quite a different direction from the rest of my family, into math and computers, and away from Great Literature that had few solutions to offer. Of course, I never quite accepted Twain’s other idea: that this progress could vanish like the mist from history, resisted by willfully ignorant humans frightened of change, unless you were lucky.
1. Godel’s Proof, Nagel and Freeman. As Shaw says in “Man and Superman”, it made me want to “think more, so that I would be more.” The idea of there being some things that I will never know or prove, is something that I am still wrestling with.
2. Spark, Ratey. The idea that we fluctuate chemically between addiction to pessimism and addiction to optimism based on whether we are channeling our hunter ancestors seeking prey by exercising in company, between learning by moderate physical stress and forgetting based on sedentary habits, between inoculating against disorders by moderately poisoning ourselves with food and moderately stressing ourselves with exercise and opening ourselves to disease and death by eating unchallenging foods and avoiding challenging exercise, seems to apply to and alter every aspect of my life.
3. The Age of Diminished Expectations, Krugman. It began to give me an ability not only to understand my sense of disconnect between conventional economics and what was happening to me in the real world, but to apply new tools to understand and improve the broad scope of my life in terms of money and generational cycles – something I’d been looking for over 20 years of college and work. Of course, I needed several additional books and articles to understand the full scope of Krugman’s approach.
4. The Fellowship of the Ring, Tolkien. I wasn’t expecting what I got when I scrounged my Dad’s library for yet more books at age 12. Suddenly, I was able to see the non-human world around me as a separate, interconnected, wonderful, alive thing. And the idea that you could frame a book or part of a life as the necessary preface leading to the beginning of a journey – like Frodo’s, stripped of his teachers, into Mordor – made me see that my life could be seen that way – and that has always given me hope.
5. Falconer, Cheever. This one is painful. I had to ask myself, after it was over, am I, like the protagonist, seeing my relationships with women too much in terms of my own needs, or can I finally grow up? I don’t know how much I’ve changed in my behavior after reading it; but I know I can never think the way I used to about relationships without far greater discomfort and dissatisfaction with myself.
6. How to Win Friends and Influence People, Dale Carnegie. There are many, many things wrong with this book, as I have come to realize. But it gave me the humility of understanding that my artsy and intellectual achievements were of very little value to others, and showed me that I really liked people, if I just listened to them. It also gave me a basis for understanding people’s social viewpoint that has allowed me to connect, slowly, over 30 years.
7. A Connecticut Yankee at King Arthur’s Court, Twain. I don’t think I recognized this at the time, but it was my introduction to what I might call the “science fiction viewpoint”: the idea that by facing scientific facts and leveraging technology, you could make a fundamental change for good in the world – the true meaning of “progress”. I can never quite shake that idea, and it has led me in quite a different direction from the rest of my family, into math and computers, and away from Great Literature that had few solutions to offer. Of course, I never quite accepted Twain’s other idea: that this progress could vanish like the mist from history, resisted by willfully ignorant humans frightened of change, unless you were lucky.
Friday, January 13, 2012
The Other BI: HP Vertica and Columnar Databases
This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about BI today.
The Importance of Vertica-Type Columnar Database Technology to BI
Last year, I wrote a blog post saying that it was likely that HP would underestimate the columnar database technology in Vertica, and if so they were missing a major opportunity. In the last year, HP has been pretty quiet about Vertica, but I have partially changed my mind, to the point where I want to call attention to Vertica as a less visible candidate for IT buyers to get the full benefits of columnar database technology over the next 2-3 years.
Let’s start with columnar technology. Here, I want to go more in-depth into Vertica’s core technology than usual, because it’s an excellent way to begin to see the benefits of columnar beyond traditional row-oriented databases.
The original idea of Vertica was to recast the relational database to focus on the (data warehousing) case where there are few if any updates. The redesign started with the idea that the data should be stored in "columns" rather than rows; the details of this are that the columns themselves (because they don't have to follow relational dogma) can be stored in a highly compressed format, with lots of compression techniques like inverted list, bit-mapped indexing, and hashing, as appropriate. Thus, (a) the database can use the column format to zero in faster on the data that the query is gathering, (b) because the data is compressed an average of 10 times (according to Vertica), more data can be crammed into main memory for faster processing. Result: a claimed 10-100 times speedup in performance, comparable to in-memory databases but far more scalable. It also means the database can handle at least 10 times more data (say, 100 terabytes instead of 5) with the same performance for a given query; or that the data center can use an order of magnitude less storage.
Now, all this does not come without a cost, and the typical cost would at first seem to be speed of updating. That is, the column storage format requires more revision of the data stored on disk when an update arrives, so update is slower. But this is counteracted by the ability to load more of localized data at once into main memory in a compressed form, for faster in-memory updating. Only at update frequencies typical of old-style operational online transaction processing (OLTP) does the row-oriented relational database have a clear edge.
The elaboration of the design in Vertica is that the basic data is also stored as "projections" (aka materialized views). That is, a set of columns in a tuple is stored one (relational) way; each column also shows up in a projection, but the projection is cross-tuple (one from tuple A, one from tuple B, etc.). This accomplishes two things: one, it gives an alternative way of querying which may be faster than basic storage, and two, it gives redundancy and therefore robustness, in a similar way to RAID 5 (projections can be "striped" across disks).
Now, here's where things get really interesting. Practically speaking, today, in data-warehousing-type databases, updates via "load windows" are becoming more and more frequent, to the point where data is pretty up-to-date and updates are a bigger part of data warehousing. To keep "write locks" from gumming up performance (especially with column update being slower), Vertica splits the storage into a write-optimized column store (WOS; effectively, a cache) and a Read-optimized Column Store (ROS). Periodically, the WOS becomes the ROS. So the write locks for the updates only interfere with reads when there’s a mass update. At the same time, such a mass update can re-store whole chunks of the ROS for optimum storage efficiency. Moreover, to gain currency, the query can be carried out across the ROS and WOS. And, because there is all this redundancy, there is no need for logs—another performance improvement. Note that because of its redundancy, Vertica doesn't need to do roll-back/roll-forward nor backup/restore.
The net of all this for IT buyers is that columnar databases in general, and Vertica in particular, should be able to deliver on average much better performance than traditional relational databases in the majority of not-highly-update-intensive cases, due mostly to its compression abilities, and that addition of other technologies like in-memory technology to both alternatives will not alter this superiority.
The Relevance of HP Vertica to BI
This kind of approach cries out for integration with or development of sophisticated admin tools, expansion beyond data warehousing and analytics to “mixed” transactions in competition with the noSQL fad, better programming tools to build up a war chest of business/industry customized solutions, and using a relational database as an OLTP complement. The resulting data-management platform would be a solid alternative for all sizes of enterprise to the “relational fits all” or “let the thousand flowers bloom” strategies of most organizations.
Once this platform is in place, it needs to become the keystone of enterprise architectures, not just an analytics or business intelligence “super-scaling” engine. That means adding integration with semi-structured and unstructured data. It also means adding major functionality for handling content, and integration with storage software for additional performance optimization. And so, anticipating that HP would not do this, I criticized the HP acquisition of Vertica last year.
Well, two things happened: HP did more than I thought it would, and competitors did less. HP bought a company called Autonomy, which added semi-structured/unstructured data support. Necessarily, this takes Vertica beyond pure data-warehousing-style analytics into a more update-intensive world, and HP’s redirection of Mercury Interactive towards agile ALM (application lifecycle management) associated Vertica with better programming tools. Meanwhile, SAP took its eye off Sybase IQ with its focus on HANA, IBM at least temporarily walked away from its Netezza semi-columnar database technology, and Oracle’s columnar-optional appliance ran into questions about its long-term hardware growth path. In other words, the result of half a loaf from HP and less than half a loaf from everyone else is that Vertica is moving towards leadership status in delivering columnar database technology to all scales of BI and analytics.
Meanwhile, of course, only the deluded think that HP will suddenly vanish, while database technology and the rest of the new software embed themselves ever deeper in HP’s DNA. HP Vertica is going to be around for quite a while; and it will be an attractive option for quite a while.
Potential Uses of Vertica-Type Columnar-Based BI for IT
The use cases of a columnar database IT is straightforward. IT should use a columnar database in new projects as an alternative or complement to a traditional relational database, unless the operations are update-intensive, in which case row-oriented relational is preferred. As a complement, columnar databases operate on a “switching” basis, in which an overall engine decides which queries should be allocated to row-oriented, which to columnar, usually on the basis of whether two or more of the “fields” involved in an operation can be compressed highly by using a columnar format. Oracle (and, until recently, IBM Netezza) takes this approach; but IT can also do its own switching mechanism.
And that’s it. Over the next 2-3 years, if not already, columnar can scale as high as querying, can integrate with as many data types and upper-level tools and applications, and can evolve to greater performance/scalability just as rapidly as the traditional row-oriented database. In the long run, in a lot of use cases, and sometimes in the short run, that favors Vertica-type columnar.
However, right now, columnar requires in some cases to “grow into” its assigned role in a new project, by adding administrative tools for particular cases. Therefore, in most applications where 24x7 operation and an adequate level of customer response time is business-critical, relational row-oriented should still be preferred. That should leave plenty of analytical and other BI uses for which Vertica-type columnar database software will deliver an important performance advantage.
The Bottom Line for IT Buyers
Over the next few years, IT buyers can take one of two views: the author of this blog post is prescient, columnar will replace row-oriented in the majority of new applications in BI and other areas, and we should include columnar in all our short lists from now on; or, the author of this blog post is wrong about the future, but columnar is useful for some things right now, and trying to standardize on one database is a fool’s game that we no longer bother to try to play. If IT buyers hold the second view, then they should be focused on applying columnar to analysis of huge amounts of structured data with “sparse” fields where high compression is achievable – like five-field customer names (Mr. John Taylor Jakes, Jr.) and product codes. Spend the resulting improvements on increased performance, lowered storage costs, or both.
Again, this is not a matter of a pre-short list, unless you have a “gray area” BI project involving somewhat update-intensive or somewhat business-critical little-downtime apps, in which case you want to wait for columnar to evolve a little. In all other cases, HP Vertica should go on the short list along with the obvious others, like Sybase IQ. Right now, Vertica appears to be ahead both in some of the needed features to adapt to new analytics needs and in speed of evolution. One never knows – but over the next year, that leadership role may continue.
Above all, IT buyers should not listen to any FUD from traditional relational vendors suggesting that this is yet another new technology, like object databases, that will eventually fall to earth with a thud. Columnar database technology proved its superiority in many situations long ago in the non-relational world, with CCA’s Model 204, and has found uses continuously since then, like bit-mapped indexing. Most times there’s a fair BI matchup, as with some of the TPC benchmarks of the last seven years, columnar comes out well ahead. Under whatever name, columnar database technology is not going away. Therefore, its markets will continue to grow relative to row-oriented relational. For IT buyers, acquiring columnar BI solutions like HP’s Vertica is simply being smart and getting a little ahead of the curve.
The Importance of Vertica-Type Columnar Database Technology to BI
Last year, I wrote a blog post saying that it was likely that HP would underestimate the columnar database technology in Vertica, and if so they were missing a major opportunity. In the last year, HP has been pretty quiet about Vertica, but I have partially changed my mind, to the point where I want to call attention to Vertica as a less visible candidate for IT buyers to get the full benefits of columnar database technology over the next 2-3 years.
Let’s start with columnar technology. Here, I want to go more in-depth into Vertica’s core technology than usual, because it’s an excellent way to begin to see the benefits of columnar beyond traditional row-oriented databases.
The original idea of Vertica was to recast the relational database to focus on the (data warehousing) case where there are few if any updates. The redesign started with the idea that the data should be stored in "columns" rather than rows; the details of this are that the columns themselves (because they don't have to follow relational dogma) can be stored in a highly compressed format, with lots of compression techniques like inverted list, bit-mapped indexing, and hashing, as appropriate. Thus, (a) the database can use the column format to zero in faster on the data that the query is gathering, (b) because the data is compressed an average of 10 times (according to Vertica), more data can be crammed into main memory for faster processing. Result: a claimed 10-100 times speedup in performance, comparable to in-memory databases but far more scalable. It also means the database can handle at least 10 times more data (say, 100 terabytes instead of 5) with the same performance for a given query; or that the data center can use an order of magnitude less storage.
Now, all this does not come without a cost, and the typical cost would at first seem to be speed of updating. That is, the column storage format requires more revision of the data stored on disk when an update arrives, so update is slower. But this is counteracted by the ability to load more of localized data at once into main memory in a compressed form, for faster in-memory updating. Only at update frequencies typical of old-style operational online transaction processing (OLTP) does the row-oriented relational database have a clear edge.
The elaboration of the design in Vertica is that the basic data is also stored as "projections" (aka materialized views). That is, a set of columns in a tuple is stored one (relational) way; each column also shows up in a projection, but the projection is cross-tuple (one from tuple A, one from tuple B, etc.). This accomplishes two things: one, it gives an alternative way of querying which may be faster than basic storage, and two, it gives redundancy and therefore robustness, in a similar way to RAID 5 (projections can be "striped" across disks).
Now, here's where things get really interesting. Practically speaking, today, in data-warehousing-type databases, updates via "load windows" are becoming more and more frequent, to the point where data is pretty up-to-date and updates are a bigger part of data warehousing. To keep "write locks" from gumming up performance (especially with column update being slower), Vertica splits the storage into a write-optimized column store (WOS; effectively, a cache) and a Read-optimized Column Store (ROS). Periodically, the WOS becomes the ROS. So the write locks for the updates only interfere with reads when there’s a mass update. At the same time, such a mass update can re-store whole chunks of the ROS for optimum storage efficiency. Moreover, to gain currency, the query can be carried out across the ROS and WOS. And, because there is all this redundancy, there is no need for logs—another performance improvement. Note that because of its redundancy, Vertica doesn't need to do roll-back/roll-forward nor backup/restore.
The net of all this for IT buyers is that columnar databases in general, and Vertica in particular, should be able to deliver on average much better performance than traditional relational databases in the majority of not-highly-update-intensive cases, due mostly to its compression abilities, and that addition of other technologies like in-memory technology to both alternatives will not alter this superiority.
The Relevance of HP Vertica to BI
This kind of approach cries out for integration with or development of sophisticated admin tools, expansion beyond data warehousing and analytics to “mixed” transactions in competition with the noSQL fad, better programming tools to build up a war chest of business/industry customized solutions, and using a relational database as an OLTP complement. The resulting data-management platform would be a solid alternative for all sizes of enterprise to the “relational fits all” or “let the thousand flowers bloom” strategies of most organizations.
Once this platform is in place, it needs to become the keystone of enterprise architectures, not just an analytics or business intelligence “super-scaling” engine. That means adding integration with semi-structured and unstructured data. It also means adding major functionality for handling content, and integration with storage software for additional performance optimization. And so, anticipating that HP would not do this, I criticized the HP acquisition of Vertica last year.
Well, two things happened: HP did more than I thought it would, and competitors did less. HP bought a company called Autonomy, which added semi-structured/unstructured data support. Necessarily, this takes Vertica beyond pure data-warehousing-style analytics into a more update-intensive world, and HP’s redirection of Mercury Interactive towards agile ALM (application lifecycle management) associated Vertica with better programming tools. Meanwhile, SAP took its eye off Sybase IQ with its focus on HANA, IBM at least temporarily walked away from its Netezza semi-columnar database technology, and Oracle’s columnar-optional appliance ran into questions about its long-term hardware growth path. In other words, the result of half a loaf from HP and less than half a loaf from everyone else is that Vertica is moving towards leadership status in delivering columnar database technology to all scales of BI and analytics.
Meanwhile, of course, only the deluded think that HP will suddenly vanish, while database technology and the rest of the new software embed themselves ever deeper in HP’s DNA. HP Vertica is going to be around for quite a while; and it will be an attractive option for quite a while.
Potential Uses of Vertica-Type Columnar-Based BI for IT
The use cases of a columnar database IT is straightforward. IT should use a columnar database in new projects as an alternative or complement to a traditional relational database, unless the operations are update-intensive, in which case row-oriented relational is preferred. As a complement, columnar databases operate on a “switching” basis, in which an overall engine decides which queries should be allocated to row-oriented, which to columnar, usually on the basis of whether two or more of the “fields” involved in an operation can be compressed highly by using a columnar format. Oracle (and, until recently, IBM Netezza) takes this approach; but IT can also do its own switching mechanism.
And that’s it. Over the next 2-3 years, if not already, columnar can scale as high as querying, can integrate with as many data types and upper-level tools and applications, and can evolve to greater performance/scalability just as rapidly as the traditional row-oriented database. In the long run, in a lot of use cases, and sometimes in the short run, that favors Vertica-type columnar.
However, right now, columnar requires in some cases to “grow into” its assigned role in a new project, by adding administrative tools for particular cases. Therefore, in most applications where 24x7 operation and an adequate level of customer response time is business-critical, relational row-oriented should still be preferred. That should leave plenty of analytical and other BI uses for which Vertica-type columnar database software will deliver an important performance advantage.
The Bottom Line for IT Buyers
Over the next few years, IT buyers can take one of two views: the author of this blog post is prescient, columnar will replace row-oriented in the majority of new applications in BI and other areas, and we should include columnar in all our short lists from now on; or, the author of this blog post is wrong about the future, but columnar is useful for some things right now, and trying to standardize on one database is a fool’s game that we no longer bother to try to play. If IT buyers hold the second view, then they should be focused on applying columnar to analysis of huge amounts of structured data with “sparse” fields where high compression is achievable – like five-field customer names (Mr. John Taylor Jakes, Jr.) and product codes. Spend the resulting improvements on increased performance, lowered storage costs, or both.
Again, this is not a matter of a pre-short list, unless you have a “gray area” BI project involving somewhat update-intensive or somewhat business-critical little-downtime apps, in which case you want to wait for columnar to evolve a little. In all other cases, HP Vertica should go on the short list along with the obvious others, like Sybase IQ. Right now, Vertica appears to be ahead both in some of the needed features to adapt to new analytics needs and in speed of evolution. One never knows – but over the next year, that leadership role may continue.
Above all, IT buyers should not listen to any FUD from traditional relational vendors suggesting that this is yet another new technology, like object databases, that will eventually fall to earth with a thud. Columnar database technology proved its superiority in many situations long ago in the non-relational world, with CCA’s Model 204, and has found uses continuously since then, like bit-mapped indexing. Most times there’s a fair BI matchup, as with some of the TPC benchmarks of the last seven years, columnar comes out well ahead. Under whatever name, columnar database technology is not going away. Therefore, its markets will continue to grow relative to row-oriented relational. For IT buyers, acquiring columnar BI solutions like HP’s Vertica is simply being smart and getting a little ahead of the curve.
Thursday, January 12, 2012
The Other Agile Development: CollabNet and Collaborative Agile Development
This blog post highlights a software company and technology that I view as potentially useful to organizations investing in agile development, agile new product development, and business agility over the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about agile development today.
The Importance of Collaborative Software Technology to Agile Development
Today, I received a presentation about IBM’s collaborative technology under the heading of Lotus that emphasized its positive effects on business agility. And yet, there was no mention of increased support for agile development tools and methodologies. Instead, the focus was on incorporating “social business” and “mobile” in collaborative technology. This approach, all too common among vendors of collaborative software solutions ranging from email to videoconferencing to Facebook-type information sharing, has already proven itself almost entirely ineffective in improving development or business agility, and sometimes has caused organizations to reinforce instead of moving away from un-agile practices.
A careful study of software development shows that only three software development approaches have had any positive impact at all recently on development TCO, ROI, percent completed on time/within budget, and customer satisfaction: agile development, collaborative development, and “hybrid” open-source/traditional development.
However, when we isolate the three, agile alone is clearly superior to hybrid alone, and far superior to collaborative alone. Add agile back to the mix, and the result is only slightly better than agile alone. The conclusion is clear: the main benefits from either collaborative or hybrid alone come from “unconscious agility”, such as the ability of hybrid to incorporate unexpected but valuable additions in the middle of the development process, or the ability of collaboration to improve a slightly agile development process such as a spiral methodology.
These results, however, do not consider the case where collaboration software is deliberately designed to focus on supporting agile development. In those cases, collaboration software’s support for distributed and multi-person development where necessary should remove much of the “drag” on agile development that such situations create, by wedding the maximum of “methodology fit”, loosely-coupled flexibility, and automation to the necessary coordination techniques like versioning and application lifecycle management. Beyond that, collaboration, with its ubiquitous presence in the organization can, once targeted to agile NPD or another agile business process, deliver the same scalability and distribution benefits. In other words, the collaborative framework of an agile, scalable business process is already there; all you need to do is infuse the collaboration technology with agile “smarts” and insert the agile methodology.
And yet, up to now, those solutions are thin on the ground. Look where you will, at IBM’s Jazz plus Lotus or HP’s Mercury Interactive extended to Application Lifecycle Management (ALM), and the major vendors are not delivering collaboration software clearly targeted to agile development methodologies. However, these solutions are indeed beginning to arrive – from the likes of CollabNet.
The Relevance of CollabNet to Agile Development
The leadership role of CollabNet in collaborative agile development is no surprise, given its original leadership role in collaborative development in the late 1990s and early 2000s. However, that did not guarantee that CollabNet would recognize the synergies between collaborative and agile development when collaborative was targeted to agile. For a while, rather, CollabNet aimed partly at the offshoring market, which in some cases decreased development agility by forcing design separated temporally and physically from the development phase – not to mention the end user. This lasted until the likes of Wipro climbed on the bandwagon of agile, often ahead of IT and the major development-tool vendors.
Still, it is surprising and heartening to see how far CollabNet has come since that time. Every other word on the product section of the web site is “agile,” and CollabNet backs that up with a SCRUM-support toolset called SCRUMworks with which its other solutions such as TeamForge and the ALM toolset are specifically integrated.
That, in turn, is why I conclude that CollabNet is settled in for the long haul, and is therefore a safe bet for IT buyers. As a Web-based collaboration and then an offshoring-focused solution provider, CollabNet was more dependent on speculative markets (not to mention smaller) than it is now. Having survived the transition from one strategic focus to another, it is centered around an industry – agile development tools and services – that has enormous “legs”, as agile development not only spreads throughout organizations but also matures and elaborates its methodology in technology to achieve far more – such as the “continous delivery” that I have mentioned in a previous blog post. Yes, the majors may roll over and crowd out CollabNet by using their clout in existing business collaboration software – but that would be to suddenly deep-six their existing high-end-development toolsets, which I believe require long periods of retrofitting before they can support agile development project management adequately. No, I think that CollabNet is well set up to be among the leaders in collaborative agile development over the next 2-3 years, and probably well beyond that.
Potential Uses of CollabNet-Type Agile Development for IT
Obviously, CollabNet is best suited to organizations of all sizes right now that require distributed team development and are adhering to a SCRUM-type agile technology. However, it is also appropriate to cloud agile development (especially public cloud), because that also emphasizes distributed development and deployment for multi-tenant SaaS in globe-serving multi-location server farms. And, of course, IT with legacy development offshoring and outsourcing will find that CollabNet, due to its experience in those areas, provides a transition to agile practices.
It may seem odd to also propose CollabNet for NPD, rather than the heavyweight lean-plus-agile or innovation-centric tools that have been used up to now. However, I believe that CollabNet should be considered as an alternative/complement to these, especially for the SMB, because its DNA lies in the Web-oriented movement that emphasizes crowdsourcing and global-consumer attention rather than in the existing “lean-focused” NPD tools that emphasize information flow within the business. For SMBs, especially those for whom software is increasingly a product differentiator, something like CollabNet offers an opportunity yet again to outmaneuver the clumsier large enterprises. This is particularly true when the SMB is already leading-edge in agile NPD.
Of course, CollabNet cannot incorporate existing and upcoming social-media input quite as well as a Lotus or perhaps a Google. That’s when the IT buyer should complement CollabNet with features from the appropriate social-media-rich-collaboration-software-plus-development-tool vendor. But combining the two, rather than tossing CollabNet, is probably the way to go.
The Bottom Line for IT Buyers
The bottom line is short and sweet: Over the next 2-3 years, IT buyers should not seek to implement collaboration and hope for increased agility, but rather buy collaboration tools focused on agile development, and be assured of bottom-line results even beyond those of agile development itself. Don’t bother with a pre-short list; get a short list, put CollabNet on it, make your choice, and get going. Hopefully, we’re past the time when anything about agile development had to prove itself by traditional cost and quality metrics – which never saw the cost/revenue/margin/customer-satisfaction forest for the “you’re wasting time and money right now” trees. The benefits are already proven. Get on with it, already.
Of course, you never know which smaller companies will fold, or be acquired and lose their technology edge as they scramble to integrate with less-agile large-vendor solutions. I never say never, but CollabNet appears to be one of the least likely to fold – and the larger vendor may also provide additional useful collaboration software and services.
If you’re still reading this, you’re not agile enough. Go look at CollabNet-type agile-aimed collaborative development solutions. Make CollabNet one of the solutions to be inspected. Go.
The Importance of Collaborative Software Technology to Agile Development
Today, I received a presentation about IBM’s collaborative technology under the heading of Lotus that emphasized its positive effects on business agility. And yet, there was no mention of increased support for agile development tools and methodologies. Instead, the focus was on incorporating “social business” and “mobile” in collaborative technology. This approach, all too common among vendors of collaborative software solutions ranging from email to videoconferencing to Facebook-type information sharing, has already proven itself almost entirely ineffective in improving development or business agility, and sometimes has caused organizations to reinforce instead of moving away from un-agile practices.
A careful study of software development shows that only three software development approaches have had any positive impact at all recently on development TCO, ROI, percent completed on time/within budget, and customer satisfaction: agile development, collaborative development, and “hybrid” open-source/traditional development.
However, when we isolate the three, agile alone is clearly superior to hybrid alone, and far superior to collaborative alone. Add agile back to the mix, and the result is only slightly better than agile alone. The conclusion is clear: the main benefits from either collaborative or hybrid alone come from “unconscious agility”, such as the ability of hybrid to incorporate unexpected but valuable additions in the middle of the development process, or the ability of collaboration to improve a slightly agile development process such as a spiral methodology.
These results, however, do not consider the case where collaboration software is deliberately designed to focus on supporting agile development. In those cases, collaboration software’s support for distributed and multi-person development where necessary should remove much of the “drag” on agile development that such situations create, by wedding the maximum of “methodology fit”, loosely-coupled flexibility, and automation to the necessary coordination techniques like versioning and application lifecycle management. Beyond that, collaboration, with its ubiquitous presence in the organization can, once targeted to agile NPD or another agile business process, deliver the same scalability and distribution benefits. In other words, the collaborative framework of an agile, scalable business process is already there; all you need to do is infuse the collaboration technology with agile “smarts” and insert the agile methodology.
And yet, up to now, those solutions are thin on the ground. Look where you will, at IBM’s Jazz plus Lotus or HP’s Mercury Interactive extended to Application Lifecycle Management (ALM), and the major vendors are not delivering collaboration software clearly targeted to agile development methodologies. However, these solutions are indeed beginning to arrive – from the likes of CollabNet.
The Relevance of CollabNet to Agile Development
The leadership role of CollabNet in collaborative agile development is no surprise, given its original leadership role in collaborative development in the late 1990s and early 2000s. However, that did not guarantee that CollabNet would recognize the synergies between collaborative and agile development when collaborative was targeted to agile. For a while, rather, CollabNet aimed partly at the offshoring market, which in some cases decreased development agility by forcing design separated temporally and physically from the development phase – not to mention the end user. This lasted until the likes of Wipro climbed on the bandwagon of agile, often ahead of IT and the major development-tool vendors.
Still, it is surprising and heartening to see how far CollabNet has come since that time. Every other word on the product section of the web site is “agile,” and CollabNet backs that up with a SCRUM-support toolset called SCRUMworks with which its other solutions such as TeamForge and the ALM toolset are specifically integrated.
That, in turn, is why I conclude that CollabNet is settled in for the long haul, and is therefore a safe bet for IT buyers. As a Web-based collaboration and then an offshoring-focused solution provider, CollabNet was more dependent on speculative markets (not to mention smaller) than it is now. Having survived the transition from one strategic focus to another, it is centered around an industry – agile development tools and services – that has enormous “legs”, as agile development not only spreads throughout organizations but also matures and elaborates its methodology in technology to achieve far more – such as the “continous delivery” that I have mentioned in a previous blog post. Yes, the majors may roll over and crowd out CollabNet by using their clout in existing business collaboration software – but that would be to suddenly deep-six their existing high-end-development toolsets, which I believe require long periods of retrofitting before they can support agile development project management adequately. No, I think that CollabNet is well set up to be among the leaders in collaborative agile development over the next 2-3 years, and probably well beyond that.
Potential Uses of CollabNet-Type Agile Development for IT
Obviously, CollabNet is best suited to organizations of all sizes right now that require distributed team development and are adhering to a SCRUM-type agile technology. However, it is also appropriate to cloud agile development (especially public cloud), because that also emphasizes distributed development and deployment for multi-tenant SaaS in globe-serving multi-location server farms. And, of course, IT with legacy development offshoring and outsourcing will find that CollabNet, due to its experience in those areas, provides a transition to agile practices.
It may seem odd to also propose CollabNet for NPD, rather than the heavyweight lean-plus-agile or innovation-centric tools that have been used up to now. However, I believe that CollabNet should be considered as an alternative/complement to these, especially for the SMB, because its DNA lies in the Web-oriented movement that emphasizes crowdsourcing and global-consumer attention rather than in the existing “lean-focused” NPD tools that emphasize information flow within the business. For SMBs, especially those for whom software is increasingly a product differentiator, something like CollabNet offers an opportunity yet again to outmaneuver the clumsier large enterprises. This is particularly true when the SMB is already leading-edge in agile NPD.
Of course, CollabNet cannot incorporate existing and upcoming social-media input quite as well as a Lotus or perhaps a Google. That’s when the IT buyer should complement CollabNet with features from the appropriate social-media-rich-collaboration-software-plus-development-tool vendor. But combining the two, rather than tossing CollabNet, is probably the way to go.
The Bottom Line for IT Buyers
The bottom line is short and sweet: Over the next 2-3 years, IT buyers should not seek to implement collaboration and hope for increased agility, but rather buy collaboration tools focused on agile development, and be assured of bottom-line results even beyond those of agile development itself. Don’t bother with a pre-short list; get a short list, put CollabNet on it, make your choice, and get going. Hopefully, we’re past the time when anything about agile development had to prove itself by traditional cost and quality metrics – which never saw the cost/revenue/margin/customer-satisfaction forest for the “you’re wasting time and money right now” trees. The benefits are already proven. Get on with it, already.
Of course, you never know which smaller companies will fold, or be acquired and lose their technology edge as they scramble to integrate with less-agile large-vendor solutions. I never say never, but CollabNet appears to be one of the least likely to fold – and the larger vendor may also provide additional useful collaboration software and services.
If you’re still reading this, you’re not agile enough. Go look at CollabNet-type agile-aimed collaborative development solutions. Make CollabNet one of the solutions to be inspected. Go.
Wednesday, January 11, 2012
Ooh, Those Hilarious Computer Industry Marketers!
Look, I know that picking on marketers is unfair. They have a really tough job, trying to explain the technology they can’t possibly fully understand to business customers and depending on techies to cover their rear flanks. But sometimes, sometimes it’s just impossible not to make fun of them. Because what they say is just so – just so … funny.
I just got an email from IBM that starts out like this: “IBM has announced the closing of its acquisition of Green Hat, which expands IBM's ability to help customers improve the quality of software applications by enabling developers to conduct testing on a software application prior to its delivery.” The italics are mine, all mine. Look, I know what they were trying to say. I know that Green Hat has valuable technology, and it can indeed improve development quality in certain cases. I know that bad grammar played a role. But still …
Gosh, IBM, this is marvelous. I never thought of testing an application before delivery before. What an incredible insight! Maybe that explains why every software application in the world before now has crashed before you can even use it! If it weren’t for the computer hardware that just sits there and blinks its lights, I bet we would never have been able to have such a vibrant and humongous computer industry. Please, IBM, don’t just stop there. Submit this for a Nobel Prize. Turn Green Hat into open source, for the good of all humankind. I beg you – or I will, when I can stop laughing.
Or maybe you’re restricting this to IBM customers. Maybe it’s only IBM customers who have never looked up, never considered the possibility of testing their applications before delivering them. 50 years in the salt mines, never able to get an application to work, always being teased by the developer down the hall who has written a Windows app to add two and two, with the dust and cobwebs accumulating all over you, and now, at last, a ray of sunlight, the mummy arises, the worm turns, and you can write your very first workable application for zEnterprise using Java and JDBC, Web-enabled out of the box, infinitely scalable, but baby steps … to let the entire corporation add three plus three. Oh, the ecstasy!
And the best part is that now I’m remembering all the other screamingly funny computer-industry marketing gems. Perhaps my favorite is a billboard I saw in Boston in the late 1990s, from Sun: “We built the Internet. Let us help you build your Internet.”
Oh, gosh, Mr. McNeally. So that was you in the government all along, pretending to be working for DARPA, physically going out to all the computers in all the telcos all over the world and secretly installing all the Internet software. All those employees at BB&N and CCA that thought up TCP/IP and newsgroups, routing and ftp, they were Sun employees, even though nominally Sun had about five people working for it at the time, supposedly on CAD/CAM! All those techies who did Berkeley UNIX, Mosaic, hypertext, HTTP, and all those administrators who installed them on existing enterprise servers and clients, they were all little Sun robots! It’s a shadow government! It’s the Illuminati! Of course I want you to come into my house and link my two PCs and publish their addresses! I wouldn’t dare not to, because you clearly control everything else in the world … Just give me a moment to stop laughing.
I remember that I actually pointed out that one out to a Sun marketer, back in the day. He was quite surprised, and insisted that it made perfect sense, because Sun had acquired some former BB&N employees. It was very hard to talk to him, because he kept making me want to crack up.
However, aside from their humor value, these marketing claims extraordinaire really do have value to me, and possibly to you, too. No, really. That is a straight face. Let me tell you a story …
Once there was a boy taking biology, faced with a test on which there was a question: When does the female rabbit ovulate? He couldn’t remember the answer. He knew it had to be something dramatic, because he knew that rabbits multiply like crazy – his family had had two in the backyard, until within a very short time of arrival they had gone from two to about 20, and his parents had moved very quickly to get rid of them all before they ate all the vegetation and multiplied again. And so he panicked, and his answer was: upon sight of the male rabbit.
The next class, the teacher handed out the graded papers. But before he did so, he made an announcement. I was grading these last night, he said, and I was getting too tired to finish. And then I ran across this answer – and, without naming the source, he related the boy’s answer to the question.
And so, he concluded, I was so amused by that answer, that it woke me up and I had enough energy to finish grading the papers. By the time the teacher said that, of course, the class was gone – gone to laughter. ROTFL.
And so, thank you, IBM. Thank you, Sun. Thank you, all the hard-working marketers who have ever made a mistake like this. Thanks to you, I can finish this blog post. Thanks to you, I will always remember that there are many amazing new technologies yet to be discovered, that will continue to enthrall me and improve the lot of humankind in the years to come. Like software testing. Like an entire global cloud infrastructure built by one person working for the Illuminati. Like female robots that ovulate on sight of the male. Like marketers that make it all happen without writing one single line of code.
It’s a wonderful world to write about. I’ll start writing immediately. Just as soon as I can manage to stop laughing.
I just got an email from IBM that starts out like this: “IBM has announced the closing of its acquisition of Green Hat, which expands IBM's ability to help customers improve the quality of software applications by enabling developers to conduct testing on a software application prior to its delivery.” The italics are mine, all mine. Look, I know what they were trying to say. I know that Green Hat has valuable technology, and it can indeed improve development quality in certain cases. I know that bad grammar played a role. But still …
Gosh, IBM, this is marvelous. I never thought of testing an application before delivery before. What an incredible insight! Maybe that explains why every software application in the world before now has crashed before you can even use it! If it weren’t for the computer hardware that just sits there and blinks its lights, I bet we would never have been able to have such a vibrant and humongous computer industry. Please, IBM, don’t just stop there. Submit this for a Nobel Prize. Turn Green Hat into open source, for the good of all humankind. I beg you – or I will, when I can stop laughing.
Or maybe you’re restricting this to IBM customers. Maybe it’s only IBM customers who have never looked up, never considered the possibility of testing their applications before delivering them. 50 years in the salt mines, never able to get an application to work, always being teased by the developer down the hall who has written a Windows app to add two and two, with the dust and cobwebs accumulating all over you, and now, at last, a ray of sunlight, the mummy arises, the worm turns, and you can write your very first workable application for zEnterprise using Java and JDBC, Web-enabled out of the box, infinitely scalable, but baby steps … to let the entire corporation add three plus three. Oh, the ecstasy!
And the best part is that now I’m remembering all the other screamingly funny computer-industry marketing gems. Perhaps my favorite is a billboard I saw in Boston in the late 1990s, from Sun: “We built the Internet. Let us help you build your Internet.”
Oh, gosh, Mr. McNeally. So that was you in the government all along, pretending to be working for DARPA, physically going out to all the computers in all the telcos all over the world and secretly installing all the Internet software. All those employees at BB&N and CCA that thought up TCP/IP and newsgroups, routing and ftp, they were Sun employees, even though nominally Sun had about five people working for it at the time, supposedly on CAD/CAM! All those techies who did Berkeley UNIX, Mosaic, hypertext, HTTP, and all those administrators who installed them on existing enterprise servers and clients, they were all little Sun robots! It’s a shadow government! It’s the Illuminati! Of course I want you to come into my house and link my two PCs and publish their addresses! I wouldn’t dare not to, because you clearly control everything else in the world … Just give me a moment to stop laughing.
I remember that I actually pointed out that one out to a Sun marketer, back in the day. He was quite surprised, and insisted that it made perfect sense, because Sun had acquired some former BB&N employees. It was very hard to talk to him, because he kept making me want to crack up.
However, aside from their humor value, these marketing claims extraordinaire really do have value to me, and possibly to you, too. No, really. That is a straight face. Let me tell you a story …
Once there was a boy taking biology, faced with a test on which there was a question: When does the female rabbit ovulate? He couldn’t remember the answer. He knew it had to be something dramatic, because he knew that rabbits multiply like crazy – his family had had two in the backyard, until within a very short time of arrival they had gone from two to about 20, and his parents had moved very quickly to get rid of them all before they ate all the vegetation and multiplied again. And so he panicked, and his answer was: upon sight of the male rabbit.
The next class, the teacher handed out the graded papers. But before he did so, he made an announcement. I was grading these last night, he said, and I was getting too tired to finish. And then I ran across this answer – and, without naming the source, he related the boy’s answer to the question.
And so, he concluded, I was so amused by that answer, that it woke me up and I had enough energy to finish grading the papers. By the time the teacher said that, of course, the class was gone – gone to laughter. ROTFL.
And so, thank you, IBM. Thank you, Sun. Thank you, all the hard-working marketers who have ever made a mistake like this. Thanks to you, I can finish this blog post. Thanks to you, I will always remember that there are many amazing new technologies yet to be discovered, that will continue to enthrall me and improve the lot of humankind in the years to come. Like software testing. Like an entire global cloud infrastructure built by one person working for the Illuminati. Like female robots that ovulate on sight of the male. Like marketers that make it all happen without writing one single line of code.
It’s a wonderful world to write about. I’ll start writing immediately. Just as soon as I can manage to stop laughing.
Tuesday, January 10, 2012
The Other BI: Birst and the Mobile Ad-Hoc Query
This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about BI today.
The Importance of Mobile Ad-Hoc Query Software Technology to BI
The mobile business market in general, and mobile BI in particular, have taken off over the last year, as end users “roaming around” the warehouse with a smart phone or using a tablet for local central-database-syncing sales calls lead to a “three-legged mobile stool”: smartphone for in-house, smartphone or tablet for local, and laptop for emerging-market instant-on “virtual offices.” And yet, what kind of mobile BI? The main focus of many vendors, including the majors, seems to be BI that analyzes mobile uses, not BI on mobile devices. Those that do aim to do BI on a tablet seem to be emphasizing reporting, not analytics. So where’s the BI that the rest of the industry seems to be focusing on? Where’s the exploratory, in-depth, ad-hoc querying on mobile devices, with a similar look-and-feel to the enterprise’s data-warehousing-type analytics?
Well, the capabilities are there. Something like Google Analytics provides the common infrastructure software for apps running on iPhone, Android, and traditional operating systems, allowing write-once-deploy-many. But Google Analytics is scant on ad-hoc query support, preferring instead to emphasize delivering flexible access to canned reports. Most other vendors seem to do likewise, at the level of their own BI interfaces.
That’s fine for now, as end users slowly get their heads around simple, routine querying. But very soon now, they will want the kind of slide-with-your-hand, pinch-and-expand way of looking at data in more depth that the new mobile interfaces presently provide for searching through a list of YouTube videos for the perfect Facebook post, or zeroing in on the best app for finding a restaurant in the area. For example, there’s birst mobile for the iPad, which can “flick through charts and tables, filter to information of interest and drill into more detail.” And once end users get that, they will indeed apply it to all three legs of the stool (as laptops move to support the new touch screens in their new generation).
The business benefits of that are very much analogous to the business benefits of desktop productivity software like Excel and PowerPoint in the 1990s. They are consumer-market-driven, ubiquitous improvements in employee productivity that, as in that generation of technology, eventually lift the entire global economy to a faster rate of increase. The business benefits of today’s mobile BI are top-down: they are about driving the business’ notion of BI to their shop floors and offices as the employee walks around or takes a trip or sets up a new office on the cheap. The business benefits of mobile ad-hoc querying are about employees piggybacking on the flexibility of those queries to generate new uses for mobile BI and analytics in general, because analytics is being used in places and situations where it never could be used before.
The Relevance of Birst to Mobile Ad-Hoc Query BI
I cannot say, as I have for other Other BI solution vendors, that Birst is a completely safe and well-established vendor. Birst is relatively new, and specializes in cloud BI, a BI market that is much more dynamic and unpredictable. What I can say is that, based on the evidence of its web page, Birst is further along than most major vendors towards delivering common-look-and-feel, fully functional mobile ad-hoc querying in a full BI suite, and that it makes sense that this would be so.
No, it’s not because Birst was able to design for mobile ad-hoc querying from the ground up; that might be true of Birst’s cloud BI, but not of its mobile BI, since Birst is not primarily aimed at mobile BI. Rather, it is because cloud BI, as is the case with cloud markets in general, have a greater emphasis on the SMB customer, and therefore on the naïve corporate end user playing around with the data. It’s relatively simple to extend that kind of user interface to ad-hoc querying on mobile platforms, and the obvious way to do so on the cheap is to provide a similar app with a similar look-and-feel on each mobile platform, based on standardized if not open-source infrastructure software.
I rather doubt that the major competition for cloud-BI-type mobile ad-hoc querying will be the major BI vendors like IBM, Oracle, SAP, and SAS, over the next 2-3 years, although I could be wrong. They seem content to follow their business customers, rather than getting ahead of consumer or SMB customers. No, it may very well be that the main competitor for cloud ad-hoc BI in 2015 is the generation of hundreds and thousands of iPhone, tablet, and Google Analytics-platform apps that together cover much of the immediate use cases of mobile ad-hoc querying, and which slowly integrate together into full-fledged business “pocket querying.” Right now, however, that’s an expensive way to go, because the business has to figure out how integrate and customize an as-yet-unknown set of apps. No, right now the best path forward is a flexible “core” mobile ad-hoc querying and BI suite with which new apps can integrate as needed. Birst, among others, would appear to have such a “core.”
Potential Uses of Mobile Ad-Hoc Querying for IT
As with desktop productivity software, IT should both support (as in, follow the lead of) end users employing mobile ad-hoc querying and also integrate their querying into an overall enterprise BI architecture. Right now, this means ensuring that a “core” platform like Birst is both fully deployed on mobile platforms (whether used or not) and integrated with existing analytics. Thus, the end user may, if he or she wishes, combine the latest Web-searching mobile app with access to corporate data using the same look-and-feel.
Over time, as the market sorts out, it is possible that a platform like Birst’s will be superseded by an agglomeration of apps, as suggested above. No harm, no foul; one can simply fade out the “core” as its integration functions are steadily replicated by the loosely-coupled apps. Or, the major vendors can catch up in their mobile ad-hoc query functionality. Sorry, a replacement is not necessarily justified in this case. Remember, this is being driven by end-user uses, so there is no point in risking the possibility that the vendor hasn’t got it right yet, and especially when the Birst-type platform being replaced depends on standardized, flexible infrastructure software that is well suited to following the ongoing changes in business-user use cases. As in the Bring Your Own Device (BYOD) case, simplicity for the end user trumps simplicity for IT.
And finally, in some cases IT may want to use this technology as a way of getting one’s feet wet in both mobile ad-hoc querying and cloud BI – a two-for-the-price-of-one deal. This does not mean that IT must tackle both topics at once; it simply means that IT can do them at least partially in parallel if it wants to. That would seem especially appropriate for medium-sized organizations that are in a driving hurry to find a kind of analytics technology where they can outflank the big guys, at minimal cost and with minimal effort.
The Bottom Line for IT Buyers
Somewhere in the next 2-3 years, some sort of mobile ad-hoc querying is going to happen, and IT is going to start seeing support calls about it. Sometime after that, it will really take off, and IT will be scrambling to keep up. At that point, IT will be really glad if it has at least a “core” Birst-type mobile ad-hoc-querying/BI platform that will allow it to sort out the confusion.
In the meanwhile, IT buyers should either start the process of creating a pre-short list, or kick the tires of and/or implement a tentative “core” platform. As I have said, Birst would seem to be one candidate for pre-short list or implementation.
It’s hard to tell at this point where some other candidates will come from. Perhaps Google will expand Google Analytics’ capabilities to mobile ad-hoc querying, and they will become a de-facto standard; perhaps SAP/Sybase will move its standout mobile infrastructure software in that direction. Or, someone may acquire Birst and add on its own superb core-platform features. Or Steve Jobs may return and make the swipe-and-pinch interface as dated as hula hoops … naaah.
So, IT buyers, you thought you were getting comfortable with buying for mobile BI? You ain’t seen nothing yet. Still, start familiarizing yourself with solutions like Birst’s, and you should be better set up for the next 2-3 years. If not …
I suppose you could pray for the return of hula hoops. One of these centuries, that should work.
The Importance of Mobile Ad-Hoc Query Software Technology to BI
The mobile business market in general, and mobile BI in particular, have taken off over the last year, as end users “roaming around” the warehouse with a smart phone or using a tablet for local central-database-syncing sales calls lead to a “three-legged mobile stool”: smartphone for in-house, smartphone or tablet for local, and laptop for emerging-market instant-on “virtual offices.” And yet, what kind of mobile BI? The main focus of many vendors, including the majors, seems to be BI that analyzes mobile uses, not BI on mobile devices. Those that do aim to do BI on a tablet seem to be emphasizing reporting, not analytics. So where’s the BI that the rest of the industry seems to be focusing on? Where’s the exploratory, in-depth, ad-hoc querying on mobile devices, with a similar look-and-feel to the enterprise’s data-warehousing-type analytics?
Well, the capabilities are there. Something like Google Analytics provides the common infrastructure software for apps running on iPhone, Android, and traditional operating systems, allowing write-once-deploy-many. But Google Analytics is scant on ad-hoc query support, preferring instead to emphasize delivering flexible access to canned reports. Most other vendors seem to do likewise, at the level of their own BI interfaces.
That’s fine for now, as end users slowly get their heads around simple, routine querying. But very soon now, they will want the kind of slide-with-your-hand, pinch-and-expand way of looking at data in more depth that the new mobile interfaces presently provide for searching through a list of YouTube videos for the perfect Facebook post, or zeroing in on the best app for finding a restaurant in the area. For example, there’s birst mobile for the iPad, which can “flick through charts and tables, filter to information of interest and drill into more detail.” And once end users get that, they will indeed apply it to all three legs of the stool (as laptops move to support the new touch screens in their new generation).
The business benefits of that are very much analogous to the business benefits of desktop productivity software like Excel and PowerPoint in the 1990s. They are consumer-market-driven, ubiquitous improvements in employee productivity that, as in that generation of technology, eventually lift the entire global economy to a faster rate of increase. The business benefits of today’s mobile BI are top-down: they are about driving the business’ notion of BI to their shop floors and offices as the employee walks around or takes a trip or sets up a new office on the cheap. The business benefits of mobile ad-hoc querying are about employees piggybacking on the flexibility of those queries to generate new uses for mobile BI and analytics in general, because analytics is being used in places and situations where it never could be used before.
The Relevance of Birst to Mobile Ad-Hoc Query BI
I cannot say, as I have for other Other BI solution vendors, that Birst is a completely safe and well-established vendor. Birst is relatively new, and specializes in cloud BI, a BI market that is much more dynamic and unpredictable. What I can say is that, based on the evidence of its web page, Birst is further along than most major vendors towards delivering common-look-and-feel, fully functional mobile ad-hoc querying in a full BI suite, and that it makes sense that this would be so.
No, it’s not because Birst was able to design for mobile ad-hoc querying from the ground up; that might be true of Birst’s cloud BI, but not of its mobile BI, since Birst is not primarily aimed at mobile BI. Rather, it is because cloud BI, as is the case with cloud markets in general, have a greater emphasis on the SMB customer, and therefore on the naïve corporate end user playing around with the data. It’s relatively simple to extend that kind of user interface to ad-hoc querying on mobile platforms, and the obvious way to do so on the cheap is to provide a similar app with a similar look-and-feel on each mobile platform, based on standardized if not open-source infrastructure software.
I rather doubt that the major competition for cloud-BI-type mobile ad-hoc querying will be the major BI vendors like IBM, Oracle, SAP, and SAS, over the next 2-3 years, although I could be wrong. They seem content to follow their business customers, rather than getting ahead of consumer or SMB customers. No, it may very well be that the main competitor for cloud ad-hoc BI in 2015 is the generation of hundreds and thousands of iPhone, tablet, and Google Analytics-platform apps that together cover much of the immediate use cases of mobile ad-hoc querying, and which slowly integrate together into full-fledged business “pocket querying.” Right now, however, that’s an expensive way to go, because the business has to figure out how integrate and customize an as-yet-unknown set of apps. No, right now the best path forward is a flexible “core” mobile ad-hoc querying and BI suite with which new apps can integrate as needed. Birst, among others, would appear to have such a “core.”
Potential Uses of Mobile Ad-Hoc Querying for IT
As with desktop productivity software, IT should both support (as in, follow the lead of) end users employing mobile ad-hoc querying and also integrate their querying into an overall enterprise BI architecture. Right now, this means ensuring that a “core” platform like Birst is both fully deployed on mobile platforms (whether used or not) and integrated with existing analytics. Thus, the end user may, if he or she wishes, combine the latest Web-searching mobile app with access to corporate data using the same look-and-feel.
Over time, as the market sorts out, it is possible that a platform like Birst’s will be superseded by an agglomeration of apps, as suggested above. No harm, no foul; one can simply fade out the “core” as its integration functions are steadily replicated by the loosely-coupled apps. Or, the major vendors can catch up in their mobile ad-hoc query functionality. Sorry, a replacement is not necessarily justified in this case. Remember, this is being driven by end-user uses, so there is no point in risking the possibility that the vendor hasn’t got it right yet, and especially when the Birst-type platform being replaced depends on standardized, flexible infrastructure software that is well suited to following the ongoing changes in business-user use cases. As in the Bring Your Own Device (BYOD) case, simplicity for the end user trumps simplicity for IT.
And finally, in some cases IT may want to use this technology as a way of getting one’s feet wet in both mobile ad-hoc querying and cloud BI – a two-for-the-price-of-one deal. This does not mean that IT must tackle both topics at once; it simply means that IT can do them at least partially in parallel if it wants to. That would seem especially appropriate for medium-sized organizations that are in a driving hurry to find a kind of analytics technology where they can outflank the big guys, at minimal cost and with minimal effort.
The Bottom Line for IT Buyers
Somewhere in the next 2-3 years, some sort of mobile ad-hoc querying is going to happen, and IT is going to start seeing support calls about it. Sometime after that, it will really take off, and IT will be scrambling to keep up. At that point, IT will be really glad if it has at least a “core” Birst-type mobile ad-hoc-querying/BI platform that will allow it to sort out the confusion.
In the meanwhile, IT buyers should either start the process of creating a pre-short list, or kick the tires of and/or implement a tentative “core” platform. As I have said, Birst would seem to be one candidate for pre-short list or implementation.
It’s hard to tell at this point where some other candidates will come from. Perhaps Google will expand Google Analytics’ capabilities to mobile ad-hoc querying, and they will become a de-facto standard; perhaps SAP/Sybase will move its standout mobile infrastructure software in that direction. Or, someone may acquire Birst and add on its own superb core-platform features. Or Steve Jobs may return and make the swipe-and-pinch interface as dated as hula hoops … naaah.
So, IT buyers, you thought you were getting comfortable with buying for mobile BI? You ain’t seen nothing yet. Still, start familiarizing yourself with solutions like Birst’s, and you should be better set up for the next 2-3 years. If not …
I suppose you could pray for the return of hula hoops. One of these centuries, that should work.
Subscribe to:
Posts (Atom)