The Other BI: Composite CIS/Studio and Agile BI

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about BI today.

The Importance of the Composite Software Technology to BI

Today’s enthusiasm for “agile BI” that often consists of slapdash applications of SCRUM to rapid roll-out of new analytics applets obscures the fact that business agility, not to mention New Product Development (NPD) agility, involves applying more of agile methodologies than a sprint, and using agile methodologies in many more areas than software development. However, this is where the market is right now. The issue at hand is not so much the better application of business agility in the future, as applying today’s “agile BI” more effectively in the next 2-3 years. This is best done by recognizing one key fact: infusing the organization’s BI with agility is not primarily about development agility – it is about information agility.

Speeding up an inflexible business process via targeted analytics simply doubles your investment in and commitment to an information-handling process that needs instead the ability to fundamentally change. As I have noted in past posts, surveys show that in the typical process of receiving data and using analytics as part of the process of turning it into the right decision by the right decision-maker, or the right change in the business, the biggest problem is not that information is delivered too slowly. The fundamental problem is (a) that “leaks” at every step in the process result in more than 2/3 of the actionable information being lost along the way, and (b) the inflexibility of the process – specifically, its lack of openness to new types of data as they arrive – keep making the “leaks” worse.

Today’s best technology for dealing with (a) and (b) is something now called “data federation” or “data virtualization”. I won’t repeat the long litany of benefits to be expected from data virtualization; here, I will simply note that data virtualization, like master data management, provides one view of related data across the enterprise to the developer and the end user, and, unlike master data management, it can actively reach out, within or beyond the enterprise (as in Composite Discovery), to discover new data sources that can likewise be part of that global view of potential information. The one view of data tackles problem (a) at several critical points by ensuring that everyone can potentially see all data, and that analytics tools can be potentially applied to all useful data. The discovery feature tackles problem (b), if used effectively, by constantly refreshing the actionable data from, let’s say, new social-media data sources not presently covered by BI – as some Composite solutions do for Big Data.

Solving problem (b), of course, improves the organization’s information agility, not just its development agility. So the quick and effective method of tackling information agility in the here and now is to make a data virtualization development tool part and parcel of agile BI projects. Tools from folks like Composite Software, Tibco, Denodo, IBM, and SAP/Sybase are among those that would seem to fit the bill.

The Relevance of Composite Software to BI

Surprisingly enough, for a while around five years ago data virtualization’s predecessor, Enterprise Information Integration, was a hot topic in BI, as a way of extending data warehousing to operational data stores in order to query certain key new data in “near real time.” The primary results of this “fad” were to finally and definitively establish a major market for data virtualization, as well as to cement BI as Composite Software’s core market. In other words, unlike most or all other data-virtualization vendors, Composite Software has ten years of experience with real-world data virtualization at medium to large scales, five years of experience with BI at the same scales, a demonstrated ability to combine with other flexible BI tools such as MicroStrategy, and a focus on handling more “complex” needs that translates naturally to the more complex in-depth analytics typical of social-media and related Big Data analytics.

It may reasonably be objected that Composite, as well as other data virtualization vendors, has shown little or no interest in agile software development methodologies up to now, and therefore use of data virtualization solutions such as Composite’s may tend to slow down agile BI development efforts. Again, I would argue, as someone who has been touting agile development nearly since its inception twelve years ago, that this misses the point on two counts. In the first place, the rule of thumb with agile tools is “lead, follow, or get out of the way.” Where a SCRUM-supporting tool leads by urging developers towards agile best practices, and a really good refactoring tool follows in its wake by adding flexibility to the code thus produced, there is a third category of tools that simply automates and simplifies the necessary tasks of software development, while not getting in the way of the developer – and that is what something like Composite Information Server (CIS) with Composite Studio does for data-accessing functions in agile-BI programs.

In the second place, data virtualization such as Composite Software’s comes with a built-in gain in any agile-BI program’s information agility, which is often just as valuable as increased analytic-program implementation speed. Out of the box, the developer is assured of not missing a key in-enterprise data source. More subtly, the developer is made aware of other data sources than the one he or she may initially have targeted, which means that, as should be expected in agile development projects, new and unexpected features with value-add surface in the middle of the project.

In other words, while Composite Software may be in the middle of the pack when it comes to agility applied to agile-BI software development, it has long taken a leadership role in the features of data virtualization that really matter to agile-BI users, including BI services expertise, discovery, and complex data virtualization.

One final caveat appears to be going away. Since their inception, data virtualization tools have been oriented towards users above a certain size of medium-sized organization. Obviously, much of the action and excitement around agile BI has centered around its use in the public cloud by SMBs, who traditionally don’t deal with lots of various data sources that could use virtualization. However, what these SMBs are finding is that some public-cloud data is inherently “multi-organization”, and therefore requires federation and virtualization – especially since many of these same SMBs are now using multiple public clouds for their analytics. Sooner or later, Big Data has to be combined with ERP data on the cloud, and then with customer data on salesforce.com or wherever, and so on. As a result, even these SMBs are probably going to find data virtualization and Composite-type technology useful in their agile-BI efforts sooner rather than later.

Potential Uses of CIS-Type Agile BI for IT

Until recently, data virtualization in the enterprise has typically been done on a per-project basis, with the result that spreading it across all in-enterprise data is typically a work in progress. Obviously, this limits a little the usefulness of data virtualization in agile-BI development in the near future; but it is also an opportunity, in those enterprises that qualify, to kick-start the “global metadata repository” effort into high gear. Slap a data virtualization engine and development tool like CIS with Composite Studio into the agile-BI development team’s toolset, and have them scream to you about needing it to cover more data. That will get corporate’s attention, fast.

For those organizations that haven’t used data virtualization before, especially the SMBs, using data virtualization on things like Big Data is an excellent opportunity to avoid the arrows that the large-enterprise data-virtualization pioneer users faced. More specifically, if you start out by federating key customer Big Data with ERP on the cloud, private or public, you have already created a repository that covers most of your likely business-critical data in the next 2-3 years, and so you will not have to retrofit it later on.

A third, often underestimated, use of data virtualization is as a tool to support the new “business analysts” on the corporate side who are also doing agile analytics. A recent survey cited by Sloan Management Review suggests that these now view IT as slower and lower than dirt. Give them and support a tool that allows them to go out and discover new data sources, and I would not be surprised if they responded, like the woman conducting a choir in the movie Love Story, with “That is absolutely, stupendously, incredibly – OK.” That is, they won’t kiss you, but they will make a big shift towards tolerating you.

However, the use of CIS-type data virtualization that is dearest to my heart, and should be dearest to yours if you ever want to accomplish business agility rather than just BI “agility,” is applying its discovery features constantly, out on the Web, embedded in your analytics applications, to find and bring in-house new data and new data types. One key finding of business surveys is that only half of respondents find out about new key data heralding new market trends on the Web in less than half a year. Data virtualization can cut that to a day. You see, business agility isn't about reacting quickly and effectively to changes in your environment that you happen to find out about because they show up eventually in your customer complaints; the critical success factor is finding out all the changes that you can, as they happen out there, and then reacting quickly and effectively.

The Bottom Line for IT Buyers

Over the next few years, IT buyers should be viewing the acquisition and use of data virtualization tools in agile BI, among other things, as a no-brainer – but they don’t. Those smart IT buyers who do, however, will find that a pre-short-list is not necessary: on closer examination, the use cases almost present themselves. At any rate, that has been users’ experience in data virtualization’s original core markets such as the armed forces in the government – another Composite installed base.
However, for those constrained by budgets even in their strategic analytics initiatives, Composite and others may well have to go on a “pre-short list.” This should be of a peculiar type, however. The IT buyer should not passively wait for corporate to give IT a mandate that requires data virtualization. Instead, the IT buyer should be actively looking for the opportunity to introduce or extend agile-BI data virtualization, like the politician looking in every budget process or special spending initiative for a way of cutting taxes.

Composite Software is one among several who should be on that short list, as well as other data-virtualization-focused firms like Denodo and Tibco, not to mention the data federation solutions of IBM, Oracle via its BEA WebLogic Liquid Data acquisition, Red Hat/Metamatrix, and SAP/Sybase. Composite Software’s edge over most of these is simply that, over the last 10 years, Composite has established its experience and a leadership role in features and services useful to agile BI, and is among the most vocal about their plans to support Big Data and agile BI. Since that leadership role hasn’t diminished over the last three years, Composite Software will probably maintain its position at or near the top of agile-BI data virtualization short lists over the next 2-3 years, as well.

The bottom line for IT buyers of Other agile-BI data virtualization in general, and Composite Software’s CIS and Composite Studio in particular, is this: buy it right now or buy it a year from now, use it for BI agility or use it for business agility, but buy it and use it for some agile-BI initiative sometime in the next 2-3 years. Like an agile development methodology, it looks like it might cost more and cut into the bottom line, but it repeatedly costs less and delivers more revenue – because, used right, it improves your information agility.