The Other BI: Progress Apama and Event Processing

This blog post highlights a software company and technology that I view as potentially useful to organizations investing in business intelligence (BI) and analytics in the next few years. Note that, in my opinion, this company and solution are not typically “top of the mind” when we talk about BI today.

The Importance of the Apama Software Technology to BI

The value-add of Apama to BI, in my opinion, is the value-add of applying analytics to “data in motion” on a very broad range of data. Apama carries out “event processing”: conceptually, I think of event processing as a “processing head” monitoring an Enterprise Service Bus (ESB). Most data entering the organization, as well as data moving between data stores and between users within the organization, is wrapped up as messages and sent across the ESB to its destination. In the process, the “processing head” monitors the whole stream of data-in-motion and performs analytics, alerting, and other processing based on the type of data being reviewed (or, in the aggregate, the “pattern” of a stream of related data). What is unprecedented about this kind of data processing is that (a) it focuses on data across organizational units, unlike the typical data warehouse or operational database; (b) it intercepts some data the moment it arrives in the organization, which is the ultimate in real-time business-critical data processing; and (c) it can draw a direct line between that data and a decision-maker by alerting, so that business-critical decisions can be made as quickly as possible.

Practically, of course, an “event processor” can do (c) only for a certain small subset of the information in the organization, because by itself an event-processing database does not scale nearly as much as a data warehouse. The event processor has much less historical “context” as it processes each datum, because it simply does not have the time to perform a “query from hell” on terabytes of historical data before the next datum must be processed. In-depth analytics will simply have to wait, often for an hour or more. Nevertheless, this kind of instantaneous response is, in the real world, enormously valuable when fast response to the type of events that the event processor detects from the data is indeed mission-critical and/or business-critical.

At the same time, (a) – the ability to correlate data across organizational units – is an often-underestimated value of the event processor. As the discipline of systems analysis understands, a collection of business units is as much a set of process flows between units as a set of stand-alone companies. The job of corporate is often to ensure that these process flows work well, and the value-add of the event processor in this case is to provide enterprise performance management (EPM) that reads the tea leaves of particular process flows and ties them back to glitches in the performance of the units and their coordination. In plain English, a good event processor goes beyond what you could do before because it lets you respond immediately to some new threats and opportunities in your environment, and because it tells you some of the things that are really happening to muck up your business’ overall performance as it coordinates business units. If you combine the two, you get the famous “360-degree view” inside and outside the organization.

Apama, like other event processors, never operates in a vacuum. All organizations already have operational and decision-support databases supporting key applications, and event processing must adapt itself to handle what these do not. Therefore, Apama’s value-add within the limits of (a)-(c) above can vary quite widely. Always, however, if the user does a careful analysis of the most important decisions that need to be speeded up and the gaps in business performance information, the BI done by an event processor like Apama has a major impact on the organization – not on its bottom line, necessarily, but always on its business risk. The out-of-the-blue event that businesses always face becomes much less risky when an event processor manages to detect it in a timely fashion.

Right now, businesses of all sizes are still in the early stages of use of event processing – you can tell, because case studies typically trumpet particular per-project uses. Therefore, the field is wide open for approaches such as the “event-driven architecture”, in which an event processor on top of an ESB becomes the focal point at which corporate can not only monitor but also direct information flows; the “complex event processor”, in which on-the-fly analytics becomes far deeper; and “data streaming”, in which the whole notion of a data-warehouse database is upended to be a “processing head” handling querying on multiple parallel “streams” of XML-type data. These, however, will often not achieve full implementation until 2-4 years from now, at the least. The key value-add of event processing in real-world BI over the next 2-3 years, I believe, will be the ongoing identification and implementation of the most important alerts, decisions, and business-unit correlations that it can handle, and their integration with the existing BI architecture.

The Relevance of Progress Software to BI
The relevance of Progress Software itself to BI, and to data processing in general, is less clear than in its “glory days,” when (imho) it was a pioneer in near-lights-out database administration, rapid application development, Software as a Service (SaaS), and the ESB. In those days, it had a gift for identifying the innovative infrastructure-software simplification that led its SMB/departmental customers to rapid implementation of the latest large-enterprise functionality, and beyond. Apama arrived at the end of that period, as part of a successful Progress effort to provide services and software to allow its departmental customers to scale their SMB-type technology to the division, the line of business, and even the “edge” in the data center. Apama first found its niche in rapid analysis of massive streaming financial-market data; Progress Software appears, with its Event Manager, Event Modeler process-design end-user tool, SmartBlocks, and Dashboard Studio, to have added Progress’ own strengths in SMB-driven simplicity of use. To put it another way: Apama was born large-enterprise-ready; Progress added the veneer and tools that makes it fully in sync with open-source or agile BI as applied, say, to Big Data.

Thus, five years ago Progress Software would have been seen as an operational and decision-support database for an SMB, and a fast-moving local-level operational adjunct to enterprise BI in the Global 1000. Today, Progress Software has much less visibility in BI, and its connection to the latest BI technology is less visible; but if you look at the actual technology, Apama is indeed innovative and can deliver value-add across a wide range of enterprises. It only remains to ask, what’s the future of Progress Software as a supplier of event processing technology, and in general?

There are two parts to my answer. First, let me note the characteristic that Progress Software shares with just about every database company: it has a core loyal base of customers whose size may shrink, but whose tendency not to migrate away from the platform ensures that Progress Software will be extraordinarily long-lived. As in the past few years, database revenues may ebb over time; but few if any database companies in my 30 years of acquaintanceship with the industry see a massive collapse of their installed base. In the next 2-3 years, as sure as the sun rises, reasonable management plus this revenue flow will see Progress Software still standing (acquired or not) and still supporting Apama plus the infrastructure software like the Progress ESB that complements Apama.

However, there is little in the past three years, where revenues have been essentially flat, to suggest that Progress Software’s glory days will return, and that it will identify another new infrastructure technology to get strong growth started again. Moreover, Progress’ strength has never been that its solutions were a key part of the mainstream of Web innovation, and so there is no obvious reason to expect that Apama can, at a minimum, transition to form the core of cutting-edge open-source BI solutions. And that brings me to the second part of my answer.

I assert that these caveats almost certainly do not matter, in the next 2-3 years and probably further out. The reason is that Progress Software’s DNA may not be Web or open-source, but it is very definitely SMB-simple and flexible, with no vendor lock-in. Apama pre-acquisition would probably be a niche financial-market BI product. Apama plus Progress Software is a uniquely flexible and easy-to-use event processor that integrates with the rest of your architecture just fine, and it will stay that way. Progress Software’s new services prowess just ensures that the simplicity scales up to the largest of enterprises.

Potential Uses of DataRush for IT
The net of the contributions of both Apama technology and Progress Software’s “approach” to the Progress Apama solution is that, whether you are an SMB tackling BI for the first time or a large enterprise trying to become more agile in your querying of Big Data, Apama offers differentiated value-add, either stand-alone or as the complement to a large-vendor event processor and database architecture like IBM Streams and IBM Information Server. In the case of an SMB, the use case is straightforward: you can fit Apama with agile-BI efforts as a front end to catch urgent alerts implicit in Big Data, raw operational data, or ongoing analytics; and you can begin to develop EPM. Large enterprises typically have bottom-up Windows/desktop computing in parallel with the massive datacenter data warehouse: they can grow Apama with the evolution of that side of the enterprise architecture, while if appropriate also driving forward their top-down, datacenter-driven event processing projects using other vendors’ event-processing solutions, and easily integrating the two.

In either use case, the key to the most rapid possible success (I think) will be to identify on an ongoing basis the key targets for alerting and rapid decision-making, and to tie Apama as much as possible to historical data in order to allow the greatest “depth” of analytics at the point of data intercept. One good detection of a major customer about to dump you and immediate, effective reaction to prevent it will make the whole exercise more than worthwhile. And, remember, we are talking low-touch, very-low-TCO event processing here.

It is unusually hard to think of things that can go wrong with Progress Apama implementation. Services? Not as much needed, and Progress Software services that may be needed are already real-world-proven from more than 20 years of rave reviews from SMB and departmental clients, plus 5 years of driving similar technology into the division and LOB. Training? Again, the Progress Software track record is that even an untrained local-office manager can handle database maintenance – which is usually the biggest concern – and any developer can handle the drag-and-drop development tools. Integration? Progress Software has what I view as the standard set of adapters and gateways. Limits to scalability? Not if the Progress ESB is any guide. Let’s face it, IT implementation of Apama is very unlikely to be rocket science – and neither is gaining BI insights with it.

The Bottom Line for IT Buyers
At this point, an IT buyer should view Progress Software’s Apama as roughly equivalent to a “diamond in the rough.” It is the center of attention neither of BI, nor streaming technology, nor even sometimes of Progress Software itself. It suffers undeservedly from questions about Progress Software’s future, the future of event processing technology, whether Apama will continue to track BI technology, and a reputation as a high-end or specialized event processor. All it has going for it is that it is a more simple, more flexible, powerful event processing tool for a wide variety of use cases and scales, and should continue to deliver for the next 3-5 years, and almost certainly longer. And that should be plenty.

I have noted that Apama’s (and event processing’s) main value-add in BI is more in the risk area than in the top or bottom line (although some implementations, like EPM, do indeed impact revenues and costs). However, this is one technology that, when it succeeds, is really, really visible. Sell it to corporate as the latest technology fad if you like; the odds are that when the IT buyer acquires and then IT implements Apama, a big success story happens in the next year. And then you can concentrate on what’s really important: integration with the rest of your BI so that it all works optimally, in harmony.

The net-net for IT buyers, therefore, is to do a “reality check” on present-day event processing in BI, and then prepare a short list of event-processing software vendors to take the next step. I see no reason why, in most if not all cases, Progress Software’s Apama should not be on that Other BI “pre-short list.”