Data Virtualization Day: A New, Useful Way To Migrate Legacy Databases

Twenty-odd years ago I compared vendor databases to boat anchors:  they (and the business-critical apps depending on them) are very difficult to sunset, and they tend to act as a drag on needed performance improvements in everything data-related.  Moreover, the basic technologies for performing “database migration” seem to be much the same as they were fifteen years ago:  conversion of dependent apps to the new interface by SQL “veneers” plus modification, reverse engineering, or full rewrite of the underlying app, one by painful one.  You can imagine my happy surprise, therefore, when a customer at this year’s Cisco/Composite Software’s DV Day testified that they were beginning to use a new technique for legacy database migrations, one that should significantly speed up and improve the safety of these migrations.

Before I go into detail about this new technique, I should mention some of the other great and useful ideas that, as usual, surfaced at DV Day.  Among these I might number:

1.       Use of data virtualization to combine traditional Big Data and the streaming events typical of the sensor-driven Web;

2.       A “sandbox” to allow real-world testing of DV apps before rollout;

3.       Additional hybrid-cloud support.

I hope to go into more detail on these in a later post.  The legacy migration idea, however, is worth its own post – indeed, worth attention from all large enterprises jaded by legacy-migration solutions that advance incrementally while the database-dependent app “legacy problem” grows apace.

The Previous State of the Art

Briefly, the problem of database migration – especially across database vendors – is typically much more a problem of migrating the applications written to take advantage of it than of migrating the data itself.  The typical such application, whether it be written using the commands of IBM IMS, CCA MODEL 204, DATACOM DB, Pervasive SQL, Sybase SQL Server, or any other such database, is 10-30 years old and not always that well documented, is partly written using database-specific commands or “tricks” in order to maximize performance, and does not share code with the tens or hundreds of other apps on the same database.  Therefore, migration time will often be projected at more than a year, no matter how much person-power one throws at it.

Broadly speaking, until now each such app migration has involved one of three approaches:

1.       Rip and replace, in which the entire app is rewritten for the new database;

2.       Emulation, in which an “old-database” veneer is placed over the new database, with rewrites only applied where this fails;

3.       Reverse engineering, in which the function of the app is described and then a new, supposedly identical app is generated from this “model” for the new database.

None of these approaches is one-size-fits-all.  Rip and replace runs the risk of missing key functionality in the old application.  Emulation often produces performance problems, as the tricks used to maximize performance in the old database can have the opposite effect in the new one.  Because of the lack of documentation, reverse engineering may be impossible to do, and it also may miss key functionality – although it often pays for itself by “doing it right the first time” in the new environment.

The New Data Virtualization Approach

As Anderson of HSBC described it at DV Day, the data virtualization approach uses a DV tool both as an integral part of a “sandbox” for apps being migrated, but also as a “recorder” of transactions fired at the old database, which serves as a test scenario for the new app.  Separately, none of these “innovations” is new; it is the combination that makes the approach novel and more useful.

Specifically, I see the “data virtualization” legacy-migration approach as new in several ways:

·         Data virtualization already creates database portability for apps, if one writes all apps to its “veneer” API.  The new approach allows the migrating app to join this ultra-portable crowd – and, don’t forget, DV has had almost 15 years of “embedded” experience in providing such a common interface to all sorts of data types and data-management interfaces.

·         The new approach allows a more flexible, staged approach to migrating hundreds of apps.  That is, the DV tool semi-automates the process of creating an performance-considering emulation, and the test scenarios then allow rollout when they indicate the app is ready for prime time, rather than when a separate full-scale test is run with fingers crossed.

·         The DV-tool process of “metadata discovery” means that migration often comes with additional knowledge – in effect, better documentation -- of the apps.

The net of these novelties, I conjecture, is faster (more parallel and more automated) migration of legacy apps, with better performance (counterintuitively, using DV can actually improve app transactional performance), with better future portability, better documentation, and better ability to share data with other apps in the enterprise’s BI-app portfolio via a global metadata repository.  Not bad at all.

The Net-Net

Is short and sweet.  I urge all large enterprises with significant legacy-database concerns to consider this new DV approach and kick the tires.  It is early days yet, but its value-add can hardly help but be significant.