Data migration is dead, long live the enterprise architecture! The days of planning and executing a large scale data migration from one enterprise system to another, while maintaining on operational business, should be gone. There are too many failure modes in a modern enterprise extraction, transformation, and load (eETL) operation. Risk management initiatives are ill-equipped to provide the aversion models necessary to ensure the availability of critical business process in the event of the inevitable migration failure. But even amidst such information technology despair there is a hope, but at what cost?
Before we get into the discussion why enterprise data migration fail, lets first spend a bit of time of the basics. Data migration is often defined as process of transferring data between storage types, formats, and/or computer systems. In essence, it is the one off selection, preparation and movement of relevant data, of the right quality, to the right place, at the right time. It is usually performed manually and/or programmatically when organizations change (extend, upgrade, adopt) information systems, where the new data storage format is not the same as the old. But not all migrations of data are data migrations.
Data migration is the irregular movement of data (an important concept that will come back to later). Regular movements of data, such as those found in data warehouse refreshes, are fully automated, highly repeatable, and take advantage of pre-mapped data schemas, are not part of this data migration space. With data migration, a single pass movement of data is often accompanied by low or inconsistent initial quality of data between disparate databases and/or applications. It is this large-scaled multi-factor characteristic (semantics, people, process, technical, workflows, data quality, and product types) of enterprise data migration that make is so complex, lending itself to numerous failure modes.
IDC has estimated that over 80% of data migration projects fail or have significant cost overruns, which is itself a type of financial failure. Within this cohort, 50% exceed their projected schedule by 75%, their budgets are exceeded by 66%, and 33% of these fail entirely. While these failure statistics are based on a wide range of projects, over have used some form of formal migration methodology. These are not good results if your looking for migrate a enterprise system while maintaining critical business function and financial predictability. Moreover, give the complexity of today’s enterprise systems, you can only reduce the risk of migration failure, but you can not eliminate it completely.
While there are no industry standards for migrating data within the enterprise that guarantee success, there are many migration methodologies designed around risk reduction. The most common (e.g., Practical Data Migration) are based on variants of proof of technologies/concepts, landscape analyses, data quality/cleansing, engaging the business early and often, testing, and automation through ETL tools. While these are good practices, or in some cases even necessary, for any project, they have limited demonstrated ability to ensure that the data is correct/error free after being transformed through complex data quality and business transformation rules.
So, with its high complexity, enterprise data migration should be dead, not because it is bad or evil, but because it the risk of failure is just too high. But, if not the enterprise migration, then what? Well, the answer lays in moving away from data migration as an unplanned irregular movement of data, something today’s enterprise architecture (EA) is all about. But what is EA in the context of migration?
Enterprise architecture is the continuous practice of organizing logic for business processes and IT infrastructure that reflect the integration and standardization requirements of the company’s business operating model (TOGAF 2009). EA is more than just structure, it is an dynamic means to realize architectural vision, business architecture, information systems architecture, technology architecture, governance, and even migration architecture. EA treats the business as an organic growing entity that evolves through a continuum of changes. Which should not be a surprise to most, given that people (organic by nature) constitute most of a business’s enterprise to begin with.
The key difference between traditional data migrations and migrations developed through enterprise architecture is that EA-driven migrations tend to be less complex and therefore more successful. EA-driven migration succeed because, by their nature, they are designed to succeed. Migrations are not an afterthought, a single one time event of irregular movements of data. Enterprise data migrations are designed in the context, not in leu of, an evolving business and its infrastructure and is the means to the businesses ends, not the ends itself.
Traditional data migration died a long time ago, we just never noticed it because we were too busy cleaning up all the failure debris. It is time for us now to start thinking differently so we don’t repeat this same migration mistakes. Enterprise Architecture-driven migration presents the best hope of success for those looking to continuously evolve their business model.
For more articles like this, please see LiquidHub.
