Everything to Know About Data Migration

Data Migration September 24, 2020

Priyanka Bhadani

Big data is perhaps the biggest buzzword of the previous decade, compelling thousands of companies to see digital information as the lifeblood of their business. Research by IDC predicts global revenues for big data and business analytics to reach US$274.3 billion in 2020 - up from $189 billion in the previous year.

This surge in investments in enterprise data systems isn't surprising. The world's leading companies know firsthand that data has transformative power. It gives them the knowledge and predictive capability to innovate, create new products and services, improve customer experience, and generate revenue.

Deciding to leverage data, however, requires having well-established processes in place, such as a data migration plan.

What is Data Migration and Why Is it Important?

In a nutshell, data migration is the process of transferring an organization's data, applications, and digital systems from one location to another. However, it can also refer to transferring data from one format to the other or one application to another.

Data migration is an essential component of an organization's overall data management strategy. It is a necessary process when upgrading or consolidating server and data storage hardware, adopting a new application, or moving to a new computing environment (e.g., the cloud).

There are many reasons to migrate data from one place to another. The most common and easiest example to explain is the moving of databases and applications from legacy, on-premise servers (typically an aging data center) to the public cloud.

In cases like this, the existing infrastructure is too old and slow, and may even take up too much space. One solution is to move datasets to a new cloud-based application or server to drive business growth and flexibility through faster, more agile processes. Data migration can also mean moving workloads and databases from one cloud platform to another provider.

Primary Types of Data Migration

Data migrations come in different types and involve different business drivers. Data migrations typically cover databases, applications, storage, and cloud environments.

Database Migration

This type of data migration occurs when moving information from one or more source databases to another target database. This migration is typically needed when updating database software, migrating to the cloud, or switching between database vendors.

This requires a lot of planning and careful testing due to the many tasks involved in the migration process, from determining the target database's storage capacity to ensuring data integrity.

Application Migration

These migrations involve moving data from one vendor application or platform to another. For example, a small business can move from an on-premises version of Microsoft Office to a cloud-based service. It can also mean migrating to a new application from a different provider.

Storage Migration

Storage migrations deal with moving data between two or more storage systems, such as an on-premise server and the cloud. These migrations are usually driven not so much by lack of storage, but by a need for technology upgrades.

Migrating to a new storage system provides an opportunity to roll out data management features, such as data validation and cloning. Cloud storage migration also enables faster and more cost-effective scaling.

Cloud Migration

This involves any migration of data from on-premises to the cloud or from one cloud environment or platform to another. Cloud migrations are arguably the most popular type of migration, with 70% of companies adopting some form of cloud computing and 71% believing it is the key to simplifying their operations, according to a Harvard Business Review Analytics Services report.

A cloud migration project, however, should not be confused with backing up data to the cloud. Migrating data to the cloud means moving data to a new native environment accessible through the Internet.

What to Consider When Building a Data Migration Strategy

Data migrations are often complex undertakings that require careful planning and attention to detail. Businesses should consider these factors when approaching their data migration project.

Workload Type

Depending on the type of workload involved, the data migration process may require some downtime. Organizations can use vendor-provided data migration tools to seamlessly move workloads like virtual machines, databases, and data backups.]

For mission-critical workloads, the best practice is to migrate data in stages, keep source and target systems running simultaneously, and test in intervals throughout the process. Alternatively, companies can also opt for a large, one-time transfer and perform it outside of production hours to minimize the impact of downtime.

Amount of Data Involved

The amount of data to be migrated to a new location is another crucial factor to consider.

For migrations smaller than 10 terabytes (TB) of data, the cheapest and most efficient way to move the information is by shipping a physical storage solution, such as an external hard drive, to its new location.

Mass migrations involving petabytes of data, however, can use a dedicated data migration device provided by a vendor.

Migration Speed

With online migrations, the amount of data and the speed of the network connection determine the length of time required for the process. For offline migrations, the time to physically move the storage solution is the primary variable.

In any case, whether data is moved online or offline, it's important to have an accurate estimate of the data transfer's timeline.

Basics of Planning a Data Migration

At its simplest, the data migration process follows a 3 step flow- extract the data, transform the data, and load the data in a new location - a procedure known as ETL. While there is no one-size-fits-all approach to moving data, there are basic steps organizations can follow to formulate an effective plan.

1. Pre-Migration Planning - Assess the stability of data to be migrated.

2. Project Initiation - Create a communication plan to brief key stakeholders in the organization.

3. Landscape Analysis - Determine the structure and context of data and brief the organization on the goals of the data migration project.

4. Solution Design - Identify what data to move and map out the source to target transformations.

5. Build and Test - Roll out the migration architecture and test it with a mirror of the production environment.

6. Migrate and Validate - Execute and log data migration activities to demonstrate compliance and determine the migrated data's viability for business use.

7. Decommission and Monitor - Sunset the legacy source environment and monitor data quality.

This checklist is by no means comprehensive and there are many more activities that can happen in between each step. But these steps should provide a good starting point for most organizations.

2 Types of Data Migration Strategies

Successful data migrations are driven by a clear strategy, which can vary from company to company. However, most strategies fall into one of two principal categories - trickle migrations and big bang migrations.

Big Bang Migrations

A big bang migration entails conducting the full data migration in a single event within a defined window of time. The draw of this approach is its relative straightforwardness, as everything happens in a brief time-boxed event.

However, the tradeoff is downtime, particularly in the case of systems migrations. Live systems will have to go down as the data is extracted from its source, processed, loaded to the target database, and switched to a new environment. This can put the business under pressure as it operates with mission-critical systems offline.

Trickle Migrations

As the name suggests, a trickle migration takes an incremental approach to moving data. Instead of aiming to finish the transfer in one fell swoop, a trickle migration typically involves running old and new environments alongside each other and moving the data in phases.

The biggest benefit of this strategy is the zero downtime it affords, making it ideal when migrating mission-critical applications that require 24/7 availability. However, the trickle approach adds a layer of complexity to the migration process as it requires monitoring which data has been migrated.

Common Risks and Challenges of Data Migration

Despite the availability of tools, providers, and platforms that make data migration easier and faster than ever, moving mission-critical systems can be a risky undertaking due to these factors.

Data Loss

Data loss is a common issue when moving data. This is often a non-issue with smaller migrations, where the data may not be missed at all or may be restored with a quick backup. However, when moving mission-critical applications and systems, data loss can be a costly mistake.

If the migration is conducted online and a short connection failure occurs, the missing information may only be noticed when a user or application needs it and it is no longer available.

Compatibility Problems

Businesses using outdated IT infrastructure with little to no vendor support can have a difficult migration process. In fact, more than 90% of companies in the UK and US run into problems when migrating legacy applications to the cloud.

Compatibility issues can also happen due to conflicting operating systems, unsupported file formats, and conflicts over user access rights between the source and target systems.

Longer Downtime

Unexpected downtime happens when the data migration process takes longer than predicted. This can pose problems to the company's operations as the source system is usually inactive (at least in the case of big bang migrations).

Poor Execution

Many companies decide to let their IT teams conduct the data migration in-house to save money. However, not all IT professionals are trained in virtualization, software architectures, and cloud security the required specializations in a migration project. When mission-critical processes and applications are on the line, companies would be foolish to cut corners.

Data Migration vs. Data Integration

While data migration and data integration are related terms, they are fundamentally different processes.

Data integration is the process of collecting and combining different data sets from disparate sources to create clean, organized, and actionable business intelligence. The idea is to have a unified view of information, including its meaning, context, and purpose. For example, customer data can be gathered from different business systems like marketing, sales, and accounts to create a single view of a customer.

The similarity to data migration comes from the use of ETL to integrate data. However, data integration is an ongoing process - as more data is collected, the more detailed the business insight becomes. In contrast, data migration is a one-time event (sometimes done in phases) where data stored internally is moved to different systems or environments.

Whatever the case, both data integration and data migration can be simplified and made more efficient with integration software. These tools allow enterprises to combine, manage, and analyze datasets from multiple sources in a single platform. This makes it easier to make business decisions based on the data.

However, with the sheer variety of data integration tools out there, choosing a solution can be tricky. Consider focusing on factors such as-

Pre-Built Connectors - The more systems and applications the data integration tool supports, the more insights it can provide.

Open-Source Architecture - These solutions are more flexible and are free of vendor lock-in periods.

User-Friendliness - While all data integration tools have a learning curve, look for a tool that is easy to understand and visualizes data effectively.

Cloud Support - The tool should support single, multi, and hybrid cloud environments.

Transparent Pricing - Look for providers with transparent pricing models and no hidden fees.

Data migrations are inevitable for most, if not all, enterprises. As more companies realize the performance and efficiency benefits of moving to the cloud, now is as good a time as any to start planning for a data migration of one form or the other.

Business leaders should take the time to assess their IT infrastructure, look for data migration solutions, and bring in the right people for the project.