Data is the most valuable asset in the world right now according to a report published by The Economist in 2017. When handling data, there are times when you may need to transfer it between different storage systems or applications. A lot can go wrong when moving huge volumes of data from one storage system or application to another.
That is why it is important to be extra careful while doing any form of data migration. There are procedures you need to follow to ensure the successful migration of your valuable data. In this article, I will discuss everything you need to know about data migration and the best strategy and practices you need to follow when doing the migration. Without any further ado, let’s jump right in!
What is data migration?
Data migration is the process of transferring data from one location, application, or storage system to another. The whole process of data migration involves selecting, preparing, extracting, and transforming the data before the actual migration is implemented.
There are a number of reasons why an organization may want to undertake data migration. Some of the common reasons for data migration include the need to overhaul a system, upgrade databases, and establish a new data warehouse. No matter the reason for the migration, there is a need to follow the best practices to avoid data loss or wasting unnecessary resources when undertaking data migrations.
According to Gartner, over 50% of data migration projects go beyond their pre-determined budgets. This is usually due to the lack of proper planning and ignoring some of the best data migration practices that have been proven to work over time. One may assume data migration simply involves copying and pasting data from one location to another.
However, it is a complex endeavour that involves several processes and following certain guidelines especially if it involves databases for mission-critical applications and systems. Having a well-defined and planned data migration strategy is very crucial to reducing or eliminating the negative impacts that could happen during the process.
Data migrations strategies
There are three common strategies used for data migration; big bang data migration, trickle data migration, and zero downtime data migration. Let me explain each of these in detail.
1. Big Bang data migration
With this strategy, all the data is transferred from one database or storage location to another at once within a fixed period. This strategy is usually implemented when there is a scheduled downtime for the databases that are involved in the migration. One of the major benefits of using this strategy is its simplicity as it doesn’t involve a lot of processes and preparation when compared to the other strategies.
It is also less costly to implement the big bang data migration strategy since it happens within a short time and doesn’t require a lot of expertise. However, this strategy is not ideal for applications and systems that are required to be up and running at all times. Organizations that run their systems 24/7 cannot afford to have any downtime scheduled for data migration unless they have backup databases that they can switch to during the migration period.
Another downside to using this strategy is that the cost of failure can be very high since all the data is migrated in a single operation. If you are to use this method, there is a need to be extra careful and also follow the right procedures to avoid or eliminate any chances of failure during the process.
2. Trickle Data Migration
This data migration strategy uses the agile approach which involves breaking down the process into small sub-migrations that are implemented separately. Each of the sub-migrations is well-defined with its own scope, goals, and deadline. The main benefit of using this method is that the cost of failure is not as high when compared to migrating all the data in a single operation.
The data migration team will also find it easier to implement and confirm the success of each sub-migration. If one of the sub-migration fails, the lessons learnt from its failure can be used while implementing future migrations to avoid making similar mistakes. Trickle data migration also requires less downtime. Sometimes trickle data migration requires zero downtime.
This creates less to no impact for organizations that require their systems to be up and running at all times. The downside to using this strategy is that it is more expensive since it requires both systems to be up and running during the migration. It is also much more complex and trickier, thanks to the complicated syncing issues involved.
3. Zero downtime data migration
With this strategy, the goal is to ensure there is no downtime during the data migration period. This requires replicating the source database to a target database to ensure the client can access and operate by using the data on the source database as the migration is happening. As you might have guessed, this is the most expensive strategy since it adds one more database to the mix.
However, the main benefit of using this data migration strategy is less business distraction and faster migration times. Zero downtime database migration also allows fallback capability in case the process is not successful. It also doesn’t put a lot of burden and responsibility on the data migration team since there is some room for error.
So, despite being more expensive than the other strategies, some organizations are willing to incur the extra cost because of the benefits the zero downtime data migration strategy has over the previous strategies. If your organization runs mission-critical systems that have to be up and running at all times, using zero-downtime database migration is a worthy investment.
Which data migration strategy should you choose?
There are a couple of things that you need to consider when choosing between the three strategies. Some of these considerations include;
- Flexibility to create downtime: If you can afford to pause the systems relying on the data being transferred, then using the big bang strategy is not a problem. However, for those dealing with systems that have to be running at all times, zero downtime or trickle database migration is the ideal choice.
- Scope and migration deadline: If there is a clear scope and deadline for migrating the data, using the big bang or zero downtime data migration strategy would be the best alternative. If the scope and deadline for migration are not strict or clearly defined, then using the trickle data migration strategy should be the way to go.
- Expertise and experience of the data migration team: The team implementing the data migration should be well conversant with the procedures involved in the strategy you intend to use. The team’s competence and expertise have a significant impact on the overall success of the data migration process.
Best practices for data migration
No matter the data migration strategy you choose, its success will significantly depend on how well the data migration team follows the well-known procedures and best practices. Here are some of the best practices that should be considered to ensure successful data migration.
1. Conduct an impact study
Conducting an impact study is one of the best strategies you should implement before undertaking the data migration process. An impact study involves assessing all the processing and determining the possible consequences if anything goes wrong. A 2017 study by Experian found out that only 36% of organizations that participated in this study conducted an impact study as part of their data migration process.
Conducting an impact study might require incurring extra costs, but it will help you plan for any negative consequences that data migration could cause. With all details of what could happen if things go wrong, the migration team can put in place plan B strategies to limit the impact of any failures in the process.
2. Determine the scope of the project
One of the elements you should include in your plan is the scope of the project. The scope includes all the data that is to be migrated from the current data location to a new one. Having a clear idea of the data to be migrated will make it easier to determine the best migration strategy to use.
3. Create and implement a clear plan and strategy
Another best practice for data migration is setting up a clear plan that the team should follow when implementing every activity in the data migration process. The plan should be formalized and written with clarification of why and how the data migration will be implemented and who is responsible for doing what.
The strategy should include the high-risk areas and the obstacles the team might face during the different stages of the data migration process. Your plan should also be shared with all the relevant stakeholders that will be involved in the process. These stakeholders should be given the chance to give feedback about the strategy.
4. Have a data backup plan
Failures are pretty common during data migration, so it is important to have to fall back plan in case things don’t go your way. Having the data backed up on an independent storage location will come in handy if files get corrupted or some portions of your data go astray during transfer. So, your data migration process should involve backing up data as one of the stages.
It is also important to have a clear map of how the data will be moving. This will allow your team to know the source of the data, where it’s going, and how and when it’s migrating. Having these details makes it easier to trace back the exact cause of failure in case it occurs.
5. Have a well-aligned and trained team
No matter how well you plan and strategize, your data migration process may not be a success if the people implementing it don’t know what to do and how. That is why it is crucial to involve the team that will be doing the actual hands-on work while drafting the data migration plan. The team needs to be well-conversant with all the processes.
All knowledge gaps within the team should be dealt with before embarking on the data migration process. You might also have to involve some external experts to help guide your team on how to go about the different stages of this process. Of course, involving experts or consultants in the process is costly. However, it is a worthy investment because it minimizes the chances of failure.
6. Have a migration time frame
Before embarking on the process, it is important to specify the time it will take for all stages of the migration to be completed. The time frame will largely rely on the size and complexity of the data plus the computing capabilities of the servers involved in the data migration process.
Your estimations of the time frame should largely be based on past data if this is not your first data migration project. If you are implementing your data migration process for the first time, you will need to get expert advice when it comes to determining the timeframe for the different stages of the migration.
7. Post-migration validation and testing
Your data migration team needs to ensure all the data has been correctly and completely replicated to the new database before destroying the data in the old database.
Final thoughts
We have covered the strategies and the best practices you must implement to reduce or possibly eliminate chances of failure during data migration. Before you embark on your next data migration project, it is important to put these practices into consideration to avoid being the next victim of data migration failures.
