Planning Data Migration the Right Way

What is data migration, really?

Let’s start with a couple basics. (I’ll be brief and get to the good stuff fast.) Data migration is the process of transferring information from one system to a new system. This can include transferring data between different file formats and different storage types. The term “data migration” is actually quite broad and covers a whole gamut of types of data, activities, and systems. Examples include:

• Migrating a file system or storage repository from on-premises to a company data center
• Migrating to the cloud
• Server upgrades
• Server consolidations
• Migrating to a document management system (DMS) or Enterprise Content Management (ECM)
• Implementing SharePoint (or increasing the use thereof)
• Implementing OpenText
• Transitioning from Novell Operating System to Windows
• Implementing Network Attached Storage (NAS) or a Storage Area Network (SAN)
• Remapping file paths into archives
• Folder restructuring
• Implementing Distributed File System (DFS).

So, how do we form a complete plan for data migration? It begins with understanding the need for data migration, creating a data migration project plan template, executing the project, and then conducting a data migration review.

What triggers a data migration?

Data migration standards that justify the transfer of files onto a new database include: • Routine upgrades of hardware and/or software. • The legacy system does not have enough storage capacity. • The business has been acquired by another company. • The business or government agency has merged with another entity that uses different software and/or different file naming conventions. • The organization splits into two or more organizations or units, each with a new name and requiring new file paths and filenames. • Someone orders all data storage moved off premises. The difficulty level of a data migration varies widely. The amount of data, the formats it’s stored in and the number and types of systems affected can all play a role. Whether you’re transferring a single database or undertaking a large-scale data migration project, it’s not just “important” to be prepared. Serious problems including downtime, lost income, extra work, and upset end-users can result from failing to adequately plan (not to mention lost weekends). The purpose of this data center relocation checklist is to help you avoid some of these perils.

Part 1: Planning Your Data Migration

Before you begin transferring data, take some time to determine exactly what data to transfer and what systems it affects. (Okay, this is kinda obvious. But, hey, it wouldn’t be a checklist if it didn’t start from the beginning. We’ll get to some lesser-known actions — and “gotchas” — before we are done.) Data migration is very strategic, so during the strategy development, assessment, and analysis stages, figure out what type of data you need to transfer, which systems use this data, and what other files the data links to. Data migration plan samples could give you some insight on how to structure your own plan, and are worth glancing over. Note: I meant it when I said, “and what other files the data links to.” On first glance, this may appear to be just innocuous advice. But in the real world, this has actually been one of the “gotchas” that many IT pros have failed to account for, sometimes to their embarrassment. It’s easy to forget just how connected data is to other data. Users all across your organization are connecting (linking) data into their files all the time. They think nothing of it. And they don’t ever mention it to anyone in IT. If you change one thing, say a folder name, for example, you could be affecting other data somewhere else your system in ways you hadn’t thought about yet. More on this will follow. Other questions you might want to ask yourself include:

• How much data is there to migrate?
• How long do I estimate the project will take?
• What technology will this project require?
• What format is the data in, and will it need alteration for the new system?
• Do I have any duplicate or obsolete content that I can omit from the migration?

Dated or redundant files frequently occur when reviewing data from old systems. So take the extra time during this stage to weed out this content. This will speed up the migration process by lightening the load of data that will need transferring. Establishing a time frame using accurate means of assessment and having the adequate tools you need are also important in pre-migration planning.

Have you checked your network for any obstacles to data transfer?

If you’re transferring data over a network, it’s important to have the most bandwidth you can. Check for firewalls that might delay data transfer. Firewalls are responsible for a large quantity of low-speed data flows instead of only a few high-speed flows. When attempting to send data through a firewall that’s faster than its internal processor can handle, the firewall has trouble handling those bursts of data. This can result in packet loss. Packet loss occurs when data traveling through your network fails to reach its destination and can create serious performance issues. In addition to adjusting your firewall to allow for full utilization of the bandwidth you have available, consider transferring data when there are fewer users — or even better, no users — online. Schedule data migration for overnight if possible. If you must share a network during data migration, consider throttling your usage to prevent slowing down the network for other users.

Have you developed a map of what data needs to be migrated?

Before beginning your data migration project, develop a map of what data needs to be migrated. Develop a data migration plan example, and then set up your map to fit that template.  Your map should include where your files are and what they’re connected to. To maintain link integrity during data migration, include parent/child files in your map. This will help you make sure that once the transfer is complete, employees aren’t stuck dealing with broken links. If you are not yet 100% familiar with what is meant by “parent file” and “child file”, click here to get a brief definition. Structure your data mapping specifications to encompass the critical attributes of each piece of information that transfers through your data migration activities. Use the following elements to form a data migration template:

• A list of all details for the original source of data.
• Connections from the original data to their corresponding targets, providing any relevant information from the data directory.
• Any rules or stipulations for data that may need to be manipulated to move correctly through the network. Settings of interest could include default and mapping values as well as fields that need to be combined.

Data migration mapping templates can also be found online and used to better visualize how to set up your own. They are most commonly done in spreadsheet software such as Microsoft Excel or Google Sheets. Traditional data mapping techniques can be supplemented with software such as our link preservation and data mapping tool that allows for easy transfer or modification of files without the risk of the file becoming broken or corrupted. Bonus Tip: Print out your map before you begin migrating your data. Yes, it’s old-school, but it’s worth having a paper backup before you begin the process. Plus, sometimes you can get more of a broad view when you lay something out on paper on your desk. This can help you catch errors.

Have you done a landscape analysis to determine what systems the migration will affect?

Data migration usually focuses on the core and target systems. However, these two systems are rarely the only ones affected. Prevent broken links during migration by doing a landscape analysis. Landscape analysis is mostly research done by an organization’s competition and industry in general. Doing this analysis is advantageous to your company since it helps you:

• Determine the best strategies and moves to make for the greatest chance of success
• Catch any design flaws before they become a nuisance
• Gain a competitive advantage over any competition
• Receive feedback on projects already underway

This will help you map out what other systems use the data you’ll be migrating, and help you see how the data migration could influence them. This is where LinkFixer Advanced comes into play. It has a feature called “Inoculate” that automatically creates associations (sort of like a secondary means of linking) between each parent file and all its child files before they’re migrated. After migration, the “Cure” function acts as a batch re-link operation. A tool that automatically handles links during your data migration can save you thousands of hours.

Have you identified software necessary to complete the migration?

Once you’ve identified the systems, amount and type of data you’ll be migrating, you’ll need to choose software to help you move the data to the target system. There is a range of different data migration software available. However, it’s a good idea to pick something specific to the type of data you’ll be migrating. There are different types of data migration. The purest form is database migration, which is just a matter of moving data from one database to another. However, you also can use data migration for applications such as a company changing customer relationship management (CRM) software. Business processes can also be migrated to better align with current operations. Storage devices are also commonly migrated onto new technology for easier accessibility. Specific processes take place while converting data that differ from one another as well. Exporting and importing is one type of process, and to expand on the example used in the paragraph above, could include uploading a list of client information from Excel to a CRM software which requires the data to be in a neutral format. Scripts prepare data and move things around to fit a new data model better. When the data set is too large for a simple script, extract, transform, and load (ETL) tools are preferred, and can automate many elements of the data conversion project plan. The best data migration software depends on what type of data migration you are trying to achieve, and which processes you wish to be assisted. Data migration managers, NAS virtualization software, and storage area networks all have their applications. Do your research to determine what best suits your needs and which are the best tools you can use for the job.

Imagine the “bad” things

I wrote the following in an article in 2015: “Most of your failed data migration projects (heck, most failed IT projects of any kind) fail due to lack of imagination in the planning stage. Notice that I didn’t say ‘lack of planning.’ (I wouldn’t insult you like that.) But there is an insufficient amount of imagining what the various bad outcomes of various actions might be so that you can prevent the bad things from happening, or at least be ready when they do.” This is still true today. Take a moment and do this exercise. It could save you untold heartache. Mistakes that IT professionals typically face include the following:

• Underestimating the time the project will take to complete
• Trying to transfer too much data or alter a large amount of code at once
• Migrating data that is outdated and no longer needed
• Failing to take advantage of data migration tools
• Forgetting to conduct baseline tests to evaluate results after migrating is complete
• Failing to institute a rollback plan in case a new version faces instability
• Understanding how to perform proper validating testing and implementing those techniques into the data conversion project plan

While sometimes data migration fails because of unavoidable reasons, more often than not it is due to human error and can easily be prevented by foreseeing common mistakes and using the correct techniques and software to allow the process to go as smoothly as possible.

Part 2: Figuring Out How to Transfer Data

Your next step in the planning process is to figure out how to transfer data. Test your hypotheses early and often during this stage. Corruption and loss of data are two of the biggest risks in any migration project. Your data transfer method should help to minimize data loss. Good test cases can also cut down on the need for time-consuming manual validation. Preventing file loss and data corruption can be achieved by backing up your files on an online cloud or solid state drive (SSD). Back-ups should be done habitually at a fixed time, rather than waiting for a data migration project to be underway. Viruses, theft, and destruction of hardware all take place commonly, so regularly backing up of your files can help dampen data migration risks and safeguard your files in general.

Have you accessed your data?

If you don’t already have the full access you need, get access to your data early in the data migration process. This will save you a lot of grief down the road. Check out individual files. See what types of security permissions they have. Will you need to change these permissions during the transfer? What types of security permissions will they need in the new databases? Are there are any quirks in how the database is used that you’ll need to account for? Actually looking at your data will let you test out these hypotheses.

Do you have a plan for validating the data transfer?

Validating the data transfer is a crucial step of a data migration; after all, corrupt or lost data is often a costly problem. Planning for data validation from the get-go is the way to go. There are three types of data validation possible:

• Validating random samples
• Validating subsets of data
• Validating the complete data set

In an ideal world, it would be possible to validate the complete data set. But in the real world — do you know any IT professionals with the time and resources to validate a complete data set? Consider validating random samples or subsets of the data to save time and sanity. Random sampling is advantageous in that it leads to a low bias of model performance, but can also lead to subsets that do not encompass all the data properly and provide high variance, resulting in an inaccurate test. Validating certain subsets of data offer a better generalization of all the types of data in the transfer. It is just important that you split the data properly to get a confident analysis.

Do you have a recovery plan for data that didn’t transfer properly?

Despite your planning and testing, some of the data might not transfer properly. This can include minimal issues like using the wrong delimiter, or the core and target columns using different units of measurement. But it can also include major issues like data loss and corruption. Two issues you should consider now include:

• Semantic data and file naming conventions: It’s common for departments to develop their own file naming conventions, or to use database fields in unusual ways. This usually isn’t a problem. But when you’re transferring data to a new system, it’s important to understand your users’ idioms. If you’re transferring data used by a particular department, consult with them now to make sure that you understand any unusual ways that they input information. Consider getting employees from these departments to visually validate samples of their data.

• Data loss and corruption: During the planning process, you should also consider what to do if data doesn’t transfer. This is a situation where prevention is the best cure. Will you be able to access a backup of any corrupted data? What can you do to prevent data corruption from occurring? Consider things like cleaning and deduplicating data before starting the transfer. You can also use tools like LinkFixer Advanced to inoculate (protect) links before the transfer.

 

Part 3: Beginning the Data Transfer: Preparing Data for a Migration

Whether it’s duplicates, bad links or outdated formats, most data has at least a few issues. Clean up your data before you transfer it. This will cut down on the amount of data you’ll need to transfer, as well as the amount that you’ll need to validate after it’s done.

Have you cleaned the data you’ll be transferring?

Cleaning data before migrating it is significantly easier than cleaning it after. It can also cut down on problems with corruption and data loss in the target system. Extract, clean and deduplicate data before you transfer it. Fix any broken links and file type transformations in this stage.

Have you checked to see whether any data should be archived instead of transferred?

Beginning a data migration process is a good time to review your data archiving procedures. You may not want to transfer all of your data. Some of it may be more suitable for archiving. This has the benefit of cutting down on work for you. Review or spot-check some of the data involved with the applicable department heads who use the particular files in question. If your organization has an Information Management (IM) department or group, you definitely should check with them as well. They will be very interested in what you are about to do. I can assure you of that. Be careful not to overreact, though. Although it’s tempting to archive data instead of transferring it, be realistic about what types of data need to be accessed regularly. Tip: Survey some of your key end users and your IM section to see what data they would be okay with archiving. They may save you a lot of time from having to restore something they regularly need. Conversely, they may also reveal some additional data that can be archived that you hadn’t considered. Have you tested the new hardware? One of the biggest mistakes in data migration is not testing the new hardware before you transfer data to it. Since most people transfer data from a smaller, outdated system to a newer, bigger system, they don’t bother to test the hardware. Unfortunately, this means that people complete a data migration, only to have the hardware break shortly after. Don’t be one of those people. Test your new hardware thoroughly before transferring data onto it.

Part 4: Data Security Concerns

Protecting data is a growing concern for executives, financial officers, information managers, and public relations people. When you take on a data migration project, expect that many stakeholders will have concerns about security. Addressing data security concerns early in the process can get your project critical support from these groups. Be sure to obtain formal agreements from any governance you need to reach out to before the project’s start date. Ignoring protocol and security procedures can delay the project, or worse, get you directly in trouble.

 Have you identified relevant stakeholders from data security or privacy?

There aren’t many ways to stop a data migration project quicker than ignoring security. Security can be compromised in several different ways while preparing for data migration. When setting up systems and installing software, all those who are working on the project usually gain administrative rights. However, this larger pool of employees who have access to these privileges increases the likelihood of malicious actions occurring, whether intentional or unintentional. If you bring in outside help to assist with an extensive database migration project, whoever is lending their aid on the project now has access to your company. They have inside knowledge and can access systems they shouldn’t be authorized to enter, or execute tasks that do not align with standard procedure. Make sure outsourced help is familiar with the foundation of your company. It’s also worth noting that you should be wary of what custom software and utilities are used throughout your data migration activities. Although they might do their job efficiently and save you time, overlooking specific security issues during the design of the application isn’t uncommon. Our software designers at Linktek are committed to ensuring that the products we offer to our customers are exemplary in doing the proposed function they are meant to help automate while not compromising any security measures. Find out now who’s concerned with data security and data privacy issues at your business, and figure out how to address their concerns. Along with the IT department, consider the following groups:

• Human Resources departments
• Accounting departments
• Legal departments
• Chief Financial Officers (CFO)
• Chief Information Officers (CIO)
• Third party software contractors
• Information Management (IM) personnel

Identify their specific concerns and the data affected by these issues.

Have you vetted the software and personnel involved in the data migration?

Many of the data privacy risks that a company faces come from careless, uninformed, or just plain disgruntled employees. That’s why it’s important to vet any personnel who will play a role in the migration. Determine the goals and responsibilities that each member of the IT team has, complete with milestones and time frames that different aspects of the job should be completed by. Do this by illustrating your team’s goals with some form of data migration plan template, documents, and PowerPoints for them to look over. This is also a good time to review who has what types of permissions to access confidential data. It’s far too common to have accounts that are either not in use (or belong to ex-employees) retain access to restricted files. Remove these permissions before beginning a data transfer to cut down on the risk of data breaches. By the same token, it’s important to vet the software you use for a data transfer. Don’t just pick up any open-source file transfer software! Make sure the software you use is legitimate and well-regarded.

Have you reviewed industry statutes to ensure that you’re conforming to best practices?

If you’re undertaking a data migration project in a specialized industry, you may have to adhere to specific requirements governing how data is protected during transfer. This is particularly true of healthcare and financial organizations. Check to make sure that any software and migration methods you use meet these standards. Industry best practices are also a good way to get management on board with the data migration.

Part 5: Getting Leaders to Buy-in to Your Data Migration

Getting business leaders to buy into a data migration project isn’t always easy. Neither executives nor users always understand the benefits of migrating data to a new format or system. More than a few IT professionals have found themselves struggling to get funding and personnel for data migration. Be sure to fully understand the reasons that data migration is necessary, and know how to communicate those reasons clearly to someone who may have less experience in the IT field. Look at your goals for the project from a business standpoint and let that shape the rest of your decision making. Use empathy, think like an executive, and try to understand what benefits they would care about.

Have you set realistic deadlines and deliverables?

More than half of all data migration projects go over the deadline. Therefore, it’s important to be upfront about realistic deadlines for your project. It can help to group the data by functional areas. Your data map can help you to identify these areas and schedule realistic deadlines. Be sure to allot more time for resource-intensive areas. If you’ve identified areas where there are variations in how the system is used, consider allotting extra time for testing and transferring data. One way to increase buy-in from executives and users is to schedule a few wins early in the data migration project. When users can see the benefits of the project, they’re more likely to support it. If you’ve identified an area that could see immediate benefits, consider pushing it to the front of the data transfer project. Look for low-hanging fruit to use as your “convincer”. For example, if your organization has an engineering department that has frequently complained of slow processing speed, and you estimate that after the move to those two super-fast servers these engineers are going to see a major speed and performance increase on their Computer-Aided-Design (CAD) files, consider moving them first and using the results as your “show car”.

Do you have an adequate budget for this project?

We know this doesn’t always happen, but it’s good to know how much a data migration project will cost up front! It can help to get leadership buy-in if you align the data migration with other company goals, like increased productivity or cost cutting. To get people to support the needs of the data migration, focus on how the new data format will help the company. If you’ll be saving money by turning off an old database or server, let people know.

Do you have the right personnel for this project?

Think beyond the IT department when you consider personnel. Try to get subject experts who can help to confirm whether data has transferred over properly. It can also be helpful to find real users to test the migrated data. These real users are your best gauge of whether the new database or system is working properly. If you’re in a large office, consider delegating this to department heads.

Do you have company buy-in?

For many business owners or organization leaders, data migration isn’t the most exciting topic (though they will suddenly get real “excited” if something ends up missing or messed up). Particularly for longer migrations or more technical topics, it’s important to get prior buy-in from leaders. One of the best ways to do this is to focus on the most important outcome: They will lose access to data that isn’t migrated. (This should get their attention.) Making sure that they still have access to their data is often enough to get users on board. To repeat, one way to get the leaders on board with migrating is to build-in some short-term wins up front in your project. These will let leaders see the benefits of the migration early on, and help keep them a bit more patient throughout the slower portions of the project.

Part 6: Planning for After the Deed is Done

After wrapping up your project, there are a few more tasks you should take care of and monitor for the foreseeable future.

Have you planned for decommissioning the old server or database?

Decommissioning the old hardware can get executives and users on board in two ways. First, phasing out the old program/hardware can be a cost-saver. Most executives are easy to get on board when there is a savings involved. Second, the fact that the old hardware will be turned off means users will no longer have access to data on the old hardware. This can help users understand that they will need to access data in a new format or via a new program. This can also make it easier to get users on board with the process of transferring data. Do you have a maintenance plan in place? The work’s not done simply because the data’s been migrated. You’ll also need to make sure that the data is maintained regularly. Some people identify employees in the relevant departments to maintain data, while others charge someone within the IT department with the responsibility of maintaining it. We’ve designed LinkFixer Advanced for both data migration and maintenance. You can even use it to repair links, in batch, that were broken during an earlier data transfer. Getting started with LinkFixer Advanced is easy. Download your free trial today!

---

Ed Clark

LinkTek COO View Bio