Behind every business is data, and one of the core elements of a records management program is organizing this data so you can apply retention policies. But data comes in obvious and not-so-obvious forms. Records managers know an effective Records and Information Management (RIM) program must deal with a wide variety of information sources. These might include current and legacy software applications, spreadsheets, homegrown databases, as well as scanned and paper records, just to name a few.
How much effort is involved in searching for information when it’s in so many different places? What happens when the original software is no longer supported and even worse, if it won’t run? Such concerns lead organizations to consolidate systems, looking to simplify and introduce the latest technology that also offers the greatest searching efficiency. Enter…the art of data migration! But what is data migration and what should you look for in a data migration specialist?
As a data migration expert with 15 years of experience, I can tell you that the words “easy” and “simple” should only be whispered, if spoken at all. Data migration boils down to converting data from one source to another. The main goal of data migration is to preserve valuable records. It’s a bonus if you can add data cleanup or reorganization to the process.
Data Migration Caveats
There is no doubt the digital age has created an amazing wealth of tools, but the dark underbelly of this age is the chaos that hides a few clicks away in the millions of boxes, trillions of folders, and bazillions of documents users create and store every day (sure, I know bazillions isn’t a number, but it sounds bigger than big, right?) How do we move this data and keep track of it when there’s so much that goes into data migration? Here are a few important points that will affect your data migration process:
- Cleanup: Data is not perfect and will probably need some cleanup.
- Complexity: Cost and time increase with the number of data sources and their complexity. Combining 15 similar Excel files into one can reduce effort by more than half. Having an internal resource knowledgeable in the source data can drastically speed up analysis and help accurately identify key data components.
- Integration: Blending data from multiple systems requires alignment of characteristics and functionality since you likely can’t rewrite the destination system to operate just like the old one. This requires understanding the destination system to identify what is possible, getting creative when necessary, and knowing what features are optional.
- Verification: Converting data from one format to another can come with unintended data loss when you apply the wrong processes, so be sure to have a good plan in place to verify the data once the migration is complete.
- Alternatives: The costly and resource-heavy alternative to data migration is manual data entry.
Now that you have a basic understanding of what data migration is and what it entails, let’s consider some of the factors involved in migrating large amounts of data.
Structured Data Transfers
The most common types of data sources involved with data migration are structured data sources (e.g. databases and spreadsheets). Structured sources are the easiest sources to manage because they are already organized in some fashion. They have obvious columns of data, structures you can search and update, and may even have some consistent form of data entry. When organizations find such amazing archives, they immediately ask, “Where’s the button to import this spreadsheet I have into this software I use?” The information is already structured, so moving it should be a breeze. Unfortunately, it’s not that simple. Whatever person or application that is responsible for moving the data must:
- Understand your data: Understand the data in the spreadsheet to identify important elements and relationships
- Find patterns: Be able to look for patterns in the data to identify cleanup concerns and compatibility issues with the destination system
- Identify the right location: Know where to put each piece of information in the destination system
- Find what doesn’t belong: Be able to identify unwanted or unnecessary data to exclude
- Think ahead: Know how the information is typically used to determine the best transformation path to match up with the functionality of the destination system
Something that seems like a 5-minute task with a single click of a button now seems a little more complex.
Defining Valuable Records
Defining what a valuable record seems obvious but can lead to an entire analysis on its own. The “data maker” (i.e. you) needs to ask questions like: Is all the data in the record I want migrated important? Do the record’s characteristics meet standards we already follow? Is this duplicating information we already have in another system? Is there enough detail to be valuable?
A database containing information on 20,000 boxes might seem important due to the volume, but what if the only valuable thing in this database is a box identifier that only means something to the person who provided it and no additional descriptive detail to rely on? As a “data maker”, knowing when to hold them and when to fold them isn’t just for poker. It is not in your best interest financially or mentally to carry on a legacy of incomplete or irrelevant data because it’s there. Draw a line in the sand to toss it and start over if that’s what is best. Alternatively, devote resources to fill in the missing pieces to make it useful again. Otherwise, you are wasting time, resources, and storage space. It’s also important that you don’t make the mistake of evaluating the value of data based on the functionality of the legacy system. Moving the same data into a system with more functionality and robust searching options can bring life to dead data.
Data Movers
Now that some of the complexities are a little clearer, what comes next? Engage the right resource either internally or with the appropriate vendor to guide you. A good “data mover” (i.e. IT specialist) will help you analyze the data and its integrity. They have experience with what to look for and are well versed in ensuring data melds as seamlessly as possible with the destination system. Imagine looking at a spreadsheet of 50,000 files to see if you have any duplicate records, inconsistent data entry, or missing information. Would you just scan the file, quickly scrolling by hundreds of records a second, and think it all looks good? Inevitably, we hear convincing declarations that the “data is clean.” The truth is, we are humans and free-form text suffers the most from overconfidence (and a myriad of typos). There are many techniques someone experienced with transforming data can use to quickly find areas that require correction. They might even know how to fix issues automatically. The same efforts might take hours or even weeks to do if you were to comb through the data manually.
There’s a fine line between data cleanup and data massaging however. Cleanup tasks, such as making sure all the box numbers are unique and in a consistent format or checking for valid dates, are important for ensuring data integrity. Trying to find all the spelling mistakes or consolidate the varying abbreviations for the same word is different. Although massaging information can provide value, it is also time consuming and this time increases as the data volume grows. The additional cost and effort may not offer the return on investment you want.
In the end, your “data mover” can provide information to help ensure the process was successful. This might include logs, validation of record counts, directions on how to locate the data in the destination for further review, and recommendations to ensure your user testing is effective.
Conclusion
It’s important to consider these topics when embarking on data migration projects. As with many things in life, you get what you pay for. An inexperienced “data mover” might get the job done, but might also miss issues, take longer to complete the task (or not long enough), not recognize data loss, and not have the forethought to provide alternative ways to efficiently organize data. Plus, it’s important to think about the impact data migration might have on system functionality. An expert can help you avoid these pitfalls. And, if the data you migrate is from a system that not many people understand, having an experienced “data mover” on the job can be invaluable. They can help reverse engineer the structure and transform data from a legacy system no one can use into a valuable resource for your business.
In the end, data migration can save hours, days, or months of error-prone manual data entry and can be a critical component when you invest in a new system. Treat it like you would any project—make sure you’re organized and involve resources that will help you succeed.
If you want to migrate from your legacy records management or records retention system to a cutting edge solution, contact us today.
Disclaimer: The purpose of this post is to provide general education on Information Governance topics. The statements are informational only and do not constitute legal advice. If you have specific questions regarding the application of the law to your business activities, you should seek the advice of your legal counsel.