No two data migration projects are the same, so the methodology used will vary from case to case. However, for Start with Data, the basis of any successful transformation project lies in the application of our four-step model.
At Start with Data, we appoint, at minimum, two key migration figures:
The Data Lead (Managing or Lead Consultant), who guides the overall scope and activities and provides quality assurance for the final deliverables aligned with the initial proposal. This person will also be accountable for delivering the entire data migration strategy,
The Data Architect (Lead or Senior Consultant), who carries out the discovery and analysis tasks, working alongside the key client stakeholders. Their insight and findings also act as the main contribution to the data migration strategy
Below is a broad outline of our four-step model for data migration processes. Our aim is always to ensure that when the new PIM or MDM solution goes live, it is fit for use and fully operational.
Current System inventory
Collating information on current systems which cover the scope of the program, including system platform, extent of data and process information.
Project Scope
Rules for the scope of the data in the identified legacy systems to be migrated. This includes absolute clarity regarding the data volumes and master data and attachments.
Risks and dependencies
Identifying internal and external risks associated with the design and build of the new system ( as well as connected systems) to mitigate the potential impact they may have on the execution of the initial data migration.
Analysis of data models
Analysis of the trading Core and legacy PIM (or storage) system and other sources. This clarifies the extent and complexity of the required transformations.
Quality assessment of input data
Quality assessment is key to gaining complete oversight of the complexity of the cleansing activities required.
Discovery – optional data profiling
Although this step is optional, we recommend its deployment. Start with Data’s consultants assess the quality of the data in scope to gain a deep understanding of the quality challenges within the current data. These profiling results help us to understand the complexity of the cleansing activities needed as a part of the initial phase of data migration.
Profiling the data effectively assumes that groundwork has been laid in the previous time period: a clear definition of the data scope and the data to be profiled provided in format(s) (frequently csv or Excel import) agreed between Start with Data and the client. Start with Data then profiles the data using the appropriate tools.
Defining the migration methodology at a high-level gives us an overview from which we can develop the more granular steps.
big bang vs migration in chunks
The ‘big bang’ approach deploys a single operation to load all data from the old system to the target. Advantages are speed, simplicity, and lower cost. However, all systems need to be down and unavailable during the migration.
Generally, this approach is more suitable for small companies with a lower volume of data, or projects which involve a limited amount of data.
File chunking uses an approach which permits detection of a larger quantity of data, allowing more duplicates be detected and deduplicated to optimise storage.
Then, the following steps are applied:
Once the preliminaries are satisfactorily completed, product data can be profiled, Validated & Cleansed and source data is audited to avoid problems later on.
Regarding data cleansing, a comprehensive review of issues should identify what needs to be resolved. This must be included in the error report. If there are deviations in quality which are addressed in the error report, this is the time to either manually repair the data in the source system or use the automated rules which have been defined by the data owner. To what extent and seriousness such issues exist, you may need more tools and specialist resources to rectify the situation.
The build then moves on to data extraction, where the required is removed from the source system. This stage is followed by data transformation, which involves data loading, source data validation, data conversion and converted data validation.
The product data can now be loaded to the target destination, and once the verification stage has taken place, business stakeholders are in a position to sign off on the build.
The cutover process refers to the rapid transition from one phase of a migration project to another. This needs to be a highly orchestrated effort between the project manager, the system administrator(s), storage administrator(s), DBA(s), and data owner(s). All key stakeholders should be taken through each step of the cutover process item by item, with clear acknowledgement of each step’s completion.
Data migration goes down to a granular level when cleansing and mapping. However, we never lose sight of the bigger picture, deploying the most appropriate and up-to-date migration tools to ensure testing responds directly to the goals of the PIM or MDM implementation project
So, as we’ve seen, the data migration process is necessarily complex, especially given the importance of starting a new era of PIm-powered product data management with the best possible chances of success.
That’s why we at Start with Data offer a PIM Consultancy service which guides and assists you towards the right outcome for your data migration process.
If you would like to find out more about how product data management, PIM and MDM can create value for your business, we’d love to hear from you – Ben Adams, CEO Start with Data