Data Migration

The migration of legacy product data into new or existing PIMs

Data Migration can be a complex process. Start with Data has a proven method for product data migration with a blueprint approach to migrating legacy product data into new or existing PIMs.

What is data migration?

Data migration is the process of transferring data from one digital storage system to another, which involves defining the scope, profiling, selecting, preparing, extracting, transforming and loading data. 

It is important to highlight the differences between data migrationdata integration and data conversion. At the start of the data migration project, data may need to be modified or transformed, which is known as data conversion. The process of combining data from different sources into a single, unified view refers to data integration. As such, conversion and integration form only a sub-category of tasks within a data migration project.

What is a Data Migration Strategy?

Data migration strategy defines the approach for migrating the data managed in the existing solution to the new one. A data migration strategy prevents a substandard migration project from causing more problems than it solves (such as missing deadlines or going over budget).

The Data Migration Process

The broad industry consensus identifies three phases of activities in the data migration process, we extend that to a fourth phase, given the importance of getting the preparation and planning right before embarking on the activity itself. At the bottom of this page, you will find a breakdown of our stages. 

1. Initiation

2. Planning and Designing

In the initiation phase, we document the strategy, scope and governance of the data migration. 

  

  • Requirements – Current System inventory & Migration Scope
  • Design – Analysis of Source & Target Data Model, Data Mapping & Data Templates, Planning for Execution Steps, Tool Selection
  • Product data governance maturity assessment: benchmarking across access, ownership, quality

3. Build

The build phase comprises the activities required to transfer the source data from existing systems to new systems

  • Profiling, Validation & Cleansing – required to gather the knowledge of the data in question; source data must be audited to avoid problems at a later stage.
  • Cleansing – All issues identified need to be resolved and must be included in the error report. All data quality deviations described in the report must be addressed by fixing the data in the source system or by the automated rules defined by the Data Owner. Depending on the scale and extent of the issues, this may require further tools and specialist resources.
  • Data Extraction – Extract required data from source system.
  • Data Transformation – This stage involves data loading, source data validation, data conversion and converted data validation
  • Data Loading – to target model
  • Verification – Data verification, sign off from business stakeholders

4. Cutover

Activities required to close off the data migration projects – the final load and validation. The final iteration

  • Data Archival
  • Data Cleanup in the old system

Bear in mind how much budget you can allocate to the migration process. Using the services of a data migration consultancy is key, as strategy-planning with them sets you up for a successful and problem-free migration. Additionally, ensure you consult closely with those who rely on the data – the end users. Successful data migration will increase user adoption of the new PIM solution.

Best Practices for Data Migration

The main problems with data migration projects tend to occur in the delivery phase. These problems are often traced back to a lack of due attention to the business engagement strategy or an inadequate key data stakeholder analysis. Adopting best practices helps guide a robust strategy towards limiting problems.

Analysis

This area covers the scope, complexity of source data, exhaustive documentation, governance implications and the impact of the data migration project on your business. Carrying out a pre-migration impact assessment is recommended, as it will identify the dependency between data migration and the new system implementation.

Data Standards

To be fit for purpose in your target application data requires comprehensive verification, consolidation and cleansing. Whether implementing new functionalities or consolidating business functions, data quality is key to your migration project.

Migration Design

The migration team requires an infrastructure enabling them to alter access and mapping components rapidly. Ideally, data and organisational requirements should have as little impact on operational business processes as possible.

Business rules (current and future)

Migration is a multi-agency project, so it is crucial to implement a set of business rules. For example, defining the time frame - all stakeholders know exactly how much time is taken on certain tasks and any cost implications.

Data governance rules

Rules and procedures are needed to track and report on data quality because this informs data integrity. Governance and oversight means having a clear chain of command and set of accountabilities.

Data quality assessment

Data quality assessment must start with a comprehensive audit of the source data. Ongoing data quality tracking is an essential design element, as are risk mitigation procedures.

Risk management and mitigation

Ensure you have access to the expertise which can guarantee the stability of the method you are using, especially relating to data integrity and knowledge of target system attributes. This avoids potential pitfalls.

Tool selection

Use a consultant to identify certain key criteria – areas like design, presentation, pricing and performance (see below).

Data migration testing, reconciliation and validation

Reconciliation means comparing target and source data to ensure the migration architecture has transferred data correctly. This guarantees usability, compatibility and integrity. Testing procedures for reconciling and validating include full or partial migration testing and acceptance testing at destination.

Data migration tools

As with any good product, you need a tool with features fit for purpose; flexibility, scalability, ease of use for non-experts (with minimal technical expertise) and intuition in its interface and suite of functionalities. There’s no one tool that fits all for migration projects. Depending on the client’s application landscape, IRM recommendations, source and target capabilities, data quality scope – the right choice should be made.

Start with Data’s partner in data tools is Conemis, a provider whose innovative approach, leveraging the latest developments in AI, has proved to be highly fit for purpose in the migration projects we have managed. To give you more context, let’s look at the types of attributes you should be looking for in a high-quality data migration tool 

What to look for in the right tool

Straightforward Data Mapping tools like a code-free, drag-and-drop, graphics-oriented user interface.

Data Integration and Transformation Capabilities which can restructure data for targeted delivery.

Enhanced Connectivity enabling seamless connection with various source and destination structures.

Automated workflow orchestration and job scheduling to streamline data processing.

Data analysis, cleansing and deduplication capabilities to verify and improve data quality prior loading data to the target system.

Data enrichment to support additional, clean content for the initial data migration, for example address data enrichment, validation or enrichment of the company information with D&B.

Recommended tools based on the source or target systems to benefit from predefined workflows, connectors, mappings.

This is not to omit hosting, user limitations, range of functionalities, automated processes, scalability and customer support services. 

Data migration challenges

A poorly executed data migration process leaves your organisation unresponsive to a rapidly evolving business ecosystem, with a fragmented and inconsistent set of tools.

Lack of Source system information (Docs, SME, access)

A landscape analysis provides an overview of the source and target systems. Without it, the project team cannot understand how each system works and how the data is structured within systems.

Inappropriate collaboration

Involving both business and IT from the outset is essential to engender project ownership. Handling management and technical aspects requires expertise, so ensure experts are available, not only to navigate the disparate data sources, but to guide the result towards users’ needs.

Lack of expertise

Assess levels of expertise in your migration team. A lack of knowledge about delivery expectations or a mismatch between people’s skills levels and the expertise required delays progress. Factor training and education into the planning phase.

IT scope creep

‘Grey’ areas of responsibility and accountability, may result in the IT department becoming the de facto project driver. Business engagement fails if IT specialists co-opt the process when enabled by a lack of leadership from the user teams.

Lack of business engagement (senior stakeholders and further down)

This has ramifications beyond the scope of the project, impacting on psychological and emotional factors connected to values like buy-in, commitment and support. This significantly affects stakeholder morale.

Lack of requirements (missing or lost)

Without clear ownership responsibility for data quality standards, the project lacks key information. Failure to correct, cleanse or enhance data makes it potentially unusable. Migrating data ‘as is’ is inherently risky when validating and reconciling data in the new system.

Validation failure (or lack of validation)

Failure to embed compliance procedures for checking cleansed data means the migration team cannot prove that source data matches that moved to the target application.

Late or delayed reconciliation

Errors in data mapping, missing or duplicated records and broken relationships across tables are common problems. They have a serious impact on data accuracy without robust and timely reconciliation.

Data quality issues in the source system

These occur when existing data quality is either poor or unknown and there is insufficient functions or standards documentation about data quality.

Ignoring the data quality aspect in the target

Data quality factors are often ignored in the target systems (the “let’s fix it later” syndrome) leading to data integrity problems and lack of user adoption.

Lack of coordination with overarching project

Data migration projects are often parts of larger digital transformation programmes. They must be closely aligned with the overarching project to ensure risk identification and mitigation, proper planning and timely execution.

Data Migration Consultancy

Start with Data has a proven method for product data migration with a blueprint approach to migrating legacy product data into new or existing PIMs. 

Our proven method for product data migration

Our 4 Phase Data Migration Approach

Initiation

Phase 1

Plan & Design

Phase 2

Build & Test

Phase 3*

Cut Over

Phase 4*

Find out more

If you would like to find out more about how product data management, PIM and MDM can create value for your business, we’d love to hear from you – Ben Adams, CEO Start with Data

Case Study

“Start with Data are helping transform product data management, laying scalable technology and data governance foundations”