Product Data Migration: Testing

Data migration testing needs to be addressed in the preparatory phase, before moving to the data migration project. Ensuring verification and validation of the product data to be migrated significantly reduces the inherent risk involved.

Pre-migration

Certain data migration testing steps should be taken before entering the main project phases:

Before extracting the source data, establish a clear understanding of the scope of the data: that is, how the extraction will be carried out, and which records, tables, and relationships will be extracted. This means accessing or developing an entity relationship diagram and commonly used data glossary
Make sure all stakeholders have a clear understanding of both the load process and destination. Entity relationships and glossary also come into play here
Ensure the data scheme is fully comprehensible and consistent. This includes mandatory fields, field labels and types, and types of data – these principles apply to both the legacy data source and the new PIM system
Establish clear protocols, procedures, and key actions for the data cleansing phase
Plan contingencies for impacts on any interfaces with other systems (such as ERP or CRM), as well as for 3rd party entities supplying data into the existing and new systems
Test the fields that uniquely link source and target records to make sure there is definitive mapping between the records

Data cleansing

In the data cleansing phase, it is imperative that it is carried out on the current system which houses and maintains product data. The extent of cleansing needed depends largely on the degree of rigor applied to previous operational norms (system development and user input). ‘Dirty’ data will almost certainly be present in the sets to be migrated.

The following cleansing tasks need to be built in as foundational components of a data migration:

Understand error types: these may include blank fields, too-long data lengths, or bad characters
Identify the data checking method using tools such as DQGlobal, Clean & Match or DTM Data Scrubber
Understand the data linkages: changing the data in one table may damage the data in another linked table if the same changes are not made across the entire system

After complete and comprehensive clarification and understanding of the factors listed above, the following step can take place:

Data interrogation of all sets to be migrated, to identify and locate data with the error categories previously established

Post-migration

Common business cases, and user cases can be used to carry out black box testing[1]. Whatever insights are drawn regarding useability can probably be reused in the acceptance testing phase at a later stage.

As well as testing the desired business flows, tests should also perform negative testing to guarantee that the data cleansing is happening at run time in the system:

Consistency between user interface validation and database validation. Lack of consistency is treated as a defect
Checks on data locks: Where data is being written, multiple users should not be able to access the same new record in the database

Conversion testing: when data is recalled from the database(s) it should be displayed accordingly to the user – the criteria are normally fulfilled when covering business cases and use cases

To conclude, it is clear how important it is to carry out specific, careful testing of product data from (before) beginning to end of the data migration process. Failure to do so provokes the risk of damaged migration (incomplete, inconsistent, or corrupt data) and the ensuing substandard user experience. This impacts not only on the system’s reputation for reliability, but also its intrinsic usability.

PIM consultancy service for product data migration

Services around Data Migration form an important part of the suite of Start with Data’s. PIM consultancy services. With our experience and expertise, you can ensure that your migration will occur in a well-planned, consistent and seamless manner.

Find out more

If you would like to find out more about how product data management, PIM and MDM can create value for your business, we’d love to hear from you – Ben Adams, CEO Start with Data

Case Study

“Start with Data are helping transform product data management, laying scalable technology and data governance foundations”

Read about it