When a utility company conducts a data migration project, quality assurance and quality control are of the utmost importance. These projects can consist of migrating to a different GIS platform, a different data model within the existing GIS platform, or integrating new data into the GIS as a result of acquisitions or company consolidation. Regardless of the type of migration, QA/QC is needed. Usually, a combination of automated tests and visual checks are performed. This post will cover the automated tools that SSP Innovations created to provide a client quality assurance in their Smallworld to Esri migration.

A client hired SSP to perform QA/QC services of their migration of gas distribution GIS data into Esri’s UPDM data model. This is a large gas provider with over 3 million customers, in multiple states. Their GIS is mature and robust, meaning there is a lot of data to migrate!

Equally, robust automated QA tools were necessary for this job. Rich Zamorski, a Senior Software Engineer at SSP developed them. He has been a part of many data migration projects over the years and knows what is needed to make useful and flexible QA tools. The tools are named Quality Checker. The initial tests were ran and summarized by SSP, and then the Quality Checker software was delivered to the client, so they can continue with the QA themselves beyond what the scope of work originally involved.

Quality Checker Details

Quality Checker compares Esri databases in either (or a mix of) the File Geodatabase or Enterprise SDE formats. For the latest project, we compared a File Geodatabase of the Smallworld data that they created in-house to the Enterprise SDE Geodatabase that the migration software was writing to.

There are 3 types of checks made:

• Key Tests: Compare the Unique ID fields for every feature in the configuration from the source to target data. This identifies data that was added or deleted during the migration process. It will also identify any duplicates.
• Spatial Tests: Validates the graphical components of every feature in the configuration by comparing each vertex in the comparable points, lines, and polygons. A tolerance is set for this comparison
• Attribute Tests: Validates attribute values between the source and target data for every feature in the configuration. Lookup tables are configured to capture attributes that are being migrated from one value to another, or to match a domain in the target geodatabase.

The user interfaces for Quality Checker is easy to work with:

Configuration of Quality Checker is kept in a formatted JSON file that is user-friendly to edit. One configuration file can be used for all 3 of the QA types.

Example below:

Test results are exported to an Excel spreadsheet:

Overview

Details

For the attribute checks, there is an option to have the mismatch errors exported to individual text files by feature class which display the source value and the target value for each error.  This added detail is very helpful when tracking down the cause of the errors.

CONCLUSION

Once configured, the Quality Checker tool is easy to use.  It supplies reports that GIS managers can use to prioritize the work and gives the GIS users the information needed to quickly locate and investigate them.  The tests can be rerun at any time as the GIS data is being updated.

With the flexibility of the configuration, Quality Checker will definitely be used for future projects at SSP.  GIS data migrations are going to keep on occurring and this tool will help these projects greatly.