At a recent event, GIS colleagues were discussing their latest work activities and project challenges. During the conversation, someone asked, “How will you do that?”, referring to a recent SSP project venture that entailed a significantly large production geodatabase plus a massive multi-year data improvement program … soon enough, SSP came up with a solution that would combine the functionality of two of our most successful solutions: All Edits and State Zero. This post describes the opportunity.
SSP is part of a talented multi-faceted project team brought together by a valued utility client. The over-arching mission is to carve out a process and solution that allows for a significant, multi-year utility GIS data improvement program to take place, all the while having the utility’s production geodatabase keep humming along by executing and posting data transactions over the course of the data improvement program.
You could be wondering….”how can a third party company be making edits to the production database while the client is making edits at the same time?”
On the surface of it all, the requisite component elements of the program were defined, quantified and procedurally planned:
- Source data for the utility data improvement program identified, gathered and prepped – check
- Established vendor to apply source data to existing production GIS database – check
- Data processing and technical specifications for data improvement program – check
- Project administration and organization established – check
- Enthusiasm to elevate utility data to the next order of completeness and quality – check
A crucial remaining element to be understood, defined and fulfilled was the need for a solution that would gather the data packages provided by the data processing vendor and incorporate those into the production geodatabase with minimal disruption to existing data management operations. SSP’s role is to provide for that “copy” from working database and “paste” to production database.
Would it be as easy as a ctrl+c and crtl+p? You must be wondering…The answer is Yes and No. There are some conditions that we must adhere to and also some core assumptions:
- The data improvement program will process massive amounts of geo-spatial and attribute data over a wide and far-ranging service area; a solution will need to operate in a batch-type mode and be repeatable, able to handle hundreds of discrete data package deliverables over multiple years given the quantity of work.
- For the program, the utility’s Service Area will be divided into “Working Areas” to localize, manage and contain data processing for the scheduled cycles. Project coordination will be used to separate work among the data vendor and the utility’s day-to-day data maintenance operations based on geography to minimize overlap and conflicts.
- The utility GIS data model will need to be frozen over the course of the program.
- The processed data, at some point in its journey, will need to account for and ensure that existing automated business rules such as ArcFM™ auto-updaters will be executed within the process lifecycle.
- Existing production data interfaces among internal business systems must be accommodated for when bringing the processed data back into the production environment.
Enough of the background information! Here is how we will enable the crtl+c and crtl+p functionality:
The approach will leverage core software stack capabilities such as the Esri versioned geodatabase framework – the data packages will be delivered, transferred and applied to the target geodatabase via geodatabase versions.
Also key to the solution will be existing tools and applications built by SSP overtime and further extended. Primary among these applications will be the SSP All Edits Reporting and QA Tool, the NBS application framework (Nightly Batch Suite) and functionality from State Zero.
The All Edits Reporting / QA tool has been evolving over a number of years and based on this particular effort, will now have the ability to replay edits from one geodatabase (the data vendors working database) and apply those edits into another geodatabase (the utility’s production database).
SSP will utilize the All Edits extraction engine to systematically extract all of the edits from specified versions as provided by the data processing vendor. The extracted data will be written to the All Edits schema for both spatial and attribute data, and from there will be replayed into versions created in the utility production geodatabase.
The SSP NBS application framework will provide for the requirements pertaining to version creation, the batch execution of the extraction engine schedule, providing notifications of status and completion, and other functions.
For more information on the All Edits Toolset, NBS and/or State Zero please see associated SSP Blog Posts. Also, please return for information and results regarding how the design was implemented and how the solution objectives were met.