You are here

Nightly Batch Suite Lets IREA Get to State Zero

IREA nightly batch suite - Get to State Zero

IREA Re-Projects Data without Losing Versions

Status: Completed


nightly batch suite ireaThe need: Intermountain Rural Electric Association (IREA) wanted to re-project their data from NAD 1927 to NAD 1983 without losing their versions. A combination of the SSP Nightly Batch Suite, the SSP All Edits Tool, several white-board sessions, and a creative vision made it all possible.

The background: IREA is a nonprofit electric distribution cooperative that serves more than 150,000 customers inside a 5,000-square-mile service territory along Colorado’s Front Range. Their headquarters is in Sedalia, Co., and district offices are in Conifer, Strasburg and Woodland Park — also in Colorado.

The Challenge: 

IREA wanted to re-project their data from NAD 1927 to NAD 1983 without losing their versions. As you can see it's a very simple request, but the complexity was challenging to say the least.

The Solution: 

A combination of the SSP Nightly Batch Suite, the SSP All Edits Tool, several white-board sessions, and a creative vision made it all possible.

How does the process work?

  1. First, we use a new configurable SSP Nightly Batch Suite (NBS) application to extract all versions into the SSP All Edits tables and feature classes. These are effectively the same tables and feature classes used by the SSP All Edits Report when extracting an individual version for re-visualizing it at a later point in time.
  2. Next, we delete all versions and compress the database. 
  3. If required, drop all relationships, all geometric networks, and un-version the GIS Data.
  4. Do whatever you need to do once you are at State 0 (In IREA’s case we had to re-project their data from NAD 1927 to NAD 1983).
  5. Use a second configurable SSP NBS application to recreate all versions extracted in step 1. This application creates the versions using the exact same name and then replays all of the edits including inserts, updates, and deletes.
  6. If you are a Schneider Electric Designer™ shop, then there is one additional SSP NBS application that is used to re-synchronize each design’s XML which includes references to the versioned object ids.
  7. Celebrate!

After recreating the versions and verifying data, IREA’s versioning lineage looked like this:

nightly batch suite irea Nightly Batch Suite - Get to State Zero

Close-up shot: 

nightly batch suite irea Nightly Batch Suite - Get to State Zero

It truly is a nice thing to have your versioning lineage as displayed above; normally you can’t even find where SDE.Default is within this view.

Now that you have a general idea of how the whole process works, there are probably a few questions popping up in your head: How long did it take to run? Was it a few hours, or days? How long was production shut down? etc., etc.

Production was shut down only for a single day (Friday) and the process took us 2 ½ days to get the whole project completed - including the re-projection which was time-consuming as well:

  • Thursday afternoon – Preparation time. Installing code, updating config files and kicking off the first SSP NBS  application “WriteVersionsToDB”.
  • Friday – Deleting versions, relationships, dropping network, un-versioning GIS Data, re-projecting Electric and Land Dataset, rebuilding the database (recreate relationships, network, version GIS Data) and kicking off the second SSP NBS application “CreateVersionsFromDB”.
  • Saturday – Validating all versions were recreated successfully, running the final SSP NBS application “UpdateDesignXML” to update each design's XML with the new object ids and full system testing.

The following statistics were gathered from running the test process in our office with a total 299 versions and from running the production process at IREA’s site with a total 242 versions.

SSP's Office

 SSPs Stats Nightly Batch Suite - Get to State Zero

IREA's Office

 IREA Stats Nightly Batch Suite - Get to State Zero

The difference in timing between our office and IREA are mainly due to system infrastructure (i.e. memory, processors, etc.) but I hope it gives you an idea as to what to expect if/when you consider getting to State 0 without losing any of your versions. You should be able to use your version- and/or edit-count from your geodatabase to come up with some approximate processing times within your environment.

The processing time at IREA took about 12 hours and 29 minutes (give or take) for 242 versions, but what if you have thousands of versions? Would you need to shut down production for more than couple of days? The answer is no - our solution is scalable. We recently made some changes so that you can process a range of versions within a multi-threaded implementation of the SSP NBS.

For example:

Let’s say you have 3,000 versions. We would set up 3 SSP NBS processes to run concurrently, each processing 1000 versions at a time.

  • NBS1 will process versions 1-1,000.
  • NBS2 will process versions 1,001-2,000.
  • NBS3 will process versions 2,001-3,000.

This will decrease the time it takes to process the entire geodatabase by splitting up the workload. The processing time may go up slightly per each edit/version because there is only a single geodatabase supporting the multiple processes, but this approach will tremendously reduce the overall down time within your production environment.

Additional NBS processes can be added as long as you have the hardware and database resources to support running the processes concurrently (i.e., additional application servers and a well-performing geodatabase).

The Results:

There you have it, reaching State 0 is no longer just a theory; the issue is now a thing of the past. You can get to State 0 without losing your versions. 

We asked IREA’s Duane Holt for some feedback about what getting to State Zero meant for him and IREA, and this is what Duane said: 

"The AEZERO project provided IREA the ability to reach State 0 and re-project the geodatabase to a NAD83 coordinate system.  Without going through this process, the antiquated NAD27 geodatabase would be restricted by “on-the-fly” projection limitations that are present in Web Mapping Services. 

"Completing the AEZERO project allows IREA to progress into the future of GIS and deploy more web GIS technologies such as ArcGIS Online.  Successful completion of the project instills confidence that the same process can be repeated if necessary to reach State 0 again.

"The AEZERO project employed a process that utilized two of SSP’s out-of-the-box products with a little customization and specific configuration.  The thorough development and testing completed by SSP and IREA resulted in a succinct step-by-step process.   Much like following a recipe to bake a cake, the AEZERO project at IREA stepped through exactly as planned in shorter time than expected. 

"Working side by side with SSP developer Dennise Ramirez, IREA’s Duane Holt was able to work through the project and at the same time gain the knowledge and confidence that the two companies had created a scalable, reusable method to reach State 0 in a versioned database without losing the versions; a far-reaching idea for any GIS with a versioned SDE geodatabase.  I am continually satisfied and surprised at how easy project rollouts have been working with SSP Innovations."

Related Content:

Confidence and Control: Managing SDE Versions

All Edits State 0 Used to Replay Offshore Edits

All Edits State 0 Used to Replay Offshore Edits, Part 2

An All Edits and State Zero Combo Platter