The very first blog post I wrote for SSP back in June of 2011 was Versioning for Dummies. It ended up being a four-part series explaining how geodatabase versioning worked behind the scenes, by detailing how state ids managed the transactions, the adds and deletes (A&D) tables tracked the changes, and reconciling, conflict detection and posting made it all magically work. Over the years, we’ve encountered various issues with versioning related to conflicts, complex state trees, and overall geodatabase performance — and we’ve been here to help educate you through all these problems. Now, over six years later, I am writing the first installment in a new series focused on Esri’s new, branch versioning model, which is being released alongside the Utility Network.
The great news is that Esri has learned from the challenges present in the traditional geodatabase versioning model and has overhauled the way versioning will work in the future.
Branch Versioning in the Utility Network: What Is It?
There are no longer the A&D tables, no longer a state id, and no longer a state tree or state_lineages table! The new versioning is called branch versioning, and it operates using a temporal moment as opposed to using tables and state ids to track delta changes. The word temporal is defined as “of or relating to time,” and that is precisely how the new branch versioning works. Some of the major benefits of the new branch versioning include:
- Improved Performance — Simplified queries need only to access a single versioned table (see below) to get to the versioned data leveraging timestamps and branch ids.
- Simplified Archiving — Those of you who have implemented Esri archiving know how this adds a lot of overhead to the traditional geodatabase. Branch versioning provides full archiving capabilities via the single, versioned table.
- No Nightly Reconcile & Compress — Say what?!? That’s right, the simplicity of the new branch versioning means that we don’t have to fully reconcile the state tree each night, and the concept of a compress is long gone.
- Support for Parallel Posting — We can now support concurrent reconcile and posting workflows (there is no longer a serialized process to post to the DEFAULT version).
- Enhanced Conflict Management — This is the ability to persist (i.e., save) a version with only SOME of the conflicts being resolved. Come back later to finish the rest. And new functionality helps you tag inspected conflicts with comments during the resolution process.
- Version Ownership Change — This is the ability to change a version’s owner, which leads to better integration with workflow management, where differing owners own the version throughout the lifecycle (currently the version ALWAYS shows as owned by the creator).
- Enhanced Editor Tracking — Esri added the ability for create/update user and date at 10.2.1, but has now extended this to track users who delete features within a version.
- Partial Posting Support — While not fully included out of the box, the new versioning model includes support for partners to build partial posting workflows — finally solving the holy grail of Esri version management. More on this in a future post.
Enough about the Benefits … How Does Versioning Actually Work in the Utility Network?
Instead of lecturing on about the terminology and benefits, let’s begin to dig in to the new branch versioning to see how it operates! Everything I am performing below is contained within a new Utility Network, administered via ArcGIS Pro. For the purposes of this post, I am using PostgreSQL (see related articles at the bottom for more on this database option), though the same concepts apply to Oracle and SQL Server.
I started off by creating a new feature dataset and a new feature class called Premise to manage my address points.
This is a simple feature class with a handful of fields as shown here (via a connection to the database):
Note that there are currently 11 columns in the feature class/table. Now, before I configure this class to be Registered as Versioned, I need to configure the database to use the new branch versioning vs. the traditional delta versioning of the past. We do this via a geoprocessing tool in the Utility Network toolbox. To verify my geodatabase is configured correctly, I can check the GDB connection properties:
Note the use of branch version vs. the traditional version type. This is purely the configuration of the geodatabase connection – we haven’t actually versioned the dataset yet. This next step is very similar to versioning in 10.2.1. We simply right-click the database and register it as versioned:
To best understand how versioning works, I have monitored the changes to the database during this operation. As noted above, no new tables are created! So, what really happened behind the scenes? This operation simply updates the premise table with six new columns to enable the branch versioning (note we now have 17 columns on the table):
These six new columns allow edits to this class to be managed via a temporal (time-based) approach. The first realization you will encounter is that branch versioning manages ALL of the versioned edits in the single (base) table. VERY different from today. To drive this home, let’s take a look at what each branch versioning column tracks:
- gdb_branch_id: Think of a branch as a version. Therefore, each new version will be assigned a branch id, and this column will track the id of the version where the edits are applied. sde.DEFAULT has a branch id of 0.
- gdb_from_date: This is the clearest indicator of temporal versioning. This column contains a date/timestamp of when the edit occurs.
- gdb_is_delete: This column is a binary/bit field which tracks a yes (1) / no (0) for whether the edit was a deletion of the record.
- gdb_deleted_at: When gdb_is_delete is set to 1 (indicating a delete), this column is set to the date/timestamp of when the delete occurred. This will be the same value as is saved into gdb_from_date and was added to be consistent with Esri’s editor tracking fields.
- gdb_deleted_by: This tracks the username of the user who deleted the record. This is based on the user’s login to Portal (where the Utility Network data is hosted) and will be maintained on the deletion edit forevermore.
- gdb_archive_oid: This field tracks a unique id of the edit that has occurred. As a single record is edited multiple times, each edit is tracked in the table with a unique gdb_archive_oid value.
Each time new edits are applied to the class, a new record is added to the table and these columns are updated accordingly. My first question (and maybe yours) was how the concept of a compress is handled. In other words, how are multiple posted edits combined into a single, default record? The very short answer is that they are never combined. As you continue to edit the underlying data, the base table will continue to grow.
I have talked to Esri extensively about this, and the reasoning lies in the simplicity of the queries used to access the versioned data. In the traditional versioning world, very complex queries were required to combine the base table, the state lineage table, and the A&D tables (and the spatial index). Now, there is much simpler query that hits the single table. The simplicity results in much faster results, which will allow these tables to grow significantly in record count with little to no impact on performance. Our customers who have gone through SSP’s Utility Network Jumpstart still had concerns, though, and Esri noted they are considering a prune task for a future release that would reduce the record counts in the base tables by combining older posted edits. More to come on this subject in the future! For now, back to how versioning works …
In my above example with the Premise table, my next steps would be to publish the premise layer to my Portal and then to consume it in ArcGIS Pro. An important note is that the traditional Versioning toolbar has been brought forward into the new Versioning ribbon in ArcGIS Pro, and as far as the user goes, all of the versioning tools work exactly like they do today:
To be clear, the change to branch versioning only impacts the underlying infrastructure in the database and the end user experience has not changed at all. You still create a version, reconcile, resolve conflicts, and post as you do in 10.2.1 today. On a side note, SSP will be providing an ArcGIS Pro plug-in to our industry leading work management system, Lifecycle Work, that will allow you to manage jobs, edits, and overall GIS workflow seamlessly with our web based software. This will allow your users to simply interact with jobs (work orders, service requests, edit sessions) without having to worry about the underlying versioning. More on that at this year’s GeoConX conference!
For now, I hope that this preview on the new Utility Network and its use of branch versioning will get you excited about where we’re headed with the core software. The result is a much simpler, efficient, and performant versioned GIS that removes many of the pain points we’ve encountered over the last 20 years.
And finally, I would be remiss without reminding you that access to the Utility Network geodatabase is ONLY available through your Portal web maps / REST web services. The days of accessing the geodatabase through SQL and MultiVersion Views (MVVs) are gone so start to get your head around using the new services oriented architecture (SOA) for your reports, integrations, and exports. This will require much more discussion, so stay tuned for more on this topic in coming months. Until then, happy versioning!