As a partner of SSP’s and president of Blair Services, Ed Blair was tapped for this guest post regarding how to migrate Smallworld data and move to ArcGIS. 

Smallworld Data Migration

If you’ve got a Smallworld GIS (SWGIS) database and you’re contemplating moving to ArcGIS, well, first of all, good choice!  We think you’ll find the quality and breadth of technology available within ArcGIS extraordinarily valuable to your organization.

Of course you’ll need to migrate the data – a task that’s been done plenty of times.  But as each company’s GIS database is unique so will be the specific steps of your migration process.  Having been through this process numerous times ourselves our advice is that you’ll want a thoughtfully crafted approach rather than a blunt force instrument.

There’s no denying that SWGIS has some sophisticated data structures of which you may have taken advantage.  There are also some aspects of an SWGIS database that are foreign to ArcGIS users – think “null geometries.”  All that said, here are six things to be thinking about.

1.      Internal Worlds.   This is probably the most well-known and most heavily discussed distinction between SWGIS and ArcGIS data structures.  The basic concept is that and SWGIS database can have a set of distinct “worlds” each of which can have its own planar map projection.   External world (“world zero”) feature geometries are described in a mapping projection that suits coverage of your service territory – such as a state plane, UTM or similar projection system.   Other worlds can be defined for equipment judged best separated from, and not displayed with the external world using a different projection, or no projection at all.

These are typically used for depicting details inside some type of container, such as an electric substation, a gas regulator station or electric switch cabinet – though the particular use depends on the needs and ambition of the Smallworld implementer.  We’ll use as an example of an electric switch cabinet to illustrate.

External and internal worlds are linked at “hypernodes” that exist in both the external and internal worlds.   These enable a SWGIS trace to originate inside the internal world and terminate in the external – or originate in the external world, pass through the internal world, and terminate at another location in the external world.

There are pros and cons to the use of internal worlds within a SWGIS database, but we’re not going to get into the details of that topic here.  We’ll just leave it that some customers we’ve worked with have been glad to get data out of them.

At any rate,  if you’ve got internal worlds and they contain equipment you need to include in your utility network they need to be migrated into the same projection (or “spatial reference” in ArcGIS terms) as the rest of your network.

The migration mechanism is to apply a transformation to internal world geometries to convert from internal to external world coordinates.  ArcGIS provides a variety of methods for transforming geometries and if the arrangement of matching internal and external hypernodes is proportional then, really, any of them will work fine.

The example below used an affine transformation, which preserves collinearity and ratios of distances.

However, if the arrangement of internal and external hypernodes is not proportional, then a different transformation may be best.   The example below illustrates this case with the affine and several other available transformation methods available within ArcGIS.

And of course in the case that after transformation an edge that snapped to an internal hypernode does not snap to an external hypernode, that edge must be extended to the hypernode post-transformation.

Hopefully you’ve now got the gist of the process.  However, here are some special cases that also need to be accounted for:

• Internal Worlds with a single hypernode
• Internal worlds with no hypernode
• Mismatched internal and external hypernodes

Finally, there’s not just one way to do this, there are options.  Here are some you’ll want to consider:

• If you have a small, number of these you might consider re-creating them by hand rather than going throughthe migration. Actually we would advise this only if you have a very small amount of them as the manual effort would have to be repeated for each iteration of the migration.
• Apply a transform of your choice as-is to all internal worlds
• Adjust hypernode locations to give the best transformation result possible.

2.      Multiple geometries per object.   Probably the second most widely known distinction – just behind internal worlds – is the fact that objects in an SWGIS database class can have multiple geometries.  The geometries can be of different types; for example a gas regulator station can have a polygon geometry field that represents the station boundary and another point geometry field that is the station centroid, or of the same type; for example a valve can have one point for its location along a pipe and another point location offset from the pipe for view at smaller scales.

There are multiple strategies for incorporating these into an ArcGIS Geodatabase.  In summary, the options are:

• ArcSchematics.    A component of core ArcGIS technology that generates and maintains a schematic representation in an associated schematic dataset.
• Cartographic Representations.  Also a core technology component, the representation is basically a graphic element with a geometry stored on the feature.  In this sense this approach is similar to the SWGIS model.
• Multiple Classes.   Keep two feature classes, one for each geometry in the source SWGIS class that are related by a common identifier.

It’s hard to say if one approach is consistently better than another as each has benefits and drawbacks.  Be sure to review each option with your vendor.

3.      Multiple connection points.  The third most widely known distinction (in our estimation anyway) is the fact that objects in an SWGIS network can have in addition to the geometry used for feature display,  multiple points at which the object is connected to the network.  The prototypical example of this is an electric transformer that is often modeled with separate “pins” for high side and low side connections (see below). 

     This is actually pretty handy since it clearly informs applications like traces what type of equipment is connected to what connection point.  To accomplish the same thing ArcFM™ Electric traces inspect all edges connected to the transformer and interrogate a VOLTAGE field to determine whether a given conductor is connected to the transformer high side or low side.

In migration we need to create an actual geometric connection between the high and low side pins.  Two options we’ve pursued in past experience are to (1) create a “jumper” edge that’s not either primary or secondary but really an additional geometry associated with the transformer, or (2) extend the secondary/service lines by adding a vertex that connects to the high side connection point.

Both approaches accomplish the task in this example.  Your specific SWGIS database may employ similar but different conventions that will need to be addressed as part of your migration design.

4.      Proposed equipment.   This one is not so much a difference in data structure, but a difference in application functionality – and nonetheless something to be accounted for in your migration if you take advantage of it.

      The crux of the matter is that SWGIS tracing applications can take into account the life cycle status of a feature and, if tracing on the “existing” network ignore any connected features that are “proposed install” or “proposed retired” and if tracing the “proposed” network take proposed statuses into account.  Neither core ArcGIS nor ArcFM™ tracing does this.

For features that are extensions to the network this is really not such a big deal.  For features that are in-line, it can be.  In the images below we see a new gas line extension (dashed pink line) to serve a new cul-de-sac.  Though this is connected to the network, we can prevent tracing into proposed by setting a field with the ArcFM™ PIPESTATUSINDICATOR model name to [0] Closed (middle panel).

The same can be accomplished in an electric network using a field with the FDRMGRNONTRACEABLE model name.  Using an ArcGIS core Find Connected trace we traverse the proposed equipment (right panel.)

For a new in-line feature the options aren’t so flexible.  Say for instance in the same gas network pictured above we wanted to insert into the existing pipeline a new proposed “closed” valve and have it optionally honored by the same ArcFM™ or ArcGIS trace, well, then some customization would be involved.

The presence of proposed equipment, the specific way its modeled in your SWGIS database and the way it will be migrated into ArcGIS should be a significant topic of discussion in your migration design.

5.      Domain mapping.   Getting to topics not so widely known, SWGIS provides the ability to define an enumeration of valid values for one or a set of fields in a manner very similar to ArcGIS coded value domains. And the mapping of these domain values is one of the most tedious but one of the most important tasks in your migration. Here are some best practices we’ve assembled along the way:

• The domain value mapping will certainly need to account for hundreds and possible thousands of values and even if you’re using a standard ArcGIS data model there’s little chance the standard will include your SWGIS domain values.  When dealing with hundreds or thousands of distinct values it’s inevitable that there will be typos and mismatches – especially at the beginning.  Your migration process should catch and report mismatched values as it operates so you need not have to weed these out after the fact.
• A domain entry in both SWGIS and ArcGIS includes a code value and a name, as illustrated below.

Be clear at the outset whether either or both of these components need to be retained in your Geodatabase.  For example, if you have an interface between your SWGIS database and another database/application that you plan to replace, and that interface passes domain NAME values (i.e., “Ductile Iron”, etc.) then make sure the domain names in your Geodatabase match those in your SWGIS database.

• The initial definition of your SWGIS database was likely performed using a CASE tool that contained definition of your classes, fields and relationships – as well as your domain values.   A CASE report from this definition can be  a very valuable place to start in documenting all of your domain values – but we’ve seen that not everyone keeps the CASE tool up to date as domain values are altered and extended once the database has gone into production.  (For what it’s worth, the same can be said of many ArcGIS Geodatabases and the UML model from which they started.)   The lesson is, don’t rely on the CASE tool report alone as the sole source of domain entries.  Look at your SWGIS database itself as well.
•   <null> values.  Many times we’ve seen SWGIS domains include entries like [9999] “Unknown” included because there is otherwise no way to revert a selection to <null> if another value is selected by mistake.  You can choose to keep these values if desired, but all ArcGIS domains include the ability to assign a <null> if needed.

6.      <Null> Geometries.   This is an example of where the SWGIS and ArcGIS databases are quite distinct.  In ArcGIS a user can’t create a feature with a <null> geometry through the user interface.  There’s just no way to do this (though it can be back-doored using ArcObjects).  In SWGIS a feature geometry is like any other attribute and can be left <null> like any other attribute.   Again, whether this is good or not is for another time, but for data migration we must have a strategy.  Here are three options we’ve pursued in our experience:

•  Track down and remove in SWGIS.  While these are possible some SWGIS customers actively seek and destroy these as they arise.  In which the migration process can ignore the possibility of their presence.
•  Track as anomalies in the migration process.  A <null> geometry is easy to identify, just harder to know what to do with.  The migration can simply report their presence in a table or report of anomaly conditions.
•  Track the “meatball”.  In some cases customers have employed a generic geometry colloquially referred to as a meatball to allow graphic selection of a feature that has a null geometry.   This can be optionally migrated as well.

Summary

Those are our top six topics, and we’ve probably gone on here long enough.  Hopefully you’ve gotten the point that while migration from SWGIS to ArcGIS is quite doable, and one that’s been successfully accomplished for many companies, there are important details to weigh in the process.

If you want information about other important SWGIS topics, like “Structure Location” objects, preserving parametric curves, connectivity rules, tracking anomalies, retaining legacy unique identifiers or anything else, please feel free to get in touch.