If you’ve kept up with SSP Innovations’ over the years, you’re aware that SSP has been primarily focused on providing Esri utility network services, building utility network products, and bringing our customer’s utility network implementations to life for a while now. Now that we’ve gone live with several very complex, highly integrated implementations, a few have asked me to give a few post-Go Live lessons learned to help organizations manage their upcoming implementations.
The GIS Footprint is Growing (and getting more complex)
The application and infrastructure footprint of the GIS continues (and will continue) to expand as utilities learn more, interesting ways to use GIS applications operationally across the organization. Even recently, we’ve seen a significant uptick in the various ways in which (and quantities of) mobile and web GIS applications are being deployed at utilities. ArcGIS Enterprise has only accelerated this evolution.
Examples include:
- Portal web maps showing real-time weather and outage information on the giant monitors in the Operations Center
- Operations Dashboards showing dynamic reports exactly as required by PHMSA regulations
- Economic development maps help decision-makers decide the best opportunities for expanding their organization’s service territory,
The utility network now compliments this digital revolution by supporting the existing Services-based architecture that many utilities have been leveraging. The Utility Network functionality enhances the enterprise allowing dynamic access of critical network GIS data across desktop, web, and mobile clients. The dynamic nature of this technology is powerful and enables a means of placing the analysis capabilities in the hands of the personnel making the decisions and performing the work. As utilities shift from making operational decisions off paper and one-off files to these highly dynamic systems, they become more and more critical and more and more complex. The reliability and performance of these systems is more important than it has ever been.
To this end, it’s important to follow some simple guidelines that will enable you to administer these systems:
Environment Management
Whether you’re a smaller co-op or a massive organization, implementing the latest in Esri platform technology enables the organization to more effectively support a much wider range of users. This naturally leads to the need for more server technology, specifically to support proper data governance, the proper level of DevOps, and a broader set of test and staging environments to support the service oriented-solution. Of course, this adds overhead in IT Administration as well as troubleshooting. Subtle differences among GIS environments (e.g., TEST vs PROD) can cause discrepancies in the source of errors, make a significant impact in performance observations that can lead troubleshooters down rabbit holes, or worse yet, reveal issues only when you get to Go Live.
To minimize these problems, try and follow some basic rules of System Infrastructure Governance (i.e., system monitoring, identifying issues, and planning, testing, executing, and documenting updates) that will help you administer the system over time:
1. Perform System Monitoring. Your GIS is now a web-based technology dispersed among several tiers of applications and servers. Points of failure are no longer limited to your desktop application stack or the database. Data and processing are spread across external services (ArcGIS Online), load balancers, web adapters, Portal servers, GIS servers, client applications, and everything in between. You will need to inspect the calls/web traffic among these servers using a tool like Fiddler (https://www.telerik.com/fiddler) to troubleshoot issues as they arrive. These tools will capture errors returned by much of the infrastructure and pinpoint the calls causing the errors. Additionally, talk to SSP or Esri about implementing ArcGIS Monitor, which will help pinpoint infrastructure bottlenecks and allow you to continuously tweak and improve the configuration of the system over time.
2. Manage your infrastructure as carefully as you manage code. You already carefully manage your source code by making changes and promoting those changes through environments. Code changes are performed, tested, promoted, tested again, and finalized long before they are made in Production, and so should your server configurations. “Infrastructure as code (IaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools” https://en.wikipedia.org/wiki/Infrastructure_as_code#cite_note-AWS_in_Action,_IaC-1 .
At SSP, we’ve been helping our customers manage their ArcGIS Enterprise deployments and upgrades through the use of a Continuous Configuration Automation (CCA) Tool like Chef or PowerShell) https://en.wikipedia.org/wiki/Continuous_configuration_automation, which will allow you to deploy your ArcGIS Enterprise consistently across distributed environments. We typically create the baseline deployment package and walk you through the installation, ensuring for consistent server configurations and an easier path for upgrades.
3. Create and maintain a duplicate of the Production (i.e., Sandbox) environment. This will help accurately troubleshoot and test changes without affecting your production users. Having another environment that is configured very similarly to your production environment in as many variables (software versions, server types, server resources, network configuration, etc.) will help isolate issues as they occur and provide a safe place to troubleshoot. You don’t necessarily need to have the exact same number of servers but be sure to align with the logical architecture (e.g., if it’s a distributed architecture with web adaptors, portal, server, and desktop machines all on different servers, be sure to setup that same architecture in the Sandbox environment).
4. Keep an active, detailed system architecture document. It’s important to have up to date location (data center) and resource (RAM, processor, number of cores, etc.) information for every server in each of your GIS environments. This includes desktop machines/servers, web tier, (application) server tier, and databases server information. Changes to server configuration should be documented and noted to help reveal differences in performance / errors form environment to environment.
Treat your deployment data model carefully
There are a lot of new capabilities and value that the utility network is bringing to utilities. SSP has worked with many customers with varying modeling goals and associated representations of their data and is developing approaches and frameworks to support common standards.
Although SSP Productivity adds valuable data management capabilities, the simple fact of potentially adding more data with greater connectivity and options for analysis could likely add additional overhead to managing that information. It’s very important to understand the impacts of these changes to know how they will impact user efficiency and performance.
Some tips to follow when sifting through the various UN model options and maintaining that model:
1. Try to make your implementation as agile as possible. In most utility network projects, there is a significant amount of technology that needs upgraded or deployed. Rather than collecting requirements, building everything in isolation, and then bringing in the end users at the final acceptance testing stage, you should make your implementation as iterative as possible. Start with the model and configuration of the core applications and use a utility network migration technology like SSP Sync to quickly get the technology and the data into the hands of the end users as soon as possible. While integrations, attribute rules, and/or custom tools are being developed, have your users perform typical workflows to provide feedback and allow for time to create solutions or make model modifications to adjust to their needs. Demonstrate each component as it becomes functional and get user feedback as early as possible. At SSP, this iterative approach is a fundamental aspect to our project methodology, called the Sustainable Quality Method or SQM. Getting end users involved early and often is always a good approach to any technology project, but it’s critical to the success of your utility network project.
2. Align your approach to the UN model with other customers. There has been a lot of work already done to test the impact of certain models (e.g., high vs low fidelity). Choose an implementation vendor who has evaluated the various permutations of the UN model, has implemented a real-world application of these models, and fundamentally understands the impacts of the changes to those models. Not only how changes can affect functionality but also how they affect performance.
3. Carefully evaluate/manage changes to the model. Create a data model council within your organization to evaluate data model requests and determine all the options (identifying existing fields, using aliases, layer names, etc.) for implementing those changes. If changes are required, it’s important to understand all the impacts and unintended consequences to changing regular fields (performance, consistency, tracing, integrations, custom tools, reports, etc.) and/or UN fields (performance impacts for validating, updating subnetworks, tracing, version processing).
Ensure Esri is involved
As you know, SSP Innovations is always working with our partners at Esri through all levels of our organization: product, sales, marketing, support, and especially professional services. On utility network Implementations especially, we strongly encourage our customers to include a careful System Architecture assessment and recommendation from Esri. This should be followed with setting up ArcGIS Monitor and a formal tuning assessment. This effort will provide a ton of value and help to ensure your users are happy at Go Live. Further, we have worked with Esri to provide premium support contracts during the duration of your implementation project (and after). This is so valuable as it will help quickly identify environment or application issues by escalating them within Esri Support. Then, if a fix is required, and your issue is identified as greatly impacting production and your users, it will be supported through the ENM (Enterprise Network Management Release).
What do you think?