Pravin V. More
Water GIS Coordinator
Water Planning Department
State of Qatar
[email protected]
Qatar General Electricity & Water Corporation (KAHRAMAA) was established in July 2000 -to regulate and maintain the supply of electricity and water to customers. Since its inception, KAHRAMAA operated as an independent corporation on a commercial basis with a total capital of four billion Qatari riyals.
Water network in the State of Qatar expanded extensively in recent years. The growth of urban areas, industry and agriculture has led to an increase in the water distribution network to 3,622 kilometres and expansion in the number of storage reservoirs to 23 with a total capacity of 259 million gallons of water a day. The electricity transmission network consists of approximately 100 primary high-voltage substations with a total of 660 kilometres of overhead lines supported by 600 kilometres of underground cables across the country. The network is coupled with 6,500 low- and medium-voltage substations (11 kV) and more than 4,500 kilometres of cable lines.
Introduction
KAHRAMAA (W) implemented “GIS Technology Migration and Transition Project” which included database design based-on ArcFM, data dictionary development based-on CGIS guidelines, data migration and QA/QC, backlog management and operational updating, customised GIS utility applications development based on ESRI ArcGIS 9.3.X release and training.
Project tracks
Database design: When KM (W) embarked upon creating a GIS system within the organisation, it was vital to ensure that the basic block of the system, the data model, would accurately represent and contain all the water network assets’ information, while at the same time remaining flexible enough to allow for any additional features to be incorporated easily. As such, KM (W) capitalised on its choice of the ESRI ArcGIS platform combined with Telvent’s ArcFM in order to create an efficient water network model that is able to accommodate maintenance and facility management solutions effectively.
The proposed data model for KM (W) was founded on the object data model that was based on ArcFM for water transmission/distribution. The ArcFM model had been adapted to suit KM (W) requirements and to allow for the migration of data available in CAD, Shapefiles and other systems to the new ArcGIS based geodatabase without losing any information previously acquired. This migration from KM (W) current format to this new and improved ESRI Geodatabase (ArcSDE) format based-on ArcFM data model was essential in order to incorporate all the water GIS database information that was previously stored in various GIS formats and thereby reducing the overhead that was necessary to maintain and keep track of the information and its updates. Furthermore, the complete data migration to the newly designed Geodatabase improved the KM (W) data update procedures and with the developed metadata auto updaters, the editing and the quality control, the overall data quality and maintenance activities have also been enhanced.
KM (W) created the data model by starting with the core ArcGIS object classes: objects, features, simple and complex edges, and simple junctions. The creation of the data model followed an intensive process of numerous workshops conducted, with the participation of a vast variety of skilled KM (W) engineers in conjunction with the GIS consultant, the purpose of which was to model the KM (W) distribution system in GIS including the identification of attributes, relationships and behaviours. Unified Modelling Language (UML) was used to present KM (W) assets in a structured object oriented manner. The modelling began with a top-down approach, where the list of network objects was conceptually divided into logical groups. The key characteristic of these groups is that they share common properties and/or behaviour. For example, valves can be grouped together because they connect pipes and they have the ability to control the flow of water in the network.
Having an accurate and structured dataset is crucial to the validity of the complex spatial analysis that is to be performed by a GIS. As such, KM (W) decided to include three datasets in its data model:
- WaterDistributionDataSet: It contains the various feature classes and GDB components that form the existing KM (W) Water Distribution Network.
- HWaterDistributionDataSet: It contains the various feature classes and GDB components that are deleted from the KM (W) Water Distribution Network
- PWaterDistributionDataSet: It contains the various feature classes and GDB components forming the proposed and under design KM (W) Water Distribution Network
Each of the datasets contained all the attributes of the water distribution network objects, whether they are points, polygons or lines, such as pump stations, valves, pipeline segments, etc. These components were grouped into four general logical categories: water network components, water line, water operation records and water network facilities.
Data dictionary: With the data model for the KM (W) Water Distribution Network completed, it was essential to develop a comprehensive document that contained information on all aspects of the model. This document would in turn assist any KM (W) user in order to gain a better understanding of the data model itself, and as such, on how to make use of its various components easily, effectively and quickly.
KM (W) recognises that many of today’s challenges involve optimising the use of existing resources and effectively managing capital improvement budgets to ensure sustainable service quality. And with the object technology at the core of any ArcGIS software that combines data and application behaviour modelling, the water data model not only includes an essential set of water object classes and properties, but it also includes rules, and relationships that define object behaviours. Therefore, it is essential to have a document that specifies not only the logical design but also the physical database design of the data model, which includes descriptions of database feature classes, business tables and properties. It also includes table layouts of all feature attributes and related tables. The Data Dictionary Document is a living document and is modified from time to time, depending on new any updates made to the data model itself.
KM (W) developed the data dictionary not only in order to provide its users with general information about Geodatabase components and an overview of the system, but also in order to provide its users with a one-stop-shop containing descriptions of the different components of the KM (W) database, including descriptions of each of the feature datasets, lists of feature classes, topology layers, geometric network, annotation feature classes, dimensions and the features’ overall hierarchy in the data model. The dictionary also describes the various features and entities (or non-geographic object classes) and their attributes in line with the CGIS Guidelines. It provided a description of all required domains, or lists of acceptable values, and subtypes of features, as well as the relationship classes that exist between the various features.
Data migration/backlog management and operational updating: The form, extent, accuracy, correctness and other characteristics of the source data are critical considerations to the development of the final integrated enterprise geodatabase. During the project KM (W) and its GIS consultant identified and evaluated all possible data sources in order to initiate the data migration process into the final KM (W) geodatabase.
At the project start-up, data was collected and catalogued for evaluation as potential sources of information to be incorporated into the data model. The data assessment considered the evaluation of these data sources, recommendations and specifications for data migration. Each feature class of the KM (W) data model was assessed with regard to its spatial and descriptive data available in all data sources.
The data migration process had several challenges associated with it, including that of backlog maintenance. However, KM (W)’s plan for the GIS migration project was developed in such a way as to leverage from the applications that were developed in order to ease and speed up the migration of all the CAD drawings and Shapefiles available into the final geodatabase. As such, KM (W) was developing applications and testing them in parallel with developing the data model and data dictionary. KM (W) data stored in CAD and previous ESRI GIS Shapefiles formats was mapped, through a one-time migration approach, to the final geodatabase format. At an intermediate phase of the migration, a data assessment based-on the quantity and numbers of assets and data migrated into an intermediate geodatabase was conducted. The assessment also ensured that migrated features were still as spatially intact as they were in the original data. This pre-processing stage of the data was necessary since there was no other point where it would be possible to compare the original data with the GIS data created. This is intrinsic to the data model itself, and can be illustrated in the following case. Valves in the GIS water distribution network will need to split the pipes to which they are snapped. However, this is not true for the previous structure. This will in turn create more pipes in the GIS database as opposed to the original data and a comparison based on the number of assets cannot be a made any longer. The same can be said about overlapping edges. These will decrease the number of edges in the GIS database as compared to the source files, since they are in conflict with the general geometric and connectivity rules that have to be upheld in the new GIS model in order to ensure the integrity of the data. Furthermore, a post-processing procedure is applied mainly to answer the need to review the whole network data topology and connectivity to detect the anomalies not implemented in the above referenced modules and in order to apply the necessary fix to rectify the network connectivity issues.
The process of data migration follows not only the spatial migration of the network assets, but also the migration of their associated attribute data. As such, KM (W) developed mappings between the old network features’ attributes and those in the newly developed GIS data model.
The fields marked in red were changed in the new data model in order to give them a more appropriate field name.
The migration of the data undergoes QA/QC routines to ensure the spatial and attribute integrity of the network assets, such as detecting overlapping edges, identifying disconnected edges, checking connectivity violations, identifying illogical attributes, especially values that are not part of the domains realised in the new GIS data model. These values can then be incorporated into the data model for KM (W) if there is a need to do so, easily and quickly. This will be done when the data is loaded to the final ESRI GDB Data Model Structure.
In addition, KM (W) needed to ensure that any backlog was collected and migrated into the new GIS model as well. As such, and during the one month that was then taken by KM (W) to assess the migrated data from the CAD format into the new GIS format, as well as the following month in which the applications were being used and tested on production, the backlog was data was being converted into GIS format, as well as any newly received data in order to ensure that no backlog would accumulate. Following this period, and once KM (W) users started using the applications themselves, the data was on track, as all backlog had been incorporated into the GIS data model and the KM (W) editors needed to only worry about the newly received data.
Customised GIS utility applications: The KM (W) applications were developed following comprehensive workshops in which functional and user requirements were collected and analyzed in order to properly define the applications’ functionalities, maintain the preferred system design and ensure the desired operability was achieved.
WebViewer Application: The WebViewer application is a web-based engine that offers KM (W) management the convenience of viewing and analyzing their water network over the web and generating condensed yet comprehensive statistical reports pertaining to the water data, independently from a desktop environment. The WebViewer application provides a highly scalable framework for GIS Web publishing that equally meets the needs of corporate Intranets as well as the demands of worldwide Internet access. The built-in GIS functionalities and spatial analysis tools, such as the zooming tools, the identifying tools, and the attributes querying tools are all integrated within this application to help the user spatially and quickly browse through and analyze the network data at hand.
In addition to these basic functionalities, the WebViewer application offers a set of customized, easy-to-access tools, which cater for more advanced spatial analysis on KM’s landbase map and water network data. These consist of a set of locator tools, such as the landbase, the landmark and the street locator tools, which enable the user to view, zoom to and identify specific locations on the map; other locator tools such as the water facility, apparatus, burst, feature and RPS locator tools are offered to allow KM (W) end users to generate a list of water components located in a specific area. With the help of the Attribute Query Builder tool, users are also offered the convenience of locating water and landbase features with specific attributes, while the Project Locator tool assists in locating areas where projects and construction works have been or are being implemented as a function of the corresponding project status. Last but not least, the statistical reports generation tool caters for KM (W) managers and decision-makers who are mostly interested in having access to condensed information on the status of their network assets and related features in a particular area, such as bar diagrams specifying the number of valves or fittings based on their diameter and water mains and pipes based on length and diameter.
In a word, the value behind the WebViewer application for KM(W) end users rests in its web-based access as well as its easy-to-use, comprehensive tools, which provide condensed spatial and statistical analysis results on the network data for KM (W) decision-makers.
Water Data Maintenance Update Application
The Water GIS Data Maintenance and Update (WDMU) application provides an editing solution geared towards the needs of the water utility’s end users.
ArcGIS Editor being the main engine for GIS data update and maintenance at KM (W), the capabilities of this technology were extended even further with the addition of several customized and sophisticated tools part of the WDMU application. These customized editing tools provide a comprehensive application for water utilities, allowing maintenance and viewing of data, specifically tailored for KM (W) and its specific business rules. The tools create a simpler editing environment for users and produce a smooth and unlaborious process to support users in drafting network assets.
The WDMU application consists of several tools, one of which is a Version Manager Module that implements a versioning strategy allowing users to simultaneously create multiple, persistent representations of the database without data replication, all while establishing a comprehensive and seamless versioning methodology, based on the business processes that involve editing. In an effort to simplify the editing environment for all involved users, KM (W) made enhancements to its business and operational workflows that include GIS editing. The Version Manager Module allows the establishment and enforcement of a specific version structure and set user roles to streamline the editing process and manage it efficiently. This is essential in ensuring that edited versions of the same geodatabase are posted to the ArcSDE KM Geodatabase, after resolving any conflicts that may have arisen between two versions edited at the same time by two different users. Since the Geodatabase inherently allows multiple users to edit the same version at the same time, reconciling, or merging two versions, may result in conflicts when the same feature or topologically related features are edited by two or more users, and the database is unclear about which representation is valid. KM (W) editors can now create versions and reconcile them once they are finished, however, editors’ versions will be posted to the ArcSDE via a super editor who is in charge of conflict resolution and version management among KM (W) GIS users and editors.
A Batch Reconcile/Post/Compress tool has been also deployed allowing the geodatabase administrator to reconcile/post versions and compress the Geodatabase from within a single interface launched from ArcCatalog.
Moreover, and in order to enhance the editing environment even further, KM (W) developed other tools as part of the WDMU application, that will assist users in the editing process, by adhering to industry standard architecture and programming environments. These tools were implemented in an easy-to-use interface that will facilitate the editing and sketching processes thereby, allowing the editors to reduce the time consumed during such processes. One such tool is the Custom Attribute Editor tool. Although ArcMap ships with an attribute editor tool that displays and edits all the attributes of features once selected, by default, the purpose of the Custom Attribute Editor tool is to display specific attribute information of the selected geographical feature(s) from the map in a more user friendly interface. As such, there is no need to show all the attributes in the display interface. The Custom Attribute Editor is supported by custom-built tools designed specifically for advanced attribute editing and analysis and are explained hereinafter.
The Utility Identifier Administrator provides users with a quick and easy way to categorize and prioritize attribute data for any of the feature classes that are part of the utility’s GIS database, as they see fit. This will allow users to sort attributes by categories according to predefined priorities between them.
Based on the attributes and their categories, the Utility Identifier tool, which is similar to the built in Identifier tool in ArcMap, identifies a selected feature on the map by returning its table of attributes and displays it in the Utility Attribute window, as shown in the following figure. The user has the possibility to display specific attributes and update the value of any field in the list based upon its dependencies. Furthermore, the attributes are displayed in their associated categories and sorted according to a predefined priority or logical manner indicated through the administrator tool.
The tool was integrated with the ESRI attributes editor in order to make use of the editing capabilities of both these tools. Other functionalities include the ability to zoom to the specified feature on the map and highlighting it in order to better assist the user in recognizing the feature by flashing it on the map.
The Snapping Manager tool allows users to easily connect features while drawing based on a predetermined and saved snapping environment. Snapping allows users to place features and connect them easily and precisely at preset distances. The users also choose to connect a new feature to a specific part of the feature (vertex, edge, or endpoint). With the Snapping Manager tool, and based on the default snapping saved in the GIS database, KM (W) gave the users a method to set and dynamically store the custom auto-snapping configuration as a default snapping environment for network (or non-network) features. This, in turn, relieves users from the need to change the snapping environment as each new feature is added. The interface of the Snapping Manager is displayed next.
The Custom Auto-Updaters automatically generate an attribute’s value at the onset of a specific event, such as adding or modifying a feature. The Auto-Updaters maintain metadata and attributes information in such a way as to logically and effectively insert attribute values for some of the features’ fields without the editor having to worry about filling them in manually, following a specific event. Hence, for example, once a fitting is created and snapped to a pipe feature, the auto-updaters will split the pipe edge and will capture the diameter and material of the fitting, among other common attributes, from the pipe feature automatically. This will, in turn, greatly increase user productivity in an editing environment.
Finally, the QA/QC tool provides KM (W) users with a tool for detecting any geometric connectivity errors or attribute and domain violations on a selected set of water network feature classes. This QA/QC tool also displays a listing of these violations in a convenient tree-like view. This tool comes in very handy in validating network and non network features based on defined connectivity rules, relationship rules, and domain rules. The displayed results clearly list the feature class name, field name, domain name, feature object ID, as well the inaccuracy type of each of the violations. KM (W) also gave the users the freedom to export the resultant violations from the Quality Assurance and Quality Control program to an excel sheet of their choosing, thus making it easier to share the results among the various stakeholders.
XData Tool: With the fast rapid economic growth and the dramatic changes resulting from expanded urbanization in Qatar over the past few years, the utility network of KAHRAMAA (Water) will surely be expanding to meet the growing water demand. As such, and in an effort to affirm KM (W) GIS system’s maturity by timely capturing major physical network changes, KM (W) developed several applications that will aid the utility in efficiently, accurately and quickly representing the abundant slew of information from the field, via shop drawings and contractors’ as-built drawings, into GIS.
With that in mind, KM (W) set standard specifications for contractors to draft shop drawings and as-built drawings in order to ensure compliance with its GIS/CAD standards and specifications, resulting in efficient and timely data migration from AutoCAD into GIS and guaranteeing data reliability and capturing of assets’ information and attributes. This in turn, would ensure the organisation of CAD data efficiently and the collection of valuable asset information from the CAD drawings.
As such, and to better assist the contractors and their AutoCAD draughtsmen, KM (W) developed an application, along with an associated template in AutoCAD format, containing the guidelines for sketching network assets and entering the assets’ information via a user-friendly interface incorporated into AutoCAD that will allow for the collection of the assets’ information as GIS-ready content. This tool enables the use of the CAD XData simply and easily through a toolbar that is installed in AutoCAD with a button for every required feature class from KM (W)’s geodatabase.
User Interface for entering the Gravity Pipe layer’s XData attributes
The tool provided a simple interface to the CAD XData including the needed configurable attributes of each asset, as well as the GIS feature’s subtype. The interface also supports domain values, or lists of valid and possible values for a particular attribute, in a drop-down list view. Since changes and updates on the GIS data structure happen frequently, the XData tool can be easily synchronized with the physical GIS model.
With the XData tool and its associated CAD template, KM (W) can now benefit from preset and configurable standard CAD Drawings from all its contractors which can be directly migrated into the KM (W) GDB, whereby both the network features and their corresponding attributes can be migrated simultaneously without the need for batch process preparation.
Survey and As-Built Drawing Verification/Positional Accuracy Check Application (SADV/PAC): KM (W) also developed an additional application that will provide its users with an easy solution for loading data from different formats (AutoCAD drawings, Survey Data in Shapefile format…) into the GIS geodatabase. Not only that, but this application also contains some Quality Assurance and Quality Control tools to detect geometric connectivity or domain violations after the data is loaded into an intermediate geodatabase, as well as performing positional accuracy checking on the survey data as compared to the data on the ArcSDE. The application can run as a standalone engine or integrated through the KM (W) Data Maintenance Framework.
Data Migration Tool used for Survey Data Migration
Furthermore, shop drawing and as-built drawings collected from contractors in AutoCAD format, can now be easily migrated into GIS data – which goes hand in hand with the XData tool that would have been used to capture the network assets’ attributes. The users can then directly migrate the CAD drawings into an intermediate geodatabase and then to the production ArcSDE GDB, of the same schema as KM (W)’s geodatabase. This is done through a predefined mapping file between the AutoCAD layers and their corresponding geodatabase feature classes, as shown in the following figure.
Once the loading is completed, users can QA/QC the results. As such, they can access the QA/QC functionality of the Application on either the migrated intermediate geodatabase or the ArcSDE production geodatabase, including geometry checks, checking for illogical attributes, disconnected junctions in the network, any overlapping edges or junctions as well as connectivity and attribute violations.
Data Migration Tool used for CAD Data Migration
Data Migration Tool used for Loading Personal GDB data into SDE GDB
The results of the QA/QC tools can be easily viewed by the user from either the standalone executable, or from within ArcMap, which will give users additional capabilities such as the advantage of zooming to the results or adding them to their map selections.
Sample Stand-alone QA/QC tool for Illogical Attributes
Data Migration Tool QA/QC Results Window
Sample QA/QC Results following Data Migration
Finally, and in order to complete the data migration exercise comprehensively, KM (W) added the functionality to perform Positional Accuracy Check (PAC) on the survey collected data. The purpose behind the tool is to allow for checking the positional and attribute discrepancies between surveyed data from the field, after it has been migrated to an intermediate geodatabase, and As-Built data available on the ArcSDE. This will enable the user from getting a holistic view of the accuracy of the data collected from the field versus the data that is currently available.
EDA application: The Engineering Drawings Archiving – Management Application provides a mechanism for users and administrators at KM (W) to have access to engineering drawings related to the different assets and facilities in the GIS water network. The application contains a complete management system that enables to add/edit/remove drawings in addition to managing links between drawings and their assets or facilities in the GIS water network. As such, the EDA application provides KM (W) end users with a dual convenience, embodied by the ability to administer and manage their engineering drawings over the web all-the-while creating and maintaining links between drawings and features/feature classes in the GIS system, in addition to the convenience of viewing and managing their GIS water network data, while keeping track of all CAD drawings that contain any particular feature/feature class in the network. For added flexibility, end users are offered the ability to administer the structure of the information managed by the application, and so by updating the categories of the drawings and their types, the names of the feature classes and subtypes as they appear in the GIS as well as the landbase locations and project names and numbers.
Training
KM (W) was wasting no time during the early phases of the project. In that sense, KM (W) users’ capacity building was initiated by training over staff on GIS as well as any applications that will be used by the users based on KM (W) data. As such, a comprehensive training plan was set-up by KM (W) to train its users on the various applications and tools and in different categories, ranging from GIS Basics, to editing tools, to tools specific to KM (W) like the XData tool and Survey and As-Built Drawing Verification/Positional Accuracy Check (SADV/PAC) applications, discussed in this paper.
The KM (W) users were distributed in groups based on their categories, for example, whether they are Analysts, Editors, Surveyors and Technicians, Water Engineers, Project Engineers, or Operation & Maintenance Engineers.
Each session was comprised of both lectures and exercises to apply the learnt knowledge. During the lectures, a demonstration of the various tools involved in the session was presented leading up to the applied and interactive exercises for the attendees to apply what they have learnt. Each of the sessions extended over a period of 3 days for 6 hours per day. These sessions mainly focused on basic GIS techniques and KM (W) specific tools that will assist the engineers in their day to day GIS activities at KM (W).
Conclusion
Kahramaa Water GIS technology migration project addressed issues related to data dictionary, database design, migration and deployment, latest technology application development and migration of existing applications, automation of data integrations from other corporate databases like SCADA and ERP, update existing procedures and specifications, development of archiving system for engineering drawings and training. It implemented GIS within KAHRAMMA with latest ESRI ArcGIS 9.3.1 GIS software.
Centralised data repository approach within this project achieved complete data migration to the newly designed GeoDatabase which has improved data update procedures. Developed metadata auto updaters, editing and quality control have enhanced data quality and maintenance activities. Development of Engineering Archiving system enhanced drawings retrieval and linkage to GIS features. It enabled drawing visualisation through developed web viewer. Water Web Viewer was developed using latest ArcGIS Server technology which displays data from the centralised data repository. Water Web viewer was published on KAHRAMAA intranet which is being used across KAHRAMAA departments and has contributed to corporate workflows for better decision making via information availability on fingertips.
