Vanessa Ann Macaspac, Mark Edwin A. Tupas, Julius M. Bangate, Romaluz A. Aligam, Ezzo Marc C. Hibionada, Micah Angeli F. Sayco
Public Assessment of Water Services, National Engineering Center,
University of the Philippines, Philippines
The Public Assessment of Water Services (PAWS) project is an independent monitoring tool of the Metropolitan Waterworks and Sewerage System (MWSS) to assess and evaluate the performance of water service concessionaires. The project employs GIS technology for the storage, retrieval, display and analysis of geospatially referenced data. PAWS has utilized a loose coupling of both open source and proprietary software to assist in consumer survey operations. Open source software lacks power tools for spatial analysis and lags in cartographic capabilities but excels in data integration and software distribution; conversely, proprietary GIS excels and lags in the opposite. GIS operations include 1) automated map generation for consumer survey operations, 2) spatial database management; 3) development/customization of GIS tools to simplify GIS-related tasks, 4) creation of desktop and Web-based data viewers for the project management and casual users; 5) spatial analysis and report generation for the purpose of decision-making. The paper will demonstrate that the fusion of open source and commercial software creates the most optimal system possible.
INTRODUCTION
On August 1, 1997, MWSS awarded concessions under 25-year agreements to two private companies, dividing the coverage into the Eastern and Western Concession Areas. Public Assessment of Water Services (PAWS) was instituted as a methodical tool to strengthen the monitoring activities of the Metropolitan Waterworks and Sewerage System– Regulatory Office (MWSS-RO). It was approved by the MWSS Board of Trustees to supplement the post-privatization regulatory arrangements. The PAWS Project is currently implemented by the UP National Engineering Center by virtue of the Memorandum of Agreement between the University of the Philippines and the MWSS-RO.
PAWS Project was created as a tool in monitoring and evaluating the two concessionaires of MWSS. The results of PAWS would: (a) assist the MWSS-RO in its decision-making process; (b) provide the Concessionaires with valuable operational and business planning information; and (c) increase public awareness and participation in the assessment of the level of water services [1].
Several components comprise the PAWS organizational team, as summarized in Figure 1. The project management, assisted by the administrative staff, plans and oversees the whole project operation and transacts with the MWSS-RO and concessionaires. The Consumer Survey (CS) group is responsible for the bulk of the survey operations: planning, survey implementation and handling of field enumerators, and recording the field data. The Service Performance (SP) group is in charge of gathering water pressure data from respondents through installation and monitoring of pressure data loggers. The Management of Information Systems (MIS) group is directly involved with the handling and processing of the data generated by the CS group, and the calculation of performance ratings and maintenance of the project website. The GIS team is tasked with the generation of maps used in consumer survey operations, data logger installations, and data analysis for reports, as well as the management of all geospatial data. Each group is headed by a specialist/consultant and backed by a team of research engineers who are primarily responsible for carrying out each team’s mandate.
The GIS team acts as a support group to all the other project components, providing assistance to each team’s geoprocessing needs: producing maps, conducting spatial analysis, training field personnel, and handling GPS data for the CS group; supplying maps for the SP group; validating performance ratings alongside the MIS group; and producing report maps for the project management as required.
OLD PAWS GIS INFRASTRUCTURE
Prior to the project’s third cycle (Year 3), PAWS’ GIS package consisted only of proprietary GIS software—two ArcView GIS 3.2a [2] licenses—manned by a single research engineer. Survey maps were generated piecewise from templates and no automated mapmaking system was in place. Customization extended only to the creation of a static desktop viewer, the use of which was limited to computer units containing licensed ArcView programs. Data layers were in shapefile format which made for the periodic updating and consolidation of data from several workstations a tedious and lengthy chore. The digitization of surveyed households and updating of spatial data were done manually, reproduced off map sketches.
The expansion of the project’s area coverage and succeeding workload triggered the need for (1) additional GIS research engineers; (2) a more efficient system that would significantly divert time spent on repetitive tasks to other endeavors such as development and customization of GIS tools; (3) setting up the groundwork for an enterprise GIS to better accommodate the influx of data from the survey operations and provide added value to the project, at a reasonable cost.
PROPRIETARY AND OPEN SOURCE GIS SOFTWARE
The project started to consider upgrading its existing GIS software around Year 3. Foremost among the possible choices was the latest of ESRI’s GIS software products at the time: ArcGIS 9.0 [4], a suite packing powerful GIS capabilities and geoprocessing functions. The project, however, decided to forgo the migration to the new software after weighing PAWS’ geoprocessing requirements—then limited to map production, georeferencing, map analysis and updating data layers—against the ArcGIS suite’s considerable cost and wide array of functionalities, many of which may not prove relevant to the project’s purposes.
It was resolved instead to continue utilizing ArcView 3.2a, maximize its capabilities and explore software development through ArcView customization and programming. The ArcView 3.x series’ native programming language–the object-oriented Avenue–allows for the creation of scripts, graphical user interfaces (GUIs), and extensions custom-built to facilitate specific operations such as map generation, digitization and data consolidation. With the software’s SQL Connect feature, non-spatial survey data from the MIS group’s MySQL [5] database are automatically extracted and merged with corresponding spatial data layers.
Another viable option that arose was to explore free and open source software (FOSS) to augment the project’s GIS competence. Specifically, PAWS wanted to introduce a more sophisticated database management system to host the operation’s increasing data accumulation, as the project has partially incorporated GPS technology into its enumeration methodology in its fourth cycle (Year 4) and has tasked its field enumerators with the geolocation of consumer household locations and other points of interest. The usage of FOSS GIS would substantially reduce project cost; FOSS also fosters easy software distribution and developer and user support from the GIS community. Active developer and community participation prompts regular software version updates and widespread availability of software documentation; also, most open data formats are supported. Additionally, the new software should be capable of connecting to and handling a spatial database management system that would allow simultaneous access to data.
The project considered several open source desktop GIS programs to be employed in conjunction with ArcView 3.2a: uDig [6], gvSIG [7], and JUMP GIS [8], among others. The team settled on Quantum GIS (QGIS) [9] for its relative ease of use; prevalence (indicative of a large community of users and contributors for support and documentation); compatibility with other software programs—GIS and otherwise—that extend its functionalities; and customization capabilities. The experience with ArcView and Avenue spilled over into QGIS–specifically, QGIS’ affinity with the object-oriented Python programming language and the interface design module set PyQt [10]. Similar to ArcView and Avenue, the combination of QGIS, Python and PyQt can be used to create programs and plugins to simplify and hasten repetitive operations or construct specialized functions.
As previously mentioned, PAWS also required a spatial database management system (DBMS) to administer all of its geospatially referenced data. The MIS group had been utilizing the open source DBMS MySQL since PAWS Year 2 to handle information gathered from the consumer survey operations. However, though MySQL has spatial extensions which are integral to the project’s GIS needs, the said spatial component hasn’t sufficiently matured yet to be used in the project. In lieu of this and ESRI’s ArcGIS Server and ArcCatalog, the project opted to implement PostgreSQL [11], another open source database program, in Year 4. The ability of QGIS to connect with PostgreSQL and its spatial database extension PostGIS [12] allowed for a multi-user environment that alleviates the integration of spatial data updates. This has been especially helpful in the consolidation and extraction of GPS data gathered from the enumeration.
Aside from desktop based FOSS GIS, Web based GIS servers were also introduced to the project. Web based GIS servers are used to “serve” GIS data over the internet (or intranet) through web browsers. Although limited in capability, GIS web apps could be deployed without additional software installations, which makes it ideal for distributed casual users. MapServer [13] and Geoserver [14] were explored for deployment while front end JavaScript programming using OpenLayers [15] were also researched.
To date, the PAWS GIS infrastructure is comprised of (1) commercial desktop program ArcView 3.2a; (2) open source desktop software Quantum GIS; (3) open source database management system PostgreSQL and its spatial component PostGIS; and (4) simultaneous instances of GeoServer and MapServer for web based applications.
PAWS GIS COMPONENT TASKS AND INNOVATIONS
A. AUTOMATED MAP GENERATION IN ARCVIEW
PAWS prefers the more established cartographic capabilities of ArcView 3.2a to generate maps for the project’s survey operations. The operation’s coverage size—amounting to about 1500 barangays for both Year 4 and Year 5—coupled with the demands of the daily survey deployment, prompted the GIS group to create a more efficient system to accommodate the production of all the necessary maps. PAWS took advantage of ArcView’s customization abilities to create extensions facilitating automated map production using Avenue. An Avenue script can be coded to produce hundreds of maps in under a few minutes, given a single map template and a list of attributes for each unit area. The maps are exported as project files instead of image files to preserve optimal graphic resolution. Minor editing of labels and other peripheral map elements take place after generating the project files.
It should be noted that the same system of automated map generation was explored with Quantum GIS, but while QGIS can be customized to support mass map production, certain cartographic attributes warrant further development to surpass or equal those of ArcView’s, for the project’s purposes.
B. SPATIAL DATABASE MANAGEMENT
The team incrementally shifted from using shapefiles and static editing to PostgreSQL + PostGIS in the middle of the project’s fourth cycle. This change was necessitated by the amount of data incurred during the daily enumeration that needs to be uploaded to the database—household locations, new landmarks, road network updates, and other additions and revisions to the PAWS base layers. The new system allowed for better data security, an enhanced system to back up data, uniformity of data layers utilized by the GIS users, and most notably, the simultaneous updating of data layers over the local intranet. Using QGIS, qualified data editors are able to apply parallel updates to the centralized database, ensuring that all GIS users are using the latest dataset version. PostgreSQL also serves as host and backend support to the PAWS data viewers, again providing users and viewers—classified and casual alike—with the real-time versions of PAWS spatial data. Full implementation of this system was applied in the succeeding cycle, Year 5.
C. CUSTOM GIS TOOLS USING ARCVIEW AND OPEN SOURCE SOFTWARE
Aside from map generation, the team regularly performs various other GIS-related tasks including but not limited to digitizing and updating data layers, management of GPS data, and landmark location. The tedious and repetitive nature of some of these tasks entails the need to customize and develop tools that simplify said undertakings. The tools are developed using both proprietary and open source technology; primarily Avenue customization for ArcView, and the combination of Python programming and PyQt interface design for Quantum GIS. Specialized query and locator tools have been developed for ArcView since Year 3, while plugins for the extraction and consolidation of GPS tracks and waypoints have been in use for QGIS since the middle of Year 4.
As shapefiles still figure largely in the project’s mapping component, tools were also developed for the seamless and almost instantaneous conversion of PostGIS base layers to shapefiles using Avenue scripting and the command line tool pgsql2shp which comes bundled with the PostGIS component. Correspondingly, shapefiles are easily convertible to PostGIS layers via QGIS’ packaged plugins.
D. WEB AND DESKTOP BASED DATA VIEWERS
Part of the GIS team’s core project deliverables is the creation of web and desktop based data viewers that would showcase the project’s output to its components and serviced sectors.
The web-based data viewer is developed using OpenLayers and GeoServer. Its principal purpose is to disseminate the results of the survey to the general public, specifically ratings on the performance of water concessionaires in their respective areas. Users are able to geographically associate the qualitative and quantitative nature of the performance ratings to their respective barangays/administrative units. Additionally, intranet-based data viewers for PAWS personnel were also developed. This includes mashups of Google Satellite imagery and PAWS data integrated for better visualization of field environments (e.g. respondent clustering is counter checked with imagery).
The desktop-based viewer is packaged as a Quantum GIS plugin, developed using Python programming and PyQt interface design. Devised as an extension as opposed to a standalone application, it makes use of QGIS’ native tools—tweaking the QGIS interface to display only the basic GIS tool set—as well as the custom instruments. The viewer caters to the different PAWS project groups and provides visualization of classified data to casual users and controlled access to the survey data. Additionally, it can serve as a decision-making aid to the consumer survey planning component and the project management.
E. SPATIAL ANALYSIS AND REPORT MAP GENERATION IN ARCVIEW
To ensure consistently unbiased results, GIS research engineers conduct visual analysis of the surveyed households by inspecting the tracks accrued by field personnel during enumeration. In the analysis, the research engineers check for the even dispersion of households within the barangay, adherence to the assigned sampling pattern methodology, and the correct situation of waypoints within residential areas. Majority/minority analysis is performed to verify data validity and detect data outliers; resurveys and recomputations are occasionally executed to confirm data integrity in case of extreme deviations. The occurrence of irregularities are immediately reported to and addressed by the CS group.
The generation of report maps and the consolidated data layer of surveyed households are the end requirements expected from the GIS component at every cycle’s conclusion. Report maps exhibiting the overall performance ratings of the concessionaires aid the concerned parties in visually interpreting the results of the survey at a glance.
CONCLUSION
Combining existing proprietary and exploratory open source GIS software proved to be highly effective and beneficial to the project, instead of solely implementing one kind over the other. The two systems complemented each other in that the features one system may lack thrive in the other. ArcView has preferable mapping capabilities; various open source components cover ground for database management, deployment of web and intranet applications, and easy distribution of applications to other project sectors. Both software components are customizable and essential for report generation and data analysis, resulted to efficient working hours and entailed almost no additional cost to the project. The existing system combination of proprietary and open source software–though still in its intermediate stages–already conforms to the structure of a functioning enterprise GIS; it satisfies conditions of a centralized data repository and the ability to support interdepartmental GIS data traffic, among others [16]. The current framework lays the groundwork for a mature and established enterprise GIS for the project’s future endeavors.
ACKNOWLEDGMENT
The authors would like to thank the Metropolitan Waterworks and Sewerage System-Regulatory Office and the concessionaires for their continued support to the PAWS Project, and the PAWS project management for constantly backing the GIS team’s development efforts.
REFERENCES
- Public Assessment of Water Services. (2011). Years I-V Accomplishment Report. Quezon City, Philippines.
- Environmental Systems Research Institute, Inc. (1992-2000). ArcView GIS 3.2a. California, USA.
- Public Assessment of Water Services, above n 1.
- Environmental Systems Research Institute, Inc. (2004). ArcGIS 9.0. California, USA.
- MySQL AB. (2005, 2006). MySQL Community Server (Version 1.2.15). Available from https://www.mysql.com/downloads/
- Various developers. uDig (Version 1.2.3). Available from https://udig.refractions.net/
- Various developers. gvSIG (Version 1.11). Available from https://www.gvsig.org/web/
- Various developers. openJump (Version 1.4.2). Available from https://www.openjump.org/
- Various developers. Quantum GIS (QGIS) (Version 1.7.1). Available from https://www.qgis.org/
- Riverbank Computing Limited. PyQt (Version 4.8.5). Available from https://www.riverbankcomputing.co.uk
- Various developers. PostgreSQL (Version 8.4). Available from https://www.postgresql.org/
- Various developers. PostGIS (Version 1.5). Available from ASIA GEOSPATIAL FORUM 2011
- OSGeo, et al. MapServer (Version 6.0.1). Available from https://mapserver.org
- Various developers. GeoServer (Version 2.1.1). Available from https://geoserver.org
- OSGeo, et al. OpenLayers (Version 2.11). Available from https://openlayers.org/
- Bowman, Scott. Evaluating Enterprise GIS Requirements. Retrieved from https://www.esri.com/news/arcuser/0205/evaluating1of2.html
