<< Evolving beyond deriving three-dimensional information from images, photogrammetry is now being integrated into various activities involving geospatial data. Here’s a look at factors bringing about this change and the increasing applications of photogrammetry >>
Photogrammetry is now being integrated into many activities involving geospatial data. This is partly due to the nature of digital data which encourages convergence of data from different sources and partly due to the explosion of different sensors and platforms to collect images. Imaging systems range from traditional aerial photography to LiDAR and radar. Platforms now include unmanned aerial vehicles (UAVs), mobile terrestrial vehicles and satellites as well as the traditional aircraft. Handheld consumer cameras are also a valuable source and images from this type of camera, uploaded on social networks, are also becoming an important source of volunteered geographic information (VGI). Software is now reliable and user friendly and there is a division between high-end software from established vendors to less expensive products from new entries in the photogrammetry software market. Many mapping companies are finding life difficult at the moment, given the weak economic conditions, but the range of applications of photogrammetry is increasing and new companies are emerging to exploit these markets.
WHAT IS PHOTOGRAMMETRY?
Twenty years ago the answer to this question would have been technical in nature, explaining that photogrammetry is the science of deriving accurate three-dimensional information from images. Now the answer is more complex because, although the principle is exactly the same, photogrammetry is embedded in many different processes and the user may be quite unaware of what photogrammetry involves, or what it is. There is less distinction between different branches of geomatics and organisations pick and choose the technique which best suits their purpose. Diagram 1 shows the basic operation of photogrammetry. The central green boxes represent the core component of photogrammetry which requires an understanding of the geometry of images, the principle of collinearity and the importance of accuracy.
The situation is further complicated by the fact that technology has produced new sensors which do not involve images but are processed in a manner similar to imagery. Laser scanners actually measure distance and direction and produce a point cloud which is identical to a point cloud generated from matching stereo images. From these point clouds, we can compute digital elevation models at scales ranging from macro, with accuracies of less than 1mm, to global. The point cloud can be processed in the same way, whatever its source; however, it is generated by different techniques. Laser scanning is done from airborne platforms, often referred to as LiDAR, and from terrestrial scanners. We will consider both laser scanning and image systems as part of photogrammetry. Synthetic Aperture Radar (SAR) also measures distance and can be used to generate DEMs and orthoimages, so we will also touch on it, but in less detail.
Diagram 2 shows the different sensors, data types, processes and products which we may consider as part of the general area of photogrammetry.
The diagram shows the stages in product generation. Note the differentiation between georeferenced images and other products – georeferencing is done in two-dimension and does not require full orientation or a DEM and is not a rigorous product. Ortho images can be accurately registered with orthoimages generated from other data sources and with maps.
ECONOMIC BACKGROUND
There is an undisputed need for accurate information from images. The industry survey carried out by the American Society for Photogrammetry and Remote Sensing (ASPRS) in 2008 (Photogrammetric Engineering & Remote Sensing, 74(11)) revealed that of the 512 responses from 58 nations (mainly from North America and Europe), the majority require geolocational accuracy of better than 1m and that the need for this is greater than the supply, as shown in Graph 1. The ASPRS Industry Forecast was based on a survey in 2007, but still gives a good idea of the current situation:
Diagram 1: Basic operations of photogrammetry
Previous Forecast reports highlight that the demand for the highest levels of resolution was clearly not met; data users wanted higher resolution content. Phase V shows that this trend continues in the global market. Digital aerial sensors, as well as the continued operation of high-resolution satellite systems, have expanded the global demand for better than half metre data. Provision of data lower in spatial resolution than half metre is now an area of potential overcapacity. LiDAR, hyperspectral and IfSAR were identified as the three data sources that need exceeded current use. Data fusion appears to be considered more in these results than reliance on a single sensor or data source. Satellite sources of data versus aerial sources are used more in developing regions of the world. Restrictions on remote sensing data were viewed as a hindrance in every region of the world. However, the survey indicated that respondents from East and South Asia, Africa and Oceania felt that effects of restrictions on data, licensing and governmental controls most limiting on their activities. The most significant issues that continue to represent large challenges and opportunities with the industry are: the demand versus cost in an uncertain economy for high spatial resolution and new sensor data, both aerial and satellite, meeting the increase in demand for higher levels of education in GIS and newer imaging technologies, and the conflicting roles of national governments in developing remote sensing platforms and products while limiting access and use of data.
CURRENT TECHNOLOGY
Sensors are evolving all the time, driven by the equipment manufacturers. Users respond to this; and software developers follow with improved software packages. Here are the main sensors and software packages.
Sensors
Images now come from many sources, ranging from mobile phones to highly sophisticated digital cameras used on aircraft or satellites. In addition, the use of laser scanning is now widespread. Radar is still the preserve of specialist users, although companies such as Intermap have used airborne SAR to generate their Nextmap products and the private public partnership of Astrium and DLR is producing global DEMs from Tan-Dem-X. These sensors are used from many types of platforms, ranging from handheld and small UAVs to mobile mapping vehicles carrying cameras and LiDAR to survey aircraft and satellites. Although images are still the main source of data for mapping, a source in Leica says that LiDAR is now a more important source of revenue than cameras.
Digital Cameras
Digital cameras are now mainstream, although there has been a significant diversification from the original cameras from Leica, Z/I, and Vexcel. The use of smaller, less expensive cameras is also widespread. Table 1 shows the characteristics of the large and medium format cameras. These are cameras which replace the 230 x 230mm film cameras from the 1990s. They typically have 9,000- 12,000 pixels across track and collect data in panchromatic and multispectral bands. These large format cameras are complemented by smaller cameras from companies such as Trimble and Optech. All of these companies now offer a suite of products acquiring image data and LiDAR. Tobias Toelg from Trimble GeoSpatial tells of a company with a small plane in Africa which first bought a small camera and gradually upgraded to have a full navigation system with a camera and LiDAR.
Satellite sensors are an important source of images for photogrammetric processing. A full review of sensors and description of techniques can be found in ‘High Resolution Optical Satellite Imagery’ by Dowman, Jacobsen and Konecny, published by Whittles Publishing.
Airborne Laser Scanning (ALS) or LiDAR
Airborne laser scanners, often referred to as LiDAR, produce a point cloud which is generated from the measured distance and orientation of a point from the sensor. Sensors for ALS all have similar characteristics – all collect a number of returns from each pulse and also record the intensity of the return signal. The important parameters which affect accuracy and density of points are the scan rate or scan frequency and the pulse rate of pulse frequency and the flying height. Most systems allow selection of these parameters and manufacturers also offer a range of systems for different applications depending on the flying height. Most systems also record the waveform which allows detailed analysis of the vegetation cover which is very useful for forestry applications.
Table 2 shows some of the airborne LiDAR systems, including the Hawkeye system used for bathymetric measurement.
Terrestrial Laser Scanning
Terrestrial laser scanning (TLS) is one of the big success stories of recent years, with a range of scanners in the market now from the major companies including scan arms that reach difficult places and are capable of close range scanning with 0.2mm accuracy. A major new application of laser scanning is to provide data for Building Information Models (BIM). At the moment, the main use of TLS is to provide point clouds of interiors from which as-built drawings can be generated, but within the BIM concept there are other applications including site detail during planning and as-built survey during construction.
Diagram 2:The stages of generating photogrammetric products
Other Sensors
There are new sensors now in the market such as Photonic Mixer Device (PMD) ‘Smart-Pixel’ sensors which are able to capture a complete 3D scene in real-time without any moving parts. The sensor works by transmitting a modulated optical signal which illuminates the scene to be measured. The reflected light is detected by the PMD sensor, which is able to determine the timeof- flight per every single pixel. There are also cameras which record texture which would assist interpretation of images.
Another new device is the SpheronVR SceneCam™ which uses high dynamic range imagery and takes two panoramic 360º images from which a 3D scene can be generated. The SceneCenter™ database can be used to manage and archive the images and is used for many applications such as forensics and indoor mapping.
Other techniques which have been around for some time but have not yet been fully exploited are thermal imaging, images sequences, video theodolites and also combinations of sensors in sensor networks.
Graph 1: Geo-location Accuracy: Current Use versus Need
PLATFORMS
Mobile Mapping Platforms
Mobile mapping systems were first developed in the 1980s using fairly coarse navigation systems and cameras. The navigation systems have improved using GNSS positioning systems and inertial navigation systems. As the technology has improved, laser scanners have been added to the platform.
Cyclomedia has its own camera systems and has 30 years of experience in developing its camera systems. It has a partnership with the University of Delft in developing the camera systems and already has a ninth generation recording system. Companies like Navteq also operate their own vehicles but the new data source which may be called a sensor is data collected by users on the routes which they are driving which can be used for updating: this is volunteered geographic information (VGI) or crowd sourced data.
Oblique Imagery
Developments in inertial navigation systems and stabilised mounts have allowed the use of oblique imagery. Pictometry is a patented system which captures aerial images of all sides of a building as well as a vertical image from above. These perspectives can then be joined together to form 3D models from which measurements can be taken of dimensions and areas on the building. Swiss company Helimap System SA operates a versatile helicopter to capture oblique images. This has found applications in many areas such as ski resorts and mountain management, cliff mapping as well as the more conventional infrastructure mapping.
Unmanned Aerial Vehicles
The use of UAVs is having a big impact on photogrammetry. Professor Armin Gruen of ETH Zurich says, “It is safe to say that in the years to come, we will see an increase in UAV making activities, both in terms of hardware and software development, a most interesting and challenging area for research, development and practice. This makes a clear transition from toys to tools.” [Geoinformatica 1-2012]. What has made this transition possible are small digital cameras and powerful software which enable large numbers of small format images to be calibrated and oriented through use of photogrammetric block adjustment. UAVs have significant advantages over traditional air photography, including being highly transportable which allows for rapid mobilisation. UAVs can also typically operate below cloud coverage, making them less dependent on weather conditions. Systems such as Gatewing and the Sensefly Swinglet Cam come as a complete package, providing mission planning and processing software. They can operate at a flying height of 150 metres above ground level and can achieve a ground sampling distance (GSD) of 5cm. UAVs however are not without problems. The need to obtain permission to fly in many countries and ensuring safe operation and the safety of the public are important issues which require close attention. Academics and amateurs have been using UAVs for a long time, but the UAVs are now being used for commercial mapping.
SOFTWARE
A number of software packages exist for photogrammetric processing. All basic photogrammetric operations can be performed including automatic interior orientation and automatic aerial triangulation with automated blunder detection and selfcalibration. Some automation of feature extraction will be provided, although most of this requires human input. Table 3 lists the main packages with key features.
Recent improvements in software packages include improved speed using new architectures and new graphical processing units (GPUs). A key feature of many of these packages is their interface with packages such as ArcGIS and Autocad which enables users of these packages to integrate accurate imagery into their workflow.
Image matching is an important component in photogrammetric software and this is constantly being improved. The semi-global matching method, which matches every pixel and assesses the quality of every match, has given improved results. Remote Sensing Solutions GmbH and its associate company 3D Reality Maps are using semi global matching on satellite images to produce spectacular 3D visualisations which are widely used by the tourism industry, overlain with useful information about walking trails, ski runs and hotels, for example.
Another key development in software has been the introduction of bundle adjustment packages which can handle large blocks collected by UAVs or basic digital consumer cameras. Photosynth is a software application from Microsoft that can take large number of images and generate a three-dimensional model from the photos and a point cloud of a photographed object. Bundler is a similar package.
Table 1: Large & Medium Format Imaging Sensors
MAPPING ORGANISATIONS
Companies like Blom, EuroSense, COWI and Fugro continue to produce data for a wide range of clients. The main activities of mapping companies have not changed a lot in a long time, and the demand is still for DEMs, digital orthoimages and the generation of data from stereo restitution. However, the techniques for doing this have changed and LiDAR is used for larger scales, particularly corridor mapping, often using helicopters. Mapping from terrestrial sensors is becoming more important and many companies report increased demand for detailed survey using terrestrial laser scanning and mobile mapping. Prof. Dr.-Ing. Ralf Schroth, CMD and Operations Manager, Eastern Europe, BLOM Romania SRL sees UAV technology closing the gap between terrestrial and airborne mapping.
National Mapping Organisations (NMOs) are major users of photogrammetry. NMOs such as OSGB are important in developing new methods for national mapping by working with hardware and software providers to meet their specific requirements in a rapidly changing environment. China has completed an update of the 1:50,000 coverage of the whole country requiring the use of many data sources, primarily images, and the integration of these data to form a high accuracy, quality checked product. This, in turn, required techniques of image matching, semantic integration, generalisation, database management and conflict resolution. Automatic processing could be used in many cases, but intelligent human interpretation and judgment remained essential.
Table 2: Airborne Lidar
The field of use of 3D data is already very wide, covering a lot of applications. A great breakthrough of any new application is not in sight, according to Ralf Schroth. There is still a debate about the use of 3D city models. Ralf Schroth sees the market for 3D city models as already mature as many cities have their own data and commercial Internet suppliers like Google, Microsoft or Blom are even partially delivering these data for free.
There has also been an increase in the number of companies primarily concerned with collecting images over wide areas as a speculative venture and selling finished products, usually orthoimages, off-the-shelf to customers. These include satellite operating companies who provide visualisation and image analysis services. The products which these companies offer make use of photogrammetric software. Saab Rapid 3D Mapping™ is of particular note because of the near real-time production of 3D models. Many of these products are aimed at the defence market. Companies such as Astrium offer special services. Astrium has announced Go Monitor which will detect changes anywhere in the world using satellite imagery and image analysts from different thematic areas.
THE FUTURE
Research in photogrammetry is still focussed on automation; the large volumes of data available today make this necessary and the cloud assists in making it possible. A number of areas of research have been active for a long time, such as building reconstruction: these are difficult areas and will probably not move to production in the near future. There is also a lot of research being undertaken in automatic building reconstruction and extraction of road networks. Processing of point clouds is a continuing area for intense research. Data fusion, of images and laser scanning data, is another important research area. Registration of data sets is important as a result of the many sources of data available.
The rapid generation of photorealistic 3D scenes has been made possible because of new image matching software. While development of this will continue, one of the challenges which users face is whether to use laser scanning to collect point clouds directly or to use photographs and image matching.
We have seen that UAVs are emerging as very important source of data. The development of this technique will continue. The use of a swarm of such platforms is one possibility and the improvement of accuracy using new techniques for camera calibration and bundle adjustment will be developed.
Table 3: The principle photogrammetric software packages
VGI will also develop. Professor Dieter Fritsch of the Institute for Photogrammetry, University of Stuttgart, opines that VGI will become increasingly important and will be able to meet NMOs’ accuracy standards; data will be stored in the cloud and software will be developed to conflate VGI data from different sources. VGI will extend to 3D and Professor Fritsch is developing an ‘app’ for 3G phones and tablets which will allow a 3D model to be generated from photographs taken of the building or object. It is not clear how VGI will fit into existing business models, nor whether the information will be used by NMOs. The new catchphrase for photogrammetry is ‘the crowd and the cloud’.
New applications are being developed all the time. Bathymetric LiDAR of coastal waters is well established but work is now on to use LiDAR over inland streams to obtain accurate channel depth to enable modelling of river flow to obtain in-depth understanding of hydrological processes. LiDAR can also be used to determine water level in turbid reservoirs and to obtain information on forest structure.
CHALLENGES
The basic principles of photogrammetry are well established and an essential part of extracting information from images and data from laser scanning and radar. It seems unlikely that there will be major changes in the techniques and implementation of photogrammetric principles. There will undoubtedly be improved sensors and improved software and research will be needed to implement these. The standard mapping programmes for photogrammetry applications will be reduced in their traditional markets like Western Europe but will still be requested in developing areas like the emerging markets in Asia and Latin America and where no general cadastre information is available.
The challenges in doing this are summarised as:
- Restrictions on data collection, for example, restrictions on flying photographic missions, collecting information on roads for in-car navigation and flying UAVs.
- Continued supply of data from earth observation satellites at an affordable price.
- Availability of software which can be used by users unfamiliar with photogrammetry to generate their own specialist information, alternatively to sell photogrammetry as a commodity: users giving data to bureaus to produce the necessary products such as orthoimages and visualisations.
- Responding to emergencies and other urgent needs to provide data in real-time.
- Educating students on the principles of photogrammetry and preventing the reinventing of photogrammetry.
- Expanding the applications of imagery.
We can see that photogrammetry is an essential component in new developments in imagery, the use of UAVs and 3D VGI depends upon accurate rigorous photogrammetric models; many applications using terrestrial laser scanning, such as BIM, require the rigorous processing deriving from photogrammetry and of course there is a universal need for orthoimages and 3D data derived from images.
The article was originally published in Geospatial World, April 2012
