One thing that’s been on my mind quite frequently over the past year is the bridge between image processing, which is often carried out purely in digital numbers, and their relationship to radiance, which is more commonly used in wider Remote Sensing. Whilst digital numbers have huge memory advantages for applications in computer vision and robotics (integers are far faster to deal with then floating point numbers), radiance seems to be all to often left at the wayside. In this blog I hope to discuss why this is the case, and in what kind of applications I hope to see the culture change.
1. UAV Photogrammetry
I’ll start with the main one for me, applications in UAV photogrammetry, which I’m specialising in. Photogrammetry is a very old discipline which has seen new life through modern structure-from-motion applications. The concepts all come from the computer science community, and the entire workflow is executed on digital numbers – difference of gaussian feature trackers (SIFT, SURF) were designed with this in mind. Lots of these applications are being done using consumer grade cameras, with operators capturing JPGs and inputting them into software packages. The simplicity is very attractive and it’s easy to see why they have become so popular, but the problem for me lies in color balance. If we take a product from an SfM survey we have a rich data source, with detailed information of normals which can be provided to calibrate satellite images and provide a reference for change detection. Indeed, if you’re using an image-based survey to do something like map glacial dynamics, I feel like looking for a product which has absolute color information (W/m-2) with a degree of uncertainty associated with it is natural, and should be the norm. One of my favorite papers I’ve read in the last year, a landmark produced by Debevec and Malik which I’ve mentioned on this blog before demonstrated how practical this is, and how it just requires some extra calibration/preparation. Given this paper was written in 1997 and is still taught widely to this day (one example, albeit from 2009) in computational photography, I don’t know why we can’t generate radiance maps as standard given the concepts are so evolved and code is widely available. While their work focuses on high dynamic range imaging, you can do it with one image from a spectrally calibrated camera.
I have a sneaking suspicion I might be missing something. I’m aware of the difficulties in atmospheric correction but that isn’t enough to dissuade me from it’s generation. From what I gather, and I have spoken with one student who had done it, all you need is a monochromator or uniform source system for the calibration.
I can imagine a calibrated camera which can replicate the spectral bands of RGB from landsat or similar, and aid in satellite image simulations. This is done with high grade industrial cameras already, and I feel the extension to consumer/academic work is actually straightforward enough to do just for the sake of it.
2. Describing features
Features are a mainstay of image alignment, and are used in many mosaiicing algorithms. Can we describe features in terms of radiance? Again, this is something which computational/architectural photographers have been looking at, and I particularly like this proceedings paper from Kontogianni on the subject. He generates tone maps to display how higher dynamic ranges can produce more points for alignment algorithms, and while not the point of this blog entry, is a nice byproduct.
He uses Debevec and Malik’s implementation for radiance mapping, and demonstrates how straightforward this could be to apply to a UAV application. While we would lose the HDR aspect, the radiance map would still be recovered and very useful.
As well as this, we could perhaps test how changing bins for radiance to digital number conversions affect the detected features. Considering how well studied and explained something like SIFT is, I would consider a study like this very worthwhile, particularly in the geoscience domain. Can we start describing features at a feature level in terms of the radiance to digital number conversion? When is a feature not a feature? How much contrast is needed for a feature to be a feature? If this is at the mercy of how the jpg is binned then it’s clearly very important. Again, I feel like I might be missing something and would love to know if this is being done anywhere.
Lastly, while radiance maps (instead of digital number orthophotos) would be much larger to store, in terms of generating an accurate historical database for SfM work which allows for studies to be directly compared and updated as algorithms evolve, I feel like it would be worth it for higher grade studies. Considering how important some of these datasets being generated at the minute will be for climate models/baselines for future studies, providing and databasing everything correctly for not only replication, but extension for accurate comparison to other datasources should be a focal point of modern geomatics in my opinion.
Absolute units are the basic unit for describing the world, and while such care is taken in many domains to impose SI units, nobody seems too concerned with it for photogrammetry applications. I imagine a future where due to the level of detail provided from the historical record that when higher-level object detectors become more developed we won’t have to scratch our heads over EXIF files and try and back project/guess what the radiance mapping is. Absolute units are the way to go on all fronts!
I hope this entry is of some interest to any readers, and I would love for anyone with literature suggestions to get in touch!