Django greyscales

Access the application here.

I’ve been learning lots about the django web framework recently as I was hoping to take some of the ideas developed in my PhD and make them into public applications that people can apply to their research. One example of something which could be easily distributed as a web application is the code which serves to generate greyscale image blocks from RGB colour images, a theme touched on in my poster at EGU 2016.

Moving from a suggested improvement (as per the poster) using a complicated non-linear transformation to actually applying it to the general SfM workflow is no mean feat. For this contribution I’ve decided to utilise django along with the methods I use (all written in python, the base language of the framework) to make a minimum working example on a public web server (heroku) which takes an RGB image as a user input and returns the same image with a number of greyscaling algorithms (many discussed in Verhoeven, 2015) as an output. These processed files could then be redownloaded and used in a bundle adjustment to test differences of each greyscale image set. While not set up to do bulk processing, the functionality can easily be extended.

web_out

Landing page of the application, not a lot to look at I’ll admit 😉

To make things more intelligible, I’ve uploaded the application to github so people can see it’s inner workings, and potentially clean up any mistakes which might be present within the code. Many of the base methods were collated by Verhoeven in a Matlab script, which I spent some time translating to the equivalent python code. These methods are seen in the support script im_proc.py.

Many of these aim to maximize the objective information within one channel, and are quite similar in design so it can be quite a difficult game of spot the difference. Also, the scale can often get inverted, which shouldn’t really matter to photogrammetric algorithms processes, but does give an interesting effect. Lastly, the second PC gives some really interesting results, and I’ve spent lots of time poring over them. I’ve certainly learned a lot about PCA over the course of the last few years.

web_out.png

Sample result set from the application

You can access the web version here. All photos are resized so they’re <1,000 pixels in the longest dimension, though this can easily be modified, and the results are served up in a grid as per the screengrab. Photos are deleted after upload. There’s pretty much no styling applied, but it’s functional at least! If it crashes I blame the server.

The result is a cheap and cheerful web application which will hopefully introduce people to the visual differences present within greyscaling algorithms if they are investigating image pre-processing. I’ll be looking to make more simple web applications to support current research I’m working on in the near future, as I think public engagement is a key feature which has been lacking from my PhD thus far.

I’ll include a few more examples below for the curious.

 

This slideshow requires JavaScript.

Sentinel bot source

I’ve been sick the last few days, which hasn’t helped in staying focused so I decided to do a few menial tasks, such as cleaning up my references, and some a little bit more involved but not really that demanding, such as adding documentation to the twitter bot I wrote.

While it’s still a bit messy, I think it’s due time I started putting up some code online, particularly because I love doing it so much. When you code for yourself, however, you don’t have to face the wrath of the computer scientists telling you what you’re doing wrong! It’s actually similar in feeling to editing writing, the more you do it the better you get.

As such, I’ve been using Pycharm lately which has forced me to start using PEP8 styling and I have to say it’s been a blessing. There are so many more reasons than I ever thought for using a very high level IDE and I’ll never go back to hacky notepad++ scripts, love it as I may.

In any case, I hope to have some time someday to add functionality – for example have people tweet coordinates + a date @sentinel_bot and have it respond with a decent image close to the request. This kind of very basic engagement for people who mightn’t be bothered going to Earth Explorer or are dissatisfied with Google Earth’s mosaicing or lack of coverage over a certain time period.

The Sentinel missions offer a great deal of opportunity for scientists in the future, and I’ll be trying my best to think of more ways to engage the community as a result.

Find the source code here, please be gentle, it was for fun 🙂

dainlptxkaajaaw

Photogrammetry rules of thumb

I’ve uploaded a CloudCompare file of some fieldwork I did last year to my website here. It uses the UK national LiDAR inventory data, mentioned in the post here. I think it espouses lots of the fundamentals discussed here, and is a good starting point for thinking about network design.

80% overlap

This dates way back, and I’m unsure of where I heard it first, but 80% overlap between images in a photogrammetric block with a nadir viewing geometry is an old rule of thumb from aerial imaging (here’s a quick example I found from 1955), and carries through to SfM surveying. I think it should likely be a first port of call for amateurs doing surveys of surfaces, as it’s very easy to jot down an estimate before undertaking a survey. For this, we should consider just camera positions orthogonal to the surface normal (see this post) and estimate a ground sample distance to aid us with camera spacing from there.

1:1000 rule

This has become superseded in recent years, but is still a decent rule of thumb for beginners in photogrammetry. It says that, in general (very general!), the surface precision of a photogrammetric block will be around 1/1000th of the distance to the surface. Thus, if we are imaging a cliff face from 30m away, we can realistically expect accuracy to within 3 cm of that cliff. This is very useful, especially if you know beforehand the required accuracy of the survey. This is also a more stable starting point than GSD, whose quality as a metric which can vary widely depending on your camera selection.

Convergent viewing geometry

Multi-angular data is intuitively desirable to gather, with the additional data comes additional data processing considerations, but recently published literature has suggested that adding these views has the secondary effect of mitigating systematic errors within photogrammetric bundles. Thus, when imaging a surface, try and add cameras at off angles from the surface normal in order to build a ‘strong’ imaging network, to avoid systematic error creeping in.

Shoot in RAW where possible

Whilst maybe unnecessary for many applications, RAW images allow the user to capture a much great range of colour within an image, owing to the fact that colours are written on 12/14 bits rather than the 8 of JPG images. Adding to this, jpg compression can impact the quality of the 3D point clouds, so using uncompressed images is advised.

Mind your motion

Whilst SfM suggests that the camera is moving, we need to bear in mind that moving cameras are subject to blur, and this is sometimes difficult to detect, especially when shooting in tough conditions where you can’t afford to look at previews. Thus, you can pre-calculate a reasonable top speed for the camera to be moving, and stick to that. We recommend a maximum of 1.5 pixels in GSD over the course of each exposure given the literature and as advised by the OS.

Don’t overparameterize the lens model

Very recently, studies have suggested that overparameterizing the lens model, particularly when poorer quality equipment is being used without good ground control, can lead to a completely unsuitable lens model being fit which will impact the quality of results. The advice – only fit f, cx, cy, k1 and k2 parameters if you’re unsure of what you’re doing. This is far from the default settings in most software packages!

Conclusion

I had a few more points in my long list, but for now these 6 will suffice. Whilst I held back on camera selection here you can read my previous camera selection post for some insight into what you should be looking for. Hope this helps!

Too much JPEG!

Having read lots about the JPEG algorithm of late in my investigations of image quality, and having written about it’s effects on image gradients in my last post, I though it would be good to include an entry about it in this blog.

Whilst I invite the more curious reader to delve into the nuances of the algorithm, which in closely related to the Fourier transform which I’ve written about previously, today I’ll be looking past the black box by testing the same key parameter as in the last post which the user has control over, the ‘quality‘ setting. One thing we will note, however, is that the JPEG algorithm operates on 8 x 8 discrete pixel windows, which is one of the more noticeable things when the algorithm is applied at lower quality settings.

Let’s have a look at the impact of varying the quality of a cropped portion (1000 x 1000 pixels) of an image:

The impact at the lower end of the JPG images is dramatic. As the quality is set to 1, 8 x 8 pixel blocks are essentially assigned the same value, and so the image will downgrade visibly. As we increase the quality parameter, this compression will start to disappear, but at quality 25 we can still see some degree of ‘blockiness’ due to the 8 x 8 pixel windows still varying to a large enough degree.

However, past around quality 50 the impact is much more subtle, and I tend not to be able to tell the difference for images cropped to this size. This elucidates the point: The JPG algorithm is amazing at the amount one can save, in terms of file size, in an image.

Let’s take a look at one more set of crops, this time the same image as above, but cropped to just 200 x 200 pixels:

The ‘blockiness’ is certainly evident at quality 50, and less subtle but notable at quality 75. I think the most astounding thing is the lack of perceptible difference between quality 92 and 100, given the file size difference. We can investigate where the difference lies using a comparison image (imagemagick’s compare function), where red pixels show different values. I will also include the difference image between the two cropped sections, which should offer some insight into the spatial distribution of pixel variations, if any exist:

So The mean variation between digital numbers for pixels in each 8 bit band is 1.5, but the file size saving is nearly 75%! The difference image shows that the digital number differences are concentrated in areas of high frequency information, such as along the cracks in the rock wall, areas which could be very important in delineating boundaries, for example.

While subtle, for work which involves photogrammetric precision these effects have not been so well documented – this is one thing I’m working towards within my PhD research. Oftentimes researchers will use JPEGs taken off the camera used, which can have custom filters applied prior to use, making reporting and replication more difficult. If we need to compare research done with different equipment under various lighting conditions on various days, this is one part of the research workflow which is crying out for standardization, as the effects, at least in the case of this one simple example, are clear.

For a visualization of a stack of every quality setting for the first set of crops, please visit this link to my website.

 

 

Digitizing Elmo (Windows)

Just today I participated in the first step of a collaborative project between departments at Kingston which involved a live demo – the product was a point cloud of of an Elmo/Monster toy as shown below. Here I’ll just go through the steps involved in the cloud generation, so hopefully readers can replicate it themselves!

IMG_3858.JPG

Our test subject

  • Images

For generating the data, I used my Canon 500D with a 50mm EF2 lens, so nothing overly fancy. I started by putting the subject on a raised platform in order to minimize the effect of reconstructing the ground, and used an aperture of f/5 or so to ensure the subject was in focus, but that the ground was not. A good example of precautions to take when dealing with low-texture objects is introduced in this blog post, but can often be limited by the amount of RAM in a computer. As I was somewhat time limited, I decided to forgo the accuracy of using a tripod and so used a fast shutter speed (1/30 s) with an ISO of 400 to compensate, and generally just tried to get a reasonable amount of coverage on the subject. I took some other images with a wider aperture (f/2) and faster shutter speed (1/50 s) also. I threw a few paintbrushes into the scene to generate a bit more texture.

The test dataset (Some images are very poor I’m aware!) can be downloaded here.

  • Model building

For convenience, Agisoft Photoscan (There’s a free 30-day trial) was used to build the model, though other open source alternatives exist, such as VisualSFM or MicMac. I’ve included a short slideshow on what the exact steps are in Photoscan below to hopefully make it easy to follow!

This slideshow requires JavaScript.

  • Level/denoise in CloudCompare

CloudCompare is an open source point cloud editing software available here. Because our model is exported without any coordinate system, it can’t tell up from down, but we can fix this! In CloudCompare we can use the leveling tool to quickly orient the model so it’s a bit easier to view. Another useful tool is the ‘Statistical outlier removal filter’ in tools-> clean-> SOR filter, though we’ll skip it in this case.

This slideshow requires JavaScript.

  • Preparing to upload

Potree is a free point cloud viewer which can be used to host datasets online. Here we’ll just used in it’s most basic form to get a minimum example out. This section gets a bit hairier than the others but hopefully it’s intelligible. We’ll need to download and unzip both the potree converter and potree in the same directory, making a new subdirectory for each; ‘Converter’ and ‘Potree’. Next we’ll add the model we saved from CloudCompare to the potree converter directory, renaming it to ‘model.las’. Then we’ll follow the slides below!

Note – the command for the fourth slide is ‘PotreeConverter.exe model.las -o ../../Potree/potree-1.3/model_out –generate-page model’

This slideshow requires JavaScript.

  • Upload to the web

While there’s instructions for Kingston Students on how to upload web pages, this is a general skill that is good to have. We use FileZilla FTP to log in to our server, and the idea is to upload the entirety of the Potree folder, which contains all resources necessary for rendering the scene. The actual HTML page where the model is located is stored in the directory potree-1.3/model_out/examples/ , and can be accessed by this once uploaded.

FZ.jpg

The final version of the model generated is viewable at the directory here –

EDIT: I’ve lost the original model for this, though will reupload a new version soon

If the Potree stuff is a bit too hairy CloudCompare is a brilliant software for toying with models, I recommend giving it some time as it’s an extremely useful software package!

  • Conclusion

This is a basic tutorial on how to rapidly get 3D models online using nothing but a handheld camera and a laptop. Including taking the images this process took around 20 minutes, but can be speeded up in many ways (including taking better but fewer images). The cloudcompare step can be skipped to speed up even further, but having a ‘ground floor’ plane is in my opinion almost a necessity for producing a model.

This is not intended to be best practice photogrammetry or even close, this is intended to give an overview on modern photogrammetric processes and how they can be applied to rapidly generate approximations to real world objects. These can be then cleaned and models generated for use in applications such as 3D printing, videogames or interactive gallerys.

  • Complete software list

Photoscan

CloudCompare

Potree

Filezilla