Gamify it

I’ve been planning and chinking away at writing up the last three years of work into a coherent thesis in the last 6 months or so. It’s very interesting to look back at the reams of planning documents, literature reviews and interim results documents I’ve produced over this time!

Knowing what and how much to write on each topic is a bit of a dark art however; the initial targets I’ve set are very loose, but I think important to form some sort of structure to grow the report into. As a bit of a tongue-in-cheek joke I produced some ‘progress bar’ style bar charts, one for each chapter planned for the final report and have been updating day on day. The satisfaction gained from seeing them creep up has actually been surprisingly effective in getting me into a writing mode each day!

I’ve gone with a traffic light colour palette, the top bar indicates how many words I planned to write, the second the word count to date and the bottom the upper limit I’ve set myself. I know obsessing over word count is a massive waste of time, and I don’t worry about them too much at all, but couldn’t pass up an opportunity for some opportunistic data visualization!

Progress.png

Standard summary report I’ve been producing

A slippy map for Sentinel bot

Over the weekend I decided to expand on what was in sentinel bot‘s portfolio by having an automatically updating slippy map, which plots where the point for which sentinel bot has found an image is in the world, as long with the basic metadata of date acquired and lat/lon. I was trying to the leaflet’s marker-clusterer to work but to no avail, couldn’t quite get the knack of it! If anyone has experience with it I’d love to hear from you. I continued with just the pins nonetheless!

One really cool github project I used was this, which allows you to cycle through basemap providers for leaflet and provides you with the javascript code for inserting into your own map. I chose Heidelberg university’s surfer roads for no reason in particular, but may change this in the future. I think I’ll be returning to that github for future slippy maps!

In any case, the product is not perfect, but gives an interesting view of what the bots activities have been for the week it’s been active. I’m not trying to reinvent Earth Explorer, so will probably spend no more time on this, but it was an enjoyable pursuit!

Check the map here.

Data visualisation

Haven’t posted in the last while, so thought I’d make a quick post about some of my favorite data visualizations I’ve come across lately. The more I read about these the more it makes me want to improve the own graphics I produce, so if you’re looking for inspiration look no further! In no particular order:

Markov Chains

Basic as the visuals are, it really gives a good feel for what finite state problems look like. Can modify with your own code too!

Markov Chains

Baye’s rule/Conditional probability

From the same blog. Bayesian stats can be a bit daunting. Let this visualization of balls dropping through a filter calm you down as you need. Interactive to boot!

Conditional Probabilty

Fourier analysis

Just beautiful graphics putting simply what so many hours of reading couldn’t. Probably my favorite in the list due to the depth it covers!

Fourier analysis

Pathfinding

Not something I’m overly familiar with but have bookmarked because of how nice the graphics are to look at. Search is such a basic concept which is such a necessity to modern computing, I love the simplicity with which it’s presented.

Pathfinding

Blend4web curiosity app

Some might call it gimmicky, but I think the ability to be able to scroll through the cameras while the robot moves is just such a cool feature.

Curiosity

Potree

I can’t believe this is freeware. It’s amongst the best tools on the internet for point cloud viewing and the design is brilliant!

Potree

Seaborn

From the DIY category – seaborn is a front end plotting library for making graphs in python. It produces some beautifully crafted graphics! I love the joint plots.

Seaborn joint plot

Bokeh

Actually a pretty standard library it seems, I can’t believe how long it took me to find. I’m preparing some interactive graphics for upcoming conferences and bokeh makes it so simple to do! I particularly like the Lorenz example!

Bokeh Lorenz

Stamen mapping skins

Some very attractive base layers for using in your mapping needs. I think I’ll have to give making a base layer a go at some stage, but for now I can appreciate the possibilities…

Stamen

100,000 stars

Last on our list, one from the astronomers. An in browser interactive environment for exploring our stellar neighborhood!

100,000 stars

 

 

 

Leaflet maps, beautiful!

I decided to try and make some sort of map to help visualize the goings on of the movement of refugees through Europe, but got distracted in the technical aspects of putting together a web map and appreciating the effort that some open source developers have gone through to make really beautiful tiles for openlayers-based maps.

As such, I’ve put together a chloropleth map showing where the 130,000 extra refugees Europe is being asked to accept would go based purely on GDP. In essence, it’s a map of EU countries GDPs but may give a bit of context as to the practicality of certain countries offering asylum. It’s got some basic javascript components (Roll over for info, with highlighting), I’ve disabled the zoom as it wasn’t very relevant considering how sparsely the dataset is populated.

You can see it in my webspace here, a printscreen is attached! Here‘s a second similar one, showing number of asylum applications per country as of December 2014.

refug

50 posts!

For my 50th post I thought I’d just present another couple of geovisualisations based off of the UK national LiDAR inventory. The first is London, and is relatively complete. The point spacing at the most zoomed level is 7m as this was the best I could get without crashing my computer! I still think it’s quite interesting to see, and I hope to add some more functionality to it as time goes on. It’s viewable here.

Secondly I wanted to present a more topographically diverse region, so I searched for the area around Snowdonia, which had no data listed. Next, I searched around Ben Nevis, which had no data either! I then searched around Scafell Pike where there was a tile with data present, though you’ll see that it’s patchy, but somewhat interesting nonetheless! See it here.

A national 3D point cloud

I was searching various forks of Potree‘s web based point cloud viewer on Github, and happen to have stumbled across a fork with a python bindings for processing huge datasets. The example given is 640billion points, and has a live search feature that is pretty damn cool. The customisable color bar is also something I’m pretty excited about, as these ideas can be used for more than just height modelling – I’m looking forward to seeing some thematic models with this in mind, with perhaps an active legend which changes with whatever level of zoom (octree) you’re on. The Netherlands, however, is pretty flat, so I may try and adapt these ideas to the UK inventory over the weekend if I have time – either way I suggest you give the webviewer a look as it’s a technical marvel if nothing else!

I’ll include 1 screen cap showing the detail at the lowest level, you can make out individual houses and trees, really impressed by it all!

From the giant dataset at the lowest level, a building set amongst trees

From the giant dataset at the lowest level, a building set amongst trees

http://ahn2.pointclouds.nl/

Validating the UK LiDAR inventory/SfM products

On September 1st the geomatics section of the UK Environment Agency released it’s LiDAR inventory for free (including commercial use). I thought I’d take the chance to compare it with an SfM survey which was carried out on a relatively flat field in Damerham, UK. It was the subject of a georeferenced point cloud I generated previously (viewable here), and I was wondering what kind of differences we would see (or would expect to see) vs. what will presumably be the new national benchmark in an area which shouldn’t change much topographically.

First, I generated a geotiff using a new function in CloudCompare for the Damerham data. I then needed to find the tile reference where the field was located and requested that data from the environment agency’s new portal. I loaded both of these into QGIS and generated a difference DEM based on these inputs, shown below.

Difference in raster grids (LiDAR vs SfM survey)

Difference in raster grids (LiDAR vs SfM survey)

Next we can do the reverse. First we load our Damerham cloud, which was made previously and georeferenced in Agisoft’s SfM package. We then convert the ascii grid to a LAS file using one of the many very handy tools found in Lastools toolbox, las2las can do this for us. Now with the two clouds ready we can use the cloud-to-cloud distance tool to measure the difference between the two.

c2cdist_DSM

Histograms for cloud-to-cloud distance of LiDAR vs SfM clouds

Interesting! There seems to be a pretty big offset between the two. I decided to filter out all points <25cm and all points >60cm as it was such a small amount of the cloud, and generated a new extract which is presented below.

C2C_Filtered_With_hist

Cloud-to-cloud distance between LiDAR data and SfM survey using GCPs

It’s a bigger difference then I was expecting to see, and would love to test a few more SfM surveys in areas of simple topography which don’t change often to see how they fair against what will become the national LiDAR.

I had one other dataset to hand today with which to try, a terrestrial LiDAR survey of a coastal cliff in Wales, featured in this paper. Here‘s an SfM cloud I produced from using the imagery from that paper. I loaded the relevant tile into QGIS, but was required to do a reprojection as the survey was done in UTM30N, a different coordinate system to the OSGB system of the LiDAR data. After performing the reprojection I continued in much the same way, though I won’t present the QGIS screengrabs as they leave something to be desired! On loading both clouds into cloud compare I was greeted with quite the difference, as shown below.

Reproj

Offset after reprojecting

This is the nature of reprojections and coordinate systems, I just did a simple shift in Z to get it to line up more or less to where it would sit to visually check the fit, it looked pretty good!

Consti_translate

SfM cloud draped on the LiDAR

The LiDAR data (This is 1m, not even the highest!) is actually really amazing, it’s accuracy rivaling the result of this survey done just 3 years ago. I’ll include 1 more screen capture of the coastal town, bonus points for whoever can tell me what the strange streaking effect across the cloud is!

LiDAR_Coast

Town beside constitution hill