Zappa

Zappa is a python library which hugely simplifies the deployment of web apps, by using AWS lambda functions (‘serverless’). In essence, the library packages up an existing app, for example a flask application, and generates the endpoints required as lambda functions.

Why is this useful?

Running servers, at least from a hobbyist level, can be pricy, especially if the app requires lots of resources. Lambda functions are perfect for applications which are used as a demo, or things which are only infrequently required, as the consumer pays only for the time the server is active, billed in ms on AWS.

The downsides?

Generally speaking, lambda functions have a start up time that’s slower than a 24/7 server. When a request comes in for a given function, if the function has not recently been called, it will need to be created before the request can be processed. This can be quite a high overhead for functions with many dependencies. Zappa helps out with this by keeping the function ‘warm’ – periodically sending a request to the function to keep it from being stripped down. If you have a lambda function which gets bursts of requests, it can take time to spin up clones of the functions, limiting its effectiveness in production environments.

Examples:

The terracotta library, which I have mentioned before on here, is a great example of how effective lambda functions can be. NDVI time series by Vincent Sarago is another great example.

Advertisements

Geodiversity

Radiant Earth, whose CEO Anne Hale Miglarese I was lucky enough to see speak at the RSPSoc conference last year, partnered with Amazon in order to provide more ‘geodiverse’ training data for machine learning models. I think this is timely, as the AI4EO paradigm sets in. The availability of Sentinel 2 Analysis Ready Data from s3, as well as the ability for partial reads of this data using gdal, is the preferred option vs. Google Earth Engine for me for geodevelopment, so I’m delighted on these continuing data releases. I’ve been reading about rastervision, and look forward to sinking my teeth into this data with that as a supporting tool to see what kind of learning can be done!

Geodiversity is required for reliable modelling (source)

Past Sentinel 2 data, there’s so much opportunity  to shift thinking on how to develop AI4EO models, extending to other metrics  such as air quality (for instance from Sentinel3 SLSTR).

Keep an eye on this space – I’ll do an jupyter notebook or similar exploring the data once I get the chance!

Earth from space

The BBC have released the first of a documentary series focusing on Remote Sensing, and how it has changed/can teach us about out changing planet. It’s definitely a tough subject to fill whole episodes with, so the style is somewhat blended between satellite imagery, and storytelling on the ground, which makes for a very different kind of wildlife documentary experience.

I’m particularly curious as to how they produced the ‘superzooms’, which involve both zooming into, and out from, individual elephants in Africa to a continent wide view, as  they’re extremely well done. I’m a bit skeptical as to how much space cameras are involved in videoing shaolin monks, and am curious which satellites would even have the capability for this – maybe Vivid-i could capture a short video sequence, but the resolution wouldn’t really be high enough to discern individuals, and the recently defunct Worldview-4 would only be able to capture stills. Regardless, it’s really a well paced, emotional episode which I enjoyed immensely.

worldview.jpg

Sample from Worldview-4, available here

The series continues next week with an episode on patterns – the dunes of Namibia are an area whose beauty I only really discovered through sentinel_bot and I’m looking forward to learning more!

 

EGU 2019

EGU this year was a bittersweet affair, as I actually didn’t make the conference myself, despite having two posters presented on my behalf. I enjoy EGU, but this year my aim is to get to a few new conferences, and having already attended the amazing big data from space conference (BiDS) in Munich in February, I’m hungry to branch out as much as possible. Also on the agenda this year is FOSS4G (I have always wanted to go!) and RSPSoc’s conference in Oxford (this is one I think I will go to every year).

That being said, I did still submit two abstracts, both for posters sessions, with colleagues
of mine presenting on my behalf. The first was another extension of my PhD work, which focused principally on image quality of data collected in the field for use in photogrammetric work and it’s effect on the accuracy and precision of photogrammetric products.

This extension used new innovations within the field to further dive into this relationship, by using Mike James’ precision maps (James et. al 2017). In essence it investigates how stable sparse point clouds are when systematically corrupted with noise (in all of the camera positions, parameters and points within the cloud). His research tries to refine a big unknown within bundle adjustment using structure-from-motion, how do we account for variability in the precision of measurement when presenting results. Due to bundle adjustment’s stochasticity, we can never guarantee that out point cloud accurately reflects real life, but by simulating this sensor variation, we can get an idea of how stable this is.

egu2019_final.png

Pdf version available here

In all, the research points to the fact that compressing data is generally a bad thing, causing point clouds to be relatively imprecise and inaccurate when compared with uncompressed data. It would be interesting to extend this to other common degradations to image data (blur, over/underexposure, noise) to see how each of those influences the eventual precision of the cloud.

Secondly, I submitted a poster regarding a simple app I made to present Sentinel2 data to a user. This uses data from an area in Greece, and geoserver to serve the imagery behind a docker-compose network on an AWS server. It’s very simple, but after attending BiDs, I think there is an emerging niche for delivery of specific types of data rapidly at regional scales, with a loss of generality. Many of the solutions at the BiDS were fully general, allowing for arbitrary scripts to be run on raw data on servers – something comparable to what Sentinel-hub offer. By pruning this back, and using tools like docker-compose, we can speed up the spin-up and delivery of products, and offer solutions that don’t need HPCs to run on.

greece

Sample of the app

Lastly, I’ve simplified my personal website massively in an attempt to declutter. I’ve just pinched a template from Github in order to not sink too much time into it, so many thanks to Ryan Fitzgerald for his great work.

aboutme

That’s all for now, I’ll be writing about KisanHub in the next blog!

Web development is my job

I think it’s high time I restarted this blog, rather than let it disappear into oblivion. For the last year I’ve been working in Cambridge with an agricultural technology company called KisanHub, who aims to introduce a new wave of efficiency into crop monitoring and the food supply chain. I’ll do a separate post on the company, but for this blog post, I’m going to give general updates on skills I’ve acquired in the year, and what new skills I would recommend budding EO web developers should accrue in an attempt to demystify some of the jargon in the webdev/EO worlds.

Django/Flask

Python is the de facto standard language for geoscientific computing, and so it makes sense to learn a web framework in this language. Django and Flask are both good options for common taks in web development – a great example of their power of flask is terracotta, where you can go from local files to a full blown interactive environment in one command.

myimage_small.gif

Terracotta let’s you make an xyz server from a directory of geotiffs

Docker

By far and away the biggest revelation in the way I work, docker takes virtual environments taken to an extreme. In a naive sense, it lets you download and run a different computer – therefore stripping down all the barriers of system dependancies (gdal!), environment dependancies (pygdal!) and operating systems. The docker-compose command lets you run many of these computers in a network, and exchange information with one another. The kartoza docker-geoserver project is a great place to start for a demonstration of how easy it is to get a niche piece of software up and running. I generated a demonstration project here based on this network (source)!

myimage_small.gif

Docker–geoserver based project

Kubernetes

The natural extension of docker-compose, kubernetes let’s you efficiently run the aforementioned docker-compose network by declaring how much resource (cpu/gpu/memory) is given to each part of the network, and define rules for how the network scales/shrinks under certain conditions. It takes away most of the headache of having to manage servers and network configuration (my nginx config knowledge needs some work!), which I am very grateful for. Minikube can be used to run kubernetes networks locally, but seem to consume far more resources than docker-compose, so I usually use that at the final stages.

Geoserver

The grandfather of tileservers, I hadn’t used Geoserver much before this year, but am impressed with the very active community (I’m on the mailing list), and once bolted on to docker how easy it is to get started. I learned quickly it’s very easy to misuse, and so have spent the last few months properly learning about what it can and can’t do, and it’s REST interface. I think it’s a good starting point for geoscientists with little development experience, as everything can be fully controlled from the GUI, which forms a good base for beginning to manage it through REST.

Summary

I think these are the most significant technologies I’ve adopted in the last year, and would encourage any budding (or budded) geoweb developers to invest time into them. In the future, I’ll be writing about my job, my continuing research interest and my now significant commute (spoiler: it’s London -> Cambridge).

SentinelBot upgraded

I’ve been on a webdev kick since starting a new job, and have recently upgraded SentinelBot as a result. It now filters snow scenes less often and can handle atmospherically corrected products – I’ll be updating the github repository, and will be writing a post about my current job soon, but for now feast your eyes on some Sentinel goodness 🙂