Technical Ramblings

In a conversation at FOSS4G, Schuyler and I sat down with Frank Warmardam to chat about the possibility of extending GDAL to be able to work more cleanly when talking to files over HTTP. After some brief consideration, he agreed to do some initial work on getting a libcurl-based VSIL wrapper built.

VSIL is an API inside of GDAL that essentially allows you to treat files which are accessed through different streaming protocols available as if they were normal files; it is used for support for accessing content inside zipped containers, and other similar data access patterns.

GDAL’s blocking strategy — that is, the knowledge of how to read sub-blocks of files in order to obtain the information it needs, rather than needing to read a larger part of the file — is designed to limit the amount of disk I/O that’s needed for rendering large rasters. A properly set up raster can limit the amount of data that needs to be read significantly, helping improve tile rendering time significantly. This type of access would also allow you to fetch metadata about remote images without the need to access an entire (possibly large) image.

As a result, we thought it might be possible to use HTTP-based access to images using this mechanism; for metadata access and other similar information over the web. Frank thought it was a reasonable idea, though he was concerned about performance. Upon returning from FOSS4G, Frank mentioned in #gdal that he was planning on writing such a thing, and Even popped up mentioning ‘Oh, right, I already wrote that, I just had it sitting around.’

When Schuyler dropped by yesterday, he mentioned that he hadn’t heard anything from Frank on the topic, but I knew that I’d seen something go by in SVN, and said so. We looked it up and found that the support had been checked into trunk, and we both sat down and built a version of GDAL locally with curl support — and were happy to find out that the /vsicurl/ driver works great!

Using the Range: header to do partial downloads, and parsing some directory listing style pages for READDIR support to find out what files are available, the libcurl VSIL support means that I can easily get the metadata about a 1.2GB TIF file with only 64kb of data transferred; with a properly overlaid file, I can pull a 200 by 200 overview of the same file while using only 800kb of data transfer.

People sometimes talk about “RESTful” services on the web, and I’ll admit that there’s a lot to that that I don’t really understand. I’ll admit that the tiff format is not designed to have HTTP ‘links’ to each pixel — but I think the fact that by fetching a small set of header information, GDAL is then able to find out where the metadata is, and request only that data, saving (in this case) more than a gigabyte of network bandwidth… that’s pretty frickin’ cool.

Many thanks to EvenR for his initial work on this, and to Frank for helping get it checked into GDAL.

I’ll leave with the following demonstration — showing GDAL’s ability to grab an overview of a 22000px, 1.2GB tiff file in only 12 seconds over the internet:

$ time ./apps/gdal_translate -outsize 200 200  /vsicurl/http://haiticrisismap.org/data/processed/google/21/ov/22000px.tif 200.tif
Input file size is 22586, 10000
0...10...20...30...40...50...60...70...80...90...100 - done.

real	0m11.992s
user	0m0.052s
sys	0m0.128s

(Oh, and what does `time` say if you run it on localhost? From the HaitiCrisisMap server:

real	0m0.671s
user	0m0.260s
sys	0m0.048s

)

Of course, none of this compares as a real performance test, but to give an example of the comparison in performance for a single simple operation:

$ time ./apps/gdal_translate -outsize 2000 2000 
     /vsicurl/http://haiticrisismap.org/data/processed/google/21/ov/22000px.tif 2000.tif
Input file size is 22586, 10000
0...10...20...30...40...50...60...70...80...90...100 - done.

real	0m1.851s
user	0m0.556s
sys	0m0.272s

$ time ./apps/gdal_translate -outsize 2000 2000 
    /geo/haiti/data/processed/google/21/ov/22000px.tif 2000.tif
Input file size is 22586, 10000
0...10...20...30...40...50...60...70...80...90...100 - done.

real	0m1.452s
user	0m0.508s
sys	0m0.124s

That’s right, in this particular case, the difference between doing it via HTTP and doing it via the local filesystem is only .4s — less than 30% overhead, which is (in my personal opinion) pretty nice.

Sometimes, I love technology.

One of the things that users often want to do within MapServer or other tools is to hook up to existing caches of high resolution aerial imagery, like VirtualEarth, Google, etc. to get an aerial basemap that is as high resolution as possible, preferably for free. This applies to cases other than serving web maps as well: for example, the EVS Islands blog has been using Google Earth imagery to derive datasets from for some time now, and is now acting as if there is a product that Digital Globe is selling which could fill this need.

Of course, many users also want a pony.

In some cases, the imagery you see in Google Earth is something you can’t even buy for use online — so with all the money in the world, you might find yourself short a high resolution dataset, or so you might think.

It turns out that for some areas of the world, there is a lot of *public* imagery available. Instead of depending on commercial providers, and the restrictions they entail, you can turn this free datasource into a source for your own maps. OpenAerialMap has begun the task of collecting and collating these images for you, so you don’t have to have your own WMS to see the latest data from the USGS dataset in your maps. (It’s still in its early stages, but it’s getting there.) Now, I understand that in the less enlightened parts of the world, this is not yet the case, but we’re getting there, slowly but surely.

But what happens when you don’t want a map, but instead want an image that you can include in your print-resolution magazine spread?

Open Source Software happens, that’s what.

GDAL has the ability to read images from a remote WMS server, and treat them as it does any other data source: which means that it can take the images, and convert them to any other format, from JPEG2000 to GeoTIFF. And it doesn’t even require any system dependencies for Windows or Linux: Just use Frank Warmerdam’s excellent FWTools. FWTools wraps up all the tools you need to work with geographic formats, be they raster or vector, into one neat package.

1. Grab FWTools: Linux, Windows
2. Unpack/extract it.
3. Run the install script in the root directory.
4. Grab http://openaerialmap.org/static/gdal_wms.xml and save it to the root of your FWTools installation.
5. Open a command line prompt in the FWTools directory.
6. Type: bin_safe/gdal_translate -projwin -93.246226 44.892179 -93.195672 44.866816 -outsize 9424 4728 -of JPEG gdal_wms.xml airport.jpg

End result? A 10000 by 5000 mosaic of the Minneapolis-St. Paul International airport, using data loaded from Public Domain USGS aerial imagery — some from Ramsey County, some from the Twin Cities overflight, both higher than 1M resolution (.15 and .3 respectively) within the past year and a half — and the speed to create it is essentially as fast as your connection to the net.

You can try this at home, and once you have the image, you can do *whatever you want with it*: The data involved is all public domain, and there is nothing to stop you from doing anything with the data. You can convert it to another format, derive vectors from it, print it out on a giant poster board, etc. No attribution or other sourcing necessary, because the data is public.

OpenAerialMap is all about the sharing of data that you should be able to get access to anyway, and as you can see here, in some cases the data that you wish you had you have access to already — you just didn’t know it yet.

One of the cool things about building a map on top of quality Open Source products is the ease with which one can extend the possible uses of your software.

GDAL 1.5 has gained the ability to read a WMS-C style remote tile cache directly: this means that any tool which uses GDAL as an image provider can read remote tile caches directly. The one that matters the most to me is MapServer: MapServer has the ability to load up a simple config file — http://openaerialmap.org/static/gdal_wms.xml — and use that in a mapfile: Using With MapServer.

Not running the bleeding edge GDAL? That’s fine: Instead, you can use the WMS that’s hosted by the OAM project: Using With WMS. This means that any WMS client — including those fancy schmancy ESRI tools that speak WMS. (There are at least some of them, I think.)

Your application only speaks in WorldWind? Well, we can take care of that for you: With a WorldWind metadata file, you should be up and running in no time. (In my experience, unfortunately, this is true in ossimPlanet, which has no problem with the file, but WorldWind itself seems a bit touchy about it once you zoom past the first couple levels. But I can only do so much.)

Using the new plugin-friendly TileCache that I rewrote last night, I was able to add support for the Mobile GMaps client — Using With MGMaps — in just 20 minutes. Similarly, a user reported success using the tiles with Maemo Mapper.

Of course, it can be used with OSM — either as a tile backdrop, or in the OSM editing application (JOSM): Using With OSM — due to the above plus TileCache’s TMS support.

As a result of the TMS support, it should be possible to load into Google Maps as well, though I’m really too lazy to test that one.

Overall, with just a few small tools — GDAL, MapServer and TileCache — it’s possible to build support for a wide variety of services from a single dataset.

In the last couple weeks I’ve contributed to two other OSGeo projects, besides OpenLayers:

• GDAL: Better documentation for new Web Map Service driver
• MapServer: Patched shp2img to allow changing image size.

Even though they are minor changes, it feels nice to participate outside the OpenLayers/MetaCarta bubble.

Having spent some time around SJ Klein the past week, I’m currently hacking on some mapping things for them. I’m also trying to find interesting activities that already exist to try them out: One of the ones I stumbled into is Kuku Anakula (Hungry Chicken). After playing it for a bit, I found it dissappointingly slow to respond to keyboard input. Since Python is a main component in OLPC, it’s written in Python, so I took it apart.

The result is that I refactored the code to make an order of magnitude fewer function calls, and in the process shaved about 50% of the CPU time off, so far as I can tell, when testing on my mac. I learned how to use hotshot, and have the original profiling data and the profile for my refactor. Getting the new version of the activity on the XO was very satisifying: the gameplay is significantly better, and changes the game from being more like a chore to more like a game.

Other things I did today for OLPC include building/installing OGR, and getting FeatureServer set up to run on it with a world borders shapefile. Still got a bit of work to do to build a world map/atlas with click-to-query attributes local on the box, but it’s not too far off.

If you didn’t see them via Slashdot, this set of photos of the OLPC in action is very cute.