Technical Ramblings

One of the most difficult thigns to do in time of disaster is to quickly organize, marshal, and present resources. This applies across all aspects of disaster response — whether it be managing and distributing food, organizing volunteers, or setting up technical resources to assist with the relief effort.

The last is the field I obviously have the most experience/ability to help with, especially with regard to mapping. In past situations, I have put some of my map expertise to work in helping to create a resource for the disaster; the last significant case for me was in 2007, when I managed a ton of imagery made available as part of the efforts with regard to the San Diego wildfires. (That map is still available, though it’s a bit worse for the wear at this point.)

When the Haiti Crisis happened, I let it slide; I figured that someone else would step up to manage the data this time. After a while, though, I saw an increased number of imagery sources, and little coherent organization of the resources by a single party — one of the key things that made the 2007 fires map successful. As a result, and combined with some data that was being more narrowly published, I decided to set up a map. The first day I did any significant work on this was over the weekend of the 15th.

At first, the map wasn’t particularly great; it was primarily just a tool to view a bunch of satellite data that was being made available. This was primarily just a quality control check for users of OSM who needed access to the data to complete the map of Haiti. Over time, more data became available — and more importantly, the OpenStreetMap map data became a primary map for the area and rescue efforts. Suddenly, the Haiti Crisis Map — then just the “UAV map” — was being used more and more.

As more and more data became available, the old map, using a simple OpenLayers layer switcher, became unwieldy; never a user-friendly layout to begin with, adding 20 layers to an OpenLayers map with an unplanned mix of base and overlay layers leaves much to be desired.

By Wednesday, it was clear that the hodge-podge of available disk space attached to the hosting machine wasn’t going to cut it; though we started with just over 4TB available spread over 3 different drives, managing the data was becoming unwieldy at the same rate as the UI. Thankfully, by Wednesday the 20th, John Graham was able to get access to another Sun X4500 and set it up, giving us a clean 16TB drive to put new and old imagery on. (About 6 hours later, the NFS machine to which all of the current data was stored began to fail, most likely due to heavier than normal load on the machine; I spent most of that day moving data off the old drive and onto the new.)

In addition to the data migration, at this time, Aaron Racicot was able to step up and offer his help in building a GeoExt based UI for the map. His efforts turned my hack into a reasonable UI for browsing the map, and it is really only because of that that I was able to keep going.

Over the weekend, at CrisisCamp, I was able to add additional features to support Ushahidi; the code was moved into Github, haitibrowser. In the middle of this week, the code was integrated into APAN, the All Partners Access Network, to support the efforts of SOUTHCOM in maintaining a high quality Central Operating Picture of events in the area.

Over the past two weeks, data has continued to pour in, in the hundreds of gigabytes a day. This is in part thanks to the wonderful availability of imagery thanks to the generosity of the commercial providers, in addition to the data made available by organizations like NOAA, companies like Google, and more. The extremely high quality imagery produced by RIT/ImageCat/WorldBank, for example, is an example of what is possible with the hard work of people with great hardware and a great team.

Using my knowledge — gleaned from my efforts in the earlier days of OpenAerialMap — I have been able to process this data and make it available as tiles and WMS to all consumers, primarily targeted towards OpenStreetMap editors. Over two dozene layers are available via what is now called the Haiti Crisis Map, each one adding a different viewpoint of data. In addition, the map contains links to other files like KML collections from Ushahidi and Sahana, and as recently as yesterday, gained the ability to create your own layers, which you can access in the map and provide as a link to someone else, as well as export as KML.

As part of the process of making the site more readily available, it is now available from haiticrisismap.org.

The most difficult part of this is attempting to manage the large sources of data. Thankfully, the resources that I have available have allowed me to be a bit lax in my conservation of disk space, CPU time, etc. Many thanks to CalIT, SDSU/SDSC, and Telascience for organizing these resources. In addition, a lot of the ‘hard work’ in the UI has been done by Aaron Racicot of Z-Pulley. I’ve done a lot of minor work, but the major UI layout and work has been done by him.

Thankfully, I’ve had the support of a lot of good people in this effort, and a lot of good tools to use. Using GDAL + OSSIM in the background for image processing, MapServer + TileCache for mosaicing and serving, OpenLayers + GeoExt for a UI, and OSM for a base map data layer have all made this effort possible.

The haiticrisismap will continue to see improvements. It shows a lot about what a dedicated small group of people can do with an investment when properly motivated; I can honestly say that because of the resources made available through these efforts, we have saved lives. Whether it is through maps produced through OSM being loaded onto Volunteer GPS systems, or the use of the data to determine an accurate location in a map by Ushahidi volunteers, this tool has been an effective aid to the relief effort in Haiti, and will continue to do so as much as is possible in the coming days and weeks.

I use the logged_in_or_basicauth snippet for a lot of my work, and had had some problems with it since I started using mod_wsgi in place of mod_python. Thanks to this post, I now know why my basic auth under mod_wsgi isn’t working: lack of WSGIPassAuthorization On in my Apache config.

Thanks to the author of that post! Also, thanks to Google, since without it, I’d never have found it.

PowerPoint is a way to make gibberish look important.

— my 12 year old daughter, Alicia

For a long time, I avoided MrSID like the plague. After trying to do *anything* useful with it, I finally gave up; the requirement for old versions of gcc, non-working on 64bit, etc. really gave me a negative impression of the SDK for MrSID reading. This was especially painful when working with OpenAerialMap, since MrSID has a practical lock on the market from ortho imagery datasources. (There are exceptions to this, but they’re usually JPEG2000 data, which was even worse to work with with the tools that I use, in general.)

However, after a set of discussions yesterday, I sat down and had a bit of a discusion about it, and Frank said that MrSID building in GDAL had gotten much easier. I didn’t really believe him, but I had the DSDK handy for other reasons, and reading the build hints, it was supposed to be easy.

Thinking I was going to prove Frank wrong, I started building. I did ./configure --with-mrsid=~/Downloads/Geo_DSDK-7.0.0.2167; confirmed MrSID ‘yes’ in the output, then make.

3 minutes later, I had a gdalinfo and gdal_translate built on my Mac with MrSID support.

My historical problems with MrSID are completely irrelevant: the effort in the new SDK to support more platforms has clearly worked, and I can say that building MrSID support even on the Mac is trivial. A big thumbs up to the LizardTech folks for their effort in this regard — and to people like Frank and Michael for egging me on into learning this about the DSDK in the first place.

Yesterday, I got to sit down and do some real performance testing with the MapServer folks. After rebuilding a local copy of the Boston Freemap on my laptop, I was able to share it with Paul, who ran it through Shark to find out where the performance killers are. The one thing we found was that this 5 year old MapServer ticket was negatively affecting performance on maps with many labels: The labelling code in MapServer right now, if you’re using outlines, draws each glyph 9 times in order to get a nice outline color. After determining this, it was determined that we are going to be working with the GD maintainers to add the support described in #1243 to GD to use Freetype’s internal stroking code to get the same behavior. (At the time, in Freetype *2.0.09*, there was a bug in this code; but we’re now on 2.3.8, so that bug has been long fixed. :)) This change will likely give a 20% increase on map drawing with many outlined labels, as can be seen in maps like the Boston Freemap.

After this, we sat down with MrSID and GDAL/MapServer to figure out if there were performance problems there. One thing we found was that the MapServer code drawing one-band-at-a-time means that there is a significant performance hit. In addition, some other performance enhancement techniques are being looked into at the GDAL level by Frank, thanks to the help of LizardTech developers participating in the sprint. He’s currently looking at improving the way that GDAL reads from MrSID, and was already able to achieve a 25% speed increase by simply changing the size of the internal GDAL buffer size for reading from MrSID to GeoTIFF. More documentation and experimentation is still in order, but there are some possible optimizations to investigate there for users of the library.

We then had a great dinner at Jack Astor’s.

Thanks to our sponsors for today: Bart van den Eijnden from OSGIS.nl and Michael Gerlek from LizardTech — performance improvements in MapServer and GDAL access for label drawing and MrSID are potentially big wins for many users of MapServer.

Mapnik is a great tool. It allows for all kinds of neat toys to happen, and the recent work in SVN has really opened up the possibility that Mapnik might be a potential solution for a rendering engine in a lot of areas that it has previously left alone. (Support for reading OGR datasources, sqlite/spatiallite plugins, etc. are all great developments that look likely to be released in the upcoming 0.6 release.)

In prep for the OpenStreetMap Mapping Party this Saturday and Sunday in Somerville, I was working on printing a big map to bring with me. A friend at the Media Lab was gracious enough to help me out.

Using Mapnik, it was trivial to produce a large — 29750 x 29750 pixel — PNG image. This was designed to fill up the 49.5″ by 49.5″ printer space at 600 dpi.

The printer prefers PDF, PS or TIFF. I was able to take that PNG and convert it to a TIFF — but the resulting tiff was DEFLATE compressed, and the printer help only mentioned LZW compression. I decided to fall back to trusty GDAL to try to fix this. I found that the imagemagick-converted TIFF had one giant block — and GDAL was not pleased with this at all. (Its internal un-blocking scheme doesn’t work with compressed tiffs.)

Thanks to a suggestion from Norman Vine, I was able to use the ossim image copy program (icp) to convert this giant tiff to a tiled tiff which gdal could easily read: icp tiff_tiled -w 256 image2.out.tiff image.icp.tiff. Once I had done this, I recompressed the tiff using LZW compression with GDAL: gdal_translate -co COMPRESS=LZW image.icp.tiff image.lzw.tiff, and was able to upload the 3GB image to the printer.

All in all, took a bit more than I was expecting, but I’ve got a 4ft by 4ft map to bring to the mapping party this weekend. In the process, I also got to wanting magnification in Mapnik… which is amusing since just 24 hours before, I’d read a thread on the MapServer list and couldn’t imagine for the life of me why such a thing mattered.

Looking forward to showing the map off to local OSMers at the mapping party!

Interested in OpenStreetMap? In the Boston area — or considering travelling here with your lucky companion for Valentine’s Day? Come to the OpenStreetMap mapping party, in Somerville, MA on Feb. 14th and 15th, and help put your house on the map… or anything else you might run across. I’m hoping to be there — in part to meet other OSM interested people in the area, in part to defend my actions in uploading all the houses in the metro-Boston area and making the map quite pretty to look at, but annoying slow to edit.

Sean Gillies just wrote a great post on the use of Python decorators to help you write prettier code. he didn’t quite go into what decorators are underneath though, which is something I think that it’s important to realize to understand how decorators work.

Decorators are just function wrappers around your functions. In all the languages I use on a regular basis, functions are just another variable: they can be passed around in the same way objects can. The pretty syntax for decorators is just a way to say “Pass this function into the function I’ve just defined, and return me a new function, called the same thing as my old one was.”

If you are happy with only supporting Python2.4 and above, that’s a great way to work. Sadly, not all of us are: for example, the current release of Jython is still at 2.2. 😉 More seriously, people who are maintaining older systems may not have newer Python functions available yet.

That doesn’t mean you can’t get the benefit of decorators — at least, I’ve never found it to mean that. Instead, it just means your code is a bit less ‘pretty’ to look at. Instead of functions being ‘deocrated’:

@logprints
def work():
    print "Starting some work."
    print "Doing some work ..."
    print "Finished the work ..."

You can simply wrap your function in the decorator:

def work():
    print "Starting some work."
    print "Doing some work ..."
    print "Finished the work ..."
work = logprints(work)

The idea of passing functions around is one of the things that took me a while to get used to, but learning it has helped me with a lot of code since then.

TileCache and FeatureServer don’t use decorators, specifically because they seek to support older Python versions. If you’re writing code that’s only forward looking, using all the advanced features of Python may fit your bill. But when you find yourself on an old machine some day, where all you have is Python 2.2, sometimes it’s nice to know a little bit about what’s going on underneath.

Earlier today, I tried Jython for the first time, because I’m doing some work that may involve interactions with Java libraries in the near future. Jython, which I’ve always avoided in the past due to an irrational fear of Java, is “an implementation of the high-level, dynamic, object-oriented language Python written in 100% Pure Java, and seamlessly integrated with the Java platform.” (I love projects that have great one-liners that I can copy paste.)

My goal for Jython was to do some work with the GeoTools EPSG registry code related to SpatialReference.org. Sadly, I didn’t get that working, but in the process, I learned that Jython now has a beta version which is up to Python 2.5 — much newer than the 2.2 that had previously been available.

With that in hand, I decided to see if I could get some of my other Python projects running under Jython. I’m the maintainer for both TileCache and FeatureServer — two pure Python projects. Theoretically, these projects should both work trivially under Jython, but I’ve always had my doubts/fears about this being the case. However, it turns out that my fears here are entirely unfounded.

I downloaded the FeatureServer ‘full’ release from featureserver.org: this includes the supporting libraries needed to get a basic FeatureServer up and running. I then tried to run the FeatureSever local HTTP server… and it worked out of the box. I was able to Load the layer, save data to it, query it, etc. with no problems whatsoever. Java has support for the DBM driver that FeatureServer uses by default, so out of the box, I was able to use FeatureServer with Jython without problems.

Next came TileCache. TileCache was originally built to support Python all the way back to 2.2, so I wasn’t expecting many problems. Getting it running turned out to be almost as easy: the only code modification that was needed was a minor change to the disk cache, because Jython doesn’t seem to support the ‘umask’ method. Once I changed that (now checked into SVN), Jython worked just as well with TileCache as it did with FeatureServer.

Clearly, there are some things which are less trivial. The reason that these libraries were so easy to use is because they were designed to be low-dependancy: TileCache and FeatureServer default paths are both entirely free of compiled code. Using something like, for example, GDAL Layers in TileCache, would be much more difficult (if it’s even possible).

However, this presents some interesting capabilities I had not previously thought of.

For FeatureServer, this means that it may be possible to write a DataSource which accesses SDE using the ArcSDE Java API, ESRI’s supported method for accessing their SDE databases. One of the purported “holy grails” of the GIS world is RESTful access to SDE databases via lightweight servers — Jython may provide a path to that, if someone is interested in it. (It may be that this has become a moot point with the release of the ESRI 9.3 REST API — I’m not really sure.) This may be a waste of time, but the fact that it *can* be done is interesting to me. Edit: Howard points out that ArcSDE read/write support exists in OGR 1.6, so this is a moot point; you can simply use OGR to do this without involving Jython/Java.

I think this might also speak to a possibility of having better answers available for people who want to use things like FeatureServer from Java platforms (though I don’t know enough about jython to be sure): the typical answer of “use GeoServer” is great, but to be able to provide something a bit more friendly would be interesting. Thankfully, the Java world is largely catching up to the advances made in TileCache/FeatureServer, so this is also less urgent than it has been in the past.

In the end, this was likely simply an interesting experiment. However, it’s nice to know that the capabilities to do things like this within Jython are improving, and that Jython continues to advance their Python. The 2.2 release being the ‘current’ one still is disappointing, but seeing a 2.5 beta available is an exciting development.

As I said, the current version of FeatureServer works out of the box with Jython, and I’ll be doing a TileCache release shortly that will work with Jython out of the box as well. It’s neat to see more possibilities for using these libraries I’ve spent so much time on.

So, the hotel I’m currently staying in uses a classed system of internet access: you can pay $n for so many hours of internet at a certain ‘service level’.

After some experimentation, it seems that what this actually does is put you in a QoS bracket for your HTTP traffic, where you’re apparently grouped with other people in the same bracket. “Bronze” gets 30k downstream, “Silver” gets 60k downstream, and “Gold” gets to max out the connection. Only HTTP traffic is limited in this fashion: other traffic simply falls into the Gold bucket by default.

What does this mean for me? Well, currently someone else in the hotel in the ‘gold’ bucket is using up all the bandwidth. As a result, I can’t use the web while in the Gold bucket. I can, however, get perfectly usable bandwidth when using the Silver and Bronze buckets.

The worst part of this is that most of my traffic that I care about goes through ssh — and ssh isn’t monitored/blocked, so it doesn’t get into a different QoS bucket. The end result is that I can use the web — but not if I use the highest service level. And no matter what I do, I can’t use ssh at all.

What a pain. I’m *so* looking forward to being back in the states in another day and a half and having usable internet again…