Technical Ramblings

So, after 7 years of being a mac laptop user, I’m strongly considering switching back to Linux.

On Sunday, the hard drive in my (18 month old) Macbook died. Nothing super-serious, but unlike in the past, replacing the hard drive isn’t a 3-screws operation, so rather than rushing out and getting a new drive, I just waited until work on Monday and grabbed myself a Linux machine.

In using Linux for the past week, there’s relatively little that I miss from OS X. (The biggest drawback may be lack of Netflix; curse my addiction to DRMed streaming media!)

Although I’m on a bit of a beast of a machine — a Dell Precision M2400, hardly the prettiest of Linux machines — I’ve found that overall, Linux has most of the things that I use on a daily basis from OS X — in many cases easier than it would be to get them via OS X. I’d visited the Computer Repairs store far too many to let the computer run on a dawdling pace.

All of my nitpicks come down to minor differences than any switch would have — for example, I miss being able to quickly change between *applications*, instead of just windows of an application; I feel a bit slower on Linux than I did on Mac still, and for some reason my several attempts to install Flash don’t seem to have actually worked. I am missing having Exchange MAPI support in a mail client; with our Nokia mail setup, this means I can’t get my mail from outside the firewall except via webmail (ewww).

But a lot of the things that were problems in the past, just haven’t been issues. Sound and Power seem to work just as well as on OS X. Since I already use VLC, not much difference there. I miss Safari a little bit — not for any particular reason, just in the ‘getting used to things’ sense — but a full screen Firefox… version-whatever-I-Have (apparently, after checking, the answer is ‘6.0’) doesn’t feel particularly different in any significant way.

After 7 years of being on mac, and now switching back to Linux for the past week — I haven’t yet found anything that feels like a huge win for mac, and I have found many small wins for Linux. I expect that in the next couple weeks I’ll make a final decision on whether I want to make the jump and get a long-term computer that’s Linux, but right now, I’m leaning strongly towards “yes.”

I’ll just have to keep my tablet handy for watching Netflix.

Nearly 7 years ago, I first started hacking software on phones. This blog was started in part to track my interest in topics around mobile hacking, a desire that I’ve never given up on.

Amusingly enough, when I joined Nokia as an employee, I was able to actually find references to some of these early works as being important in the space of the Symbian Python developments. It was kinda cool to see my name in some PowerPoint presentation that had been put together 6 years before by someone I’d never met or even heard of.

But you’ll notice that since then, I’ve had a serious dropping off; there’s been a dearth of development in that space for a lot of years. The reason for this is simple: on the platform that I used the most for many years, development was a pain. Shipping applications to users was not possible, and as time went on, the number of people who even used the same OS or platform as I did was shrinking drastically, due to Nokia’s tiny marketshare in the US.

Up until two weeks ago, I had never owned a non-Nokia phone. Sure, when we were acquired by Nokia, I got an opportunity to use other phones — both to test out our software on them, and to compare how they succeeded and failed — but I never owned anything but a Nokia phone, from summer of 2003 when I got my first 3650, to April of 2010 when MetaCarta was acquired and I still had my N95 (with a 6600 in the middle).

That’s not true anymore, though.

I recently attended a Microsoft Windows Phone developer event, and I’m — almost 7 years later — hacking phones again. While there still isn’t anything quite as easy as Symbian Python — it’s really hard to beat something in a programming language I speak all the time, presenting a beautiful API to the widget sets that the platform makes available with almost no need for platform specific knowledge — the Windows Phone platform combines a strong toolkit, extremely broad set of documentation and examples, and another key thing that I never had in the Symbian world: the ability to get the applications that I write to users.

Nine days ago, just three days after getting Visual Studio installed in my Windows 7 VM, I was able to not only build, but deploy to my phone and upload to the Windows Phone Marketplace — where just a few days later, my app was available for all to download (well, at least everybody on Mango).

Windows Phone development is *fun*. The tools are helpful, intuitive, and a breeze to get started with. The documentation and support community is huge — building on top of one of the things Microsoft does best. And thanks to the developer event, I now own my first ever non-Nokia phone: an LG Quantum, won in a raffle among people who were able to build and demo an app in just a day at the Microsoft developer event.

While nothing can compete with Symbian Python for pure quick-hack ability — at least, nothing I’ve seen yet — the Windows Phone platform provides a great development ecosystem, with a strong community backing, and the ability to quickly and easily get together apps that do things. Once you do — not only to put them on your phone, but you can also get them in a store so everyone else can install them… and that’s just damn cool.

Many thanks to Joe Healy, and the whole MS Events team who were able to put together the Mango Boston event a couple weeks ago. It was pretty awesome to not only be able to learn about Windows Phone — but to be able to *do* Windows Phone development, and really get my hands dirty, and ship an app less than a week after I even got the dev tools installed on my system.

иконописSo, on Sunday, we released the new version of OpenLayers, with updated mobile support. This included the ability to do dragging and panning and even editing of features on mobile devices, including Android and iOS.

Last week, I finished up a quick project, called olhttp. olhttp is a demo of how to use the OpenLayers HTTP protocol to create a simple, straightforward UI for editing features — but one that is easily customizable.

This afternoon, I decided to make an attempt to put these two things together — specifically, to make it possible to demo feature editing on a mobile device like my new tablet, saving the features and then being able to display them on my Nexus S or a laptop. So, I wanted a quick and easy deployment plan for a GeoDjango app — after my experience with DotCloud, I’ve come to the realization that hosting my own shit is for chumps (unless I really need some specific high-level SLAs for uptime, which I almost never do for personal projects).

Luckily, I happened to be at a table full of FOSS4G Django Hackers, so they were able to suggest DjangoZoom, which recently added support for GeoDjango/PostGIS. I was able to apply for an invite, and later that afternoon, I had one, and was able to start playing with it.

Now, at the time, I was in a talk, so the only thing I had with me was my tablet; I figured I’d set my account up, and see how it went. Turned out, the answer was “really well”.

The way DjangoZoom works by default is that you give it a Github repository URL, and it will automatically fetch the code for you, and set up a Django database, appserver, etc., deploying your application’s code to the DjangoZoom servers. What does this mean in practice? Well, for me, it meant that I was able to continue with the setup process for DjangoZoom — all the way up through actually getting a working application deployed, without even switching from the tablet to the laptop. I provided my Django app to the platform, and it worked right out of the box.

After getting my application deployed in just minutes, I then moved onto modifying my app to specifically target touch devices. This included modifying the UI to be more touch friendly — larger editing icons, for example, since the defaults are very difficult to hit on a tablet or phone screen, with 200 ppi. This work (in a new github repo for the time being, olhttp-django) complete, I now have a simple, easy to use tablet editing UI. It works on phones like iOS and Android, it works in all browsers on the desktop, and it provides an easy to use data-input mechanism — and I never had to touch an Apache config.

That’s what I call “success”.

Of course, in addition to the good — solid OS, and overall positive experience from all sides — there is some bad that comes with the owning of my new tablet.

Lacking Apps

• No Netflix. I said about 2 months ago that I’d run into my first real pain from DRM — not being able to move files from a broken Wii to a new Wii. This Netflix disaster is another one. Apparently there’s some software issues that prevent this, but the biggest reason that things seem to be stuck is simply because ensuring playback remains DRM-safe is harder than it should be. Netflix support exists on certain Android platforms, and there are a bunch of hacks out there to make it work on the tablet I have, but it’s not currently possible without rooting it.
• No Hulu. Same as above. Hulu just gives a “No streaming support for your device” message when loading it in the browser, and there’s no App for Android. I’ve really enjoyed the fact that over the past year, more and more of my video watching can be done completely legally, and the lack of support for the tablet is forcing me to back off of that position overnight, which is a shame. (I’m happy to watch adverts, let people track me, even give feedback on how much ads appeal to me — if I can actually get my content. Since I can’t…)
• No Facebook app? Not really sure what’s up with that, since it doesn’t seem like Facebook would be that hard, but there doesn’t seem to be an official Android Tablet app. I did grab a copy of “Friend Me”, which seems like it’s providing a reasonable replacement.
• No support for DivX video playback out of the box — but easily solved with a number of other video players on the device, it seems. (Sadly, my personal favorite, VLC, isn’t yet publicly available for Android.

Hardware/Tablet Nits

The Tablet doesn’t seem to support USB mass storage. It does support the “MTP”/Media Transfer Protocol, so I was able to use an app called “XNJB” to copy files over to it, but not being able to use the built in filesystem tools in the OS is pretty unfortunate.

It’s not possible to connect to ad-hoc wireless networks, like those created on mobile devices using software like Joikuspot. It seems like there are (again) some workarounds to this — replacing wpa_supplicant on a rooted device — but nothing that you can do out of the box. (I hate turning ‘toys’ into ‘work’, so I’m trying to avoid rooting if I can.)

No charging, not even trickle charging, over USB. I really think this is unacceptable: even if you can’t increase battery power over USB, there is no reason that you can’t prevent my battery from draining as fast if I’m plugged into USB.

I’m sure I’ll run into more; this is just what I’ve run into so far as I go through. Overall, I’m still 100% happy with the purchase; it is so far looking like it will definitely be worth it for me. But I also recognize that it’s not a perfect tool, and there are improvements I wish could be made.

(As an aside: does anyone have any recommendations for a good Flickr *browsing* application, both my photos and others, on Android Tablets? “Flickr Viewer HD” is … suboptimal in the way it navigates, and browsing contacts photos seems hard to impossible.)

(Cross-posted from my LiveJournal.)

Yesterday, I bought myself an Android-based tablet.

I’ve been considering this purchase for a long time. Tablets are one of those things that I have always been wary of, because I’m not as much of a media user as a tablet really targets, I think. However, after a recent flight to Berlin, I realized that I have at least one solid use case for a tablet, in the form of a media player on flights. With that in mind, I quickly moved onto others: showing photos I’d taken to family members, an easy way to get computerized video to the TV, etc.

In addition, it would be a chance to explore Android. I’ve played with a lot of other mobile platforms over the past 18 months, but not Android — and although it’s not quite a phone, the Android tablet experience is still an Android experience.

I’ve been biding my time for a while, while the marketplace kind of leveled out. Handily, my coworkers have done my research for me: over the past 3 months, 3 other people in the office have bought the same Tablet, so I joined the crowd yesterday, and bought an Acer Iconia Tab.

So far, I love it.

Using it makes me feel like I’m in Star Trek, or Minority Report. *whoosh* goes the desktop! *Whoosh* Slicing some fruit!

The application support is obviously much broader than WP7 or Symbian’s; more quick toys, more real tools, more access to third party services that make using the thing easier. The OS really feels clean and relatively solid — I’ve used Android on phones before, but not a recent version, and the tablet experience feels so much ‘cleaner’ than any of the phones I’ve used. In the browser, things like OpenLayers support dragging, pinch-zooming, drawing in the browser, which is awesome. (I should really see if I can dig up whether/when that is happening or not happening in the 2.x series… or maybe 4.x will just fix it for everyone.) The browser really feels nice and relatively solid. The desktop, and widgets, are both nifty and useful.

The Acer was what I picked because it was essentially the same as the Xoom and Galaxy Tab 10.1 in functionality, form factor, etc. — but it was $100 less. I still need to explore sleeve/case for it to protect it from my stupidity, but for now: I have a new shiny toy, and it is Nifty.

икониFor a long time, I’ve seen complaints that the OpenLayers API is ‘too complicated’ — that there are too many options, that it’s too difficult to configure, etc. I’ve said for a long time that although OpenLayers offers a lot of power, it’s also possible to make things simpler with a relatively thin layer on top of OpenLayers — that the power doesn’t necessary have to compete with simplicity, but that maintaining a simple wrapper at the OpenLayers level is probably not ideal, for a lot of reasons.

I’m obviously not a good target for making a simpler API — I don’t consider OpenLayers difficult to use, so it’s really hard for me to be the one to implement an ‘easier’ to use API! Thankfully, only 3.5 years after I started saying someone should write a simpler API, there appears to be a significant competitor to OpenLayers which is being praised for its ease of use: Leaflet, from Cloudmade.

So, after watching people continually praise the Leaflet API, I thought I’d see what it took to make OpenLayers as easy to use as Leaflet. I started with the Quick Start example, and worked my way down the features of the page until I got to the end.

The result is the ol-leaflet-wrapper quick-start — a substantially similar page, which looks the essentially the same as the Leaflet Quick Start guide (minus popups). Total code to implement: 99 lines.

Now, I’m not saying this is a full-fledged wrapper: in fact, anyone with half a brain would be able to observe that this particular example is missing one big feature, in the form of popups. (In fact, I think this kind of demonstration is one very easy way to make it clear where the OpenLayers *library* is deficient: things that are hard to wrap between one library and another are a clear target for API refactoring and cleanup.) Any deeper look would show that I’ve only minimally wrapped Leaflet, and that a more complete wrapper would still be a significant investment.

However, I do think that it’s possible to make the kinds of changes that people desire most from OpenLayers *without* completely reimplementing the library — and even without a refactoring of the (admittedly bulky) core library; instead, approaching a new API as something built on top of the existing is a way to rapidly prototype a new approach, and integration or improvements can move into the core library as needed.

And if people who like OpenLayers really want a simpler API like Leaflet — well, I’m happy to help develop the Leaflet API wrapper into something more real, to help direct what a suitably friendly API for working with OpenLayers might actually look like.

иконописПравославни икониToday, my 13-year old daughter was flying from Manchester to Greenville/Spartanburg, transferring in Dulles. She was flying without the unaccompanied minor service. Her connection time in Dulles was short (only 1hr15m), but should have been sufficient.

At 4:50PM, her flight status was updated, making it clear that her flight was going to be an hour late in arriving, arriving at 4:54PM. (Admittedly, I should have been keeping closer track and making some phone calls earlier, but I doubt it would have helped.) She pocket-dialed me at 5:00, so I knew she was at least somewhere with cell reception; unfortunately, calling her back didn’t get any response.

At 5:05, I called United, and asked if they could confirm her status — whether she made her flight. While navigating their phone menus, I observed that the flight she was supposed to take had taken off, 5 minutes early. I gave United her name + flight information, and they then proceeded to take 5+ minutes to tell me that she had left from Manchester earlier that day. (Thanks for telling me information I gave you.) They also told me her flight was late (thanks again), and that she was rebooked for the next morning at 8:20AM.

Now, for me, cue a bit of panic; I made the decision not to purchase unaccompanied minor service for her trip based on the idea that she could make her flights, and even if she didn’t, there was another flight later in the day she could pick up (still the case here: there were two more flights out of IAD to her destination 30 minutes later). This may have been a mistake, but that was a decision. Having United rebook her on a flight the next day is an entirely different beast, obviously.

I asked the agent whether they could confirm she had *not* made her flight: I was told that she could tell me that she had been rebooked the next morning, which they would not have done if she had made her flight, but she repeated that she could not tell me if she actually made her flight. (What?)

I called a couple other numbers, including the airport traveller assistance number, who told me that they could not do anything other than page in the baggage claim area; we punted on that for the time and kept trying United, still trying to get a confirmation that she had or had not made it on her flight.

In my next call, I got the same basic story: She’s rebooked for tomorrow. Since she is rebooked for tomorrow, United will be putting her up in a hotel tonight. No, I can’t tell you what hotel. No, I can’t tell you a number you can call for that information. No, I can’t tell you whether she made it on the flight. No, I can’t tell you hotels we use for putting passengers up. I can tell you that she was rebooked for tomorrow morning.

At this point, we had her paged in the airport — which ended up producing no response, though they were attempting to be very helpful. They did clarify that they can’t page in the gate area, and we should contact the airline for that.

Again, we called a different United number, only to be shunted back to the main United phone disaster. No, we can’t contact our gate agents because of 9/11. No we can’t page in the gate are. No, we can’t tell you what hotels we use. No, we can’t give you a number to find out where this passenger has gone.

Kristan and I at this point started calling hotels — this is one hour after she has landed in Dulles and we have been told she will be staying overnight in DC and that we could get no information from anyone. We made it through about 10… at which point we got a call from Alicia. She had not answered her cell phone (which she did have on her, and charged)… because she was on her plane.

She was fine, and on her plane, the whole time. United put her on the next flight, but rebooked her anyway. She’s not in IAD, and had no problems… other than our complete lack of ability to communicate with United at all.

The first response to this will naturally be: “You should have paid for unaccompanied minor service.” While this may be true, I’m not convinced that it would have helped: specifically, the problem was not that Alicia did not make her flight — she did. The problem is that *no one could tell us this* — despite the fact that, presumably, someone scanned her boarding pass. Since we did not have a phone number — and I looked for an ‘unaccompanied minor contact number’ on United’s site — and she made her flight on time, I doubt we would have been contacted by United. (In fact, unaccompanied minor service might well have meant the difference between her making her flight and not in this case!)

Overall, I’m not upset that United didn’t know where my daughter was: that’s my fault. I’m not really that upset about most of the question they couldn’t answer in this case — though if she *was* stuck overnight, I’d be damn upset. Instead, I’m only upset about one thing: United told me she missed her flight, rebooked her on a flight tomorrow morning, and was completely unable to tell us that *she actually made her flight*. That has to be seen as a failure somewhere in the system, in my opinion, and I think the complete lack of being able to get to anyone other than reservations via the phone on short notice is completely unreasonable.

I do want to thank people who helped reassure me on twitter and elsewhere, and the IAD Travelers Aid desk, who did everything within their limited power to help us out, and the most important thing is that Alicia is at her destination + safe.

икониOne of the interesting things that I’ve noticed in exploring various cloud providers is the limited amount of support for strong geo integration. This isn’t particularly surprising — Geo on the web is still a niche market, even if it is a niche market I tend to care about a lot. Here are some things that I think that might make people more eager to adopt cloud-y services for geo, specifically in the context of DotCloud, where I’ve been investing most of my time recently.

DotCloud has the idea of ‘services’ — independent units which work together to form an overall operating picture. The idea behind these services is that they are simple to set up, and perform one particular task well. So rather than having a service which combines PostGIS, GeoDjango, and MapServer, you have three separate services: one for PostGIS, one for GeoDjango, and one for MapServer, or map rendering, and you connect between them. This way, if your application needs to scale at the frontend, you can easily simply deploy additional GeoDjango services; if you need more map rendering ‘oomph’, same deal. (Deploying additional service for databases won’t magically scale them, but you do have the ability to use the software’s internal scaling across your multiple instances.)

So, again, using DotCloud, there are three types of ‘services’ that I think would be interesting to users who are working in Geo:

• PostGIS — Supporting PostGIS on most debian+debian-derivatives platforms is pretty simple; the tools are all easily installable via packages. (pgRouting is another step beyond — something which might actually have some benefit simply because it’s more difficult to get started with, so having it preconfigured could make a big difference in adoption.) Nothing really special needed here other than apt-get install postgis and some wrapper tools to make setting up PostGIS databases easy. There probably isn’t any reason why this couldn’t be included in the default Postgresql service type in DotCloud.
• GeoDjango — This one is even easier than PostGIS. GeoDjango already comes with the appropriate batteries included. The primary blocker on this is to get the GEOS, proj, and GDAL libraries installed on the Django service type. I don’t think there is any need for a separate service type for GeoDjango.
• Map Rendering — This one is a big one for a lot of people, and I’m not entirely sure the best way to work it within DotCloud. Map Rendering — taking a set of raster or vector data, and making it available as rendered tiles based on requests via WMS or other protocols — is one of the things that is not pursued as often by the community right now, and I think a lot of that is in the difficulty of setup. As data grows large, coping with it all on the client side becomes more difficult; some applications simply never get built because the jump from ‘small’ to ‘big’ is too expensive.
  There are three different ‘big’ map rendering setups that I can think of that might be worth trying to support:
  • MapServer — MapServer is a tried and true map rendering tool. It primarily exists as a C program with a set of command line tools around it; it is usually run under CGI/FastCGI. Configuration exists as a set of text-based configuration files (mapfiles); rendering uses GDAL/OGR for most data interactions, and GD or AGG (plus more esoteric output formats) for output. MapServer is often combined with TileCache, for caching purposes; TileCache is based on Python.
  • GeoServer — GeoServer is a Java program, which runs inside a servlet container like Tomcat. Like MapServer, it supports a variety of input/output formats; configuration is typically via its internal web interface. Caching is built in (via geowebcache). I think GeoServer would probably run as is under the ‘java’ service type that exists on DotCloud, assuming the appropriate PostGIS support exists for the database side.
  • OSM data rendering — This one is a bit less solid. OpenStreetMap data rendering has a number of different rendering frontend environments, but the primary one that I think people would tend to set up these days is a stack of mod_tile (Apache-side frontend) talking to tirex (renderer backend) which calls out to/depends on Mapnik, the actual software which does tile rendering. Data comes from a PostGIS database — though in the case of OSM, even that requires some additional finagling, since getting a worldwide OSM dump is… pretty big. (It’s probably safe to set that point aside as a starting point, and concentrate instead on targeting localized OSM rendering deployments — solve the little problems first, scale up later.)

One thing that all of these tools have in common is that they really like having fast access to lots of disk for quickly reading and writing small files. I’m not sure what the right way to do that within the DotCloud setup is — I don’t see an obvious service type which is designed for this — so that might be another component in the overall picture. (Things like the redis service try to solve this problem I think, but since the tools primarily intend to write to disk as is, adopting them to support other ways of storing tile data persistently would require modifying the upstream libraries.)

I think that there is room to significantly simplify deployment of some components of geographic applications by centralizing installation in cloud-based services; the above sketches out some of the components that it might make sense to consider as a first pass. These components would let someone create a relatively complex client + server side geographic data application; exploring and expanding on these — especially the OSM data rendering component — could make deploying to the cloud easier than deploying locally, with the net effect of increased adoption of cloud-based services… and more geodata in the world to consume, which is something I’m always in favor of. 🙂

My previous attempts to make GDAL work in the cloud were sort of silly; compiling directly on the instance is sort of counter to the design of DotCloud, which lets you scale primarily by assuming that a ‘push’ to the server establishes all of the configuration you’ll need. (sshing into the machine is certainly possible, but it seems it is designed more for debugging than it is for active work.)

So, with a few suggestions from the folks in #dotcloud, I started exploring how to set up my dotcloud service in a bit more of a repeatable way — such that if I theoretically needed to scale, I could do it more easily. Rather than manually installing packages on the host, I moved all of that configuration into a ‘postinstall’ file — fenceposted so it runs only once.

After a bit of experimentation (and reporting a bug in GDAL), I was able to get a repeatable installation going; this means that I no longer have any manual steps in creating and setting up a new service doing OpenAerialMap thumbnailing; in fact, the entire process — right down to setting up redis as a distributed cache — can now be completed automatically.

The postinstall is pretty simple: download and configure, then install, curl; Download and configure, then install, GDAL. In both cases, use the respective -config tool as a fencepost so we don’t install multiple times.

Once this is done, setting up a new deployment with the appropriate services is trivial: that’s outlined in setup.sh — configure a redis service, grab the config information, and deploy a wsgi service using that config and the rest of the code.

In fact, while I was writing this post, I followed that very set of instructions, running ‘./setup.sh openaerialmap’ — before I finished the post, I had a running www.openaerialmap.dotcloud.com instance, which I could then commit in place of the silly-named ‘pepperoni’ service that oam.osgeo.org was using before.

The big thing for me about these one-off deployments is that by making available better tools for doing iterative deployment, they allow me to be more disciplined in how I configure services. Normally, the act of installing GDAL or curl is a one-time event: without reformatting my system (or setting up a VM and reinstalling/rolling back every time), it’s non-trivial to test what happens on a fresh system. With DotCloud, deploying another service literally takes seconds — and when I do, it’s a green field, ready for me to test my deployment scripts all over again. No more mistakes in one-off apt-get install commands that I’ll never remember if my system goes down. Now, I’ve got the ability to script deployments of software.

This is of course possible with EC2 directly as well — building custom AMIs, redeploying them, etc. For me, though, DotCloud took the pain out of having to do such a thing. They did the hard part for me — and I got to do the more interesting and fun parts.

One of the key components of the OpenAerialMap design that I have been working on since around FOSS4G of last year is to use distributed services rather than a single service — the goal being to identify a way to avoid a single point of failure, and instead allow the data and infrastructure to be shared among many different people and organizations. This is why OpenAerialMap does not provide hosting for imagery directly — instead, it expects that you will upload images to your own dedicated servers and provide the information about them to the OpenAerialMap repository.

One of the things that many people want from OpenAerialMap is (naturally) a way to see the images in the repository more easily. Since the images themselves are quite large — in some cases, hundreds of megabytes — it’s not as easy as just browsing a directory of files.

Since OAM does not host the imagery itself, there is actually nothing about this type of service that would be better served by hosting it centrally; the meta-information about the images is small, and easily fetched by HTTP, and the images themselves are already remote from the catalog.

As a result, when I saw people were interested in trying to get OpenAerialMap image thumbnails, I decided that I would write it as a separately hosted service. Thankfully, Mike Migurski had already made a branch with the ‘interesting’ parts of the problem solved: Python code using the GDAL library to fetch an overviewed version of the image and return it as a reasonably sized thumbnail. With that in mind, I thought that I would take this as an opportunity to explore setting this code up in the ‘cloud’.

A requirement for such a cloud deployment is that it needs to have support for GDAL: this severely limits the options available. In general, most of the ‘cloud hosting’ that is out there is to host your applications, using pre-installed libraries; very few of the sites out there would let you compile and run code in them, at least that I can find. However, I remembered that during my vacation week, I happened to sign up for an account with ‘DotCloud’, a service which is designed to help you “Assemble your stack from pre-configured and heavily tested components.” — instead of tying you to some specific setup, you’re able to build your own.

At first, I wasn’t convinced that I could do what I needed, but after reading about another migration, I realized that I could get access to the instance I was running on directly via SSH, and investigated more deeply.

I followed the instructions to create a Python deployment; this got me a simple instance running a webserver under uwsgi behind nginx. I was able to quickly SSH in, and with a bit of poking about, was able to compile and run curl and GDAL via the SSH command line into my local home:

$ wget http://curl.haxx.se/download/curl-7.21.6.tar.gz; tar -zvxf curl-7.21.6.tar.gz
$ wget http://download.osgeo.org/gdal/gdal-1.8.0.tar.gz; tar -zvxf gdal-1.8.0.tar.gz
$ cd curl-7.21.6.tar.gz
$ ./configure --prefix=$HOME --bindir=$HOME/env/bin
$ make
$ make install
$ cd ../gdal-1.8.0;
$ ./configure --prefix=$HOME --bindir=$HOME/env/bin --with-curl --without-libtool --with-python
$ make
$ make install
$ cd ..
$ LD_LIBRARY_PATH=lib/ python


So this got me a running python from which I could import the GDAL Python libraries; a great first step. Then I had to figure out how to make the webserver know about that LD_LIBRARY_PATH — a much more difficult step.

After some mucking about and poking in /etc/, I realized that the way that the uwsgi process was actually being run was under ‘supervsisord’; basically, whenever I deployed, my Python process running under supervisord was restarted, and was running my Python code inside it. So then all I had to do was figure out how to tell the supervisor on the DotCloud instance to run with my environment variables.

I found the ‘/etc/supervisor/conf.d/uwsgi.conf’ which was actually running my uwsgi process; after some research, I found that DotCloud will let you append to the supervisor configuration by simply creating a ‘supervisord.conf’ into your application install directory; it hooks this config at the end of all your other configs. So, I copied the contents of uwsgi.conf into my own supervisord.conf, dropped it in my app directory, and added: environment=LD_LIBRARY_PATH=”/home/dotcloud/lib” to the end of it.

Pushed my app to the server, and now, instead of “libgdal.so.1: cannot open shared object file: No such file or directory”, I was able to successfully import GDAL; I then easy_installed PIL, and with a little bit more work, I got the image thumbnailing code working on my DotCloud instance.

I’ve still got some more to do to clean this up — I’m not sure I’m actually using DotCloud ‘right’ in this sense — but I was able to get GDAL 1.8.0 in a Python server running on a DotCloud instance and use it to run a service, without having to install or screw with the running GDAL libraries on any of my production servers, so I’ll consider that a win.