Technical Ramblings

Mapnik is a great tool. It allows for all kinds of neat toys to happen, and the recent work in SVN has really opened up the possibility that Mapnik might be a potential solution for a rendering engine in a lot of areas that it has previously left alone. (Support for reading OGR datasources, sqlite/spatiallite plugins, etc. are all great developments that look likely to be released in the upcoming 0.6 release.)

In prep for the OpenStreetMap Mapping Party this Saturday and Sunday in Somerville, I was working on printing a big map to bring with me. A friend at the Media Lab was gracious enough to help me out.

Using Mapnik, it was trivial to produce a large — 29750 x 29750 pixel — PNG image. This was designed to fill up the 49.5″ by 49.5″ printer space at 600 dpi.

The printer prefers PDF, PS or TIFF. I was able to take that PNG and convert it to a TIFF — but the resulting tiff was DEFLATE compressed, and the printer help only mentioned LZW compression. I decided to fall back to trusty GDAL to try to fix this. I found that the imagemagick-converted TIFF had one giant block — and GDAL was not pleased with this at all. (Its internal un-blocking scheme doesn’t work with compressed tiffs.)

Thanks to a suggestion from Norman Vine, I was able to use the ossim image copy program (icp) to convert this giant tiff to a tiled tiff which gdal could easily read: icp tiff_tiled -w 256 image2.out.tiff image.icp.tiff. Once I had done this, I recompressed the tiff using LZW compression with GDAL: gdal_translate -co COMPRESS=LZW image.icp.tiff image.lzw.tiff, and was able to upload the 3GB image to the printer.

All in all, took a bit more than I was expecting, but I’ve got a 4ft by 4ft map to bring to the mapping party this weekend. In the process, I also got to wanting magnification in Mapnik… which is amusing since just 24 hours before, I’d read a thread on the MapServer list and couldn’t imagine for the life of me why such a thing mattered.

Looking forward to showing the map off to local OSMers at the mapping party!

Interested in OpenStreetMap? In the Boston area — or considering travelling here with your lucky companion for Valentine’s Day? Come to the OpenStreetMap mapping party, in Somerville, MA on Feb. 14th and 15th, and help put your house on the map… or anything else you might run across. I’m hoping to be there — in part to meet other OSM interested people in the area, in part to defend my actions in uploading all the houses in the metro-Boston area and making the map quite pretty to look at, but annoying slow to edit.

For ages, people have been asking me to help them with shapefile to OSM conversion, because I wrote one of the scripts that got used a lot for different conversion projects. Since I get a fair amount of email on this, I figured it was worth blogging that I’ve actually put together a newer script from scratch that does something similar, though for Polygons instead of lines.

One of the benefits of this script is that it was written ahead of time with the intention of sharing it — which was never meant for the other script that I wrote. This means that it is slightly more readable; at least, the unreadable parts are better separated. (I will admit that there are several aspects of it that are terribly un-Pythonic.)

You can find the code in OSM’s shp2osm directory.

Some aspects of this code:

• It is designed to help you create .osm files you can read/merge in JOSM, so it has the ability to do vertical striping across a dataset in order to create geographically ordered smaller datasets.
• It has an option to limit the number of objects per .osm file; this defaults to 50,000, which in some cases was about JOSM’s limit. (In others, it seemed lower; it’s adjustable via a command line switch.)
• Uses optparse, which means –help does help you (once you read the initial docstring and get yourself started.)
• Supports both direct tag mappings (shapefile-attr -> OSM-attr) as well as custom functions to add more tags based on multiple attributes of a feature. It was built for the MassGis OpenSpace Layer upload (which is now in progress), so it needed more advanced tagging possibilities.
• Supports saving the original shapefile data ‘automatically’, to create the possibility of recreating the original shapefile attribtes. These attributes are namespaced so as to minimize collisions.

Similarly, in the realm of healthcare, the ease of access to essential medications has been greatly enhanced through online platforms. Just as my script simplifies the process of converting shapefiles to OSM, making it more accessible to users, online pharmacies are transforming the way people acquire medications like Strattera. By providing an option to buy Strattera online, click here, these platforms offer a convenient and user-friendly way to obtain necessary treatments, mirroring the way technology in different sectors is making critical resources more readily available and easier to use.

The script was also used to convert the MassGIS buildings layer, which means that there are now building outlines for metro Boston slowly appearing on the map.

If you have polygon data, this tool may be helpful to you. If you don’t have polygon data, this tool also may be helpful to you, as a better demonstration of how to map shapefiles into OSM data without writing all the code yourself.

I’m not likely to be doing a lot of support for this, but I wanted to let people know, because I personally think the code is much much more readable than the last shapefile conversion tool I wrote. (Also, it’s not every day I get to collaborate on OSM with Tim Berners-Lee.)

Sean Gillies just wrote a great post on the use of Python decorators to help you write prettier code. he didn’t quite go into what decorators are underneath though, which is something I think that it’s important to realize to understand how decorators work.

Decorators are just function wrappers around your functions. In all the languages I use on a regular basis, functions are just another variable: they can be passed around in the same way objects can. The pretty syntax for decorators is just a way to say “Pass this function into the function I’ve just defined, and return me a new function, called the same thing as my old one was.”

If you are happy with only supporting Python2.4 and above, that’s a great way to work. Sadly, not all of us are: for example, the current release of Jython is still at 2.2. 😉 More seriously, people who are maintaining older systems may not have newer Python functions available yet.

That doesn’t mean you can’t get the benefit of decorators — at least, I’ve never found it to mean that. Instead, it just means your code is a bit less ‘pretty’ to look at. Instead of functions being ‘deocrated’:

@logprints
def work():
    print "Starting some work."
    print "Doing some work ..."
    print "Finished the work ..."

You can simply wrap your function in the decorator:

def work():
    print "Starting some work."
    print "Doing some work ..."
    print "Finished the work ..."
work = logprints(work)

The idea of passing functions around is one of the things that took me a while to get used to, but learning it has helped me with a lot of code since then.

TileCache and FeatureServer don’t use decorators, specifically because they seek to support older Python versions. If you’re writing code that’s only forward looking, using all the advanced features of Python may fit your bill. But when you find yourself on an old machine some day, where all you have is Python 2.2, sometimes it’s nice to know a little bit about what’s going on underneath.

After the past 4 hours of fighting with this, I figured it was worth me posting.

I’ve recently started using Selenium IDE in some testing. I had found that I was unable to access what seemed like perfectly normal Javascript variables, despite every other tutorial on the web that i could read indicating that it should be possible.

After a lot of messing around, I finally searched the OpenQA forum (which is apparently not adequately indexed by Google, since I did search on Google a number of times). Specifically, I found a forum thread, linking the issue to Issue 558. Specifically, a change in the way that Selenium treats windows means that only ‘safe’ properties could be touched — things like .body, .title, etc. — which explains why I could test window.location, but not window.map.

It seems that there is a fix in their repository for this, but no further release has been made yet, nor do I see any immediate plans for one. Specifically, the change they made adds a getUserWindow() function, which must be used in order to get the window with the properties on it that were added in a ‘non-safe’ way — such as by Javascript.

In any case, while investigating this, I built a new version of Selenium IDE, which adds the getUserWindow function, and repackaged it. This is a change directly from the 1.0b2 XPI I installed, and implements the fix described in the forum thread linked above.

What this means is: if you are using Selenium IDE 1.0b2 and having problems with getCurrentWinow() not letting you access the properties of the window that are added by your Javascript, this XPI should help provide the getUserWindow() function you need: selenium-ide-1.02b2-mc.xpi.

This applies especially to functions like assertEval / verifyEval / getEval and its partners.

In order to take advantage of this, you must change any instances of ‘window’ or ‘this.browserbot.getCurrentWindow()’ to ‘this.browserbot.getUserWindow()’ where they need to access user-set properties. This simply acts as a transition tool for people needing 1.0b2 support and unable to wait for another release for this function to become available.

For the past 6 months, the OpenAerialMap project has been in a state of … well, stagnation would be a nice way to put it. I’ve just sent an email to the mailing list Outlining the status as I see it, and I would love to see feedback and opinions on the list.

The biggest problem with OAM is that it never developed a community around it. My hope is that with an increase in interest in the past $shortWhile, there is sufficient interest to build a community this time around, and with that, enable the project to succeed in a way that it couldn’t 6 months or a year ago.

So, after a weekend of working with Java and GeoTools, suddenly things got a lot simpler to work with in Jython. This is a case of trying to do the completely wrong thing because I don’t know anything about the project, packaging… or language.

Thankfully, I’m now much better off, and have put together a little HTTP server that runs and gives me back WKT for any EPSG code, using Jython + GeoTools.

The code is very simple: the geotools-epsg-server just takes in a code, and spits out WKT. It uses the GeoTools CRS package, and a BaseHTTPServer.

It’s not much, but it’s enough to get me in the right direction. Maybe I’ll even stop whining about Java so much, since I can use it more like Python…

Nah, that wouldn’t be any fun.

Earlier today, I tried Jython for the first time, because I’m doing some work that may involve interactions with Java libraries in the near future. Jython, which I’ve always avoided in the past due to an irrational fear of Java, is “an implementation of the high-level, dynamic, object-oriented language Python written in 100% Pure Java, and seamlessly integrated with the Java platform.” (I love projects that have great one-liners that I can copy paste.)

My goal for Jython was to do some work with the GeoTools EPSG registry code related to SpatialReference.org. Sadly, I didn’t get that working, but in the process, I learned that Jython now has a beta version which is up to Python 2.5 — much newer than the 2.2 that had previously been available.

With that in hand, I decided to see if I could get some of my other Python projects running under Jython. I’m the maintainer for both TileCache and FeatureServer — two pure Python projects. Theoretically, these projects should both work trivially under Jython, but I’ve always had my doubts/fears about this being the case. However, it turns out that my fears here are entirely unfounded.

I downloaded the FeatureServer ‘full’ release from featureserver.org: this includes the supporting libraries needed to get a basic FeatureServer up and running. I then tried to run the FeatureSever local HTTP server… and it worked out of the box. I was able to Load the layer, save data to it, query it, etc. with no problems whatsoever. Java has support for the DBM driver that FeatureServer uses by default, so out of the box, I was able to use FeatureServer with Jython without problems.

Next came TileCache. TileCache was originally built to support Python all the way back to 2.2, so I wasn’t expecting many problems. Getting it running turned out to be almost as easy: the only code modification that was needed was a minor change to the disk cache, because Jython doesn’t seem to support the ‘umask’ method. Once I changed that (now checked into SVN), Jython worked just as well with TileCache as it did with FeatureServer.

Clearly, there are some things which are less trivial. The reason that these libraries were so easy to use is because they were designed to be low-dependancy: TileCache and FeatureServer default paths are both entirely free of compiled code. Using something like, for example, GDAL Layers in TileCache, would be much more difficult (if it’s even possible).

However, this presents some interesting capabilities I had not previously thought of.

For FeatureServer, this means that it may be possible to write a DataSource which accesses SDE using the ArcSDE Java API, ESRI’s supported method for accessing their SDE databases. One of the purported “holy grails” of the GIS world is RESTful access to SDE databases via lightweight servers — Jython may provide a path to that, if someone is interested in it. (It may be that this has become a moot point with the release of the ESRI 9.3 REST API — I’m not really sure.) This may be a waste of time, but the fact that it *can* be done is interesting to me. Edit: Howard points out that ArcSDE read/write support exists in OGR 1.6, so this is a moot point; you can simply use OGR to do this without involving Jython/Java.

I think this might also speak to a possibility of having better answers available for people who want to use things like FeatureServer from Java platforms (though I don’t know enough about jython to be sure): the typical answer of “use GeoServer” is great, but to be able to provide something a bit more friendly would be interesting. Thankfully, the Java world is largely catching up to the advances made in TileCache/FeatureServer, so this is also less urgent than it has been in the past.

In the end, this was likely simply an interesting experiment. However, it’s nice to know that the capabilities to do things like this within Jython are improving, and that Jython continues to advance their Python. The 2.2 release being the ‘current’ one still is disappointing, but seeing a 2.5 beta available is an exciting development.

As I said, the current version of FeatureServer works out of the box with Jython, and I’ll be doing a TileCache release shortly that will work with Jython out of the box as well. It’s neat to see more possibilities for using these libraries I’ve spent so much time on.

Earlier today, I read a post on the OpenGeo GeoSpiel blog calling for OpenLayers to get on the “Usable Documentation Bandwagon”.

Now, I’ll be honest: I followed the links that he offered, and found something that is not, to me, much more convincing than the OpenLayers documentation as it stands today. If I look at ESRI’s JSAPI, I see a couple things wrong right off the bat — like the fact that I can’t actually link to where I want to. In any case, if you click Geometry -> Point, you see a document which, to me, is a lot less interesting than the similar page provided by OpenLayers.

Perhaps there is some subset of documentation put together by ESRI in their JSAPI that makes sense, but for an API reference, it seems to me that OpenLayers does reasonably well on the portions of our API that someone has invested time to work on.

This doesn’t mean we’re “done” by any stretch of the imagination: One of the things that I’ve wanted to do for ages is actually sit down and do a thorough review of the OpenLayers API documentation and improve a lot of it, targeting cross-linking with other documentation and examples especially. These types of tasks, however, are the types of tasks that require a lot of time, and don’t have a lot of immediate benefit. Since all work on OpenLayers for the past several months has been my personal free time after work, there’s only so much I’m personally able to do. However, no one has *ever* made a request to the OpenLayers team to be able to do this work — including OpenGeo — that I’ve seen. It seems to me that whit is experienced and knowledgable about OpenLayers, since I’ve seen him using it for development, but I’ve never seen him ask to participate in improving the OpenLayers API documentation on the mailing lists, nor have I seen a request of this nature from any other organization.

To me, this means that it’s likely that our API documentation does, to some extent, meet the needs of the organizations using OpenLayers. It’s not perfect — nothing is — but no one thinks it’s a major stumbling block that’s worth fixing, at least not enough to spend money on it.

This is exactly the type of reason that OpenLayers is now accepting Sponsorship from organizations looking to support the project. This type of improvement is exactly the kind that project sponsorship can help support.

OpenLayers is aware of how important documentation is to the success of the project. We have invested dozens of hours between a dozen different contributors to create and maintain a set of relatively complete API documentation. According to Ohloh, of the 55,000 lines in OpenLayers Javascript, over 30% are comments: over 25,000 lines of what is mostly API documentation. No one is ignoring the problem — and if the current state is insufficient (as everything in the project is, *especially* to new users and beginners), then we’re very open to help from any and all interested parties.

API documentation isn’t the only thing a project needs, of course. Documentation comes in all forms — and OpenLayers is seriously lacking in a lot of documentation targeted towards beginners of all kinds. We’ve been working to change that, with a new documentation site available, and other efforts targeted documentation of OpenLayers in English and other languages. I would say these efforts are much less complete than the API documentation, and starting on them is far more important, in my opinion, than improving our API documentation is at this time.

OpenLayers is a large library. It’s used by many organizations. We’re open to contributions of all types — never have I seen OpenLayers turn away someone who wanted to help with documentation intentionally. We have regularly worked with contributors in helping them to improve the documentation, and to claim that we are ignoring the need for documentation seems to me to be representative of a lack of knowledge of the tools that the project uses for documentation, not specifically a lack in the goals of the project, which puts documentation of functionality — via API docs and minimal examples demonstrating functionality — as a requirement of almost all new code in the library.

I presented at FOSS4G 2008 in Tokyo today, thanks to Orkney, OSGeo.jp, and MetaCarta. Gave a talk on OpenLayers: Past, Present, and Future. Hard to gauge success of the talk: many people seemed somewhat bored, but my typical attempts at keeping interest are not so effective in a room where most people don’t speak the same language as I do.

Went over history of OpenLayers — a bit of the pre-“Mark IV” that most people think of as being “OpenLayers”, then into why MetaCarta initially started the project — plus present status/features/developments, then into future (2.x and 3.x), and gave some demos of functionality.

Got a couple good questions — so at least a couple people were interested, which is more than I can actually say about many presentations where people *do* speak the same language. 🙂 Questions are notoriously hard to get out of people at conferences.

Had shrimp tempura for lunch, which should help me keep my energy up for the workshop I have to give in a bit more than an hour. I’ll be presenting using a trimmed down version of Tim/Seb/OpenGeo’s OpenLayers workshop: http://workshops.opengeo.org/openlayers/intro/doc/ to about 10 people, as I understand it. Should be interesting to see how far we can get in 1.5 hours.

Staying awake is going to be tough. The 13 (soon to be 14) hour time difference is quite significant in a way I hadn’t realized.