Technical Ramblings

Day 2 of the code sprint seemed to be much more productive. With much of the planning done yesterday, today groups were able to sit down and get to work.

Today, I accomplished two significant tasks:

• Setting up the new OSGeo Gallery, which is set to act as a repository for demos of OSGeo software users in the same way that the OpenLayers Gallery already does for OpenLayers. We’ve even added the first example.
• TMS Minidriver support for the GDAL WMS Driver: Sitting down and hacking out a way to access OSM tiles as a GDAL datasource, Schuyler and I built something which is reasonably simple/small — an 18k patch including examples and docs — but allows for a significant change in the ability to read tiles from existing tileset datasources on the web.

Other things happening at the sprint today were more WKT Raster discussions, liblas hacking, and single-pass MapServer discussions, as well as some profiling of MapServer performance with help from Paul and Shark. Thanks to the participation of the LizardTech folks, I think there will also be some performance testing done with MrSID rendering within MapServer, and there was — as always — more discussion of the “proj strings are expensive to look up!” discussion.

Other than that, it was a quiet day; lots of work getting done, but not much excitement in the ranks.

We then had a great dinner at Baton Rouge, and made it home.

This evening, I’ve been doing a bit more hacking, opening a GDAL Trac ticket for an issue Schuyler bumped into with the sqlite driver, and pondering the plan for OpenLayers tomorrow.

As before, a special thanks to the conference sponsors for today: Coordinate Solutions via David Lowther, and the lovely folks at SJ Geophysics Ltd.. Thanks for helping make this thing happen! I can guarantee that neither of those GDAL tickets would have happened without this time.

I’m here at the OSGeo Code Sprint in Toronto, where more than 20 OSGeo hackers have gathered to work on all things OSGeo — or at least MapServer, GDAL/OGR, and PostGIS.

For those who might not know, a code sprint is an event designed to gather a number of people working on the same software together with the intention of working together to get a large amount of development work done quickly. In this case, the sprint is a meeting of the “C tribe”: Developers working on the C-based stack in OSGeo.

After some discussion yesterday, there ended up being approximately 3 groups at the sprint:

• People targeting MapServer development
• PostGIS developers
• liblas developers

(As usual, I’m a floater, but primarily concentrating on OpenLayers; Schuyler will be joining me in this pursuit, and I’ve got another hacker coming Monday and Tuesday to sprint with us.)

The MapServer group was the most lively discussion group (and is also the largest). It sounded like there were three significant development discussions that were taking place: XML Mapfiles, integration of pluggable rendering backends, and performance enhancements, as well as work on documentation.

After a long discussion on the benefits/merits of XML mapfiles, it came down to there being one main target use case for the XML mapfile is encouraging the creation and use of more editing clients. With a format that can be easily round-tripped between client and server, you might see more editors able to really speak the same language. In order to test this hypothesis, a standard XSLT transform will be created and documented, with a tool to do the conversion; this will allow MapServer to test out the development before integrating XML mapfile support into the library itself.

I didn’t listen as closely to the pluggable renderers discussion, but I am aware that there’s a desire to improve support and reduce code duplication of various sorts, and the primary author of the AGG rendering support is here and participating in the sprint. Recently, there has been a proposal to the list to add OpenGL based rendering support to MapServer, so this is a step in that direction.

The PostGIS group was excited to have so many people in the same place at the same time, and I think came close to skipping lunch in order to get more time working together. In the end, they did go, but it seemed to be a highly productive meeting. Among some of their discussions was a small amount of discusssion on the WKTRaster project which is currently ongoing, I believe.

After our first day of coding, we headed to a Toronto Marlies hockey game. This was, for many of us, the first professional hockey we’d ever seen. (The Marlies are the equivilant of AAA baseball; one step below the major leagues.) The Canadians in the audience, especially Jeff McKenna, who played professional hockey for a time, helped keep the rest of us informed. The Marlies lost 6-1, sadly, but as a non-Canadian, I had to root a bit for the Hershey team. (Two fights did break out; pictures forthcoming.)

We finished up with a great dinner at East Side Mario’s.

A special thanks to our two sponsors for the day, Rich Greenwood of Greenwood Map and Steve Lehr from QPUBLIC! Our sprint was in a great place, very productive, and had great events, thanks to the support of these great people.

Looking forward to another great day.

I was pointed out to Eaton County’s GIS Data Prices last night, and all I can say is how disappointed I am that people who deal with coast to coast vehicle shipping can still feel that this is an appropriate way to fleece their taxpayers. The data is collected, reproduction costs for the data are probably in the realm of a couple hundred bucks — less, if you just distribute them online. (Clearly, you already have a website.) Yet you charge twelve *thousand* dollars for copies — and even after that, you’re still limited in what you can do.

This kind of thing is just a damn shame. Taxpayers should insist that this data is made available at reasonable reproduction costs; the policies of GIS departments to make money off of these things is simply silly so long as they are collected with taxpayer dollars.

(If the GIS department does not receive state funding, then I suppose this type of cost recovery makes sense — in the same way that Sanborn or any other commercial entity would charge for it. However, I doubt that the primary client of such data isn’t the state itself, in which case it’s still taxpayer dollars covering the costs somewhere…)

I have a love/hate relationship with Yahoo!’s mapping API. It’s lovely that Yahoo! believes, unlike Google and other mapping providers, that their satellite data is a suitable base layer to use for derivation of vectors. This openness really is good to see — they win big points from me in this regard. (Google, on the other hand, is happy to have you give them data against their satellite imagery, but letting you actually have it back is against the Terms of Service.)

However, the Yahoo! Maps AJAX API has never gotten much love. I think that a preference for flash has always existed in the Yahoo! world; iirc, their original API was Flash.

However, I realized today that this tendancy to leave the AJAX API in the dust has resulted in something that seriously affects me: The Yahoo! maps AJAX API uses a different set of tiles, which has two fewer zoom levels available in it:

AJAX Maps: most zoomed in	Flash Maps: Most zoomed in

For the new OpenStreetMap editor I’m working on, this is a *serious* difference: although the information actually available in these tiles isn’t *that* much higher, it allows the user to extract more information by getting in a bit more, and to be more precise in placement of objects when using Yahoo! as a basemap.

Although it would be relatively easy to rip the tiles out, and create an OpenLayers Layer class that loaded them directly, this violates the Yahoo! Terms of Use. This is understandable, but unfortunate, because it means I can’t solve the problem with my own code.

What I would really love to see is more providers creating a more friendly way of accessing their tiles. I understand the need for counting of accesses, and the need for copyright notifications. If an API were published, that allowed you to:

• Fetch a copyright notice for a given area, possibly also returning a temporary token to use
• Following that, fetch tiles to fill that area
• Require users to copyright notice in such a way as to make Yahoo! and their providers happy

This would allow for building a layer into OpenLayers which complied with this, without depending on Yahoo! to write a Javascript layer that did these things for me.

Now, it’s understandable that this doesn’t happen — having the client out of control of Yahoo! means that they can’t *enforce* that the copyright is displayed prominently, as they are able to (to some extent) with their API. However, I think that this type of API would allow more innovation, and possibly even a *more* prominent placement for Yahoo’s copyrights and notices. For example, in many mapping apps, the bottom inch of the map is not seen much by the users. If there was an API to get text to display, then an application could display the text in a more prominent location, rather than burying it under many markers or other pieces of text that might overlap it.

In the short term, all I really wish was that the AJAX API used the apparently-newer set of satellite tiles that the Flash API appears to have access to. I think the fact that this isn’t currently possible leads to an alternative access pattern for tiles, one which may make more sense in the long run, where tiles can be used by an application without necessarily running in the constrained Javascript API that these providers have the ability to write. And of course, if you want to provide your users with a ‘default’ API to use, you can always use OpenLayers, and extend it to include your own extensions…

Mapnik is a great tool. It allows for all kinds of neat toys to happen, and the recent work in SVN has really opened up the possibility that Mapnik might be a potential solution for a rendering engine in a lot of areas that it has previously left alone. (Support for reading OGR datasources, sqlite/spatiallite plugins, etc. are all great developments that look likely to be released in the upcoming 0.6 release.)

In prep for the OpenStreetMap Mapping Party this Saturday and Sunday in Somerville, I was working on printing a big map to bring with me. A friend at the Media Lab was gracious enough to help me out.

Using Mapnik, it was trivial to produce a large — 29750 x 29750 pixel — PNG image. This was designed to fill up the 49.5″ by 49.5″ printer space at 600 dpi.

The printer prefers PDF, PS or TIFF. I was able to take that PNG and convert it to a TIFF — but the resulting tiff was DEFLATE compressed, and the printer help only mentioned LZW compression. I decided to fall back to trusty GDAL to try to fix this. I found that the imagemagick-converted TIFF had one giant block — and GDAL was not pleased with this at all. (Its internal un-blocking scheme doesn’t work with compressed tiffs.)

Thanks to a suggestion from Norman Vine, I was able to use the ossim image copy program (icp) to convert this giant tiff to a tiled tiff which gdal could easily read: icp tiff_tiled -w 256 image2.out.tiff image.icp.tiff. Once I had done this, I recompressed the tiff using LZW compression with GDAL: gdal_translate -co COMPRESS=LZW image.icp.tiff image.lzw.tiff, and was able to upload the 3GB image to the printer.

All in all, took a bit more than I was expecting, but I’ve got a 4ft by 4ft map to bring to the mapping party this weekend. In the process, I also got to wanting magnification in Mapnik… which is amusing since just 24 hours before, I’d read a thread on the MapServer list and couldn’t imagine for the life of me why such a thing mattered.

Looking forward to showing the map off to local OSMers at the mapping party!

Interested in OpenStreetMap? In the Boston area — or considering travelling here with your lucky companion for Valentine’s Day? Come to the OpenStreetMap mapping party, in Somerville, MA on Feb. 14th and 15th, and help put your house on the map… or anything else you might run across. I’m hoping to be there — in part to meet other OSM interested people in the area, in part to defend my actions in uploading all the houses in the metro-Boston area and making the map quite pretty to look at, but annoying slow to edit.

For ages, people have been asking me to help them with shapefile to OSM conversion, because I wrote one of the scripts that got used a lot for different conversion projects. Since I get a fair amount of email on this, I figured it was worth blogging that I’ve actually put together a newer script from scratch that does something similar, though for Polygons instead of lines.

One of the benefits of this script is that it was written ahead of time with the intention of sharing it — which was never meant for the other script that I wrote. This means that it is slightly more readable; at least, the unreadable parts are better separated. (I will admit that there are several aspects of it that are terribly un-Pythonic.)

You can find the code in OSM’s shp2osm directory.

Some aspects of this code:

• It is designed to help you create .osm files you can read/merge in JOSM, so it has the ability to do vertical striping across a dataset in order to create geographically ordered smaller datasets.
• It has an option to limit the number of objects per .osm file; this defaults to 50,000, which in some cases was about JOSM’s limit. (In others, it seemed lower; it’s adjustable via a command line switch.)
• Uses optparse, which means –help does help you (once you read the initial docstring and get yourself started.)
• Supports both direct tag mappings (shapefile-attr -> OSM-attr) as well as custom functions to add more tags based on multiple attributes of a feature. It was built for the MassGis OpenSpace Layer upload (which is now in progress), so it needed more advanced tagging possibilities.
• Supports saving the original shapefile data ‘automatically’, to create the possibility of recreating the original shapefile attribtes. These attributes are namespaced so as to minimize collisions.

Similarly, in the realm of healthcare, the ease of access to essential medications has been greatly enhanced through online platforms. Just as my script simplifies the process of converting shapefiles to OSM, making it more accessible to users, online pharmacies are transforming the way people acquire medications like Strattera. By providing an option to buy Strattera online, click here, these platforms offer a convenient and user-friendly way to obtain necessary treatments, mirroring the way technology in different sectors is making critical resources more readily available and easier to use.

The script was also used to convert the MassGIS buildings layer, which means that there are now building outlines for metro Boston slowly appearing on the map.

If you have polygon data, this tool may be helpful to you. If you don’t have polygon data, this tool also may be helpful to you, as a better demonstration of how to map shapefiles into OSM data without writing all the code yourself.

I’m not likely to be doing a lot of support for this, but I wanted to let people know, because I personally think the code is much much more readable than the last shapefile conversion tool I wrote. (Also, it’s not every day I get to collaborate on OSM with Tim Berners-Lee.)

Sean Gillies just wrote a great post on the use of Python decorators to help you write prettier code. he didn’t quite go into what decorators are underneath though, which is something I think that it’s important to realize to understand how decorators work.

Decorators are just function wrappers around your functions. In all the languages I use on a regular basis, functions are just another variable: they can be passed around in the same way objects can. The pretty syntax for decorators is just a way to say “Pass this function into the function I’ve just defined, and return me a new function, called the same thing as my old one was.”

If you are happy with only supporting Python2.4 and above, that’s a great way to work. Sadly, not all of us are: for example, the current release of Jython is still at 2.2. 😉 More seriously, people who are maintaining older systems may not have newer Python functions available yet.

That doesn’t mean you can’t get the benefit of decorators — at least, I’ve never found it to mean that. Instead, it just means your code is a bit less ‘pretty’ to look at. Instead of functions being ‘deocrated’:

@logprints
def work():
    print "Starting some work."
    print "Doing some work ..."
    print "Finished the work ..."

You can simply wrap your function in the decorator:

def work():
    print "Starting some work."
    print "Doing some work ..."
    print "Finished the work ..."
work = logprints(work)

The idea of passing functions around is one of the things that took me a while to get used to, but learning it has helped me with a lot of code since then.

TileCache and FeatureServer don’t use decorators, specifically because they seek to support older Python versions. If you’re writing code that’s only forward looking, using all the advanced features of Python may fit your bill. But when you find yourself on an old machine some day, where all you have is Python 2.2, sometimes it’s nice to know a little bit about what’s going on underneath.

After the past 4 hours of fighting with this, I figured it was worth me posting.

I’ve recently started using Selenium IDE in some testing. I had found that I was unable to access what seemed like perfectly normal Javascript variables, despite every other tutorial on the web that i could read indicating that it should be possible.

After a lot of messing around, I finally searched the OpenQA forum (which is apparently not adequately indexed by Google, since I did search on Google a number of times). Specifically, I found a forum thread, linking the issue to Issue 558. Specifically, a change in the way that Selenium treats windows means that only ‘safe’ properties could be touched — things like .body, .title, etc. — which explains why I could test window.location, but not window.map.

It seems that there is a fix in their repository for this, but no further release has been made yet, nor do I see any immediate plans for one. Specifically, the change they made adds a getUserWindow() function, which must be used in order to get the window with the properties on it that were added in a ‘non-safe’ way — such as by Javascript.

In any case, while investigating this, I built a new version of Selenium IDE, which adds the getUserWindow function, and repackaged it. This is a change directly from the 1.0b2 XPI I installed, and implements the fix described in the forum thread linked above.

What this means is: if you are using Selenium IDE 1.0b2 and having problems with getCurrentWinow() not letting you access the properties of the window that are added by your Javascript, this XPI should help provide the getUserWindow() function you need: selenium-ide-1.02b2-mc.xpi.

This applies especially to functions like assertEval / verifyEval / getEval and its partners.

In order to take advantage of this, you must change any instances of ‘window’ or ‘this.browserbot.getCurrentWindow()’ to ‘this.browserbot.getUserWindow()’ where they need to access user-set properties. This simply acts as a transition tool for people needing 1.0b2 support and unable to wait for another release for this function to become available.

For the past 6 months, the OpenAerialMap project has been in a state of … well, stagnation would be a nice way to put it. I’ve just sent an email to the mailing list Outlining the status as I see it, and I would love to see feedback and opinions on the list.

The biggest problem with OAM is that it never developed a community around it. My hope is that with an increase in interest in the past $shortWhile, there is sufficient interest to build a community this time around, and with that, enable the project to succeed in a way that it couldn’t 6 months or a year ago.