Technical Ramblings

So, after a weekend of working with Java and GeoTools, suddenly things got a lot simpler to work with in Jython. This is a case of trying to do the completely wrong thing because I don’t know anything about the project, packaging… or language.

Thankfully, I’m now much better off, and have put together a little HTTP server that runs and gives me back WKT for any EPSG code, using Jython + GeoTools.

The code is very simple: the geotools-epsg-server just takes in a code, and spits out WKT. It uses the GeoTools CRS package, and a BaseHTTPServer.

It’s not much, but it’s enough to get me in the right direction. Maybe I’ll even stop whining about Java so much, since I can use it more like Python…

Nah, that wouldn’t be any fun.

Earlier today, I tried Jython for the first time, because I’m doing some work that may involve interactions with Java libraries in the near future. Jython, which I’ve always avoided in the past due to an irrational fear of Java, is “an implementation of the high-level, dynamic, object-oriented language Python written in 100% Pure Java, and seamlessly integrated with the Java platform.” (I love projects that have great one-liners that I can copy paste.)

My goal for Jython was to do some work with the GeoTools EPSG registry code related to SpatialReference.org. Sadly, I didn’t get that working, but in the process, I learned that Jython now has a beta version which is up to Python 2.5 — much newer than the 2.2 that had previously been available.

With that in hand, I decided to see if I could get some of my other Python projects running under Jython. I’m the maintainer for both TileCache and FeatureServer — two pure Python projects. Theoretically, these projects should both work trivially under Jython, but I’ve always had my doubts/fears about this being the case. However, it turns out that my fears here are entirely unfounded.

I downloaded the FeatureServer ‘full’ release from featureserver.org: this includes the supporting libraries needed to get a basic FeatureServer up and running. I then tried to run the FeatureSever local HTTP server… and it worked out of the box. I was able to Load the layer, save data to it, query it, etc. with no problems whatsoever. Java has support for the DBM driver that FeatureServer uses by default, so out of the box, I was able to use FeatureServer with Jython without problems.

Next came TileCache. TileCache was originally built to support Python all the way back to 2.2, so I wasn’t expecting many problems. Getting it running turned out to be almost as easy: the only code modification that was needed was a minor change to the disk cache, because Jython doesn’t seem to support the ‘umask’ method. Once I changed that (now checked into SVN), Jython worked just as well with TileCache as it did with FeatureServer.

Clearly, there are some things which are less trivial. The reason that these libraries were so easy to use is because they were designed to be low-dependancy: TileCache and FeatureServer default paths are both entirely free of compiled code. Using something like, for example, GDAL Layers in TileCache, would be much more difficult (if it’s even possible).

However, this presents some interesting capabilities I had not previously thought of.

For FeatureServer, this means that it may be possible to write a DataSource which accesses SDE using the ArcSDE Java API, ESRI’s supported method for accessing their SDE databases. One of the purported “holy grails” of the GIS world is RESTful access to SDE databases via lightweight servers — Jython may provide a path to that, if someone is interested in it. (It may be that this has become a moot point with the release of the ESRI 9.3 REST API — I’m not really sure.) This may be a waste of time, but the fact that it *can* be done is interesting to me. Edit: Howard points out that ArcSDE read/write support exists in OGR 1.6, so this is a moot point; you can simply use OGR to do this without involving Jython/Java.

I think this might also speak to a possibility of having better answers available for people who want to use things like FeatureServer from Java platforms (though I don’t know enough about jython to be sure): the typical answer of “use GeoServer” is great, but to be able to provide something a bit more friendly would be interesting. Thankfully, the Java world is largely catching up to the advances made in TileCache/FeatureServer, so this is also less urgent than it has been in the past.

In the end, this was likely simply an interesting experiment. However, it’s nice to know that the capabilities to do things like this within Jython are improving, and that Jython continues to advance their Python. The 2.2 release being the ‘current’ one still is disappointing, but seeing a 2.5 beta available is an exciting development.

As I said, the current version of FeatureServer works out of the box with Jython, and I’ll be doing a TileCache release shortly that will work with Jython out of the box as well. It’s neat to see more possibilities for using these libraries I’ve spent so much time on.

Earlier today, I read a post on the OpenGeo GeoSpiel blog calling for OpenLayers to get on the “Usable Documentation Bandwagon”.

Now, I’ll be honest: I followed the links that he offered, and found something that is not, to me, much more convincing than the OpenLayers documentation as it stands today. If I look at ESRI’s JSAPI, I see a couple things wrong right off the bat — like the fact that I can’t actually link to where I want to. In any case, if you click Geometry -> Point, you see a document which, to me, is a lot less interesting than the similar page provided by OpenLayers.

Perhaps there is some subset of documentation put together by ESRI in their JSAPI that makes sense, but for an API reference, it seems to me that OpenLayers does reasonably well on the portions of our API that someone has invested time to work on.

This doesn’t mean we’re “done” by any stretch of the imagination: One of the things that I’ve wanted to do for ages is actually sit down and do a thorough review of the OpenLayers API documentation and improve a lot of it, targeting cross-linking with other documentation and examples especially. These types of tasks, however, are the types of tasks that require a lot of time, and don’t have a lot of immediate benefit. Since all work on OpenLayers for the past several months has been my personal free time after work, there’s only so much I’m personally able to do. However, no one has *ever* made a request to the OpenLayers team to be able to do this work — including OpenGeo — that I’ve seen. It seems to me that whit is experienced and knowledgable about OpenLayers, since I’ve seen him using it for development, but I’ve never seen him ask to participate in improving the OpenLayers API documentation on the mailing lists, nor have I seen a request of this nature from any other organization.

To me, this means that it’s likely that our API documentation does, to some extent, meet the needs of the organizations using OpenLayers. It’s not perfect — nothing is — but no one thinks it’s a major stumbling block that’s worth fixing, at least not enough to spend money on it.

This is exactly the type of reason that OpenLayers is now accepting Sponsorship from organizations looking to support the project. This type of improvement is exactly the kind that project sponsorship can help support.

OpenLayers is aware of how important documentation is to the success of the project. We have invested dozens of hours between a dozen different contributors to create and maintain a set of relatively complete API documentation. According to Ohloh, of the 55,000 lines in OpenLayers Javascript, over 30% are comments: over 25,000 lines of what is mostly API documentation. No one is ignoring the problem — and if the current state is insufficient (as everything in the project is, *especially* to new users and beginners), then we’re very open to help from any and all interested parties.

API documentation isn’t the only thing a project needs, of course. Documentation comes in all forms — and OpenLayers is seriously lacking in a lot of documentation targeted towards beginners of all kinds. We’ve been working to change that, with a new documentation site available, and other efforts targeted documentation of OpenLayers in English and other languages. I would say these efforts are much less complete than the API documentation, and starting on them is far more important, in my opinion, than improving our API documentation is at this time.

OpenLayers is a large library. It’s used by many organizations. We’re open to contributions of all types — never have I seen OpenLayers turn away someone who wanted to help with documentation intentionally. We have regularly worked with contributors in helping them to improve the documentation, and to claim that we are ignoring the need for documentation seems to me to be representative of a lack of knowledge of the tools that the project uses for documentation, not specifically a lack in the goals of the project, which puts documentation of functionality — via API docs and minimal examples demonstrating functionality — as a requirement of almost all new code in the library.

I presented at FOSS4G 2008 in Tokyo today, thanks to Orkney, OSGeo.jp, and MetaCarta. Gave a talk on OpenLayers: Past, Present, and Future. Hard to gauge success of the talk: many people seemed somewhat bored, but my typical attempts at keeping interest are not so effective in a room where most people don’t speak the same language as I do.

Went over history of OpenLayers — a bit of the pre-“Mark IV” that most people think of as being “OpenLayers”, then into why MetaCarta initially started the project — plus present status/features/developments, then into future (2.x and 3.x), and gave some demos of functionality.

Got a couple good questions — so at least a couple people were interested, which is more than I can actually say about many presentations where people *do* speak the same language. 🙂 Questions are notoriously hard to get out of people at conferences.

Had shrimp tempura for lunch, which should help me keep my energy up for the workshop I have to give in a bit more than an hour. I’ll be presenting using a trimmed down version of Tim/Seb/OpenGeo’s OpenLayers workshop: http://workshops.opengeo.org/openlayers/intro/doc/ to about 10 people, as I understand it. Should be interesting to see how far we can get in 1.5 hours.

Staying awake is going to be tough. The 13 (soon to be 14) hour time difference is quite significant in a way I hadn’t realized.

A friend of mine recently had his account ‘restricted’ by Flickr for “potentially offensive content”. His comments on Flickr’s treatment of the issue are concerning to me: as he describes in his entry:

They still haven’t gotten around to telling me what this content IS or why it is potentially offensive. … “There are millions of people visiting Flickr who may not see the world the same way you do” is certainly true, but a dangerous way to run a community.

I understand Flickr’s needs for the adequate use of controls on users accounts in order to make the site they want it to be, but threats of account termination on a service like Flickr are very concerning to me. I’ve invested a lot of work in my photos on Flickr, and although I’d like to continue to use the service, the feedback of anonymous users being able to lead to account termination with no human feedback in the loop is a scary prospect.

I’d love to hear that this is an oversight somewhere, but if I don’t see Dan get some feedback on his issues, maybe I should consider abandoning Flickr as the primary place to host my photos. Convenience isn’t really worth the possible risk of having an account terminated — presumably taking all its content with it — without even having feedback from a human on why. Clearly, that risk always exists with the Terms of Service that exist on the site — almost every site I’ve seen which has lawyers behind it has some sort of “We can do whatever we want, and in that case, you’re screwed” clause in their terms of use — but to have it become clear that they follow this path in something that seems a routine matter to resolve is a somewhat scary precedent.

In the Flickr authentication API documentation regarding their Authentication Specification, the example in section 9.2 seems to have an invalid value.

They provide the following:

• API Key: 987654321
• Parameters: method=flickr.auth.getFrob
• Secret: foobarbaz

According to section 8, this should result in taking the md5sum of:

foobarbazapi_key987654321methodflickr.auth.getFrob

Which is 91216d07a9e945a8e2bf2b2660e7ba86 . Hoewever, the md5sum used in the api_sig parameter in the example is 5f3870be274f6c49b3e31a0c6728957f. (I’ve tried several different combinations, and been unable to figure out what this value might be based on.)

This yws-flickr post also confirms that the documentation appears to be incorrect.

I couldn’t find the appropriate place to put this given a short search, so I’m publishing it here in hopes that it will remind me to inform someone who might be able to fix it. (There are 5 authors listed on the spec: Emailing all five of them feels like overkill, especially since I’m pretty sure that not all of them still actively work on flickr.) If you know anyone who might be involved, feel free to pass it on…

My most ‘interesting’ photo

One of the things that has always confused me is how Flickr’s ‘interestingness’ score worked. It’s clearly not based directly on views — my most ‘interesting’ photos have ~500 views, while my most popular ones have ~5000. The same is true of comments, tags, groups, etc: Some of my most interesting photos are, all in all, the least obviously ‘popular’ ones.

However, reading through the patent on Interestingness, I see:

[0027] The statistics engine generates statistics and other metrics based upon aggregated metadata. In one embodiment, the statistics engine determines the popularity of metadata (e.g., tags) within a grouping of media objects over a predetermined time period. For example, the statistics engine may determine the number of different users that have assigned a particular tag to one or more media objects within all groups on the system, within a single group, or within a set of media objects, over the last 24 hours. The aggregation engine may determine (and display) a histogram of the tags, and may determine the most frequently assigned tags (at any point in time or over a predetermined time period) by determining those tags either having a frequency exceeding a minimum threshold frequency or belonging to a predetermined number of the most popular tags.

The patent application doesn’t directly claim that this is used in the interestingness calculation, but this type of time-period based tag aggregation/valuation is clearly a non-obvious metric that can’t be calculated by looking directly at user-visible information on a single photo.

Still, none of this really changes the fact that I wish that flickr had a per-person ‘most interesting this week’ sort: my ‘eastern bunny’ photo above has been at the top of the interesting list for ages, and I want new interesting photos, damnit!

One of the most common complaints I’ve heard about FOSS4G this year is the lack of information coming out of the conference. As far as I’m concerned, that’s down to one simple fact: being held in a developing country leaves much to be desired with regard to connecting to the rest of the world in any number of ways.

Whether it be the lack of SMS (too costly to use idly — messages ending up in the US at a service like Twitter end up being $0.25 apiece, even with a local SIM) leading to less Twitter activity, limited internet bandwidth making even getting to a website — much less updating one — far more difficult, lack of decent hotel internet, and paying an arm and a leg when it does exist… the end result is a situation where simply getting online to chat on IRC was far more than many people had an opportunity to do during the conference, and the more regular ‘datastream’ that comes flowing out of most conference attendees during a conference in a more well-connected part of the world was simply nonexistent.

A slightly less technical and more social aspect to this may be the fact that Cape Town bars stay open til 4am, and I know I wasn’t the only FOSS4G attendee they were kicking out at closing time every night.

Combine that with the distance of the trip, and right now, I bet a lot of people are still either here in Africa (as I am: Posting from a Cape Town hotel) or still recovering from their trip, and haven’t had the opportunity to do the detailed breakdowns that people had hoped for.

My take on the conference is that it was great for the people who attended. There was a huge crowd of people who had limited or no experience with Open Source software, or had just recently heard about it and were not really understanding what it meant, or how it worked, or why it mattered. I think that there were a number of people who were interested who were able to learn more — even people who work in communities which are not traditionally open to that kind of solution.

There were a number of presentations on many things that I don’t think we would have seen at a more software developer oriented conference. The uses of GIS — to track water pipeline breakages, to visualize the spread of malaria, to track the best places to plant grapes for growing better wine — are something that we saw much more of at this conference than at previous conferences. I think that this was directly tied to the fact that this conference was clearly a merging of two generally distant groups, and I think the end result was productive for both groups.

The attendance was, from what I heard, 50% ‘locals’, from around South Africa and the rest of Africa. There were many new faces who clearly would not have been able to attend a conference in the US or Europe, and for that alone, there was a benefit to the community as a whole to have the conference in Cape Town.

The organizers did a great job arranging the event: the venue, activities, and sessions were, on the whole, well set up. Also, the food that was provided was some of the best I’ve ever had — at a conference or anywhere else.

Overall, the conference was educational, helped to establish Open Source GIS software in a new location, helped to spread the worldwide knowledge and usage of Open Source software, and introduced the concepts that we in the OSGeo world would like to push onto others wherever appropriate. There was a huge cross breeding of information and cultures that wouldn’t have been possible in another setting, and — despite the distance, cost, and other possible pitfalls that could have befallen the conference — I think that placing the conference in Cape Town was a huge success. There have been a number of doubts voiced as to whether this was really the right decision — certainly, it made attending more difficult for a number of people who would have been at a North American or European event. These doubts and complaints are not to be ignored: It’s true that FOSS4G is meant to be a meeting of the tribes of OSGeo, and that aspect of it was more limited this year than in the past. However, the end result is that hundreds of people who would have had absolutely no chance of making it to an event in the Northern Hemisphere did so, and a number of them have been introduced to and hooked on Open Source software in a way no other event would have been able to.

Placing the conference in Cape Town had a specific goal of spreading Open Source software to someplace that it had not been as successful as possible in the past. With that in mind, this Cape Town conference succeeded in more ways than I can count, and that’s a big win for the conference and for OSGeo as a whole.

I finally finished uploading all the photos I took at/around the FOSS4G conference.

FOSS4G 2008 Set

Includes photos from:

• Various nights hanging out at the bars/hotels during the conference
• A trip up Table Mountain with OpenLayers/OpenGeo folks
• GeoDjango Workshop
• Sessions, exhibition hall, etc.
• Pictures from the Gala Dinner at Moyo
• Closing Session
• OSGeo AGM
• OpenLayers Workshop
• GeoServer Workshop

Photos that are taken at the conference center/of conference proceedings are also tagged with foss4g2008:

foss4g2008 tagged

And most photos of people who I recognized are tagged with first name in the title, and with a username (as used on IRC or other unique identifier) attached as a tag:

Photos of Arnulf
Photos of Steven

Photos of the OSGeo AGM are also tagged as such:

OSGeo AGM Photos

I’ve made my tagging settings as open as possible, but I believe you still need to be a contact on flickr in order to add notes/tags to photos: simply add me as a contact, and I’ll add you back.

Any names I got wrong, please let me know, either via email or by simply commenting on the flickr photo.

Thanks to all for helping to create such a photogenic conference.

So, the hotel I’m currently staying in uses a classed system of internet access: you can pay $n for so many hours of internet at a certain ‘service level’.

After some experimentation, it seems that what this actually does is put you in a QoS bracket for your HTTP traffic, where you’re apparently grouped with other people in the same bracket. “Bronze” gets 30k downstream, “Silver” gets 60k downstream, and “Gold” gets to max out the connection. Only HTTP traffic is limited in this fashion: other traffic simply falls into the Gold bucket by default.

What does this mean for me? Well, currently someone else in the hotel in the ‘gold’ bucket is using up all the bandwidth. As a result, I can’t use the web while in the Gold bucket. I can, however, get perfectly usable bandwidth when using the Silver and Bronze buckets.

The worst part of this is that most of my traffic that I care about goes through ssh — and ssh isn’t monitored/blocked, so it doesn’t get into a different QoS bucket. The end result is that I can use the web — but not if I use the highest service level. And no matter what I do, I can’t use ssh at all.

What a pain. I’m *so* looking forward to being back in the states in another day and a half and having usable internet again…