Technical Ramblings

Interested in OpenStreetMap? In the Boston area — or considering travelling here with your lucky companion for Valentine’s Day? Come to the OpenStreetMap mapping party, in Somerville, MA on Feb. 14th and 15th, and help put your house on the map… or anything else you might run across. I’m hoping to be there — in part to meet other OSM interested people in the area, in part to defend my actions in uploading all the houses in the metro-Boston area and making the map quite pretty to look at, but annoying slow to edit.

Sean Gillies just wrote a great post on the use of Python decorators to help you write prettier code. he didn’t quite go into what decorators are underneath though, which is something I think that it’s important to realize to understand how decorators work.

Decorators are just function wrappers around your functions. In all the languages I use on a regular basis, functions are just another variable: they can be passed around in the same way objects can. The pretty syntax for decorators is just a way to say “Pass this function into the function I’ve just defined, and return me a new function, called the same thing as my old one was.”

If you are happy with only supporting Python2.4 and above, that’s a great way to work. Sadly, not all of us are: for example, the current release of Jython is still at 2.2. 😉 More seriously, people who are maintaining older systems may not have newer Python functions available yet.

That doesn’t mean you can’t get the benefit of decorators — at least, I’ve never found it to mean that. Instead, it just means your code is a bit less ‘pretty’ to look at. Instead of functions being ‘deocrated’:

@logprints
def work():
    print "Starting some work."
    print "Doing some work ..."
    print "Finished the work ..."

You can simply wrap your function in the decorator:

def work():
    print "Starting some work."
    print "Doing some work ..."
    print "Finished the work ..."
work = logprints(work)

The idea of passing functions around is one of the things that took me a while to get used to, but learning it has helped me with a lot of code since then.

TileCache and FeatureServer don’t use decorators, specifically because they seek to support older Python versions. If you’re writing code that’s only forward looking, using all the advanced features of Python may fit your bill. But when you find yourself on an old machine some day, where all you have is Python 2.2, sometimes it’s nice to know a little bit about what’s going on underneath.

Earlier today, I tried Jython for the first time, because I’m doing some work that may involve interactions with Java libraries in the near future. Jython, which I’ve always avoided in the past due to an irrational fear of Java, is “an implementation of the high-level, dynamic, object-oriented language Python written in 100% Pure Java, and seamlessly integrated with the Java platform.” (I love projects that have great one-liners that I can copy paste.)

My goal for Jython was to do some work with the GeoTools EPSG registry code related to SpatialReference.org. Sadly, I didn’t get that working, but in the process, I learned that Jython now has a beta version which is up to Python 2.5 — much newer than the 2.2 that had previously been available.

With that in hand, I decided to see if I could get some of my other Python projects running under Jython. I’m the maintainer for both TileCache and FeatureServer — two pure Python projects. Theoretically, these projects should both work trivially under Jython, but I’ve always had my doubts/fears about this being the case. However, it turns out that my fears here are entirely unfounded.

I downloaded the FeatureServer ‘full’ release from featureserver.org: this includes the supporting libraries needed to get a basic FeatureServer up and running. I then tried to run the FeatureSever local HTTP server… and it worked out of the box. I was able to Load the layer, save data to it, query it, etc. with no problems whatsoever. Java has support for the DBM driver that FeatureServer uses by default, so out of the box, I was able to use FeatureServer with Jython without problems.

Next came TileCache. TileCache was originally built to support Python all the way back to 2.2, so I wasn’t expecting many problems. Getting it running turned out to be almost as easy: the only code modification that was needed was a minor change to the disk cache, because Jython doesn’t seem to support the ‘umask’ method. Once I changed that (now checked into SVN), Jython worked just as well with TileCache as it did with FeatureServer.

Clearly, there are some things which are less trivial. The reason that these libraries were so easy to use is because they were designed to be low-dependancy: TileCache and FeatureServer default paths are both entirely free of compiled code. Using something like, for example, GDAL Layers in TileCache, would be much more difficult (if it’s even possible).

However, this presents some interesting capabilities I had not previously thought of.

For FeatureServer, this means that it may be possible to write a DataSource which accesses SDE using the ArcSDE Java API, ESRI’s supported method for accessing their SDE databases. One of the purported “holy grails” of the GIS world is RESTful access to SDE databases via lightweight servers — Jython may provide a path to that, if someone is interested in it. (It may be that this has become a moot point with the release of the ESRI 9.3 REST API — I’m not really sure.) This may be a waste of time, but the fact that it *can* be done is interesting to me. Edit: Howard points out that ArcSDE read/write support exists in OGR 1.6, so this is a moot point; you can simply use OGR to do this without involving Jython/Java.

I think this might also speak to a possibility of having better answers available for people who want to use things like FeatureServer from Java platforms (though I don’t know enough about jython to be sure): the typical answer of “use GeoServer” is great, but to be able to provide something a bit more friendly would be interesting. Thankfully, the Java world is largely catching up to the advances made in TileCache/FeatureServer, so this is also less urgent than it has been in the past.

In the end, this was likely simply an interesting experiment. However, it’s nice to know that the capabilities to do things like this within Jython are improving, and that Jython continues to advance their Python. The 2.2 release being the ‘current’ one still is disappointing, but seeing a 2.5 beta available is an exciting development.

As I said, the current version of FeatureServer works out of the box with Jython, and I’ll be doing a TileCache release shortly that will work with Jython out of the box as well. It’s neat to see more possibilities for using these libraries I’ve spent so much time on.

So, the hotel I’m currently staying in uses a classed system of internet access: you can pay $n for so many hours of internet at a certain ‘service level’.

After some experimentation, it seems that what this actually does is put you in a QoS bracket for your HTTP traffic, where you’re apparently grouped with other people in the same bracket. “Bronze” gets 30k downstream, “Silver” gets 60k downstream, and “Gold” gets to max out the connection. Only HTTP traffic is limited in this fashion: other traffic simply falls into the Gold bucket by default.

What does this mean for me? Well, currently someone else in the hotel in the ‘gold’ bucket is using up all the bandwidth. As a result, I can’t use the web while in the Gold bucket. I can, however, get perfectly usable bandwidth when using the Silver and Bronze buckets.

The worst part of this is that most of my traffic that I care about goes through ssh — and ssh isn’t monitored/blocked, so it doesn’t get into a different QoS bucket. The end result is that I can use the web — but not if I use the highest service level. And no matter what I do, I can’t use ssh at all.

What a pain. I’m *so* looking forward to being back in the states in another day and a half and having usable internet again…

MetaCarta Labs is looking for a new employee again.

MetaCarta Labs is a small team of enthusiastic, fast paced developers working directly under the CTO to develop prototypes and new products and facilitate developer outreach. MetaCarta Labs is taking a key role in geo-enabling the Web — helping existing content owners to add geographic information to their content — and has a long history of working with external developer communities through open source projects and conferences.

ландшафтoptical communications
Most people who’ve been long time readers know that I’ve been working at MetaCarta for two years, and I’ve really had a great time there. Some of the smartest people I know are my coworkers, and I’ve learned more working with them than I have anywhere else. I’ve travelled to conferences in other countries, met people from around the world, worked on several open source projects, and directly participated in a number of interesting revenue-building things within the company.

Interest in geo a must: experience in geo optional. We’re looking for a web hacker, primarily: interest and experience in HTML and JS is pretty much a prerequisite for what we’re looking for at the moment.

It’s a cool gig: if you know anyone who might be interested, feel free to send them my way, either directly (crschmidt@metacarta.com) or via the link above.

So, a couple people have asked me why Google thinks that my site may harm your computer.

On Wednesday, I received an email from “Google Search Quality”:

We recently discovered that some of your pages can cause users to be infected with malicious software. We have begun showing a warning page to users who visit these pages by clicking a search result on Google.com.

I looked through, and sure enough, the links they offered were indeed ‘infected’: I’ve always used WordPress despite my knowledge of the fact that it has security exploits more often than I get around to fixing them (though I do try to keep up). They had has a small iframe included, which claimed to be ‘stats tracking’: Instead, there was Javascript included which, presumably, was malicious.

To the best of my knowledge, I solved this problem on Wednesday night, by removing the mal-links that were pointed out, and patching the security holes I could find fixes to in WordPress. (I just upgraded to WordPress 2.5; In the past, upgrading has been painful, but it wasn’t so bad this time, and there are not yet any known security holes for 2.5 that I’m aware of.)

All in all, not a bad thing: Google emailed me, I fixed the problem, everyone wins. Except…

Following Google’s FAQs, I went to Webmaster Tools, signed up, verified my site, went to their tools…

And in the site management tools, found no such link as they described. Great.

At the time, I assumed only Google was using stopbadware: I’ve since discovered that other things are using it, so I’ve requested reconsideration there.

Still, Google now tells users that my site may be dangerous, despite the fact that it no longer is, and there appears to be no tool in the website management ‘tools’ panel to have them check it out again. Stretches the definition of ‘Do No Evil’ a bit… (Edit: Okay, not really, but it always works when you really want to get a response out of Google to just tell them they’re being evil: People get defensive and help you out ;))

In any case, my web site should be safe. Sorry that people have been confused by the problem.

Edit: JohnMu in comments pointed out why I was having a problem: Since crschmidt.net/blog/ was the only thing listed as ‘infected’, I had to sign up and verify for crschmidt.net/blog/ *seperately* from crschmidt.net. Certainly not exactly intuitive, but doing so allowed me to request a review of my site, so hopefully soon people will be able to view my site again in FF3, and won’t be caught out by Google’s warning (assuming I got all my malware off).

It’s my Birthday!

Mapping related gifts I’ve picked up so far:

• Mapping Boston, which is gigantic and heavy and beautiful.
• A Wonder Globe, which is small and spinny and will go on my desk at work
• A National Geo map of Germany, for hanging on the walls at work
• Transit Maps of the World, which a friend who is staying with us picked up from off my recently revamped Amazon Wishlist 🙂

Not a bad haul, since I’ve only been 24 for 18 minutes 😉

I’m in DC on Wednesday morning for a meeting: Flying in tonight, arriving around 6, staying at the Holiday Inn in Reston. If there’s anyone who I should be stopping over and having a beer with tonight, let me know: at the moment, my plan is just to wander around DC monuments for a couple hours tomorrow night, since I Haven’t been able to find anyone I should be meeting. (I’m booked until my plane leaves on Wednesday; the trip is not about seeing DC, just a quick hop down for work.)

OSGeo Users Group meeting in Boston tomorrow (well, it’s already tomorrow, but you know what I mean) night, 7pm at MIT Museum in Cambridge.

When: Wednesday, Oct 17th, 7PM
Where: MIT Museum, Downstairs, “MIT360” space,
265 Massachusetts Ave., Cambridge, MA
http://web.mit.edu/museum/
Boston Freemap
What: First Meeting of Boston OSGeo users group: intro, and maybe review of FOSS4G?
Who: Anyone interested in open source geo software
Why: Because it’s there!

I figure we’ll spend an hour or two getting to know people, talking about FOSS4G (for those of us who went… that might just be me), etc.

If you plan to be there, please respond to the list so people know to expect you! (A ‘maybe’ is fine.)

My first attempt to do something useful with IronPython:

>>> import urllib
>>> urllib.urlopen("http://crschmidt.net/")
Traceback (most recent call last):
  File httplib, line unknown, in getreply
  File httplib, line unknown, in getresponse
  File httplib, line unknown, in __init__
  File System, line unknown, in set_ReceiveBufferSize
  File System, line unknown, in SetSocketOption
WindowsError: Invalid arguments

Note that I’m working on OS X, and my exception is a WindowsError. Fancy.

(I was inspired by Bill Thorp’s efforts to get TileCache working on IronPython: Round 1, Round 2. However, I’m not all that inspired now.)

Still, it is kind of cool that IronPython just ran — I didn’t expect that to work. Maybe there’s something to this mono business after all.