Technical Ramblings

OpenLayers testing for new users was always felt a bit odd at things like code sprints: because the OpenLayers tests use XMLHttpRequest, Popup windows, and the like, there was always an issue of a few tests that would fail outside of being run on an HTTP server. For a product where almost all the tests pass just fine without it, I always found it sort of annoying that a few minor XMLHttpRequest restrictions forced me to shell out to a server.

This weekend, as I was helping at the OpenHatch Open Source workshop at RPI, I found myself in a position where a new developer was running the tests, and asking me why they failed. I was pointing out that in order for them to pass, they’d have to be run from a webserver, and someone else in the room helpfully pointed out that if you have Python installed, you have a webserver available to you with just one line of code.

“What?” I said, incredulously. I mean, I believed them — in the same way that python -mjson.tool has become a daily part of my life, I’m not entirely surprised by Python modules offering useful command line interactions that help make my life easier. Still, this was a new one to me.

“Sure”, came the reply. “Just use python -m SimpleHTTPServer in the directory you want to serve.”

And I `cd`’d into the root of my OpenLayers checkout, and typed python -m SimpleHTTPServer, and went to http://localhost:8000/tests/run-tests.html — and ‘lo, the tests did pass, and the developer did say it was Good.

(I probably learned more tips and tricks in the two day workshop about git, and other helpful tools, than I do in a week of doing my own development. Kids these days, teaching me new things!)

Occasionally, I think back to when we started writing OpenLayers, and some of the tools we didn’t have when I started programming JavaScript. Then I feel old, and start to yell at kids to get off my lawn.

In May of 2006, when we started working on OpenLayers:

• Internet Explorer was 63% of w3cschools web traffic. (Today? 19%.)
• IE7 wouldn’t be released for another 5 months.
• SVG support was only available via the Adobe SVG plugin, and only in IE on most platforms.
• Safari was at version 1.2/1.3.
• Firefox was not yet at version 1.5, which would bring in SVG support, but disabled by default.
• There was no Firebug. “Real men use Venkman!” (I believe that as part of the rewrite of OpenLayers that we eventually shipped, we did bump into Firebug 0.3/0.4. 1.0 wouldn’t be released for another 6 months.)
• jQuery was still 6 months from being released.

In addition to the JavaScript world changing, the Maps world has changed. Although I was originally interested in OpenLayers because of OpenStreetMap, there wasn’t a lot there back in 2006. That isn’t the only way the world has changed:

• When OpenLayers started, OpenStreetMap had approximately 2000 registered users. (Today? 500,000.) At the time, there was no regular dump, and the map that existed was… ‘interesting’ 🙂 (Mapnik wouldn’t come until later.)
• Installing PostGIS on most platforms was… touchy at best. (Things like pgRouting, though coming into existence around that time, were far from practical to install, even more than a year later.)
• ka-Map and Community Map Builder were still the de facto web mapping software.
• There was no one in the open source world caching XYZ tiles yet. (The FOSS4G discussion on tile caching in September of 2006 was the first real discussion of that.) TileCache was developed later that year — after a discussion where we all agreed that WMS-style strings were a good idea, and then someone left the room and immediately started talking about TMS 🙂
• All map rendering software was somewhat difficult to install at the time — things like GeoServer’s current wonderful web UI were… not as complete then as they are now 🙂
• Nobody knew how to render things in ‘Spherical Mercator’ so that they matched up to Google. Spatial Reference codes like 41001, 900913, 3857/3875 were all quite a ways down the road.
• Software that hasn’t changed much: GDAL/OGR. GDAL was an extremely useful tool in 2006 — pretty much the same as it is today. Although GDAL has certainly grown many features, and more complete over the years, it still has the same general shape as it did back then. 🙂

(Other things OpenLayers predates: Twitter, open access to Facebook.)

As you would expect, the world has changed. People sometimes comment that OpenLayers feels a bit long in the tooth — something I can certainly sympathize with. I have always prioritized maintaining API compatibility for existing applications over any thing else in my personal investment in OpenLayers: the most important thing to do is not to break existing applications. This stability has allowed many people to use OpenLayers, and I don’t think that violating those principles is a good thing. (I am happy with the solution that has grown over the past 6 months in OpenLayers — moving code to the “deprecated.js” file is a great way to let people maintain backwards compatibility with a path forward as well.)

I’m happy to have other people take the principles created by OpenLayers over the past half decade and do something exciting with them. Competition is good. Options for applications are good. The fact that OpenLayers effectively sucked all of the air out of the room from 2006-2010 was not good for the rest of the web mapping world: without competition, it’s really hard for any innovation to take place.

But the fact that a piece of JavaScript software written in a world before jQuery, before Firebug, while OpenStreetMap was still getting off the ground, is still useful today — I think that’s a testament to what OpenLayers became, and I’m happy to see what it has become and continues to be for many people.

In 2002, I got an account on LiveJournal. In order to help my friends get accounts, I started doing LiveJournal support. (I got 5 support points in my first week, which was a reasonably big deal back then. (Warning: Reading other posts from that era of my life is not recommended :))

Over time, as a somewhat more technical member of the support team, I started attempting to explore links to this ‘zilla’ thing that people kept mentioning, LiveJournal’s bug tracker. However, I always found that I never really understood what was going on: looking at the content above the fold in a typical Bugzilla bug is a bunch of confusing looking metadata to a new developer. I remember at some point when someone told me to look at the comment — and I remember saying something along the lines of “Wait, there’s more content in this page if I scroll down?” Yes, I had been reading bugs for weeks, and never realized that … there was information other than the metadata.

I remember somewhat later, as I got more into development, that I would actually read patch files and attempt to understand what they were doing. In one case, I found a patch file written by a friend in the support community that had a pretty clear typo in it. So, I downloaded the patch file, opened it in an editor, added the new line that was missing, and added the “+” at the beginning of the line — like he had for all of his lines, of course — and re-uploaded the patch.

Of course, anyone with a passing knowledge of tools will know that adding a single line to a patch file … isn’t going to go well. 🙂 But to someone who didn’t know that patchfiles were produced and applied with ‘patch’, ‘diff’, etc., this concept was a strange one.

So, in my start in open source, I didn’t know how to read bug reports; I didn’t know how patchfiles worked. Yet I still was able to learn, improve, and eventually, do a little bit of good in the open source world, thanks to the help of a lot of people along the way.

We all start somewhere.

This weekend, I sat down and got comfy with Scalatron, a programming tutorial based around building a competitive bot in a game environment in Scala.

At first, I was a bit stymied; with no Java-friendly IDE, and the instructions being really IDE targeted, I had some problems getting my development environment set up. Thanks to some help from godwinsgo on Twitter, I got set up with an environment where I was able to have:
– Automatic compiling on file save, which was super helpful (rather than having to do something else)
– The ability to quickly swap back over, tap a button, and have Scalatron reload my bot and test it right away.

After some playing, I wrote a bot that is pretty good: it gets ~22000 in the ‘reference’ (bot on its own, 5000 steps, 100×100 field). I have a better understanding of some of the simple mistakes that I made — adding a println at the end of my function when I have a return type expected will return a somewhat cryptic “Expected Type, Got Unit” error message pointing to a spot in my code that it doesn’t make sense to me; same with really any case where I’m not paying careful attention to return types, since without an explicit ‘return’ statement, what gets returned is confusing to me.

I probably wrote terrible Scala; I didn’t bother to learn much about the language, just enough to actually hack my bot into something that worked. (My bot is on github, if people want to critique my terrible Scala — or code in general.) However, I was able to get working, learned some things about Scala, and wrote a cool digital robot. (Hooray!)

Thanks to David Winslow for giving me the bits to running, and to Scalatron for helping me have a fun project for this weekend!

“When a junior developer has decided they need to build a whole new framework to solve a relatively simple problem, and they’ve started building it instead of fixing the problem, that’s an unpleasant surprise. When they’ve a week in to it, and have been making difficult-to-undo changes to support it, that’s a nasty surprise.” — Scrum, the Good Bits: Daily Standups

So true. And it points to Picture Hanging — a metaphor that a couple years ago would have been completely foreign to me, but is now so much a part of my daily life that I take it for granted…

So, in general, I’ve avoided multithreaded processing; it’s one of those things that historically has been tricky to get right, and I don’t typically have embarrassingly parallel problems.

Today, however, I was parsing a set of 25,000 HTML files using BeatifulSoup, to pull out a small set of data (~500 bytes of JSON per HTML file). I briefly tried to simplify some of the code, but then realized that the lion’s share of the CPU time was being spent on the initial parse; there wasn’t going to be a way to clean up my code enough to make the script that much faster, no matter how good the rest of my code was.

Enter multiprocessing. With a 5 line change to my Python code, I was able to move from one core to four. Instead of:

def handle_place():
     for i in glob.glob("beerplaces/*"):
           # Do stuff with I
     return data

I have:

from multiprocessing import Pool
def handle_place(filename):
    # Do stuff with filename
if __name__ == "__main__":
    p = Pool(4)
    data = p.map(handle_place, glob.glob("places/*"))

Once I made the change, I went from using one CPU fully to using all four — and instead of taking 25 minutes to generate my output, the total time was under 7.

multiprocessing is cool.

One revelation I had yesterday, when participating at BarCamp is… I still actually care about maps.

Now, those people who observe me probably know this. Although I have moved out of the map creation business — my day job is no longer hacking on OpenLayers, though amusingly, I actually end up *using* OpenLayers more now than I did most of the time at MetaCarta — I still enjoy maps, and still work in a field where maps play a big role, for example. However, I’ve felt for a long time like I was burned out on maps: After 5-6 years of working on the same thing, I was tired of the same discussions over and over again.

Turns out — that probably isn’t the case. Instead, what I got bored of was dealing with questions about Javascript-based web mapping. I got tired of only existing as “The guy people ask OpenLayers questions to”; I got tired of being a one-trick wonder, even if that trick earned me the title, for a while, of “Boy Genius” (literally: it was on my business cards).

I ran into Bill Wendel at Barcamp Boston yesterday, and when I pulled out my array of phones (Nexus S, Nokia C7, Nokia N9, Lumia 710), he commented that he remembered me talking at Barcamp Boston last year about the need to constantly reinvent yourself: if you’re not learning something new, you’re falling behind. I kept one hat on for what was — for me! — quite a while: Web mapping technology was the core of my existence in a lot of my development for 5 years. (Also changed in those 5 years: Got married. Child moved from elementary school to applying to high schools. Moved to current apartment. etc.) 5 years is a long time. A really long time. Being “the OpenLayers guy” just isn’t cutting it anymore.

But the mapping world still interests me: even today, it seems I have a lot of information in my head that isn’t ‘common knowledge’ about the mapping industry. (For example: Yahoo’s mapping platform is almost entirely provided by Nokia these days; something that most people aren’t aware of.) Even if I’m no longer as actively interested in doing the software development side of things, I still love talking maps — I love sharing information about how the world used to be, and how we’re seeing it change.

To me, this is news. I know that this is probably a sign that I’m insufficiently good at self-examination, but I really just thought that I was ‘done’ with Maps: finding out that indeed, I’m not done with maps, but just with mapping software, is an interesting revelation. I’m looking forward to Bill reaching back out and helping build up a group of people in Boston interested in mapping again. Back in 2006, we used to meet up regularly — and this was the early days of OpenLayers, so the meetups were more informal, and less code. (This is before we were part of OSGeo, for example.) As I moved more into the software development side, so did the meetups — but I realize that in doing that, I missed a key factor: the software of maps isn’t the interesting part. The *stories* matter. Software is the boring part. Leaving out the users and targeting the developers was the wrong way to go, and I want to see if going back away from that opens more interesting doors… again.

These days, the things I do have changed pretty drastically from what they were a couple years back. Since I’m attending BarCampBoston this weekend, I figured I should make a So, here’s a collection of things I know and do:

• Django: I have worked with Django since long before 1.0, using it for many different projects. The longest running project type I have used it for is managing distribution of Mechanical Turk style annotation tasks among a large pool of annotators; At MetaCarta, this was ~10-15 people; at Nokia, it’s more than 140 part-time workers around the world, producing thousands of responses to task judgements each day.
• Data Analysis — At Nokia, I also help do analysis against logs for the Nokia Map Search service, analyzing client trends, usage, and extracting information from search and click logs to help improve the overall quality of Map Search
• Web Development — a bit part of what I do at Nokia, as it has been in every job, is the creation of web UIs to make information more easily accessible. Typically using simple, plain HTML + Javascript interfaces, I work to make it easier for other people to get their job done by presenting information in a more easily accessible form; this can include anything from basic data aggregation to more complex map related display interfaces.
• AWS — At Nokia and outside of Nokia, I have done a fair amount of work around Amazon Web Services, using everything from EC2 to Cloudwatch to SNS to run services at a larger scale than I have before. Using Amazon and other ‘cloud’ services has been a big step forward in my knowledge of software deployment and development.

At Nokia, I work primarily on improving Map Search. This runs the gamut from annotation data and measurement to deployment — if there’s a part of the Nokia Map Search backend that I don’t know at least a little bit about, I probably will soon. At home, I dabble in photography, phone geekery, and various random hacking projects. I’m no longer quite the Open Source GIS guru that I was once — I’m no longer the go-to guy on questions about OpenLayers, for example — but I can now tell you a lot more about AWS than I could in those days 🙂

So, if you meet me at BarCampBoston and get to my blog: now you know. I’m Christopher Schmidt, and I’m a web hacker.