Technical Ramblings

Earlier this evening, Atrus pointed out that DC has a bunch of cool data in their GIS Data Catalog. I decided I would play with it a bit and see what I could come up with.

I grabbed the Street Centerlines, played with it in QGIS to do a bit of cartography, and then (eventually) got it exported to a MapServer .map file (which describes styling info). I was then able to set the file up in MapServer, serve it out to OpenLayers, and then to stick TileCache in the mix. The result isn’t the prettiest thing in the world, but it works.
After going through it once, I decided I’d go through it all again, to see how long it took.

• 12:15AM: Open Firefox to the DC Data Catalog to find some data to map.
• 12:16AM: Pick out Structures Polygons.
• 12:17AM: Download complete, open QGIS
• 12:18AM: Open file in QGIS
• 12:19AM: Save QGIS project file, save map file from project file
• 12:20AM: Copy both shapefile and mapfile to server
• 12:21AM: Tweak mapfile: adjust PNG output to not be interlaced (for TileCache usage), change background color
• 12:22AM: Test mapfile in mapserv CGI. Find out I misspelled something, fix it.
• 12:23AM: Edit TileCache config to add new layer information.
• 12:24AM: Copy an existing tile URL, ensure that it works in TileCache with the different layer.
• 12:25AM: Edit OpenLayers config to include additional layer
• 12:26AM: Edit OpenLayers config to include layerswitcher.
• 12:27AM: Marvel at the result

In less than 15 minutes I was able to turn a dataset into a browsable, lazily cached web viewable data set, using qgis, OpenLayers, and TileCache. Not bad at all.

A bit more than a year ago, when I was just learning how to use the Google Maps API, I put together a patch for the OpenGuides software, adding Google Maps support. It seemed the logical way to go: It wasn’t perfect, since Google Maps are obviously non-free, but it seemed like a better way to get the geographic output from OpenGuides out there than anything else at the time.

Since I did that, I’ve learned a lot. Remember that 18 months ago, I’d never installed MapServer, had no idea what PostGIS was, and didn’t realize that there were free alternatives to some of the things that Google had done. Also, 9 months ago, there was no OpenLayers, or any decent open alternative to the Google Maps API.

In the past 18 months, that’s all changed. I’ve done map cartography, I’ve done setting up of map servers, and I worked full time for several months on the OpenLayers project. Although my direction has changed slightly, I still work heavily with maps on a daily basis, and spend more of my time on things like TileCache, which lets you serve map tiles at hundreds of requests/second.

So, about a month ago, I went back to the Open Guide to Boston, and converted all the Google Maps API calls to OpenLayers API calls. The conversion took about an hour, as I replaced all the templates with the different code. (If I was writing it again, it would have taken less time, but this was my first large scale open source Javascript undertaking, long before I gained the knowledge I now have from working with OpenLayers.) In that hour, I was able to convert all the existing maps to use free data from MassGIS, rather than the copyrighted data from Google, and to have Google as a backup: a Map of Furniture Stores can show you the different. You’ll see that there are several layers — one of which is a roadmap provided by me, one from Google — and one from the USGS, topographic quad charts.

It’s possible that some of this could have been done using Google as the tool. There’s nothing really magical here. But now, the data in the guide is no longer displayed by default on top of closed source data that no one can have access to. Instead, it’s displayed on top of an open dataset provided by my state government.

This is how the world should work. The data that the government collects should be made available to the people for things exactly like this. It shouldn’t require a ‘grassroots remapping’: There are examples out there of how to do it right. I find it so depressing to talk to friends in the UK, who not only don’t have the 1:5000 scale quality road data that Massachusetts provides, but doesn’t even provide TIGER-level data that the geocoder on the Open Guide to Boston uses.

Free Guides, with Free Maps. That’s the way it should be. The fact that it isn’t everywhere is sad, but at least it’s good to know that the technology is there. Switching from Google to OpenLayers is an easy task — it’s what happens next that is a problem. You need the data from somewhere, and it’s unfortunate that that ‘somewhere’ needs to be Google for so many people. I’m thankful to MassGIS and to the US Government for providing the data I can use, and to all the people who helped me learn enough to realize that using Google for everything is heading the wrong way when you want to not be beholden to a specific set of restrictions placed on a corporate entity.

So it seems clear to me that the Pipes application is a step in a really cool direction. I don’t know if there’s anything incredibly innovative in the idea of making programming easy, but Yahoo! has gone a long way towards the goals that other people have put into place. Ning thought that letting people code would be the way forward: give them a sandbox, let them copy paste, and they’ll build applications. The idea was right: there are a lot more people out there who want to be builders that aren’t. It turned out that the people who want to be builders didn’t have the skill level that they needed to build PHP code, even with mix/match and copy/paste.

Yahoo! Pipes is the followthrough on that idea: make it possible for people to take a set of input, and get a set of output, passing it through multiple filters.

The next step is obvious: Let people turn the filter settings into a web page, with the output being another web page. Search for all content 5 miles from a given Craigslist location: Take the user input as drop down boxes or something in an HTML form, and make the output a Yahoo! Map. Boom: you’ve turned everyone who can create a pipe into a web application builder. Stick ads along the bottom, and you’ve done one of the things that Ning tried to do: make money off applications in the same way that so many have made money off content.

I’m sure that Yahoo! already has this in mind, whether they’ve written about it or done it yet or not. It’s only a matter of time. It does make me wonder if someone could build something that did this without needing Yahoo! to do it… It seems like at the moment it would require altering a pipe on the fly, which I don’t see a way to do, so either there needs to be a further API, or we’ll all just need for it to get done 🙂

Update: Looking today, you can control the input of text inputs from the URL that you fetch the RSS with. This means that I can go ahead and build the pipe thingy for my own pipes as is. That’s pretty cool. I’ll show one with MetaCarta stuff on Monday.

Perhaps I’ll build an OpenLayers based Yahoo Pipe output viewer. It wouldn’t be that different from the GeoRSS viewer… but it would need a way to visualize non-Geo content. Ponder ponder.

Recently, an OpenLayers contributor posted about his recent development of an interactive zoom for OpenLayers:

Animated Zoom Demo (Drag the scroll bar up and down to see the feature in action.)

I love this demo, because it’s a demonstration of one of the things you can do with OpenLayers that you simply can’t do with Google Maps. Looking forward to seeing OpenLayers work with the developer in question to integrate his patches and hard work.