Technical Ramblings

Earlier this evening, Atrus pointed out that DC has a bunch of cool data in their GIS Data Catalog. I decided I would play with it a bit and see what I could come up with.

I grabbed the Street Centerlines, played with it in QGIS to do a bit of cartography, and then (eventually) got it exported to a MapServer .map file (which describes styling info). I was then able to set the file up in MapServer, serve it out to OpenLayers, and then to stick TileCache in the mix. The result isn’t the prettiest thing in the world, but it works.
After going through it once, I decided I’d go through it all again, to see how long it took.

• 12:15AM: Open Firefox to the DC Data Catalog to find some data to map.
• 12:16AM: Pick out Structures Polygons.
• 12:17AM: Download complete, open QGIS
• 12:18AM: Open file in QGIS
• 12:19AM: Save QGIS project file, save map file from project file
• 12:20AM: Copy both shapefile and mapfile to server
• 12:21AM: Tweak mapfile: adjust PNG output to not be interlaced (for TileCache usage), change background color
• 12:22AM: Test mapfile in mapserv CGI. Find out I misspelled something, fix it.
• 12:23AM: Edit TileCache config to add new layer information.
• 12:24AM: Copy an existing tile URL, ensure that it works in TileCache with the different layer.
• 12:25AM: Edit OpenLayers config to include additional layer
• 12:26AM: Edit OpenLayers config to include layerswitcher.
• 12:27AM: Marvel at the result

In less than 15 minutes I was able to turn a dataset into a browsable, lazily cached web viewable data set, using qgis, OpenLayers, and TileCache. Not bad at all.

Edward Ho responds regarding callbacks in Pipes:

Apologies for the inconsistency here. JSON Callback is something we believe in for our APIs and we plan on adding it to Pipes shortly.
Thanks!

Woot.

So, I went through this morning and created GeoRSS output for MetaCarta search results. It seems likely that for the forseeable future, the way to get data in/out of Yahoo! Pipes is to build a URL-input, RSS output format. Since we already do KML, adding RSS took about 20 minutes — just changing some tag names, etc.

After doing that, I started playing with some pipes. After some missteps — dragging modules into the interface seemed to not work in a way that later did, presumably just part of the fun of dealing with these funky interfaces — I got something that I think is pretty cool coming out.

• Take NY Times input. Recent news is relevant and interesting.
• Pass the NY Times input through the content analyzer. This gets a set of relevant keywords.
• For each item, take the keywords and pass them through a subpipe…
• the subpipe goes to the MetaCarta search interface, and does a keyword search, taking the session information and bounding box from the text inputs for the pipe, returning 5 most relevant item.
• Add the 5 results to the item, and then send the output to the end of the pipe.

I’ve published the pipe, and after running it you can get the JSON output. (It seems that the sub-attributes are not available in the RSS output, at least as far as I can tell.)
Next steps:

• Create an OpenLayers interface, and use the current bounding box as the bounding box for the queries.
• Have the interface set the session and token variables: the pipe currently works with the defaults, but only for the next couple hours, since the tokens are time limited. (Currently, the MetaCarta Web Services are wide open, but eventually, they won’t be.)

The Query MC subpipe is probably useful for anyone who wants to do something like this: I’d recommend checking it out if you’re interested. Next step is to add RSS support to the other services, specifically the GeoTagger. I think that this would let me do what I want: take description text from an RSS item, tag it, and set the location attributes of the returned item.

I’m hoping that someone can answer how to use a callback on JSON output of pipes, since that would let me skip the need for a proxy, and make it entirely client side so that other people could run it on their own sites without a proxy.

A bit more than a year ago, when I was just learning how to use the Google Maps API, I put together a patch for the OpenGuides software, adding Google Maps support. It seemed the logical way to go: It wasn’t perfect, since Google Maps are obviously non-free, but it seemed like a better way to get the geographic output from OpenGuides out there than anything else at the time.

Since I did that, I’ve learned a lot. Remember that 18 months ago, I’d never installed MapServer, had no idea what PostGIS was, and didn’t realize that there were free alternatives to some of the things that Google had done. Also, 9 months ago, there was no OpenLayers, or any decent open alternative to the Google Maps API.

In the past 18 months, that’s all changed. I’ve done map cartography, I’ve done setting up of map servers, and I worked full time for several months on the OpenLayers project. Although my direction has changed slightly, I still work heavily with maps on a daily basis, and spend more of my time on things like TileCache, which lets you serve map tiles at hundreds of requests/second.

So, about a month ago, I went back to the Open Guide to Boston, and converted all the Google Maps API calls to OpenLayers API calls. The conversion took about an hour, as I replaced all the templates with the different code. (If I was writing it again, it would have taken less time, but this was my first large scale open source Javascript undertaking, long before I gained the knowledge I now have from working with OpenLayers.) In that hour, I was able to convert all the existing maps to use free data from MassGIS, rather than the copyrighted data from Google, and to have Google as a backup: a Map of Furniture Stores can show you the different. You’ll see that there are several layers — one of which is a roadmap provided by me, one from Google — and one from the USGS, topographic quad charts.

It’s possible that some of this could have been done using Google as the tool. There’s nothing really magical here. But now, the data in the guide is no longer displayed by default on top of closed source data that no one can have access to. Instead, it’s displayed on top of an open dataset provided by my state government.

This is how the world should work. The data that the government collects should be made available to the people for things exactly like this. It shouldn’t require a ‘grassroots remapping’: There are examples out there of how to do it right. I find it so depressing to talk to friends in the UK, who not only don’t have the 1:5000 scale quality road data that Massachusetts provides, but doesn’t even provide TIGER-level data that the geocoder on the Open Guide to Boston uses.

Free Guides, with Free Maps. That’s the way it should be. The fact that it isn’t everywhere is sad, but at least it’s good to know that the technology is there. Switching from Google to OpenLayers is an easy task — it’s what happens next that is a problem. You need the data from somewhere, and it’s unfortunate that that ‘somewhere’ needs to be Google for so many people. I’m thankful to MassGIS and to the US Government for providing the data I can use, and to all the people who helped me learn enough to realize that using Google for everything is heading the wrong way when you want to not be beholden to a specific set of restrictions placed on a corporate entity.

So it seems clear to me that the Pipes application is a step in a really cool direction. I don’t know if there’s anything incredibly innovative in the idea of making programming easy, but Yahoo! has gone a long way towards the goals that other people have put into place. Ning thought that letting people code would be the way forward: give them a sandbox, let them copy paste, and they’ll build applications. The idea was right: there are a lot more people out there who want to be builders that aren’t. It turned out that the people who want to be builders didn’t have the skill level that they needed to build PHP code, even with mix/match and copy/paste.

Yahoo! Pipes is the followthrough on that idea: make it possible for people to take a set of input, and get a set of output, passing it through multiple filters.

The next step is obvious: Let people turn the filter settings into a web page, with the output being another web page. Search for all content 5 miles from a given Craigslist location: Take the user input as drop down boxes or something in an HTML form, and make the output a Yahoo! Map. Boom: you’ve turned everyone who can create a pipe into a web application builder. Stick ads along the bottom, and you’ve done one of the things that Ning tried to do: make money off applications in the same way that so many have made money off content.

I’m sure that Yahoo! already has this in mind, whether they’ve written about it or done it yet or not. It’s only a matter of time. It does make me wonder if someone could build something that did this without needing Yahoo! to do it… It seems like at the moment it would require altering a pipe on the fly, which I don’t see a way to do, so either there needs to be a further API, or we’ll all just need for it to get done 🙂

Update: Looking today, you can control the input of text inputs from the URL that you fetch the RSS with. This means that I can go ahead and build the pipe thingy for my own pipes as is. That’s pretty cool. I’ll show one with MetaCarta stuff on Monday.

Perhaps I’ll build an OpenLayers based Yahoo Pipe output viewer. It wouldn’t be that different from the GeoRSS viewer… but it would need a way to visualize non-Geo content. Ponder ponder.

A prize to anyone who can make a simple Atom entry or a simple RSS entry get geocoded by the Yahoo! Pipes Location Extractor. I’ve spent the last 30 minutes on it, and failed.

Non-working pipe is my attempt. GeoRSS works, but location extraction doesn’t.

Update:

1. Location Extractor seems to work against the HTML pages referenced by the feed, not the content in the feed.
2. Minor changes to the HTML page seem to break the parsing — it seems to be very targeted towards Craigslist postings.  A page with my address, but a different map link seems to extract the Sebastapool address, while a page with just a map link doesn’t seem to extract at all.

I guess I don’t need to sell my MetaCarta stock yet… unless I’m way off, this shows that MetaCarta is significantly ahead of the game for extracting locations from unstructured text. Not that this is a surprise to me 😉

Is it possible to make your own module with Yahoo! Pipes? I was looking around and didn’t see anything… I’d really like to be able to hook up something that grabs locations from the MetaCarta Web Services, and then let people drop it into their own pipelines… I’d be willing to bet that the Location Extractor pipe module wouldn’t pull out “20 miles north of London”, but with the MetaCarta GeoTagger, I could…

It’s cool when software works.

So, I needed to test the latest TileCache in Windows. I’ve got a relatively clean Windows laptop I keep around for this. None of the niceties that I would typically want on a machine I used full time — no Firefox, no cygwin, no vim, etc. Just a straight up Windows install, with things like Google Earth installed.

So, starting from that, I started by downloading Apache from the Apache Download Page. I downloaded “Win32 Binary (MSI Installer): apache_2.2.4-win32-x86-no_ssl.msi“, and ran the installer directly. This got http://localhost/ serving a page that says “It works!”.

Next, I downloaded the Python 2.5 Windows installer , and again selected all the defaults.

Next, I downloaded TileCache 1.4, as a .zip file, since I know how well Windows supports .tar.gz files. I extracted the zip file into C:\Program Files\Apache Software Foundation\Apache2.2\cgi-bin.

Lastly, I edited C:\Program Files\Apache Software Foundation\Apache2.2\cgi-bin\tilecache-1.4\tilecache.cgi so that the first line of the file read “#!C:/Python25/python.exe -u”, and visited http://localhost/cgi-bin/tilecache-1.4/tilecache.cgi/1.0.0/basic/0/0/0.png …

And it worked! I got an image. I never expected setting up Apache, Python, and TileCache to just… well… work.

I just spent 30 minutes or so trying to understand the Python Packaging Policy for Debian. I’m no more informed than I was an hour ago, and my head hurts a lot more. :/

This isn’t helped by the fact that I want to support etch++ (because that’s what most people use) and sarge (because that’s what I’m working on at work). Sigh. Packaging is hard.

For most of the stuff I package, I just dump things in someplace in /usr/lib/, but now I need to figure out how to get my python code into the right place in /usr/lib/pythonX.Y/site-packages … but my modules support any version of Python that debian ships! So where do they go? I have no clue. Do I need to build multiple binary packages — one for each version of Python — for my single source pacakge? I have a feeling that something is supposed to take care of this for me, but I don’t know if it’s dh_pycentral, dh_python, or dh_pythonsupport or what have you, and I don’t know how to get any of them to kick in. Augh.

Oh well. Packaging this one will just have to wait a bit longer.

This has been posted to a couple mailing lists, but posting it here can’t hurt:

The FOSS4G (Free and Open Source Software for Geospatial) conference is pleased to announce the Call for Workshops for the 2007 conference, being held September 24-27 in beautiful Victoria, British Columbia, Canada.

FOSS4G is the premier conference for the open source geospatial community, providing a place for developers, users, and people new to open source geospatial to get a full-immersion experience in both established and leading edge geospatial technologies.

This is your chance to showcase your favorite application, integration solution, or other topic. You will use your superior classroom skills to lead a group of attendees through your chosen topic in either a half-day or ninety minute lab or classroom format. Half-day workshops will be delivered on Monday, September 24 (the Workshops day), while the ninety minute workshops will run concurrently with the presentations during the remainder of the conference.

While we are open to workshops on a wide range of topics, we strongly encourage workshop submissions on the following topics:

• Practical Introduction to __________
• Interoperability
• NeoGeography and NovelGeography
• Using a Software Stack
• 3D Worlds

In the tradition of previous FOSS4G events, we expect that the majority of workshops will be “hands on”, with participants seated in front of computers and able to follow along with the instructor, working directly with the software and applications under discussion.

Be prepared to spend considerable effort in creating your workshop. Past experience has shown that a high quality workshop requires about three days of preparation for each hour of presentation time. As part of this preparation you will be expected to develop material for attendees to take away with them, such as handouts, a ‘workbook’, CDROM, etc.

In recognition of this effort, workshop presenters will receive a reduction in the price of conference registration:

• free registration for delivering a half-day workshop
• half-price registration for delivering a 90-minute workshop

Because of limited space, you may want to consider submitting two versions of your topic, one for each length format.

Please visit the workshops page on our website to download the submission templates and instructions for sending them in:

http://www.foss4g2007.org/workshops.html

The deadline for workshop submissions is February 28, 2007. Submit early, submit often!