Archive for the 'Software' Category

Update on Near Term Goals for TileCache

Posted in Locality and Space, TileCache on July 20th, 2007 at 14:56:00

Threshold bug: Fixed. 1.9-rc1,2 released, and a user even confirmed that it fixed his threshold problem (yay)! This will probably be the last 1.9 release, and includes significantly improved error reporting.

Refactor for drop in Layers: done, in the ‘refactor’ branch. I think I’ve maintained backwards compatibility at the ‘using a config file’ level, even, though it does need an additional option for mod_python support, so it will definitely be a 2.0. I also added drop in support for caches.

Add ArcXML layer: Done, though I have no clue how people actually use ArcXML, so this support may actually be total crap. I got it to work with the MassGIS site though. (Would love to talk to an actually ArcIMS/ArcXML user and figure out what I should be doing that I’m not.) I bet you’ve never seen ArcIMS look so fast.

Add KML Region Support: Not done, and not really too excited about it. I plan to get the 1.9 final out the door, then release the 2.0 without it, and maybe loop around for the KML region support later.

Overall, not bad.

Near term goals for TileCache

Posted in Locality and Space, TileCache on July 18th, 2007 at 19:29:05
  • Fix threshhold bug, release 1.9, call it done
  • Refactor to make support for dynamic loading of layer types easier (a la FeatureServer)
  • Add ArcXML Layer (so that I can skip having to do it in OpenLayers)
  • Add KML Region support

First Attempt at IronPython

Posted in default, IronPython, Locality and Space, Python, TileCache on June 20th, 2007 at 06:53:26

My first attempt to do something useful with IronPython:

>>> import urllib
>>> urllib.urlopen("http://crschmidt.net/")
Traceback (most recent call last):
  File httplib, line unknown, in getreply
  File httplib, line unknown, in getresponse
  File httplib, line unknown, in __init__
  File System, line unknown, in set_ReceiveBufferSize
  File System, line unknown, in SetSocketOption
WindowsError: Invalid arguments

Note that I’m working on OS X, and my exception is a WindowsError. Fancy.

(I was inspired by Bill Thorp’s efforts to get TileCache working on IronPython: Round 1, Round 2. However, I’m not all that inspired now.)

Still, it is kind of cool that IronPython just ran — I didn’t expect that to work. Maybe there’s something to this mono business after all.

Telling Stories Via Maps

Posted in FeatureServer, OpenLayers, TileCache on May 15th, 2007 at 01:49:25

screenshot of OpenLayers-based story MapWent out for a late night walk tonight: though my path was a bit unclear, I think I made a decently accurate map:

Walking around MIT

If you hover over the circles, you’ll see points of interest along the route. I recommend zooming into the main MIT building, to see the confused meandering I was forced to.

This demo uses OpenLayers, TileCache, and FeatureServer. It was created entirely via a web interface.

PostGIS and Stored Functions in… Python?

Posted in PostGIS, Spatial Databases on April 27th, 2007 at 20:40:44

This week, I had the oppourtunity to work with Schuyler in writing stored procedures in Postgres/PostGIS for the first time.

At first, we were writing in plpgsql, but found it didn’t suit our needs… and switched to Python.

Yes, Python. Our database now has stored procedures which decode a cPickle pickle structure from a column (go go unstructured data) and return the output of a key/value pair (based on the key).

It was a very weird thing to see this actually work.

From Data To Map

Posted in Locality and Space, OpenLayers, QGIS, TileCache on February 14th, 2007 at 00:47:25

Earlier this evening, Atrus pointed out that DC has a bunch of cool data in their GIS Data Catalog. I decided I would play with it a bit and see what I could come up with.

I grabbed the Street Centerlines, played with it in QGIS to do a bit of cartography, and then (eventually) got it exported to a MapServer .map file (which describes styling info). I was then able to set the file up in MapServer, serve it out to OpenLayers, and then to stick TileCache in the mix. The result isn’t the prettiest thing in the world, but it works.
After going through it once, I decided I’d go through it all again, to see how long it took.

  • 12:15AM: Open Firefox to the DC Data Catalog to find some data to map.
  • 12:16AM: Pick out Structures Polygons.
  • 12:17AM: Download complete, open QGIS
  • 12:18AM: Open file in QGIS
  • 12:19AM: Save QGIS project file, save map file from project file
  • 12:20AM: Copy both shapefile and mapfile to server
  • 12:21AM: Tweak mapfile: adjust PNG output to not be interlaced (for TileCache usage), change background color
  • 12:22AM: Test mapfile in mapserv CGI. Find out I misspelled something, fix it.
  • 12:23AM: Edit TileCache config to add new layer information.
  • 12:24AM: Copy an existing tile URL, ensure that it works in TileCache with the different layer.
  • 12:25AM: Edit OpenLayers config to include additional layer
  • 12:26AM: Edit OpenLayers config to include layerswitcher.
  • 12:27AM: Marvel at the result

In less than 15 minutes I was able to turn a dataset into a browsable, lazily cached web viewable data set, using qgis, OpenLayers, and TileCache. Not bad at all.

Free Maps for Free Guides

Posted in Locality and Space, Mapserver, OpenGuides, OpenLayers, TileCache on February 11th, 2007 at 08:46:05

A bit more than a year ago, when I was just learning how to use the Google Maps API, I put together a patch for the OpenGuides software, adding Google Maps support. It seemed the logical way to go: It wasn’t perfect, since Google Maps are obviously non-free, but it seemed like a better way to get the geographic output from OpenGuides out there than anything else at the time.

Since I did that, I’ve learned a lot. Remember that 18 months ago, I’d never installed MapServer, had no idea what PostGIS was, and didn’t realize that there were free alternatives to some of the things that Google had done. Also, 9 months ago, there was no OpenLayers, or any decent open alternative to the Google Maps API.

In the past 18 months, that’s all changed. I’ve done map cartography, I’ve done setting up of map servers, and I worked full time for several months on the OpenLayers project. Although my direction has changed slightly, I still work heavily with maps on a daily basis, and spend more of my time on things like TileCache, which lets you serve map tiles at hundreds of requests/second.

So, about a month ago, I went back to the Open Guide to Boston, and converted all the Google Maps API calls to OpenLayers API calls. The conversion took about an hour, as I replaced all the templates with the different code. (If I was writing it again, it would have taken less time, but this was my first large scale open source Javascript undertaking, long before I gained the knowledge I now have from working with OpenLayers.) In that hour, I was able to convert all the existing maps to use free data from MassGIS, rather than the copyrighted data from Google, and to have Google as a backup: a Map of Furniture Stores can show you the different. You’ll see that there are several layers — one of which is a roadmap provided by me, one from Google — and one from the USGS, topographic quad charts.

It’s possible that some of this could have been done using Google as the tool. There’s nothing really magical here. But now, the data in the guide is no longer displayed by default on top of closed source data that no one can have access to. Instead, it’s displayed on top of an open dataset provided by my state government.

This is how the world should work. The data that the government collects should be made available to the people for things exactly like this. It shouldn’t require a ‘grassroots remapping’: There are examples out there of how to do it right. I find it so depressing to talk to friends in the UK, who not only don’t have the 1:5000 scale quality road data that Massachusetts provides, but doesn’t even provide TIGER-level data that the geocoder on the Open Guide to Boston uses.

Free Guides, with Free Maps. That’s the way it should be. The fact that it isn’t everywhere is sad, but at least it’s good to know that the technology is there. Switching from Google to OpenLayers is an easy task — it’s what happens next that is a problem. You need the data from somewhere, and it’s unfortunate that that ‘somewhere’ needs to be Google for so many people. I’m thankful to MassGIS and to the US Government for providing the data I can use, and to all the people who helped me learn enough to realize that using Google for everything is heading the wrong way when you want to not be beholden to a specific set of restrictions placed on a corporate entity.

TileCache Under Windows… It Just Works ™

Posted in Apache, Python, TileCache on February 7th, 2007 at 10:53:06

It’s cool when software works.

So, I needed to test the latest TileCache in Windows. I’ve got a relatively clean Windows laptop I keep around for this. None of the niceties that I would typically want on a machine I used full time — no Firefox, no cygwin, no vim, etc. Just a straight up Windows install, with things like Google Earth installed.

So, starting from that, I started by downloading Apache from the Apache Download Page. I downloaded “Win32 Binary (MSI Installer): apache_2.2.4-win32-x86-no_ssl.msi“, and ran the installer directly. This got http://localhost/ serving a page that says “It works!”.

Next, I downloaded the Python 2.5 Windows installer , and again selected all the defaults.

Next, I downloaded TileCache 1.4, as a .zip file, since I know how well Windows supports .tar.gz files. I extracted the zip file into C:\Program Files\Apache Software Foundation\Apache2.2\cgi-bin.

Lastly, I edited C:\Program Files\Apache Software Foundation\Apache2.2\cgi-bin\tilecache-1.4\tilecache.cgi so that the first line of the file read “#!C:/Python25/python.exe -u”, and visited http://localhost/cgi-bin/tilecache-1.4/tilecache.cgi/1.0.0/basic/0/0/0.png …

And it worked! I got an image. I never expected setting up Apache, Python, and TileCache to just… well… work.

Packaging Python is Hard

Posted in Debian on February 7th, 2007 at 01:21:04

I just spent 30 minutes or so trying to understand the Python Packaging Policy for Debian. I’m no more informed than I was an hour ago, and my head hurts a lot more. :/

This isn’t helped by the fact that I want to support etch++ (because that’s what most people use) and sarge (because that’s what I’m working on at work). Sigh. Packaging is hard.

For most of the stuff I package, I just dump things in someplace in /usr/lib/, but now I need to figure out how to get my python code into the right place in /usr/lib/pythonX.Y/site-packages … but my modules support any version of Python that debian ships! So where do they go? I have no clue. Do I need to build multiple binary packages — one for each version of Python — for my single source pacakge? I have a feeling that something is supposed to take care of this for me, but I don’t know if it’s dh_pycentral, dh_python, or dh_pythonsupport or what have you, and I don’t know how to get any of them to kick in. Augh.

Oh well. Packaging this one will just have to wait a bit longer.

Writing Code…

Posted in Software on November 20th, 2006 at 03:17:28

I think I wrote my first open source C code last night, in a fit of frustration over lack of error feedback in the Mapserver in-image error exception handling. Turns out that the in-image handling only displays the first error: XML service exceptions everything in the list by comparison. I opened a bug for better error reporting via inimage exception handling, and added a patch and example output that the patch results in.

I’m typically a Python/Javascript hacker at this point: TileCache is Python, OpenLayers is Javascript. I did write some Perl code at some point in the last year, for the Open Guide to Boston.

I don’t even write PHP much anymore, though I did write some S2 earlier today to redo the look of my LiveJournal.

So, Open Source code I’ve written… patch for Mapserver: C. Patches for OpenGuides: Perl. Patches for LiveJournal: Perl. Random Semweb utilities: Python. Random web utilities: PHP? I’m pretty sure I’ve released some of that stuff anyway. OpenLayers: Javascript. MetaCarta open source stuff: Python. I can’t think of anything else I’ve done that’s open source… I wonder what I might be forgetting.