Technical Ramblings

A MapServer developer (Thomas Bonfort) pinged me this morning, using my Boston Freemap as remote testing of a patch for MapServer’s agg rendering. (It was a very minor difference: little rounding errors were causing small black pixels in some places on the map.) In the process of testing a patch for him, he also pointed out that the newest SVN has added AGG symbol support.

The end result is that my comparison from the other day has now gotten even better:

MS 4.10:

SVN Rev 6432 + minor patch:

Now the only thing that’s left is to fix those labels… Wonder if I can learn enough C++ in the next week or two to help get that in.

Nicely done work on the symbols! Better every day.

With Steve Lime’s help on my previous post on the topic, I’ve been able to turn my previously somewhat dissappointing map into something that is definitely coming out way ahead of where it was. (Click the image to see a bigger version of the same area).

First trick: Removing ‘antialias true’/'transparency alpha’ lines from my mapfile, as Zac and Steve both mentioned on the previous post. This actually did most of the work. Looking back, had I actually removed them all, that would have done all of the work, but I missed one, so my polygon fills were still not working. Thankfully, Steve is a great man, and committed fixes for that to trunk, so now I’ve got real pretty images serving up out of the Boston Freemap. With these changes, the label problems aren’t nearly as pronounced as I’d seen them be on other maps.

I definitely feel like this is a huge step in the right direction.

Old:

New:

Pay extra attention to the T lines, which are one of the more obvious failings of the GD rendering.

Many many thanks to Steve for his help in turning the rather disappointing map into something that I’m really very happy with. I’m glad to be proven wrong — in exactly the way I hoped I would be — on the problems with MapServer’s AGG rendering. I’m looking forward to converting more maps in the near future.

The MapServer 5.0 beta is out. For a while, people had been mentioning the AGG support, and I even saw a pretty great demo put together by DM Solutions that demonstrated the functionality, and I had been impressed.

I was disappointed to learn recently that the fonts are still going to continue to be rendered by GD for the 5.0 release: really, the biggest case where I care about smooth lines is in fonts, because it makes the difference between the ability to use an 8px font, and the corresponding increase in number of labels, and being forced to use a 10px/11px font. (Small fonts become less and less readable at smaller sizes without sub-pixel rendering.) However, I was hoping that the rest of the support would be sufficient that I would be happy with switching the Boston Freemap over to using AGG instead of GD as the renderer.

This morning, I grabbed MapServer, and figured out how to build it after some stumbling. MapServer’s default of “Don’t include anything” is probably a great idea for most people, but for me, it definitely was at stumbling block. Just an example: I was trying to set up –with-agg. I knew I had installed it, and files were in /usr/include/agg2 and /usr/lib/libagg.a. But there’s only one directory! So I tried just pointing to /usr/lib, and got:

error: “could not find agg_rendering_buffer.h or libagg.a in /usr/lib.”

Well, obviously agg_rendering_buffer.h wasn’t there, but libagg.a was! I got there eventually by reading the configure script. It actually wants you to point to /usr, which would have been clearer if it asked for ‘AGG *prefix*’ instead of just ‘AGG directory’. I’m sure this stuff is obvious to people who roll their own, but I love my debian, damnit!

I then ran into an error in building mapagg.cpp, #2188, which I was able to fix by commenting out the offending lines, and an error in building php_mapscript, which I solved by removing that from the build.

I eventually got it built, and bumped into what I eventually filed as a bug: #2187: WMS Request for DRIVER AGG/PNG claims ‘invalid output format’. FrankW was kind enough to help me fix mapwms.c so I could continue to play.

I now have an agg demo of the Boston Freemap. However, I find it very disappointing. The lines actually look *more* jagged in some cases, and for fonts without an outlinecolor, the text is almost unreadable. Additionally, it seems that the blue of my water has changed to be a weird grey color. You can compare against 5.0’s GD rendering and 4.10’s GD rendering — the two differ only by what likely amounts to rounding errors.

The agg rendering is simply not up to snuff. I’ll admit I’ve put more than my fair share of work into getting the Boston Freemap to look good with previous versions of MapServer, but this definitely seems like a step backwards.

I’d love to be told that I’m missing some obvious “Make Output Pretty” option, but if there is one, the AGG RFC doesn’t mention it.

I’d hoped to have a brilliant demo of the power of MapServer. Instead, I’ve got disappointment and pain. It’s a shame that it has to end this way…

The mapfile for the Boston Freemap is, of course, available, if someone feels like trying to understand better why the problems exist. The data is also available, though it’s likely easier to just get it direct from MassGIS.

Threshold bug: Fixed. 1.9-rc1,2 released, and a user even confirmed that it fixed his threshold problem (yay)! This will probably be the last 1.9 release, and includes significantly improved error reporting.

Refactor for drop in Layers: done, in the ‘refactor’ branch. I think I’ve maintained backwards compatibility at the ‘using a config file’ level, even, though it does need an additional option for mod_python support, so it will definitely be a 2.0. I also added drop in support for caches.

Add ArcXML layer: Done, though I have no clue how people actually use ArcXML, so this support may actually be total crap. I got it to work with the MassGIS site though. (Would love to talk to an actually ArcIMS/ArcXML user and figure out what I should be doing that I’m not.) I bet you’ve never seen ArcIMS look so fast.

Add KML Region Support: Not done, and not really too excited about it. I plan to get the 1.9 final out the door, then release the 2.0 without it, and maybe loop around for the KML region support later.

Overall, not bad.

• Fix threshhold bug, release 1.9, call it done
• Refactor to make support for dynamic loading of layer types easier (a la FeatureServer)
• Add ArcXML Layer (so that I can skip having to do it in OpenLayers)
• Add KML Region support

My first attempt to do something useful with IronPython:

>>> import urllib
>>> urllib.urlopen("http://crschmidt.net/")
Traceback (most recent call last):
  File httplib, line unknown, in getreply
  File httplib, line unknown, in getresponse
  File httplib, line unknown, in __init__
  File System, line unknown, in set_ReceiveBufferSize
  File System, line unknown, in SetSocketOption
WindowsError: Invalid arguments

Note that I’m working on OS X, and my exception is a WindowsError. Fancy.

(I was inspired by Bill Thorp’s efforts to get TileCache working on IronPython: Round 1, Round 2. However, I’m not all that inspired now.)

Still, it is kind of cool that IronPython just ran — I didn’t expect that to work. Maybe there’s something to this mono business after all.

Went out for a late night walk tonight: though my path was a bit unclear, I think I made a decently accurate map:

Walking around MIT

If you hover over the circles, you’ll see points of interest along the route. I recommend zooming into the main MIT building, to see the confused meandering I was forced to.

This demo uses OpenLayers, TileCache, and FeatureServer. It was created entirely via a web interface.

This week, I had the oppourtunity to work with Schuyler in writing stored procedures in Postgres/PostGIS for the first time.

At first, we were writing in plpgsql, but found it didn’t suit our needs… and switched to Python.

Yes, Python. Our database now has stored procedures which decode a cPickle pickle structure from a column (go go unstructured data) and return the output of a key/value pair (based on the key).

It was a very weird thing to see this actually work.

Earlier this evening, Atrus pointed out that DC has a bunch of cool data in their GIS Data Catalog. I decided I would play with it a bit and see what I could come up with.

I grabbed the Street Centerlines, played with it in QGIS to do a bit of cartography, and then (eventually) got it exported to a MapServer .map file (which describes styling info). I was then able to set the file up in MapServer, serve it out to OpenLayers, and then to stick TileCache in the mix. The result isn’t the prettiest thing in the world, but it works.
After going through it once, I decided I’d go through it all again, to see how long it took.

• 12:15AM: Open Firefox to the DC Data Catalog to find some data to map.
• 12:16AM: Pick out Structures Polygons.
• 12:17AM: Download complete, open QGIS
• 12:18AM: Open file in QGIS
• 12:19AM: Save QGIS project file, save map file from project file
• 12:20AM: Copy both shapefile and mapfile to server
• 12:21AM: Tweak mapfile: adjust PNG output to not be interlaced (for TileCache usage), change background color
• 12:22AM: Test mapfile in mapserv CGI. Find out I misspelled something, fix it.
• 12:23AM: Edit TileCache config to add new layer information.
• 12:24AM: Copy an existing tile URL, ensure that it works in TileCache with the different layer.
• 12:25AM: Edit OpenLayers config to include additional layer
• 12:26AM: Edit OpenLayers config to include layerswitcher.
• 12:27AM: Marvel at the result

In less than 15 minutes I was able to turn a dataset into a browsable, lazily cached web viewable data set, using qgis, OpenLayers, and TileCache. Not bad at all.

A bit more than a year ago, when I was just learning how to use the Google Maps API, I put together a patch for the OpenGuides software, adding Google Maps support. It seemed the logical way to go: It wasn’t perfect, since Google Maps are obviously non-free, but it seemed like a better way to get the geographic output from OpenGuides out there than anything else at the time.

Since I did that, I’ve learned a lot. Remember that 18 months ago, I’d never installed MapServer, had no idea what PostGIS was, and didn’t realize that there were free alternatives to some of the things that Google had done. Also, 9 months ago, there was no OpenLayers, or any decent open alternative to the Google Maps API.

In the past 18 months, that’s all changed. I’ve done map cartography, I’ve done setting up of map servers, and I worked full time for several months on the OpenLayers project. Although my direction has changed slightly, I still work heavily with maps on a daily basis, and spend more of my time on things like TileCache, which lets you serve map tiles at hundreds of requests/second.

So, about a month ago, I went back to the Open Guide to Boston, and converted all the Google Maps API calls to OpenLayers API calls. The conversion took about an hour, as I replaced all the templates with the different code. (If I was writing it again, it would have taken less time, but this was my first large scale open source Javascript undertaking, long before I gained the knowledge I now have from working with OpenLayers.) In that hour, I was able to convert all the existing maps to use free data from MassGIS, rather than the copyrighted data from Google, and to have Google as a backup: a Map of Furniture Stores can show you the different. You’ll see that there are several layers — one of which is a roadmap provided by me, one from Google — and one from the USGS, topographic quad charts.

It’s possible that some of this could have been done using Google as the tool. There’s nothing really magical here. But now, the data in the guide is no longer displayed by default on top of closed source data that no one can have access to. Instead, it’s displayed on top of an open dataset provided by my state government.

This is how the world should work. The data that the government collects should be made available to the people for things exactly like this. It shouldn’t require a ‘grassroots remapping’: There are examples out there of how to do it right. I find it so depressing to talk to friends in the UK, who not only don’t have the 1:5000 scale quality road data that Massachusetts provides, but doesn’t even provide TIGER-level data that the geocoder on the Open Guide to Boston uses.

Free Guides, with Free Maps. That’s the way it should be. The fact that it isn’t everywhere is sad, but at least it’s good to know that the technology is there. Switching from Google to OpenLayers is an easy task — it’s what happens next that is a problem. You need the data from somewhere, and it’s unfortunate that that ’somewhere’ needs to be Google for so many people. I’m thankful to MassGIS and to the US Government for providing the data I can use, and to all the people who helped me learn enough to realize that using Google for everything is heading the wrong way when you want to not be beholden to a specific set of restrictions placed on a corporate entity.