Technical Ramblings

Day 2 of the code sprint seemed to be much more productive. With much of the planning done yesterday, today groups were able to sit down and get to work.

Today, I accomplished two significant tasks:

• Setting up the new OSGeo Gallery, which is set to act as a repository for demos of OSGeo software users in the same way that the OpenLayers Gallery already does for OpenLayers. We’ve even added the first example.
• TMS Minidriver support for the GDAL WMS Driver: Sitting down and hacking out a way to access OSM tiles as a GDAL datasource, Schuyler and I built something which is reasonably simple/small — an 18k patch including examples and docs — but allows for a significant change in the ability to read tiles from existing tileset datasources on the web.

Other things happening at the sprint today were more WKT Raster discussions, liblas hacking, and single-pass MapServer discussions, as well as some profiling of MapServer performance with help from Paul and Shark. Thanks to the participation of the LizardTech folks, I think there will also be some performance testing done with MrSID rendering within MapServer, and there was — as always — more discussion of the “proj strings are expensive to look up!” discussion.

Other than that, it was a quiet day; lots of work getting done, but not much excitement in the ranks.

We then had a great dinner at Baton Rouge, and made it home.

This evening, I’ve been doing a bit more hacking, opening a GDAL Trac ticket for an issue Schuyler bumped into with the sqlite driver, and pondering the plan for OpenLayers tomorrow.

As before, a special thanks to the conference sponsors for today: Coordinate Solutions via David Lowther, and the lovely folks at SJ Geophysics Ltd.. Thanks for helping make this thing happen! I can guarantee that neither of those GDAL tickets would have happened without this time.

I’m here at the OSGeo Code Sprint in Toronto, where more than 20 OSGeo hackers have gathered to work on all things OSGeo — or at least MapServer, GDAL/OGR, and PostGIS.

For those who might not know, a code sprint is an event designed to gather a number of people working on the same software together with the intention of working together to get a large amount of development work done quickly. In this case, the sprint is a meeting of the “C tribe”: Developers working on the C-based stack in OSGeo.

After some discussion yesterday, there ended up being approximately 3 groups at the sprint:

• People targeting MapServer development
• PostGIS developers
• liblas developers

(As usual, I’m a floater, but primarily concentrating on OpenLayers; Schuyler will be joining me in this pursuit, and I’ve got another hacker coming Monday and Tuesday to sprint with us.)

The MapServer group was the most lively discussion group (and is also the largest). It sounded like there were three significant development discussions that were taking place: XML Mapfiles, integration of pluggable rendering backends, and performance enhancements, as well as work on documentation.

After a long discussion on the benefits/merits of XML mapfiles, it came down to there being one main target use case for the XML mapfile is encouraging the creation and use of more editing clients. With a format that can be easily round-tripped between client and server, you might see more editors able to really speak the same language. In order to test this hypothesis, a standard XSLT transform will be created and documented, with a tool to do the conversion; this will allow MapServer to test out the development before integrating XML mapfile support into the library itself.

I didn’t listen as closely to the pluggable renderers discussion, but I am aware that there’s a desire to improve support and reduce code duplication of various sorts, and the primary author of the AGG rendering support is here and participating in the sprint. Recently, there has been a proposal to the list to add OpenGL based rendering support to MapServer, so this is a step in that direction.

The PostGIS group was excited to have so many people in the same place at the same time, and I think came close to skipping lunch in order to get more time working together. In the end, they did go, but it seemed to be a highly productive meeting. Among some of their discussions was a small amount of discusssion on the WKTRaster project which is currently ongoing, I believe.

After our first day of coding, we headed to a Toronto Marlies hockey game. This was, for many of us, the first professional hockey we’d ever seen. (The Marlies are the equivilant of AAA baseball; one step below the major leagues.) The Canadians in the audience, especially Jeff McKenna, who played professional hockey for a time, helped keep the rest of us informed. The Marlies lost 6-1, sadly, but as a non-Canadian, I had to root a bit for the Hershey team. (Two fights did break out; pictures forthcoming.)

We finished up with a great dinner at East Side Mario’s.

A special thanks to our two sponsors for the day, Rich Greenwood of Greenwood Map and Steve Lehr from QPUBLIC! Our sprint was in a great place, very productive, and had great events, thanks to the support of these great people.

Looking forward to another great day.

In the last couple weeks I’ve contributed to two other OSGeo projects, besides OpenLayers:

• GDAL: Better documentation for new Web Map Service driver
• MapServer: Patched shp2img to allow changing image size.

Even though they are minor changes, it feels nice to participate outside the OpenLayers/MetaCarta bubble.

Today, I finally made the switch from the 7 month-running Mapnik based homepage map on the boston freemap to a MapServer 5.0-based AGG rendering.

The new map has had more cartography put into than any existing Boston Freemap to date. Maps like:

Cambridgeport
The state of Massachusetts
Downtown Boston

All demonstrate some of the qualities offered by the new MapServer.

Among other things, the new maps:

• Have had the labels lowercased, to fit more text in (Python OGR script I wrote, help from #gdal)
• Have had sortshp run over them, to order them so that you don’t get weird overlaps (help from danmo)
• Have had additional cartography work done in order to create a better fit for labels
• Benefit from the excellent AGG rendering work I’ve been describing here, by tbonfort.

In my opinion, the look and feel of these maps really does approach the level of quality that the commercial services have set, as far as rendering goes. MapServer’s leaps and bounds in the past couple weeks in rendering have changed the game from being a “Work on the rendering” game back to a “Work on the cartography” game.

I want to send a special thanks to tbonfort and all the work he’s done recently to make this happen! The rendering is simply incredible in comparison — truly night and day. I’m so happy to be back in MapServer land where the hard work I’ve done on this cartography hasn’t gone to waste. He and the other MapServer developers have really brought me back into the fold with the latest work, and I’m happy to be there.

The font work really doesn’t show up well in a small screenshot — it’s a total affect that makes a difference. Comparing AGG to GD is probably pretty informative though.

A MapServer developer (Thomas Bonfort) pinged me this morning, using my Boston Freemap as remote testing of a patch for MapServer’s agg rendering. (It was a very minor difference: little rounding errors were causing small black pixels in some places on the map.) In the process of testing a patch for him, he also pointed out that the newest SVN has added AGG symbol support.

The end result is that my comparison from the other day has now gotten even better:

MS 4.10:

SVN Rev 6432 + minor patch:

Now the only thing that’s left is to fix those labels… Wonder if I can learn enough C++ in the next week or two to help get that in. 😉

Nicely done work on the symbols! Better every day.

With Steve Lime’s help on my previous post on the topic, I’ve been able to turn my previously somewhat dissappointing map into something that is definitely coming out way ahead of where it was. (Click the image to see a bigger version of the same area).

First trick: Removing ‘antialias true’/’transparency alpha’ lines from my mapfile, as Zac and Steve both mentioned on the previous post. This actually did most of the work. Looking back, had I actually removed them all, that would have done all of the work, but I missed one, so my polygon fills were still not working. Thankfully, Steve is a great man, and committed fixes for that to trunk, so now I’ve got real pretty images serving up out of the Boston Freemap. With these changes, the label problems aren’t nearly as pronounced as I’d seen them be on other maps.

I definitely feel like this is a huge step in the right direction.

Old:

New:

Pay extra attention to the T lines, which are one of the more obvious failings of the GD rendering.

Many many thanks to Steve for his help in turning the rather disappointing map into something that I’m really very happy with. I’m glad to be proven wrong — in exactly the way I hoped I would be — on the problems with MapServer’s AGG rendering. I’m looking forward to converting more maps in the near future.

The MapServer 5.0 beta is out. For a while, people had been mentioning the AGG support, and I even saw a pretty great demo put together by DM Solutions that demonstrated the functionality, and I had been impressed.

I was disappointed to learn recently that the fonts are still going to continue to be rendered by GD for the 5.0 release: really, the biggest case where I care about smooth lines is in fonts, because it makes the difference between the ability to use an 8px font, and the corresponding increase in number of labels, and being forced to use a 10px/11px font. (Small fonts become less and less readable at smaller sizes without sub-pixel rendering.) However, I was hoping that the rest of the support would be sufficient that I would be happy with switching the Boston Freemap over to using AGG instead of GD as the renderer.

This morning, I grabbed MapServer, and figured out how to build it after some stumbling. MapServer’s default of “Don’t include anything” is probably a great idea for most people, but for me, it definitely was at stumbling block. Just an example: I was trying to set up –with-agg. I knew I had installed it, and files were in /usr/include/agg2 and /usr/lib/libagg.a. But there’s only one directory! So I tried just pointing to /usr/lib, and got:

error: “could not find agg_rendering_buffer.h or libagg.a in /usr/lib.”

Well, obviously agg_rendering_buffer.h wasn’t there, but libagg.a was! I got there eventually by reading the configure script. It actually wants you to point to /usr, which would have been clearer if it asked for ‘AGG *prefix*’ instead of just ‘AGG directory’. I’m sure this stuff is obvious to people who roll their own, but I love my debian, damnit! 😉

I then ran into an error in building mapagg.cpp, #2188, which I was able to fix by commenting out the offending lines, and an error in building php_mapscript, which I solved by removing that from the build.

I eventually got it built, and bumped into what I eventually filed as a bug: #2187: WMS Request for DRIVER AGG/PNG claims ‘invalid output format’. FrankW was kind enough to help me fix mapwms.c so I could continue to play.

I now have an agg demo of the Boston Freemap. However, I find it very disappointing. The lines actually look *more* jagged in some cases, and for fonts without an outlinecolor, the text is almost unreadable. Additionally, it seems that the blue of my water has changed to be a weird grey color. You can compare against 5.0’s GD rendering and 4.10’s GD rendering — the two differ only by what likely amounts to rounding errors.

The agg rendering is simply not up to snuff. I’ll admit I’ve put more than my fair share of work into getting the Boston Freemap to look good with previous versions of MapServer, but this definitely seems like a step backwards.

I’d love to be told that I’m missing some obvious “Make Output Pretty” option, but if there is one, the AGG RFC doesn’t mention it.

I’d hoped to have a brilliant demo of the power of MapServer. Instead, I’ve got disappointment and pain. It’s a shame that it has to end this way… 😉

The mapfile for the Boston Freemap is, of course, available, if someone feels like trying to understand better why the problems exist. The data is also available, though it’s likely easier to just get it direct from MassGIS.

A bit more than a year ago, when I was just learning how to use the Google Maps API, I put together a patch for the OpenGuides software, adding Google Maps support. It seemed the logical way to go: It wasn’t perfect, since Google Maps are obviously non-free, but it seemed like a better way to get the geographic output from OpenGuides out there than anything else at the time.

Since I did that, I’ve learned a lot. Remember that 18 months ago, I’d never installed MapServer, had no idea what PostGIS was, and didn’t realize that there were free alternatives to some of the things that Google had done. Also, 9 months ago, there was no OpenLayers, or any decent open alternative to the Google Maps API.

In the past 18 months, that’s all changed. I’ve done map cartography, I’ve done setting up of map servers, and I worked full time for several months on the OpenLayers project. Although my direction has changed slightly, I still work heavily with maps on a daily basis, and spend more of my time on things like TileCache, which lets you serve map tiles at hundreds of requests/second.

So, about a month ago, I went back to the Open Guide to Boston, and converted all the Google Maps API calls to OpenLayers API calls. The conversion took about an hour, as I replaced all the templates with the different code. (If I was writing it again, it would have taken less time, but this was my first large scale open source Javascript undertaking, long before I gained the knowledge I now have from working with OpenLayers.) In that hour, I was able to convert all the existing maps to use free data from MassGIS, rather than the copyrighted data from Google, and to have Google as a backup: a Map of Furniture Stores can show you the different. You’ll see that there are several layers — one of which is a roadmap provided by me, one from Google — and one from the USGS, topographic quad charts.

It’s possible that some of this could have been done using Google as the tool. There’s nothing really magical here. But now, the data in the guide is no longer displayed by default on top of closed source data that no one can have access to. Instead, it’s displayed on top of an open dataset provided by my state government.

This is how the world should work. The data that the government collects should be made available to the people for things exactly like this. It shouldn’t require a ‘grassroots remapping’: There are examples out there of how to do it right. I find it so depressing to talk to friends in the UK, who not only don’t have the 1:5000 scale quality road data that Massachusetts provides, but doesn’t even provide TIGER-level data that the geocoder on the Open Guide to Boston uses.

Free Guides, with Free Maps. That’s the way it should be. The fact that it isn’t everywhere is sad, but at least it’s good to know that the technology is there. Switching from Google to OpenLayers is an easy task — it’s what happens next that is a problem. You need the data from somewhere, and it’s unfortunate that that ‘somewhere’ needs to be Google for so many people. I’m thankful to MassGIS and to the US Government for providing the data I can use, and to all the people who helped me learn enough to realize that using Google for everything is heading the wrong way when you want to not be beholden to a specific set of restrictions placed on a corporate entity.

One of the problems I’m running into with mapserver right now is related to its rendering of LINE elements which are wider than one as they run into tile boundaries at acute angles. It seems that mapserver is drawing the centerlines for these elements up to the side of the image — but in cases where a line is approaching a boundary at an acute angle, this means that the ‘outer’ edge of the rendered line stops away from the edge.

In non anti-aliased lines, this is less visible as a problem (especially if you’re not looking at the images as tiles) because the lines just stop — and visually, it’s hard to tell if it’s at the image edge. However, it becomes very obvious in cases where anti-aliasing is on because the edges of the left and right boundary are ‘tied’ together by a curving, anti-alised line: resulting in a bubbled look at tile boundaries.

I’m not sure if this is a known bug, or something that other people have run into: I’m mostly recording it here so that I have a description of it.

Right now, it seems like the workarounds are:
* Use thinner lines (so it’s less visible)
* Don’t have roads near boundaries at acute angles — although there’s not much I can do about this one!

Caching headers. Geoserver has an outstanding JIRA item about this, and a geoserver mailing list post describes the problem:

Putting squid in front of geoserver can help tremendously, but squid is very reluctant to cache anything without proper caching meta-information in the appropriate http headers.

This applies to mapserver as well.

Additionally, when running mapserver behind squid, libcurl sends “Pragma: no-cache” headers, so even if a remote mapserver instance supported caching (which it doesn’t), it wouldn’t be cached when behind a proxy. I think this is fixable by adding ‘Pragma:’ to the header setting code in maphttp.c, but I tried that and couldn’t get it to work.

The combinations of these would make tiling map apps much more realistic, since it would reduce load when requesting tiles from the map server.