Technical Ramblings

Having spent some time around SJ Klein the past week, I’m currently hacking on some mapping things for them. I’m also trying to find interesting activities that already exist to try them out: One of the ones I stumbled into is Kuku Anakula (Hungry Chicken). After playing it for a bit, I found it dissappointingly slow to respond to keyboard input. Since Python is a main component in OLPC, it’s written in Python, so I took it apart.

The result is that I refactored the code to make an order of magnitude fewer function calls, and in the process shaved about 50% of the CPU time off, so far as I can tell, when testing on my mac. I learned how to use hotshot, and have the original profiling data and the profile for my refactor. Getting the new version of the activity on the XO was very satisifying: the gameplay is significantly better, and changes the game from being more like a chore to more like a game.

Other things I did today for OLPC include building/installing OGR, and getting FeatureServer set up to run on it with a world borders shapefile. Still got a bit of work to do to build a world map/atlas with click-to-query attributes local on the box, but it’s not too far off.

If you didn’t see them via Slashdot, this set of photos of the OLPC in action is very cute.

Today, I finally made the switch from the 7 month-running Mapnik based homepage map on the boston freemap to a MapServer 5.0-based AGG rendering.

The new map has had more cartography put into than any existing Boston Freemap to date. Maps like:

Cambridgeport
The state of Massachusetts
Downtown Boston

All demonstrate some of the qualities offered by the new MapServer.

Among other things, the new maps:

• Have had the labels lowercased, to fit more text in (Python OGR script I wrote, help from #gdal)
• Have had sortshp run over them, to order them so that you don’t get weird overlaps (help from danmo)
• Have had additional cartography work done in order to create a better fit for labels
• Benefit from the excellent AGG rendering work I’ve been describing here, by tbonfort.

In my opinion, the look and feel of these maps really does approach the level of quality that the commercial services have set, as far as rendering goes. MapServer’s leaps and bounds in the past couple weeks in rendering have changed the game from being a “Work on the rendering” game back to a “Work on the cartography” game.

I want to send a special thanks to tbonfort and all the work he’s done recently to make this happen! The rendering is simply incredible in comparison — truly night and day. I’m so happy to be back in MapServer land where the hard work I’ve done on this cartography hasn’t gone to waste. He and the other MapServer developers have really brought me back into the fold with the latest work, and I’m happy to be there.

The font work really doesn’t show up well in a small screenshot — it’s a total affect that makes a difference. Comparing AGG to GD is probably pretty informative though.

So, OpenLayers wants to use a single well-known number to describe the ‘Google Projection’ I talked about earlier. 54004/54005/41001 are all ‘well known’ as being a standard Mercator projection. Which is not the Google Projection.

So, in addition to http://spatialreference.org/ref/user/google-projection/ / http://spatialreference.org/ref/user/6/ , we need something we can use in “SRS” in WMS — an EPSG code, such as it was.

In order to pick one that doesn’t conflict.

It’s the ‘Google’ projection — so we picked 900913. (Google. In 1337-speak.)

Use it. Love it. Promote it.

Rybelsus authorises the Food and Drug Administration (FDA) to treat type 2 diabetes. It functions best when combined with a balanced diet and regular exercise. Rybelsus may help promote weight loss even though it is not FDA-approved as a medication for weight loss. Additionally, individuals with type 1 diabetes are not permitted to use it… read more

A MapServer developer (Thomas Bonfort) pinged me this morning, using my Boston Freemap as remote testing of a patch for MapServer’s agg rendering. (It was a very minor difference: little rounding errors were causing small black pixels in some places on the map.) In the process of testing a patch for him, he also pointed out that the newest SVN has added AGG symbol support.

The end result is that my comparison from the other day has now gotten even better:

MS 4.10:

SVN Rev 6432 + minor patch:

Now the only thing that’s left is to fix those labels… Wonder if I can learn enough C++ in the next week or two to help get that in. 😉

Nicely done work on the symbols! Better every day.

With Steve Lime’s help on my previous post on the topic, I’ve been able to turn my previously somewhat dissappointing map into something that is definitely coming out way ahead of where it was. (Click the image to see a bigger version of the same area).

First trick: Removing ‘antialias true’/’transparency alpha’ lines from my mapfile, as Zac and Steve both mentioned on the previous post. This actually did most of the work. Looking back, had I actually removed them all, that would have done all of the work, but I missed one, so my polygon fills were still not working. Thankfully, Steve is a great man, and committed fixes for that to trunk, so now I’ve got real pretty images serving up out of the Boston Freemap. With these changes, the label problems aren’t nearly as pronounced as I’d seen them be on other maps.

I definitely feel like this is a huge step in the right direction.

Old:

New:

Pay extra attention to the T lines, which are one of the more obvious failings of the GD rendering.

Many many thanks to Steve for his help in turning the rather disappointing map into something that I’m really very happy with. I’m glad to be proven wrong — in exactly the way I hoped I would be — on the problems with MapServer’s AGG rendering. I’m looking forward to converting more maps in the near future.

So, I’ve just made some updates to spatialreference.org:

• Added comments to EPSG reference pages
• Added human readable wkt output
• Added pagable listing of EPSG codes
• Searching of spatial reference titles
• Searching of WKT

With this, I think I have removed my need to ever have to touch /usr/share/proj/epsg again. The reasons I touched it in the past were usually to search based on title or to add a new projection. Once the new GDAL/OSR is out and installed where I need it, I will be able to use SR.org as references rather than needing to edit proj/epsg to add my own.

Also, for the record, I love Django. This has been the easiest web coding of my life. I spent yesterday porting a half-baked app that was spread over 4 different SVN directories into one Django app. Took me a day, but now instead of a bunch of half-baked stuff that can only be put together by me, I can actually hand the code to someone else and have them run it. (It’s still a bit hard to get up and running on some platforms, because it depends on PostGIS, Python bindings for Postgres, MapScript, OGR, and GDAL. Which are non-trivial on non-Linux platforms.)

It’s great. Thanks to hobu for getting me started with it.

The MapServer 5.0 beta is out. For a while, people had been mentioning the AGG support, and I even saw a pretty great demo put together by DM Solutions that demonstrated the functionality, and I had been impressed.

I was disappointed to learn recently that the fonts are still going to continue to be rendered by GD for the 5.0 release: really, the biggest case where I care about smooth lines is in fonts, because it makes the difference between the ability to use an 8px font, and the corresponding increase in number of labels, and being forced to use a 10px/11px font. (Small fonts become less and less readable at smaller sizes without sub-pixel rendering.) However, I was hoping that the rest of the support would be sufficient that I would be happy with switching the Boston Freemap over to using AGG instead of GD as the renderer.

This morning, I grabbed MapServer, and figured out how to build it after some stumbling. MapServer’s default of “Don’t include anything” is probably a great idea for most people, but for me, it definitely was at stumbling block. Just an example: I was trying to set up –with-agg. I knew I had installed it, and files were in /usr/include/agg2 and /usr/lib/libagg.a. But there’s only one directory! So I tried just pointing to /usr/lib, and got:

error: “could not find agg_rendering_buffer.h or libagg.a in /usr/lib.”

Well, obviously agg_rendering_buffer.h wasn’t there, but libagg.a was! I got there eventually by reading the configure script. It actually wants you to point to /usr, which would have been clearer if it asked for ‘AGG *prefix*’ instead of just ‘AGG directory’. I’m sure this stuff is obvious to people who roll their own, but I love my debian, damnit! 😉

I then ran into an error in building mapagg.cpp, #2188, which I was able to fix by commenting out the offending lines, and an error in building php_mapscript, which I solved by removing that from the build.

I eventually got it built, and bumped into what I eventually filed as a bug: #2187: WMS Request for DRIVER AGG/PNG claims ‘invalid output format’. FrankW was kind enough to help me fix mapwms.c so I could continue to play.

I now have an agg demo of the Boston Freemap. However, I find it very disappointing. The lines actually look *more* jagged in some cases, and for fonts without an outlinecolor, the text is almost unreadable. Additionally, it seems that the blue of my water has changed to be a weird grey color. You can compare against 5.0’s GD rendering and 4.10’s GD rendering — the two differ only by what likely amounts to rounding errors.

The agg rendering is simply not up to snuff. I’ll admit I’ve put more than my fair share of work into getting the Boston Freemap to look good with previous versions of MapServer, but this definitely seems like a step backwards.

I’d love to be told that I’m missing some obvious “Make Output Pretty” option, but if there is one, the AGG RFC doesn’t mention it.

I’d hoped to have a brilliant demo of the power of MapServer. Instead, I’ve got disappointment and pain. It’s a shame that it has to end this way… 😉

The mapfile for the Boston Freemap is, of course, available, if someone feels like trying to understand better why the problems exist. The data is also available, though it’s likely easier to just get it direct from MassGIS.

So, as a result of Custom Projection 6: Google Projection, I got pinged by SharpGIS on IRC, and told that the projection I had there was wrong.

He helped me understand what he was using, and I let him know it still wasn’t helping me. So I set up a Virtual Earth + TileCache demo, and he helped me find the missing piece, in the proj4 FAQ:

In this case, and many other cases using spherical projections, the desired approach is to actually treat the lat/long locations on the sphere as if they were on WGS84 without any adjustments when using them for converting to other coordinate systems. The solution is to “trick” PROJ.4 into applying no change to the lat/long values when going to (and through) WGS84. This can be accomplished by asking PROJ to use a null grid shift file for switching from your spherical lat/long coordinates to WGS84.

+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs

The key here was the +nadgrids=@null, which I’d never seen anywhere before.

The end result?

• Fixing my OSM Tiles
• WMS and TMS overlaid in OpenLayers, as described in an email
• A slightly better understanding of what I didn’t understand for so long.

So. Nice. To not feel lost anymore.

Three cheers for sharpgis! Hooray hooray hooray!

From a post by Howard Butler to gdal-dev:

Chris Schmidt and I have been working on a website that reflects the EPSG spatial reference codes database and user-contributed codes in a variety of output formats. The EPSG code database is never as up to date as you would like, and we frequently tell people to hack their /usr/share/proj/epsg database file with made up codes to support alternative projections. If we could consume URL-based spatial references with OSR, all that would be required is someone adds the custom coordinate system definition to the site and then everyone could refer to it at that location (hopefully indefinitely).

As for the website itself, if it has uptake and people find it useful, we will work to put it somewhere it can live for a very long time — for example OSGeo. The source code (it’s written in Django) will be made available so anyone can stand up their own version of the site.

I’ve been having a lot of fun: this is my first real foray into Django, and I wish I’d done it long ago.

Threshold bug: Fixed. 1.9-rc1,2 released, and a user even confirmed that it fixed his threshold problem (yay)! This will probably be the last 1.9 release, and includes significantly improved error reporting.

Refactor for drop in Layers: done, in the ‘refactor’ branch. I think I’ve maintained backwards compatibility at the ‘using a config file’ level, even, though it does need an additional option for mod_python support, so it will definitely be a 2.0. I also added drop in support for caches.

Add ArcXML layer: Done, though I have no clue how people actually use ArcXML, so this support may actually be total crap. I got it to work with the MassGIS site though. (Would love to talk to an actually ArcIMS/ArcXML user and figure out what I should be doing that I’m not.) I bet you’ve never seen ArcIMS look so fast.

Add KML Region Support: Not done, and not really too excited about it. I plan to get the 1.9 final out the door, then release the 2.0 without it, and maybe loop around for the KML region support later.

Overall, not bad.