Technical Ramblings

A bit more than a year ago, when I was just learning how to use the Google Maps API, I put together a patch for the OpenGuides software, adding Google Maps support. It seemed the logical way to go: It wasn’t perfect, since Google Maps are obviously non-free, but it seemed like a better way to get the geographic output from OpenGuides out there than anything else at the time.

Since I did that, I’ve learned a lot. Remember that 18 months ago, I’d never installed MapServer, had no idea what PostGIS was, and didn’t realize that there were free alternatives to some of the things that Google had done. Also, 9 months ago, there was no OpenLayers, or any decent open alternative to the Google Maps API.

In the past 18 months, that’s all changed. I’ve done map cartography, I’ve done setting up of map servers, and I worked full time for several months on the OpenLayers project. Although my direction has changed slightly, I still work heavily with maps on a daily basis, and spend more of my time on things like TileCache, which lets you serve map tiles at hundreds of requests/second.

So, about a month ago, I went back to the Open Guide to Boston, and converted all the Google Maps API calls to OpenLayers API calls. The conversion took about an hour, as I replaced all the templates with the different code. (If I was writing it again, it would have taken less time, but this was my first large scale open source Javascript undertaking, long before I gained the knowledge I now have from working with OpenLayers.) In that hour, I was able to convert all the existing maps to use free data from MassGIS, rather than the copyrighted data from Google, and to have Google as a backup: a Map of Furniture Stores can show you the different. You’ll see that there are several layers — one of which is a roadmap provided by me, one from Google — and one from the USGS, topographic quad charts.

It’s possible that some of this could have been done using Google as the tool. There’s nothing really magical here. But now, the data in the guide is no longer displayed by default on top of closed source data that no one can have access to. Instead, it’s displayed on top of an open dataset provided by my state government.

This is how the world should work. The data that the government collects should be made available to the people for things exactly like this. It shouldn’t require a ‘grassroots remapping’: There are examples out there of how to do it right. I find it so depressing to talk to friends in the UK, who not only don’t have the 1:5000 scale quality road data that Massachusetts provides, but doesn’t even provide TIGER-level data that the geocoder on the Open Guide to Boston uses.

Free Guides, with Free Maps. That’s the way it should be. The fact that it isn’t everywhere is sad, but at least it’s good to know that the technology is there. Switching from Google to OpenLayers is an easy task — it’s what happens next that is a problem. You need the data from somewhere, and it’s unfortunate that that ‘somewhere’ needs to be Google for so many people. I’m thankful to MassGIS and to the US Government for providing the data I can use, and to all the people who helped me learn enough to realize that using Google for everything is heading the wrong way when you want to not be beholden to a specific set of restrictions placed on a corporate entity.

In the vein of my previous post, I bring you another nifty trick: This time integrating Google Maps and Mapserver (kind of).

Visit static renderer. Browse Google Map. Then, when you’re looking where you want to be, hit the link up top, and off you go — transported to a world where data is public domain or licensed for re-use 🙂

I’m still trying to find a happy medium level of size of markers — when you’re zoomed way out, they’re too big, but when you zoom way in, they’re too small, so I don’t know what to do, but it *works* and that’s the important part.

Interface suggestions welcome — perhaps a side by side view is better? (I think I may be having a bit too much fun…)

After hours of fighting with Postgis, Schuyler finally got bia into a state where she would do the right thing when told to install it, so I was able to load the data from the Open Guide to Boston into postgis, and from there, to talk to it with mapserver. The result is a couple of pretty cool looking maps: Boston Metro, and Boston Metro Big, 1000×800 and 2000×1600 respectively.

The maps underneath are provided by Public Domain datasets, wrapped up in a tidy little easy to use package by the folks at OpenPlans through their “Sigma” project, and I’m extremely grateful to them for their efforts! They’ve saved me a ton of work, and allowed me to produce something that looks pretty damn cool.

If that’s not an advertisement for Open Geodata, I don’t know what is.

I’ve set up a mailing list for the Open Guide to Boston: sign up if you’re interested.

Likely topics on the list:
* Local get togethers to discuss the guide — technical, social, intro to beginners, etc.
* Requests for help in some aspect of the guide — looking for images, or help maintaining some specific feature.
* Discussion of changes made to the guide technically
* Discussion of changes made to the guide socially: policies, etc.

Thus far, there has been no traffic, but I’m about to change that, because I’m working on redesigning the site, and I’m a very bad designer. (I have a few tricks and tips I learned from working as one for far too long, but nothing that makes me qualified.)

So, if you’re interested, please feel free to join the list.

I’ve just implemented a fair amount of memcached support into CGI::Wiki, the Wiki software that powers OpenGuides. Now, any time a node is fetched, it will come from memcached if possible, or be loaded and stored into memcached, and when nodes are edited, the cached version is deleted.

This is in addition to adding caching to the main map pages, which were previously taking up to 20 seconds to load, and are now in the ~1 sec/range. (There is much more slowness in the browser side of this page, due to the javascript proccessing time required to load hundreds of location markers).

Additionally, the Open Guide to Boston is now running under mod_perl after my Message to the OG-Dev list with a patch that fixed the major stumbling block. This has increased speed somewhere between 5 and 10 fold over using CGI.

About 15 minutes ago, I restarted memcached to clear out old stats. I then visited the main index pages for both the Milton Keynes and Boston guides, to prefill the caches. There were 3535 items which were loaded (GET\_MISSes). In that time, there have been 4 new objects added — and 946 GET\_HITs. With the stability of memcached, I expect that this cache will be able to run for a long time with no maintenance, and I’ll be able to come back in a few weeks to see that memcached has saved me thousands of hits to the mysql database, therefore increasing the speed with which the Open Guide to Boston loads, something that I’m sure everyone can appreciate.

Most of the slowness that memcached is fixing in this case is not caused by MySQL being slow, but rather by the processing done by OpenGuides after the fact, due to the way that it is designed. The software was never really built to be the most efficient Wiki software out there, but to scale to the extent that it needed to, a task which it has completed admirably. However, this has left some corner cases where the performance was less than perfect — typically cases which were seldom used. The example of the “All nodes” map that the guide has is a good one: I built that page out of a corner case (the ?action=index view) which is not linked from the guide itself. As a result, I discovered a case where the software did not perform as expected. There was a roundtrip to the database to load the data about each node. This round trip time combined with a complex map { } function brought performance way below acceptable levels. So, I sought to improve that. First I simply cached all the data after it was processed, and today implemented caching of the individual nodes. Although the second change is minor and may not offer significant benefits on the small scale, it does help to build up the larger cache that it’s supporting: Roundtrips to memcached are quicker than roundtrips to the database.

This was originally posted to a local geo-hackers list, but I haven’t
received any response yet. It was long enough that I wanted to publish
it in a Google-able form as well.

I’m Christohper Schmidt, local mapping, geo, mobile application, web
development, RDF, RSS, and otherwise technically oriented hacker. Jo
Walsh mentioned my name on this list a few months back, as the organizer
of the Open Guide to Boston.

I’ve been working for the past couple months on boostrapping the content
in the Open Guide via the zami.com sites for Boston, Cambridge, and
Boston-Metro. Since there is no database-protection on copyright in the
US, pulling the data from these sites for repurposing it is a (as far as
I know) valid legal use of the data, and the Boston OpenGuide has
prospered significantly from these additions in terms of Google hits due
to the sheer size of the dataset.

However, it’s now coming to a point where adding more random content
isn’t helping to expand the guide in a way that’s entirely beneficial.
What is needed to expand the Open Guide to Boston is, quite simply, more
content about the places that are described.

A relative few of the nodes have full content — reviews, pictures, and
so on. Judy Jetson ,
Full Moon ,
Mojo Music . However, the large majority
are relatively devoid of life. I want to bring life to them, because I
want this to be even better than citysearch, even better than anything
else there is out there as a guide to what is around Boston, and I want
to encourage people to use it everywhere.

I have a couple reasons why. First, I hate closed systems with
unneccesary licensing and all the crap that goes along with things like
CitySearch and the other similar city directories. More importantly,
however, I feel that an open source of awesome information on what is in
andd around Boston lets people build more interesting things that pull
that data in. I have several personal goals that would use this data,
enabling users to get more information about what’s around them at
wireless hotspots, or when using a mobile device.

I would love to see more interest from the geo community in and around
Boston and especially my hometown of Cambridge. I’ve pushed for quite a
while to get the content to the level it’s at, but I simply can’ty visit
every restaurant in town. I would love to see more interest from
external parties because it might encourage me to keep on working at
what seems to be a very gargantuan task of documenting every thing I can
in town. An annotated Yellow Pages is a great goal, but requires far
more resources than I can bring to bear.

I’d be willing to demonstrate or pontificate on any of my desires or
goals, if persons are interested in meeting up to do this sometime in
the next couple weeks (or anytime, really). I’d also love to hear what
the Open Guides software isn’t providing that you would like to see: the
Google Maps component is something that I already put together and
committed back to the main tree, and I’ve got a couple other hacks in
the guides hosted on my server that aren’t in the main branch.

Looking forward to hearing from anyone who’s interested in helping,
evangalizing, or suggesting things that might be done better.

So, tonight I took it upon myself to redo the javascript behind the map for the Open Guide to Boston. The reason for this is relatively simple: when I first wrote the Google Maps interface, the guide had about 50 nodes. With that in mind, drawing them all was acceptable. As the guide got bigger, it got a bit more time consuming, but was still much easier than paging. However, the guide has recently doubled in size as I’ve increased the rate at which I pull data from Zami.com (specifically for the purpose of trying to build a free database of all the churches in the area with user interaction) — up to 2400 nodes (making it the single largest Open Guide to date by pure node size, as far as I can tell).

With this change, the map was no longer just difficult to use: it was impossible. Attempting to open the page crashed my web browser.

So, I took it upon myself to learn enough JavaScript to do paging… but then decided that rather than paging, I should attempt a bit more friendly solution to start. So I started hacking, and got into the Google Maps getBoundsLatLng() function, which allows me to determine the area of the map.

There are two basic options to go from here, depending on performance you expect in various situations, and the scalability you want to have.

* Load all data. Create Javascript variables to store it all. When the map moves, iterate over the data, adding markers for only points which are not already visible, and are in the new span.
* Load only the original data from the database. All additional data to be loaded via XMLHttpRequest.

The first is obviously something that can be done entirely client side, and in fact turned out to be the (much easier than I expected) path I chose. The largest reason for this is simply ease of integration. OpenGuides code is rather convoluted (to me), and modifying it is not something that I enjoy spending a lot of time doing. I’ll do it when I need to, but I prefer to avoid it. The other option would be to simply query the database directly via Perl or some other language, without using the OG framework. This would probably be slightly faster, but with the size of data I’m using, iterating over it is a minimal time compared to the time drawing the GMaps Markers. As a result, I chose to go with the slower, less scalable, but quicker-to-implement and merge to other guides way of doing things. The biggest benefit here is that I can merge it back into the other quickly growing guides — Saint Paul is now growing leaps and bounds alongside Boston, due to help from pulling Zami.com’s database, and I plan to do similar things with other guides. Also, London probably will not be able to use the mapping stuff in any useful way without this kind of code being added, so I went for the quickest thing that could possibly work.

There were a few gotchas that I had to end up avoiding:

* Removing Google Maps Markers takes *much* longer than adding them. Like, 3-4 times longer in extremely informal testing. With one or two, or even 20-30, this doesn’t matter, but with 100, this starts to be a significant barrier to user interaction.
* There are a number of words which are reserved in Javascript, but are not treated that way in Firefox. This leads to confusing error messages in Safari (although they are slightly improved in the latest webkit): If you see a ParseError, you should check the Firefox Reserved Words list — this is in one of their bugs. For example, the variable “long” is reserved, and can not be used to pass something like, say, longitude.

Discovering the ParseError, working out why it was there, and how to fix it, was greatly helped by the #webkit folks: I have never seen a more dedicated, hardworking, friendly, open and inviting group of Open Source Software developers in my history as a programmer, either in closed or open source products. Many thanks to them for helping me work through why the error was happening, and how to avoid it in the future.

Lastly (although this is not quite chronological, as I did this first), I had to modify the code to the guide to zoom in father to start, so it wouldn’t try to load so many nodes. I think a next step is to allow users to set a default lat/long for themselves, a la Google Local, from which they can start browsing, rather than always dropping them into the main map. But that’s a problem for a nother late night hack session.

Over the past couple months, I’ve been importing a large chunk of data, largely from Zami, into the Open Guide to Boston. Tonight, the guide crossed the 1000 place mark, and I decided that showing everything on one map was no longer making much sense.

The result? Boston Mashup – pick the categories you want, and they’ll be dropped onto a map, with a different color for each type of marker (up to 7 – I don’t have that many colored markers yet).

I’ve still got more I want to do with it, but it works, and works pretty well: much better than I expected it to. Thanks to perigrin for helping me work around my Perl ineptitude and solve the problems I needed to.

Not really much to say about it other than that: try it out, let me know what you think. I’m happy with it, at least as a starting point, so I think that’s probably a good sign.

Since I’m now in a bigger, better, brighter city, I restarted some of the work I’d done on creating an OpenGuide for where I lived, previously Manchester, now Boston.

There was some discussion over simply using MediaWiki in a local LiveJournal community for the storage of the content, but I made an argument for structured metadata. After creating a Google Maps View of Things in Boston, I think that I convinced the detractors that the structured metadata technique evangelised by OpenGuides can be a much more useful for expansion.

As usual, however, the wiki nature of the OpenGuide has led to it lapsing as a data source. I can only spend so many hours a day adding and editing nodes, and with limited help, the resource is unfortunately limited in content.

This is a cry for help. If you are in Boston, or know of things in Boston, or know people who know of things in Boston, have them hop on the bandwagon. I’d love to see the Boston Guide take over from places like CitySearch for top Google hits. A service where people can contribute directly to the content, good or bad, is much more worthwhile to me as a user than one where all I get is the phone book description from an agency which probably charges to list there.

Help show the power of wiki! Help make the Open Guide to Boston great!

Geolocation is the technique of determining a user’s geographic latitude, longitude and, by inference, city, region and nation. There are a number of ways to do this: one of the common ones discussed on the internet (according to a Google search for “geolocation”, as we all know Google is the Answer) is geolocation via IP address. The kind I’m interested in is much more accurate: geolocation via GPS device.

I want to be able to know where I am. I want this for a lot of reasons, most of them geeky rather than actually reasonable. However, it would be nice to offer more specific statistics on where my pictures are actually taken with fine grain granularity that a GPS can offer. Additionally, some of my alternative projects – cell based geolocation and the like – could benefit from actual coordinates on which to base everything from restaurant locations to searches. Openguides is, in particular, one area that could benefit from this.

I want something that works over bluetooth. My laptop and phone both speak bluetooth, and something with an actual display is out of my price range, for the most part, so I want something I can use my phone to get data out of. (USB / serial obviously doesn’t work for that.) From what I understand, most GPS devices which support NMEA are going to work okay for communication, as there are tools out there which support them. (Whether I can get the thing to talk over bluetooth is a different concern, but one I’m becoming more proficient at every day.)

For a long time, all I could find for Bluetooth GPS devices were 200-250 USD and up. However, while discussing it with someone in #mobitopia on Freenode, I found the Delorme Bluelogger, a Bluetooth GPS device for $150. Matt already posted about our discussion, but I hadn’t yet.

I have some cash left over from Christmas, and I know that I almost never actually buy anything for myself. So, I’m going to splurge, and I’m going to get it. I’m going to learn to use it, and I’m going to do all kinds of neat things with it. Plans include:

• GPS Annotation of Photos – This rolls into my photo annotation project, and is part of the reason I was keen to get it done: I want to actual have some fun queries for normal people (rather than just RDQL-aware people) over my photos.
• Location Based Description of things for Openguides – Describing where things are with GPS coordinates allows searches by distance. Once I have that, the guide allows more niftyness.
• Association of Cell IDs to Geo locations – Tied to the previous, this allows me to know where I am based on a Cell ID: Useful for “what’s nearby”, as well as useful for the general “where are you” that I like to be able to do – with just my cell phone.

All in all, some of the apps I have in mind seem nifty, some geeky, some just demonstrative of something bigger. Some are RDF related, some are just fun. The Bluelogger seems like a decent tool to achieve everything I need to.