Technical Ramblings

I’ve just implemented a fair amount of memcached support into CGI::Wiki, the Wiki software that powers OpenGuides. Now, any time a node is fetched, it will come from memcached if possible, or be loaded and stored into memcached, and when nodes are edited, the cached version is deleted.

This is in addition to adding caching to the main map pages, which were previously taking up to 20 seconds to load, and are now in the ~1 sec/range. (There is much more slowness in the browser side of this page, due to the javascript proccessing time required to load hundreds of location markers).

Additionally, the Open Guide to Boston is now running under mod_perl after my Message to the OG-Dev list with a patch that fixed the major stumbling block. This has increased speed somewhere between 5 and 10 fold over using CGI.

About 15 minutes ago, I restarted memcached to clear out old stats. I then visited the main index pages for both the Milton Keynes and Boston guides, to prefill the caches. There were 3535 items which were loaded (GET\_MISSes). In that time, there have been 4 new objects added — and 946 GET\_HITs. With the stability of memcached, I expect that this cache will be able to run for a long time with no maintenance, and I’ll be able to come back in a few weeks to see that memcached has saved me thousands of hits to the mysql database, therefore increasing the speed with which the Open Guide to Boston loads, something that I’m sure everyone can appreciate.

Most of the slowness that memcached is fixing in this case is not caused by MySQL being slow, but rather by the processing done by OpenGuides after the fact, due to the way that it is designed. The software was never really built to be the most efficient Wiki software out there, but to scale to the extent that it needed to, a task which it has completed admirably. However, this has left some corner cases where the performance was less than perfect — typically cases which were seldom used. The example of the “All nodes” map that the guide has is a good one: I built that page out of a corner case (the ?action=index view) which is not linked from the guide itself. As a result, I discovered a case where the software did not perform as expected. There was a roundtrip to the database to load the data about each node. This round trip time combined with a complex map { } function brought performance way below acceptable levels. So, I sought to improve that. First I simply cached all the data after it was processed, and today implemented caching of the individual nodes. Although the second change is minor and may not offer significant benefits on the small scale, it does help to build up the larger cache that it’s supporting: Roundtrips to memcached are quicker than roundtrips to the database.

This was originally posted to a local geo-hackers list, but I haven’t
received any response yet. It was long enough that I wanted to publish
it in a Google-able form as well.

I’m Christohper Schmidt, local mapping, geo, mobile application, web
development, RDF, RSS, and otherwise technically oriented hacker. Jo
Walsh mentioned my name on this list a few months back, as the organizer
of the Open Guide to Boston.

I’ve been working for the past couple months on boostrapping the content
in the Open Guide via the zami.com sites for Boston, Cambridge, and
Boston-Metro. Since there is no database-protection on copyright in the
US, pulling the data from these sites for repurposing it is a (as far as
I know) valid legal use of the data, and the Boston OpenGuide has
prospered significantly from these additions in terms of Google hits due
to the sheer size of the dataset.

However, it’s now coming to a point where adding more random content
isn’t helping to expand the guide in a way that’s entirely beneficial.
What is needed to expand the Open Guide to Boston is, quite simply, more
content about the places that are described.

A relative few of the nodes have full content — reviews, pictures, and
so on. Judy Jetson ,
Full Moon ,
Mojo Music . However, the large majority
are relatively devoid of life. I want to bring life to them, because I
want this to be even better than citysearch, even better than anything
else there is out there as a guide to what is around Boston, and I want
to encourage people to use it everywhere.

I have a couple reasons why. First, I hate closed systems with
unneccesary licensing and all the crap that goes along with things like
CitySearch and the other similar city directories. More importantly,
however, I feel that an open source of awesome information on what is in
andd around Boston lets people build more interesting things that pull
that data in. I have several personal goals that would use this data,
enabling users to get more information about what’s around them at
wireless hotspots, or when using a mobile device.

I would love to see more interest from the geo community in and around
Boston and especially my hometown of Cambridge. I’ve pushed for quite a
while to get the content to the level it’s at, but I simply can’ty visit
every restaurant in town. I would love to see more interest from
external parties because it might encourage me to keep on working at
what seems to be a very gargantuan task of documenting every thing I can
in town. An annotated Yellow Pages is a great goal, but requires far
more resources than I can bring to bear.

I’d be willing to demonstrate or pontificate on any of my desires or
goals, if persons are interested in meeting up to do this sometime in
the next couple weeks (or anytime, really). I’d also love to hear what
the Open Guides software isn’t providing that you would like to see: the
Google Maps component is something that I already put together and
committed back to the main tree, and I’ve got a couple other hacks in
the guides hosted on my server that aren’t in the main branch.

Looking forward to hearing from anyone who’s interested in helping,
evangalizing, or suggesting things that might be done better.

Looks like the hosted Wordpress service, Wordpress.com, is broken. All the blogs hosted there are not showing any posts: Scobleizer just tells me “Sorry, no posts matched your criteria.”

Wonder what happened…

Edit: As of 11AM EST, it’s back.

Julie’s at home these days - since we found out she’s deaf in one ear, we pulled her out of school to do more one-on-one learning at home.

She’s good at associating words with the visual representation of the letter - she’ll see “S”, and say “That’s what Sienna’s name starts with!” - but bad at names of letters. So, I hacked up a quick wxPython script from the wxPython samples that will just show an uppercase and lowercase letter, which she can click through. It’s basically a quick-and-dirty flashcard program: source available, of course.

Because I wanted to track her progress, I hacked up a little Ning App to track her learning. As of today, she only recognizes J (for Julie) and X on the first try. So, we’ll track it and see how she does. LearningLetters. 20 minute app, probably less. It’s crude, but it works.

It’s easy to make this work for just about anything you want to track. Just modify the ‘choices’ array.

A long time ago, when I first started working in the Ning playground, the first thing that impressed me about it was the fact that it had ready built components for so much *stuff*. Comments. Ratings. Flickr. Amazon. Calendars. Questionairres (Which is a very difficult word to spell right.)

Last night, after setting up X\_Query, I thought “hm, what if people want to comment?” And as I did, I remembered those components. I went into the developer documentation and looked up XNC\_Comment, which had a great, simple example of how to add comments to any content object.

But I didn’t have any content objects. The app is built as an API, so there was no need for Content Objects (at least not until I build up the documentation).

So I built a couple quick lines to add Content where I need it:

<?php
$d = XN\_Query::create(’content’)->filter(”owner”)->filter(”type”, “=”, “Page”)->filter(’title’, ‘=’, $\_SERVER['SCRIPT\_FILENAME'])->uniqueResult();
if (!$d) {
$d = XN\_Content::Create(”Page”, $\_SERVER['SCRIPT\_FILENAME']);
$d->save();
}
?>

Once this is done, the $d is an object for the current page. Once you’ve done that, you can just slightly modify the example from XNC_Comment:

require\_once ‘XNC/Comment.php’;
$newComment = new XNC\_Comment($d);
// Handle any form submission of adding a new comment
if ($newComment->willProcessForm()) {
    $newComment->processForm();
} elseif ($newComment->lastError() != XNC\_Comment::ERROR\_FORM\_ABSENT) {
    print $newComment->lastError();
}
// Display the add a comment form (unless it was just submitted and saved)
if ($newComment->canProcessForm()) {
    echo $newComment->buildForm();
}
// Display a list of comments belonging to a parent object
if ($d->my->content($newComment->referenceAttribute,true)) {
    foreach ($d->my->content($newComment->referenceAttribute,true) as $comment) {
        echo new XNC\_Comment($comment);
    }
}

Now, you’ve got comments on any page you want! Just include these two chunks of code at the bottom of each page. This is one of the easiest things I’ve done. Now anyone can take these code snippets and use them in any Ning app. Hm… maybe the next step is to build a code snippets repository…

I said a while ago that the coolest part of Ning was the content store. I’m now working on a proof of concept to prove it.

So, there’s a bunch of Roshambo (Rock Scissors Paper) apps. Or at least a couple. Or at least two: Roshambo and Fifteen, my personal 15-option version.

So, I know that I’ve played a handful of games, and don’t want to scroll back through them. The app itself doesn’t offer an API itself, so there’s no easy way to get the data out, and I hate screenscraping.

So, I built a query app. XQuery. The homepage kind of documents what it can do so far, but it’s nowhere near done. It was just a one hour hack. Then I built the Roshambo Results Viewer. I also then extracted the logic into a simple Roshambo Python Script — usage ‘python roshambo.py crschmidt’.

Expect more to come of this.

The crschmidt.net site now uses in-page validation to provide a visual indication of whether the source is valid. See the smiley face in the upper right corner, which is an indicator of validation or not. It’s linked to the validator, so you can see information about it by clicking the link.

Code stolen from sbp’s ‘valid’ project.

So, tonight I took it upon myself to redo the javascript behind the map for the Open Guide to Boston. The reason for this is relatively simple: when I first wrote the Google Maps interface, the guide had about 50 nodes. With that in mind, drawing them all was acceptable. As the guide got bigger, it got a bit more time consuming, but was still much easier than paging. However, the guide has recently doubled in size as I’ve increased the rate at which I pull data from Zami.com (specifically for the purpose of trying to build a free database of all the churches in the area with user interaction) — up to 2400 nodes (making it the single largest Open Guide to date by pure node size, as far as I can tell).

With this change, the map was no longer just difficult to use: it was impossible. Attempting to open the page crashed my web browser.

So, I took it upon myself to learn enough JavaScript to do paging… but then decided that rather than paging, I should attempt a bit more friendly solution to start. So I started hacking, and got into the Google Maps getBoundsLatLng() function, which allows me to determine the area of the map.

There are two basic options to go from here, depending on performance you expect in various situations, and the scalability you want to have.

* Load all data. Create Javascript variables to store it all. When the map moves, iterate over the data, adding markers for only points which are not already visible, and are in the new span.
* Load only the original data from the database. All additional data to be loaded via XMLHttpRequest.

The first is obviously something that can be done entirely client side, and in fact turned out to be the (much easier than I expected) path I chose. The largest reason for this is simply ease of integration. OpenGuides code is rather convoluted (to me), and modifying it is not something that I enjoy spending a lot of time doing. I’ll do it when I need to, but I prefer to avoid it. The other option would be to simply query the database directly via Perl or some other language, without using the OG framework. This would probably be slightly faster, but with the size of data I’m using, iterating over it is a minimal time compared to the time drawing the GMaps Markers. As a result, I chose to go with the slower, less scalable, but quicker-to-implement and merge to other guides way of doing things. The biggest benefit here is that I can merge it back into the other quickly growing guides — Saint Paul is now growing leaps and bounds alongside Boston, due to help from pulling Zami.com’s database, and I plan to do similar things with other guides. Also, London probably will not be able to use the mapping stuff in any useful way without this kind of code being added, so I went for the quickest thing that could possibly work.

There were a few gotchas that I had to end up avoiding:

* Removing Google Maps Markers takes *much* longer than adding them. Like, 3-4 times longer in extremely informal testing. With one or two, or even 20-30, this doesn’t matter, but with 100, this starts to be a significant barrier to user interaction.
* There are a number of words which are reserved in Javascript, but are not treated that way in Firefox. This leads to confusing error messages in Safari (although they are slightly improved in the latest webkit): If you see a ParseError, you should check the Firefox Reserved Words list — this is in one of their bugs. For example, the variable “long” is reserved, and can not be used to pass something like, say, longitude.

Discovering the ParseError, working out why it was there, and how to fix it, was greatly helped by the #webkit folks: I have never seen a more dedicated, hardworking, friendly, open and inviting group of Open Source Software developers in my history as a programmer, either in closed or open source products. Many thanks to them for helping me work through why the error was happening, and how to avoid it in the future.

Lastly (although this is not quite chronological, as I did this first), I had to modify the code to the guide to zoom in father to start, so it wouldn’t try to load so many nodes. I think a next step is to allow users to set a default lat/long for themselves, a la Google Local, from which they can start browsing, rather than always dropping them into the main map. But that’s a problem for a nother late night hack session.

Over the past couple months, I’ve been importing a large chunk of data, largely from Zami, into the Open Guide to Boston. Tonight, the guide crossed the 1000 place mark, and I decided that showing everything on one map was no longer making much sense.

The result? Boston Mashup - pick the categories you want, and they’ll be dropped onto a map, with a different color for each type of marker (up to 7 - I don’t have that many colored markers yet).

I’ve still got more I want to do with it, but it works, and works pretty well: much better than I expected it to. Thanks to perigrin for helping me work around my Perl ineptitude and solve the problems I needed to.

Not really much to say about it other than that: try it out, let me know what you think. I’m happy with it, at least as a starting point, so I think that’s probably a good sign.

Internet Explorer is kind of like a spooked horse. You have to talk to it *very* slowly… And if you move too fast, it’ll run.

I can just imagine someone tossing some CSS3 selectors at IE and having it go run and cower in a corner…