Technical Ramblings

Working with Schuyler on hacking my cell/gps data — got it displayed on top of tiger/line data (At which point I said to schuyler, “Okay, so now we’ve got an extremely crappy version of what I already have with Google Maps. What’s next?”) Next step is to work on a voronoi translation of some kind to make centroids out of the data and see what that does.

In the process, have discovered that v.in.ascii appears to be completely broken in openosx grass.

We have also discovered that v.surf.idw can produce really pretty pictures… but on OS X, it gives something more like:

GRASS 6.0.0 (bostonlatlong):~/cell > v.surf.idw in=cell out=cell\_idw col=cid npoints=1
v.surf.idw:
8468 records selected from table
3538 points loaded
Interpolating raster map <cell_idw> … 247 rows … Segmentation fault

Hooray for segfaults, eh?

However, Grass is still really fucking cool.

Per a request on the Milton Keynes guide, I took up the matter of looking at making searches include more than just the text of the entry, but instead also listing the metadata in the nodes. I successfully completed this in the following manner:

* Modify the rename.pl in the dist to be more along the lines of: http://crschmidt.net/openguides/reindex.pl
You can see here that this indexes phone address city postcode website fax summary, as well as categories and locales.
* Tested this to make sure the results worked. Number searching seems broken, but other than that it’s in good shape.
* Head in to try and make this change to the underlying guide code.
* Realize that searching is basically hardcoded into write\_node, with no easy way around it. Resign self to this fact and decide that double-indexing is the way to proceed.
* Write a CGI::Wiki Plugin. Bang head against wall until realizing you need to set @ISA yourself.
* Write reindexer similar to reindex.pl above.
* Find “write_node” in Openguides.pm. Place Reindexer plugin above write\_node (see code snippet at end of email).
* Test it out. Seems to work okay.
* Upload Indexer.pm code to http://crschmidt.net/openguides/Indexer.pm
* Post to mailing list.

I’m not sure if anyone else will find this interesting, but I certainly did, so I figured I’d share the knowledge in case anyone else out there wants it.

my $indexer = CGI::Wiki::Plugin::Indexer->new;
$wiki->register_plugin( plugin => $indexer );
$indexer->reindex($node, $content, \%metadata);
my $written = $wiki->write\_node($node, $content, $checksum, \%metadata );

I’ve just implemented a fair amount of memcached support into CGI::Wiki, the Wiki software that powers OpenGuides. Now, any time a node is fetched, it will come from memcached if possible, or be loaded and stored into memcached, and when nodes are edited, the cached version is deleted.

This is in addition to adding caching to the main map pages, which were previously taking up to 20 seconds to load, and are now in the ~1 sec/range. (There is much more slowness in the browser side of this page, due to the javascript proccessing time required to load hundreds of location markers).

Additionally, the Open Guide to Boston is now running under mod_perl after my Message to the OG-Dev list with a patch that fixed the major stumbling block. This has increased speed somewhere between 5 and 10 fold over using CGI.

About 15 minutes ago, I restarted memcached to clear out old stats. I then visited the main index pages for both the Milton Keynes and Boston guides, to prefill the caches. There were 3535 items which were loaded (GET\_MISSes). In that time, there have been 4 new objects added — and 946 GET\_HITs. With the stability of memcached, I expect that this cache will be able to run for a long time with no maintenance, and I’ll be able to come back in a few weeks to see that memcached has saved me thousands of hits to the mysql database, therefore increasing the speed with which the Open Guide to Boston loads, something that I’m sure everyone can appreciate.

Most of the slowness that memcached is fixing in this case is not caused by MySQL being slow, but rather by the processing done by OpenGuides after the fact, due to the way that it is designed. The software was never really built to be the most efficient Wiki software out there, but to scale to the extent that it needed to, a task which it has completed admirably. However, this has left some corner cases where the performance was less than perfect — typically cases which were seldom used. The example of the “All nodes” map that the guide has is a good one: I built that page out of a corner case (the ?action=index view) which is not linked from the guide itself. As a result, I discovered a case where the software did not perform as expected. There was a roundtrip to the database to load the data about each node. This round trip time combined with a complex map { } function brought performance way below acceptable levels. So, I sought to improve that. First I simply cached all the data after it was processed, and today implemented caching of the individual nodes. Although the second change is minor and may not offer significant benefits on the small scale, it does help to build up the larger cache that it’s supporting: Roundtrips to memcached are quicker than roundtrips to the database.

This was originally posted to a local geo-hackers list, but I haven’t
received any response yet. It was long enough that I wanted to publish
it in a Google-able form as well.

I’m Christohper Schmidt, local mapping, geo, mobile application, web
development, RDF, RSS, and otherwise technically oriented hacker. Jo
Walsh mentioned my name on this list a few months back, as the organizer
of the Open Guide to Boston.

I’ve been working for the past couple months on boostrapping the content
in the Open Guide via the zami.com sites for Boston, Cambridge, and
Boston-Metro. Since there is no database-protection on copyright in the
US, pulling the data from these sites for repurposing it is a (as far as
I know) valid legal use of the data, and the Boston OpenGuide has
prospered significantly from these additions in terms of Google hits due
to the sheer size of the dataset.

However, it’s now coming to a point where adding more random content
isn’t helping to expand the guide in a way that’s entirely beneficial.
What is needed to expand the Open Guide to Boston is, quite simply, more
content about the places that are described.

A relative few of the nodes have full content — reviews, pictures, and
so on. Judy Jetson ,
Full Moon ,
Mojo Music . However, the large majority
are relatively devoid of life. I want to bring life to them, because I
want this to be even better than citysearch, even better than anything
else there is out there as a guide to what is around Boston, and I want
to encourage people to use it everywhere.

I have a couple reasons why. First, I hate closed systems with
unneccesary licensing and all the crap that goes along with things like
CitySearch and the other similar city directories. More importantly,
however, I feel that an open source of awesome information on what is in
andd around Boston lets people build more interesting things that pull
that data in. I have several personal goals that would use this data,
enabling users to get more information about what’s around them at
wireless hotspots, or when using a mobile device.

I would love to see more interest from the geo community in and around
Boston and especially my hometown of Cambridge. I’ve pushed for quite a
while to get the content to the level it’s at, but I simply can’ty visit
every restaurant in town. I would love to see more interest from
external parties because it might encourage me to keep on working at
what seems to be a very gargantuan task of documenting every thing I can
in town. An annotated Yellow Pages is a great goal, but requires far
more resources than I can bring to bear.

I’d be willing to demonstrate or pontificate on any of my desires or
goals, if persons are interested in meeting up to do this sometime in
the next couple weeks (or anytime, really). I’d also love to hear what
the Open Guides software isn’t providing that you would like to see: the
Google Maps component is something that I already put together and
committed back to the main tree, and I’ve got a couple other hacks in
the guides hosted on my server that aren’t in the main branch.

Looking forward to hearing from anyone who’s interested in helping,
evangalizing, or suggesting things that might be done better.

Looks like the hosted WordPress service, WordPress.com, is broken. All the blogs hosted there are not showing any posts: Scobleizer just tells me “Sorry, no posts matched your criteria.”

Wonder what happened…

Edit: As of 11AM EST, it’s back.

Julie’s at home these days – since we found out she’s deaf in one ear, we pulled her out of school to do more one-on-one learning at home.

She’s good at associating words with the visual representation of the letter – she’ll see “S”, and say “That’s what Sienna’s name starts with!” – but bad at names of letters. So, I hacked up a quick wxPython script from the wxPython samples that will just show an uppercase and lowercase letter, which she can click through. It’s basically a quick-and-dirty flashcard program: source available, of course.

Because I wanted to track her progress, I hacked up a little Ning App to track her learning. As of today, she only recognizes J (for Julie) and X on the first try. So, we’ll track it and see how she does. LearningLetters. 20 minute app, probably less. It’s crude, but it works.

It’s easy to make this work for just about anything you want to track. Just modify the ‘choices’ array.

A long time ago, when I first started working in the Ning playground, the first thing that impressed me about it was the fact that it had ready built components for so much *stuff*. Comments. Ratings. Flickr. Amazon. Calendars. Questionairres (Which is a very difficult word to spell right.)

Last night, after setting up X\_Query, I thought “hm, what if people want to comment?” And as I did, I remembered those components. I went into the developer documentation and looked up XNC\_Comment, which had a great, simple example of how to add comments to any content object.

But I didn’t have any content objects. The app is built as an API, so there was no need for Content Objects (at least not until I build up the documentation).

So I built a couple quick lines to add Content where I need it:

<?php
$d = XN\_Query::create(‘content’)->filter(“owner”)->filter(“type”, “=”, “Page”)->filter(‘title’, ‘=’, $\_SERVER[‘SCRIPT\_FILENAME’])->uniqueResult();
if (!$d) {
$d = XN\_Content::Create(“Page”, $\_SERVER[‘SCRIPT\_FILENAME’]);
$d->save();
}
?>

Once this is done, the $d is an object for the current page. Once you’ve done that, you can just slightly modify the example from XNC_Comment:

require\_once ‘XNC/Comment.php’;
$newComment = new XNC\_Comment($d);
// Handle any form submission of adding a new comment
if ($newComment->willProcessForm()) {
    $newComment->processForm();
} elseif ($newComment->lastError() != XNC\_Comment::ERROR\_FORM\_ABSENT) {
    print $newComment->lastError();
}
// Display the add a comment form (unless it was just submitted and saved)
if ($newComment->canProcessForm()) {
    echo $newComment->buildForm();
}
// Display a list of comments belonging to a parent object
if ($d->my->content($newComment->referenceAttribute,true)) {
    foreach ($d->my->content($newComment->referenceAttribute,true) as $comment) {
        echo new XNC\_Comment($comment);
    }
}

Now, you’ve got comments on any page you want! Just include these two chunks of code at the bottom of each page. This is one of the easiest things I’ve done. Now anyone can take these code snippets and use them in any Ning app. Hm… maybe the next step is to build a code snippets repository…

I said a while ago that the coolest part of Ning was the content store. I’m now working on a proof of concept to prove it.

So, there’s a bunch of Roshambo (Rock Scissors Paper) apps. Or at least a couple. Or at least two: Roshambo and Fifteen, my personal 15-option version.

So, I know that I’ve played a handful of games, and don’t want to scroll back through them. The app itself doesn’t offer an API itself, so there’s no easy way to get the data out, and I hate screenscraping.

So, I built a query app. XQuery. The homepage kind of documents what it can do so far, but it’s nowhere near done. It was just a one hour hack. Then I built the Roshambo Results Viewer. I also then extracted the logic into a simple Roshambo Python Script — usage ‘python roshambo.py crschmidt’.

Expect more to come of this.

This post is to test WordPress 1.5’s ability to check posts via email.
I oftentimes wish, when responding on a mailing list, to send my
thoughts or whatever directly to the weblog. Hopefully this will let me
do that.

jcowan wanted to do something like this about 2 years ago when I first
met him, but as far as I know, he never did. We’ll see how it works for
me.

In reality, this is actually my 5th post-via-email, but the first 4
didn’t work. I think this one will. (Who knew that WP1.5 shipped with
such a broken mail->post interface?)

A couple weeks ago, I was lent a rebranded RoyalTek BlueGPS by Jo Walsh. She and Schuyler had it for something for Wireless Hacks, and hadn’t used it since. I’ve been using it to do a number of fun things. If you still do not have a car o truck so you can run in to new adventures with your gps, a ford van or transit custom might be just for you according to the local road expert, find his explanation at the last link.

Getting it working was pretty easy. Just go through the “Setup a Bluetooth Device” process, enter ‘0000’ in the passcode box. Set up a Serial Port in the Bluetooth System Preferences Panel. Then I’ve got a ‘/dev/tty.BlueGPS27E8B5-SerialPort-1’ device that I can play with.

There’s a couple things I’ve played with or looked at:
* BlueGPS Log Downloader — although it’s only Bluez+Linux, and my last dongle broke recently, so I can’t use it.
* gpsd — turning any serial device into something accessible over TCP in a regularized format.
* gpsdrive — uses gpsd to display your position and track on a map. Biggest problem with this is that downloading maps seems to be difficult at best. I couldn’t get it to work how I thought it should.
* gpsconnect, a mac os x program that lets me initiate conversation with the GPS, after which I can use `cat` to spit the output to a file. Set the laptop in the back seat, the gps on the dashboard, and cat the NMEA to a file, then use GPS Visualizer to generate a pretty map.

I’m also doing some nifty celltracking stuff, but I’ll save that for another entry.

The BlueGPS is nifty. I recommend it, although I have no clue how much it costs 🙂