Technical Ramblings

I’ve set up a mailing list for the Open Guide to Boston: sign up if you’re interested.

Likely topics on the list:
* Local get togethers to discuss the guide — technical, social, intro to beginners, etc.
* Requests for help in some aspect of the guide — looking for images, or help maintaining some specific feature.
* Discussion of changes made to the guide technically
* Discussion of changes made to the guide socially: policies, etc.

Thus far, there has been no traffic, but I’m about to change that, because I’m working on redesigning the site, and I’m a very bad designer. (I have a few tricks and tips I learned from working as one for far too long, but nothing that makes me qualified.)

So, if you’re interested, please feel free to join the list.

I just closed comments on all posts older than this one. In the past 3 days, I have received 892 spam attempts-at-comments. Everything goes through moderation, so this doesn’t end up being an issue for anyone except for me, who has to sort through all the notifications and spam to try and find the wheat in the chaff.

I hate sites that do this, but it was getting *really* annoying, and nobody comments anyway.

So, this weekend I was showed Google Earth-on-Windows has a “Browser Pane”, where you can do searches and so on from within Google Earth, and the application will send back extra requests when the Google Earth client moves and so on.

I’m looking into doing some stuff with this, but when I looked, the mac client has no such browser pane. I then looked up WINE’s records for Google Earth and found it doesn’t run under Ubuntu Breezy, which is what I’m running.

So, does anyone have any suggestions as to how I could either 1. Get a copy of Windows Google Earth running on OS X or Ubuntu 5.10, for the Browser Pane feature, or 2. Get an account on a windows box I can RDC to? 🙂

Starting around April 1st, I’m going to be looking for a new fulltime position somewhere. If you have someplace you think might be good for me, and you know they’re looking for people, feel free to let me know.

Some of my previous work is pretty much the best resume I’ve got — the simple format I’ve got a text/pdf resume isn’t cleaned up yet.

I’m most interested in projects related to mapping, and take a secondary interest in anything open source. I primarily work in PHP and Python.

Contact can be made via email.

Working with Schuyler on hacking my cell/gps data — got it displayed on top of tiger/line data (At which point I said to schuyler, “Okay, so now we’ve got an extremely crappy version of what I already have with Google Maps. What’s next?”) Next step is to work on a voronoi translation of some kind to make centroids out of the data and see what that does.

In the process, have discovered that v.in.ascii appears to be completely broken in openosx grass.

We have also discovered that v.surf.idw can produce really pretty pictures… but on OS X, it gives something more like:

GRASS 6.0.0 (bostonlatlong):~/cell > v.surf.idw in=cell out=cell\_idw col=cid npoints=1
v.surf.idw:
8468 records selected from table
3538 points loaded
Interpolating raster map <cell_idw> … 247 rows … Segmentation fault

Hooray for segfaults, eh?

However, Grass is still really fucking cool.

Per a request on the Milton Keynes guide, I took up the matter of looking at making searches include more than just the text of the entry, but instead also listing the metadata in the nodes. I successfully completed this in the following manner:

* Modify the rename.pl in the dist to be more along the lines of: http://crschmidt.net/openguides/reindex.pl
You can see here that this indexes phone address city postcode website fax summary, as well as categories and locales.
* Tested this to make sure the results worked. Number searching seems broken, but other than that it’s in good shape.
* Head in to try and make this change to the underlying guide code.
* Realize that searching is basically hardcoded into write\_node, with no easy way around it. Resign self to this fact and decide that double-indexing is the way to proceed.
* Write a CGI::Wiki Plugin. Bang head against wall until realizing you need to set @ISA yourself.
* Write reindexer similar to reindex.pl above.
* Find “write_node” in Openguides.pm. Place Reindexer plugin above write\_node (see code snippet at end of email).
* Test it out. Seems to work okay.
* Upload Indexer.pm code to http://crschmidt.net/openguides/Indexer.pm
* Post to mailing list.

I’m not sure if anyone else will find this interesting, but I certainly did, so I figured I’d share the knowledge in case anyone else out there wants it.

my $indexer = CGI::Wiki::Plugin::Indexer->new;
$wiki->register_plugin( plugin => $indexer );
$indexer->reindex($node, $content, \%metadata);
my $written = $wiki->write\_node($node, $content, $checksum, \%metadata );

I’ve just implemented a fair amount of memcached support into CGI::Wiki, the Wiki software that powers OpenGuides. Now, any time a node is fetched, it will come from memcached if possible, or be loaded and stored into memcached, and when nodes are edited, the cached version is deleted.

This is in addition to adding caching to the main map pages, which were previously taking up to 20 seconds to load, and are now in the ~1 sec/range. (There is much more slowness in the browser side of this page, due to the javascript proccessing time required to load hundreds of location markers).

Additionally, the Open Guide to Boston is now running under mod_perl after my Message to the OG-Dev list with a patch that fixed the major stumbling block. This has increased speed somewhere between 5 and 10 fold over using CGI.

About 15 minutes ago, I restarted memcached to clear out old stats. I then visited the main index pages for both the Milton Keynes and Boston guides, to prefill the caches. There were 3535 items which were loaded (GET\_MISSes). In that time, there have been 4 new objects added — and 946 GET\_HITs. With the stability of memcached, I expect that this cache will be able to run for a long time with no maintenance, and I’ll be able to come back in a few weeks to see that memcached has saved me thousands of hits to the mysql database, therefore increasing the speed with which the Open Guide to Boston loads, something that I’m sure everyone can appreciate.

Most of the slowness that memcached is fixing in this case is not caused by MySQL being slow, but rather by the processing done by OpenGuides after the fact, due to the way that it is designed. The software was never really built to be the most efficient Wiki software out there, but to scale to the extent that it needed to, a task which it has completed admirably. However, this has left some corner cases where the performance was less than perfect — typically cases which were seldom used. The example of the “All nodes” map that the guide has is a good one: I built that page out of a corner case (the ?action=index view) which is not linked from the guide itself. As a result, I discovered a case where the software did not perform as expected. There was a roundtrip to the database to load the data about each node. This round trip time combined with a complex map { } function brought performance way below acceptable levels. So, I sought to improve that. First I simply cached all the data after it was processed, and today implemented caching of the individual nodes. Although the second change is minor and may not offer significant benefits on the small scale, it does help to build up the larger cache that it’s supporting: Roundtrips to memcached are quicker than roundtrips to the database.

This was originally posted to a local geo-hackers list, but I haven’t
received any response yet. It was long enough that I wanted to publish
it in a Google-able form as well.

I’m Christohper Schmidt, local mapping, geo, mobile application, web
development, RDF, RSS, and otherwise technically oriented hacker. Jo
Walsh mentioned my name on this list a few months back, as the organizer
of the Open Guide to Boston.

I’ve been working for the past couple months on boostrapping the content
in the Open Guide via the zami.com sites for Boston, Cambridge, and
Boston-Metro. Since there is no database-protection on copyright in the
US, pulling the data from these sites for repurposing it is a (as far as
I know) valid legal use of the data, and the Boston OpenGuide has
prospered significantly from these additions in terms of Google hits due
to the sheer size of the dataset.

However, it’s now coming to a point where adding more random content
isn’t helping to expand the guide in a way that’s entirely beneficial.
What is needed to expand the Open Guide to Boston is, quite simply, more
content about the places that are described.

A relative few of the nodes have full content — reviews, pictures, and
so on. Judy Jetson ,
Full Moon ,
Mojo Music . However, the large majority
are relatively devoid of life. I want to bring life to them, because I
want this to be even better than citysearch, even better than anything
else there is out there as a guide to what is around Boston, and I want
to encourage people to use it everywhere.

I have a couple reasons why. First, I hate closed systems with
unneccesary licensing and all the crap that goes along with things like
CitySearch and the other similar city directories. More importantly,
however, I feel that an open source of awesome information on what is in
andd around Boston lets people build more interesting things that pull
that data in. I have several personal goals that would use this data,
enabling users to get more information about what’s around them at
wireless hotspots, or when using a mobile device.

I would love to see more interest from the geo community in and around
Boston and especially my hometown of Cambridge. I’ve pushed for quite a
while to get the content to the level it’s at, but I simply can’ty visit
every restaurant in town. I would love to see more interest from
external parties because it might encourage me to keep on working at
what seems to be a very gargantuan task of documenting every thing I can
in town. An annotated Yellow Pages is a great goal, but requires far
more resources than I can bring to bear.

I’d be willing to demonstrate or pontificate on any of my desires or
goals, if persons are interested in meeting up to do this sometime in
the next couple weeks (or anytime, really). I’d also love to hear what
the Open Guides software isn’t providing that you would like to see: the
Google Maps component is something that I already put together and
committed back to the main tree, and I’ve got a couple other hacks in
the guides hosted on my server that aren’t in the main branch.

Looking forward to hearing from anyone who’s interested in helping,
evangalizing, or suggesting things that might be done better.

Looks like the hosted WordPress service, WordPress.com, is broken. All the blogs hosted there are not showing any posts: Scobleizer just tells me “Sorry, no posts matched your criteria.”

Wonder what happened…

Edit: As of 11AM EST, it’s back.

Julie’s at home these days – since we found out she’s deaf in one ear, we pulled her out of school to do more one-on-one learning at home.

She’s good at associating words with the visual representation of the letter – she’ll see “S”, and say “That’s what Sienna’s name starts with!” – but bad at names of letters. So, I hacked up a quick wxPython script from the wxPython samples that will just show an uppercase and lowercase letter, which she can click through. It’s basically a quick-and-dirty flashcard program: source available, of course.

Because I wanted to track her progress, I hacked up a little Ning App to track her learning. As of today, she only recognizes J (for Julie) and X on the first try. So, we’ll track it and see how she does. LearningLetters. 20 minute app, probably less. It’s crude, but it works.

It’s easy to make this work for just about anything you want to track. Just modify the ‘choices’ array.