Technical Ramblings

At the Google Developer Day. Sitting in the Google Gears session: it’s pretty frickin cool. I just created an OpenLayers Map, wrote 15 lines of code, and that page will now load, even when I’m online.

Other things it does:

• WorkerPool — run code in the background in your browser. Demoing finding prime quadruplets: user interface continues to be responsive — can run multiples at the same time, and user experience is not bad. (Oh man, how I wish this were  built in by default.)
• Storage — local storage in a sqlite database, including full text search: millions of documents, working on fts3, which will be 10s of millions.
• Local server — caches data for offline use. 

There are tons of limitations — very early release — but it’s available for Firefox and IE, and can be built for Safari.

Made a ‘screencast’ (at least, I think that’s the right buzzword) of OpenLayers + QGIS today:

Mapping Your Data: Using QGIS, OpenLayers, and MapServer to serve open data

Tried uploading it to YouTube: the resolution was shrunk to the point of being unsable. Tried uploading to Google Video: despite the upload completing 11 hours ago, the upload is still ‘processing’. So I went through with ffmpeg and converted it to Flash myself, and set up the environment for playing it on openlayers.org.

To create the video itself, I used:

• Snapz Pro X (Shareware: $60)
• ffmpegx (Shareware: $15)
• Flash Video Player (Free for Non-commercial use: $15 commercial license)

In the video, I used TextMate (Shareware), QGIS (Open Source + Free), Firefox, and OpenLayers.

(There is no audio in the presentation, because I felt like my voice would be a distraction. I suppose next time I should put some fancy soundtrack in place.)

At BarCampBoston today, I got to talking to the OLPC folks. I was demoing our new vector editing support in OpenLayers, and talking to them about it. They wanted to see if it would run on their laptop, and we got an error that SVG wasn’t supported in their browser. After a bit of effort after getting home tonight, I was able to find out how to check for SVG 1.0 instead of 1.1 — after modifying the OpenLayers code to do a 1.0 check instead of a 1.1 check, I got a confirmation that vector display works on the One Laptop Per Child laptop! Woot. Filed a ticket with a patch for OpenLayers: 540, just need to get someone to review it and check it in.
So, the next step is to help the OLPC folks get up and running with OpenLayers. 🙂

So, last week was an OpenLayers hack week. One of the things that we did was make adding support for new vector formats trivial. Instead of modifying several parts of the code, you only need to create two functions: a ‘read’, which takes a set of data — XML, strings, Javascript object, what have you — and returns a list of OpenLayers.Feature.Vector objects, and a ‘write’ which does the reverse — takes a list of objects and returns a string or object or XML.

To prove this, I set out to write some additional vector format support last night. I decided to add one read, and one write.

• Read: KML. I added support for KML point display in about 20 minutes, including the time to find data and write a demo HTML page loading some example data. Adding LineString support was another 15 minutes.
• Write: GeoRSS. Support for writing georss:simple points, lines, and polygons was simple… once I found data. I asked for a live example, and was unfortunately unable to find any valid line data outside the GeoRSS website, so I just generated something that was as close as I could come to the examples. I’m lazy, so the export is just RSS 2, and I’m sure that someone will come along and criticize it, but that’s one of the benefits of Open Source: Anyone can offer up a patch. Time from when I created the file stub to when I committed the code was 27 minutes, again, including a demo.

Altogether, the Format support in the new OpenLayers is pretty cool. Because of the way it’s built, I can even do something that is pretty damn ridiculous: Import KML, and export GeoRSS (or GML), all from the browser. Certainly, this is an incredibly crazy thing to do, but OpenLayers is a pretty crazy project.

I’m convinced that there’s nothing in the code that would make it difficult for someone who’s comfortable working with Javascript to write support for any simple-to-parse format. Now, to get the code back to trunk and get the patches rolling in.

Ever since I had some problems with my server and Crucial Paradigm, I’ve been having too much spam in my inbox. I thought I had the exact same spamassasin config, but either I missed something, or the settings have just gotten less effective.

This morning, when I woke up to check the mail on expired domains from www.spamzilla.io, I finally got fed up and decided it was time to do something about the fact that my inbox was about 90% junk mail which was hiding the essential mails.

I’ve had a Gmail account since around August 2004, but I never use it. However, I do know that Google has better spam prevention than I do: something about having hundreds or thousands of humans acting as bayesian filters helps them. (Mm, scale.) After having recently implement Akismet for spam checking on the MetaCarta Labs weblog — with the result being that about 98% of the 3000 spams it gets every week are now gettin caught — I decided it was time to do something about my inbox spam.

So, I set up crschmidt@crschmidt.net to act as a forward to Gmail, and then set up my fetchmail to fetch from Gmail instead of from localhost.

Results so far are extremely positive:  Of the 85 emails that made it past my local spamassasin filter and into Gmail, 75 of them were spam. Gmail caught all but two. So, instead of 85 messages in my inbox, 10 real, I ended up with 12, and 10 of them were real.

If this stays true, my personal email address may actually become a reasonable way to contact me again. Sorting through the 200 spam I get every day and finding the wheat was becoming an almost impossible task, and this may have at least given me a leg up temporarily.

So, for the past several months, one of my main projects has been working inside of MetaCarta to put together the tools neccesary to let our customers integrate MetaCarta search results into all sorts of web pages inside their networks. Over the past several weeks, my main project has been taking the work we did in that regard, and making it available to the public.

So, if you’ve ever had an interest in finding out what MetaCarta does, the best way to find out is to:

• Check out MetaCarta Web Services
• Create an account
• Follow the step by step tutorial on adding a MetaCarta geographic search interface to a web page

Sure, it may not seem the most glamorous thing in the world, but we’ve definitely put a lot of effort into making these tools available, and I’d love to see some people try it out.

I can’t seem to find any places that I can get to that have many slrn users — I’m looking to figure out if it’s possible, in the article listing, to ask slrn to display already-deleted messages.

Anyone have any advice? Or tips/tricks on slrn in general?

Went to BarCampBoston today. Took Schuyler, Jo, and Gregor with me. Had a great time, the venue provided by Monster was absolutely incredible in a lot of ways. I’d really like to thank the people at Monster for their time and effort pulling things together: everything at the conference *just worked*. (It’s like it was a mac or something!)

Schuyler and I gave a presentation on mapping on the web, centered primarily around OpenLayers and the uses for it:

Ever since Google first ‘solved’ interactive web-based mapping for the masses, providing an API to put your data on top of, people have explored the space at an ever increasing rate. This same appetite for data-driven insights has led services like crypto casinos UK to incorporate mapping technology into their platforms, enabling them to track user trends and compliance across different jurisdictions. From Google Maps Mashups to all new datasets created out of the existing data, and hacked into that ever-lovable interactive mapping interface, people are creating new, exciting, and sometimes even innovative ways of looking at existing data.

Come discuss the things you’ve done with maps, the things you’re doing with maps, and what you think of the ‘mapping revolution’. See what alternatives there are to Google… and why you might not want to be quite so dependant on the corporate data silos.

The links which were shared during the meeting are available from my BarCamp Presentation page.

A bunch of people have emailed or commented here with things that they think I should be checking out. They’re probably right. However, I’m short on time, and working on a project heavily at the moment, so if you’re waiting on an email from me, please be aware that I probably saw it, I just haven’t gotten around to actually answering everything yet.

I’ll get back to you soon, I promise.

I’ve just implemented a fair amount of memcached support into CGI::Wiki, the Wiki software that powers OpenGuides. Now, any time a node is fetched, it will come from memcached if possible, or be loaded and stored into memcached, and when nodes are edited, the cached version is deleted.

This is in addition to adding caching to the main map pages, which were previously taking up to 20 seconds to load, and are now in the ~1 sec/range. (There is much more slowness in the browser side of this page, due to the javascript proccessing time required to load hundreds of location markers).

Additionally, the Open Guide to Boston is now running under mod_perl after my Message to the OG-Dev list with a patch that fixed the major stumbling block. This has increased speed somewhere between 5 and 10 fold over using CGI.

About 15 minutes ago, I restarted memcached to clear out old stats. I then visited the main index pages for both the Milton Keynes and Boston guides, to prefill the caches. There were 3535 items which were loaded (GET\_MISSes). In that time, there have been 4 new objects added — and 946 GET\_HITs. With the stability of memcached, I expect that this cache will be able to run for a long time with no maintenance, and I’ll be able to come back in a few weeks to see that memcached has saved me thousands of hits to the mysql database, therefore increasing the speed with which the Open Guide to Boston loads, something that I’m sure everyone can appreciate.

Most of the slowness that memcached is fixing in this case is not caused by MySQL being slow, but rather by the processing done by OpenGuides after the fact, due to the way that it is designed. The software was never really built to be the most efficient Wiki software out there, but to scale to the extent that it needed to, a task which it has completed admirably. However, this has left some corner cases where the performance was less than perfect — typically cases which were seldom used. The example of the “All nodes” map that the guide has is a good one: I built that page out of a corner case (the ?action=index view) which is not linked from the guide itself. As a result, I discovered a case where the software did not perform as expected. There was a roundtrip to the database to load the data about each node. This round trip time combined with a complex map { } function brought performance way below acceptable levels. So, I sought to improve that. First I simply cached all the data after it was processed, and today implemented caching of the individual nodes. Although the second change is minor and may not offer significant benefits on the small scale, it does help to build up the larger cache that it’s supporting: Roundtrips to memcached are quicker than roundtrips to the database.