Technical Ramblings

Chad writes:

I know their “motto” is “Don’t Be Evil” .. but I think it should be “Don’t Be Smart” instead.. this is some dumb thinking from Google. Trust me.. I know better than Google on how I want to download and install my software.

This is just the latest in a whole lot of similar statements I’ve seen from many people across the web in a variety of situations talking about how “I know how to manage my machine”, with the underlying meaning being something like “You should act as if people who are working with your software know how to work their computers.”

When I put it that way, does it really sound right? Is there anyone who thinks that the *majority* of users of Google Earth actually know how to run their machines? Is there anyone who thinks that it makes sense for Google to build and QA two different install mechanisms — one for technical users who know what they’re doing, and one for those who don’t?

Very few companies the size of Google do anything on a whim. I expect that some thought went into the development of the Google Earth downloader. The fact that the thinking is not centered around technically competent users is just evidence that Google doesn’t need to target the early adopters; it’s not a sign that what they are doing is ‘Bad’ or ‘Stupid’.

Technical users are few and far between in the mass market. Google Earth is targeted towards the mass market. Just like all software that is targeted towards a mass market, there is nothing ’stupid’ about removing tools that the majority of users don’t need or care about: By doing so, you limit the number of people who are going to end up confused by your tool, and that’s not a bad thing when you care about the majority of people instead of a technical elite.

One of the problems I’m running into with mapserver right now is related to its rendering of LINE elements which are wider than one as they run into tile boundaries at acute angles. It seems that mapserver is drawing the centerlines for these elements up to the side of the image — but in cases where a line is approaching a boundary at an acute angle, this means that the ‘outer’ edge of the rendered line stops away from the edge.

In non anti-aliased lines, this is less visible as a problem (especially if you’re not looking at the images as tiles) because the lines just stop — and visually, it’s hard to tell if it’s at the image edge. However, it becomes very obvious in cases where anti-aliasing is on because the edges of the left and right boundary are ‘tied’ together by a curving, anti-alised line: resulting in a bubbled look at tile boundaries.

I’m not sure if this is a known bug, or something that other people have run into: I’m mostly recording it here so that I have a description of it.

Right now, it seems like the workarounds are:
* Use thinner lines (so it’s less visible)
* Don’t have roads near boundaries at acute angles — although there’s not much I can do about this one!

WMS is great… when it works. But having all my maps go busted because a server outside of my control decides that it doesn’t want to work today (today, the MassGIS WMS server, yesterday, a TIGER/LINE source I was using) is really annoying.

Oh well. At least one of my two basemaps is up.

So, tonight I took it upon myself to redo the javascript behind the map for the Open Guide to Boston. The reason for this is relatively simple: when I first wrote the Google Maps interface, the guide had about 50 nodes. With that in mind, drawing them all was acceptable. As the guide got bigger, it got a bit more time consuming, but was still much easier than paging. However, the guide has recently doubled in size as I’ve increased the rate at which I pull data from Zami.com (specifically for the purpose of trying to build a free database of all the churches in the area with user interaction) — up to 2400 nodes (making it the single largest Open Guide to date by pure node size, as far as I can tell).

With this change, the map was no longer just difficult to use: it was impossible. Attempting to open the page crashed my web browser.

So, I took it upon myself to learn enough JavaScript to do paging… but then decided that rather than paging, I should attempt a bit more friendly solution to start. So I started hacking, and got into the Google Maps getBoundsLatLng() function, which allows me to determine the area of the map.

There are two basic options to go from here, depending on performance you expect in various situations, and the scalability you want to have.

* Load all data. Create Javascript variables to store it all. When the map moves, iterate over the data, adding markers for only points which are not already visible, and are in the new span.
* Load only the original data from the database. All additional data to be loaded via XMLHttpRequest.

The first is obviously something that can be done entirely client side, and in fact turned out to be the (much easier than I expected) path I chose. The largest reason for this is simply ease of integration. OpenGuides code is rather convoluted (to me), and modifying it is not something that I enjoy spending a lot of time doing. I’ll do it when I need to, but I prefer to avoid it. The other option would be to simply query the database directly via Perl or some other language, without using the OG framework. This would probably be slightly faster, but with the size of data I’m using, iterating over it is a minimal time compared to the time drawing the GMaps Markers. As a result, I chose to go with the slower, less scalable, but quicker-to-implement and merge to other guides way of doing things. The biggest benefit here is that I can merge it back into the other quickly growing guides — Saint Paul is now growing leaps and bounds alongside Boston, due to help from pulling Zami.com’s database, and I plan to do similar things with other guides. Also, London probably will not be able to use the mapping stuff in any useful way without this kind of code being added, so I went for the quickest thing that could possibly work.

There were a few gotchas that I had to end up avoiding:

* Removing Google Maps Markers takes *much* longer than adding them. Like, 3-4 times longer in extremely informal testing. With one or two, or even 20-30, this doesn’t matter, but with 100, this starts to be a significant barrier to user interaction.
* There are a number of words which are reserved in Javascript, but are not treated that way in Firefox. This leads to confusing error messages in Safari (although they are slightly improved in the latest webkit): If you see a ParseError, you should check the Firefox Reserved Words list — this is in one of their bugs. For example, the variable “long” is reserved, and can not be used to pass something like, say, longitude.

Discovering the ParseError, working out why it was there, and how to fix it, was greatly helped by the #webkit folks: I have never seen a more dedicated, hardworking, friendly, open and inviting group of Open Source Software developers in my history as a programmer, either in closed or open source products. Many thanks to them for helping me work through why the error was happening, and how to avoid it in the future.

Lastly (although this is not quite chronological, as I did this first), I had to modify the code to the guide to zoom in father to start, so it wouldn’t try to load so many nodes. I think a next step is to allow users to set a default lat/long for themselves, a la Google Local, from which they can start browsing, rather than always dropping them into the main map. But that’s a problem for a nother late night hack session.

For the past 4 weeks or so, I’ve been working on a project known previoiusly as 24 Hour Laundry.

Now, it’s no longer 24HL: Welcome Ning.

A development playground with all kinds of neat and nifty toys, Ning is attempting to do to application and code sharing what other apps have done to photos, bookmarks or other arenas. Allowing people to clone, mix, and create new apps.

There’s a lot of cool things here, and I’ve got a pretty bad headache, so I’m not going to be able to cover all the things that I would like to here, but here’s some of the cooler things about the site:

* System wide content store. Public content which is created can be accessed by any application. This content store is well abstracted, and has a content creation and query system. You don’t have to worry about scaling up: You can leave that to the professionals in the backend. At the same time, you can collect data from all the other apps in the playground. You want to create a book reviews site? First, grab everything that’s known as a Book from the site, and then use the built in classes for ratings and comments to build a discussion board. The possibilities for content mix and match are really spectacular. However, if you don’t want others touching your data, you can mark it as “private” and use it only in your app - but why would you want to?
* Built in classes for lots of things. Build a calendar. Interact with Flickr. Make a GMap. Talk to Amazon. The code’s all done for you, you just use it. Bookshelf makes extensive use of the Amazon classes, Restaurant Reviews With Maps uses Google Maps to show where you’re going — Bay Area Hiking Trails shows you how to get there.
* RSS feeds of content. The Ning Pivot is a really cool way of looking at the content flowing by, but not only can you watch it, you can watch it flow by.

There’s about a half doezn other really nifty things here that I can’t even think of at the moment because it’s 5am and I’ve been walking like a Zombie for two weeks to get this stuff complete.

But the coolest thing is:
* All data added is placed under CC By-SA license. (If you don’t like this, ning isn’t for you.)
* All app code is completely open, and you can make it your own in 2 seconds.

Screw Ruby on Rails: who needs a 2 minute app, when you can write a 2 second app? All depends on how fast you can click.

If you run into problems with ning, feel free to drop them here: You can never fix all the bugs before release, but I think that the team working on Ning has done an absolutely incredible job with all the work they’ve put together here. I’ll pass them on as best as possible.

There’s a lot of other stuff I want to write — one that others here might find interest in is how similar Ning’s content store is to RDF, and why I think that there’s no functional difference. Of course, Marc and I got into a nice “discussion” on that one on IRC the other night, so maybe I’ll wait til I’m a bit less exhausted and can adequately express my points on the topic.

crschmidt.net now features a new colorscheme: I’m still not sure how much I like it, but the old black/grey/white scheme was really starting to grate on me. (Note that the weblog uses a different stylesheet which I haven’t updated yet.)

Additionally, all pages now have a feature to allow commenting from users. So you can now leave a comment on any page! This is taken from Eikeon’s websites, which have this feature (although it requires logging in first). I’ve done some very basic escaping of script tags, and I do my best to add newlines if they are appropriate, but if you want to make your content look right, you’re best off just formatting it with HTML yourself.

However, this means that it’s really easy to offer feedback on any page of the site now. If you’re interested in my semantic web tools, you can leave comments on the various ones there. You can comment on the code for any of my Python tools, on my symbian stuff, on pretty much anything. Soon, and very soon, I’ll be writing an RSS feed generator for this. Right now I’m just happy it works, and would love to see people commenting on any page on the site they’d like more info about or would like to offer feedback on.

Every now and then, I’ll try and type a difficult to type word on IRC, and curse the lack of auto-complete built into my IRC client. I’ve always thought “I should really look into fixing that.” Well, tonight I was sleepy and browsing through the entire list of irssi scripts (obtained via `rsync -avz main.irssi.org::irssiweb/scripts/scripts/\*.pl ~/.irssi/scripts/official’`), and I discovered that there is a “wordcompletion” script, which pulls data from a MySQL database.

“Nifty!” I thought, and poked at it a bit more, finding that it simply stored words you used in messages into a MySQL database. So, I got to thinking. Wouldn’t it be nice to take the words from /usr/share/dict and dump them into there?

So I did.

for i in `cat /usr/share/dict/american-english”`; do export v=`echo $i | perl -pe “s/’/\\\\\\\\’/”`; echo $v; echo “INSERT INTO words (word, prio) VALUES (’$v’, 1)” |mysql -u irssi -pPASSHERE irssi ; done

And since I did it, I saved you the work: You can fetch the entire database dump (in compact, minimal impact one-insert form) from odds and ends, a new section on crschmidt.net. Additionally, you can grab my new version of the script from there, which changes the script to read all messages rather than just ones which were typed by you. In the process, I became interested enough to work out how to store these fields in a setting - the new version of the script features a number of improvements, such as saving the database password, user, and dsn in a setting, as well as offering help, so people who don’t know Perl enough to even change simple variables can use it.

I’ve contacted the author to let him know about these changes so he can roll them into the official version if he wishes. If I don’t hear back within a week, I’ll submit my version as an update to the original script at irssi.org.

Programs which are easy to script make a great wya to keep yourself occupied late at night, and let you occasionally release something that seems impressive which otherwise wouldn’t. Thanks to the original author of the script (Jesper Lindh) as well as the authors of all irssi scripts for their help in getting this one out the door.

While many people these days are switching to annotation-in-XHTML, there’s still at least one file format out there which has extremely useful metadata annotation using RDF/XML inside the document: SVG.

The Scalable Vector Graphcs format has a Metadata element, which is expected to contain RDF/XML. This is great news for people who might wish to create a directory of SVG images: the metadata can be stored in the actual images, something that the Open Clip Art Library takes advantage of, using a number of tools to extract statistics and aggregate metadata from SVG files.

To take an example from the library, Autos_01.svg (SVG file, requires SVG viewer) contains 23 RDF statements. These triples are given a base of a cc:Work with the URL of the file of itself, meaning that a simple query about the predicates and objects with http://openclipart.org/clipart/transportation/autos_01.svg as a subject returns the important aspects of this document. This includes description, creator, keywords, and license. The license is “Public Domain” — adding the images to the Open Clip Art Library requires placing them into the Public Domain.

For working with this data, developers of the project created the Perl module SVG::Metadata - a module for annotating SVG files with this metadata, as well as making change to the metadata which already exists in such files.

The maintainer just announced on the Clipart Discussion list that he has released 0.28, which includes the changes from previous releases 0.26 and 0.27 which were mostly maintenance releases. (The message will eventually appear in the August threads, but hasn’t yet.)

The RDF generation in versions prior to 0.24 was broken, but was fixed in the 0.25 release - OCAL is now using this release in their scripts, so many of the more recent images in the library are valid RDF, meaning that you can simply pass it to Redland with the http://feature.librdf.org/raptor-scanForRDF feature set. In the Python bindings, that is:

p.set_feature(”http://feature.librdf.org/raptor-scanForRDF”, “1″)

In rapper:

[crschmidt@creusa ~]$ rapper -c -f scanForRDF=1 http://www.openclipart.org/incoming/cat_scrathing_post_benji_01.svg
rapper: Parsing URI http://www.openclipart.org/incoming/cat_scrathing_post_benji_01.svg
rapper: Parsing returned 30 statements

I think this is a great example of how to work with structured metadata without dealing with the crappy aspects of RDF/XML syntax corner cases: simply write a library which parses the metadata, fills your variables up, and lets you modify them with a standard API, then lets you resync the data to the file. Congrats to Bryce for his hard work on the module, and on making the metadata for these SVG files accurate and useful to external users.

At some point, on the FOAFnet mailing list, David Sifry asked, ” What is DOAP? What is GRDDL? What’s the use case?” in response to Danny Ayers’ post on the mailing list.

My reply was pretty simple and succinct in my opinion, but I never put it anywhere public to read. I think it’s a relatively succinct explanation of why GRDDL can be useful for moving between the microformats, with tiny datasets, and RDF, with the large dataset behind it.

What is DOAP? What is GRDDL? What’s the use case?

DOAP - Description of a Project. DOAP is to Open Source software what FOAF is to people. http://usefulinc.com/doap

GRDDL - Gleaming Resource Descriptions from Dialects of Language. This is a way of performing transormations from (x)HTML documents to an RDF document. This transformation takes place via an XSLT transformation, specified either implicitly through the “profile” link in the <head> of the HTML document, or via a meta rel=”transformation”, which parses can then apply to the document to get something with real meaning out of.

http://www.w3.org/TeamSubmission/grddl/

Use Case:

“Microformats” and storing data in HTML may be good for people who are writing specialized tools for each format they want to deal with, but GRDDL allows for people who want to support all these to simply write one translation, using XSLT, and take advantage of the underlying data model of RDF. This allows you to merge existing data sources with the newly emergent small-s semantic technologies, and to use existing tools for combining these sets of data and discovering correlations between them, using the already existing RDF data access framework.

If I have a datastore of RDF data (which I do), and someone wants to find if the maintainer of a certain project has contact information in that database, using hBlah pages, I can’t really do that. However, if I merge the datasets with the already existing FOAF data out there, I can find out that a person named Christopher Schmidt, who is a maintainer of “julie, aka redlandbot”, also has an email address of crschmidt@crschmidt.net.

That’s a pretty simplified use case, but the general idea is simply to take the tiny sources of data that the h* formats provide, and integrate them with the millions of pieces of data out there already in FOAF, RSS, and everything else Semantic.

I mentioned the other day the release of Apple’s WebKit, WebCore, and JavascriptCore (the latter two of which were already publicly available). Naturally, the first thing I wanted to do was download it and give it a try. This post will outline the steps I took to get as far as possible in the build process at this point. First, I would like to mentioned that this project has the cleanest build steps I have ever seen. It is well documented all the way through, and for Tiger users on Xcode 2.0, the build process went off without a hitch, the first time through. (Xcode 2.1 problems have since been fixed.) The members of the supporting IRC channel are helpful and intelligent, and the mailing list has already taken multiple patches from non-employees into the source tree. This is, quite simply, the best opening for an open source project that I have ever been aware of.

However, the build process currently favors those with Tiger, and the current CVS does not support those who are using Panther. Apple developers have expressed an interest in correcting this once the WWDC, being held this week in San Francisco, is over. So, I took it upon myself to report bugs in bugzilla where they are applicable, to help out developers when they get a chance to breathe.

First problem: Building returned a problem with “CarbonSound.h not available”. This was as a result of not yet installing the QuickTime 7 SDK. (It has been in software update, I just hadn’t touched it yet.) Updating fixed that.

Second Problem: 10.3.9 Build Failure: NSString may not respond to `+stringWithCString:encoding:’. This is a method which was not available in Panther. Maciej has said he is working on a patch to have this use CFString instead, where it is available. (I am tossing about some terms I don’t know here, so please excuse any incorrect terminology.) Workaround for the time being - copy the last two build commands before the crash (a cd line and a gcc-3.3 line) and past them, altering the gcc-3.3 line slightly to remove the -Werror. This means that it may cause problems later on, but will compile for the time being.

Third Problem: isnan failure in kjs_window.cpp: This one boggles me a bit, especially since (as I mention in the bug) there seems to be explicit knowledge in the code of the problem. However, a workaround is now offered in the bug in comment 1: replace using std::isnan; with extern “C” int isnan(double); This fixed the problem for me.

Fourth Problem: XSLT Headers not installed - This one is more systematic of the way that Apple releases updates, and is something that dajobe has brought up with building Redland in the past: “Headers don’t match libraries”. This is true here as well, but I now (thanks to toby from #webkit) know that the reason for this is that Apple does *not* ship updated headers with libraries updated through Software Update. Since libxslt is new in 10.3.9, there are no development headers. Dave Hyatt, of the WebKit team, mentioned that the whole team, when building on Panther, had to install libxslt and libxml from the source. Once I did this, it made this problem go away.

Fifth Problem: libxml headers are wrong - this was before I installed libxml, which also fixed this problem. It is, again, related to the fact that Apple does not update headers with System Update.

Once you get through these, you will have built both JavascriptCore and WebCore. Congratulations! You now have two completely useless frameworks which the new Webkit will depend on when you can build it!

WebKit is the previously unreleased Apple-specific Framework which is the “pretty” part of WebCore - it’s what ties everything together. It has a few more issues building on Panther, but most of them can be worked around by simply copy pasting build lines without the -Werror flag. (Note that this will produce possibly unstable results! These builds are not designed for production, and I do not advise doing this and filing bug reports on Safari crashing.)

npapi headers not available - for some reason, building on Panther does not find the appropriate headers from the in-process WebKit build. I really have no clue why this is, and neither did anyone else when I was building. My workaround was to copy the headers out of the framework and into ~/build/include (a directory I had to make), which was already on the path. cp ~/build/WebKit.Framework/Versions/A/Headers/* ~/build/include, cp ~/build/WebKit.Framework/Versions/A/PrivateHeaders/* ~/build/include, then continuing the build. I am not sure why this is neccesary, but it does seem to work.

Missing 10.4 Method -setCompositingOperation for WebImageRenderer - Two parts of the code require: (void)setCompositingOperation:(NSCompositingOperation)operation;
(NSCompositingOperation)compositingOperation; — this function was added in 10.4. This can be resolved by following the above -Werror removal steps. You will have to do this several times.

Missing 10.4 Method CFMakeCollectible - CFMakeColelctible is new in 10.4. Building with no -Werror allows the build to continue.

And, the current showstopper: Missing SecurityNssAsn1 headers — This comes from the libWebKitSystemInterface.a file, which is currently Tiger-specific. Once WWDC is over, a Panther binary file will be released. Until then, this is where the ride stops: you can build WebCore and JavascriptCore, but WebKit is out of your reach until you get your hands on Tiger.

Luckily for me, I’m going to be in Cupertino this weekend, so I’ll pick up a copy and get it installed soon