Technical Ramblings

Prototype of a worldwide seamless multi-resolution baselayer powered by open data:

OpenAerialMap

Jess’s Powerbook died last night. Trying to start it up, I found that it was unable to mount the drive read/write. The symptom of this that we were able to see, by the way, was that the computer just ‘hung’ at the Apple with the spinning logo under it, because some file was unable to be written to and therefore startup hung waiting for it to come back.

I figured out how to get into single-user mode (apple-S during startup) and ran an fsck on the disk. (Before this, the disk could be mounted read-only, but not read-write.) It seemed to work, for the most part, though it complained of a bad journal, so I couldn’t mount the drive read/write. Figuring I’d start over after the fsck and see if that let it boot any farther, I restarted… and found that my fsck had let the HFS volume figure out that the Journal really *was* bad, and that it was no longer going to even mount read only. (Sigh.) The error message was “journal magic is bad”, followed by “nfs_boot_init failed with 6”. I was royally pissed at myself for failing to buy the Firewire cable and get data off of it first, but figured I’d fix it in the morning.

I went out and bought a Firewire cable this morning and mounted the disk to my laptop. It showed up in Disk Utility, and I ran “Repair Disk” on it. Disk Utility reported that it had repaired the disk successfully, and even claimed it could mount the disk successfully, but never did. Trying to boot the computer still paniced immediately, with the same “journal magic is bad” message. So, following the advice of Some Guy On the Internet, I went ahead and bought DiskWarrior, and he’s 100% right: “Dont waste any more time. Get Disk Warrior. Run it on the volume. It just works. Done.”

Of course, for some people, it’s not so simple: disable_journal.c was written by an Apple user who couldn’t get even DiskWarrior to fix the problem, so instead wrote a small C app to hack the bits of the drive. (Ouch!) But for me, DiskWarrior was just the best $100 I’ve ever spent. (Though I do wish I could have skipped the $9 shipping, since I just bought it online and I don’t really need or want a copy in the mail.)

Thanks to DiskWarrior, Jess now has her laptop back, with nary a file misplaced. You can’t ask for anything better than that — and all told, it only took about 15 minutes, including the run to the store to buy the Firewire cable.

San Diego Wildfires 2007

Lots of different data here:

• Landsat7: i-Cubed processed landsat7 WMS hosted by Telascience.
• USGS Urban Orthos, hosted by Telascience.
• 2006 DigitalGlobe imagery of the area, hosted by Telascience — built last night specifically for this project
• MODIS fire map (last 7 days), hosted by Telascience, put together by Martin Spott
• OpenStreetMap Hybrid tiles built by me and hosted by TelaScience
• Destroyed houses: WFS served by FeatureServer, initial population from KML file
• 2 KML files, from Google MyMaps: KPBS and LA Times
• Geotagged Photos tagged ‘wildfires’ from Flickr, converted to WFS by FeatureServer.

OSGeo Users Group meeting in Boston tomorrow (well, it’s already tomorrow, but you know what I mean) night, 7pm at MIT Museum in Cambridge.

When: Wednesday, Oct 17th, 7PM
Where: MIT Museum, Downstairs, “MIT360” space,
265 Massachusetts Ave., Cambridge, MA
http://web.mit.edu/museum/
Boston Freemap
What: First Meeting of Boston OSGeo users group: intro, and maybe review of FOSS4G?
Who: Anyone interested in open source geo software
Why: Because it’s there!

I figure we’ll spend an hour or two getting to know people, talking about FOSS4G (for those of us who went… that might just be me), etc.

If you plan to be there, please respond to the list so people know to expect you! (A ‘maybe’ is fine.)

OpenStreetMap display of Massachusetts. (This will soon enter the main OSM caches — I’m just impatient.)

Thanks to Brandon Martin-Anderson, one of MetaCarta Labs’s new minions, who helped me get started by writing MassGIS shapefile -> OSM file converters, and to all the great people who have made the OSM Server and code so much more usable since the last time I tried to do this.

REST is really about just one thing: making your resources available in a way that everyone nows how to get to them, change them, and remove them. What does that mean? Well, this is what REST is really about:

66.249.66.20 - - [07/Oct/2007:07:34:06 -0500] "GET /mapping/wpserverdemo/featureserver/featureserver.cgi/scribble/67.html HTTP/1.1" 200 658 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
72.14.199.40 - - [07/Oct/2007:07:36:43 -0400] "GET /atompub/featureserver/featureserver.cgi/scribble/726.atom HTTP/1.1" 200 1182 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=14743344383889883903)" "-"

Google has been slowly crawling through the wpserverdemo HTML pages overnight, and several people have subscribed to the feed or individual features from the AtomPub demo I put together under MetaCarta Labs.

Addressability. Linkability. Shareability. All these things are what the principles of REST are attempting to bring about. Making it so that the things that have worked so well for the web can work for you — things like Google searching, things like syndication, things like web browsers. They’ve worked well for lots of things over the years, and if you’re working on the web, you should consider how to take advantage of that.

The WPServer demo is more complete now: WPServer Demo.

New functionality:

• More operations
• Control over size of buffer
• Control over tolerance of simplify
• Ability to select specific features to operate on
• More complete set of OpenLayers Feature editing tools

Note that everything here is included in the stock version of what will be OpenLayers 2.5. All the code that you need to set the client up is displayed in the web page — there’s no magic behind it.

There is one additional thing to note: The demo now has the ability to load features from the FeatureServer demo. This is one of the most important things about working over the web: the various pieces of data that exist in disparate data sources can be integrated.

Unfortunately, this is made harder than it should be because of the limitations of the browser. For security reasons, it is not possible to load data from any URL on the web: in this case, the WPServer ‘noop’ (no operation) is used essentially as a GeoJSON proxy, fetching the features from FeatureServer to bring them into the local javascript domain, where we can fetch them. However, once that proxy is set up, we can fetch data from multiple locations, and put it together on the client, creating new features from the old, and taking them out to put them wherever we like.

Note that although this demo uses GeoJSON extensively — it’s the only language WPServer speaks at the moment — there’s no reason for that to be the case. OpenLayers has the ability to load data from WFS, as well as the ability to load other things like KML documents. You can load these datasources into the same layer, and use the WPServer functionality to put things together.

Want your shapefiles on a map? Use FeatureServer. Want to buffer each of the points in your FeatureServer-served data? Serialize them, and pass them up to WPServer, then display the data that comes back. Want to mix in KML data, to see the intersections? Add a KML layer to OpenLayers, and use WPServer to do the intersections.

… Crap. I think what OpenLayers can do now might actually be something people would refer to as GIS.

There’s a couple of different efforts going on to do web processing over the web — in an effort to understand what it’s all about, I put together a little web processing server and a demo to go along with it.

WPServer Demo

Using OGR and the GEOS operators built into it, it allows one to do operations I’m told are common: buffering, centroid, convex hull, and dissolve.

The source for these operations is simple: as few as 8 lines to define your own operation, as the OGRGeos operators and the Shapely buffer operation show.

The Boston Routing Demo is also now using WPServer to do its routing, via the PGrouting action.

Although WPServer shares an acronym with OGC’s WPS, it doesn’t yet implement WPS — however, like FeatureServer, it has a pluggable frontend architecture for determining the types of actions to perform… but only REST + GeoJSON are supported so far 🙂

I think it’s a pretty looking demo, even if I’m not sure if it’s useful at all.

Andreas posted earlier today with a followup on the SLD Rules front: Though I had consigned myself to expressions, he had not, and went on to create a very nice looking rules system which supports the full range of SLD operators.

More information is available in his post to the mailing list: the code is in his sandbox (though this URL will move once it moves to trunk).

Three rule types are used to build up rules: comparison rules, for comparing values against feature attributes, FeatureID rules, for limiting by fid, and Logical rules, which group the other rule types together. Logical rules can be AND, OR, or NOT.

One key benefit to this — which I had not thought about until Andreas brought it up — is the ability to serialize these rules back out to SLD. It’s possible to use something like yacc/lexx (C, etc.) or eval() to evaluate an expression — but in Javascript, at least, it’s hard to turn that expression back into rules. This saves us that problem, and means we can serialize out SLD from user-created styling rules — clearing the way for the in-browser SLD editor that many people have been talking about for a long time.

I’m curious about how others store/evaluate SLD rules/filters. At the moment, in OpenLayers SLD rendering, we have an ‘applies’, which is a string something like:

“feature[‘attributename’] == ‘foo’ || feature.fid == 1”

Which we eval. Yuck, I know.

But how else do you do it? What’s the internal storage look like?

I originally thought of something like a set of tuples:

(‘operation’, ‘value’, ‘type’, ‘extra’):

• (‘equals’, ‘foo’, ‘attribute’, ‘attributename’)
• (‘equals’, 1, ‘fid’)

But then Andreas pointed out that these are not just ‘and’ or ‘or’, but either of the two, and can also be nested… it seems like it quickly goes complex enough that we would need to implement a full boolean logic thingy of some kind, and it seems like the wrong way to go… but then again, so does eval.

Would love to have advice from someone else as to how they implement SLD filters and their application.