Technical Ramblings

A better release announcement, which I wrote over on the Labs Weblog:

MetaCarta Labs is proud to announce the release of FeatureServer, an open source, Python based RESTful feature server.

FeatureServer allows you to store points, lines, and polygons in a number of backends, and get the data out all at once or one piece at a time. You can get the data out as KML, JSON, GeoRSS, GML/WFS or even as HTML.

FeatureServer is primarily designed as a lightweight vector feature storage companion to the new OpenLayers vector capabilities.

A demo of FeatureServer in OpenLayers is available at the FeatureServer Demo page. You can see features from this demo in:

• HTML
• KML
• GeoRSS (Atom, simple)
• JSON

FeatureServer is the first project we’ve released under our copyright-only open source license, a move designed to protect both MetaCarta and project contributors. We’re moving forward with getting this license approved by OSI, and plan to release future open source software under this license.

I’ve been looking forward to this for a long time, and am looking forward to demoing it for people at Where 2.0. I think that FeatureServer really is the first step forward in the world of interacting with geodata on a feature level rather than a giant-database level. Per-feature interaction — view, update, delete — as well as easy-to-use attribute querying are both huge boons when working with lots-o-data: instead of loading 700,000 features, or even just loading features based on a bounding box, you can talk about a feature based on its unique identifier.

It also makes a really great translator — for example, it makes it possible to load up your Flickr photos as a WFS server which you can drop into OpenLayers. No need to depend on the external interfaces: anything which provides a geospatial data API can be converted to a FeatureServer datasource.

The ability to read in GeoRSS and KML are great from the point of view of an aggregator: simply pipe the output of a curl GeoRSS download into a FeatureServer upload, and you’ve got a GeoRSS post collector.

There’s a ton of possibilities, and I’m looking forward to helping people explore them now that the software is available and open source.

I wonder how many people are going to the MetaCarta Public Sector User Group Meeting today…

Went out for a late night walk tonight: though my path was a bit unclear, I think I made a decently accurate map:

Walking around MIT

If you hover over the circles, you’ll see points of interest along the route. I recommend zooming into the main MIT building, to see the confused meandering I was forced to.

This demo uses OpenLayers, TileCache, and FeatureServer. It was created entirely via a web interface.

Can You Copyright the Globe?

Originally uploaded by crschmidt.

When at the Museum of Science, I noticed that the giant globe in the atrium near the entrance of the museum is copyrighted by Rand McNally. Although I understand that the interpretation of a lot of raw data to make a 3D model of the Earth is definitely a creative work, it was still strange to read ‘Globe copyright Rand McNally…’

I think from now on, I’m not going to try to answer questions on mailing lists which don’t follow the guidelines of “How to Ask Questions the Smart Way“. I spend way too much time trying to extract information that’s needed to solve any
problems. Obviously, in some cases, it’s hard to know which information is important, but with two open source projects with growing user populations, it’s very hard for me to take the time to coax every person through getting the information that is important to solving problems.

I think it is a sign of the growth in popularity of the OpenLayers and TileCache projects that I’ve come to this stage — it’s only when projects reach a certain level of maturity that people start asking enough questions that I don’t feel like I can just answer everything. Certainly, for the first 6 months of the OpenLayers project, I never felt that answering questions on the mailing list was something that was a burden to do, but over the past couple months, it’s been an increasing level of traffic from users who haven’t taken the time to investigate the cause of their problems fully. I think the time has come to stop devoting as much energy to solving *every* problem, and instead solve the problems that can’t be solved by other people.

Of course, an additional step is to work to create a forum in which users can offer cash to solve their problems — you put up a question, with a cash bounty on it, and you either increase the amount of money you’re willing to spend to get your question answered, or you answer it yourself.

This seems like the kind of thing that there should already be software to do. I’ve never heard of any, though. I wonder why that is. Probably something obvious I’m missing.

So, I’ve created a lot of maps with Open Source tools. In fact, I put together my first online map with MapServer and a not-yet-released OpenLayers about a year ago this month. (Before that, I’d never actually done anything with geospatial tools.)

I usually stick to my local Open Source world — reading Planet OSGeo, rather than PlanetGS — but every now and then, I wander across the line to see what the rest of the world is using.

What I find is a hell of a lot of Arc* names, and no real understanding of what benefit they provide. Although I understand that Analysis is difficult with existing open source tools — not that it’s necessarily better in proprietary tools —  it seems that much of what people want to do with software like ArcGIS is just make a map.

So, what’s the benefit to using these tools? What doesn’t open source do… and how do we make them do them?

This week, I had the oppourtunity to work with Schuyler in writing stored procedures in Postgres/PostGIS for the first time.

At first, we were writing in plpgsql, but found it didn’t suit our needs… and switched to Python.

Yes, Python. Our database now has stored procedures which decode a cPickle pickle structure from a column (go go unstructured data) and return the output of a key/value pair (based on the key).

It was a very weird thing to see this actually work.

So, who should I be on the lookout for at Where 2.0?

As a follow up to sgillies’ post on Feature Paging, I’d just like to say that I was thinking of this a couple days ago for FeatureServer, and implemented it last night for the datasources that FeatureServer provides for which it makes sense. (The WFS DataSource is one of the ones that doesn’t.) The implementation is pretty immature — for everything other than postgis, it’s just a ‘skip’ counter, but it’s there.

(Note that I did this before I even saw posts about it — I think this is just more evidence that all the things we’re talking about here are well understood and not novel, though it is interesting that we’re all implementing them at the same time.)

So now you can do:

FeatureServer/layername/all.georss?bbox=-180,-90,0,0&startfeature=10&maxfeatures=10

And on a PostGIS layer, it will translate into a LIMIT, OFFSET block, and in a DBM layer, it will just iterate and not do anything, and in OGR, it will iterate and skip over the values that it doesn’t care about.

I think that’s cool, anyway.

More followup from the list: It seems that there is a reason (in addition to routing) for topological behavior: editing.

If you have a junction described in “features”, what you get are two roads that somehow happen to meet in one place (two lines crossing, or meeting). Right? In terms of storage, both lines will usually have their own coordinates in the data base; the information that they intersect, or meet, is not there explicitly, it just comes from the maths applied when drawing them, and if your database has spatial support you can ask the database for the point where they intersect.

However if someone wants to move that intersection, he will have to edit each feature separately – if you move one road away from the junction, then the junction is no more.

(From the mailing list.)

It’s a valid point, and not one that should be easily tossed aside. However, I’m not arguing against topology in all cases: I think that this is the exact kind of thing for which topology should be used — in the client.

If you have two nodes that are on top of each other, they should merge together. When you drag one, any feature that includes that node should be updated. When things don’t actually intersect, they should be separated. The canonical example is a bridge-over-river situation: If I want to draw a bridge over a river, and need two nodes in the exact same place, how do I do that without joining them?

The next piece needed beyond simple join-on-node is a set of logic which tells you when *not* to join data. OpenStreetMap already has the concept of a ‘layer’ — layers determine rendering order, but they also (essentially) determine the ordering on the earth. A bridge has a higher layer than a street, so they shouldn’t be merged topologically.

But what if they’re at the same layer, but not connected? (I can’t envision this, but suppose it happens.) The way I look at it, the answer is that this is a special case, and an additional ‘junction’ (or ‘not-junction’) node can be appropriate here.

Maybe I’m wrong. I’ve been wrong plenty of times before 🙂 However, this is how I’ve worked so far, and it’s worked out for the data I’m working with. I know that Frederik and I disagree on it — and that’s fine. Brings vitality to the discussion. 🙂 It’s good to see others so interested in solving the succinctly stated problems brought to the fore by the paper he has put together with Jochen (available from remote.org).

There’s an additional concern, stated by Steve on the list:

And yes you’re right, topological is really useful since OSM is a wiki and we track changes in nodes. Otherwise moving an intersection of many roads would mean updating many linestrings not one node.

another mailing list post

This one is more interesting to me, because I feel like moving topology to the client actually turns editing a topological operation — after which, grouping the edits in the API should be simple. Updating multiple features at once is a ‘simple matter of programming’ that I can solve (given time/effort on my part). Certainly I’ve built a RESTful Feature API that supports this — but it’s Simple Feature-based, not topology based.

Does anyone have experience of creating GIS data from aerial imagery? This is the thing that I have the most experience with, after mapping out a decent sized city using Yahoo! Maps as a backdrop. What do you use in editing? What do you use for export? How does it all tie together?