Technical Ramblings

So, I’ve created a lot of maps with Open Source tools. In fact, I put together my first online map with MapServer and a not-yet-released OpenLayers about a year ago this month. (Before that, I’d never actually done anything with geospatial tools.)

I usually stick to my local Open Source world — reading Planet OSGeo, rather than PlanetGS — but every now and then, I wander across the line to see what the rest of the world is using.

What I find is a hell of a lot of Arc* names, and no real understanding of what benefit they provide. Although I understand that Analysis is difficult with existing open source tools — not that it’s necessarily better in proprietary tools —  it seems that much of what people want to do with software like ArcGIS is just make a map.

So, what’s the benefit to using these tools? What doesn’t open source do… and how do we make them do them?

This week, I had the oppourtunity to work with Schuyler in writing stored procedures in Postgres/PostGIS for the first time.

At first, we were writing in plpgsql, but found it didn’t suit our needs… and switched to Python.

Yes, Python. Our database now has stored procedures which decode a cPickle pickle structure from a column (go go unstructured data) and return the output of a key/value pair (based on the key).

It was a very weird thing to see this actually work.

So, who should I be on the lookout for at Where 2.0?

As a follow up to sgillies’ post on Feature Paging, I’d just like to say that I was thinking of this a couple days ago for FeatureServer, and implemented it last night for the datasources that FeatureServer provides for which it makes sense. (The WFS DataSource is one of the ones that doesn’t.) The implementation is pretty immature — for everything other than postgis, it’s just a ‘skip’ counter, but it’s there.

(Note that I did this before I even saw posts about it — I think this is just more evidence that all the things we’re talking about here are well understood and not novel, though it is interesting that we’re all implementing them at the same time.)

So now you can do:

FeatureServer/layername/all.georss?bbox=-180,-90,0,0&startfeature=10&maxfeatures=10

And on a PostGIS layer, it will translate into a LIMIT, OFFSET block, and in a DBM layer, it will just iterate and not do anything, and in OGR, it will iterate and skip over the values that it doesn’t care about.

I think that’s cool, anyway.

More followup from the list: It seems that there is a reason (in addition to routing) for topological behavior: editing.

If you have a junction described in “features”, what you get are two roads that somehow happen to meet in one place (two lines crossing, or meeting). Right? In terms of storage, both lines will usually have their own coordinates in the data base; the information that they intersect, or meet, is not there explicitly, it just comes from the maths applied when drawing them, and if your database has spatial support you can ask the database for the point where they intersect.

However if someone wants to move that intersection, he will have to edit each feature separately – if you move one road away from the junction, then the junction is no more.

(From the mailing list.)

It’s a valid point, and not one that should be easily tossed aside. However, I’m not arguing against topology in all cases: I think that this is the exact kind of thing for which topology should be used — in the client.

If you have two nodes that are on top of each other, they should merge together. When you drag one, any feature that includes that node should be updated. When things don’t actually intersect, they should be separated. The canonical example is a bridge-over-river situation: If I want to draw a bridge over a river, and need two nodes in the exact same place, how do I do that without joining them?

The next piece needed beyond simple join-on-node is a set of logic which tells you when *not* to join data. OpenStreetMap already has the concept of a ‘layer’ — layers determine rendering order, but they also (essentially) determine the ordering on the earth. A bridge has a higher layer than a street, so they shouldn’t be merged topologically.

But what if they’re at the same layer, but not connected? (I can’t envision this, but suppose it happens.) The way I look at it, the answer is that this is a special case, and an additional ‘junction’ (or ‘not-junction’) node can be appropriate here.

Maybe I’m wrong. I’ve been wrong plenty of times before 🙂 However, this is how I’ve worked so far, and it’s worked out for the data I’m working with. I know that Frederik and I disagree on it — and that’s fine. Brings vitality to the discussion. 🙂 It’s good to see others so interested in solving the succinctly stated problems brought to the fore by the paper he has put together with Jochen (available from remote.org).

There’s an additional concern, stated by Steve on the list:

And yes you’re right, topological is really useful since OSM is a wiki and we track changes in nodes. Otherwise moving an intersection of many roads would mean updating many linestrings not one node.

another mailing list post

This one is more interesting to me, because I feel like moving topology to the client actually turns editing a topological operation — after which, grouping the edits in the API should be simple. Updating multiple features at once is a ‘simple matter of programming’ that I can solve (given time/effort on my part). Certainly I’ve built a RESTful Feature API that supports this — but it’s Simple Feature-based, not topology based.

Does anyone have experience of creating GIS data from aerial imagery? This is the thing that I have the most experience with, after mapping out a decent sized city using Yahoo! Maps as a backdrop. What do you use in editing? What do you use for export? How does it all tie together?

Lars Aronsson on the OSM list said:

The result from this is Steve’s current data model and the fact that the rest of us accept this as a viable solution. Those who don’t, because they know more of GIS, like Christopher Schmidt, are repelled by everything they find under the hood of OSM.

Part of my response:

I’m actually not repelled by everything. It’s simply a different choice than I would make. Specifically:

• OSM uses topology as its base storage. Topology is good for making graphs, which is important when you need to do routing. For this reason, (it seems to me) that OSM was built towards the goal of creating driving directions. Great goal for a project to have. However,
• Most geo-software uses Simple Features — not topology — for handling data. The result is very different — Simple Features are designed for making maps. If you’d like evidence, look at how the mapnik maps are built: the topology is turned into simple features, and stored in PostGIS. My MapServer demos just under a year ago worked the same way.

The difference to me is simple:

If I want to drawn an OSM feature on a map, I have to fetch a large number of pieces of data fromm the API individually, and combine them to create a geographic feature.

Example:

Way ID 4213747:

• 1 way.
• 21 segments.
• 22 nodes.

So, to visualize this one way, I have to make 44 fetches to the API.

Now, if I switch to a simple features model:

JSON Simple Feature output of same geometry

I’m given a geometry (“Line”), list of coordinates, and list of properties. (This is JSON output: you can also see it as html by adding ‘.html’ to the end, or as atom by adding ‘.atom’ to the end.)

“Line” can also be “Polygon”, or “Point”. (Or “MULTIPOLYGON”, etc., though FeatureServer doesn’t support those.)

This is one fetch. I can now draw the feature. I can also query for other features which have the same name, and get the information for those, too:

Attribute query on name
This shows me that there is also a feature, ID 4213746, which has the same name. I can draw all these features on a map with the output of one query.

In OSM, that would be 88. 88 queries to the API, just so I can display two features — not to mention the fact that at the moment, there’s no way to query attributes quickly.
If there was a strong reason for storing topology — that is, if OSM was really not about maps, and was instead about making driving directions — this could make sense. In fact, it may make sense: I may have a serious lack of understanding about how the project data is being used. However, I think that the most common usage of OSM is *making maps* —
in fact, Steve even backs me up on this, in his post:

OpenStreetMap is driven by this principle that we just want a fscking map.

Topology makes a graph, not a map. This is the reason why I’m in favor of a simple features-based data model: Features-based models are what you use for making maps. Topology is what you use for doing analysis.

The upshot of this? The tools to make topology out of simple features *already exists*: GRASS will do it. PostGIS + pgdijkstra will do it. Any application out there which needs topology knows how to get it, because mapping data is almost always distributed as something that isn’t topological. These are all technical problems: mapping back and forth is possible. The best way to do it is hard to determine, but the OSM project has no shortage of hard-working participants, and I’m sure that over time we will see easier to use UIs and editors for editing and creating data.