Technical Ramblings

This week, I had the oppourtunity to work with Schuyler in writing stored procedures in Postgres/PostGIS for the first time.

At first, we were writing in plpgsql, but found it didn’t suit our needs… and switched to Python.

Yes, Python. Our database now has stored procedures which decode a cPickle pickle structure from a column (go go unstructured data) and return the output of a key/value pair (based on the key).

It was a very weird thing to see this actually work.

So, who should I be on the lookout for at Where 2.0?

As a follow up to sgillies’ post on Feature Paging, I’d just like to say that I was thinking of this a couple days ago for FeatureServer, and implemented it last night for the datasources that FeatureServer provides for which it makes sense. (The WFS DataSource is one of the ones that doesn’t.) The implementation is pretty immature — for everything other than postgis, it’s just a ‘skip’ counter, but it’s there.

(Note that I did this before I even saw posts about it — I think this is just more evidence that all the things we’re talking about here are well understood and not novel, though it is interesting that we’re all implementing them at the same time.)

So now you can do:

FeatureServer/layername/all.georss?bbox=-180,-90,0,0&startfeature=10&maxfeatures=10

And on a PostGIS layer, it will translate into a LIMIT, OFFSET block, and in a DBM layer, it will just iterate and not do anything, and in OGR, it will iterate and skip over the values that it doesn’t care about.

I think that’s cool, anyway.

More followup from the list: It seems that there is a reason (in addition to routing) for topological behavior: editing.

If you have a junction described in “features”, what you get are two roads that somehow happen to meet in one place (two lines crossing, or meeting). Right? In terms of storage, both lines will usually have their own coordinates in the data base; the information that they intersect, or meet, is not there explicitly, it just comes from the maths applied when drawing them, and if your database has spatial support you can ask the database for the point where they intersect.

However if someone wants to move that intersection, he will have to edit each feature separately – if you move one road away from the junction, then the junction is no more.

(From the mailing list.)

It’s a valid point, and not one that should be easily tossed aside. However, I’m not arguing against topology in all cases: I think that this is the exact kind of thing for which topology should be used — in the client.

If you have two nodes that are on top of each other, they should merge together. When you drag one, any feature that includes that node should be updated. When things don’t actually intersect, they should be separated. The canonical example is a bridge-over-river situation: If I want to draw a bridge over a river, and need two nodes in the exact same place, how do I do that without joining them?

The next piece needed beyond simple join-on-node is a set of logic which tells you when *not* to join data. OpenStreetMap already has the concept of a ‘layer’ — layers determine rendering order, but they also (essentially) determine the ordering on the earth. A bridge has a higher layer than a street, so they shouldn’t be merged topologically.

But what if they’re at the same layer, but not connected? (I can’t envision this, but suppose it happens.) The way I look at it, the answer is that this is a special case, and an additional ‘junction’ (or ‘not-junction’) node can be appropriate here.

Maybe I’m wrong. I’ve been wrong plenty of times before 🙂 However, this is how I’ve worked so far, and it’s worked out for the data I’m working with. I know that Frederik and I disagree on it — and that’s fine. Brings vitality to the discussion. 🙂 It’s good to see others so interested in solving the succinctly stated problems brought to the fore by the paper he has put together with Jochen (available from remote.org).

There’s an additional concern, stated by Steve on the list:

And yes you’re right, topological is really useful since OSM is a wiki and we track changes in nodes. Otherwise moving an intersection of many roads would mean updating many linestrings not one node.

another mailing list post

This one is more interesting to me, because I feel like moving topology to the client actually turns editing a topological operation — after which, grouping the edits in the API should be simple. Updating multiple features at once is a ‘simple matter of programming’ that I can solve (given time/effort on my part). Certainly I’ve built a RESTful Feature API that supports this — but it’s Simple Feature-based, not topology based.

Does anyone have experience of creating GIS data from aerial imagery? This is the thing that I have the most experience with, after mapping out a decent sized city using Yahoo! Maps as a backdrop. What do you use in editing? What do you use for export? How does it all tie together?

Lars Aronsson on the OSM list said:

The result from this is Steve’s current data model and the fact that the rest of us accept this as a viable solution. Those who don’t, because they know more of GIS, like Christopher Schmidt, are repelled by everything they find under the hood of OSM.

Part of my response:

I’m actually not repelled by everything. It’s simply a different choice than I would make. Specifically:

• OSM uses topology as its base storage. Topology is good for making graphs, which is important when you need to do routing. For this reason, (it seems to me) that OSM was built towards the goal of creating driving directions. Great goal for a project to have. However,
• Most geo-software uses Simple Features — not topology — for handling data. The result is very different — Simple Features are designed for making maps. If you’d like evidence, look at how the mapnik maps are built: the topology is turned into simple features, and stored in PostGIS. My MapServer demos just under a year ago worked the same way.

The difference to me is simple:

If I want to drawn an OSM feature on a map, I have to fetch a large number of pieces of data fromm the API individually, and combine them to create a geographic feature.

Example:

Way ID 4213747:

• 1 way.
• 21 segments.
• 22 nodes.

So, to visualize this one way, I have to make 44 fetches to the API.

Now, if I switch to a simple features model:

JSON Simple Feature output of same geometry

I’m given a geometry (“Line”), list of coordinates, and list of properties. (This is JSON output: you can also see it as html by adding ‘.html’ to the end, or as atom by adding ‘.atom’ to the end.)

“Line” can also be “Polygon”, or “Point”. (Or “MULTIPOLYGON”, etc., though FeatureServer doesn’t support those.)

This is one fetch. I can now draw the feature. I can also query for other features which have the same name, and get the information for those, too:

Attribute query on name
This shows me that there is also a feature, ID 4213746, which has the same name. I can draw all these features on a map with the output of one query.

In OSM, that would be 88. 88 queries to the API, just so I can display two features — not to mention the fact that at the moment, there’s no way to query attributes quickly.
If there was a strong reason for storing topology — that is, if OSM was really not about maps, and was instead about making driving directions — this could make sense. In fact, it may make sense: I may have a serious lack of understanding about how the project data is being used. However, I think that the most common usage of OSM is *making maps* —
in fact, Steve even backs me up on this, in his post:

OpenStreetMap is driven by this principle that we just want a fscking map.

Topology makes a graph, not a map. This is the reason why I’m in favor of a simple features-based data model: Features-based models are what you use for making maps. Topology is what you use for doing analysis.

The upshot of this? The tools to make topology out of simple features *already exists*: GRASS will do it. PostGIS + pgdijkstra will do it. Any application out there which needs topology knows how to get it, because mapping data is almost always distributed as something that isn’t topological. These are all technical problems: mapping back and forth is possible. The best way to do it is hard to determine, but the OSM project has no shortage of hard-working participants, and I’m sure that over time we will see easier to use UIs and editors for editing and creating data.

OpenLayers now has its own blog.

Thank the creation of Planet OSGeo for the final inspiration to actually put it together.

I finally got fed up with Planet GeoSpatial this weekend. Too much Google, too many formatting mess ups, and in general, too much crap. Although I appreciate that those who are working in the GeoSpatial space have a large interest in ESRI, Google Earth and Maps, and the general ‘state of the industry’, my target interests are much smaller. I just want to know about what’s going on in the world which affects Open Source GeoSpatial software: I wanted what is essentially “Planet OSGeo”, rather than “Planet Geo”.

(Note that this is a commentary mostly on my specific interests, rather than on Planet GeoSpatial, which James Fee has done a wonderful job on maintaining for the wider target audience he has.)

To that end, and with Sean Gillies’ excellent recommendation of Venus (omg, a version of Planet that *works*?), I’ve set up Planet OSGeo, a collection of Open Source GIS blogs.

Of note, I’ve not included several topics here, even though they are at least tangentially related to Open Source Geo:

• GeoRSS. There are a half dozen people who are regularly blogging about GeoRSS — not least, the GeoRSS blog. However, GeoRSS is not in and of itself “software”, which is my target interest, and I think the primary field in which OSGeo has thus far expressed an interest.
• OpenStreetMap. Although collection of Open Geodata is within the realm of OSGeo, at the moment I’m targeting software, and most of OSM’s blogging is not about software development.

(I’m happy to take feedback on my choices — or suggestions for more blogs!)

Notably lacking in this Planet:

• Blog for OpenLayers. Thus far, I’ve not set up a seperate infrastructure for OpenLayers blogging, sticking instead to the MetaCarta Labs blog. I think the time has come to grow out of that, and move into some OpenLayers infrastructure for blogging.
• Blog for GDAL. The library at the base of much of the Geo software on the web doesn’t have an RSS feed — for either announcements or general project discussion. Some of this is probably representative of the stability of the project: certainly after as many years as GDAL has been around, there’s limited content in terms of “Rapid Development” that many other tools like web mapping clients are still undergoing. Still, an announcements log with an RSS feed would be cool.
• MapServer. MapServer doesn’t seem to have a blog or RSS feed out of its website that provides interesting announcement-style updates, which would be good to see.

I think the things I see as ‘missing’ clearly demonstrate my bias in development and usage of tools — so I’m sure more people can point out what else I’m missing.

Looking forward to any feedback you might have.

Made a ‘screencast’ (at least, I think that’s the right buzzword) of OpenLayers + QGIS today:

Mapping Your Data: Using QGIS, OpenLayers, and MapServer to serve open data

Tried uploading it to YouTube: the resolution was shrunk to the point of being unsable. Tried uploading to Google Video: despite the upload completing 11 hours ago, the upload is still ‘processing’. So I went through with ffmpeg and converted it to Flash myself, and set up the environment for playing it on openlayers.org.

To create the video itself, I used:

• Snapz Pro X (Shareware: $60)
• ffmpegx (Shareware: $15)
• Flash Video Player (Free for Non-commercial use: $15 commercial license)

In the video, I used TextMate (Shareware), QGIS (Open Source + Free), Firefox, and OpenLayers.

(There is no audio in the presentation, because I felt like my voice would be a distraction. I suppose next time I should put some fancy soundtrack in place.)

At BarCampBoston today, I got to talking to the OLPC folks. I was demoing our new vector editing support in OpenLayers, and talking to them about it. They wanted to see if it would run on their laptop, and we got an error that SVG wasn’t supported in their browser. After a bit of effort after getting home tonight, I was able to find out how to check for SVG 1.0 instead of 1.1 — after modifying the OpenLayers code to do a 1.0 check instead of a 1.1 check, I got a confirmation that vector display works on the One Laptop Per Child laptop! Woot. Filed a ticket with a patch for OpenLayers: 540, just need to get someone to review it and check it in.
So, the next step is to help the OLPC folks get up and running with OpenLayers. 🙂

So, last week was an OpenLayers hack week. One of the things that we did was make adding support for new vector formats trivial. Instead of modifying several parts of the code, you only need to create two functions: a ‘read’, which takes a set of data — XML, strings, Javascript object, what have you — and returns a list of OpenLayers.Feature.Vector objects, and a ‘write’ which does the reverse — takes a list of objects and returns a string or object or XML.

To prove this, I set out to write some additional vector format support last night. I decided to add one read, and one write.

• Read: KML. I added support for KML point display in about 20 minutes, including the time to find data and write a demo HTML page loading some example data. Adding LineString support was another 15 minutes.
• Write: GeoRSS. Support for writing georss:simple points, lines, and polygons was simple… once I found data. I asked for a live example, and was unfortunately unable to find any valid line data outside the GeoRSS website, so I just generated something that was as close as I could come to the examples. I’m lazy, so the export is just RSS 2, and I’m sure that someone will come along and criticize it, but that’s one of the benefits of Open Source: Anyone can offer up a patch. Time from when I created the file stub to when I committed the code was 27 minutes, again, including a demo.

Altogether, the Format support in the new OpenLayers is pretty cool. Because of the way it’s built, I can even do something that is pretty damn ridiculous: Import KML, and export GeoRSS (or GML), all from the browser. Certainly, this is an incredibly crazy thing to do, but OpenLayers is a pretty crazy project.

I’m convinced that there’s nothing in the code that would make it difficult for someone who’s comfortable working with Javascript to write support for any simple-to-parse format. Now, to get the code back to trunk and get the patches rolling in.