Technical Ramblings

I have a love/hate relationship with Yahoo!’s mapping API. It’s lovely that Yahoo! believes, unlike Google and other mapping providers, that their satellite data is a suitable base layer to use for derivation of vectors. This openness really is good to see — they win big points from me in this regard. (Google, on the other hand, is happy to have you give them data against their satellite imagery, but letting you actually have it back is against the Terms of Service.)

However, the Yahoo! Maps AJAX API has never gotten much love. I think that a preference for flash has always existed in the Yahoo! world; iirc, their original API was Flash.

However, I realized today that this tendancy to leave the AJAX API in the dust has resulted in something that seriously affects me: The Yahoo! maps AJAX API uses a different set of tiles, which has two fewer zoom levels available in it:

AJAX Maps: most zoomed in	Flash Maps: Most zoomed in

For the new OpenStreetMap editor I’m working on, this is a *serious* difference: although the information actually available in these tiles isn’t *that* much higher, it allows the user to extract more information by getting in a bit more, and to be more precise in placement of objects when using Yahoo! as a basemap.

Although it would be relatively easy to rip the tiles out, and create an OpenLayers Layer class that loaded them directly, this violates the Yahoo! Terms of Use. This is understandable, but unfortunate, because it means I can’t solve the problem with my own code.

What I would really love to see is more providers creating a more friendly way of accessing their tiles. I understand the need for counting of accesses, and the need for copyright notifications. If an API were published, that allowed you to:

• Fetch a copyright notice for a given area, possibly also returning a temporary token to use
• Following that, fetch tiles to fill that area
• Require users to copyright notice in such a way as to make Yahoo! and their providers happy

This would allow for building a layer into OpenLayers which complied with this, without depending on Yahoo! to write a Javascript layer that did these things for me.

Now, it’s understandable that this doesn’t happen — having the client out of control of Yahoo! means that they can’t *enforce* that the copyright is displayed prominently, as they are able to (to some extent) with their API. However, I think that this type of API would allow more innovation, and possibly even a *more* prominent placement for Yahoo’s copyrights and notices. For example, in many mapping apps, the bottom inch of the map is not seen much by the users. If there was an API to get text to display, then an application could display the text in a more prominent location, rather than burying it under many markers or other pieces of text that might overlap it.

In the short term, all I really wish was that the AJAX API used the apparently-newer set of satellite tiles that the Flash API appears to have access to. I think the fact that this isn’t currently possible leads to an alternative access pattern for tiles, one which may make more sense in the long run, where tiles can be used by an application without necessarily running in the constrained Javascript API that these providers have the ability to write. And of course, if you want to provide your users with a ‘default’ API to use, you can always use OpenLayers, and extend it to include your own extensions…

JOSM is the ‘advanced’ OpenStreetMap editor, used by most technical users of OpenStreetMap. It is written in Java, but despite that 😉 it works reasonably well. Jokes about Java aside, JOSM is an excellent example of the type of ‘advanced’ editor that most GIS professionals would feel comfortable with* after some work understanding OSM: it has familiar interfaces for drawing lines, displaying and editing attributes, etc.

It works well on the eeepc: Java comes pre-installed, so it’s simple to get started; just download josm-latest.jar from the josm homepage and run (from a terminal) ‘java -jar josm-latest.jar’. You’ll be presented with a message that you can’t read, but it’s not really important. (The reason you can’t read it is that the message is apparently laid out with ‘fixed’ border sizes of ~350px… meaning the message only has about 100px across on the eee’s screen. “Oops.”)

First, we’ll set up the interface so it has more room for attribute values on the right hand side, by hiding the command history (Alt-O) and the selection list (Alt-E). You can bring these back at any time using the same shortcuts.

Next, we’ll download some data. The easiest way to download data for an area you’re interested in is to navigate one of the ‘slippy maps’ that OpenStreetMap has: my personal preference is to use Information Freeway, since it has a full page map. To see the area I’ve been mapping in, check out Grand Cayman; you can use this URL in JOSM by copying it, then selecting “File”, “Download From OSM”.

Navigating the map once you’ve downloaded it isn’t too difficult: by default, you’re in ‘zoom’ mode, which will do a ‘rubber band zoom’ (As we call it in OpenLayers) by default. You can switch modes using hte keyboard: simply hit ‘s’ (for select), ‘a’, for add, ‘d’, for delete, or ‘z’ to go back to the default ‘zoom’ mode. Moving the map can be done with Right click->Drag, and the arrow keys can also be used for navigation if you hold down ‘Control’. Zooming in and out can be done with the ‘scrollwheel’, which on the Eee is the right hand side of the trackpad.

In general, the editing experience of JOSM on the eee is actually significantly better than my mac. The reason for this is simple — the Mac doesn’t have a right click, which means that navigating by dragging the map doesn’t work. Additionally, one of the ways to get information about nodes near your cursor is the middle click. On the Eee, this is as simple as tapping two fingers on the trackpad.

That said, there are some significantly lacking aspects in using JOSM on such a small screen that don’t come up on the Mac:

• Toolbar is too tall — can’t select buttons towards the bottom of the list
• Preferences dialog is too small: can’t see the ‘okay’ button, so can’t enable plugins (one of the coolest aspects of JOSM)
• Inability to resize right hand side control panels: this means that the ‘layer switcher’ panel is as tall as the tags panel, which isn’t really neccesary for me. Similarly, ‘relations’ (Which are seldom used, at least at this point) share equal play time with tags/attributes, which is somewhat unneccesary

All in all, JOSM doesn’t work out too bad on the EeePC, but the lack of plugins due to the preferences panel being ‘too tall’ is somewhat annoying, and I haven’t yet figured out how to get around it. It’s possible that installing the plugins manually will work okay, but it’s been so long since I’ve installed them from within JOSM that I don’t even know how anymore!

* Of course, many GIS professionals working with OSM are going to have a steep learning curve, due to the nature of OSM’s data model: the majority of the vectorization software (at least, the stuff that I’ve seen) works with features, whereas OSM is topological, which makes interacting with the data a very different experience.

Got my EeePC. It’s so tiny! I love it already; I haven’t even pulled out the macbook tonight. (I’m sure this won’t keep up forever, since there are some things I really can’t do on a screen this small.) I did some playing with OpenStreetMap editing with Potlatch (painful, but not horribly so), figured out how to get a terminal up and running (ctrl-alt-t; handy), got subversion installed after setting up some extra repositories based on instructions in the eeeuser wiki, etc.

I will want to see if I can start building some tools targeted at the small screen resolution in OpenLayers or what have you, so that I can edit maps more effectively, and do other things like hook up GPS traces. I’m leaning more and more towards building myself some nice custom UIs for editing OSM, just to figure out how to do things that work well on my hacky platforms.

I have no plans (for the time being) to switch away from the ‘friendly’ interface, which seems (to me) to work just peachy keen. I am happy with my decision: even if I only use this thing intermittently from today forward, it’s a worthwhile and nifty toy to have, and I’m liking it a lot.

OpenStreetMap display of Massachusetts. (This will soon enter the main OSM caches — I’m just impatient.)

Thanks to Brandon Martin-Anderson, one of MetaCarta Labs’s new minions, who helped me get started by writing MassGIS shapefile -> OSM file converters, and to all the great people who have made the OSM Server and code so much more usable since the last time I tried to do this.

Andreas posted earlier today with a followup on the SLD Rules front: Though I had consigned myself to expressions, he had not, and went on to create a very nice looking rules system which supports the full range of SLD operators.

More information is available in his post to the mailing list: the code is in his sandbox (though this URL will move once it moves to trunk).

Three rule types are used to build up rules: comparison rules, for comparing values against feature attributes, FeatureID rules, for limiting by fid, and Logical rules, which group the other rule types together. Logical rules can be AND, OR, or NOT.

One key benefit to this — which I had not thought about until Andreas brought it up — is the ability to serialize these rules back out to SLD. It’s possible to use something like yacc/lexx (C, etc.) or eval() to evaluate an expression — but in Javascript, at least, it’s hard to turn that expression back into rules. This saves us that problem, and means we can serialize out SLD from user-created styling rules — clearing the way for the in-browser SLD editor that many people have been talking about for a long time.

So, as a result of Custom Projection 6: Google Projection, I got pinged by SharpGIS on IRC, and told that the projection I had there was wrong.

He helped me understand what he was using, and I let him know it still wasn’t helping me. So I set up a Virtual Earth + TileCache demo, and he helped me find the missing piece, in the proj4 FAQ:

In this case, and many other cases using spherical projections, the desired approach is to actually treat the lat/long locations on the sphere as if they were on WGS84 without any adjustments when using them for converting to other coordinate systems. The solution is to “trick” PROJ.4 into applying no change to the lat/long values when going to (and through) WGS84. This can be accomplished by asking PROJ to use a null grid shift file for switching from your spherical lat/long coordinates to WGS84.

+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs

The key here was the +nadgrids=@null, which I’d never seen anywhere before.

The end result?

• Fixing my OSM Tiles
• WMS and TMS overlaid in OpenLayers, as described in an email
• A slightly better understanding of what I didn’t understand for so long.

So. Nice. To not feel lost anymore.

Three cheers for sharpgis! Hooray hooray hooray!

More followup from the list: It seems that there is a reason (in addition to routing) for topological behavior: editing.

If you have a junction described in “features”, what you get are two roads that somehow happen to meet in one place (two lines crossing, or meeting). Right? In terms of storage, both lines will usually have their own coordinates in the data base; the information that they intersect, or meet, is not there explicitly, it just comes from the maths applied when drawing them, and if your database has spatial support you can ask the database for the point where they intersect.

However if someone wants to move that intersection, he will have to edit each feature separately – if you move one road away from the junction, then the junction is no more.

(From the mailing list.)

It’s a valid point, and not one that should be easily tossed aside. However, I’m not arguing against topology in all cases: I think that this is the exact kind of thing for which topology should be used — in the client.

If you have two nodes that are on top of each other, they should merge together. When you drag one, any feature that includes that node should be updated. When things don’t actually intersect, they should be separated. The canonical example is a bridge-over-river situation: If I want to draw a bridge over a river, and need two nodes in the exact same place, how do I do that without joining them?

The next piece needed beyond simple join-on-node is a set of logic which tells you when *not* to join data. OpenStreetMap already has the concept of a ‘layer’ — layers determine rendering order, but they also (essentially) determine the ordering on the earth. A bridge has a higher layer than a street, so they shouldn’t be merged topologically.

But what if they’re at the same layer, but not connected? (I can’t envision this, but suppose it happens.) The way I look at it, the answer is that this is a special case, and an additional ‘junction’ (or ‘not-junction’) node can be appropriate here.

Maybe I’m wrong. I’ve been wrong plenty of times before 🙂 However, this is how I’ve worked so far, and it’s worked out for the data I’m working with. I know that Frederik and I disagree on it — and that’s fine. Brings vitality to the discussion. 🙂 It’s good to see others so interested in solving the succinctly stated problems brought to the fore by the paper he has put together with Jochen (available from remote.org).

There’s an additional concern, stated by Steve on the list:

And yes you’re right, topological is really useful since OSM is a wiki and we track changes in nodes. Otherwise moving an intersection of many roads would mean updating many linestrings not one node.

another mailing list post

This one is more interesting to me, because I feel like moving topology to the client actually turns editing a topological operation — after which, grouping the edits in the API should be simple. Updating multiple features at once is a ‘simple matter of programming’ that I can solve (given time/effort on my part). Certainly I’ve built a RESTful Feature API that supports this — but it’s Simple Feature-based, not topology based.

Does anyone have experience of creating GIS data from aerial imagery? This is the thing that I have the most experience with, after mapping out a decent sized city using Yahoo! Maps as a backdrop. What do you use in editing? What do you use for export? How does it all tie together?

Lars Aronsson on the OSM list said:

The result from this is Steve’s current data model and the fact that the rest of us accept this as a viable solution. Those who don’t, because they know more of GIS, like Christopher Schmidt, are repelled by everything they find under the hood of OSM.

Part of my response:

I’m actually not repelled by everything. It’s simply a different choice than I would make. Specifically:

• OSM uses topology as its base storage. Topology is good for making graphs, which is important when you need to do routing. For this reason, (it seems to me) that OSM was built towards the goal of creating driving directions. Great goal for a project to have. However,
• Most geo-software uses Simple Features — not topology — for handling data. The result is very different — Simple Features are designed for making maps. If you’d like evidence, look at how the mapnik maps are built: the topology is turned into simple features, and stored in PostGIS. My MapServer demos just under a year ago worked the same way.

The difference to me is simple:

If I want to drawn an OSM feature on a map, I have to fetch a large number of pieces of data fromm the API individually, and combine them to create a geographic feature.

Example:

Way ID 4213747:

• 1 way.
• 21 segments.
• 22 nodes.

So, to visualize this one way, I have to make 44 fetches to the API.

Now, if I switch to a simple features model:

JSON Simple Feature output of same geometry

I’m given a geometry (“Line”), list of coordinates, and list of properties. (This is JSON output: you can also see it as html by adding ‘.html’ to the end, or as atom by adding ‘.atom’ to the end.)

“Line” can also be “Polygon”, or “Point”. (Or “MULTIPOLYGON”, etc., though FeatureServer doesn’t support those.)

This is one fetch. I can now draw the feature. I can also query for other features which have the same name, and get the information for those, too:

Attribute query on name
This shows me that there is also a feature, ID 4213746, which has the same name. I can draw all these features on a map with the output of one query.

In OSM, that would be 88. 88 queries to the API, just so I can display two features — not to mention the fact that at the moment, there’s no way to query attributes quickly.
If there was a strong reason for storing topology — that is, if OSM was really not about maps, and was instead about making driving directions — this could make sense. In fact, it may make sense: I may have a serious lack of understanding about how the project data is being used. However, I think that the most common usage of OSM is *making maps* —
in fact, Steve even backs me up on this, in his post:

OpenStreetMap is driven by this principle that we just want a fscking map.

Topology makes a graph, not a map. This is the reason why I’m in favor of a simple features-based data model: Features-based models are what you use for making maps. Topology is what you use for doing analysis.

The upshot of this? The tools to make topology out of simple features *already exists*: GRASS will do it. PostGIS + pgdijkstra will do it. Any application out there which needs topology knows how to get it, because mapping data is almost always distributed as something that isn’t topological. These are all technical problems: mapping back and forth is possible. The best way to do it is hard to determine, but the OSM project has no shortage of hard-working participants, and I’m sure that over time we will see easier to use UIs and editors for editing and creating data.