Technical Ramblings

If someone sets up a way for me to obtain, nightly, the worldwide output of the getnodes function for the whole world, and the corresponding getlines call from those returned nodes (as described in the file dao.rb), before the 10th of June, I will donate $200 towards the OpenStreetMap project.

I will also, from this data, set up a rendered version of OpenStreetMap, in a tiled instance of OpenLayers, on hardware that can support a lot of traffic, and build code that will convert this data into a shapefile every night, to be used by whatever projects people want to use them for, releasing the shapefile under the same license as OSM.

Anselm Hook has offered to match that contribution.

Anyone else who is in, please comment here, and I will communicate it to the OpenStreetMap developers.

A Windows Live Local truck just drove down the street next to the park I’m on. Looked like it had a camera on top of it. Seems that Cambridge may be picking up some more content from Windows Live Local in the near future…

Was a brown Ford Explorer, with some kind of camera-type mount on the top. Sadly, I didn’t snap a picture before it turned the corner.

I’m going to be giving a lightning talk called ‘Geolocation with GSM Cells’ at O’Reilly’s Where 2.0 conference, on June 13th, 2006.

Depart: 12 June, Boston, 4.40 p.m. and arrive in San Jose at 8.05 p.m. (JetBlue flight #471)

Conference: 13th->14th, Where 2.0, Fairmont Hotel, San Jose. (I still am not exactly sure where I’m staying during this time period.)

Staying in San Francisco on the 15th->16th.

Return: 16 June, 10.35PM, Oakland, (JetBlue flight #476)

I hope to get to do a bunch of things in SF: I love that city, and I did not get nearly enough time there last time.

Anyone out that way, either for the conference, or just in the San Francisco area, that I should plan on meeting up with? I know that there’s a lot of West Coast hackers that I talk to regularly, but the West Coast is big 🙂 I’d love to meet some people I’ve only known online in person while I’ve got the chance.

A bunch of people have emailed or commented here with things that they think I should be checking out. They’re probably right. However, I’m short on time, and working on a project heavily at the moment, so if you’re waiting on an email from me, please be aware that I probably saw it, I just haven’t gotten around to actually answering everything yet.

I’ll get back to you soon, I promise.

About 6 months ago, when I was trying to kickstart the Open Guide to Boston project, I mapped out each of the stations on the subway lines in Boston by hand, by looking up the info on the MBTA website, and hand clicking on a satellite map, in an effort to avoid concerns of copyrighted material making it into a CC-licensed dataset.

I later learned several things that frustrated me, such as the fact that there’s no database protection in the United States like there is in the UK, so a collection of the location of T Stops is just a collection of facts, and is not copyrightable — although a particular representation of those may be copyrighted, for example, the display of them on a Google Map in a specific way.

However, this was not the most frustrating event. The most frustrating thing that happened with regard to all the manual labor I put into it is that there was a much easier way: The state government of Massachusetts posts all its data under an open license, allowing for free reuse (with source credit). This includes, amazingly enough, a full listing of all the T Stations, including their name, what line they’re in, and their geographic location.

Yeah. Those 6 hours I spent could have easily been compacted into one without a problem, with more accuracy, and with nicer results. There would have been no copyright concerns, and I could have done it entirely with free software.

So why didn’t I?

I made the same mistake that many neogeographers are making, over and over again: Thinking I was doing something new. The idea that there might be information out there which could make my life easier never crossed my mind. Researching the standard GIS formats, and how to work with them, meant nothing to me: I seriously thought ESRI Shapefiles were a proprietary format that no one would know how to do anything with. Hell, I took at look at them at one point, and couldn’t figure them out because they didn’t return anything useful via `cat`.

A lesson here? Learn the tools of your trade. Every neogeographer should take a lesson in what ‘ogrinfo’ does. Look up information from your local planning board: See if they post files online. See what data is available for you to play with. See if maybe you can save yourself a bit of work by taking what already exists, and not doing the same work by hand. Because there’s a lot of geographic information out there, and recreating it all yourself is not really worth the effort.

GIS is a big field. Neogeography is the cutting edge of it, but right now so much of it is about reinvention of the wheel. There are solved problems that keep getting resolved with every new web mapping API. Learn from others mistakes, and use what they provide to you, and you may end up saving yourself a lot of work. I know I could have.

Summary of Geomancers Meeting:

I showed up late, as usual, but luckily Allan was there early to catch
people as they came in. Attendance peaked about about 9 — enough to
fill two tables of people. The meeting examined and discussed the
Camberville Greenmap, being published by Jerrad Pierce, and the various
things that can be done to make it more interesting and interactive:

* Web based format (mapserver/WMS/tiling something or other)
* Linking locations to the Open Guide to Boston or a similar site —
allowing people to click on something and finding out more about it,
and possibly adding their own information to it
* Allowing display of more information, since you don’t have to
overwhelm people with everything at once like you do on paper:
various layers let you put the pieces together to build up your own
interesting greenmap

There was some discussion of how the data was gathered, and about the
MrSID format that the aerial photography from MassGIS is in (mostly me
whining about it). There was discussion of some of the symbol choices —
how a chevron facing in the direction of the road can be misleading and
make people feel like it’s indicating a direction. All in all, the
greenmap is looking really cool, and I’m looking forward to getting a
version of it online and seeing what more can be done with it.

After my computer got on the network, we were able to talk about the
Boston Freemap that Schuyler and I built, some of the issues we were
having with it, and how to solve them. There was discussion of the
problems with using a WMS for the soure of a tiled clientside WMS
browser: labels intersecting tile boundaries, as well as roads or road
labels being drawn at weird angles. We also discussed (as was mentioned
on the list) the various tradeoffs of using different image formats,
some of which comes down to the fact that mapserver doesn’t quite do
what it would be nice for it to do.

There was some discussion of the work that went into the mumbai freemap
project: hand-digitization of vector information obtained from the local
planning board, and the attempts to convert said data into a reasonable
projection which would be overlaid on top of landsat satellite imagery,
something that’s not typically done.

There was some discussion of the Platial platform, what works about it,
what doesn’t, where the value is, and how it can be used for the Real
Estate field. Apparently, the housing market statewide is in a serious
decline: we’re passing the peak of the bubble, and many houses are
selling at 20-30% below their assessed value. This is a having a huge
impact on the real estate industry, and is something that may be able to
be alleviated in part by the sharing of local information: turn your
entire block into a cheerleader about a location. They want the value of
your house to be higher so that the value of *their* house can be
higher, and teh local view is a view that no realtor or anyone else can
give you but the people who live there.

There was also discussion of various ideas about what you could do with
the various data out there from a geographic point of view: the ability
to churn out a ‘local rap-sheet’ would be a nice idea for a startup
targetted towards Buying agents for real estate.

We’ll probably do another gathering in a couple weeks: Possibly in
someplace quieter, and maybe not on a friday evening to attract a
different crowd. The Muddy Charles is fine at 5, but towards 7 you
really have to start shouting to get yourself heard, and I’d like to go
someplace with food for a change at some point, since I don’t drink beer
🙂 Any recommendations or suggestions for date/time/place would be
welcome!

Links below are what was left in my browser history, or what was left
behind, and what I could remember. Feel free to add to it.

Link-based Summary of meeting:
* Greenmap
* Mumbai Freemap
* Boston Freemap
* Open Guide to Boston
* Platial
* MLS Listing
* Lowell Deeds Blog
* Boston World66
* World Flooding Map
* MassGIS Raster imagery

One of the problems I’m running into with mapserver right now is related to its rendering of LINE elements which are wider than one as they run into tile boundaries at acute angles. It seems that mapserver is drawing the centerlines for these elements up to the side of the image — but in cases where a line is approaching a boundary at an acute angle, this means that the ‘outer’ edge of the rendered line stops away from the edge.

In non anti-aliased lines, this is less visible as a problem (especially if you’re not looking at the images as tiles) because the lines just stop — and visually, it’s hard to tell if it’s at the image edge. However, it becomes very obvious in cases where anti-aliasing is on because the edges of the left and right boundary are ‘tied’ together by a curving, anti-alised line: resulting in a bubbled look at tile boundaries.

I’m not sure if this is a known bug, or something that other people have run into: I’m mostly recording it here so that I have a description of it.

Right now, it seems like the workarounds are:
* Use thinner lines (so it’s less visible)
* Don’t have roads near boundaries at acute angles — although there’s not much I can do about this one!

I’m not busy, and have some interesting stuff to talk about this Friday, so I’ll be leading a meeting of the Geomancers Geo Interest group this Friday, May 5th, from 5pm-7pm at the Muddy Charles Pub, located just inside 142 Memorial Drive in Cambridge.

This time, I’ll try to actually be on time 🙂

Some things I’m interested in showing off:

* A web map of Boston put together from MassGIS sources
* Some work done on Openstreetmap map data to create a usable web mapping service out of the monthly data exports
* Talking about what I’ll be presenting at Where 2.0
* A demo of http://www.wayfinderearth.com/ on my cell phone

Anyone is welcome to come by: Bring laptops, and you can get online via MIT’s network, bring cash (no cards!) and you can grab a drink — beer or soda.

I’ll try and grab us two tables, since last time we kind of overflowed the one we had.

(Note that I’m perfectly happy to have other people arrange these meetings: depending solely on me as admin contact is probably flakey at best. I still haven’t heard what time is best for other people, so until I do I’m just going to send ou t these emails when I have nifty things to show off and time to show them off in.)

Mapping is fun. Picking out the way things should look, when information should be displayed, what info should be on a map — it’s pretty nifty. Looking at the various resolutions of data, making rendering decisions based on time and quality of data needed at that level… it’s nifty.

Schuyler and I have been working with the Openstreetmap data, trying to figure out how to make a fast WMS server available. Seems like we’re getting pretty close now: generating various levels of polyline simplification ranging in degree, giving a set of 5 layers to be used at scales ranging from above 3 million all the way down to 20,000 without significant loss of data, then displaying the full quality data below that.

GRASS is being used to create the polylines and clean the data, and I’m using qgis to view the different layers. qgis, btw, is a great little GIS program for testing out various layer files you might find on the intarweb. It’s a bit slow for huge (dozens of thousands) numbers of polylines, but for most purposes, it’s great, has nice zooming features, and lets you check out some of the fields that are stored in a file without having to resort to things like ogrinfo.

All in all, I like this mapping stuff. Now, I need to figure out how to make money doing this.