Technical Ramblings

Wherecamp is the closest thing to a perfect conference going experience I have had in my history of going to conferences. The organizers — Ryan and Anselm — did a great job of pulling everything together, but I think that the single entity most responsible for making this happen was Google, and their representatives on the ground, Nate and Dusty.

Key things to good unconferences:

• Constant stream of food. Geeks like to eat, and eating keeps us happy, healthy, and wise. Snacks are great, both healthy and unhealthy, and drinks and so on are also good. I think this, more than anything else, is the kind of thing that a well-funded social event should try to have. The meals at Google were also great, and having the space we did for serving was super useful for avoiding frustration at lines. In general, for the number of people who were present, Google’s facilties were significantly better prepared for serving people than my experience at Where 2.0 and FOSS4G events.
• Big open space for organization, small cozy meeting spaces for sessions. One thing that repeatedly came up was that the conference spaces we were in for sessions were full — and, believe it or not, I find sitting on a makeshift chair in the back of the room *great*. The participation in the sessions was better than at any other unconference I’ve been too, because of the close-knit situation. Occasionally things were a bit too small — the Processing talk was one example — but in general, it was a good thing to be in close quarters.
• Great wireless. Coverage of the entire space was perfect, including outside, and given that there were 300 people sitting around banging on it, that was far more impressive. I’ve seen networks fall apart under significantly less load, even at Barcamp Boston 2, at MIT.
• Open Space: The central auditorium and outdoor areas were great places to chill out and enjoy your time. Not having to be stuck in a hallway, or running back and forth between sessions, was a wonderful end result, and the tables/chairs/etc. setup was just great.

The last thing that can make or break any conference is the people, and I couldn’t have asked for any better. I met and hung out with a bunch of Python Geo hackers — more so than I ever have in the past. We stayed up late, hacked all night (in my case, literally) and resolved a ton of issues. I got to hang out with some of the people that I really enjoy spending time around, and meet some new ones.

Thanks to all of you who made this weekend totally worth the trip. Special thanks to the organizers, and Google for being such a great host.

Next up is FOSS4G, I hope; I hope that I can take what I learned here and make an effort to turn FOSS4G into (at least partially) the event that Wherecamp was so successfully.

Notes about Processing, from processing.org, and other related things.

“Drawing images is not about telling which pixels to turn on or off, it’s about telling to draw lines.”

In addition to processing being pretty, you can draw the entire map into any unit you want. Drew a bunch of lines out in state plane projected unit system, and then cropped image.

Drawing all of the streets in King County plus all of transit lines

4d expansion of transit data — time out of the map on photo

Processing is great at generating static images; looping over a dataset and drawing lines is what processing is for.

Demoing a bunch of demos of processing from Tom Carden; cabspotting, travel time tube map, etc. etc.

Processing.js: Processing in JS, by John Resig. Uses “Processing.js is like developing Processing on a 400mhz celeron”

Source code to London Tube Map is available; Processing makes source code available by default, Google for “powered by processing” for examples.

Stamen typically does Prototype in Processing, final client in Flash.

Processing is Java; it should be possible to pull in Java libraries; can import .jar files, can drag in data (images, etc.); loading data is a one liner.

NodeBox: “Processing in Python”

Obsessing.org: Processing.js GUI.

Same processing source code in browser and applet using Processing.js

Context Free Art: here

The default mode of processing is:
* Setup
* Draw function that’s called as much as possible

Going to publish cabspotting example to wherecamp wiki

Brandon demos OpenStreetMap data rendered on the fly with processing data.

NodeBox demo using a springgraph. Proxmiity fixes of local bluetooth data.

Processing is 4-5 years old now

Collaboration is typically “Copy an example, create your own, keep going”

Tonight, starting around 7:30, there is a plan to descend on The Dubliner, in Noe Valley, San Francisco, for beer and hackery. If the plan changes, I’ll update twitter. If you want to join us, please:

• Comment here
• Email me (crschmidt@crschmidt.net)
• or Text or call me (603.264.2294)

I’m off to Wherecamp tomorrow. I’ll be in town and have no plans yet as of Friday, and I’ll be at wherecamp all weekend, planning to spend the night hacking at the Googleplex.

Who’s going to be there? Anyone interested in meeting up on Friday, or doing any hacking ahead of time or during?

So, a couple people have asked me why Google thinks that my site may harm your computer.

On Wednesday, I received an email from “Google Search Quality”:

We recently discovered that some of your pages can cause users to be infected with malicious software. We have begun showing a warning page to users who visit these pages by clicking a search result on Google.com.

I looked through, and sure enough, the links they offered were indeed ‘infected’: I’ve always used WordPress despite my knowledge of the fact that it has security exploits more often than I get around to fixing them (though I do try to keep up). They had has a small iframe included, which claimed to be ‘stats tracking’: Instead, there was Javascript included which, presumably, was malicious.

To the best of my knowledge, I solved this problem on Wednesday night, by removing the mal-links that were pointed out, and patching the security holes I could find fixes to in WordPress. (I just upgraded to WordPress 2.5; In the past, upgrading has been painful, but it wasn’t so bad this time, and there are not yet any known security holes for 2.5 that I’m aware of.)

All in all, not a bad thing: Google emailed me, I fixed the problem, everyone wins. Except…

Following Google’s FAQs, I went to Webmaster Tools, signed up, verified my site, went to their tools…

And in the site management tools, found no such link as they described. Great.

At the time, I assumed only Google was using stopbadware: I’ve since discovered that other things are using it, so I’ve requested reconsideration there.

Still, Google now tells users that my site may be dangerous, despite the fact that it no longer is, and there appears to be no tool in the website management ‘tools’ panel to have them check it out again. Stretches the definition of ‘Do No Evil’ a bit… (Edit: Okay, not really, but it always works when you really want to get a response out of Google to just tell them they’re being evil: People get defensive and help you out ;))

In any case, my web site should be safe. Sorry that people have been confused by the problem.

Edit: JohnMu in comments pointed out why I was having a problem: Since crschmidt.net/blog/ was the only thing listed as ‘infected’, I had to sign up and verify for crschmidt.net/blog/ *seperately* from crschmidt.net. Certainly not exactly intuitive, but doing so allowed me to request a review of my site, so hopefully soon people will be able to view my site again in FF3, and won’t be caught out by Google’s warning (assuming I got all my malware off).

In order to use TileCache to cache tiles for use in the Google Maps API, you need three things:

• A WMS server which supports the spherical mercator projection
• A properly configured TileCache pointing to it
• A small snippet of code to add a custom TileLayerOverlay to your Google Map
• A small snippet of code to add a custom TileSource to your Virtual Earth Map

WMS Server

My experience is with MapServer, so that’s what I’m going with here.

MapServer uses proj.4 for its reprojection support. In order to enable reprojection to Spherical Mercator in MapServer, you must add the definition for the projection to your proj.4 data directories.

On Linux systems, edit the /usr/share/proj/epsg file. At the bottom of that file, add the line:

<900913> +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 
               +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs

After you do this, you must add the projection to your wms_srs metdadata in your map file:

map
  web
    metadata
      wms_srs "EPSG:4326 EPSG:900913"
    end
  end
  # Layers go here
end

This will allow you to request tiles from your MapServer WMS server in the Spherical Mercator projection.

Configuring TileCache

Your TileCache configuration will need to point to your WMS installation, using the parameters suggested for Spherical Mercator in the default tilecache.cfg.

[google-tiles]
type=WMS
url=http://labs.metacarta.com/wms/vmap0
layers=basic
extension=png
bbox=-20037508.3427892,-20037508.3427892,20037508.3427892,20037508.3427892
maxResolution=156543.0339
srs=EPSG:900913

Here, you can see that I’ve used the MetaCarta Labs vmap0 WMS. If you’re using a standard MapServer WMS, you might have a url more like:

url=http://example.com/cgi-bin/mapserv?map=/mapdata/mapfiles/vmap0.map

Setting up Google Maps

Finally, you must make a GTileLayerOverlay for your tiles.

 
    var myTileLayer = new GTileLayerOverlay(new GTileLayer(null,null,null,{ 
      tileUrlTemplate: 'http://example.com/tilecache/1.0.0/google-tiles/{Z}/{X}/{Y}.png?type=google', 
      isPng:true})); 
    var map = new GMap2(document.getElementById("map_canvas"));
    map.setCenter(new GLatLng(0,0), 0);
    map.addControl(new GSmallMapControl());
    map.addOverlay(myTileLayer);

The ‘type=google’ flag on the end of the URL tells TileCache to use the “Google-style” 0,0 in the upper left corner.

Once you’ve done this, you should have a TileCache layer on top of your Google Maps base layer. You can see an example of this setup, just for proof that I’m not putting you on. 🙂

Virtual Earth Javascript

“But I don’t like Google!” you say. “I want VE!” Well then, why aren’t you using OpenLayers already? I mean, that’s what it’s for, right? 🙂

More seriously, VE isn’t much more difficult:

    vemap = new VEMap('myMap');
    vemap.LoadMap(new VELatLong(0, 0), 0 );
    //Add layer
    var tileSourceSpec = new VETileSourceSpecification("mclabs", null, 1, 
       [new VELatLongRectangle(new VELatLong(-86,-180),new VELatLong(86,180))], 
       1, 16, function (tileContext)
{
   if(tileContext != null && tileContext != "undefined")
   {
      var key = tileContext.ZoomLevel+ "/" + tileContext.XPos + "/" + tileContext.YPos + ".png?type=google";
      var path = "http://example.com/tiles/1.0.0/google-tiles/" + key;
      return path;
   }
}, 0.8, 100 );
     vemap.AddTileSource(tileSourceSpec);
    
     var tileLayer = new VELayerSpecification(VELayerType.VETileSource,"mclabs","mclabs");
     vemap.AddLayer(tileLayer);

And, for your viewing pleasure: an example of the same tileset in use.

Using Client Side Storage for Mapping — a proof of concept which saves locally drawn features either via HTML5 Offline Storage or the same from Google Gears.

After a long, long haul, r6945 tags OpenLayers 2.6 as a final release.

The OpenLayers Development Team is proud to announce release of OpenLayers 2.6. As of this final release, the OpenLayers 2.6 release closes 294 outstanding tickets. This is the largest of any OpenLayers release to date.

Client side reprojection! Smooth commercial layer panning! KML styling support and pretty, auto-sizing popups! Improved styling support! And lots more.

Six months in the making, this is the best OpenLayers release ever. (And only two RCs! A new record. Here’s hoping that’s not just because we had no testers… ;))

Use 2.6 today!

Just a reminder: If you use OpenLayers in your applications, Now Is The Time to test out 2.6 and make sure that it doesn’t break anything. We’re currently in RC, and we’re getting pretty close to cutting a final release if no one complains.

The RC2 Announcement has more info on how to get set up with OpenLayers 2.6.

OpenLayers 2.6 adds some cool mechanisms for doing your styling on the client side: allowing you to use attributes of your data to create styling information on the fly.

A couple days ago, there was a post about using GeoJSON for thematic mapping. The conclusion of that post puts the blame in the wrong place: “Conclusion: GeoJSON has a lot of potential, but is currently not suitable for world maps due to browser restrictions.” was the conclusion, but this is the case for all formats: nothing here is specific to GeoJSON. It also stated that Firefox is the best browser for in-browser vector display: this is also wrong, as both Safari and Opera do significantly better with SVG rendering than Firefox or IE.

I’ve moved beyond that, though, and wanted to look at various ways to style the data.

The original map was essentially only styled by using data built into the GeoJSON for storing attribute color. Clearly this is not ideal: embedding styling information with the data is great when you want to control the user experience, but seperating it allows application developers more control. Luckily, with the upcoming release of OpenLayers (OpenLayers 2.6), you can do this styling in the browser.

This choropleth map uses the same data as the example in the Thematic Mapping blog post, but instead of taking the style rules from the data, creates a graduated color set. (I don’t know what this is actually called: Thematic mapping isn’t my gig.) Looking at the code, it’s easy to see the color ranges: 0 -> 10, 10-> 20, 20->50, 50->100. (The theme is separated into a separate function for readability.) No base map: we don’t need one for the visual effect to be reasonably pronounced. We get a worldwide, colored map of internet users (in 2005), with attribution and the ability to hover over a country to see its statistics.

But wait, there’s more!

OpenLayers 2.6 has reprojection support: the ability to change the projection of data. So, we can reproject the map into mercator. This is actually a useful educational map: you can see that different projections show significantly different percentages of the world as being covered with high-percentage of internet users.

Anyway, threw it together, and thought it was cool.