Technical Ramblings

So, a couple people have asked me why Google thinks that my site may harm your computer.

On Wednesday, I received an email from “Google Search Quality”:

We recently discovered that some of your pages can cause users to be infected with malicious software. We have begun showing a warning page to users who visit these pages by clicking a search result on Google.com.

I looked through, and sure enough, the links they offered were indeed ‘infected’: I’ve always used WordPress despite my knowledge of the fact that it has security exploits more often than I get around to fixing them (though I do try to keep up). They had has a small iframe included, which claimed to be ‘stats tracking’: Instead, there was Javascript included which, presumably, was malicious.

To the best of my knowledge, I solved this problem on Wednesday night, by removing the mal-links that were pointed out, and patching the security holes I could find fixes to in WordPress. (I just upgraded to WordPress 2.5; In the past, upgrading has been painful, but it wasn’t so bad this time, and there are not yet any known security holes for 2.5 that I’m aware of.)

All in all, not a bad thing: Google emailed me, I fixed the problem, everyone wins. Except…

Following Google’s FAQs, I went to Webmaster Tools, signed up, verified my site, went to their tools…

And in the site management tools, found no such link as they described. Great.

At the time, I assumed only Google was using stopbadware: I’ve since discovered that other things are using it, so I’ve requested reconsideration there.

Still, Google now tells users that my site may be dangerous, despite the fact that it no longer is, and there appears to be no tool in the website management ‘tools’ panel to have them check it out again. Stretches the definition of ‘Do No Evil’ a bit… (Edit: Okay, not really, but it always works when you really want to get a response out of Google to just tell them they’re being evil: People get defensive and help you out ;))

In any case, my web site should be safe. Sorry that people have been confused by the problem.

Edit: JohnMu in comments pointed out why I was having a problem: Since crschmidt.net/blog/ was the only thing listed as ‘infected’, I had to sign up and verify for crschmidt.net/blog/ *seperately* from crschmidt.net. Certainly not exactly intuitive, but doing so allowed me to request a review of my site, so hopefully soon people will be able to view my site again in FF3, and won’t be caught out by Google’s warning (assuming I got all my malware off).

In order to use TileCache to cache tiles for use in the Google Maps API, you need three things:

• A WMS server which supports the spherical mercator projection
• A properly configured TileCache pointing to it
• A small snippet of code to add a custom TileLayerOverlay to your Google Map
• A small snippet of code to add a custom TileSource to your Virtual Earth Map

WMS Server

My experience is with MapServer, so that’s what I’m going with here.

MapServer uses proj.4 for its reprojection support. In order to enable reprojection to Spherical Mercator in MapServer, you must add the definition for the projection to your proj.4 data directories.

On Linux systems, edit the /usr/share/proj/epsg file. At the bottom of that file, add the line:

<900913> +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 
               +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs

After you do this, you must add the projection to your wms_srs metdadata in your map file:

map
  web
    metadata
      wms_srs "EPSG:4326 EPSG:900913"
    end
  end
  # Layers go here
end

This will allow you to request tiles from your MapServer WMS server in the Spherical Mercator projection.

Configuring TileCache

Your TileCache configuration will need to point to your WMS installation, using the parameters suggested for Spherical Mercator in the default tilecache.cfg.

[google-tiles]
type=WMS
url=http://labs.metacarta.com/wms/vmap0
layers=basic
extension=png
bbox=-20037508.3427892,-20037508.3427892,20037508.3427892,20037508.3427892
maxResolution=156543.0339
srs=EPSG:900913

Here, you can see that I’ve used the MetaCarta Labs vmap0 WMS. If you’re using a standard MapServer WMS, you might have a url more like:

url=http://example.com/cgi-bin/mapserv?map=/mapdata/mapfiles/vmap0.map

Setting up Google Maps

Finally, you must make a GTileLayerOverlay for your tiles.

 
    var myTileLayer = new GTileLayerOverlay(new GTileLayer(null,null,null,{ 
      tileUrlTemplate: 'http://example.com/tilecache/1.0.0/google-tiles/{Z}/{X}/{Y}.png?type=google', 
      isPng:true})); 
    var map = new GMap2(document.getElementById("map_canvas"));
    map.setCenter(new GLatLng(0,0), 0);
    map.addControl(new GSmallMapControl());
    map.addOverlay(myTileLayer);

The ‘type=google’ flag on the end of the URL tells TileCache to use the “Google-style” 0,0 in the upper left corner.

Once you’ve done this, you should have a TileCache layer on top of your Google Maps base layer. You can see an example of this setup, just for proof that I’m not putting you on. 🙂

Virtual Earth Javascript

“But I don’t like Google!” you say. “I want VE!” Well then, why aren’t you using OpenLayers already? I mean, that’s what it’s for, right? 🙂

More seriously, VE isn’t much more difficult:

    vemap = new VEMap('myMap');
    vemap.LoadMap(new VELatLong(0, 0), 0 );
    //Add layer
    var tileSourceSpec = new VETileSourceSpecification("mclabs", null, 1, 
       [new VELatLongRectangle(new VELatLong(-86,-180),new VELatLong(86,180))], 
       1, 16, function (tileContext)
{
   if(tileContext != null && tileContext != "undefined")
   {
      var key = tileContext.ZoomLevel+ "/" + tileContext.XPos + "/" + tileContext.YPos + ".png?type=google";
      var path = "http://example.com/tiles/1.0.0/google-tiles/" + key;
      return path;
   }
}, 0.8, 100 );
     vemap.AddTileSource(tileSourceSpec);
    
     var tileLayer = new VELayerSpecification(VELayerType.VETileSource,"mclabs","mclabs");
     vemap.AddLayer(tileLayer);

And, for your viewing pleasure: an example of the same tileset in use.

Using Client Side Storage for Mapping — a proof of concept which saves locally drawn features either via HTML5 Offline Storage or the same from Google Gears.

After a long, long haul, r6945 tags OpenLayers 2.6 as a final release.

The OpenLayers Development Team is proud to announce release of OpenLayers 2.6. As of this final release, the OpenLayers 2.6 release closes 294 outstanding tickets. This is the largest of any OpenLayers release to date.

Client side reprojection! Smooth commercial layer panning! KML styling support and pretty, auto-sizing popups! Improved styling support! And lots more.

Six months in the making, this is the best OpenLayers release ever. (And only two RCs! A new record. Here’s hoping that’s not just because we had no testers… ;))

Use 2.6 today!

Just a reminder: If you use OpenLayers in your applications, Now Is The Time to test out 2.6 and make sure that it doesn’t break anything. We’re currently in RC, and we’re getting pretty close to cutting a final release if no one complains.

The RC2 Announcement has more info on how to get set up with OpenLayers 2.6.

OpenLayers 2.6 adds some cool mechanisms for doing your styling on the client side: allowing you to use attributes of your data to create styling information on the fly.

A couple days ago, there was a post about using GeoJSON for thematic mapping. The conclusion of that post puts the blame in the wrong place: “Conclusion: GeoJSON has a lot of potential, but is currently not suitable for world maps due to browser restrictions.” was the conclusion, but this is the case for all formats: nothing here is specific to GeoJSON. It also stated that Firefox is the best browser for in-browser vector display: this is also wrong, as both Safari and Opera do significantly better with SVG rendering than Firefox or IE.

I’ve moved beyond that, though, and wanted to look at various ways to style the data.

The original map was essentially only styled by using data built into the GeoJSON for storing attribute color. Clearly this is not ideal: embedding styling information with the data is great when you want to control the user experience, but seperating it allows application developers more control. Luckily, with the upcoming release of OpenLayers (OpenLayers 2.6), you can do this styling in the browser.

This choropleth map uses the same data as the example in the Thematic Mapping blog post, but instead of taking the style rules from the data, creates a graduated color set. (I don’t know what this is actually called: Thematic mapping isn’t my gig.) Looking at the code, it’s easy to see the color ranges: 0 -> 10, 10-> 20, 20->50, 50->100. (The theme is separated into a separate function for readability.) No base map: we don’t need one for the visual effect to be reasonably pronounced. We get a worldwide, colored map of internet users (in 2005), with attribution and the ability to hover over a country to see its statistics.

But wait, there’s more!

OpenLayers 2.6 has reprojection support: the ability to change the projection of data. So, we can reproject the map into mercator. This is actually a useful educational map: you can see that different projections show significantly different percentages of the world as being covered with high-percentage of internet users.

Anyway, threw it together, and thought it was cool.

It’s my Birthday!

Mapping related gifts I’ve picked up so far:

• Mapping Boston, which is gigantic and heavy and beautiful.
• A Wonder Globe, which is small and spinny and will go on my desk at work
• A National Geo map of Germany, for hanging on the walls at work
• Transit Maps of the World, which a friend who is staying with us picked up from off my recently revamped Amazon Wishlist 🙂

Not a bad haul, since I’ve only been 24 for 18 minutes 😉

A couple people have asked me over the years what I plan to do with GSMLoc.org. I think that every time I’ve answered, I’ve wished I could say “I have great plans!”… but I don’t.

Until now, I didn’t have a good dump of the data made automatically available. This was silly. I’ve now created a cronjob that will, at midnight, create a gzipped csv dump of the data.

I make no claims as to the quality of this data, and place no restrictions on it, though credit to gsmloc.org or the gsmloc project would be cool.

Sorry that I’ve been so lazy about not making this available. I may still clean up the website, and I’ll gladly link to other projects who are doing the same thing, and I’m sorry that I’ve been such a bad steward over the past couple years.

Jo Cook writes about ESRI’s crackdown on licensing:

In what can only be described as a noble act of self-sacrifice, ESRI have told us that as an educational charity we are no longer allowed to have an educational discount for using their software and, not only that, our license codes will cease to work at the end of this month.

This is why Open Source software is so important. So you think you have a stable relationship with your vendor? Maybe you think that you’ve come to a great licensing agreement that you’re happy with? Remember that so long as you’re working in an environment where someone else controls the tools you use, you’re not able to make your own rules.

Why does Open Source software matter? If you’re not using it, you’re handing over control of your use of your tools — possibly important ones — to people who aren’t under your control. In the end, that lack of control may end up hurting you far more than you’d expect.

KML has become the “HTML” of the Geographic Web. With limited semantic meaning, a combination of mostly-human understandable XML tags for the majority of the usages, widespread use and abuse for purposes far beyond the original thoughts and intentions of the designers, and more, KML fits well into the geographic version of the niche filled by HTML in more generalized content publishing.

Google Earth loads *content* over the web, rather than loading code like Google Maps mashups do. The difference is important: Google Earth ‘mashups’ generate content which is, to some extent reusable within other applications. Google Maps mashups don’t: they tend to use hidden databases without public APIs, and the HTML and code that you do see is executable code, not content.

The content that is generated is reasonably easy to follow: KML is reasonably close to human readable. There doesn’t tend to be a lot of extra XML that confuses users: similar to HTML, it can be edited, for the most part, in a text editor with limited difficulty. There’s a caveat to this, however: editing geographic information — the actual coordinates behind a ‘FlyTo’ — is something that humans don’t have an innate ability to do.

To this problem, Google Earth offers a solution: In addition to being a KML viewer, it’s also a KML *generator* — draw any type of feature you like, style it how you like, then just select the item and ‘copy’ the item, and paste it into a text editor: You immediately have a working KML file, with coordinates that you created. WYSIWYG editing, built right into the data browser. Google Earth acts as the ‘view source’ button for the geoweb.

To some extent, this is extremely handy: users now have the ability to trivially create styled content in Google Earth, export it, and mix and mash these items together to create their own customized feed of data. However, this democratization is exactly the reason that KML does fit the bill of the “HTML of the GeoWeb”, with its pitfall as well as its benefits.

One of the key aspects of the difference between “paleogeographers” and neogeographers is the difference in the importance of accuracy. (I use the terms lightly here, since I don’t tend to categorize users in that way, but it provides a spectrum which is useful to understand.) In general, GIS professionals — regardless of what toolset they’re using, be it Google Earth or ESRI ArcMap — have a need for an understanding of the quality of their data. It doesn’t have to be perfect for all cases, but knowing how good it is or isn’t is an important distinction when you’re going to make decisions based on it. Google Earth doesn’t make including information like this a part of the KML workflow, nor did KML accommodate this type of usage very well until relatively recently.

Information like attributes of a feature, source data — in fact, metadata at all — and so on, is largely left out of the realm of KML. Similar to HTML, it’s largely treated as a ‘presentation’ language by the majority of the users of it. (FeatureServer uses it in a different way, and OpenLayers mimicks that usage, but that’s the exception, not the rule.) Although HTML in an ideal world separates style from substance — using CSS, semantically correct tags, etc. — I think that most web developers have seen too many uses of <table> for layout to believe that that ideal world actually exists. Similarly, KML has support for many features that users will never use. Separation of content and style is possible, but not as widely used, and mixing presentation with content (in terms of application-style control) is the best practice behavior for KML.

I’ve seen continued belief by some users that KML is a geographic data interchange format. Although exchanging geographic data via KML is possible, it’s not an interchange format for data: it’ s primarily an interchange format for presentation. Tools like GDAL and other geodata libraries do their ‘best guess’ interpretation of this data as geographic data, but are not well suited for presentation, only data interchange.

OpenLayers, on the other hand, is in a position where presentation can be taken into account. The newest version of OpenLayers has support for a fair amount of KML styling: other than differing default styles, this KML file looks pretty similar in OpenLayers and Google Maps. However, even here you can see the comparative limitations: Since OpenLayers is primarily about rendering maps, the ‘extras’ around the outside are lost, a somewhat significant loss in cases where most of the information is provided via additional display.

In the end, Google Earth is currently the only reasonably complete KML browser (that I’m aware of). Google Maps and Virtual Earth are slowly improving, and OpenLayers is moving forward as well. I look forward to seeing continued development, and am hopeful that the open source geospatial community can build UIs around data like KML that allow the feature rich presentational options that are currently available via KML in Google Earth — even if we’re only in 2d space. Solving the problems of implementing a format as far-reaching as KML is quite a task, and I’m enjoying watching as users demonstrate their interest in a particular set of features by helping to implement them, and look forward to continuing to see the trend of high quality presentations of geographic material grow, both in KML and other arenas.