Technical Ramblings

The HTC HD7, with Windows Phone 7, is a damn cool phone.

I don’t say this because I’m some kind of corporate shill: I gave a talk last year at Boston’s BarCamp which was titled “Why I love my N900 — and you probably wouldn’t.” Despite working for Nokia, I realize that their products are not for everyone, and comparing Symbian to some of its competitors shows some obvious flaws.

Windows Phone 7, on the other hand, is a really fun operating system.

This phone is fast. Lightning fast. Blazing fast. There’s no ‘loading’ screens for seconds while an app starts up. Every time you press a button, something happens. On sheer ‘toy’ factor alone, the phone and its OS are very impressive. The first reaction that you have when you use the OS is “Wow”, all over.

The integration between apps and the OS is pretty nice. I took a picture, and I could share it via Flickr, Facebook, or Twitter, through their existing apps. Facebook news updates integrate into the contacts application, as do others (like Windows Live) though right now it seems that this is done at the OS level and not at the app level. Facebook integration into contacts includes everything you’d expect, pictures and so on — including phone numbers, something I never got from the N900 ‘facebook’ (Jabber) integration.

The calendar app is slick; the ‘Agenda’ view gives me all the information that I want, quickly and easily, and the home screen includes a clock and my next calendar appointment, making it so I don’t even need to

The browser is fast, like the rest of the system. Unlike most prior experiences in IE, the browser is competitive with most other mobile browsing experiences. (It doesn’t seem to have any touch event support, but the only thing I’ve ever used that does is iOS, so I’m biased here.)

Wifi handling is good; it switches easily to Wifi when it sees a saved access point, with no obvious problems.

The phone app is solid: getting to it is easy, answering calls has a good UI, and the voice quality is solid. Getting to the phone easily is something that the N900 definitely did not give me, and I was surprised by how much it annoyed me.

Voice search/activation is cool, and actually worked pretty well. (Getting to it is stupid — holding down the “Windows” button turns it on rather than, say, “Search” — but it works pretty well.)

The Marketplace application is nice; support for updates, and a reasonable UI for quickly finding what you’re looking for. (The search leaves something to be desired; I oftentimes find myself searching in ‘Apps’ and getting back music/videos first, which isn’t really what I want. But when I know what I’m looking for, it works relatively well.) Compared to the Ovi Store which I’m used to, it’s a breath of fresh air: again, fast, easy to find things, and it has lots of content. Of the top 10 free apps on the android market, 7 of them are also in the Microsoft Marketplace — everything from Kindle to Facebook to Twitter is available in the existing marketplace.

Live tiles are neat. Seeing them on TV/advertisements, I didn’t really care. But they actually work well, include information that’s helpful and useful at a quick glance. The homescreen UI also makes it easy to get easy access to the applications, contacts, etc. that I want — and doesn’t have an arbitrary limit on a number of home screens like other devices do.

Text messaging is easy to use. Text all over, in fact, is really pretty — the entire UI is just easy on the eyes, which helps a lot.

Pictures are easy. Picture quality is solid — as much as I expect from any point and shoot.

Despite the lack of ‘multitasking’, the app ‘stack’ is well managed — much better than iOS, which is always centered around the home screen. I can open an email, have it take me to a link, follow it to a YouTube video in the YouTube app, and then hit back to go all the way back to the original email.

Overall, I feel like the WP7 OS running on the HTC HD7 really feels like a solid competitor in the next generation of smartphones. It’s fast, clean, and smooth all over, and for everything from sending email to watching movies on Netflix, it’s a fun phone to use, and lets me get done everything that I want to do quickly and easily.

Now if only I could get cell reception in the damn office.

Earlier this weekend, I was pointed to a post about MBTiles, a new portable map tile distribution format. After a bit of hemming and hawing, I realized what it actually was, and realized that there wasn’t really much to hem and haw about.

This ‘new’ format is really nothing new, nor is it actually a ‘file format’ in the strictest sense of the word. What it is instead is:

1. An OS X binary that takes a bunch of files in directories on disk and writes them into a sqlite database.

Some of you may remember the old commercials for the iMac: “Step 1: Plug in. Step 2: Get connected. There is no step 3.” In this case, there isn’t even a step two. There’s nothing else here: not a specification document, not a reader of any kind, not even a description of what this magical ‘file format’ is.

Okay, so it’s not much of anything, but it’s not really a *bad* idea, it’s just that… it’s not the way I’d go about it.

First of all, if you actually want people to actually use something, you kind of have to have a reader and a writer. A promise of an iPad app improvement someway down the line along with some vague handwavey “saved 90% of your disk space!” statistics might be good for a flash in the pan in the Twitter-world echo chamber, but if you want success, you gotta do a bit better than that.

So, to outline what the MBTiles ‘file format’ actually seems to be: It’s a sqlite database with two simple tables.


CREATE TABLE metadata (name text, value text);
CREATE TABLE tiles (zoom_level integer, tile_column integer, tile_row integer, tile_data blob);

The metadata is not required to contain anything (so far as I can tell; possibly some reader tools might require it).

Now, a smart reader might notice that there is nothing very complex about this: any programming langauge that can interact with sqlite can create or access the MBTiles data — a very good thing. (It’s possible that not publishing this simple fact is because the Development Seed/MapBox folks plan to extend it and don’t want to actually make it so that other people are using it; dunno.) However, excepting that, as it is, there’s no reason (that I can see) that the code to create a cache should be in C!

For prototyping, or if your goal is to create and develop a ‘standards’ thing, you really want to be working in a language which is more widely understood and easier to prototype. I realize that this is a judgement call on my part, but for things where you encourage people to check it out and use it, you should be working in a language with a wider audience. For example, using the mb_tiles_importer on my tiles produced by TileCache gives me a database that has 4 rows… even though my directory only has two tiles in it, and 2 of the 4 rows are entirely empty. If the code were in Python, I might take a look and offer some feedback, but with it being an OS X only binary, or even a thinly documented C script, there’s almost nothing I can do to figure out what’s going on or help.

Add to this the fact that although this format has a writer, it has no open source reader of any kind that I can find. There’s some chat about various MapBox related software reading it, but no separation of this from MapBox — and with no description of the format, there’s not an excuse that it’s designed for people to write their own clients…

That said, I’ve gone ahead and written support into TileCache for the ‘format’ such as it is; I’m not convinced it’s the ideal thing to do, but the core concept of delivering a single file in the form of an SQLite database for tile data is a pretty solid goal.

Overall, the idea is reasonably sound: Delivering tiles in a single file is important, and sqlite is a nice, lightweight format for that that’s accessible from most C-based languages. Writing a quick cache format to read these things in TileCache was easy enough — because, as I said, there really isn’t much there. I didn’t write write support, because doing so seemed like it could be a waste of the MapBox folks want to ‘own’ this format (Hello, GeoPDF, how are you today) and are still developing it, and the only way that I was able to even do what I did was using a tool that I had to grab from a Github link I got over Twitter (and doesn’t appear to work right).

B+ for the idea. It’s a bit iffy on the implementation, but the core goal is sound. However, the way that it’s approached is a somewhat typical approach that I see lately: Publish first, actually create the thing you’re publishing about later. That type of attitude is the kind of thing that drives me — as a creator who puts a lot of time and thought into community interaction first and foremost — absolutely bonkers.

Clean it up, make it a spec, and describe some of the benefit and utility in a way that’s not tied directly to MapBox, and I can see this actually becoming a pretty regular thing for distributing files around. I can definitely see the value and benefit — with some metrics at larger scale — of doing this kind of thing for distributing larger tilesets. I just don’t want to fall into the Admiral Ackbar problem: “It’s a trap!”

In a conversation at FOSS4G, Schuyler and I sat down with Frank Warmardam to chat about the possibility of extending GDAL to be able to work more cleanly when talking to files over HTTP. After some brief consideration, he agreed to do some initial work on getting a libcurl-based VSIL wrapper built.

VSIL is an API inside of GDAL that essentially allows you to treat files which are accessed through different streaming protocols available as if they were normal files; it is used for support for accessing content inside zipped containers, and other similar data access patterns.

GDAL’s blocking strategy — that is, the knowledge of how to read sub-blocks of files in order to obtain the information it needs, rather than needing to read a larger part of the file — is designed to limit the amount of disk I/O that’s needed for rendering large rasters. A properly set up raster can limit the amount of data that needs to be read significantly, helping improve tile rendering time significantly. This type of access would also allow you to fetch metadata about remote images without the need to access an entire (possibly large) image.

As a result, we thought it might be possible to use HTTP-based access to images using this mechanism; for metadata access and other similar information over the web. Frank thought it was a reasonable idea, though he was concerned about performance. Upon returning from FOSS4G, Frank mentioned in #gdal that he was planning on writing such a thing, and Even popped up mentioning ‘Oh, right, I already wrote that, I just had it sitting around.’

When Schuyler dropped by yesterday, he mentioned that he hadn’t heard anything from Frank on the topic, but I knew that I’d seen something go by in SVN, and said so. We looked it up and found that the support had been checked into trunk, and we both sat down and built a version of GDAL locally with curl support — and were happy to find out that the /vsicurl/ driver works great!

Using the Range: header to do partial downloads, and parsing some directory listing style pages for READDIR support to find out what files are available, the libcurl VSIL support means that I can easily get the metadata about a 1.2GB TIF file with only 64kb of data transferred; with a properly overlaid file, I can pull a 200 by 200 overview of the same file while using only 800kb of data transfer.

People sometimes talk about “RESTful” services on the web, and I’ll admit that there’s a lot to that that I don’t really understand. I’ll admit that the tiff format is not designed to have HTTP ‘links’ to each pixel — but I think the fact that by fetching a small set of header information, GDAL is then able to find out where the metadata is, and request only that data, saving (in this case) more than a gigabyte of network bandwidth… that’s pretty frickin’ cool.

Many thanks to EvenR for his initial work on this, and to Frank for helping get it checked into GDAL.

I’ll leave with the following demonstration — showing GDAL’s ability to grab an overview of a 22000px, 1.2GB tiff file in only 12 seconds over the internet:

$ time ./apps/gdal_translate -outsize 200 200  /vsicurl/http://haiticrisismap.org/data/processed/google/21/ov/22000px.tif 200.tif
Input file size is 22586, 10000
0...10...20...30...40...50...60...70...80...90...100 - done.

real	0m11.992s
user	0m0.052s
sys	0m0.128s

(Oh, and what does `time` say if you run it on localhost? From the HaitiCrisisMap server:

real	0m0.671s
user	0m0.260s
sys	0m0.048s

)

Of course, none of this compares as a real performance test, but to give an example of the comparison in performance for a single simple operation:

$ time ./apps/gdal_translate -outsize 2000 2000 
     /vsicurl/http://haiticrisismap.org/data/processed/google/21/ov/22000px.tif 2000.tif
Input file size is 22586, 10000
0...10...20...30...40...50...60...70...80...90...100 - done.

real	0m1.851s
user	0m0.556s
sys	0m0.272s

$ time ./apps/gdal_translate -outsize 2000 2000 
    /geo/haiti/data/processed/google/21/ov/22000px.tif 2000.tif
Input file size is 22586, 10000
0...10...20...30...40...50...60...70...80...90...100 - done.

real	0m1.452s
user	0m0.508s
sys	0m0.124s

That’s right, in this particular case, the difference between doing it via HTTP and doing it via the local filesystem is only .4s — less than 30% overhead, which is (in my personal opinion) pretty nice.

Sometimes, I love technology.

At the OSGeo Board meeting in Barcelona, we discussed many things, but one of the topics of special interest to me is the simple question: “What is OSGeo all about?”

The first place to look for that, of course, is the website; although many parts of the website address many specific problems, there is one place that we define what OSGeo is really about: the mission statement. It says that the Mission of OSGeo is:

To support the collaborative development of open source geospatial software, and promote its widespread use.

When we started our board discussions, there was one word missing there: the “collaborative” is something we voted to add, something I was very supportive of. There are many organizations (Sencha being a significant example in the space I work in) where organizations are developing Open Source software that is not openly developed. OSGeo is not about that: instead, it’s about encouraging exactly the opposite.

One of the most important things that OSGeo incubation does is ensure that a project is collaboratively developed. We seek for projects with a reasonably broad base of support, in terms of both developers and users. We seek to encourage community; our default project setup uses open, widely available collaborative development tools.

We host dozens of mailing lists. We have a single login account that gives access to the bug trackers for more than a dozen projects. We seek the broadest interaction between projects possible in order to foster a collaborative environment.

OSGeo is a really interesting case for this type of foundation work, because we have such a broad collection of projects despite the narrow scope. Databases. Web servers — both Map and other GIS related. Clients. Data manipulation libraries. Metadata catalogs. All of them interact at almost every stage of the process. Interoperability of this software is a key way to make the Open Source geospatial world more successful, and something we do relatively well.

So, if anyone ever asks you: What does OSGeo do? The answer, at its heart, is: “Support the collaborative development of open source geospatial software.” And I’m pretty thrilled with both the goal, and the success so far.

In the past, for tiling, we discussed tiling on an EOGEO list. In the meantime, OSGeo has grown up, EOGEO has moved on, and it seems that there isn’t a very good home for future tiling discussions.

As a result, I have added a tiling list to the OSGeo mailing list server.

Tiling List @ OSGeo

Projects that I hope to see people joining from: TileCache, Tirex, Mapproxy, GWC, others, etc.

This list will be discussing general tiling ideas — how to cache tiles, how to manage caches, how to work with limited caches, where to put your tiles, things like S3, etc. etc. If you are at all interested in tiling — not at the level of a specific application, but in general — please join the list.

Additionally, if you are interested in discussing providing feedback to the OGC regarding the WMTS spec — especially if you are an implementer, but also if you are a user — I would encourage you to join the standards list at OSGeo:

http://lists.osgeo.org/mailman/listinfo/standards

Several people have expressed interest in coordinating a response to the OGC regarding the spec, and we would like to work together on this list to coordinate.

OpenLayers is approaching the ‘Brave New World’ of a major API breakage for the first time since OpenLayers 2.0 in October of 2006.

Things that have changed in OpenLayers since then:

• Support for vector drawing.
• The existence of ‘spherical mercator’
• reprojection of objects
• More than a dozen new layer types
• The existence of TileCache, FeatureServer, and RESTful HTTP APIs for geodata
• A tile-serving service from OpenStreetMap (also, for OpenStreetMap: 4 major API changes)
• Over four dozen additional contributors to the source code
• Over 2000 trac tickets
• 10 2.x releases

Overall, is it any shock that it’s time for a pretty major change?

In order to facilitate rapid development, we’ve shifted development of OpenLayers ‘3.0’ to github; you can follow along (or fork you own) on the OpenLayers github project. To be honest, git scares the crap out of me; every time I’ve used it, I have consistently lost data as a result of not understanding the tool and using the wrong command. However, I fully realize I am a fuddy duddy who needs to get over his problems at this point. 🙂

Looking forward to seeing the future of the world where OpenLayers is even more awesome than it already is!

(The following opinions are my own, and not representative of my employer or anyone else.)

A friend asked me via email:

I noticed you’ve had your N900 for a while. I am thinking about upgrading from my iPhone 3G (it’s really slow — lots of lagging on UI) to a Nexus One or a N900. Have you had a chance to compare the two at all? Where did you get your N900? Any “killer apps” or major problems with it?

I don’t know if ‘5 days’ is ‘a while’ yet. 🙂 As for how I got it… well, perhaps you missed my previous post…

I think that upgrading from an iPhone — if you generally like the way it works, and are just upset about the UI speed/performance — will be a disappointment. I have an iPod Touch and the n900, and when I want to run an app real quick, the iPhone will win, hands down. The N900 has far more functionality to be excited about, but in general, there are still aspects of the iPhone that win. Basically: If you didn’t jailbreak your iPhone, you’re not looking for the ‘right’ kind of thing for an N900 to be the solution. (I didn’t jailbreak my iPod, but it’s a media player, not a communications device.)

I don’t have experience with the Nexus One. The G1 users in the office have commented repeatedly on how ‘fast’ the N900 response is by comparison: my understanding is that the Nexus One is a significant hardware step up from the G1, so that experience may be irrelevant. (G1 users also have said “If they just came out with a ‘G2’ that just had double the CPU + double the memory, I’d buy it in a heartbeat.”)

Before the recent events, I was considering the N900 or the G1. After playing with a G1 for a while, I realized that it — sort of like the iPhone, though to a lesser extent — is also not a general purpose computer. It’s closer, because it’s more open, but in the end, it’s designed as a ‘platform’, and that’s obvious in many aspects of how it works. The Maemo OS was always developed for Internet Tablets, and that shows in many aspects of the design. (This is also why it makes a somewhat sub par phone; although the OS has grown well into many aspects of the device, the ‘phone’ app is recent enough that it hasn’t had time to mature into that role.)

Overall: If you’re the type of person who wants to run a shell script, and display the results of a shell script on your phone’s desktop, the N900 may well be for you. (Really.) If you want a platform, but for something that’s designed originally as a phone, and with a much broader ecosystem and community — but more limitations ‘out of the box’ — the Nexus One may be for you, though that’s mostly hearsay. (I would consult a G1/Nexus One user for anything resembling a serious opinion here.) And if you want something pretty that isn’t going to require you to pop open a terminal to fix it when something breaks — the Apple way may be the right way for you.

Some cool aspects of the N900: two-way video calling with Google talk (but only when initiated from a computer, sadly). Two way audio-chat over Jabber. Built-in always-on IM clients. Exchange mail push support built in. (Exchange support in general is excellent; this includes both ‘real’ exchange and clones like Zimbra.) ‘apt-get install openssh’. Doom on the phone. High res screen, great for video playback.

Some things it doesn’t do well: Making it trivial to open the actual ‘phone’ part (hardware button) would be great. The lack of Facebook app is disappointing; the iPhone one was so good that I was spoiled by it. (I’m not a huge Facebook user, so this isn’t a deal breaker by any stretch.) The fact that there was not, until recently, a well-supported way of distributing non-open applications, means that there isn’t a lot of non-open application development — so fewer professionally done games, free or pay, than on a platform like the iPhone (and I assume fewer than on Android as well). That automatically implies that you cannot use Wordscape answers you get off of the internet in your iPhone. Ovi Maps, does not compare favorably to Google Maps in many cases, and running apps via the browser just doesn’t feel right. (This is a start, but not really ideal.) Battery life is poor, but that’s probably my own fault. 🙂

As a final point of comparison, the app community for the other devices doesn’t even compare to Apple’s in quantity. However, overall, the ‘decent, usable, free apps’ communities do seem somewhat larger on N900 than on Apple, in my experience. Finding something free on iTunes that’s worth downloading is hard — but the Maemo garage is full of fun, useful apps, and they’re all free.

I like the N900. It’s certainly not for everyone, but for me, I like having my communications device being a general purpose computer (so long as the phone part keeps working). There’s plenty of room to improve, but having a very open ecosystem reassures me that I may be able to contribute some of that, and overall, it fits the bill for being the most awesome thing to carry around in my pocket.

As of April 9th, MetaCarta has been acquired by Nokia, and I am now an employee of Nokia working on local search in the Ovi services. (Woohoo!)

When using the Python ‘boto’ library for accessing Amazon Web Services, to enable logging to a file at the ‘debug’ level, simply use the logging module’s configuration:

import logging
logging.basicConfig(filename="boto.log", level=logging.DEBUG)


Place this line near the top of your script, and logging will take place to a file in your current directory called “boto.log”.

I’m sure that this is obvious for most people who use the Python logging module, but this is new code to me, and it took me a fair bit of looking to find out how to enable logging; hopefully other people find it more easily now.

KML is linked. It is self-descriptive, and can rely entirely on following of links to obtain more information, whether that is styles or additional data.

However, the most common way of packaging KML is as KMZ — which is sort of like packaging an HTML page inside a zip file with all of its component parts. When this is done, web-based tools — like the Javascript support in browsers — lose all access to the data other than through a server side proxy (and even that isn’t a trivial thing to achieve). Styling information and related parts are not stored as separate resources on the web. The information available in the KML has suddenly become just another application-specific format.

If this were uncommon, it wouldn’t be such a shame; it’s certainly possible to distribute data like this for use cases where it is necessary, including offline use and other use cases. However, this is not a limited situation — in fact, more than 80% of KML made available on the web tends to be primarily available as KMZ. This packaging of KML leaves much to be desired, and limits the use of such data in web-based tools.

The web already has ways to compress data — gzip-based compression is common on many web servers (a tradeoff of CPU time for bandwidth), and works fine in all KML clients I’m aware of (including Google Earth and Google Maps). This lets your data exist on the web of resources and documents, rather than in a zipped up bundle.

My interest in this matter should be obvious: I work with mapping on the web. Ideally, I work with tools that don’t require server-side code — every piece of server side code you have to build is another heavy requirements placed on the users of any software. Browsers, as a common platform across which developers can code, are a worthwhile target, and trapping your data in KMZ hides it from browsers.

Free your KML! Publish on the Web! Don’t use KMZ!