Technical Ramblings

Make a map of my trip to North Carolina, including route information and foursquare checkins; using the trip log I recorded, make a graph of average speed over various spans. Highlight all the points along the trip where something went terribly wrong.

Create a game out of taking photos of POIs — sort of like foursquare, but for photos. Come up with an algorithm to suggest what places should get photos based on number of times they show up in things like search results, or are clicked, and which ones don’t have photos. (I think this is probably what that Google thing is, but I don’t really know.)

Try to build a tool which can decide what place a particular photo is attached to via text recognition or by associated metadata on a photo sharing site like flickr. Start with geocoded photos; see if it’s possible to extend somehow to non-geocoded photos (though I can’t think of anything obvious).

Build a dotmap-style map of every point address that NAVTEQ has in its database, possibly adding in POIs or something. Also, see if it’s possible to rewrite the dot map as something that looks as good, but can be dynamically generated for lower zoom levels.

Using the streetview-style imagery that Nokia has, build a tool which would let users pick a ‘good’ view for POI, using a suggestion starting from a geocoded coordinate. This would probably involve building a slippymap based viewer for Nokia’s streetview imagery, because as far as I know, there’s no dynamic scriptable viewer for that data.

These are just a few of the things that I’ve thought of over the past month or so, but the motivation to do any of them is always pretty low. Maybe I should get a new hobby. Or weed the garden instead.

Ed note: Since there has been some concern, I think I should preface this post with this: I think that my employer has a very solid business position and a solid line and business plan around what we *actually do*; I think there’s serious misunderstanding internally about what we do though, which leads to a lot of this. I also don’t see this as an indictment of the team I work for or what I actually do; my day job is productive and adding usefully to our platform. It just isn’t ‘fun’ in the same way as hacking up a demo in a weekend is.

A long time ago, I used to do fun things with maps.

I would make a map of Mario levels. Or build a draggable routing system. Or work on an editable map display on top of Google Maps.

I would plot out my route, or put my photos on a map, or other fun things.

But now, I don’t do those things anymore. I don’t make cool toys, or experiment with maps, or do anything particularly interesting with maps at all. Observant observers would probably (rightly) observe that this started approximately when I started working for my current employer — but probably not for the reasons that most would think (or at least, not the reason I’d have expected when I started working there).

Frankly, working with the rest of the stuff in the world is just depressing when you have to go back to working inside of a giant corporation when the weekend is over. I see experiments like I used to do as having two options: following the company line, or really pushing the envelope and doing something cool in the world.

Neither is as exciting, because frankly, doing something cool — but not using our corporate tools — means that I can’t share it with the people I spend the rest of my week with. “Why are you using Google Maps?” is the canonical question for almost all things — even when its clear that the answer is “Because we doesn’t provide a competitive offering.”

Using our offerings to do cool things is like pulling teeth. In the space of API availability, design, and support, we still living years in the past, and there is no major shift towards the future; as an enterprise-targeted company, our interests really don’t match that of the consumer market, but there’s a lack of acceptance of that, which leads to people thinking that we *should* be competitive.

I feel differently: I feel that we’re a successful enterprise company, and we should capitalize on that. We have an offering which no one can really compete with in that space — Google’s current enterprise story is poor, and their worldwide story is less good than ours — but it’s hard to convince people we shouldn’t try to compete with Google at everything we do.

In either case, I find working with maps a million times more frustrating now than I used to. The idea of making a cool demo is completely tamped down by the knowledge that using OpenStreetMap as a base map — even when it obviously presents a significantly improved experience — is considered disloyal.

In the end, it’s easier just to not bother. While I really enjoy what I do as my day job — to the extent that I spend a lot of off-hours work on it — it’s not a *cool* thing; my job is to make a small portion of the maps experience a tiny percent better. (To me, this is actually pretty exciting — every time we make a problem 1% better, we probably make thousands of people’s lives slightly easier every day, something I couldn’t possibly have said before.) But There’s very little of external interest in what I do on a day to day basis. And the things that I used to do which *were* interesting to others, I no longer do, because I feel stymied by the environment I work in.

It’s also why I seldom am aware of the latest news in the geospatial world. It’s just frustrating; to work for a company which has significant resources, but constantly feel like I’m watching us miss opportunities so that we aren’t competitive; to constantly see questions go by about “why can’t we do X?” when the answer is “Because we have massively misunderstood that market, and don’t want to try to compete.”

Working for a big company is also frustrating in and of itself. As we work on problems, then hand them over to new people whenever one group fails, there is no *learning* process. I watch as we make the same mistakes over and over — and I’m sure that others would say the same about the work that I do, though I feel like we’re at least making progress — and don’t learn from them. I watch as people seriously misunderstand how far ahead in some areas the rest of the geographic data space is, and learn the wrong lessons — and still think that we’re competing in the consumer space, when in reality, we should capitalize on our *successes* instead.

The lesson of OpenStreetMap is not “You should attempt to build your own collaborative mapping platform” — OSM has taken 9 years to build what they have, and there is no practical way that anybody else will be seriously competitive in that space.

The lesson of MapBox is not “People like prettier maps, so we’re going to make our maps be the ultimate in pretty.” The lesson is that people want *different* maps — and they want the process of creating them to be easy, all the way down to adding new data and details to it.

The lesson of the success of the Google Maps API — and later charging for it — is not “Mapping APIs are a valuable distinguishing factor in map data use, and we should charge an arm and a leg for access to the data.” People want data in their own applications, and if you provide a sufficiently compelling service, you may be able to get people to pay for it — so long as they can use it the way they need to.

Watching the company I make for make so many mistakes, to some extent, I just bury my head in the sand. I no longer do fun things with maps because every fun thing will be questioned, and I never liked the questioning part; I just wanted to build something cool.

I loved working at MetaCarta because I had a cool platform by which to create and have some people see the work I’d made. I had a group of coworkers who were interested in similar things, and would comment on them, and I worked in an environment where “anything goes” was the watchword. I always had the support of the company in spending time building a cool demo that used MetaCarta infrastructure to produce something was new and exciting.

I miss that about working for MetaCarta. I miss working in an environment which really was “Anything goes”; I miss working in a place where creativity was rewarded with praise instead of complaint; I miss working in a place where it was accepted that the best tool for the job isn’t always the tool that we have built.

That lack of insight from the rest of the company I work for is disappointing, and leaves me wishing that I had a bit of that magic back. Since I don’t: I don’t do fun things with maps anymore, because it just isn’t as much fun.

I’ll admit it: I’m obsessed with map data. Not the maps themselves — not how they look, or anything else about them — but the data, the bits that make up the way we find our way in the world.

This wasn’t always the case. In the past, I didn’t care much about the data — I just wanted pretty pictures so I could show things. But as I’ve changed from being a map maker to working on the search side of ovi Nokia HERE Maps, I’ve moved away from caring how the map looks, to caring what is underneath.

Every time I walk into a place, my brain immediately tries to think about how I’d map it. Is the Central Cafe in the middle of Union Station a single entity, or is the upstairs eating area a second floor — even though the structure is standalone? What is the right way to represent the curving staircases up to the second level of shops in the main station?

Every time I walk into a complex place like this, I just want to spend a week mapping out every detail. Where are the stores? What is in them? Can we attach a frontage photo of each? How would you represent the Godiva chocolate — do you mark the path through it as a public hallway, since it’s used that way, or as part of the store?

I want every complex building in the world to have a fully annotated set of data about it, not so I can look at it, but so that I can be routed along it.

It’s not that the technology and approach to do this don’t exist: the data behind things like Bing’s Venue Maps (http://binged.it/ZGAGfT), sold through Nokia as Destination Maps, are typically completely covered for routing. Public spaces are demarcated. Entrances are noted. Every piece of info that you could want is there. But these maps exist for so few places, and they’re so poorly integrated with the rest of the mapping experience.

I’m tired of wandering around for 20 minutes looking for the luggage lockers. Of not knowing where the restrooms are. Of being trapped in the maze that is the International Spy Museum, looking for the way out. (By the way: International Spy Museum — awesome place, definitely worth the price of admission. Great mix of gadgets and pop culture, artifacts and pop culture, plus a huge exhibit on James Bond.) I love these places, but I hate not knowing where I am!

“More than 4230 venues”, says Bing. Well, great, but it appears that you have less than 10 places in Washington DC. No Union Station, no Air and Space Museum, no American History Museum. No National Archives, no White House.

These are the easy places. Every one of these places has a map — most of them produced by the Smithsonian, and if they’re not public domain, they probably would be perfectly happy to help publish the data more. There are 18 Smithsonian Museums in DC alone, and pretty much every one of them has an interior map they hand you when you walk in the door.

I know that this isn’t something that will happen top down, not realistically. This is a case for something like OpenStreetMap, for a crowd-sourced approach. Because I’m tired of “4230 venue maps!” I want them for every strip mall, for every department store, for every place where I’ve ever had to follow a sign to restrooms, asked for directions, and gotten lost anyway.

Until we have that — and until I’m carrying it in my pocket — I’m never going to be able to stop thinking “Damn, I wish that I could sit in here for a week and make a damn map.”

I guess there’s some sort of budget-reducing activity that happened as a result of the lack of a signed budget, which is referred to as the sequester. I was vaguely aware of this, but for the most part, it doesn’t have any impact on my life, so I didn’t really care.

That changed today, when I decided to look into what it takes to go on a White House tour: “Due to staffing reductions resulting from sequestration, we regret to inform you that White House Tours will be canceled effective Saturday, March 9, 2013 until further notice.”

Well that sucks.

(In reality, this has a minimal impact on me: I’m looking to travel about 3 weeks from now, and White House tours require notice through your congressperson a minimum of 21 days in advance. But now I know that even if I had known that and done something about it, I’d still be out of luck.)

I guess sometimes the Federal Government budget does actually have an impact on me (other than via higher taxes). I guess that means I should pay more attention to these things. (Yes, I am a terrible citizen. Whaddya want from me.)

Small World is a game I’ve been interested in since I saw the first episode of Tabletop; it seemed a somewhat complex, but interesting game.

When thinking up gifts this Christmas, Kristan and I decided that it would be an interesting thing to get for Julie.

Today, we played for the first time, and it was fun! It was definitely a bit long for the first game — about two hours — but I think it’ll be a lot quicker the second time around. (It’s also a type of game that I could see working well on a computer — the physical interactions with all the options requires a fair amount of interpretation that isn’t trivially internalized.)

The basic idea of the เว็บพนันบอล ดีที่สุด game isn’t much different from Risk: You have territories. You conquer other territories. As yo are conquered, you lose pieces. Once you lose enough pieces, you pick a different race and start over; the end-game goal is to get as many ‘tokens’ (game points) as possible.

The race choices you have are limited to five at any given time, rotating through a set of 15; each race gets combined with a special power — things like “Flying”, which gives the ability to conquer non-adjacent territories, or “Mounted” which lowers the cost of conquering Farmland and Hills.

Since the race + power combinations are random (shuffle the deck of cards), you can get interesting combinations. Bivouwacking gives you 5 extra defense tokens to deploy as you wish, combined with Halflings, whose initial two regions can’t be conquered at all.

For Alicia, Kristan and I, the game was quite enjoyable and picked up throughout; Julie was a bit slower to pick it up, and lagged slightly towards the end of the game, but that doesn’t surprise me with ~2 hours of back and forth.

For people who like board games, I think that Small World would be a positive addition to any gaming collection, and I look forward to the chance to play with some friends who might be interested in the future.

This morning, a couple of my friends shared their job descriptions in a text editor designed to only allow you to use the 1000 (or ‘ten hundred’) most commonly used words, inspired by an xkcd comic describing the Saturn V rocket using the same conditions.

I tried to do the same, describing my job working on the Local Search team for Nokia:

I try to take the words that people type into their phones and find the places they are looking for. Sometimes people can not type very well, which makes it harder. Sometimes the places that people are looking for are not places we know about, which also makes finding them harder.

I work on finding which places we show people, and which places we do not show people. There are many people who work on this problem. They use computers from their homes to tell me which searches show the right place.

Once we know which places are right, we tell a computer to try to show the right place more often. The computer looks at all the places it knows about, and tries to guess which way it should order the places to make sure as many people find the right place as possible.

Searching for places is a hard job. You need to know about places, know how people type when they try to find places, and put the best place first on a phone.

Edited based on feedback from a friend on Facebook to change the idiomatic use of the word “return”; original.

It was actually a fair amount easier than I thought — perhaps I didn’t go into as much detail as I would otherwise, and the writing certainly feels a bit stilted, but there was no part of my job that I felt I couldn’t describe reasonably well.

Aaron Swartz was an incredible guy. He was constantly successful in making me feel completely inadequate — which is generally a pretty hard thing to do — and I can claim more success in my life than I would otherwise have had thanks to Aaron’s influence.

The world is worse off without him. My best to all his friends and family.

As a result of Aaron’s passing, I am going to change my recent practice of doing many things on Facebook only. Before, I would also have ensured that my content was made available in places that weren’t Facebook, because I felt that the freedom that other platforms offered me — as well as long term stability — were important. Of late, I have not stuck to that ideal — but the fact that I haven’t is a regression from a belief that I have always had, that sharing things only in walled gardens hurts everyone.

I think this is the kind of thing that I would have frowned upon in myself a decade ago, and there is no less reason now that it should upset me. Sharing information only in a single closed platform is bad for everyone. It’s time to go back to sticking to those principles, and making my information as free as it can be. (There are practical limits to anything, but “I’m a lazy bum” isn’t a good enough excuse.)

I get a handful of recruiters who are looking to find me a role in their companies. (Sometimes they are also looking for people who aren’t me to fill roles — which I usually pass on to others by saying “Anyone looking for a job?”, getting a chorus of “Nope”, and moving on.)

While responding to one of these recently, I ran down the checklist I have in my head for what is important to me in looking at a new job. I think that the list of items on this list are essentially a log-scale order of binary predictors for how likely I am to consider a switch to another position; for example, I don’t think that it’s plausible to imagine that I’d consider any position that didn’t have the first two conditions met.

• Work from Cambridge, MA, ideally in a local office or some other employer-sponsored working space. (Things that are close enough: Cambridge, Boston. Things that are not close enough: Lexington, Waltham, Billerica.)
• Working in a working environment which supports flexibility in work schedule, and is supportive of work/life balance.
• Working on projects that I don’t personally consider dishonest or immoral.
• Working with user data — the bigger the better.
• Working on projects which are visible to the public.
• Working on interesting new technologies, especially technologies which can be open sourced and shared.
• Working with maps, or geospatial data.

(Compensation also plays a role, but I don’t think I’ve ever not responded to a recruiter based on that fact.)

I’m not actively looking for a job — despite Nokia’s overall poor performance, I work under the ‘Location and Commerce” group inside Nokia that is still making a healthy profit on our overall activities. Most importantly to me, I work with the same team I’ve worked with for more than 6 years now, so switching jobs would be a painful transition that is unlikely to be enticing without a really strong offer.

That said, I often read the engineering blogs of places like Yelp, Netflix, and Foursquare and think “Man, wouldn’t it be cool to work someplace where maybe I couldn’t put out fires all the time? Where occasionally, I could actually work on cool stuff?” (Note that my brief research into Netflix indicates that it fails *both* of the first items on my list, so it’s evident that “Companies doing cool things” is not synonymous with companies for whom I would want to work.)

I just miss the days of MetaCarta when occasionally, I got to put together something interesting without spending 75% of my time fighting against people inside my own company, and I dream that somewhere out there, there must be other cool companies to work for where that’s not the case. I’m not convinced this isn’t just a ‘grass is always greener’ thought, though. 🙂

(If you are looking for a senior software developer, and think your company can meet all of the criteria above and be cooler than where I work now, feel free to drop me a line.)

An article (Exploiting Hardware Heterogeneity within the Same Instance Type of Amazon EC2) linking to a paper from HotCloud ’12 has some information about mixed instance types for Amazon EC2 machines. I found it interesting, so browsed through the article. Here are some observations I had when looking:

– “Furthermore, the high-memory instances use identical Intel X5550 processors” — Not true, from what I can tell. E5-2665 processors are used across at least us-east availability zones for all m2 instances sizes — m2.xlarge, m2.2xlarge, and m2.4xlarge. In fact, in several thousand instances spun up, it seems that these instances are used up to 70% of the time in one availability zone (though almost not at all in another.)
– The CPUBench test was done across 20 instances, but the Redis test appears to have only been done against one of each type, as far as I can read. I’m not totally convinced — given the variability in performance between node types — that this is entirely explained by instance differences — though given the CPUBench scores, it’s clear that some of the variablity could well be coming from that.

Anyway, I primarily wanted to comment on the high memory instances all using X5550s — since it’s clear that they don’t, at least not in US-East 🙂

The OpenLayers that everyone knows today, born in what seems like the dawn of time of the modern Javascript age, was not born of whole cloth. The early development of OpenLayers recorded in our SVN history represents some of the very very early work in OpenLayers as it is today, but the project had a life for a year before that that is largely unknown.

I’ll admit that I’m not the best person to tell this story: Most of it is also before my time. I originally started working with the MetaCarta team in March of 2006, working on some server-side KML hacking. When I joined the company, there had already been three versions of OpenLayers.

“But Chris!”, the educated illuminati among you might say, “OpenLayers wasn’t released until May of 2006! What do you mean, there had been three versions of OpenLayers?”

Well, my friends, it’s a sad, but not shocking tale, all too common in software development: the premature demo.

After the Where conference in 2005, John Frank reached out to several interested parties to help build an open source alternative to the Google Maps API. (Or so I’m told.) MetaCarta had map based interfaces, and it was clear to John then that this new fangled mapping thing was going to be the future — not just for Google, but for all map interfaces. (In fact, John has even been credited with one of the early definitions of Slippy Map, in June of 2005: “A “slippy map” is type of web-browser based map client that allows you to dynamically pan the map simply by grabbing and sliding the map image in any direction. Modern web browsers allow dynamic loading of map tiles in response to user action _without_ requiring a page reload. This dynamic effect makes map viewing more intuitive.”

Revamping MetaCarta’s ‘enterprise’ UI to be more user friendly was the primary thing on John’s mind. Switching from a form with *11* form fields to a more understandable one-box search. Improving the experience of map interactions. But for a long time, that was essentially all it was: while there was a core idea behind each of these approaches — the idea of making an open source library out of the results, and distributing it widely — in each case, the demo came first.

Instead of concentrating on building a solid library which could be made into an open source base for many different projects, the first incarnations of OpenLayers were all libraries designed for a single application — something that never works well for creating a more general purpose tool.

This was a major misunderstanding of the market demand: one that could not be overcome by any amount of technical success. What the world needed at the time was not another client/server component; it wasn’t another application that allowed them to do pretty things with maps. When OpenLayers succeeded, it succeeded largely because it avoided solving anything other than the most basic problem; it avoided doing anything other than the one, simple thing of having a draggable map on a web page, and being able to load data from multiple sources. This was crucial to the success of the library we know as OpenLayers today.

Some of the flaws in previous iterations that I saw as a result of this:

• Core functionality based around parsing WMS GetCapabilities documents. Although many have criticized OpenLayers for not reading WMS Capabilities documents, reading XML from a remote domain in the browser is intended to be impossible (due to the same origin policy). Though there are now common workarounds for these types of problems, at the time, this was essentially a showstopper for client-side-only deployment: a key missing ingredient in some of the early OpenLayers work. It was only by throwing away capability parsing — by reiterating data in more than one place — that it became trivial to use OpenLayers to talk to remote servers. Note that the problem here has nothing to do with WMS: The problem has everything to do with ‘entirely client side’ vs. ‘requiring a server-side proxy’.
• Centralized hosting of a ‘service’ instead of an API. At one point, there was a thought that one of the things OpenLayers could provide was a ‘mapviewerservice’ — a simple, hosted way to present data online by simply modifying HTTP parameters. (I don’t think this was ever at the core of any of the OpenLayers versions that were written, but it was something we supported even after the transition to the all-public “Mark IV” of OpenLayers.) In the end, nobody at the time really wanted this.
• Concentration on pretty. OpenLayers, to this day, is ugly as sin out of the box, and is more annoying to customize than some other solutions might be. That said, the core functionality of OpenLayers is designed to *hide*. There are very few things that OpenLayers does — and it tries to hide as much of them as possible. Several previous incarnations had a lot of user-targeted UI — making them more applications than libraries. This was a mistake. What the world needed really was a library.

(Now, I’m sure that others who were ‘there’ as it were, might have more commentary. And certainly there were many flawed aspects of technical implementation. But these were biggies at the social level, which would have prevented uptake even if the technical flaws had been worked out.)

This post is written in large part because just this last week, I had a conversation in a bar with someone who claimed he helped start a Javascript mapping project called OpenLayers. We went back and forth for a bit, and then I realized he was right: He *did* participate in something that helped set the stage for OpenLayers. (The earlier incarnations, though usable, never really were the thing that people think of as OpenLayers today.) I just didn’t know he did — and he certainly didn’t know that OpenLayers grew up, got legs, and walked away from MetaCarta and into the hands of thousands of adoring fans.

And I didn’t even know his name before last week. But last weekend, I walked into RPI, where I met a couple of college students from RCOS — people who were still in high school when OpenLayers started — and they knew what the OpenLayers project was, and were excited to meet a guy who helped get it started.

So just remember: Though the OpenLayers you know and love today was put largely put together over a three day weekend, hacking in a darkened room, with a projector on the wall, and Venkman at our side: before we got there, mistakes were made by us, and others. And even before that, a guy with a vision of easier open source maps saw a future where all maps would be slippy.

So a brief thank you, from me, to all the people who came before me in the OpenLayers history; to OpenLayers Mark 3, 2, and 1, and especially to John Frank, who helped push the project from a vision to a reality.