Technical Ramblings

In 2002, I got an account on LiveJournal. In order to help my friends get accounts, I started doing LiveJournal support. (I got 5 support points in my first week, which was a reasonably big deal back then. (Warning: Reading other posts from that era of my life is not recommended :))

Over time, as a somewhat more technical member of the support team, I started attempting to explore links to this ‘zilla’ thing that people kept mentioning, LiveJournal’s bug tracker. However, I always found that I never really understood what was going on: looking at the content above the fold in a typical Bugzilla bug is a bunch of confusing looking metadata to a new developer. I remember at some point when someone told me to look at the comment — and I remember saying something along the lines of “Wait, there’s more content in this page if I scroll down?” Yes, I had been reading bugs for weeks, and never realized that … there was information other than the metadata.

I remember somewhat later, as I got more into development, that I would actually read patch files and attempt to understand what they were doing. In one case, I found a patch file written by a friend in the support community that had a pretty clear typo in it. So, I downloaded the patch file, opened it in an editor, added the new line that was missing, and added the “+” at the beginning of the line — like he had for all of his lines, of course — and re-uploaded the patch.

Of course, anyone with a passing knowledge of tools will know that adding a single line to a patch file … isn’t going to go well. 🙂 But to someone who didn’t know that patchfiles were produced and applied with ‘patch’, ‘diff’, etc., this concept was a strange one.

So, in my start in open source, I didn’t know how to read bug reports; I didn’t know how patchfiles worked. Yet I still was able to learn, improve, and eventually, do a little bit of good in the open source world, thanks to the help of a lot of people along the way.

We all start somewhere.

This weekend, I sat down and got comfy with Scalatron, a programming tutorial based around building a competitive bot in a game environment in Scala.

At first, I was a bit stymied; with no Java-friendly IDE, and the instructions being really IDE targeted, I had some problems getting my development environment set up. Thanks to some help from godwinsgo on Twitter, I got set up with an environment where I was able to have:
– Automatic compiling on file save, which was super helpful (rather than having to do something else)
– The ability to quickly swap back over, tap a button, and have Scalatron reload my bot and test it right away.

After some playing, I wrote a bot that is pretty good: it gets ~22000 in the ‘reference’ (bot on its own, 5000 steps, 100×100 field). I have a better understanding of some of the simple mistakes that I made — adding a println at the end of my function when I have a return type expected will return a somewhat cryptic “Expected Type, Got Unit” error message pointing to a spot in my code that it doesn’t make sense to me; same with really any case where I’m not paying careful attention to return types, since without an explicit ‘return’ statement, what gets returned is confusing to me.

I probably wrote terrible Scala; I didn’t bother to learn much about the language, just enough to actually hack my bot into something that worked. (My bot is on github, if people want to critique my terrible Scala — or code in general.) However, I was able to get working, learned some things about Scala, and wrote a cool digital robot. (Hooray!)

Thanks to David Winslow for giving me the bits to running, and to Scalatron for helping me have a fun project for this weekend!

“When a junior developer has decided they need to build a whole new framework to solve a relatively simple problem, and they’ve started building it instead of fixing the problem, that’s an unpleasant surprise. When they’ve a week in to it, and have been making difficult-to-undo changes to support it, that’s a nasty surprise.” — Scrum, the Good Bits: Daily Standups

So true. And it points to Picture Hanging — a metaphor that a couple years ago would have been completely foreign to me, but is now so much a part of my daily life that I take it for granted…

So, in general, I’ve avoided multithreaded processing; it’s one of those things that historically has been tricky to get right, and I don’t typically have embarrassingly parallel problems.

Today, however, I was parsing a set of 25,000 HTML files using BeatifulSoup, to pull out a small set of data (~500 bytes of JSON per HTML file). I briefly tried to simplify some of the code, but then realized that the lion’s share of the CPU time was being spent on the initial parse; there wasn’t going to be a way to clean up my code enough to make the script that much faster, no matter how good the rest of my code was.

Enter multiprocessing. With a 5 line change to my Python code, I was able to move from one core to four. Instead of:

def handle_place():
     for i in glob.glob("beerplaces/*"):
           # Do stuff with I
     return data

I have:

from multiprocessing import Pool
def handle_place(filename):
    # Do stuff with filename
if __name__ == "__main__":
    p = Pool(4)
    data = p.map(handle_place, glob.glob("places/*"))

Once I made the change, I went from using one CPU fully to using all four — and instead of taking 25 minutes to generate my output, the total time was under 7.

multiprocessing is cool.

One revelation I had yesterday, when participating at BarCamp is… I still actually care about maps.

Now, those people who observe me probably know this. Although I have moved out of the map creation business — my day job is no longer hacking on OpenLayers, though amusingly, I actually end up *using* OpenLayers more now than I did most of the time at MetaCarta — I still enjoy maps, and still work in a field where maps play a big role, for example. However, I’ve felt for a long time like I was burned out on maps: After 5-6 years of working on the same thing, I was tired of the same discussions over and over again.

Turns out — that probably isn’t the case. Instead, what I got bored of was dealing with questions about Javascript-based web mapping. I got tired of only existing as “The guy people ask OpenLayers questions to”; I got tired of being a one-trick wonder, even if that trick earned me the title, for a while, of “Boy Genius” (literally: it was on my business cards).

I ran into Bill Wendel at Barcamp Boston yesterday, and when I pulled out my array of phones (Nexus S, Nokia C7, Nokia N9, Lumia 710), he commented that he remembered me talking at Barcamp Boston last year about the need to constantly reinvent yourself: if you’re not learning something new, you’re falling behind. I kept one hat on for what was — for me! — quite a while: Web mapping technology was the core of my existence in a lot of my development for 5 years. (Also changed in those 5 years: Got married. Child moved from elementary school to applying to high schools. Moved to current apartment. etc.) 5 years is a long time. A really long time. Being “the OpenLayers guy” just isn’t cutting it anymore.

But the mapping world still interests me: even today, it seems I have a lot of information in my head that isn’t ‘common knowledge’ about the mapping industry. (For example: Yahoo’s mapping platform is almost entirely provided by Nokia these days; something that most people aren’t aware of.) Even if I’m no longer as actively interested in doing the software development side of things, I still love talking maps — I love sharing information about how the world used to be, and how we’re seeing it change.

To me, this is news. I know that this is probably a sign that I’m insufficiently good at self-examination, but I really just thought that I was ‘done’ with Maps: finding out that indeed, I’m not done with maps, but just with mapping software, is an interesting revelation. I’m looking forward to Bill reaching back out and helping build up a group of people in Boston interested in mapping again. Back in 2006, we used to meet up regularly — and this was the early days of OpenLayers, so the meetups were more informal, and less code. (This is before we were part of OSGeo, for example.) As I moved more into the software development side, so did the meetups — but I realize that in doing that, I missed a key factor: the software of maps isn’t the interesting part. The *stories* matter. Software is the boring part. Leaving out the users and targeting the developers was the wrong way to go, and I want to see if going back away from that opens more interesting doors… again.

These days, the things I do have changed pretty drastically from what they were a couple years back. Since I’m attending BarCampBoston this weekend, I figured I should make a So, here’s a collection of things I know and do:

• Django: I have worked with Django since long before 1.0, using it for many different projects. The longest running project type I have used it for is managing distribution of Mechanical Turk style annotation tasks among a large pool of annotators; At MetaCarta, this was ~10-15 people; at Nokia, it’s more than 140 part-time workers around the world, producing thousands of responses to task judgements each day.
• Data Analysis — At Nokia, I also help do analysis against logs for the Nokia Map Search service, analyzing client trends, usage, and extracting information from search and click logs to help improve the overall quality of Map Search
• Web Development — a bit part of what I do at Nokia, as it has been in every job, is the creation of web UIs to make information more easily accessible. Typically using simple, plain HTML + Javascript interfaces, I work to make it easier for other people to get their job done by presenting information in a more easily accessible form; this can include anything from basic data aggregation to more complex map related display interfaces.
• AWS — At Nokia and outside of Nokia, I have done a fair amount of work around Amazon Web Services, using everything from EC2 to Cloudwatch to SNS to run services at a larger scale than I have before. Using Amazon and other ‘cloud’ services has been a big step forward in my knowledge of software deployment and development.

At Nokia, I work primarily on improving Map Search. This runs the gamut from annotation data and measurement to deployment — if there’s a part of the Nokia Map Search backend that I don’t know at least a little bit about, I probably will soon. At home, I dabble in photography, phone geekery, and various random hacking projects. I’m no longer quite the Open Source GIS guru that I was once — I’m no longer the go-to guy on questions about OpenLayers, for example — but I can now tell you a lot more about AWS than I could in those days 🙂

So, if you meet me at BarCampBoston and get to my blog: now you know. I’m Christopher Schmidt, and I’m a web hacker.

My daughter was recently accepted to the Emma Willard School. I’m glad for her getting the opportunity to be in a welcoming, academically rigorous all-girls environment, with what seems to be a huge dedication to fine arts. I think that it will be an excellent environment for her, and I look forward to the opportunities this will afford her, now and later in life.

That said: holy crap, this kid is going to live in a *frickin’ castle with gargoyles*. I hope that she appreciates it!

One thing I never knew was different between the Midwest and the East Coast until I moved here: the slicing of hot dog buns.

Where I grew up, hot dogs had smooth sides — like the outside of a loaf of bread. The slice was cut in what I think of as the ‘top’ of the bun — the bun was rounded on three sides, and looked like a loaf, except on the bottom, which was white, bread-like material.

In the midwest, they’re sliced (or made entirely) a different way — the sides that you put your hand on are not smooth, outside-of-load like material, but instead bread-filler-like white material, with the loaf-like stuff on top — looking more like two pieces of bread.

The words for these appear to be “side-sliced” (which is what I grew up with):

and “top-sliced”:

I just never knew that hot dog buns could be sliced a different way.

Last night, at my weekly Drink With Geeks event at Grendel’s Den in Cambridge, I enjoyed the start of my birthday with a lively argument about whether the Open Source development model is fundamentally superior for most use cases to proprietary software. (I’d shorten it here, but I realize that it really deserves a full post.)

I finished it off with a free shot of the Bartender’s choice — which happened to be Jameson. So far, in my 28th year, I have learned one thing only: Holy crap do I not like Jameson. (But hey, gotta try new things!)

Happy Birthday to me!

Most of my reading on the internet is quick; 6 paragraph news articles, compressing down into something I can read in less than 5 minutes, because I seldom have the time to spend longer than that. I do read in my spare time, but it’s usually 1950s-1970s sci-fi — not non-fiction news articles.

This morning though, I was linked to this Slate article about some of the best stories about the early computer industry which led me to a 1983 Playboy interview with Steve Jobs — and I’m amazed at some of what is said there.

I wasn’t one of the mourners at Steve Jobs’ death — I just didn’t have a lot of interaction with Jobs as anything other than a marketing personality. I got into computers in the mid 90s — my first home computer was Christmas of ’93, and the Mac, although ever-present in my early educational career, was always a slightly dated concept at best. It was never a core part of my life, and although I’ve been using Apple for my laptop hardware for the past 7 years, I never really bought into the Cult of Mac the way some people did.

Reading the Playboy interview though, I was struck by how much, in 1983, Jobs’ role and interactions in Apple played out exactly as he thought they would. His idea that IBM PCs, if they succeeded, would limit innovation in hardware for a “Dark Age” of 20 years, though not exactly spot on, is probably something that he would say came true — and that Apple, with OS X and later with the iPhone/iPad revolution, was really exactly the kind of thing I can see as being a fulfilled vision there.

To the rest of the world, the hardware revolution stopped mattering for a long time in personal computing — or at least, I think of it that way. Comparing the computers of today to the computers of 2000 — hell, even the computers of 1995 — didn’t innovate very much in changing user interactions. The “smartphone” started to make that change in early 2000s, a bit, but I think that the iPhone and iPad have really changed computing in a fundamental way for a lot of people. (Out of that has come other technology — like the Kindle — that is also changing the way that people interact with computers, in my mind.)

There were also a number of other things he talked about — like weaving in and out of Apple as he continued his life, or like predicting the death of what I guess were the other significant computer manufacturers then (Radio Shack, Wang, TI, Xerox, etc.) — a fact which shocked the interviewer.

It really changes my perspective on Jobs, and to a lesser extent, on Apple as a whole, to see that 30 years ago, Jobs could see pretty much exactly the way things worked out. It’s a pretty weird thing to read to me.