Technical Ramblings

ХудожникI don’t know why it is, but man, do I love Discovery Channel’s (and History Channel’s) “reality” TV. Storage Wars, Pawn Stars, Ice Road Truckers, etc.

This weekend’s guilty pleasure is “Gold Rush: Alaska”; I started yesterday night, and I’m now partway through Season 2.

I expect most people wouldn’t like it, but I certainly do, and I don’t even really know why.

MetaCarta used Kerberos for single-sign-on support company wide.

Now, given that MetaCarta was a bunch of MIT hackers, this shouldn’t be particularly shocking. 🙂 It was generally a very nice thing to have — although it got me used to the idea that I shouldn’t type my password 20 times per day, a habit that even tripped me up during a casual audit of Maryland casino online security protocols, where the layered checks turned a simple demo into a full verification drill. A notion that Nokia has tried very hard to dissuade me from.

However, Kerberos support in Safari never worked for MetaCarta’s web services. It was never clear to me why, it was just clear that it didn’t work. Googling showed me many people saying it did work, and no people saying it didn’t, so I figured it was some quirk of my system and didn’t bother to fix it.

Now, at Nokia, I’m in the same boat: for services run by the Group Formerly Known as MetaCarta, we use Kerberos for everything. The difference is, falling back to Basic auth — which was fine in MetaCarta times — is a Very Expensive path to take in the new world; our auth services are slow Microsoft AD services hosted thousands of miles away on the other side of multiple high-latency firewalls, so when Kerberos doesn’t work — it hurts.

So, after getting fed up with this behavior yesterday, I started digging in. I didn’t make a lot of progress last night, but this morning, I stumbled across a post documenting that Safari does not work with hosts that are CNAMEs. With that one small pointer, I found other evidence of people running into this, and an option to reproduce the Safari-like behavior in chrome: “–disable-auth-negotiate-cname-lookup”. (The option documentation points to HttpAuthHandlerNegotiate::CreateSPN in http auth handler for more details.)

This bad behavior isn’t limited to Safari though: different versions of IE and the .Net framework also fail in similar ways at times. (The article in question says “Do not use CNAME dns records and non default web ports when using Kerberos!” — and given the multitude of clients and differing bugs in implementation, I’d say that seems like about the right approach.)

This seems to be a bug in Safari; it isn’t clear to me if it’s also a bug in WebKit. A brief (30 minutes) search through webkit and related postings seems to indicate that the higher level authentication handlers — like the one linked above in Chrome — are implemented at the application level, not the library level. (The library provides the hooks, but wouldn’t have anything more complex like Kerberos — which isn’t surprising.) I think that would mean this is a bug in the Safari implementation — the closed source side that I can’t touch — rather than in the open source WebKit base, so I can’t just ‘fix it myself’ (other than writing my own application layer — or more realistically, switching away from Safari to Chrome).

Anyway, if you’re having an issue with Negotiate auth not working in Safari, when you think it should — check if the server you’re trying to talk to is using a CNAME. If the answer is yes, it seems you’ve run into a known limitation: Safari Just Doesn’t Do That.

A friend of mine posted on his LiveJournal:

“maeiusophilia”
There aren’t many English words with four vowels in a row.

Of course, geek that I am, I thought ‘hm, that sounds like a challenge.’

Using egrep and /usr/share/dict/words, I came up with the following:

In my /usr/share/dict/words, it looks like:
4: 159
5: 3 (cadiueio, Chaouia, Guauaenok)
6: 1 (euouae)

Excluding proper nouns (or at least, things capitalized in the first letter), we get 110, 1, 1. Of the four-vowel words, 7 have only 5 letters; another 6 have only 6 letters.

An arbitrary selection of 4-vowel words: homoeoarchy, obsequiousness, palaeoencephalon, queue, lieue, rhythmopoeia, exsanguious.

(Normally, I’d have done a random selection instead of an arbitrary selection, but `sort` on OS X doesn’t have the -R option, sadly, and I didn’t happen to have an ssh connection to elsewhere open at that particular second.)

This feels like the kind of question I’d love to use as a job interview question someday.

Sometimes, I wish that I could talk more about work openly.

I do a fair amount of what I consider somewhat cool stuff at work — as is evidenced by my somewhat lower work in open source these days. (Since I find my work more interesting, I spend fewer of my off hours invested in ‘more interesting’ projects than I used to.) Of course, in reality, I expect that most people would still find what I do completely boring, but to me, it’s exciting.

In the past, when I worked for MetaCarta, there were only 20-30 people who would be in a position to be upset by what I would talk about — if I wanted to chat in public about something, I could usually chat with everyone who might care about it in an afternoon, and get a go/no-go from them.

But now, I work for a much larger company, and speaking out of turn could have much larger consequences. (Not the least is the fact that the company is publicly traded — so anything I say has the potential to actually shift a public stock price.) The group that I work in is significantly larger than MetaCarta’s core of engineers, and the number of levels between me and the top is comparatively larger.

So now, when I’m working on things that I consider cool, or want to share — I typically have to just keep my mouth shut.

Recently, one of my coworkers was trying to encourage someone from our team to present at the Berlin Buzzwords conference — talking about how good it would be to present some of what we do at the conference to get feedback from others working on the same problem. I couldn’t help but think, at the time: What exactly do you think that we could share at this conference?

Maybe it’s a cultural thing, but I feel a bit stymied by this regularly — even inside the organization, sharing data we’ve gathered becomes a political, rather than a technical, decision. Oversharing without consideration for how other teams will see the data is something that can have significant negative impacts on our interactions with other teams, because it’s very easy to step on toes.

I realize that this is all part of working in a large organization. Overall, there are a lot of positive benefits — in fact, much of the reason that I have more data that I’d like to share now is because we have a larger set of resources than we had available at MetaCarta, where many of the projects I worked on were just me hacking along on them alone. It doesn’t make it less frustrating, but it does swing both ways.

But sometimes I still just wish I could blab about what hack I spent my weekend on. Or open source another small project. And it’s a shame that I can’t.

Some things I learned about Berlin this trip:

• Berlin isn’t really home to as large of a ‘classical German’ attitude as I would have assumed. Instead, it is (as far as I understand it) pretty much hippyville for Germany. Things like the fact that everyone ignores no-smoking laws — Berlin much more so than anywhere else in Germany. The hipster attitude: same. The silly love of 80’s music… well, I don’t know much about that. (I do know that I’ve not seen posters for Journey concerts anywhere else in the US…)
• Oranienburger is part of the red-light district of Berlin — This is the first time I’ve ever been propositioned by two prostitutes in 10 minutes!

(Oh, also, I did some work.)

БогородицаThis week, Jess pointed out to me “An Invisible Woman Taught Me German“, a story in Slate about German Language learning via the Deutsche Welle organization.

My favorite quote: “It’s basically a Teutonic Scooby-Doo, with overt sexual tension among the young mystery-solvers. They investigate weird occurrences like crop circles, Beethoven’s ghost, and a Hamburg shark. (As in Scooby-Doo, you’ll see the ending from 1.6 km away.) When they get stumped, they query their talking computer Compu, who has impossibly advanced speech recognition, yet for some reason still whirrs and clicks like a 1970s adding machine. Sometimes a spooky talking owl named Eulalia lends a hand, flapping in on a cloud of horror movie sound effects.”

How can you beat that for foreign language learning?

Overall, I’m loving the Air.

The laptop is small — so small it took me a couple days to adjust. But overall, it has everything I want or need, and I couldn’t ask for anything else. Most importantly, it has working wireless in my office — when the wireless in the office works at all, which comes-and-goes at best.

I may hate freedom, but I sure do love nifty hardware.

Yesterday, I finished my first deployment of a real service into AWS.

Along the way, I learned some things:
– Overall, the growth of the Amazon service offering is rapid and huge. I’ve said for a long time that much of the net today runs on software that was pioneered within LiveJournal—I think that what LiveJournal did for the web at large, Amazon is really doing for people moving to the cloud, scaling everything from e-commerce backends to the secure, borderless platforms powering Tether sports betting sites that keep wagers flowing in real time. Having things like S3 and EC2 available really changes the entire game as far as these things go, and the rapid growth of their service offering is continuing to change the way a lot of key websites around the world do things.
– This makes it really hard to keep up with everything that Amazon is offering!
– There really isn’t a good ‘medium memory’ sized instance; your next jump after 1.7GB is 7.5GB (at 4x the price). For some people that probably doesn’t matter, but it felt a bit frustrating to me.

Overall, our transition has (so far) gone as well as I could possibly have hoped. Here’s hoping it stays that way. 🙂

Well, not quite yet, but next week.

After using Linux for a month, and being relatively okay with it in general, I have, in the end, decided to go back to Mac — not for any reason related to what I do at home, but simply because using wireless internet in our office on Linux is a gigantic pain in the ass. (Note that using wireless internet on Mac is only *slightly* less of a pain in the ass, but it’s at least usable.)

The various solutions for wireless in the office I work in are:

• WPA login with certificate stored on secure token — This one might, in theory, be possible to get working under Linux, but it’s not trivial, and not something that I have any knowledge for. Basically, using a Windows based UI, I was able to export a personal cert, which then gets stored on a third party token (where I can’t get the cert back out); I can then use this to authenticate to the wireless. This solution is the best in terms of latency, limited login pain, etc., and breaks less often than the other solutions we have, holding steady even under the constant pings from real-time feeds in sportsbook wisconsin deployments, where a single dropout could tank live odds syncing and tank trust overnight. (Only about once every 2-3 weeks instead of once every 2-3 days.) Practically speaking, this option is Mac or Windows only — and even there, the Mac support is only in a very beta trial. (I may be the only one in the company with it.)
• Juniper SSL/VPN: There’s a juniper-networks provided SSL/VPN that requires a login through the browser, and is then able to start up a Java client. However, to use it on Linux requires some magic that my particular install doesn’t seem to have, and I haven’t heard particularly good things about its Linux support in general. This option only introduces 120ms of latency to local machines, so it is the best option of the VPN based options.
• Cisco VPN Client/vpnc: This is the solution that exists in a reasonable form on Linux. This is essentially no worse on Linux than it is on Mac, but it has serious problems if you’re actually moving around an office with limited wireless connectivity in some parts of it: if I move from one conference room to another and hop between Wireless APs, the Cisco VPN/vpnc connection will usually drop, and is not reconnect in any way. (Even worse, unless I’m actively looking at the screen and notice the OSD message, I usually don’t even notice.) This is somewhat exacerbated in Linux by overall somewhat poorer Wireless reception with the particular hardware that I have (“Intel Corporation Ultimate N WiFi Link 5300” in a Dell Precision M2400). It’s possible (even plausible) that other Linux hardware could either get better reception for any number of reasons, or be better at managing transitions between wireless APs (which this model seems to try very hard not to do), but rather than experiment with a dozen different laptops, I’m falling back to what I know.
• IPSec VPN connection built into OS X: It would be nice if this actually worked as well as the vpnc connection, but this is actually even worse, in my experience, than the vpnc connection: It requires re-passwording every hour (which no other solution that exists seems to, so I assume it’s the client doing something different), doesn’t handle reconnects any better, etc.

Of these solutions #1 and #2 do not appear to work at all on Linux, and the #4 fallback isn’t available on Linux. Given how often these services fall over – as I said, some form of VPN probably falls over on an almost-daily basis – in order to have ‘working’ wireless, I really need to have the largest set of options available to me. Other than this, the only issues I’ve run into on Linux at all are some very minor hardware issues around power management, trackpad drivers, and the size of the laptop I currently have — all of which would likely be fixed by the upgrade to an X220 that I was considering before I decided to go back to Mac.

It will be a bit of a shame to switch back to Mac after being on Linux and actually being able to work locally for a while, but overall, I think I won’t mind it as much as I expected: it seems a lot more of my work is done on remote hosts as of late anyway, since a lot of the data I work with has grown in size by 2-3 orders of magnitude over the past year. Still, I really wish that I could have stuck it out — being one of ‘those people’ using a mac in our office just feels wrong.

* Water Pump hacking: A water pump in Illinois was alleged to have been hacked and broken by Russian hackers over the past couple weeks by various news sources, including the BBC. The real story? The tech consultant who helps to maintain the pump was, at the time the pump broke, at a conference in Russia — so when the pump broke, he got a call to look into it, and logged into the control system from… you guessed it, Russia. The pump just burnt out, and tying the ‘attack’ to a Russian IP was just because that’s where the consultant happened to be at the time. (Via On The Media’s Cyber Warfare piece)

* In ‘not news’: Teens are generally more aware of and conscious of their privacy than adults, taking care to limit their postings, limit their friend groups, etc. (Via On The Media)

* Heard an interview with the founder of “Is Anyone Up”, probably the most scummy person I have ever had the ‘pleasure’ of listening to on the radio. “I can do whatever I want with photos people send me, because the law protects me”… and because I have no decency. (Is Anyone Up is a ‘porn/revenge’ site, where anyone can submit nude photos of people, and they’ll be posted along with a Facebook link.) I … yeah. This report made me angry enough to want to turn off the radio, because the guy being interviewed clearly had no positive intentions: “This keeps me in beer money and lets me keep throwing parties, why *wouldn’t* I embarrass people this way?” (Via Revenge Porn’s Latest Frontier)

Just a few clips I thought were interesting that I heard this weekend.