Technical Ramblings

Ever since I had some problems with my server and Crucial Paradigm, I’ve been having too much spam in my inbox. I thought I had the exact same spamassasin config, but either I missed something, or the settings have just gotten less effective.

This morning, when I woke up to check the mail on expired domains from www.spamzilla.io, I finally got fed up and decided it was time to do something about the fact that my inbox was about 90% junk mail which was hiding the essential mails.

I’ve had a Gmail account since around August 2004, but I never use it. However, I do know that Google has better spam prevention than I do: something about having hundreds or thousands of humans acting as bayesian filters helps them. (Mm, scale.) After having recently implement Akismet for spam checking on the MetaCarta Labs weblog — with the result being that about 98% of the 3000 spams it gets every week are now gettin caught — I decided it was time to do something about my inbox spam.

So, I set up crschmidt@crschmidt.net to act as a forward to Gmail, and then set up my fetchmail to fetch from Gmail instead of from localhost.

Results so far are extremely positive:  Of the 85 emails that made it past my local spamassasin filter and into Gmail, 75 of them were spam. Gmail caught all but two. So, instead of 85 messages in my inbox, 10 real, I ended up with 12, and 10 of them were real.

If this stays true, my personal email address may actually become a reasonable way to contact me again. Sorting through the 200 spam I get every day and finding the wheat was becoming an almost impossible task, and this may have at least given me a leg up temporarily.

So, for the past several months, one of my main projects has been working inside of MetaCarta to put together the tools neccesary to let our customers integrate MetaCarta search results into all sorts of web pages inside their networks. Over the past several weeks, my main project has been taking the work we did in that regard, and making it available to the public.

So, if you’ve ever had an interest in finding out what MetaCarta does, the best way to find out is to:

• Check out MetaCarta Web Services
• Create an account
• Follow the step by step tutorial on adding a MetaCarta geographic search interface to a web page

Sure, it may not seem the most glamorous thing in the world, but we’ve definitely put a lot of effort into making these tools available, and I’d love to see some people try it out.

I can’t seem to find any places that I can get to that have many slrn users — I’m looking to figure out if it’s possible, in the article listing, to ask slrn to display already-deleted messages.

Anyone have any advice? Or tips/tricks on slrn in general?

Went to BarCampBoston today. Took Schuyler, Jo, and Gregor with me. Had a great time, the venue provided by Monster was absolutely incredible in a lot of ways. I’d really like to thank the people at Monster for their time and effort pulling things together: everything at the conference *just worked*. (It’s like it was a mac or something!)

Schuyler and I gave a presentation on mapping on the web, centered primarily around OpenLayers and the uses for it:

Ever since Google first ‘solved’ interactive web-based mapping for the masses, providing an API to put your data on top of, people have explored the space at an ever increasing rate. From Google Maps Mashups to all new datasets created out of the existing data, and hacked into that ever-lovable interactive mapping interface, people are creating new, exciting, and sometimes even innovative ways of looking at existing data.

Come discuss the things you’ve done with maps, the things you’re doing with maps, and what you think of the ‘mapping revolution’. See what alternatives there are to Google… and why you might not want to be quite so dependant on the corporate data silos.

The links which were shared during the meeting are available from my BarCamp Presentation page.

A bunch of people have emailed or commented here with things that they think I should be checking out. They’re probably right. However, I’m short on time, and working on a project heavily at the moment, so if you’re waiting on an email from me, please be aware that I probably saw it, I just haven’t gotten around to actually answering everything yet.

I’ll get back to you soon, I promise.

I’ve just implemented a fair amount of memcached support into CGI::Wiki, the Wiki software that powers OpenGuides. Now, any time a node is fetched, it will come from memcached if possible, or be loaded and stored into memcached, and when nodes are edited, the cached version is deleted.

This is in addition to adding caching to the main map pages, which were previously taking up to 20 seconds to load, and are now in the ~1 sec/range. (There is much more slowness in the browser side of this page, due to the javascript proccessing time required to load hundreds of location markers).

Additionally, the Open Guide to Boston is now running under mod_perl after my Message to the OG-Dev list with a patch that fixed the major stumbling block. This has increased speed somewhere between 5 and 10 fold over using CGI.

About 15 minutes ago, I restarted memcached to clear out old stats. I then visited the main index pages for both the Milton Keynes and Boston guides, to prefill the caches. There were 3535 items which were loaded (GET\_MISSes). In that time, there have been 4 new objects added — and 946 GET\_HITs. With the stability of memcached, I expect that this cache will be able to run for a long time with no maintenance, and I’ll be able to come back in a few weeks to see that memcached has saved me thousands of hits to the mysql database, therefore increasing the speed with which the Open Guide to Boston loads, something that I’m sure everyone can appreciate.

Most of the slowness that memcached is fixing in this case is not caused by MySQL being slow, but rather by the processing done by OpenGuides after the fact, due to the way that it is designed. The software was never really built to be the most efficient Wiki software out there, but to scale to the extent that it needed to, a task which it has completed admirably. However, this has left some corner cases where the performance was less than perfect — typically cases which were seldom used. The example of the “All nodes” map that the guide has is a good one: I built that page out of a corner case (the ?action=index view) which is not linked from the guide itself. As a result, I discovered a case where the software did not perform as expected. There was a roundtrip to the database to load the data about each node. This round trip time combined with a complex map { } function brought performance way below acceptable levels. So, I sought to improve that. First I simply cached all the data after it was processed, and today implemented caching of the individual nodes. Although the second change is minor and may not offer significant benefits on the small scale, it does help to build up the larger cache that it’s supporting: Roundtrips to memcached are quicker than roundtrips to the database.

About a month ago, I participated in a couple of meetings at the Berkman Blog Group regarding Wikimania, the Wikimedia annual conference, which will be held this year in Boston. Talking with the local organizer, I was interesting in helping in a number of ways where I’m more useful than many other people — having a place to stay and possibly offer to other people, as well as technical skills which not everyone has that might be of assistance.

During the second meeting, I had the oppourtunity to engage with other Wikimedians who would be participating, and to get some of their feelings about the conference and the Berkman Blog Group’s attempt at trying to make things easier for them here in Boston during their convention.

I got a negative vibe from one of the people I spoke to, and later attempted to determine whether my understanding – that Wikimania was dedicated to Wikimedians, and local help in a significant way by people who were not active Wikimedia participants would be appreciated only so long as it didn’t affect the conference – was correct. Although I don’t have an exact quote of my question and the answer, the feeling was, again, negative.

The local organizer here asked me to clarify why I felt pushed away by the Wikimedia participants I had talked to, and this is what I wrote in response:

When I asked if it was the case that the purpose of Wikimania was to further the goals of Wikimedians, and that local people were not being invited to participate in the same way, I was informed that yes, this was the case: assistance from non-Wikimedians was unneccesary. (Sadly, I don’t have the log of this due to a client crash, because it’s the one thing that was said more than anything else that convinced me not to participate.)

Wikipedia has always had a very distinct aura of excluding outsiders. Although the wiki-nature of Wikipedia would seem to act as a counter to that point, many friends who have in the past been heavily involved in Wikipedia have left due to issues relating to negative personal interactions between themselves and other contributors. I have always held Wikipedia at arm’s length due to this, but given the local oppourtunity presenting itself, I thought that maybe I could be convinced otherwise.

What I stumbled into was exactly what I would have been led to expect – an elitist attitude taken by core contributors towards anyone who is not one of the “good ole boys”. Although you were extremely positive towards external help from the people on the ground, every other person I spoke to who in any way represented Wikimedia only left me with a bitter taste in my mouth.

I was hoping that Wikimedia was really a good group of people, done wrong by the masses who consider themselves to be “better than you” because they participate more. Instead, I found out that that attiude is perpetrated all the way to the top, at least from my external point of view.

As a result of the general vibe and the specific statements requesting that non-Wikimedians not take part in the planning and activities surrounding Wikimania, I’ve decided simply not to bother.

Another friend of mine put it best: “The only way to participate in Wikipedia is to just edit, and ignore all those people behind the ‘Discussion’ link.”

Something I was previously unaware of:

The reason for the two different variations of “and” and “or” operators is that they operate at different precedences.

PHP’s && vs. and have different precedence? Who thought this was a smart idea? What end or purpose could it possibly serve?

Yet another one of the things that just makes me smack my head. It does explain a few things — why I couldn’t do variable setting as I might in Perl, a la:

$var = $test || $default;

In this case, the test occurs first, setting $var = 1; Had I used:

$var = $test or $default;

All would have been fine.

How can someone think this is sane?

Today I did some work improving one of the Ning example apps, making it work faster, better, stronger. The app is Bookshelf, and it’s a ton of fun to watch. Basically, users add books (via the Amazon API), rate them, and comment on them.

Some aspects of the app were running way too slow, since it was designed and tested when the system was under no load at all, a situation which makes load testing very difficult. I won’t go into the technical details, but the long and short of it is that I took the book adding process and quartered the time it takes.

Apparently my changes were important, since Gina blogged about it. Although I’ll admit I had no idea who Marion Jones was until I Googled it, it sure made me feel special.

Gina also passed along the good news to me that helping out other projects in the free time I’ve got isn’t a problem from her end – so long as Ning projects come first. So, if you have interest in working to get an app running on Ning, but don’t have the spare time to learn all the ins and outs, feel free to drop me a line.

This is such a freedom after my last job, where doing any work outside the company was considered treason. It simply reassures me, yet again, that taking the job working for Ning is the best thing that has happened to me since I have left school — I’m working on fun projects, in a forward-looking environment, with great people, and a great management team above it all. While wedu is busy building the next version of We Don’t Get The Web, I’m helping to build the future, and I’m having the time of my life doing it.

Now to get some of my other apps up to snuff: a recent change in the core means that I can get access to pages without the use of cookies, which means it’s time to write that location updater client for my cell phone.

Updates as they come!

I’ve got a number of books on my “To read” list, most which were given to me as part of my birthday or as random gifts. One of them which was gifted to me by Tom Croucher from my Amazon wishlist (as a thank you for help in his dissertation, I believe). The book is Where Wizards Stay Up Late: The Origins of the Internet, and it’s one of my favorite non-fiction computer-related books.

The book is a relatively detailed study of how the internet came to be: the development of the theories behind it, the actual hardware, the proposals from the Defense Department for the creation. The origin of RFCs, the way TCP/IP was invented, things like AlohaNet and ethernet as well.

It’s because of this book that I still carry around Vinton Cerf’s business card, which I obtained at a 10th anniversary discussion of Mosaic. Ever since I acquired the card, I was fascinated with the design that was printed. I then planned on getting some for my business from www.dxprintingperth.com.au with exactly the same layout. Cerf was there discussing the idea of an interplanetary internet and how it would work. I was there drooling at the fact that I was in the same room as Vinton Cerf, and actually got up on stage afterwards and shook his hand. I keep that thing well – it used to be in my wallet, now it stays seperate – because it means that much to me, and it was inspired by this book.

So, a big thank you to Tom for the book, and a suggestion that you all go out and read it.