Technical Ramblings

Desperate times call for desperate measures.

After doing some log watching yesterday, I realized that almost 25% of the hits to my site were people attempting to spam a now-defunct weblog with trackback spam, as well as using spam referrers. After watching this for a while, I got pretty tired of seeing all these ads for poker and so on, so I started going on a rampage. No more Mr. Nice Guy, I thought. These machines aren’t even hitting valid URIs: they’re spamming something that hasn’t even existed for the past two months. I’m tired of it.

So I started blocking them. Each and every IP that came in bearing trackback spam, spam referer headers, or anything else was no longer going to touch my server. Unfortunate, yes, becase many of the machines which are doing these things have owners who are completely unaware of it. In some cases, the IPs may have been dynamic: there was more than one comcast IP address in there, and I’m pretty sure they don’t give out completely static IP addresses all that often. So, it may be that in blocking this spam, I’ve ended up blocking some legitimate users of my site, and that sucks.

However, I am very tired of deleting spam. I am tired of sorting through crap. I am absolutely tired of people talking the problem to death, because until the people doing these things actually start getting in trouble, nothing at all is going to change. The only way to stop people from spamming, be it email, weblogs or what have you, is to get to the people behind these attempts, and make it unprofitable. Sue them, charge them, whatever, but make it so that if they try and go about it in a way that no one thinks is right, they lose money. Because spammers still make plenty of money on spam. If they didn’t, they wouldn’t keep doing it.

Overnight, I received 90 trackback spams. Approximately 15 of these were new variants on spellings and so on to the point that they made it past my spam moderation filters. As a result, when Iwoke up this morning, I had a major spam problem to deal with. (Luckily for me, it’s my day off work. Unluckily, I’ve been spending most of it fighting spam fires. Annoying, really.)

I’ve turned on comment moderation. All comments must be approved by me. I’m not dealing with this stuff anymore. There is no no way that a spammer can get a comment onto the site and public without me seeing it first. I don’t like it, it’s annoying, and it’s troublesome to comment authors, but it’s worth it to me to be able ot know that no one is going to get a higher search engine ranking because of this site. I’ve started dropping access to any IP address which sends spam. If someone can’t get to my site, hopefully they can email me or something, but I just can’t take the crap that I get anymore. And it’s been helping. Hits to my now-defunct weblog have gone from a major hit count to a few silly aggregators which haven’t adapted yet. The trackback spam flood has slowed down (although not perfectly: I’ve gotten 6 while writing this post.) The message I’m trying to send is pretty simple: I don’t have the money to find you and sue you, but I’ll be damned if my site is going to be a way for your to peddle your wares.

Desperate times call for desperate measures. And I’m taking them.

On the plus side, I did notice a few things yesterday while watching my logs. First, I’ve somehow ended up as the first hit on Google for passive aggressive emo. I think Google heard me shouting, and decided I really meant it. I’m 7th for paris hilton phone hacked – which is amusing, because the post in question doesn’t contain any of the words other than “phone”: it’s a post about RDF toys on the phone. Sounds to me like Google could have used a bit more RDF in its storage mechnism there, seperating spam comments from the actual post content. Metadata would have fixed that bad search result.

I’m going to be looking into a more permanant solution to the problem soon. Right now, I’m passing IP addresses to iptables for dropping: as I said, I’m being a bit of a vigilante about the whole thing. I do want to make it so people can at least navigate the site without problems though, so I will be changing the way this is done around. I just wish it wasn’t neccesary.

Apparently, when the TrafficCam flash program was released, Justin was opening a can of worms that was bigger than I could have imagined.

After my example of quick development on the Python app, I got a lot of interest in my own TrafficCam application. Suddenly, there was a London version. And a Dubai version. And every time I mentioned it, someone else wanted to create their own version and load it in so that they could use the same nifty features. If there’s one thing that I do right, it’s listen to what my users are telling me. So, this afternoon, upon the arrival about my new Nokia 6600, I got to work.

First step: Build a file loader. This function should take a file of predefined format and read it in over the web, letting you specify some parameters to the program. This data should then be returned in a way that the application can use. We can’t make the file format too complex: the default Python install comes with no XML support, remember, so we’re using a very basic, tab delimited layout for this. The format is pretty simple: It’s described in the TrafficCam Format Documentation, for those of you who may want to use it.

Step two: Build the app around the data. Given a specific URL, construct the entire application setup, from the tabs to the title to listings, from that returned data. Not too hard: required a little bit of changing how I did things so that it could be reloaded easily, but as always, Python was cooperative.

Step Three: Build a frontend to choosing URLs to load data from. Store a title an a URI, and let people choose which to load. Not too bad: using the popup_menu that symbian provides, can easily associate the resulting choice with your earlier list.

Step Four: Add support for reloading. Once I’m done with one set of cameras, I want to view another without having to exit and restart. This is a bit more complex: it requires me to move some of the logic around so that the application flow stays mostly the same. In the end, I ended up cleaning up some ugly repetition of the code this way, which was useful.

Step Five: Make it more user friendly. Add an “Other” for choosing their own URLs, add progress meters and information boxes, put in exception support for when a URI doesn’t load correctly, and in general, make the app work better.

All in all, I spent six hours yesterday working on the application, and basically rewrote it from top to bottom. It’s now easy to use, and extendable to do whatever people want. I can admit that it’s probably the single most user-friendly application I’ve ever written: almost all my work in the past has been command line based, but this is truly a cool application.

If you have a phone which supports Python, I highly recommend this application. Although I’m sure there are better apps out there, this one is my personal favorite: lets you get a glimpse of the world through your phone. Of course, you should be aware that this is not a low bandwidth application: the camera listings are only about 3-4k apiece, but each camera image can be anywhere from 10-15k, sometimes more depending on the cameras you’re using. Yesterday, while doing development, I used up a megabyte of GPRS bandwidth – luckily, I have unlimited GPRS through my provider.

If you live in an area where there are traffic cameras, and you’d like to see them added, simply construct a file according to the format documentation, and drop me a line.

Have I mentioned lately that I love Python?

Now, to get working on that contact database export I had in mind…

While I’ve spent a lot of time talking about MeNow-type services, I haven’t spent a lot of time doing anything with them as far as location goes. The main reason for this is there has always been something “Just over the horizon” which is going to take care of all the location stuff for me.

For a long time, nothing really happened. However, thanks to Norman Walsh’s efforts, there now is a “Where in the World” web service. It’s easy to use, although documentation is a bit weak at this point. (I say this with a smile because it’s not gearing to be a complete web service for the public: It’s mostly so that he can learn the way to make a proper web service.)

If you want to try it out, first use the WITW register form to register with the service, which will generate a userid for you and allow you to use your account. You can then use the WITW web form to alter your position. There are also other ways, more oriented to the Web Services way of doing things: these are covered in the WITW Part 1 post on Norm’s weblog.

Attached to this, for results, are a couple URIs: http://norman.walsh.name/2005/02/witw/is/$username is the most interesting, for example, WITW is Christopher Schmidt , which returns XML formatted in-browser with XSLT. (Note that this will only work in IE and Firefox at the moment.) Even more interesting things happen when you’re actually near someone, however, there’s not anyone close enough to show up by default. Norm set up a temporary page demonstrating this, by extending the “nearby” distance to 100 miles. You could see this on another WITW is Christopher Schmidt page, but it’s down now.

If you look at the resulting XML, you can see it’s pretty simple, and easy to use: Easy enough that even I could figure out what to do with it. So, I did: the freenode redlandbot install now supports a “witw” command:

< crschmidt> ^witw crschmidt
< julie> At 2005-02-18T23:51:33Z, crschmidt was at long: -71.4369, lat: 42.9813

The next step is to get it to update the site, as well as to store the data into the RDF store. That way, I can start to build up a MeNow database, which will let me work on furthering that project as well.

That’s not the only thing there is out there for location: There’s also Plazes, which aims to store a much wider range of data and be much more of a web site than a web service. You can see the place where I am right now: The Commune, the place I call home. Soon, the site will be exporting the data as RDF: something that I look forward to. Tying together Plazes and simpler things like WITW are pretty simple.

All in all, I’ve got a lot of fun toys to play with for location, and I’m expecting more will come out of them in the future. For now though, I’m happy that I can update something to let others see where I am. Now I’ve just got to make my computer start doing the hard work for me a bit more simply…

So, a long time ago, I posted about how I planned to update the FOAF that LiveJournal spits out to become more friendly, since I really didn’t know anything about RDF at all when I was first creating the format. So, I wanted to do some updates, to match things that I had discussed with members of the FOAF community over time.

However, I got a real job, lost my free time, blah blah blah, these things happen. The patch has sat around in various incarnations for at least 6 months, probably longer. Recently, someone asked me what had happened to it, and I responded that I had gotten tired of pushing for changes to FOAF, and given up. He said he had time to spare on pushing people, so I cleaned up my patch, and posted about it – a detailed rundown of the changes the patch makes, for evaluation by the community.

Since the community involved is larger than LiveJournal, I’m here to encourage absolutely anyone who might have an interest or knowledge in the arena to read through the explanation or the patch, and let me know what they think. I really would love to see this happen, but like I said, I’m not going to fight for it anymore: it took a lot of hard work to get the previous FOAF patch accepted, and I just don’t have the energy or free time to do that again. So, this one is going to depend on community feedback and demonstration of interest. If you think that LiveJournal’s FOAF data might be suboptimal, say something about it. That entry is a great place to comment and discuss. Open a dialouge with developers about what you’d like to see, because if enough people want to see it, someone may actually push it in.

So, recently I’ve been watching a little bit of a FeedBurner problem arise between some friends on IRC, over at Mobtiopia. Seems that MobileTech uses Feedburner for his feed, via http://feeds.feedburner.com/Mobiletech.

However, Erik was having some problems, where every time the feed loaded, all items would look new. He dug into the problem a bit, and found what the root of the problem most likely is: Every 30 minutes or so, the “Last-Modified” header for Feedburner updates. It caches it per IP for about 30 minutes to an hour, it seems, but once you get past that, it just snaps back to the current time.

Now, I’m not one to criticize over small mistakes like this, but I feel bad for Feedburner: Their conditional get mechanism is completely broken by this difference. Rather than returning a 304 not modified, Feedburner is returning the full feed every hour someone asks for it. That’s got to be quite a hit on their bandwidth.

In this case, there were a couple other issues relating to the problem: Tarek’s feed was previously powered by RDF, which FeedBurner seemed to chew up and spit out in a quite ugly way in his case. However, I haven’t seen it happen the same way in other cases. In Tarek’s case, the feed was actually using what looked like uniquely generated local IDs for an rdf:about – which, although fine in RDF-parlance, is not allowed according to the RSS 1.0 specification, and doesn’t pass the feed validator. Most likely, Erik’s newsreader, newzcrawler, saw these funky looking IDs and didn’t treat them as permalinks, contributing to the problem.

Regardless of other issues though, I’ve checked a few other feedburner feeds, and every single one of them has a Last-Modified header in the past hour. This is simply not a good plan for your bandwidth, or for RSS in general: you’re dealing with a lot more traffic than you need to. 304 Not Modified is your friend, either via Etags or If-Modified-Since. RSS readers are doing good at cleaning up their act and using these headers – if the servers don’t support them, that’s just going to discourage such use in the future, and contribute to the load problem that RSS has become for so many people.

I’m going to let Feedburner know about this in more detail, and this is really not a slight against them. Headers for HTTP/RSS are hard to get right, not something that just “works” out of the box typically. So, I understand the difficulties attached to them. Getting them wrong, however, has some major consequences on all parties, so I hope they can figure out what’s up and get it fixed, both for their sake and for the sake of people that use them.

Russ posted about a pretty cool Flash Lite application that was developed: a way to look at the NYC traffic cameras using your phone. It’s an extremely cool app – if I lived in New York, I’d buy Flash Lite just to be able to use that application. One thing that Russ mentioned was the development time for the project: 20 hours of development time, when something in Java would have been way larger.

Well, I’m a Python man, not a Flash man, so I can’t get much out of this yet. However, I do think it’s a cool use case: so I did a little research, found out where the data that the Flash app uses was coming from, and did a little hacking. The result? TrafficCam version 0.1, in Python. This little app took me 45 minutes to develop a fully functional prototype: this included taking the HTML from the NY Transportation site, building it into a Python file, creating a user interface, and downloading and displaying the image in the built in phone image viewer.

Not only did I do all this in 45 minutes, I did it without even having a phone to test with. Passing it off to the owner of a 6600 and a 6630, both say it works just fine, as is.

(Note that I think it probably doesn’t, but in ways that aren’t visible: There has been no testing done yet.)

So, although Flash is great for pretty apps – the Python app is *nothing* like the Flash app, which is a great user interface and something that’s really fun to use, even in a browser – but Python can be really great for *quick* apps, especially on the phone.

Update: With another 35 minutes of work, I now have fully functioning tabbed lists, one for each borough. So, with a total of less than an hour and a half of development, I have an application which allows you to download any of the traffic cam images and view them on your phone. It’s no flash, but I call that pretty damn impressive.

Hula – Open Source Groupware for the Corporate Soul.

There’s lots more out there on all aspects of this than I care to even think about, but some of the things that Nat has written about it in his weblog sound pretty cool. Personally, I have only one wish for this product: for it to become something that I can use in the workplace for task/appointment management. Currently, we’re in a 20% Windows/80% Mac world, but all the appointment management is done through Outlook and ACT!. The number of times I’ve heard the people who use it cursing ACT! is way too high, and it has the obvious problem that the rest of the office can’t use it.

I need to learn to accept that not everything is going to validate, and I need to learn that hacks and workarounds are a way of life. I need to accept that Javascript really is here to stay, and do more to learn how to make good Javascript applications, because really, they can make the entire user experience way better. Software isn’t about perfection, it’s about doing the simplest thing that could possibly work, and making things better for the user.

I wonder what I could start with… a project to learn some of the more advanced Javascript technology, while still keeping the interface usable for non-Javascript enabled browsers, allowing for linking, and so on. Google’s interfaces don’t work so well for this: Gmail, Orkut, and Google Maps all fail miserably without Javascript. I don’t like that.

Ponder ponder…

Around the time that Nokia Python was released, someone mentioned that it would be very interestig to see a replacement for the built in Nokia Notepad program. The Notepad program has many limitations: you can’t fit much text on a screen, there’s no easy way to browse or link notes you’ve written, all the typical limitations that you see when writing in a small paper notepad.

It was discussed that a small, wiki-like implementation might be a way to improve the usability of the notepad. Allow people to link between pages or nodes, let them type what they want and link it together. This would greatly improve the usability of having a phone along with you: if you can take notes on it that you can actually usee, there’s less chance you’ll lose them than paper, and you’re far more likely to be able to use something that’s connected than just write one note to yourself.

For a while, I didn’t develop anything on the phone, mostly due to technical issues related to my development process. However, after the recent Python SIG meeting, I decided that the next time I saw people, I wanted the phone to do something useful. So, I took on the challenge: a wiki-like scratchpad.

I started last night, and didn’t get too far before I ran into some symbian related problems: I couldn’t get anydbm to save anything. The file would open, it would stay open, but it wouldn’t sync, and it wouldn’t close. This morning, I finally looked at some code, and realized – wait. I need a full path. And once I got that far, things started looking up. In the hour I had this morning before work, I hacked out the basics: open a node, write to it, save, exit. Paavo on #mobitopia pointed out some other issues: the exit key handler, for one.

Then, over lunch, I created a webpage, and fleshed it out a bit more: adding a feature such that when you hit the action button, the word underneath the cursor is opened as a new node. This was about another two hours of work, spread over lunch and 10 minute breaks from work throughout the day. This included writing a program that catches keypresses and displays their associated keycodes, including learning a bit more about lambdas than I used to. I then had a working wikipad, and have since been adding minor cleanup tweaks.

In the past hour, I’ve created a beta, which allows uploading the wiki database to a webpage as POST data, from which it can be done with what you wish. (I’m currently storing the data in print_r() form, just for the sake of doing so.) From here, it’s pretty easy to imagine how this could allow reimporting the data.

So, the next step is, of course, making this work in a collaborative sense: allowing multiple people to edit one wiki via their phones. However, for the time being, it’s a great little scratchpad app, that’s only likely to get greater. You can get the code from the WikiPad Homepage.

I posted recently about Zeroconf, which allows for automated service discovery. One aspect of this which has already been exploited is the ability to have a local network of presence information, something like what iChat does to announce users who are online. However, it also passes along some simple FOAF information with it. I can’t remember who wrote the tool. If I recall correctly, it was shellac/Damien… ah, here we go. foaffinger.

Small HTTP server combined with zeroconf shares data.

Now, to expand this out a bit… now that I’m starting to know how to deal with Python a bit better, I might be able to do something with pyzeroconf and pyrple. I’d probably want to learn something about graphics too… wxPython requires an install. Are there any “built in” GUIs for Python? I don’t think so.

Anyway, local FOAF browsing of full data rather than just a subset would be nice. Not too hard, I don’t think, since Python makes almost everything simple. Of course, there is the problem that there’s no one that I share my network with… 😉

Ah well, I do have 2 macs, 2 linux boxes, and 2 windows laptops in the house now. (Wow! An even split. Nifty.) I can try it out somehow or another.

I’m kind of rambling. Usually I’d reserve this for noets, but I wanted a tie in from the Zeroconf post I just made.

There’s not a lot of webpages out there on the neat things you can do with Zeroconf. For some reason, this surprises me. I feel like it’s a pretty interesting technology, but it doesn’t seem like anyone’s done much with it, despite the fact it’s been around for quite a while. For example, AaronSw wrote about zeroconf more than two years ago.

For those who may have missed the boat, the primary thing that I mean when I talk about Zerconf is automated service discovery (DNS-SD). This is the kind of thing that allows iTunes to see the other computers locally that are also sharing iTunes music. Apple has put a lot of work into making this stuff simple, to the point that it’s one of their more popular open source releases. (Their mDNSResponder code, that is.) I know that Gnome is putting a lot of effort into making things like this work, but I don’t run Gnome (I’m a minimalist) so I don’t know where that’s really at.

Earlier tonight, I was having some problems announcing services over Rendezvous, which is the only way that iTunes will attempt to talk to a DAAP share – there’s no way to tell iTunes “go to this IP address”. This is probably by design: this way I can’t open up someone’s music share that’s not announcing it locally. Kind of annoying for me, since I’d like to be able to play my music from home at work, which I would be able to do with DAAP otherwise After some trouble, I eventually got my machine to announce the iTunes share (which is done via mt-daap) on my Linux box, so I can now play that music. I also added a few other Services: HTTP and SSH. Still, I don’t really know what I can do with these. Why doesn’t putty or some other app capitalize on this by offering to search local hosts for SSH? Does no one really use local services that this could make easier?

My Zeroconf stuff comes from Howl, which offers a multi-platform open-source solution to providing Zeroconf services. It’s a decent library, with decent tools. It is a bit iffy on the user-level: The error messages are pretty unspecific. For example, if you try to announce a service before first running mDNSResponder, it will give an error message that it is unable to connect: but won’t tell you where or why, nor is it mentioned in any documentation (that I can find). The code itself seems solid, however, and is working well here. So, what cool ideas are there for Rendezvous that no one has done anything with? And who’s going to step up to the plate to start?

One example I can think of: set up Rendezvous to announce your local MySQL servers. Then, if you’re in an office where you do development, you can just pick from a drop down of development servers: you don’t have to depend on static IP addresses, you can just scroll through a list. You can even use the text fields of the Rendezvous to include a Guest login. Seems like something that would work well for CocoaMySQL. Of course, it doesn’t look like it’s been touched for more than a year, so it’s not likely to change in the near future. Just an idea.

What zeroconf does not do, which many people don’t seem to understand, is provide any real help in actually creating the connections. Lots of people, myself included, seem to think “Rendezvous” is this magical tool that does all the network communication for you. (That is, I used to think that way – I know better now.) Really, all it does is give you an IP address – nothing that a relatively tech savvy person couldn’t do on their own, at the application level. The communication actually takes place over the typical channels – regular socket communication.

So, although Zeroconf/Rendezvous is cool, it’s not an “end all/be all” to network communication. Just something to keep in mind.