Technical Ramblings

Wombat takes charge : learning how easy it is to work with location stuff in Python, he starts work on a project that sounds a lot like the work I’ve started on locative stuff. Looks like I need to get back to that…

As FrankK reports, Darla will record an orgasm ringtone if she gets requests on her blog for one. Think she’ll follow through? I’m interested in finding out.

Flickr stuff is close to done. I’ve got a user interface put together, using code stolen from the included file browser provided by Nokia. You can browse through, select a picture to upload, and upload it, using the given tags, security, and stored username/password. It’s not ready for prime time yet, because I want to set up a way to do multiple uploads at once. I’d also like to be able to type in titles/descriptions, but the limitations of the UI widgets available blocked me there. I don’t have anything that I can type t9 into with ease, so getting the data out would be a pain. Is it so much to ask for a t9 enabled appuifw.query window?

Jess asked me yesterday if I’ll ever make any money hacking these things together. My answer is probably not: I don’t like the idea of charging people money for something. I’ve never even asked for donations, because I just think it’s not fair to expect people to pay me for the work I do. Despite this, if anyone wants to offer me money to do what I’m doing, I’d be glad to take it. I really need to move this stuff to a server that isn’t hosted in a house with four cats running around: they have a tendancy to pull out the ethernet cable nightly, which means downtime for everyone.

I’m looking to move stuff soon, I’ve had my eyes on one of the Sagonet bargain servers for a while. (The reason I’m going for my own box rather than a virtual box type solution is memory based: I want to run julie off a different machine than my own, and that requires me to use a hell of a lot more memory than any virtual hosting solution I’ve yet found.) The price is good, it’s just that initial extra setup fee that’s putting me off.

However, every day that goes by is another day that I realize I really just need to move stuff and be done with it. Waking up every morning and having to restart my menagerie of bots is getting old. I want something stable.

Is anyone looking for hosting? $10/ a month, and I’ll do all the work to install anything you want! 🙂

I’ve worked out how to post images to flickr using HTTPlib and Python from the flickrup source code. I have no idea if this is the best way to do it, but it seems to work, so I’m writing an application to surround it in a UI. In the process, I’m creating a pretty generalized application framework that I should be able to use on some of my other applications. So, I’m enjoying that.

Some things that it’s done so far:
* Established a couple generalized option configuration methods, which let me load and save from a file
* Created what i hope is a UI for changing those options (which will change per-app)
* Added in some general stuff to make sure the UI works, like locks to make sure the application doesn’t just quit

I’m probably not going to release the framework on its own, at least not until I come out with the Flickr app that I’m writing it for, simply because I’m writing the functions as I go along. If it ends up being worthwhile and reusable enough, I may list it seperately on my page of Symbian Projects.

I think that this could be really fun for me to play with, and I hope I get it working soon, because it seems like a cool application idea. The basic idea behind it is to combine the filebrowser code that is included with the examples in the distribution with a flickr uploading option, as well as the option to view the images. Granted, I can’t do anything other than open them in the default viewer right now, but then at least people can see what they’re getting into.

I’m starting to get to the point where I could see how having a bit more advanced symbian UI widgets would be helpful: something like Putty’s menu, where things can actually be in submenus and the like. I can also see how it’d be way more difficult to code, so I’m not lamenting the lack, just noticing it more than I used to as I get into more and more complex applications. One thing that I have learned is that the Text widget makes a really good way to keep messages and so on on the screen without any complex threading or anything, which is nice. Little things like a progress bar (which seems to be underway) might still be nice though.

Just some random meandering thoughts while I’m writing. Hopefully by the end of the week I’ll have an app at least starting to take shape for Flickr, and maybe I can devote some more time to the traffic cam project too.

I’m looking to find out whether anyone has any desire to see new applications based on RDF. What would you like to see that hasn’t been done yet? What do you think would be a cool demonstration?

This can be web apps, local apps, whatever. Most of my stuff these days are in Python, most of my RDF stuff is based around Redland (when it’s not dying because of something I broke) and most of my platform-specific knowledge is Linux based.

Given these constraints, what “cool thing” hasn’t been done to pieces yet? Would a redux of Plink be in order, perhaps with better tools for automatic information removal to avoid the situation of angry users? A web service allowing you to use julie’s database for queries? Mail filtering? Image searching by all kinds of fun stuff, a la MortenF’s application, only with a user-upload section?

I really am low on ideas at this point, so I’m interested in hearing what thoughts other people might have.

When I got back into the mobile world recently, with a couple of apps I wrote and hanging out in the #mobitopia IRC channel again, I wondered why there wasn’t any central aggregation for the mobile world. The Semantic Web has PlanetRDF, and there are more than a dozen other similar aggregation services for different portions of the web, from Apache to Gnome to Adium. Despite the large weblogging community surrounding the Mobile world, however, there was no similar aggregation.

I say was, because there is now. PlanetMobile is an aggregator for all things mobile related. It features posts from a variety of sources, from Engadget to the mobile platform category on my weblog. It also shows the recent flickr images tagged with “mobile”, as well as the latest posts from both del.icio.us and Technorati, all aggregated in a handy little sidebar.

The CSS is taken in large part from PlanetRDF, which has already gotten me some compliments (which I’ve suggested be forwarded onto the chaps). The Planet aggregator is running the whole shebang behind the show, along with the Magpie RSS Aggregator for the sidebar.

If you can think of another Mobile feed that should be added, I’d be interested in hearing from you. If you think that it’s a cool idea, and want one for your own specialty interest, also feel free to let me know. If you think that I’ve somehow impeded upon your copyright, also feel free to let me know. If you just think I’m a weirdo, I’m always open to that too.

Desperate times call for desperate measures.

After doing some log watching yesterday, I realized that almost 25% of the hits to my site were people attempting to spam a now-defunct weblog with trackback spam, as well as using spam referrers. After watching this for a while, I got pretty tired of seeing all these ads for poker and so on, so I started going on a rampage. No more Mr. Nice Guy, I thought. These machines aren’t even hitting valid URIs: they’re spamming something that hasn’t even existed for the past two months. I’m tired of it.

So I started blocking them. Each and every IP that came in bearing trackback spam, spam referer headers, or anything else was no longer going to touch my server. Unfortunate, yes, becase many of the machines which are doing these things have owners who are completely unaware of it. In some cases, the IPs may have been dynamic: there was more than one comcast IP address in there, and I’m pretty sure they don’t give out completely static IP addresses all that often. So, it may be that in blocking this spam, I’ve ended up blocking some legitimate users of my site, and that sucks.

However, I am very tired of deleting spam. I am tired of sorting through crap. I am absolutely tired of people talking the problem to death, because until the people doing these things actually start getting in trouble, nothing at all is going to change. The only way to stop people from spamming, be it email, weblogs or what have you, is to get to the people behind these attempts, and make it unprofitable. Sue them, charge them, whatever, but make it so that if they try and go about it in a way that no one thinks is right, they lose money. Because spammers still make plenty of money on spam. If they didn’t, they wouldn’t keep doing it.

Overnight, I received 90 trackback spams. Approximately 15 of these were new variants on spellings and so on to the point that they made it past my spam moderation filters. As a result, when Iwoke up this morning, I had a major spam problem to deal with. (Luckily for me, it’s my day off work. Unluckily, I’ve been spending most of it fighting spam fires. Annoying, really.)

I’ve turned on comment moderation. All comments must be approved by me. I’m not dealing with this stuff anymore. There is no no way that a spammer can get a comment onto the site and public without me seeing it first. I don’t like it, it’s annoying, and it’s troublesome to comment authors, but it’s worth it to me to be able ot know that no one is going to get a higher search engine ranking because of this site. I’ve started dropping access to any IP address which sends spam. If someone can’t get to my site, hopefully they can email me or something, but I just can’t take the crap that I get anymore. And it’s been helping. Hits to my now-defunct weblog have gone from a major hit count to a few silly aggregators which haven’t adapted yet. The trackback spam flood has slowed down (although not perfectly: I’ve gotten 6 while writing this post.) The message I’m trying to send is pretty simple: I don’t have the money to find you and sue you, but I’ll be damned if my site is going to be a way for your to peddle your wares.

Desperate times call for desperate measures. And I’m taking them.

On the plus side, I did notice a few things yesterday while watching my logs. First, I’ve somehow ended up as the first hit on Google for passive aggressive emo. I think Google heard me shouting, and decided I really meant it. I’m 7th for paris hilton phone hacked – which is amusing, because the post in question doesn’t contain any of the words other than “phone”: it’s a post about RDF toys on the phone. Sounds to me like Google could have used a bit more RDF in its storage mechnism there, seperating spam comments from the actual post content. Metadata would have fixed that bad search result.

I’m going to be looking into a more permanant solution to the problem soon. Right now, I’m passing IP addresses to iptables for dropping: as I said, I’m being a bit of a vigilante about the whole thing. I do want to make it so people can at least navigate the site without problems though, so I will be changing the way this is done around. I just wish it wasn’t neccesary.

Apparently, when the TrafficCam flash program was released, Justin was opening a can of worms that was bigger than I could have imagined.

After my example of quick development on the Python app, I got a lot of interest in my own TrafficCam application. Suddenly, there was a London version. And a Dubai version. And every time I mentioned it, someone else wanted to create their own version and load it in so that they could use the same nifty features. If there’s one thing that I do right, it’s listen to what my users are telling me. So, this afternoon, upon the arrival about my new Nokia 6600, I got to work.

First step: Build a file loader. This function should take a file of predefined format and read it in over the web, letting you specify some parameters to the program. This data should then be returned in a way that the application can use. We can’t make the file format too complex: the default Python install comes with no XML support, remember, so we’re using a very basic, tab delimited layout for this. The format is pretty simple: It’s described in the TrafficCam Format Documentation, for those of you who may want to use it.

Step two: Build the app around the data. Given a specific URL, construct the entire application setup, from the tabs to the title to listings, from that returned data. Not too hard: required a little bit of changing how I did things so that it could be reloaded easily, but as always, Python was cooperative.

Step Three: Build a frontend to choosing URLs to load data from. Store a title an a URI, and let people choose which to load. Not too bad: using the popup_menu that symbian provides, can easily associate the resulting choice with your earlier list.

Step Four: Add support for reloading. Once I’m done with one set of cameras, I want to view another without having to exit and restart. This is a bit more complex: it requires me to move some of the logic around so that the application flow stays mostly the same. In the end, I ended up cleaning up some ugly repetition of the code this way, which was useful.

Step Five: Make it more user friendly. Add an “Other” for choosing their own URLs, add progress meters and information boxes, put in exception support for when a URI doesn’t load correctly, and in general, make the app work better.

All in all, I spent six hours yesterday working on the application, and basically rewrote it from top to bottom. It’s now easy to use, and extendable to do whatever people want. I can admit that it’s probably the single most user-friendly application I’ve ever written: almost all my work in the past has been command line based, but this is truly a cool application.

If you have a phone which supports Python, I highly recommend this application. Although I’m sure there are better apps out there, this one is my personal favorite: lets you get a glimpse of the world through your phone. Of course, you should be aware that this is not a low bandwidth application: the camera listings are only about 3-4k apiece, but each camera image can be anywhere from 10-15k, sometimes more depending on the cameras you’re using. Yesterday, while doing development, I used up a megabyte of GPRS bandwidth – luckily, I have unlimited GPRS through my provider.

If you live in an area where there are traffic cameras, and you’d like to see them added, simply construct a file according to the format documentation, and drop me a line.

Have I mentioned lately that I love Python?

Now, to get working on that contact database export I had in mind…

While I’ve spent a lot of time talking about MeNow-type services, I haven’t spent a lot of time doing anything with them as far as location goes. The main reason for this is there has always been something “Just over the horizon” which is going to take care of all the location stuff for me.

For a long time, nothing really happened. However, thanks to Norman Walsh’s efforts, there now is a “Where in the World” web service. It’s easy to use, although documentation is a bit weak at this point. (I say this with a smile because it’s not gearing to be a complete web service for the public: It’s mostly so that he can learn the way to make a proper web service.)

If you want to try it out, first use the WITW register form to register with the service, which will generate a userid for you and allow you to use your account. You can then use the WITW web form to alter your position. There are also other ways, more oriented to the Web Services way of doing things: these are covered in the WITW Part 1 post on Norm’s weblog.

Attached to this, for results, are a couple URIs: http://norman.walsh.name/2005/02/witw/is/$username is the most interesting, for example, WITW is Christopher Schmidt , which returns XML formatted in-browser with XSLT. (Note that this will only work in IE and Firefox at the moment.) Even more interesting things happen when you’re actually near someone, however, there’s not anyone close enough to show up by default. Norm set up a temporary page demonstrating this, by extending the “nearby” distance to 100 miles. You could see this on another WITW is Christopher Schmidt page, but it’s down now.

If you look at the resulting XML, you can see it’s pretty simple, and easy to use: Easy enough that even I could figure out what to do with it. So, I did: the freenode redlandbot install now supports a “witw” command:

< crschmidt> ^witw crschmidt
< julie> At 2005-02-18T23:51:33Z, crschmidt was at long: -71.4369, lat: 42.9813

The next step is to get it to update the site, as well as to store the data into the RDF store. That way, I can start to build up a MeNow database, which will let me work on furthering that project as well.

That’s not the only thing there is out there for location: There’s also Plazes, which aims to store a much wider range of data and be much more of a web site than a web service. You can see the place where I am right now: The Commune, the place I call home. Soon, the site will be exporting the data as RDF: something that I look forward to. Tying together Plazes and simpler things like WITW are pretty simple.

All in all, I’ve got a lot of fun toys to play with for location, and I’m expecting more will come out of them in the future. For now though, I’m happy that I can update something to let others see where I am. Now I’ve just got to make my computer start doing the hard work for me a bit more simply…

So, a long time ago, I posted about how I planned to update the FOAF that LiveJournal spits out to become more friendly, since I really didn’t know anything about RDF at all when I was first creating the format. So, I wanted to do some updates, to match things that I had discussed with members of the FOAF community over time.

However, I got a real job, lost my free time, blah blah blah, these things happen. The patch has sat around in various incarnations for at least 6 months, probably longer. Recently, someone asked me what had happened to it, and I responded that I had gotten tired of pushing for changes to FOAF, and given up. He said he had time to spare on pushing people, so I cleaned up my patch, and posted about it – a detailed rundown of the changes the patch makes, for evaluation by the community.

Since the community involved is larger than LiveJournal, I’m here to encourage absolutely anyone who might have an interest or knowledge in the arena to read through the explanation or the patch, and let me know what they think. I really would love to see this happen, but like I said, I’m not going to fight for it anymore: it took a lot of hard work to get the previous FOAF patch accepted, and I just don’t have the energy or free time to do that again. So, this one is going to depend on community feedback and demonstration of interest. If you think that LiveJournal’s FOAF data might be suboptimal, say something about it. That entry is a great place to comment and discuss. Open a dialouge with developers about what you’d like to see, because if enough people want to see it, someone may actually push it in.

So, recently I’ve been watching a little bit of a FeedBurner problem arise between some friends on IRC, over at Mobtiopia. Seems that MobileTech uses Feedburner for his feed, via http://feeds.feedburner.com/Mobiletech.

However, Erik was having some problems, where every time the feed loaded, all items would look new. He dug into the problem a bit, and found what the root of the problem most likely is: Every 30 minutes or so, the “Last-Modified” header for Feedburner updates. It caches it per IP for about 30 minutes to an hour, it seems, but once you get past that, it just snaps back to the current time.

Now, I’m not one to criticize over small mistakes like this, but I feel bad for Feedburner: Their conditional get mechanism is completely broken by this difference. Rather than returning a 304 not modified, Feedburner is returning the full feed every hour someone asks for it. That’s got to be quite a hit on their bandwidth.

In this case, there were a couple other issues relating to the problem: Tarek’s feed was previously powered by RDF, which FeedBurner seemed to chew up and spit out in a quite ugly way in his case. However, I haven’t seen it happen the same way in other cases. In Tarek’s case, the feed was actually using what looked like uniquely generated local IDs for an rdf:about – which, although fine in RDF-parlance, is not allowed according to the RSS 1.0 specification, and doesn’t pass the feed validator. Most likely, Erik’s newsreader, newzcrawler, saw these funky looking IDs and didn’t treat them as permalinks, contributing to the problem.

Regardless of other issues though, I’ve checked a few other feedburner feeds, and every single one of them has a Last-Modified header in the past hour. This is simply not a good plan for your bandwidth, or for RSS in general: you’re dealing with a lot more traffic than you need to. 304 Not Modified is your friend, either via Etags or If-Modified-Since. RSS readers are doing good at cleaning up their act and using these headers – if the servers don’t support them, that’s just going to discourage such use in the future, and contribute to the load problem that RSS has become for so many people.

I’m going to let Feedburner know about this in more detail, and this is really not a slight against them. Headers for HTTP/RSS are hard to get right, not something that just “works” out of the box typically. So, I understand the difficulties attached to them. Getting them wrong, however, has some major consequences on all parties, so I hope they can figure out what’s up and get it fixed, both for their sake and for the sake of people that use them.

Russ posted about a pretty cool Flash Lite application that was developed: a way to look at the NYC traffic cameras using your phone. It’s an extremely cool app – if I lived in New York, I’d buy Flash Lite just to be able to use that application. One thing that Russ mentioned was the development time for the project: 20 hours of development time, when something in Java would have been way larger.

Well, I’m a Python man, not a Flash man, so I can’t get much out of this yet. However, I do think it’s a cool use case: so I did a little research, found out where the data that the Flash app uses was coming from, and did a little hacking. The result? TrafficCam version 0.1, in Python. This little app took me 45 minutes to develop a fully functional prototype: this included taking the HTML from the NY Transportation site, building it into a Python file, creating a user interface, and downloading and displaying the image in the built in phone image viewer.

Not only did I do all this in 45 minutes, I did it without even having a phone to test with. Passing it off to the owner of a 6600 and a 6630, both say it works just fine, as is.

(Note that I think it probably doesn’t, but in ways that aren’t visible: There has been no testing done yet.)

So, although Flash is great for pretty apps – the Python app is *nothing* like the Flash app, which is a great user interface and something that’s really fun to use, even in a browser – but Python can be really great for *quick* apps, especially on the phone.

Update: With another 35 minutes of work, I now have fully functioning tabbed lists, one for each borough. So, with a total of less than an hour and a half of development, I have an application which allows you to download any of the traffic cam images and view them on your phone. It’s no flash, but I call that pretty damn impressive.