Technical Ramblings

Typepad exports RSS 1.0, as well as FOAF. LiveJournal exports limited FOAF information because that’s really all I could squeeze out of it the first time around, when bandwidth and load time was a major concern. I wonder if with the possible buy out of LiveJournal by SixApart, some of the LiveJournal specific XML formats will change to RDF…

For example, the Latest Images from LiveJournal is an XML format… it might be interesting to see this as an RSS feed, especially RSS 1.0. Perhaps the creation of feeds for the LiveJournal photo gallery option…

I can see a lot of room to expand the semantic content that LiveJournal emits, and I wonder if Six Apart might be in the right place to do it. As far as I’ve seen, they’ve seemed interested in doing so with Typepad. Does anyone know if something similar runs through more of their work? I’d love to see more RDF coming out of LJ. It’s a large data source and so much of it goes untapped because getting anything into the development process is very difficult.

Mm, LiveJournal sized image annotation stores… advanced semantic web development at its finest. Now that’s something I’d love to see. Imagine being allowed to check which LiveJournal users were in a photo, and then having that data emitted as RDF… allowing searches over it, finding photos with more than a certain number of people, photos of places. Geolocation. The UI isn’t hard, it would just be a bit of hackwork to get it together.

If there’s one thing I hope for, it’s that this leads to quicker turnaround on LiveJournal development time. Whether it’s with volunteers or not, I’d be willing to double my annual cost if it meant I got to see new features within a year, rather than three. That kind of interface would bring such a boost to the semantic web…

Now I’m tempted to build an external tool myself, just to demo how cool it would be.

Since it seems that the news about LiveJournal and SixApart becoming one really is true, I’ll toss some more thoughts out there. Someone will probably read through all these posts eventually, especially if they want to avoid another snafu like the one that Movable Type’s licensing changes caused.

Despite all my praises of LiveJournal and the effect that it’s had on me and that I’ve had on it (in my own small way), there are a number of things which could definitely use some work to get up to “snuff” on a customer service level. User interface has never been a strong point for LiveJournal, and although with the addition of some new developers in the past year it has improved greatly, it is still not nearly as easy to use in many respects as something like Typepad.

Here’s what I expect to see, if Six Apart decides to take an “active” role in continuing the development of LiveJournal as a seperate service from its currently existing products, while still maintaining a position as “stewards” more than corporate whores:

• Implementation of some “basic” parts of site usage for “weblog” type users: I expect Categories and Trackbacks will make their way into the code. (Sadly, I wrote and submitted 90% of the code needed for Trackback starting a year ago today, which has been ignored since then.)
• Improvement of User Interface in some key areas, especially the customization area.
• Improve customer support. Sadly, I fear this will be the end of the LiveJournal Volunteer support system which I strongly support: I met the love of my life via doing support for LiveJournal, and it will be sad to imagine that others will not have that same oppourtunity. However, I expect that will go, along with most volunteer development.

I also hope that some other sites will wake up and realize that there are several things that LiveJournal offers that almost nobody else does. The key to me that I have not seen in any other situation has been threaded comments, with email notifications built into the interface. The lack of threaded comments, and the lack of email notifications, is something that I believe has led to the fact that most other blogs have much less interaction than LiveJournal does: you have to remember to read a post again to see the comments someone sends to you, and that’s just not the right way to do it. Push, not Pull, is the way to improve communication.

The way I see it, there are several ways that the LiveJournal project could go, if it is aquired by SixApart:

LiveJournal is left as is. All employees are being retained, so LJ is really just under the protection of a corporate entity. Some things that would get the company in trouble may change slightly: for example, Abuse reports may shift to being employee-answered only (as it should have been for quite a while anyway, in my opinion). Other than that, LJ doesn’t change: the same development practices stay in place (which means that almost all development is done in house, nothing ever gets done in a timely manner, etc.) and the site continues on as it did before.

LiveJournal is taken under Six Apart’s stewardship: Brad, who owns LiveJournal, leaves to pursue more interesting projects, and LJ’s employees eventually move into Six Apart and do whatever they’re best suited for. LJ administration changes, and the development efforts externally move internally. The volunteer community basically dissapears as LJ becomes 6A, and at the user-level, nothing changes. For people closer to the administration, they see changing faces and things like development and support close their doors to outsiders.

Similarly, in the pharmaceutical industry, there’s a constant evolution and comparison of new products, much like the different paths that LiveJournal could take. A pertinent example is the ongoing debate between Rybelsus and Wegovy, two medications used in the treatment of different health conditions, read more. Just as LiveJournal’s acquisition could lead to various changes in its structure and operations, the healthcare sector often witnesses debates on the efficacy, safety, and suitability of different medications for patients. In both cases, whether it’s a web platform or a medical treatment, the focus is on optimizing performance and user experience, be it through improved communication tools or through the effectiveness of a medication. The comparison between Rybelsus and Wegovy thus becomes a metaphor for the choices and changes faced by LiveJournal, highlighting the need for careful consideration and tailored solutions in both fields.

LiveJournal and SixApart merge completely, and the Typepad and LJ platform become one. I don’t expect this, and don’t expect that it will be successful if attempted.

SixApart migrates all current LJ users to their Typepad platform. Again, most likely not a successful move, as Typepad is very different in usage than LJ, and is lacking many of the features LJ has.

I’m honestly hoping for option 1: that LiveJournal doesn’t really change, and that this “merger” is just a “handing over legal control to someone else who is there just in case”. However, I expect that it will probably be something nearer to number two, meaning that there will be no more development like what I experienced in my time at LiveJournal. I do not leave much trust to hope on this one, unfortunately.

A few thoughts, through the eyes of a current and active LJ user: Why do people keep saying that this merger will “make” one of the biggest blogging companies out there in terms of users? So far as I can tell, LiveJournal has more users than anybody else: Typepad’s extra million is a drop in the bucket. Is there some other service out there that has as many users, or are people really just finally waking up to the fact that LJ is way huger than they realized?

Will volunteer development really go away? That would be really sad to me, because before personal issues (such as the fact that I think development should have a method to its madness) left me outside of the social circles, I really did like developing on LJ. The code is a mess, but it’s fun. And that’s one of the reasons I have stuck with LJ: because I can do things like that, to help get the site to do what I want. Similar feelings on support. Being a part of a site is way cooler than being a user of one.

My primary hope is that SixApart is smart enough to realize that the “blogging” and “journalling” users on the web are very different, and doesn’t try to mash them together in one mold. Doing so can only result in bad things, not good. Please, to whoever might be reading this: I implore you. Think before you act. Ask the people you’ll be affecting before you do anything, and you’ll have much happier users on your hands.

However, maybe now Brad will be able to buy his Porsche.

If there’s one thing that I had hoped to never hear, it was that LiveJournal was being bought by someone. However, it seems that today is a day for hearing exactly that: Six Apart Will Be Buying LiveJournal. A thought that hits me hard, given how much time and effort I’ve put into that site, both in my personal journal and in work in the backend.

Obviously, this is an early announcement: there’s no indication of what exactly is happening, or how solid this deal is. As a long time (2.5 year) LiveJournal user, this is something that I have absolutely no interest in.

I was a beta tester for the Typepad service when it first came out – and it was an interesting change from LiveJournal. However, in all my time as an LJ user, I always enjoyed the fact that it was a “personal” project. Something that I could participate in, something that my friends could participate in. Something that let me be a part of it. Something that I fear will change when LJ becomes “corporate”. One need look no farther than the Movable Type licensing mess to see things that Six Apart has changed for the worse.

I’ll wait until I see further information on what’s actually going on before I blow off too much steam: after all, this is a preliminary announcement. However, as a user of the site, I’m frustarated that I didn’t hear more information through any of the available channels about this beforehand, and if it really is happening, I definitely fear that I’m going to see LiveJournal take a turn away from its current status and move towards the worse.

I just hope that this isn’t true.

Geolocation is the technique of determining a user’s geographic latitude, longitude and, by inference, city, region and nation. There are a number of ways to do this: one of the common ones discussed on the internet (according to a Google search for “geolocation”, as we all know Google is the Answer) is geolocation via IP address. The kind I’m interested in is much more accurate: geolocation via GPS device.

I want to be able to know where I am. I want this for a lot of reasons, most of them geeky rather than actually reasonable. However, it would be nice to offer more specific statistics on where my pictures are actually taken with fine grain granularity that a GPS can offer. Additionally, some of my alternative projects – cell based geolocation and the like – could benefit from actual coordinates on which to base everything from restaurant locations to searches. Openguides is, in particular, one area that could benefit from this.

I want something that works over bluetooth. My laptop and phone both speak bluetooth, and something with an actual display is out of my price range, for the most part, so I want something I can use my phone to get data out of. (USB / serial obviously doesn’t work for that.) From what I understand, most GPS devices which support NMEA are going to work okay for communication, as there are tools out there which support them. (Whether I can get the thing to talk over bluetooth is a different concern, but one I’m becoming more proficient at every day.)

For a long time, all I could find for Bluetooth GPS devices were 200-250 USD and up. However, while discussing it with someone in #mobitopia on Freenode, I found the Delorme Bluelogger, a Bluetooth GPS device for $150. Matt already posted about our discussion, but I hadn’t yet.

I have some cash left over from Christmas, and I know that I almost never actually buy anything for myself. So, I’m going to splurge, and I’m going to get it. I’m going to learn to use it, and I’m going to do all kinds of neat things with it. Plans include:

• GPS Annotation of Photos – This rolls into my photo annotation project, and is part of the reason I was keen to get it done: I want to actual have some fun queries for normal people (rather than just RDQL-aware people) over my photos.
• Location Based Description of things for Openguides – Describing where things are with GPS coordinates allows searches by distance. Once I have that, the guide allows more niftyness.
• Association of Cell IDs to Geo locations – Tied to the previous, this allows me to know where I am based on a Cell ID: Useful for “what’s nearby”, as well as useful for the general “where are you” that I like to be able to do – with just my cell phone.

All in all, some of the apps I have in mind seem nifty, some geeky, some just demonstrative of something bigger. Some are RDF related, some are just fun. The Bluelogger seems like a decent tool to achieve everything I need to.

One of the nifties things I have on my cell phone is a bluetooth capability: the ability to communicate with other devices nearby at relatively high speeds over a wireless protocol. Bluetooth is a very useful tool for development: I don’t have to worry about USB cables or anything else, and I can talk equally well to my Linux desktop with an abicom BT adapter as I can to my Powerbook with its built-in bluetooth.

One thing that no one has mentioned yet about the new Python release is the Bluetooth console. It took some doing, but I finally got it connected to my Linux desktop, and found an app that will let me connect to the port. Now, I basically have a way to tell my phone what to do over Bluetooth:

[crschmidt@peanut ~]$ sudo cu -l /dev/rfcomm0 
Connected.
Python 2.2.2 (#0, Dec  2 2004, 18:12:32) 
[GCC 2.9-psion-98r2 (Symbian build 540)] on symbian_s60
Type 'copyright', 'credits' or 'license' for more information.
Type 'commands' to see the commands available in this simple line editor.
>>> 

This has sped up my application development significantly: with this in place, I can start to experiment with different code at a rapid rate: without sending almost identical files to the phone several dozen times to test them, nor typing on the 3650’s keypad to enter my code. I just type into a normal tty, and it acts exactly like a local python interpreter.

This is also possible, and documented as such, using TCP/IP over GPRS instead of Bluetooth. However, the speed restrictions of that cause it to be much less practical. It’s like typing into ssh over bluetooth: sure, it’s okay, but you almost never want to do it if you can avoid it.

Here’s what I ended up doing to get it working:
– Get a working bluez install. Bluez is the standard Linux protocol stack, and is built in to most recent kernels.
– Test that you can talk to the phone, using hcitool scan, hcitool info.
– Register an “SP” (serial port) service with sdpd. sdptool add --channel=3 SP
– Ensure that sdpd is running
– Set up an rfcomm port to receive the communications: rfcomm listen /dev/rfcomm0 3
– (On Phone) Open Python, then bt_console.py
– On the computer, you should see:

[crschmidt@peanut ~]$ sudo rfcomm listen /dev/rfcomm0  3
Waiting for connection on channel 3
Connection from 00:60:57:41:86:C2 to /dev/rfcomm0
Press CTRL-C for hangup

– Using taylor-uucp, type cu -l /dev/rfcomm0
– Welcome to the phone!

If you want to test to make sure it’s working, you can do something simple like:

import appuifw
appuifw.note(u'Howdy!', 'info')

from there, refer to the excellent Nokia documentation for more tips and tricks on what you can do. I think this is definitely a great example of what power the distribution has, and I’m surprised that more people haven’t been writing about it. Has anyone besides me (and Nokia employees) gotten this working?

With the advent of a language I can program on my phone, my first goal was to learn the language, and my second goal was to rewrite something that I think is an almost-great service, but missing a couple key features. With that in mind, I headed to work, first with my FOAF app. My work after that concentrated on a locative service based around cell towers, and learning the user interface aspects of the Python interface.

Last night, I got my first victory in both: I wrote a decent (although minimal) user interface, which was able to communicate information about my location based on cell towers I was near. You can see the start of the work at http://crschmidt.net/cell/ . As I just described it to another mobile user, it’s going to be “Like CellSpotting, without the suck.” Using services for anything is completely unneccesary, especially with how simple python makes it to build user interfaces. I don’t know how hard it is with the Symbian interface, but I want to make it better.

My code is pretty simple so far, but it works. I will be continuing work on it tonight, but I’m already seeing other people start similar work, and I believe in shared efforts. So, the code is available: the .py.txt is the Nokia python program, and the .php is the server-side script I’m using.

I will be working on changing the way things work a lot, but I just want to show some of the cool things that can be done via the exposed Python interfaces on the mobile platform. Note that the actual Nokia code is only a dozen lines: which builds a nice interface, allows you to edit the form, and to send the data off. There’s not much funtionality yet: not even a way to update your location without exiting and reentering the program. However, it’s a start, upon which others can build.

Over the weekend, I had some fly time to work on my image annotation application. I had asked for a bit of help on the way to get input in Python, and sbp pointed me towards “raw_input()”, which is what my application is based around.

Originally, it was going to be written using the Redland python bindings. I had prepared myself for the flight by installing Redland, and browsing a bunch of pages with annotation examples, which Slogger grabbed for me and stuck into a local log. With this, I thought I was prepared. So, I got on the plane, got past 10,000 feet, opened the laptop, executed my program (I already had about 20 lines of code)… and smacked myself in the head as it complained that there was no module RDF.

You see, Redland has two parts: the library itself and the language bindings. You kinda need both for working in Python.

So, after a little bit of thinking, I remembered that I had installed rdflib for testing of n3p, and decided that I would convert my existing code to that. In about 15 mintues, I was back up and running.

The program is simple, although it’s still lacking some important functionality. It basically just asks a series of questions about the image you tell it to annotate. Sample program input is available, as well as the sample output, and the sample passed through cwm to demonstrate how it looks when cleaned up.

You’ll notice that there’s data there that I didn’t enter: that data is brought in from a FOAF file. This file is only specified in the code at the moment. The code intelligently works on the name you give and checks for either a nickname or name matching: if there are multiples, it provides you a list from which you can choose a number given. In any entry form, you can just skip enter to either accept the default or skip past it.

The source is available, as depiction.py. This source code is messy, the way it’s laid out is very procedural, and you’ll have to modify things inside the code in order to get it to work for you. (Specifically, the foaf.rdf file is hardcoded to the location of mine.) The wordnet features are the newest and the least complete. I’m going to continue working on it, and the application is not considered even alpha-level release yet. However, I know that other people are interested in the arena, so feel free to take the code. You will need RDFLib to run it. It is MIT licensed. Share and enjoy!

An update to my previous post about bad documentation:

I’m blind.

Quite clearly described on page 18 in the API documentation is the popup_menu feature. I’m not sure how I missed it, as I thought I went over the entire UI section 2 or 3 times looking for it, but it’s right there, plain as the nose on my face. Thanks to Jukka Laurila for pointing this out to me.

Secondly, the additional documentation referenced in the references section of the API doc are included in the distribution: I had lost them in the sea of documents in the folder I unzipped into, since the zip file that I downloaded didn’t have the files inside it stored in a seperate folder. (Yet the Examples zip file inside that folder did). I still think that at least mentioning that they are included would be a good idea, for people who don’ot realize it.

So, my two issues with documentation are mostly resolved: both were due to me not seeing what was right in front of me.

However, the error with hitting return while on a word in t9 is definitely reproducible, so I don’t think that one is something I’m just making a mistake on.

Nokia has earned more than I’ve given them credit for. Nice job, and thanks again.

Although I love being able to develop things quickly and easily on my 3650, there’s a few major sticking points for me that I don’t understand. I know that my phone is considered an “untested” phone for this application: it’s almost the oldest Series 60 around, so I don’t expect everything to work nicely, or even at all. I was totally psyched when I learned that I was on the accepted list. Yet there are some things that bug me.

Lack of Documentation: Nokia has done an admirable job at documenting what their Python does and does not do, and has done pretty awesome, except for a couple things. First of all, there is completely undocumented UI funtionality in the Examples code that they give out. (Specifically, for the pop up choices: appuifw.popup_menu). I don’t know how much it bugs me that I looked through their entire PDF documentation 3-4 times looking for this feature described and found nothing. For a great project otherwise, to skip things like this when you’re releasing a public version is just a bit annoying. Additionally, 3 of the 6 references from the API reference don’t go anywhere: they’re just titles of publications, with no additional information. I believe two of these files are included with the distribution (I can’t check right now: The batteries of the computer I downloaded it to are dead), howeverm, to not even mention that and only list a title is again, quite silly.

Clear Lack of Testing: Again, as I said, the 3650 is not a “well supported” phone in this endeavour. However, there is a relatively major crasher bug in the Python distribution: when running in the Interactive Shell, if you press the “return” key equivilant while still on a word while using predictive text… it crashes.

That’s right, if you’re using a built in feature of the interface, it crashes. For no apparent reason. Doing something that most python programmers will do first, before anything. (The interactive shell in python is impressive, and is a good “sandbox” type environment to test small scripts out.) Granted, on a cell phone, the interactive shell users are going to be relatively slim: who wants to type on a phone when you can just type on a computer and send it over? However, to leave a standard python feature so broken is just silly.

Other than these two minor niggles, I’m very impressed, and hope to see the available UI interfaces grow over time: I’d like to be able to have as much control over user interface tools as Symbian native apps do, and I think Nokia would like to see that as well. I’m especially happy with the “location” module, which allows you to retrieve Network, Area and Cell IDs: I can now write my own tool to do MiniGPS-type cell storage, as well as uploading data to MeNow.

A list of standard modules which are included:
anydbm, atexit, base64, bdb, binascii, cmd, code, codecs, codeop, copy, copy_reg, cStringIO, dis, errno, exceptions, __future__, httplib, imp, keyword, linecache, marshal, math, md5, mimetools, operator, os, pdb, quopri, random, re, repr, rfc822, socket, sre, string, StringIO, struct, sys, thread, threading, time, traceback, types, urllib, urlparse (urlsplit only), uu, warnings, whichdb, xreadlines, as well as a “location” module for determining GSM cell info and “messaging” for sending SMS messages. Additionally, there is a UI Specific module, appuifw, the e32 Module, which offers some Symbian related utilities, e32db, a relational database module, and the e32dbm module, which offers an API to the Symbian RDBMS.

All in all, an impressive an useful subsection of the modules which most people use. The socket library has been extended to include a number of bluetooth functions, which apparently work quite well (I wouldn’t know, haven’t tried them yet.) I’m very pleased. It’s fun. But I think nokia could have waited just a little longer to ship this, to make sure that some things weren’t broken.

Today, Nokia released Python for Series 60 based phones. I’d been hearing about this stuff for months, but it had been in closed beta, so I hadn’t paid much attention to it. In addition, when this stuff was being discussed originally, back in April, I’d never had a hand in any Python coding: I was a Perl and PHP hacker only.

However, when mattb asked today who would be the first to write an RDF parser that would run on the phone, I asked what he was talking about, and received a pointer to Nokia’s announcement. I immediately started playing. (A bad thing, given that I had work today.)

So, in 5-10 minute increments, plus hacking after the kids went to bed, some help from sbp in the form of an ntriples parser written in Python, some help from Dave Beckett and Redland in the form of an easy RDF/XML->ntriples conversion, I now have a working (although clunky) FOAF application for the Series 60 phone running Python.

So, without further ado, I explain to you the steps behind the process:

First, the user runs Python. This is installed the way any other application is. Once Python is installed, the user must send two files to their phone: an ntriples parser and the actual FOAF Service script. The ntriples script is installed as a “library”: this means that it is installed on the include path, so that it can be imported as a module. The FOAF script is installed as a script.

The user opens Python, and chooses to run a script. They choose the script that they installed, and run it. A standard query box opens, and asks the user for a URL. Here, they enter the URL of a FOAF file. Once they do, the file is passed through an rdf/xml->ntriples web service: this converts most FOAF files to a format the phone can understand. An example URL for this webservice is http://crschmidt.net/semweb/xml2nt.cgi?uri=http://crschmidt.net/foaf.rdf : It is created using Redland, and source is available at http://crschmidt.net/semweb/xml2nt.cgis . It uses the Redland Python bindings. You can feel free to use this web service: however, if you’re going to be exceeding 10 hits/minute, please let me know so I can find someplace better to host it.

Much of the time for the script to run is simply fetching this content: my FOAF file generates 70k of ntriples, which is then parsed into a simple list of triples. Some post processing sorts the data, and looks for a personal profile document to whom the document can be attributed. If it is found, then the script prints out a list of contact information for that person, including email and contact IDs.

Many thanks to mattb for pointing out the Python release, sbp for his invaluable ntriples parser and his help getting it working, and to other people who muttered encouragement throughout the day.

The source is available at my Symbian Semantic Web Page. More images are available in my Symbian Hacking gallery.