Technical Ramblings

Set up mapserver for an interested web-hostee over the weekend. (If you’re interested in hosting a domain with mapserver available, please contact me. Barring extra needs, cost will probably be $10/month.) He set up a WMS+Shapefile layer to demonstrate the power of mapserver for me: this is now available at massradar. Radar imagery is provided via an Iowa State WMS Server, which also has data for the Entire country. Now I really want to set up Google Maps with a radar layer.

The crschmidt.net site now uses in-page validation to provide a visual indication of whether the source is valid. See the smiley face in the upper right corner, which is an indicator of validation or not. It’s linked to the validator, so you can see information about it by clicking the link.

Code stolen from sbp’s ‘valid’ project.

If you’re reading this, welcome to Bia, the new home of crschmidt.net and associated webhosting.

Athena services are slowly switching to bia. I’ll do a rundown of what I did post-friday, when I’m sure I’ve actually got everything working, but one cool trick I pulled is to use:

for i in 25 110 143 587 993 995; do
iptables -t nat -A PREROUTING -p tcp –dport $i -j DNAT –to-destination 64.92.170.181:$i; iptables -t nat -A POSTROUTING -p tcp -d 64.92.170.181 –dport $i -j SNAT –to-source 65.110.51.60;
done

The biggest problem I had been having up until this point was how to migrate mail. Web stuff is easy by comparison: You just tell people not to update their MySQL content in the meantime, and have them switch their DNS. But mail is hard. During the time when DNS has been switched, hours or days can go by with cached MX records clogging up older systems. This system basically turned athena into a beefy NAT router for bia on all mail ports: Thus, when someone sends mail, it opens port 25 on athena, which actually opens port 25 on bia, and talks to it. Immediately after this, I did a final rsync of all the /home/vmail/ dirs, and turned off Postfix on Athena.

Pretty nifty solution to what seemed to be a hard problem.

Crucial Paradigm, by the way, has been an extremely helpful ally in the migration process. Many thanks to them for helping me get set up in my new home.

More details on the move and the reasons for it coming soon.

Thursday night at the Merrimack Linux Users Group, we met at Marthas, in Nashua., as per our usual. We accidentally ended up split into two groups to start, but shortly after I arrived at 7:20, adjourned upstairs.

It was one of the larger meetings I’ve been too, with about 15-20 people in attendance all told. The informal discussion was centered around theWRT54G, and OpenWRT. Our friend from the Great White North (“That’s Maine, right?”) showed off his WRT router, which he had hacked to bitsand back again. Hardware changes include:

* Addition of two serial ports, one for the console, one for standard serial comms.
* Addition of SD Memory card
* Modification of USB GPS to work as serial GPS
* Modification of power input to work from cigarette lighter.

The router software was then modified to work as a wardriver-in-a-box: GPS and wifi hotspot locations are recorded and stored to the SD card.

There was then some discussion about geo stuff: GPSes, how to work withthem under Linux, how the GPS system works, and more.

Some stuff which were tossed around:
* gpsd, the gps daemon that makes communicating with GPS devices easier under Linux. This software basically turns a GPS device into something you can telnet to and ask for a current position.
* GPSDrive. This is mapping software, which downloads free maps from the web, and displays your current location on the map.
* Kismet, a wardriving program.
* The increase of macs in the Linux users culture.
* Open Guide to Boston (http://boston.openguides.org/)
* Lots of hardware mumbo jumbo I didn’t understand
* Some software stuff about the WRT54G, a la discussion of NVRAM.

Afterwards, we adjourned for dessert, and discussed the incompetencies of the Mass RMV (http://crschmidt.livejournal.com/311494.html), the interesting ways of escaping West Berlin without a passport, and more geo geekery.

I think that’s a good summary of what was discussed: I had a few pictures that I’ll put on Flickr of the hacked device and the flock of geeks adoring it, at some point.

(Originally posted to GNHLUG mailing list. GNHLUG is the Greater New Hampshire Linux Users Group, one of the most intelligent and helpful bunches of Linux people around. I trek up to Nashua from Cambridge once a week to meet with these people because I can’t imagine a more interesting bunch of people to hang around.)

One of my biggest problems so far in my writing of mobile software is that I don’t have any form of regular communication with the users of these applications. Support queries, questions, etc. get posted in blogs and so on that I never read, feedback is posted to wikis I never use, or worse, it goes into email and conversations are lost forever.

In a possibly misguided attempt to fix this, I’ve set up a Symbian Hacks mailing list. This mailing list is designed to serve as a feedback mechanism for the code that I’ve written for the mobile platform, and hopefully will encourage me to give back to the community in more meaningful ways.

If you’ve ever used something I’ve written on your phone, please join the list and just send an email saying “I used $foo and I loved it!” You will really make my day.

Today, there are a large number of cheap bluetooth GPS devices on the market. These devices allow you to connect to the device wirelessly, which is great for when you’re driving and don’t want cables draped all over the car.

However, what happens when you can’t drag your computer out to act as a display of your position? No bluetooth GPS on the market today for under $500 has a display of any kind. When you consider that these things can be had for $70, that makes purchasing one go from likely, to ridiculous. What’s the point of a handheld, easy to use GPS if you can’t use it to see where you are?

If you’ve been asking this question, I’ve got software which has an answer for you. GPSDisplay will allow you to connect to NMEA compatible Bluetooth GPS devices and display your position fix. It is written entirely in Python, using NMEA code from Forum Nokia. It requires only that you first install Python for Series 60 on your device, and should work on all first and second generation Series 60 phones. Simply download the .sis file, send it to your phone, and you’ll be all set to go — you can use your phone as a display for that new cheap Bluetooth GPS you bought, and stop dragging your laptop out into the woods to go Geocaching.

I’m prepping my first actual .sis file release for the Symbian platform today. I’ve actually been doing it for the past couple days, but today is when I’m going to put in the brunt of the effort.

Most people don’t realize that for me, a release is a process, similar to the way any software company would do it. Prior to a release, I do many things, like checking in code to a revision control system, creating a webpage for the application, gathering screenshots, writing instructions and simple informational blurbs on a product, and so on.

It also leads up to me doing a markeitng blitz for the “product”, even if it’s free: doing my best to inform the persons who would be most interested in the newly released product that it is available. This includes posting in weblog, listing in IRC channels, etc.

Really, to create a full release, it takes me about a half day to get code from “Working on my phone” to the condition where I want to declare a release point, with ample documentation of what I’ve done, why I’ve done it, and why I think it might be useful.

This is going to be my first release using some new tools: specifically Py2SisNG, an *excellent* tool for creating .sis files out of Python applications. The documentation for the app is very good, and if you read the README in the distribution, it really does tell you step by step everything you need to do. I can’t thank the creators of this application enough. It has allowed me to turn my amatuerish Python script into an actual application, something I thought I’d never be able to do on a non-Windows platform.

Of course, there is still one limitation to what I can do: I can’t figure out how to create a .mbm image for the icon of the application. It seems like this little-used format is popular in the mobile world for storing multiple bitmaps in one file, but that there is little information on how to create these images under non-Windows platforms. I’m not sure if I’m going to delay the release until I have an icon for the application, but I’m considering it.

If anyone has any experience in creating .mbm images under non-Windows platforms, please let me know! I’d love to be able to put an icon onto my application to make it look just a little more complete.

Many different projects (Redland, svn, and wxWindows included) have seen cases where users have attempted to compile, and seen the errors:

/usr/bin/libtool: for architecture: cputype (16777234) cpusubtype (0) file: -lSystem is not an object file (not allowed in a library)

or similar, posted to the mailing lists of these projects.

Only one place that I’ve been able to find so far (and not easily!) has the answer:

This is the typical error you get when you do an upgrade install of
Panther -> Tiger but you don’t install the Tiger Developer Tools
(Xcode 2.0). Don’t do that (Do upgrade your dev tools)

(From darwinports mailing list.)

Thank you ssen! Now, anyone else who has seen this will hopefully be able to read this post for now and into the future.

Ran into this attempting to compile gpsd.

Thanks to Schuyler and Rich Gibson, I now have a spatially aware postgres database.

Later today, thanks to Schuyler and zool, I’ll have a copy of Mapping Hacks, and a bluetooth GPS.

Last night, I learned how to use centroid(), astext(), and distance_spheroid(), and calculated the distance from my house to zool’s house, and from there to Darwin’s, where I ate lunch and used the wireless yesterday. I loaded some data, learned the frustration of having data in different projections, and learned a little bit about the various types of geometry. I loaded data from an ESRI shapefile. I found that “” in Postgres is equivilant to “ in mysql — that is, “GEOMETRY” means ‘the value from column Geometry’, not ‘GEOMETRY’, which is the literal. (If you ever get “Error: column “Foo” does not exist, that might be a good thing to check.)

Last night, I made my first foray into spatial databases.

Last night, I took control of space on my machine.

Tomorrow, I take control of space in the world!

But today, I need to work on things that I’m actually paid for at the moment. 😉

So, tonight I took it upon myself to redo the javascript behind the map for the Open Guide to Boston. The reason for this is relatively simple: when I first wrote the Google Maps interface, the guide had about 50 nodes. With that in mind, drawing them all was acceptable. As the guide got bigger, it got a bit more time consuming, but was still much easier than paging. However, the guide has recently doubled in size as I’ve increased the rate at which I pull data from Zami.com (specifically for the purpose of trying to build a free database of all the churches in the area with user interaction) — up to 2400 nodes (making it the single largest Open Guide to date by pure node size, as far as I can tell).

With this change, the map was no longer just difficult to use: it was impossible. Attempting to open the page crashed my web browser.

So, I took it upon myself to learn enough JavaScript to do paging… but then decided that rather than paging, I should attempt a bit more friendly solution to start. So I started hacking, and got into the Google Maps getBoundsLatLng() function, which allows me to determine the area of the map.

There are two basic options to go from here, depending on performance you expect in various situations, and the scalability you want to have.

* Load all data. Create Javascript variables to store it all. When the map moves, iterate over the data, adding markers for only points which are not already visible, and are in the new span.
* Load only the original data from the database. All additional data to be loaded via XMLHttpRequest.

The first is obviously something that can be done entirely client side, and in fact turned out to be the (much easier than I expected) path I chose. The largest reason for this is simply ease of integration. OpenGuides code is rather convoluted (to me), and modifying it is not something that I enjoy spending a lot of time doing. I’ll do it when I need to, but I prefer to avoid it. The other option would be to simply query the database directly via Perl or some other language, without using the OG framework. This would probably be slightly faster, but with the size of data I’m using, iterating over it is a minimal time compared to the time drawing the GMaps Markers. As a result, I chose to go with the slower, less scalable, but quicker-to-implement and merge to other guides way of doing things. The biggest benefit here is that I can merge it back into the other quickly growing guides — Saint Paul is now growing leaps and bounds alongside Boston, due to help from pulling Zami.com’s database, and I plan to do similar things with other guides. Also, London probably will not be able to use the mapping stuff in any useful way without this kind of code being added, so I went for the quickest thing that could possibly work.

There were a few gotchas that I had to end up avoiding:

* Removing Google Maps Markers takes *much* longer than adding them. Like, 3-4 times longer in extremely informal testing. With one or two, or even 20-30, this doesn’t matter, but with 100, this starts to be a significant barrier to user interaction.
* There are a number of words which are reserved in Javascript, but are not treated that way in Firefox. This leads to confusing error messages in Safari (although they are slightly improved in the latest webkit): If you see a ParseError, you should check the Firefox Reserved Words list — this is in one of their bugs. For example, the variable “long” is reserved, and can not be used to pass something like, say, longitude.

Discovering the ParseError, working out why it was there, and how to fix it, was greatly helped by the #webkit folks: I have never seen a more dedicated, hardworking, friendly, open and inviting group of Open Source Software developers in my history as a programmer, either in closed or open source products. Many thanks to them for helping me work through why the error was happening, and how to avoid it in the future.

Lastly (although this is not quite chronological, as I did this first), I had to modify the code to the guide to zoom in father to start, so it wouldn’t try to load so many nodes. I think a next step is to allow users to set a default lat/long for themselves, a la Google Local, from which they can start browsing, rather than always dropping them into the main map. But that’s a problem for a nother late night hack session.