Technical Ramblings

My most ‘interesting’ photo

One of the things that has always confused me is how Flickr’s ‘interestingness’ score worked. It’s clearly not based directly on views — my most ‘interesting’ photos have ~500 views, while my most popular ones have ~5000. The same is true of comments, tags, groups, etc: Some of my most interesting photos are, all in all, the least obviously ‘popular’ ones.

However, reading through the patent on Interestingness, I see:

[0027] The statistics engine generates statistics and other metrics based upon aggregated metadata. In one embodiment, the statistics engine determines the popularity of metadata (e.g., tags) within a grouping of media objects over a predetermined time period. For example, the statistics engine may determine the number of different users that have assigned a particular tag to one or more media objects within all groups on the system, within a single group, or within a set of media objects, over the last 24 hours. The aggregation engine may determine (and display) a histogram of the tags, and may determine the most frequently assigned tags (at any point in time or over a predetermined time period) by determining those tags either having a frequency exceeding a minimum threshold frequency or belonging to a predetermined number of the most popular tags.

The patent application doesn’t directly claim that this is used in the interestingness calculation, but this type of time-period based tag aggregation/valuation is clearly a non-obvious metric that can’t be calculated by looking directly at user-visible information on a single photo.

Still, none of this really changes the fact that I wish that flickr had a per-person ‘most interesting this week’ sort: my ‘eastern bunny’ photo above has been at the top of the interesting list for ages, and I want new interesting photos, damnit!

I finally finished uploading all the photos I took at/around the FOSS4G conference.

FOSS4G 2008 Set

Includes photos from:

• Various nights hanging out at the bars/hotels during the conference
• A trip up Table Mountain with OpenLayers/OpenGeo folks
• GeoDjango Workshop
• Sessions, exhibition hall, etc.
• Pictures from the Gala Dinner at Moyo
• Closing Session
• OSGeo AGM
• OpenLayers Workshop
• GeoServer Workshop

Photos that are taken at the conference center/of conference proceedings are also tagged with foss4g2008:

foss4g2008 tagged

And most photos of people who I recognized are tagged with first name in the title, and with a username (as used on IRC or other unique identifier) attached as a tag:

Photos of Arnulf
Photos of Steven

Photos of the OSGeo AGM are also tagged as such:

OSGeo AGM Photos

I’ve made my tagging settings as open as possible, but I believe you still need to be a contact on flickr in order to add notes/tags to photos: simply add me as a contact, and I’ll add you back.

Any names I got wrong, please let me know, either via email or by simply commenting on the flickr photo.

Thanks to all for helping to create such a photogenic conference.

A bit more than a year ago, when I was just learning how to use the Google Maps API, I put together a patch for the OpenGuides software, adding Google Maps support. It seemed the logical way to go: It wasn’t perfect, since Google Maps are obviously non-free, but it seemed like a better way to get the geographic output from OpenGuides out there than anything else at the time.

Since I did that, I’ve learned a lot. Remember that 18 months ago, I’d never installed MapServer, had no idea what PostGIS was, and didn’t realize that there were free alternatives to some of the things that Google had done. Also, 9 months ago, there was no OpenLayers, or any decent open alternative to the Google Maps API.

In the past 18 months, that’s all changed. I’ve done map cartography, I’ve done setting up of map servers, and I worked full time for several months on the OpenLayers project. Although my direction has changed slightly, I still work heavily with maps on a daily basis, and spend more of my time on things like TileCache, which lets you serve map tiles at hundreds of requests/second.

So, about a month ago, I went back to the Open Guide to Boston, and converted all the Google Maps API calls to OpenLayers API calls. The conversion took about an hour, as I replaced all the templates with the different code. (If I was writing it again, it would have taken less time, but this was my first large scale open source Javascript undertaking, long before I gained the knowledge I now have from working with OpenLayers.) In that hour, I was able to convert all the existing maps to use free data from MassGIS, rather than the copyrighted data from Google, and to have Google as a backup: a Map of Furniture Stores can show you the different. You’ll see that there are several layers — one of which is a roadmap provided by me, one from Google — and one from the USGS, topographic quad charts.

It’s possible that some of this could have been done using Google as the tool. There’s nothing really magical here. But now, the data in the guide is no longer displayed by default on top of closed source data that no one can have access to. Instead, it’s displayed on top of an open dataset provided by my state government.

This is how the world should work. The data that the government collects should be made available to the people for things exactly like this. It shouldn’t require a ‘grassroots remapping’: There are examples out there of how to do it right. I find it so depressing to talk to friends in the UK, who not only don’t have the 1:5000 scale quality road data that Massachusetts provides, but doesn’t even provide TIGER-level data that the geocoder on the Open Guide to Boston uses.

Free Guides, with Free Maps. That’s the way it should be. The fact that it isn’t everywhere is sad, but at least it’s good to know that the technology is there. Switching from Google to OpenLayers is an easy task — it’s what happens next that is a problem. You need the data from somewhere, and it’s unfortunate that that ‘somewhere’ needs to be Google for so many people. I’m thankful to MassGIS and to the US Government for providing the data I can use, and to all the people who helped me learn enough to realize that using Google for everything is heading the wrong way when you want to not be beholden to a specific set of restrictions placed on a corporate entity.

So it seems clear to me that the Pipes application is a step in a really cool direction. I don’t know if there’s anything incredibly innovative in the idea of making programming easy, but Yahoo! has gone a long way towards the goals that other people have put into place. Ning thought that letting people code would be the way forward: give them a sandbox, let them copy paste, and they’ll build applications. The idea was right: there are a lot more people out there who want to be builders that aren’t. It turned out that the people who want to be builders didn’t have the skill level that they needed to build PHP code, even with mix/match and copy/paste.

Yahoo! Pipes is the followthrough on that idea: make it possible for people to take a set of input, and get a set of output, passing it through multiple filters.

The next step is obvious: Let people turn the filter settings into a web page, with the output being another web page. Search for all content 5 miles from a given Craigslist location: Take the user input as drop down boxes or something in an HTML form, and make the output a Yahoo! Map. Boom: you’ve turned everyone who can create a pipe into a web application builder. Stick ads along the bottom, and you’ve done one of the things that Ning tried to do: make money off applications in the same way that so many have made money off content.

I’m sure that Yahoo! already has this in mind, whether they’ve written about it or done it yet or not. It’s only a matter of time. It does make me wonder if someone could build something that did this without needing Yahoo! to do it… It seems like at the moment it would require altering a pipe on the fly, which I don’t see a way to do, so either there needs to be a further API, or we’ll all just need for it to get done 🙂

Update: Looking today, you can control the input of text inputs from the URL that you fetch the RSS with. This means that I can go ahead and build the pipe thingy for my own pipes as is. That’s pretty cool. I’ll show one with MetaCarta stuff on Monday.

Perhaps I’ll build an OpenLayers based Yahoo Pipe output viewer. It wouldn’t be that different from the GeoRSS viewer… but it would need a way to visualize non-Geo content. Ponder ponder.

In the vein of my previous post, I bring you another nifty trick: This time integrating Google Maps and Mapserver (kind of).

Visit static renderer. Browse Google Map. Then, when you’re looking where you want to be, hit the link up top, and off you go — transported to a world where data is public domain or licensed for re-use 🙂

I’m still trying to find a happy medium level of size of markers — when you’re zoomed way out, they’re too big, but when you zoom way in, they’re too small, so I don’t know what to do, but it *works* and that’s the important part.

Interface suggestions welcome — perhaps a side by side view is better? (I think I may be having a bit too much fun…)

After hours of fighting with Postgis, Schuyler finally got bia into a state where she would do the right thing when told to install it, so I was able to load the data from the Open Guide to Boston into postgis, and from there, to talk to it with mapserver. The result is a couple of pretty cool looking maps: Boston Metro, and Boston Metro Big, 1000×800 and 2000×1600 respectively.

The maps underneath are provided by Public Domain datasets, wrapped up in a tidy little easy to use package by the folks at OpenPlans through their “Sigma” project, and I’m extremely grateful to them for their efforts! They’ve saved me a ton of work, and allowed me to produce something that looks pretty damn cool.

If that’s not an advertisement for Open Geodata, I don’t know what is.

I’ve set up a mailing list for the Open Guide to Boston: sign up if you’re interested.

Likely topics on the list:
* Local get togethers to discuss the guide — technical, social, intro to beginners, etc.
* Requests for help in some aspect of the guide — looking for images, or help maintaining some specific feature.
* Discussion of changes made to the guide technically
* Discussion of changes made to the guide socially: policies, etc.

Thus far, there has been no traffic, but I’m about to change that, because I’m working on redesigning the site, and I’m a very bad designer. (I have a few tricks and tips I learned from working as one for far too long, but nothing that makes me qualified.)

So, if you’re interested, please feel free to join the list.

I just closed comments on all posts older than this one. In the past 3 days, I have received 892 spam attempts-at-comments. Everything goes through moderation, so this doesn’t end up being an issue for anyone except for me, who has to sort through all the notifications and spam to try and find the wheat in the chaff.

I hate sites that do this, but it was getting *really* annoying, and nobody comments anyway.

I’ve just implemented a fair amount of memcached support into CGI::Wiki, the Wiki software that powers OpenGuides. Now, any time a node is fetched, it will come from memcached if possible, or be loaded and stored into memcached, and when nodes are edited, the cached version is deleted.

This is in addition to adding caching to the main map pages, which were previously taking up to 20 seconds to load, and are now in the ~1 sec/range. (There is much more slowness in the browser side of this page, due to the javascript proccessing time required to load hundreds of location markers).

Additionally, the Open Guide to Boston is now running under mod_perl after my Message to the OG-Dev list with a patch that fixed the major stumbling block. This has increased speed somewhere between 5 and 10 fold over using CGI.

About 15 minutes ago, I restarted memcached to clear out old stats. I then visited the main index pages for both the Milton Keynes and Boston guides, to prefill the caches. There were 3535 items which were loaded (GET\_MISSes). In that time, there have been 4 new objects added — and 946 GET\_HITs. With the stability of memcached, I expect that this cache will be able to run for a long time with no maintenance, and I’ll be able to come back in a few weeks to see that memcached has saved me thousands of hits to the mysql database, therefore increasing the speed with which the Open Guide to Boston loads, something that I’m sure everyone can appreciate.

Most of the slowness that memcached is fixing in this case is not caused by MySQL being slow, but rather by the processing done by OpenGuides after the fact, due to the way that it is designed. The software was never really built to be the most efficient Wiki software out there, but to scale to the extent that it needed to, a task which it has completed admirably. However, this has left some corner cases where the performance was less than perfect — typically cases which were seldom used. The example of the “All nodes” map that the guide has is a good one: I built that page out of a corner case (the ?action=index view) which is not linked from the guide itself. As a result, I discovered a case where the software did not perform as expected. There was a roundtrip to the database to load the data about each node. This round trip time combined with a complex map { } function brought performance way below acceptable levels. So, I sought to improve that. First I simply cached all the data after it was processed, and today implemented caching of the individual nodes. Although the second change is minor and may not offer significant benefits on the small scale, it does help to build up the larger cache that it’s supporting: Roundtrips to memcached are quicker than roundtrips to the database.

This was originally posted to a local geo-hackers list, but I haven’t
received any response yet. It was long enough that I wanted to publish
it in a Google-able form as well.

I’m Christohper Schmidt, local mapping, geo, mobile application, web
development, RDF, RSS, and otherwise technically oriented hacker. Jo
Walsh mentioned my name on this list a few months back, as the organizer
of the Open Guide to Boston.

I’ve been working for the past couple months on boostrapping the content
in the Open Guide via the zami.com sites for Boston, Cambridge, and
Boston-Metro. Since there is no database-protection on copyright in the
US, pulling the data from these sites for repurposing it is a (as far as
I know) valid legal use of the data, and the Boston OpenGuide has
prospered significantly from these additions in terms of Google hits due
to the sheer size of the dataset.

However, it’s now coming to a point where adding more random content
isn’t helping to expand the guide in a way that’s entirely beneficial.
What is needed to expand the Open Guide to Boston is, quite simply, more
content about the places that are described.

A relative few of the nodes have full content — reviews, pictures, and
so on. Judy Jetson ,
Full Moon ,
Mojo Music . However, the large majority
are relatively devoid of life. I want to bring life to them, because I
want this to be even better than citysearch, even better than anything
else there is out there as a guide to what is around Boston, and I want
to encourage people to use it everywhere.

I have a couple reasons why. First, I hate closed systems with
unneccesary licensing and all the crap that goes along with things like
CitySearch and the other similar city directories. More importantly,
however, I feel that an open source of awesome information on what is in
andd around Boston lets people build more interesting things that pull
that data in. I have several personal goals that would use this data,
enabling users to get more information about what’s around them at
wireless hotspots, or when using a mobile device.

I would love to see more interest from the geo community in and around
Boston and especially my hometown of Cambridge. I’ve pushed for quite a
while to get the content to the level it’s at, but I simply can’ty visit
every restaurant in town. I would love to see more interest from
external parties because it might encourage me to keep on working at
what seems to be a very gargantuan task of documenting every thing I can
in town. An annotated Yellow Pages is a great goal, but requires far
more resources than I can bring to bear.

I’d be willing to demonstrate or pontificate on any of my desires or
goals, if persons are interested in meeting up to do this sometime in
the next couple weeks (or anytime, really). I’d also love to hear what
the Open Guides software isn’t providing that you would like to see: the
Google Maps component is something that I already put together and
committed back to the main tree, and I’ve got a couple other hacks in
the guides hosted on my server that aren’t in the main branch.

Looking forward to hearing from anyone who’s interested in helping,
evangalizing, or suggesting things that might be done better.