Technical Ramblings

Edward Ho responds regarding callbacks in Pipes:

Apologies for the inconsistency here. JSON Callback is something we believe in for our APIs and we plan on adding it to Pipes shortly.
Thanks!

Woot.

So, I went through this morning and created GeoRSS output for MetaCarta search results. It seems likely that for the forseeable future, the way to get data in/out of Yahoo! Pipes is to build a URL-input, RSS output format. Since we already do KML, adding RSS took about 20 minutes — just changing some tag names, etc.

After doing that, I started playing with some pipes. After some missteps — dragging modules into the interface seemed to not work in a way that later did, presumably just part of the fun of dealing with these funky interfaces — I got something that I think is pretty cool coming out.

• Take NY Times input. Recent news is relevant and interesting.
• Pass the NY Times input through the content analyzer. This gets a set of relevant keywords.
• For each item, take the keywords and pass them through a subpipe…
• the subpipe goes to the MetaCarta search interface, and does a keyword search, taking the session information and bounding box from the text inputs for the pipe, returning 5 most relevant item.
• Add the 5 results to the item, and then send the output to the end of the pipe.

I’ve published the pipe, and after running it you can get the JSON output. (It seems that the sub-attributes are not available in the RSS output, at least as far as I can tell.)
Next steps:

• Create an OpenLayers interface, and use the current bounding box as the bounding box for the queries.
• Have the interface set the session and token variables: the pipe currently works with the defaults, but only for the next couple hours, since the tokens are time limited. (Currently, the MetaCarta Web Services are wide open, but eventually, they won’t be.)

The Query MC subpipe is probably useful for anyone who wants to do something like this: I’d recommend checking it out if you’re interested. Next step is to add RSS support to the other services, specifically the GeoTagger. I think that this would let me do what I want: take description text from an RSS item, tag it, and set the location attributes of the returned item.

I’m hoping that someone can answer how to use a callback on JSON output of pipes, since that would let me skip the need for a proxy, and make it entirely client side so that other people could run it on their own sites without a proxy.

So it seems clear to me that the Pipes application is a step in a really cool direction. I don’t know if there’s anything incredibly innovative in the idea of making programming easy, but Yahoo! has gone a long way towards the goals that other people have put into place. Ning thought that letting people code would be the way forward: give them a sandbox, let them copy paste, and they’ll build applications. The idea was right: there are a lot more people out there who want to be builders that aren’t. It turned out that the people who want to be builders didn’t have the skill level that they needed to build PHP code, even with mix/match and copy/paste.

Yahoo! Pipes is the followthrough on that idea: make it possible for people to take a set of input, and get a set of output, passing it through multiple filters.

The next step is obvious: Let people turn the filter settings into a web page, with the output being another web page. Search for all content 5 miles from a given Craigslist location: Take the user input as drop down boxes or something in an HTML form, and make the output a Yahoo! Map. Boom: you’ve turned everyone who can create a pipe into a web application builder. Stick ads along the bottom, and you’ve done one of the things that Ning tried to do: make money off applications in the same way that so many have made money off content.

I’m sure that Yahoo! already has this in mind, whether they’ve written about it or done it yet or not. It’s only a matter of time. It does make me wonder if someone could build something that did this without needing Yahoo! to do it… It seems like at the moment it would require altering a pipe on the fly, which I don’t see a way to do, so either there needs to be a further API, or we’ll all just need for it to get done 🙂

Update: Looking today, you can control the input of text inputs from the URL that you fetch the RSS with. This means that I can go ahead and build the pipe thingy for my own pipes as is. That’s pretty cool. I’ll show one with MetaCarta stuff on Monday.

Perhaps I’ll build an OpenLayers based Yahoo Pipe output viewer. It wouldn’t be that different from the GeoRSS viewer… but it would need a way to visualize non-Geo content. Ponder ponder.

A prize to anyone who can make a simple Atom entry or a simple RSS entry get geocoded by the Yahoo! Pipes Location Extractor. I’ve spent the last 30 minutes on it, and failed.

Non-working pipe is my attempt. GeoRSS works, but location extraction doesn’t.

Update:

1. Location Extractor seems to work against the HTML pages referenced by the feed, not the content in the feed.
2. Minor changes to the HTML page seem to break the parsing — it seems to be very targeted towards Craigslist postings.  A page with my address, but a different map link seems to extract the Sebastapool address, while a page with just a map link doesn’t seem to extract at all.

I guess I don’t need to sell my MetaCarta stock yet… unless I’m way off, this shows that MetaCarta is significantly ahead of the game for extracting locations from unstructured text. Not that this is a surprise to me 😉

Is it possible to make your own module with Yahoo! Pipes? I was looking around and didn’t see anything… I’d really like to be able to hook up something that grabs locations from the MetaCarta Web Services, and then let people drop it into their own pipelines… I’d be willing to bet that the Location Extractor pipe module wouldn’t pull out “20 miles north of London”, but with the MetaCarta GeoTagger, I could…