Technical Ramblings

A long time ago, when I first started working in the Ning playground, the first thing that impressed me about it was the fact that it had ready built components for so much *stuff*. Comments. Ratings. Flickr. Amazon. Calendars. Questionairres (Which is a very difficult word to spell right.)

Last night, after setting up X\_Query, I thought “hm, what if people want to comment?” And as I did, I remembered those components. I went into the developer documentation and looked up XNC\_Comment, which had a great, simple example of how to add comments to any content object.

But I didn’t have any content objects. The app is built as an API, so there was no need for Content Objects (at least not until I build up the documentation).

So I built a couple quick lines to add Content where I need it:

<?php
$d = XN\_Query::create(‘content’)->filter(“owner”)->filter(“type”, “=”, “Page”)->filter(‘title’, ‘=’, $\_SERVER[‘SCRIPT\_FILENAME’])->uniqueResult();
if (!$d) {
$d = XN\_Content::Create(“Page”, $\_SERVER[‘SCRIPT\_FILENAME’]);
$d->save();
}
?>

Once this is done, the $d is an object for the current page. Once you’ve done that, you can just slightly modify the example from XNC_Comment:

require\_once ‘XNC/Comment.php’;
$newComment = new XNC\_Comment($d);
// Handle any form submission of adding a new comment
if ($newComment->willProcessForm()) {
    $newComment->processForm();
} elseif ($newComment->lastError() != XNC\_Comment::ERROR\_FORM\_ABSENT) {
    print $newComment->lastError();
}
// Display the add a comment form (unless it was just submitted and saved)
if ($newComment->canProcessForm()) {
    echo $newComment->buildForm();
}
// Display a list of comments belonging to a parent object
if ($d->my->content($newComment->referenceAttribute,true)) {
    foreach ($d->my->content($newComment->referenceAttribute,true) as $comment) {
        echo new XNC\_Comment($comment);
    }
}

Now, you’ve got comments on any page you want! Just include these two chunks of code at the bottom of each page. This is one of the easiest things I’ve done. Now anyone can take these code snippets and use them in any Ning app. Hm… maybe the next step is to build a code snippets repository…

Attended my old Linux Users Group in NH last night: there was no scheduled speaker, so it was mostly just a “hang out and talk” type of meeting at Martha’s, in Nashua, NH.

Here’s a summary of how the meeting went from my point of view:

Had fun with all the MerriLUG folks last night. I also got to experience driving out of the Boston area during rush hour last night for the first time, which was significantly less fun.

Some things which were discussed:

* Results of the recent quarterly meeting, and location of the next one
* General questions on Linux:
** Why won’t my screen turn off when the computer goes into standby?
** How can I do load testing of MySQL and network traffic?
* Raffling off of books from Ken
* How Vendor/Client relationships are like teenage sex: They’re hormone driven and have no basis whatsoever in reality.

I also led some discussion on Ning. The reason I brought it up was in part because it is following the model of the open source world so much more closely than many other
services out there:

* Ning provides a “hosted” PHP framework
* All code, by default, is “open source” — can be viewed by anyone with an account.
* Users can “clone” applications: take the current application and customize it to their own liking from the same code
* Data is stored in a universal content store, and data is (by default) accessible to all applications across the server. So, my application “gnhlugbookshelf” can also read from the “restaurantreviewswithmaps application
* Income comes in via advertisements sold on the sidebar of the applications, as well as premium services (more space, removal of view source links, and the like)

Most of the time when this is described, it’s described as an “experiment” – can a company make money solely off ads to run their servers? Can premium services pay for all this? My experience with LiveJournal says yes: LiveJournal makes all its money off premium services (no ads), and they gross several million a year. However, their employees are paid much less than Ning’s are, so who knows.

I also mentioned the creation of GNHLUG Bookshelf, an application which “aims to store the suggestions and recommendations of the New Hampshire Linux Users Group on technical books that are the most useful of the bunch.”

Lots of interesting discussion on the business model behind Ning, where it could go from here, and how service is really where the money is these days. Give away the code: sell the service. RedHat learned from this model, and others are doing so too – Ning is simply a widescale demonstration of “give away the code”.

There are aspects of Ning which aren’t given away: the code that runs the playground itself is not open source, but the applications that run on it are. The “secret bits” are still probably important enough to Ning that open sourcing would give away a competitive advantage, but the PHP bits aren’t, since they’re easily reproducible.

Another thing that came up is how much control you have over code you write for Ning and taking it elsewhere. This is a question that has come up in developer discussion before, about how to take your application code elsewhere. What it comes down to is that Ning provides a lot of functionality: shared content store, tagging, user auth, etc., that doesn’t exist anywhere else. There’s no clone of the functionality which you can drop in and replace with something else. There’s nothing to stop people from mirroring the API and creating a way to drop Ning apps into your own webserver, it just hasn’t been done yet. So although you can take the code with you – you own it – it doesn’t do you much good without a lot of work to reproduce the functionality that Ning already provides for you.

Something I was previously unaware of:

The reason for the two different variations of “and” and “or” operators is that they operate at different precedences.

PHP’s && vs. and have different precedence? Who thought this was a smart idea? What end or purpose could it possibly serve?

Yet another one of the things that just makes me smack my head. It does explain a few things — why I couldn’t do variable setting as I might in Perl, a la:

$var = $test || $default;

In this case, the test occurs first, setting $var = 1; Had I used:

$var = $test or $default;

All would have been fine.

How can someone think this is sane?

Most of my work on Ning thus far has been in editing other apps, a large portion of it in testing and QA work. Since the launch, I’ve been taking a couple days “off” — still playing with ning, but at my own pace rather than at a “Almost time to release” pace.

Today, I built my first complete and clonable app from scratch.

Planet Ning is a planet style aggregation tool, supporting RSS 2.0 fields and the ability to rate any RSS item. This app took a total of about 1.5 hours to put together, a large part of that simply playing with design to get it to look semi decent. It uses the RSS alpha component for RSS parsing, XNC_HTML for the page bottom navigation, XNC_Rating for ratings, and the “bot” option to load new posts.

Pretty cool little app. One spot of trouble I ran into: XNC_Rating is designed to have only one form field per page. I overrode just one function to fix it though, testament to the flexibility of the code behind some of the XNC components. This is also a testament to Object Oriented code, and PHP 5 in general: in PHP4, I’m sure this would have been much harder, but PHP5’s exceptional class support, along with exception handling, make it so much more fun to work with than PHP4 was for me.

You can feel free to clone the app and play with it: it’s not ‘complete’ by any means, but it definitely works.

For the past 4 weeks or so, I’ve been working on a project known previoiusly as 24 Hour Laundry.

Now, it’s no longer 24HL: Welcome Ning.

A development playground with all kinds of neat and nifty toys, Ning is attempting to do to application and code sharing what other apps have done to photos, bookmarks or other arenas. Allowing people to clone, mix, and create new apps.

There’s a lot of cool things here, and I’ve got a pretty bad headache, so I’m not going to be able to cover all the things that I would like to here, but here’s some of the cooler things about the site:

* System wide content store. Public content which is created can be accessed by any application. This content store is well abstracted, and has a content creation and query system. You don’t have to worry about scaling up: You can leave that to the professionals in the backend. At the same time, you can collect data from all the other apps in the playground. You want to create a book reviews site? First, grab everything that’s known as a Book from the site, and then use the built in classes for ratings and comments to build a discussion board. The possibilities for content mix and match are really spectacular. However, if you don’t want others touching your data, you can mark it as “private” and use it only in your app – but why would you want to?
* Built in classes for lots of things. Build a calendar. Interact with Flickr. Make a GMap. Talk to Amazon. The code’s all done for you, you just use it. Bookshelf makes extensive use of the Amazon classes, Restaurant Reviews With Maps uses Google Maps to show where you’re going — Bay Area Hiking Trails shows you how to get there.
* RSS feeds of content. The Ning Pivot is a really cool way of looking at the content flowing by, but not only can you watch it, you can watch it flow by.

There’s about a half doezn other really nifty things here that I can’t even think of at the moment because it’s 5am and I’ve been walking like a Zombie for two weeks to get this stuff complete.

But the coolest thing is:
* All data added is placed under CC By-SA license. (If you don’t like this, ning isn’t for you.)
* All app code is completely open, and you can make it your own in 2 seconds.

Screw Ruby on Rails: who needs a 2 minute app, when you can write a 2 second app? All depends on how fast you can click.

If you run into problems with ning, feel free to drop them here: You can never fix all the bugs before release, but I think that the team working on Ning has done an absolutely incredible job with all the work they’ve put together here. I’ll pass them on as best as possible.

There’s a lot of other stuff I want to write — one that others here might find interest in is how similar Ning’s content store is to RDF, and why I think that there’s no functional difference. Of course, Marc and I got into a nice “discussion” on that one on IRC the other night, so maybe I’ll wait til I’m a bit less exhausted and can adequately express my points on the topic. 🙂

After some thinking this morning, I converted the current PHP-based crschmidt.net templating system to a Cheetah Python template. This means that some more of my tools can move to being Python powered, rather than PHP powered.

“So what?”

Currently, the interface to Redland that I have available in PHP is significantly less good than Python. It’s coded by yours truly, and it’s basically only designed for my use cases, so every time I want to use something new, I have to go and code it, or use a closer-to-native C-style interface translated into PHP. Neither of those are particularly enjoyable.

Python is a much more comfortable language for me to use. It is more intuitive for me. It feels more natural, not to mention the fact that I keep forgetting semicolons in my PHP code. It has an awesome binding for Redland, which is one of the things that I’ve been working with most over the past while.

In the past, all my scripts had been either 1. PHP or 2. Python with no site theming. Hopefully the new Cheetah template will help make me create some more tools in Python, which is the language I feel most comfortable in.

With that in mind, I’ve created a new crschmidt.net web service: an RDF Validator. A number of times, I have found that the official RDF validator will puke, but won’t give much of a reason why. This tool uses Redland, which has a tendancy to return what I consider better error messages on worse RDF. It’s designed as a one-off example of the new templating system, and should not be considered indicitive of most of the expected output of such scripts. Just a first attempt at getting myself into more code.

I’ve been doing some playing with goofy Javascript stuff lately to try to get my head wrapped around it, since I’m going to be needing to implement it in a few tools at work in the near future.

I’ve so far used it in
1. An admin interface for Athena’s email accounts,
2. An inventory listing for a work project
3. The newest one, a “suggestion” field for Wordnet searches against the RDF store I just imported this morning.

Danny alerted me to the existence of a new Wordnet dataset. I grabbed the full set, dropped it into Redland, and set up a sparql search against it. The top box there is the nifty one though: type in a string (say, apple) and watch the right side as a list of suggestions is populated.

I still need to get it actually doing a Google Suggest-like dropdown box, but haven’t had the time to hack WICK to do what I want as far as that goes.

I’m still learning, and as such, the code is sucky. I wouldn’t recommend reading it for an example: it’s a quick hack, but it works. Still many bugs to work out – for example, if you type apple, it still searches for app, appl, apple in the process. But I’ll get there. (Okay, so I just did a few bug fixes that make it much better, and switched the search mechanism to use MySQL rather than an 11 Meg PHP array. Much better now.)

Anyway, I think it’s cool. RDF people can mark it down in the “another SPARQL datastore”, Javascript people can mark it down as “Another idiot trying to use XmlHttpRequest and doing it wrong.”

Lemme know if you’ve got suggestions!

Recently, I moved most of my serving to a colocated machine, so I finally have a “Testing” machine and a “stable” machine, leaving me more free to play around locally. This has led to me installing a Rasqal nightly release and installing it, in an attempt to get the newer SPARQL query syntax working in my RDF bot, so that I can test query type detection and the like.

I had no problems installing it: very simple, just download the nightly, ./configure, make, make install. I got it working in my local “julietest”, although I’m waiting until the next release before I consider installing it on the remote server.

I got it working in PHP from the command line, no problems.

However, no matter what I do, the web version still seems to be using the old query syntax, and I don’t have any clue why. If you go to http://zeus.crschmidt.net/julie/sparql, you can test it out, and it only returns data if you use the old query format. However, if I copy the same script locally, and run the exact same query, it doesn’t work, requiring the new format.

I don’t understand it, and I don’t know if anyone else does either. The PHP in Apache2 and CLI both have almost exactly the same phpinfo(), they both have the same extension directory, and there isn’t a second copy of redland.so for the Apache version to load anyway! If anyone has run into this problem before and knows how to fix it, I’d appreciate it, because right now I’ve given up and am waiting for a release before I debug further.

(This post brought to you in part by the effort to bump all of Danny’s off of PlanetRDF while he’s on vaccation. ;))

One of the larger sources of RDF data that I’ve loaded into a database, the GovTrack RDF data is an interesting set with all kinds of information on congressmen adn so on. I recently started paying a bit more attention to the Gargonza Experiment, and found a link to their data source via the wiki.

I’ve been playing with setting up SPARQL stuff all day, and have a couple simple pages set up from my new GovTrack page. Loading the entire dataset (the RDF/XML, at least: the n3 bits I left out for the time being) took a long time, and I did some tweaking of MySQL in the process to allow me to load data faster. Some things I learned, for optimizing loading time with Redland:

1. MySQL’s key cache size is important when loading large data stores.
2. When loading statements, if you really want to optimize your load time, load with contexts. Redland will not check for duplicate statements in this case: This can be a major time saver. However, this may slow down later work, so it will probably not be worth it in the long term.
3. Loading into an already existing Redland database, even in a new model, will not increase speed: since Bnodes, Literals, and Resource tables are database wide, the selects to determine existing statements will still be just as slow as if you were loading into the existing models.

I also discovered that my QueryResults->result() method was returning actual Redland nodes, rather than the wrapped Redland.php::Node. I suppose at one point I probably realized that, but it had slipped my mind. This made it really difficult to do things like deal with optionals: calling the librdf_node_to_string in the PHP bindings causes them to segfault if the node is NULL, and there’s no decent way to check if the node is null that I found.

To compensate, I created a new way to create nodes (basically a copy constructor). This allowed me to check at node creation time whether it was a Resource/Bnode/Literal, which are the only types of Nodes there are. If it’s none of the above, I make it a PHP NULL, which I can check for, and it won’t crash PHP.

I have learned the many different ways to segfault PHP over the past week working on Redland. Of course, they all relate to PHP doing funky things with a SWIG wrapper, but it’s still one of the more interesting experiences I’ve had.

With the new PHP, all of the SPARQL interfaces I’ve got set up: one for Julie, one for XTech, one for GovTrack support Optionals. This has allowed me to create things like the GovTrack Senators page, (example for New Hampshire), listing some profile information about all the Senators from your state. (Representatives are more difficult. I’m still working on that.)

Anyway, the GovTrack data is fun to play with, although I really need to develop some more interesting interfaces over the data. I plan to do that: just haven’t gotten there yet. These tools take time to develop, but they do feel really nifty. I would go into the why’s of why I feel it’s nifty, but I almost always end up feeling like a complete and utter geek when I do it, and it makes people look at me strange, so I’ll skip it this time.

Today, I was working with the XTech Stuff, and decided I wanted to offer some fun Redland-based queries against it. Since the entire website is in PHP, I decided to stick with that theme, and write some PHP code.

I had the PHP bindings installed from a couple days ago, for… something I don’t exactly remember. I had some grand goal in mind… oh, right, I was going to provide my logo information in RDF, and parse it out using PHP.

Something I realized today is that there is no decent Redland Wrapper class like there is for Python and Perl. SWIG provides interfaces, but that basically just gets you to the level of the C API, which is something that is a bit low level for me.

To resolve this, I’ve written a PHP Wrapper class, which I hope to maintain and improve upon. It is stored in a subversion repository: you can check it out using:

svn co http://crschmidt.net/svn/redland/

Please feel free to use the trac project to help with the project.

Status: Beta Quality. Has only been tested using included test.php script. Does not do proper memory checks in any/most cases.
License: This wrapper is released under the same license as Redland itself.
Homepage: phpwrapper.