Technical Ramblings

Anselm just posted what appears to be a random thought on twitter:

Are you more generative than consumptive in your particular field? … Create more than you consume?

In open source, I often rephrase this question as “Are you a source, or a sink?”

There are many people in the community who contribute more than they consume. Organizations, individuals, etc. There are also many sinks in the community — since entropy is every increasing, this seems a forgone conclusion — and one of the key things that causes an open source project to succeed or fail is the number of sources or sinks.

I personally try very hard to be a source in all that I do, rather than a sink. One way that I do this is that I try very hard to always followup any question I ask — for example, on a mailing list, on an IRC channel, or what have you — with at least two answers of my own. This means that, for example, when I hopped into #django to ask about best practices for packaging apps, I stuck around, and helped out two more people — one who was asking a question about PIL installation, and one about setting up foreign keys to different models.

Now, in the end, my answers were simple — no one with even a basic knowledge of Django would have had problems answering them. But by sticking around and answering them, I was able to make up to some extent for the time/energy that I consumed from someone more familiar with the project, by saving them from needing to answer as well.

It is often the case that users trying to get help will claim that once they get help, they will ‘contribute back’ to the community by, for example, writing documentation. This never happens. Though there are exceptions to every rule, it is almost always the case that users who ask a question, prefacing it with “I will document this for other users”, never follow through on the latter half. The exceptions to this — or rather, the alternate cases — are cases where a user has already investedлегла significant research, and likely already started writing documentation. Unless the process is started before the problem is solved, it is almost universally true — in my experience — that the user will act as a sink, taking the information from the source and disappearing with it.

I work very hard on supporting a number of open source projects that I work on. Though my involvement lately has been more hands off — by doing things like writing documentation instead of answering questions, acting as a release manager instead of fixing bugs, and so on — I work very hard to keep the karmic balance of my work on the positive side. I believe that this pays off in the long run — I have somewhat of a reputation of being helpful, which is beneficial to me since it means I’m more likely to receive help when I need it. I also work to keep karmic balance high on the part of the organization I work for, since many of the other people in the organization are less able to keep karmic balance high.

These rules don’t apply solely to open source — I have the same karmic balance issues going on in my work inside of MetaCarta — but I maintain the same attitude there. Coming in with the idea that it is okay to be a sink can lead to a nasty precedent. In the end, I think that everyone loses. Sinks — both in open source and other karmic ventures — will eventually use up the karma they start with, and be left out to dry. It is the case for more than one person that they have extended their information seeking without contributing back beyond the point where I am willing to continue to support their information entropy.

I joke sometimes about giving out “crschmidt karma points”. Though I don’t have an actual system in this regard, I do quite clearly delineate between constant sinks, and regular sources, and grey areas in-between. I try to stay on the source side, and I encourage anyone else to do the same — even if it’s only by answering easy questions on the mailing list, or doing a bit more research on your own. Expecting other people to fix your problems, in open source or otherwise, is simply a false economy of help, since in the end, it simply doesn’t work.

Earlier today, I read a post on the OpenGeo GeoSpiel blog calling for OpenLayers to get on the “Usable Documentation Bandwagon”.

Now, I’ll be honest: I followed the links that he offered, and found something that is not, to me, much more convincing than the OpenLayers documentation as it stands today. If I look at ESRI’s JSAPI, I see a couple things wrong right off the bat — like the fact that I can’t actually link to where I want to. In any case, if you click Geometry -> Point, you see a document which, to me, is a lot less interesting than the similar page provided by OpenLayers.

Perhaps there is some subset of documentation put together by ESRI in their JSAPI that makes sense, but for an API reference, it seems to me that OpenLayers does reasonably well on the portions of our API that someone has invested time to work on.

This doesn’t mean we’re “done” by any stretch of the imagination: One of the things that I’ve wanted to do for ages is actually sit down and do a thorough review of the OpenLayers API documentation and improve a lot of it, targeting cross-linking with other documentation and examples especially. These types of tasks, however, are the types of tasks that require a lot of time, and don’t have a lot of immediate benefit. Since all work on OpenLayers for the past several months has been my personal free time after work, there’s only so much I’m personally able to do. However, no one has *ever* made a request to the OpenLayers team to be able to do this work — including OpenGeo — that I’ve seen. It seems to me that whit is experienced and knowledgable about OpenLayers, since I’ve seen him using it for development, but I’ve never seen him ask to participate in improving the OpenLayers API documentation on the mailing lists, nor have I seen a request of this nature from any other organization.

To me, this means that it’s likely that our API documentation does, to some extent, meet the needs of the organizations using OpenLayers. It’s not perfect — nothing is — but no one thinks it’s a major stumbling block that’s worth fixing, at least not enough to spend money on it.

This is exactly the type of reason that OpenLayers is now accepting Sponsorship from organizations looking to support the project. This type of improvement is exactly the kind that project sponsorship can help support.

OpenLayers is aware of how important documentation is to the success of the project. We have invested dozens of hours between a dozen different contributors to create and maintain a set of relatively complete API documentation. According to Ohloh, of the 55,000 lines in OpenLayers Javascript, over 30% are comments: over 25,000 lines of what is mostly API documentation. No one is ignoring the problem — and if the current state is insufficient (as everything in the project is, *especially* to new users and beginners), then we’re very open to help from any and all interested parties.

API documentation isn’t the only thing a project needs, of course. Documentation comes in all forms — and OpenLayers is seriously lacking in a lot of documentation targeted towards beginners of all kinds. We’ve been working to change that, with a new documentation site available, and other efforts targeted documentation of OpenLayers in English and other languages. I would say these efforts are much less complete than the API documentation, and starting on them is far more important, in my opinion, than improving our API documentation is at this time.

OpenLayers is a large library. It’s used by many organizations. We’re open to contributions of all types — never have I seen OpenLayers turn away someone who wanted to help with documentation intentionally. We have regularly worked with contributors in helping them to improve the documentation, and to claim that we are ignoring the need for documentation seems to me to be representative of a lack of knowledge of the tools that the project uses for documentation, not specifically a lack in the goals of the project, which puts documentation of functionality — via API docs and minimal examples demonstrating functionality — as a requirement of almost all new code in the library.

Chad writes:

I know their “motto” is “Don’t Be Evil” .. but I think it should be “Don’t Be Smart” instead.. this is some dumb thinking from Google. Trust me.. I know better than Google on how I want to download and install my software.

This is just the latest in a whole lot of similar statements I’ve seen from many people across the web in a variety of situations talking about how “I know how to manage my machine”, with the underlying meaning being something like “You should act as if people who are working with your software know how to work their computers.”

When I put it that way, does it really sound right? Is there anyone who thinks that the *majority* of users of Google Earth actually know how to run their machines? Is there anyone who thinks that it makes sense for Google to build and QA two different install mechanisms — one for technical users who know what they’re doing, and one for those who don’t?

Very few companies the size of Google do anything on a whim. I expect that some thought went into the development of the Google Earth downloader. The fact that the thinking is not centered around technically competent users is just evidence that Google doesn’t need to target the early adopters; it’s not a sign that what they are doing is ‘Bad’ or ‘Stupid’.

Technical users are few and far between in the mass market. Google Earth is targeted towards the mass market. Just like all software that is targeted towards a mass market, there is nothing ’stupid’ about removing tools that the majority of users don’t need or care about: By doing so, you limit the number of people who are going to end up confused by your tool, and that’s not a bad thing when you care about the majority of people instead of a technical elite.

Jo Cook writes about ESRI’s crackdown on licensing:

In what can only be described as a noble act of self-sacrifice, ESRI have told us that as an educational charity we are no longer allowed to have an educational discount for using their software and, not only that, our license codes will cease to work at the end of this month.

This is why Open Source software is so important. So you think you have a stable relationship with your vendor? Maybe you think that you’ve come to a great licensing agreement that you’re happy with? Remember that so long as you’re working in an environment where someone else controls the tools you use, you’re not able to make your own rules.

Why does Open Source software matter? If you’re not using it, you’re handing over control of your use of your tools — possibly important ones — to people who aren’t under your control. In the end, that lack of control may end up hurting you far more than you’d expect.

Three quotes from my presentation earlier today, on Getting Help With Open Source:

• On talking to Open Source developers and asking for help: “Be Polite, Be Concise, Be Gracious.”
• On diving into the code: “Code is pickled, executable documentation.”
• On choosing your requests carefully: “A developer’s time is a limited resource. Don’t waste it!”

Talk went well. Transcript recorded by Arnulf on IRC (original). Many thanks to Arnulf, and all the attendees!

As of 5PM today, I will be unavailable on the web until July 9th. I’m going on a honeymoon, and I’m leaving my computer offline for the entire trip. (Yes, I am a workaholic. Heck, I committed to OpenLayers less than 2 hours before my wedding.) So, although I will be at home slightly more than that, you should not expect to hear from me until July 9th. If you have something that requires my urgent attention, please get it to me today

My actual honeymoon will include a stay at the Mount Washington Hotel, in the White Mountains of New Hampshire. Not to pat myself on the back too much, but I think I’ve earned it

Don’t break anything while I’m gone!

Maps tell stories. They tell all sorts of stories, but one of the stories that they tell the best are processions: series of photos taken over a wide area over a period of time, but with the same principle actors.

The most important procession to me right now is the one that I’ve been working up to for four years, as of tomorrow. At the church shown (using the Boston Freemap TileCache, FeatureServer for feature translation from the Flickr API, and OpenLayers, of course, as the map interface) on the map I put together, I’ll be getting married tomorrow to the lovely Jessica Allan.

From Chicago to Champaign-Urbana to Manchester, NH to Cambridge, MA, with more late nights and late flights than I care to remember, more love and devotion than I can describe, and more acceptance of my tendencies to show utter obsession with anything I’m doing than I could possibly have expected, my relationship with Jess has flourished, and I couldn’t be happier to be tying the knot tomorrow.

Okay, so the relation this has to mapping or technical ramblings is tentative at best, but I still think it’s cool.

Can You Copyright the Globe?

Originally uploaded by crschmidt.

When at the Museum of Science, I noticed that the giant globe in the atrium near the entrance of the museum is copyrighted by Rand McNally. Although I understand that the interpretation of a lot of raw data to make a 3D model of the Earth is definitely a creative work, it was still strange to read ‘Globe copyright Rand McNally…’

I think from now on, I’m not going to try to answer questions on mailing lists which don’t follow the guidelines of “How to Ask Questions the Smart Way“. I spend way too much time trying to extract information that’s needed to solve any
problems. Obviously, in some cases, it’s hard to know which information is important, but with two open source projects with growing user populations, it’s very hard for me to take the time to coax every person through getting the information that is important to solving problems.

I think it is a sign of the growth in popularity of the OpenLayers and TileCache projects that I’ve come to this stage — it’s only when projects reach a certain level of maturity that people start asking enough questions that I don’t feel like I can just answer everything. Certainly, for the first 6 months of the OpenLayers project, I never felt that answering questions on the mailing list was something that was a burden to do, but over the past couple months, it’s been an increasing level of traffic from users who haven’t taken the time to investigate the cause of their problems fully. I think the time has come to stop devoting as much energy to solving *every* problem, and instead solve the problems that can’t be solved by other people.

Of course, an additional step is to work to create a forum in which users can offer cash to solve their problems — you put up a question, with a cash bounty on it, and you either increase the amount of money you’re willing to spend to get your question answered, or you answer it yourself.

This seems like the kind of thing that there should already be software to do. I’ve never heard of any, though. I wonder why that is. Probably something obvious I’m missing.

Lately, I’ve seen a couple of attempts of large standards organizations to migrate in some way or another to the more community-driven specification creation model demonstrated by the FOAF and GeoRSS communities. I think that these communities are a great model of the right way to create application-level specifications, but I don’t think that large standards orgs are going to find it possible to migrate to this framework without a very significant change in the way they look at what they do.

Community based standard creation offers a great number of benefits. Without a large corporation behind the specification, it seems to have been possible to create an environment where the ’survival of the fittest’ in terms of inclusion into the spec is much more likely than in a case where large competing interests don’t have any impetuous to create a small specification. This environment is likely created by the specification creators being implementors. Implementors are more likely to head to the simplest thing that could possibly work. Implementors care about how difficult a spec is to understand, because if they don’t understand it, they can’t implement it.

Large corporations don’t have the same feeling. A representative of Sun, IBM, HP is going to try to solve a lot more cases than he or she will personally implement: The reason for this is that in those large organizations, a specification will be used by dozens or hundreds of different implementors, to solve different goals. This is great when the result is a specification which is well suited to all of those needs — however, typically what happens is that a specification grows in complexity with each additional use case. You end up with a large specification which can be used to solve lots of different needs, but doesn’t solve any of them perfectly, and is difficult to implement — or at least to understand how to implement — due to the added complexity of addressing dozens, or more, of use cases rather than a single simple case. The single representative of a large organization is going to be speaking as a representative of far more people than a single use case.

There are certain specifications which are better built this way. The fact that so many people have spent time thinking about XML has resulted in a format which is extremely flexible, and can play the roles needed in thousands of applications. The specification has been around long enough that tools which understand XML are very mature, and this maturity makes XML a useful tool for information exchange among a wide number of tools. I think an implementor of a new toolkit to work with XML would, however, argue that XML is not easy at all: there are many edge cases which need to be handled, and the fact that these tools are already created hides these complexities.

Application-level specifications are not well-suited for large-organization standardization. Application-level specifications should start by addressing the simplest thing that could possibly work. Large organizations don’t have an interest in creating the simplest thing that could possibly work. Starting small and expanding into something larger is a pathway that large standards bodies have thus far successfully demonstrated the ability to do.

It is possible that someplace like the W3C could succeed at this, but it would require changing the way these standards are generally created.

* Open communication. Even in the Geospatial Incubator group, the primary communications channels are currently closed: private mailing lists, private telecons, and private wikis. Implementors have not been invited to participate. This is a large mistake: standards for application use need to involve applications. There are a number of applications developers who would love to take part in standards development who have not yet been afforded the opportunity to.
* Changing development strategy. Rather than spending months going through 17 calls for comments, last calls, etc. to get to a recommendation, get something out as quickly as possible and iterate on it.
* Require that a specification have implementations before it is complete. I’m still not aware of a good, W3C released or endorsed XML parser. No tools to convert some existing data format to any kind of XML. No way to test how the specification is supposed to work and make sure you’ve got it right. Conformance testing is a good part of this, but is not all of it. When working on RSS 1.1, we had an implementation for Wordpress and support for XML::Parser patched in Perl before we even released the spec. We included testing, Relax-NG testing, and a validator service. All of these and more were important to the rapid (though limited) uptake the technology received: no one had any serious questions about RSS 1.1 that couldn’t be answered by reading through the example implementations.

These changes, however, are antithetical to the way that large standards organizations work. In general, grassroots implementors aren’t part of large organizations which can afford to be members of these standards organizations, or when they are, they aren’t often implementing these standards as part of their work for that company. Since large standards organizations depend on large corporations memberships for revenue, choosing to allow small-potatoes implementors participate means that they give up possible revenue stream. When this happens, the larger customers start wondering what the benefit there is to being a member: if anyone can just come in on their own time and influence the standard, what is the benefit to pay large sums of money to participate?

Part of the reason for this shift is that organizational costs for creating standards have been headed downward. For $500/yr, or less, I can get my own webserver with lots of bandwidth. I can run mailing lists, Web pages, Wikis, IRC channels. With VoIP setups, I can achieve relatively cheap realtime communication, and the grassroots standards developers typically have more of a preference for somewhat real-time online communication anyway. When I can obtain all the organizational materials I need to create a standard for less than $500/yr, what purpose does a $50,000 a year membership to a standards org get me?

Typically, the answer is “Access to the right people”. Getting the members of the RDF developer community in the same place for discussions is not easy, and doing it with telecons does take work. However, as the grassroots development practices mature, they are demonstrating the ability to create specifications in ways that do not work the same way as XML was developed. Good specifications limit the need for specification ‘marketing’ — if a spec does what you need, you’ll find it, and you’ll use it, if it’s good.

So, with organizational costs heading downwards far enough that people are willing to contribute the resources out of pocket, with developers tending to group together without the need of an organizational support, and with the resulting specification having the support of application implementors, what do standards organizations offer to application specification development?

I think you can guess what I think the answer to that is.