Open Source Documentation Framework

Waldo’s announcement on a documentation framework for open source projects caught my attention for being both needed and challenging. While I haven’t participated in meetings concerning this, I here try to take a stab at it.

My immediate thought is that it has been tried before. The Linux Documentation Project(TLDP) collects documentation in a presentation neutral, organized way and has been doing so for long. In what way is TLDP insufficient? Perhaps a more concrete question is: why isn’t KDE and other open source organizations publishing via TLDP? Why isn’t TLDP considered sufficiently authoritative? Perhaps it’s a marketing problem? Those are questions that perhaps leads to what actually needs to be done.

One thing that always pulls me up on my marks is projects duplicating, perhaps down right ignoring each other. Perhaps the easiest way forward for a documentation framework is to rally TLDP-people to help extend and pitch TLDP in a direction such that it also can handle the problems OSDL identifies. Not that the OSDL-initiative necessarily would overlap TLDP, of course.

The problem domain Waldo summarizes can perhaps be extended:

• Should the framework be able to export content for offline viewing? On Free Desktop’s xdg mailing list it has at least once been brought up to standardize a framework for manuals. Perhaps the OSDL-initiative should also cover the requirements desktop environments have.
• In what way should the content be accessible? Should it be presentational or semantics-oriented? Is accessibility and device independence a priority?

One way is to “reduce” the documentation framework to a link database. Instead of serving content, it serves links to where the content is. I don’t think that holds because I believe many advantages stems from being able to serve content in a unified way: it is presented to and treated by the user in a consistent manner. Therefore, I bet my money on the more ambitious, complex approach of serving content.

“Serving content” does not necessarily mean that all content is accessed from a central server in my book, it can be that content is served through a standardized interface. Perhaps an immediate thought is that the content would be delivered from a decentralized cluster of servers(“a standardized documentation infrastructure that the various stakeholders can tap into […] as a provider of content”, as Waldo summed it up), but not necessarily. It could also be that each consumer acts as a fat client by having an XML/XSL-T application installed that knows how to fetch and deliver from the framework. It is perhaps attractive for reasons of scalability.

This documentation framework is for me a breath taking task, because its goal is very broad. It is not intended to exclusively handle book content, it is formulated to also handle API documentation, for instance. What format — what model — can be used encompass such a diversity?

Someone can surely mention a technology for that, but what makes it even trickier is that everyone would not be able to serve or consume content in that medium, nor in any other. The framework would probably end up with serving and receiving in a wealth of different formats.

Let’s say the “base” format for this framework is the latest incarnation of Docbook. The content would be accessible as Docbook(for arbitrary consumption, such as customized PDF output) but also as XHTML, for example. Content providers incapable of serving Docbook would have its content transformed into Docbook. Content would be validated.

Even with such a quickly sketched idea, major questions remains:

• Given a term that the user wants to read about, how does one find the relevant text from the framework, and from where to fetch it? A central OASIS Catalog that clients fetch?
• How should such a framework scale? With caching? How would the caches be invalidated?
• Should the license for content be communicated semantically?

“A key concern raised with respect to any portal is the maintenance burden,” says Waldo’s summary. I wonder how a documentation framework can be done that meets major goals while not turning into a large, labour intensive project.