One customer told about how sub-classes of QObject are used for representing data, and are converted to such from XML. So, why not add a little helper class to Qt for this?

The class, which currently only is a research idea and missed the feature freeze for Qt 4.5, is called QXmlToQObjectCreator, and hopefully the documentation explains it all:

QXmlToQObjectCreator API Documentation

In other words, it’s a very simple class that builds a QObject tree corresponding to the output of QXmlQuery. The current sketched code is pasted here, for those interested.

In what way can this class be made more useful?

Using it is should straightforward. Either on the command line:

xmlpatterns yourStylesheet.xsl yourInputDocument -param myParam=myValue

Or using the C++ API[1]:

QXmlQuery myQuery(QXmlQuery::XSLT20);
myQuery.bindVariable("myParam", QVariant("myValue");
myQuery.setQuery("http://example.com/myStylesheet.xsl");
QFile out("outFile.xml");
out.open(QIODevice::WriteOnly);

myQuery.evaluateTo(&out);

See the documentation for the QXmlQuery class on the overloads available for setQuery() and evaluateTo(), for instance.

However, due to the beast XSL-T 2.0 is — I agree that it’s larger than XQuery — we’ve decided to do this according to the “release early release often” approach. The first, in Qt 4.5, will carry a subset, and subsequently be complemented in Qt 4.6. The current status is documented in the main page for the QtXmlPatterns module, which can be viewed in the documentation snapshot.

Therefore, while the current implementation probably falls short on more complex applications(such as Docbook XSL), it can run simpler things, users can plan ahead, and we trolls can receive feedback on what features/APIs that are missing, and what needs focus. So feel free to do that: send a mail to qt-bugs@trolltech.com, or say hello on IRC(FransE, on Free Node).

The code is accessible through the Qt snapshots.

What is XSL-T anyway?

XSL-T is a programming language for transforming XML into XML, HTML or text. Some implementations,  such as QtXmlPatterns or Saxon, provides mechanisms to map XML to other data sources and hence widens the scope of the language by letting the XML act as an abstract interface. Wikipedia has a good article on XSL-T. Version 2.0 of XSL-T extends the language heavily by putting a rigid type system and data model in the backbone, and adds many features that was a pain to miss when programming in XSL-T 1.0. XSL-T 2.0 use XPath 2.0, and shares the same large function library as XQuery.

1.

Over time, Java bindings through QtJambi and ECMAScript bindings through QtScript, will likely arrive.

At the top of the list of things people tend to ask on the forums around is “How do I get XML represented as a sequence of bytes in Java/C++ into my query?”, whose result is clear — a tree fragment for the query to operate on — but whose method for reaching is not that given if you ask me.

There is no “bytestream” type in XQuery. Should the user build the tree herself and then pass the tree to the query? Should the implementation in some voodoish way be instructed how to treat a string or custom type? Shouldn’t the query engine do it such that its scope of analysis is increased and its done the way it prefers it?

What I sense have been the problem with some solutions is that they mix the data, the bytestream, with interpretation.

In Qt this manifestate itself with that the content of a QIODevice should appear in a QXmlQuery. The way it’s now provided, is that when a QIODevice is bound to a variable using QXmlQuery::bindVariable(), the query sees a URI(an instance of xs:anyURI) which behind the scenes maps to the QIODevice the user bound. Hence, if the purpose is to build an XML document, one passes the URI to the builtin fn:doc() function.

I hope this is clean. Since it’s handled like any other URI, custom extensions stays at a minimum, error reporting is consistent, and the interpretation hasn’t been coupled with the data. For instance, later on I hope to merge in support for XInclude and XQuery Update, and in those cases the URI is again simply passed to for instance fn:put().

One can weight quite well on URIs and the abstraction the XPath Data Model provides, it seems.

Any decent implementation(such as XQilla or Saxon) provide that nowadays in someway or another, but I’d say Patternist’s approach is novel, with its own share of advantages. So let me introduce what Qt’s snapshot carries.

<ul>
    {
        for $file in $exampleDirectory//file[@suffix = "cpp"]
        order by xs:integer($file/@size)
        return <li>
                    {string($file/@fileName)}, size: {string($file/@size)}
                  </li>
    }
</ul>

and the query itself was set up with:

QXmlQuery query;FileTree fileTree(query.namePool());
query.setQuery(&file, QUrl::fromLocalFile(file.fileName()));
query.bindVariable("exampleDirectory", fileTree.nodeFor(QLibraryInfo::location(QLibraryInfo::ExamplesPath)));
if(!query.isValid())
     return InvalidQuery;
QFile out;
out.open(stdout, QIODevice::WriteOnly);
query.serialize(&out);

These two snippets are taken from the example found in examples/xmlpatterns/filetree/, which with about 250 lines of code, has virtualized the file system into an XML document.

In other words, with the tree backend FileTree that the example has, it’s possible to query the file system, without converting it to a textual XML document or anything like that.

And that’s what the query does: it finds all the .cpp files found on any level in Qt’s example directory, and generate a HTML list, ordered by their file size. Maybe generating a view for image files in a folder would have been a tad more useful.

The usual approach to this is an abstract interface/class for dealing with nodes, which brings disadvantages such as heap allocations and that one need to allocate such structures and hence the possibility to affect the implementation of what one is going to query.

But along time ago Patternist was rewritten to use Qt’s items & models pattern, which means any existing structure can be queried, without touching it. That’s what the FileTree class does, it subclasses QSimpleXmlNodeModel and handles out QXmlNodeModelIndex instances, which are light, stack allocate values.

This combined with that the engine tries to evaluate in a streamed and lazy manner to the degree that it thinks it can, means fairly efficient solutions should be doable.

So what does this mean? It means that if you would like to, you can relatively cheaply be able to use the XQuery language on top of your custom data structure, as long as it is somewhat hierarchical.

For instance, a backend could bridge the QObject tree, such that the XQuery language could be used to find Human Interface Guideline-violations within widgets; molecular patterns in a chemistry application can concisely be identified with a two or three liner XPath expression, and the documentation carries on with a couple of other examples. No need to convert QWidgets to nodes, or force a compact representation to sub-class an abstract interface.

A to me intriguing case would be a web robot that models the links between different pages as a graph, and finds invalid documents & broken links using the doc-available() function, or reported URIs that a website shouldn’t be linking to(such as a public site referencing intranet pages).

Our API freeze is approaching. If something is needed but missing, let me know.

qt/src/xml/query/expr/qcastingplatform.cpp: In member function 'bool CastingPlatform::prepareCasting():
qt/src/xml/query/expr/qcastas.cpp:117: instantiated from here
qt/src/xml/query/expr/qcastingplatform.cpp:85: error: no matching function for call to 'locateCaster(int)'
qt/src/xml/query/expr/qcastingplatform.cpp:93: note: candidates are: locateCaster(const bool&)

Typically compiler messages have been subject to crude printf approaches and dignity has been left out: localization, translation, consistency in quoting style (for instance), adapting language to users (e.g, to not phrase things preferred by compiler engineers), good English, and just generally looking sensible.

To solve that it requires quite some work, and that’s probably the explanation to why it often is left out. To have line numbers, error codes, names of functions, and whatever available and flowing through the system requires quite some plumbing and room in the design.

Another thing is that nowadays we really should expect that compiler messages within IDEs or other graphical applications should be sanely typeset. If not, we’ve lost ourselves in all this UNIX stuff. Keywords and important phrases should be italic, emphasized, colorized depending on the GUI style.

For shuffling compiler messages around it is customary to pass a set of properties: a URI, line number, column number, a descriptive string, and possibly an error code. Apart from that it falls short reaching the goals outlined in this text, it encounters a problem which I think is illustrated in the above example with GCC. What does one do if the message involves several locations?

Even if a message involves several locations, it is still one message and should be treated so, and presented as so. The approach of using a struct with properties falls short here, and chops the message into as many parts as it has locations.

For Patternist I wanted to make an attempt at improving messages. So far it is an improvement at least. For instance, for this message that the command line tool patternist outputs:

the installed QAbstractMessageHandler was passed a QSourceLocation and a message which read:

<p>Operator <span class='XQuery-keyword'>+</span> is not available between atomic values of type <span class='XQuery-type'>xs:integer</span> and <span class='XQuery-type'>xs:string</span>.</p>

It was subsequently converted to local encoding and formatted with ECMA-48 color codes. (The format is not spec’d yet, it will probably be XHTML with specified class ids.)

While using markup for the message is a big improvement, it opens the door for formatting and all, this API still has the problem of dealing with multiple locations.

What is the solution to that?

Striking the balance between programmatic interpretation(such that for instance source document navigation is doable) and that the message reads naturally as one coherent unit is to… maybe duplicate the information, but each time tailored for a particular consumer?

<p xmlns:l="http://example.com/">In my <l:location href="myDocument.xml" line="57" column="3">myQuery.xq at line 57, column 3</l:location>, function <span class="XQuery-keyword">fn:doc()</span> failed with code <span class="XQuery-keyword">XPTY0004</span>: the file <l:location href="myDocument.xml" line="93" column="9">myDocument.xml failed to parse at line 93, column 9</l:location>: unexpected token <span class="XQuery-keyword">&</span>.</p>

This is complicated by that language strings cannot be concatenated together since that prevents translation. But I think the above paragraph is possible to implement. As above, the message reads coherently, but still allows programmatic extraction. A language string and formatted data sits in opposite corners of extremity, and maybe markup is the balance between them.

Would this give good compiler messages and allow slick IDE integration? If not, what would?

Being part of the XML library, the idea is that Qt 4.4 will ship with a C++ API for running and evaluating such queries. On the side too, is a command line tool called patternist, for quickly testing queries, scripting and old-school web solutions. But who cares, blogs with screenshots is the thing:

Stronger XML support in Qt has been consistently asked for by users over a long time, with XPath being one of the main requests. Hopefully Patternist, with the help of KDE folks, users, and customers expressing what’s missing, will please those needs. Considering the similarities of XQuery and XSL-T, Patternist also serves as a foundation for implementing XSL-T, if so decided.

For KDE folks all this might ring a bell. Patternist was indeed first developed for a long time in the KDE repository, as part of KDOM. We just thought it would make a lot more use as part of Qt.

And I think exactly that makes this exciting. W3C’s XQuery working group has registered an astonishing number of exciting implementations. But for users, reliability is what matter in the end. Whether bugs will be fixed, whether people can answer questions, whether the piece is maintained and documented. Persistency. Trolltech swiftly carries this on its shoulders(assuming I brush my teeth and all that).

Combined with that Qt is open source and the Patternist SDK used for development is as well, this is like eating some nasty chocolate while at the same time singing a little duet with Miss Piggy. I can’t sing, nor can Piggy (although she tries), but you get my point.

Humble modesty aside, it is worth to mention that this still needs work. About 94% of the test suite is passed, the API needs more work, and there is performance issues.

Nailing test cases and trimming code paths are problems that have known solutions (though typically horrible to carry out). Harder is to know what people need and how they need it. It’s hard to guess what kind of APIs or extensions Amarok or KOffice or a GNOME or web application need.

If you got input, feel free to add a comment to the blog, send a report to Trolltech, grab me(FransE) on the Open Projects IRC network, or ask a question or two on the qt-interest mailing list.

The documentation starts over here.

This blog entry is tinkering — although without accompanying code — on how to represent XML.

Introduction

For the purpose of writing a pleasing backend, I started reading up on the research. There is a lot of work in this area and I have still no trouble finding contributing papers. The focus of the papers I’ve read so far(and if taking a broader perspective, I suspect) is representing XML with respect to storage size and efficient evaluation of XPath paths, with a strong eye on axes.

One thing worth to notice, even though XQuery’s support for static typing of sequences’ cardinality is ardently supported and readily implemented, I fail to see the literature that utilizes it. Well, perhaps that will arrive later on.

Although it may sound obvious, “storing XML” is not a homogen term, which surely must be taken into account when starting to pick and choose between the different proposals. There is a clear dominance towards persistent, relational storage(not that it is surprise to anyone), which tailors the discussions — table normalization, what to index, data base optimizations to exploit, etc — in a direction that typically isn’t that useful for the more old-fashioned in-memory approaches.

However, one do find significant amounts in other directions. For example, Efficient Path Query Processing on Encoded XML discusses how to query streamed XML on resource-constrained devices, and Efficient XPath Axis Evaluation for DOM Data Structures discusses pre/post numbering close to DOM.

A common pattern through all is diversity, which perhaps can be derived from the genericy of XML and its organic structure. People use XQuery and XML in such different ways that ideas poorly travels between different scenarios.

Sometimes I think this goes so far as losing touch with reality; for example, one paper, in the competition of implementing XPath axes the best way, does so at the cost of as much as four full-document indexes and subsequent size ratio. From my impression document size is one of the most common problems people have, judging from for example vendor forums. Even if a certain proposal for axes is very fast, who can in practice take the storage hit it requires?

Therefore, the techniques one decides for is extremely dependent on what the scenarios are. If one is so lucky to constrain what kind of documents one are reading — count out Patternist — the best one can do is perhaps to adapt at runtime.

One of the key elements that the “XML era” of data management brings, is the increased freedom and diversity, which is as concrete as XML’s syntax compared to a tabular format. “While it’s generally very easy to map relational data into XML, trying to map XML, into relational data can be incredibly difficult.“

However, one trend is not to support XML’s unpredictability, but the opposite way, through excluding node types, XPath axes and features such as pessimistic typing of cardinalities. What makes me excited about XML, is this increased flexibility it gives users on behalf of computing power. And that’s the right direction, although it does mean implementors have to make an effort.

Requirements

For Patternist, the requirements are as follows:

• Implement the XPath Data Model. This means for example that for a given node, it must be able to determine the namespaces in scope, that processing instructions and comments are represented and that text nodes are text nodes, not only the string value of an element. However, CDATA and regular text nodes aren’t distinguished(which is a good thing) and system entities and system entity references are not part of the data model.
• Be XML 1.0/Namespaces 1.0 based, as opposed to 1.1, for a plethora of reasons.
• Support serialization/roundtripping to the degree XSLT 2.0 and XQuery 1.0 Serialization says.
• Memory consumption must for users be acceptable, which means that it must be a radical improvement compared to QDom. However, it’s not the only requirement, and it hence can’t go blind on this at the cost of the other requirements.
• The faster XQuery queries are, the better. For example, if the time of sorting in document order for two arbitrary nodes is linear to the distance between the two in document order(as in the case with QDom and nost other DOM implementations), one can pretty much declare game over.
• It’s about representing and writing arbitrary documents, in arbitrary locations, in arbitrary amounts. For that reason bulk loading into a persistent storage is not an option.
• Updates must be possible to the degree that the XQuery Update Facility requires. Why not even more fine grained write support? Because I believe XQuery Update makes a good job at taking into account users needs while providing implementors with good optionsfor doing things efficiently.

What tradeoffs to do on this game board will be readily discussed. Except for conformance.

Solutions

Node Numbering Schemes

“Node Numbering is one of the more interesting, and important design choices in an XML database,” says Anatomy of an XML Database: Oracle Berkeley DB XML(which gives a good overview of databases and XML) and I agree. It seems to be the key, and largely affects how the rest is wired.

The reason stems from XPath’s requirement of that results of path expressions should be free from duplicates and be sorted in document order. In order to optimize those operation(de-duplication, sorting), implementors attempt to code the structural relationship(e.g, if a node is a forward-sibling of another, and so forth) in the node idenfifier in order to avoid additional lookups. The otherwise needed indirection is relatively cheap for in-memory models(since it’s typically a pointer being dereferenced, or somekind of array lookup), while for implementations on top of relation databases it requires an additional join for performing a structural join.

One widely popular node numbering scheme is Dietz’s pre-post numbering, which is excellently explained on the MonetDB project’s site. Its major advantage is that the node relationship can be determined in constant time, but does so at the cost of requiring a re-numbering of the whole tree when nodes are inserted.

There are many variants of pre-post numbering, such as replacing the post number with an appromixated child count to reduce the number of re-numberings on updates. The problem with the different variants is that they all have somekind of flaw, still leaving one with a bad taste.

However, the Dynamic Level Numbering(DLN) scheme, nicely explained on eXist’s site, which essentially extend Dewey’s Decimal Classification with support for incremental updates and encodings for efficient storage, seems to be freed from the other schemes’ issues, at the cost of being of variable length. In practice this means that each identifier needs a heap allocation to support the identifier, which is costly considering how node identifers are used.

However, in C++ I believe this can be tackled quite well with a classic union-trick: it is first at abnormal tree depths or very large node counts that identifiers grows to considerable sizes, meaning one can store it in 63 bits and let the last bit signal whether the first word is used as a pointer to a heap allocated structure(otherwise the identifier is simply stored in the 63 bits).

The XML Web: a First Study, says that of the roughly 200 000 XML documents they scanned, “99% of the documents have fewer than 8 levels. The average depth is 4,” which sounds promising for the DLN scheme. The hit taken by large documents is more realistic, but on the other hand first occurs when having run out of “local” bits.

Compression & Text Nodes

KDE developer Ariya Hidayat reach impressive size ratios in his KoXmlReader experiments with the use of heavy compression(among other things). The requirements for KoXmlReader and an XQuery implementation are vastly different. Although for functionality that is supposed to actually access(query) the nodes, one could quickly conclude that the tradeoff of compression(slow access) is strongly present, but I don’t think that necessarily holds.

I believe it was Michael Kay who somewhere wrote that lazy parsing of source documents could be sensible for XSL-T transforms because often not all nodes are needed(and hence doesn’t need to be represented). By the same reasoning compressing text nodes could be sensible as well. Afterall, selecting a whole document is not a very useful query.

It’s also worth considering what is being compressed. If I’ve read the KoXmlReader experiments correctly, it was compressing western codepoints in UTF-16 encoding, which wouldn’t surprise me if is thankful input for many compression algorithms. Another possibility is to store UTF-8 instead of UTF-16 if that encoding as well as western content is common — A Web Study reporting on encoding distribution would be helpful.

Another possibility is to let the parser not decode at all(performing validity checks only) and store the text in the document’s encoding, which possibly could speed up serialization, which is notably expensive. However, doing so requires rather heavy surgery, and rules out features such as system entities(which on the other hand are absent in the XPath Data Model anyway), to mention one of the few things one can consider.

Another possibility is to compress text nodes consisting of only whitespace and gain by that knowledge when doing string matches as well. However, implementing this with Qt/C++ is tricky, considering QString’s non-polymorphic design. Unfortunately The XML Web: a First Study doesn’t discuss the content of text nodes, which makes it difficult to tell how relevant this discussion is, although folklore and sporadic tests suggests whitespace is of significant size.

One approach is to compensate the difficulty of accessing the compressed nodes with a full-text index, which as upside is designed for actually querying text nodes. Or rather, perhaps the compressed nodes is the payment for a full text feature. One could also attempt to base primitive searches such as fn:contains() on the full-text index. Not that I have started implementing Full-Text.

Name Pools

Name pools, typically implemented as that each name is stored as a string only once and subsequently referred to with an id, are popular both in relation databases and in-memory representations. The reason seems to simply be that structural content is overwhelming in an XML document(as the Web Study discusses in section 3.2 Node Distribution), as well as that name comparisons are optimized. Name pooling seems to one of the most efficient space optimizations one can apply.

Worth to comment on is populating the name pool. Saxon’s NamePool loops over the name index each time an id is requested, in order to determine the index of the previously identical inserted string, if any. This is likely as slow as an insert gets, which others decide to optimize by keeping a hash with the string index as value.

The index costs space(although not much since it’s linear to the amount of different names, as opposed to count of all names), which possibly could be skipped if one could approximate what nodes that appear often. For example, the html element in an XHTML element appears only once and the same applies more or less for the following 3-4 elements, but they are nevertheless inserted first in the name index and is therefore iterated over for every following name for no useful reason.

If one could approximate where the common elements starts — the Web Study finds many intriguing, strong patterns — one could offset the first names further up the index, or to store in two different indexes(“rare” and “common”).

Being able to approximate what names that will appear often is also of interest for doing pre-allocations. For example, as can be seen in the xmlstat blog entry, there is about 13 names that occurs a vast amount of times. An index that has name as key and node identifiers as values will likely use growth strategies that are non-optimistic for those names.

File Size as Parameter

For a wide variety of choices, the XML representation consists of integers whose required range they are required to express, depends on the document size. For example, the maximum amount of nodes, name distribution or tree depth. For example, using 64 bits for a document order numbers is excessive if the actual amount of nodes easily fits in 16 bits.

One approach here could be to use the file size(which is relatively cheap to retrieve for a local file which is to be parsed), if such a one is available, as hint for the XML representation which subsequently could use more appropriate storage types. More practically, this could for example be implemented in C++ with a template implementation which is instantiated for a set of common sizes, “small”, “medium”, and “large”.

This space optimization applies for probably small documents, which can be significant for resource constrained devices or when many small documents are opened, such as collection of small documents used as a database(a usage scenario which Apache’s XIndice seems to be optimized for).

The first step would be to determine what parameters in the implementation that are dependent on the document size. For example, to be able to answer the question “What is the maximum file size that can be handled if I store node count in 16 bits?” and other peculiar pussles with the XML syntax.

Even if one decides to trust ones calculations, there is the possiblity of that a file is encoded in a codec with a smaller per-character encoding than a byte(possibly as a malicious attack by installing such a codec), so it is probably a risky business to skip bounds checking for fixed sizes.

File size can also be used as a hint for pre-allocations, especially if ratios for the different stores in the representation are known.

Indexes

Due to the requirements for Patternist’s backend, indexes aren’t that relevant, except for a particular case: a map from name to attribute and element nodes.

One reason is it allows threaded execution of path expressions such as a/b/c/d because a/b and c/d can be executed concurrently, to afterwards be joined. Many XPath operators can be threaded in typical implementations(not that I can name any who does) because the operands do not depend on each, but that is the case with path expressions without any such index.

Another reason is the common abuse but also valid use of the descendant axis which should be signficantly faster for many scenarios when using a name index.

It is hard to tell whether the storage for such an index is worth it, for several reasons. For starters, the size of such an index could be of significant size; for the document in the xmlstat blog-entry it would probably be around 2 MB assuming each node identifier consumes 64 bits(although it should be taken into account that the source document is relatively big: 8.5 MB).

Query as Parameter

One way to improve both space and time consumption is to use query analysis to adapt the XML representation for documents to be loaded. Examples follows.

Node Construction

Path expressions tells in a very clear way what nodes that will be used. Hence one can selectively build representations for processing instructions, comments, attributes, and text nodes. For instance, the Web Study concludes that “for documents larger than 4096 bytes, there are proportionally more attribute nodes than element nodes(51.13% vs. 37.83%),” which suggests that significant space gains can be made.

Doing such a query analysis is straight forward, but needs consideration. For instance:

• If positional predicates are present, skipped nodes must be compensated by “artificial” place holders in order to preserve node numberings.
• Access to text nodes or the string value property is commonly implicit through operators and functions, which is one reason why coding as much as possible of an XQuery implementation in XQuery itself is of interest, since it simplifies this kind of analysis(as discussed in a previous blog entry).

Index Building

Selectively building name indexes based on what’s needed can reduce the associated space costs. For example, if no nametests for attributes are present but the attribute axis is nevertheless used in some way, a name index doesn’t have to be populated for attributes.

If one assumes that the attribute and child axes are relatively fast, a node test such as @name or name are relatively inexpensive. However //name or //*[@name] are of a different magnitude. One way to improve the decision for index building is to use weighting, as I call it, of constructs.

For example, considering the high representation of attribute nodes in documents, notably expensive path expressions involving attributes must be present before populating an index with attribute nodes. This could make the cost of a name index bearable. Similarly, if a query containing a path expression with elements tests that can be evaluated by a sequential walk(no complex axes, no sorting or de-duplication) it perhaps neither qualifies for an index build.

Where to Go

Vaporware and speculation is what characterizes this text. Inventing approaches is not difficult, the hard, time consuming but not very interesting part is implementing and measuring. Possibly, for the mere result of realizing that the long path led to a dead end.

A yet wide issue is how users should access the result of a query(as verbosely discussed in XML APIs). A traditional approach as seen in W3C’s XPath DOM Module or XQuery API for Java is the use of iterators. Another approach could be more in the direction of the XProc pipeline language, where for a query one would declare and name output pipes which could be sent to with a to-pipe("qualified-name-of-pipe", expression) function. I believe it could possibly:

• Allow more operations to be carried out within the query instead of that the user inspects nodes from an iterator, for example
• With an improved understanding of how the results will be used, hence have a stronger foundation for further optimizations
• Designing practical APIs for retrieving the result.

However, I think one thing can be strongly concluded: declarative approaches is the way to go. One reason I am opposed to exposing functionality to users through the traditional tree API or low-level APIs in general is that they kill every attempt at doing clever things, in addition to delegating the task of conforming to standards to the user.

One can’t apply a pipeline/early-exit strategy if the user can retrieve child nodes as a list; one cannot attempt efficient lookups if the user is doing it with a for-loop; threading becomes the user’s responsibility if a higher perspective is absent; and it is difficult to adapt document building if all the user has told is that a document needs to be loaded, and so on. That’s why I believe XQuery and pipeline languages like XProc are good ways to go.

With XML documents, as with many other things, it is difficult to perceive actual circumstances. Hence we construct measuring devices. Abc. Here’s the output from invoking xmlstat on the catalog file that describes the tests in W3C ‘s XQuery Test Suite:

Document:         file:///home/fenglich/kde/src/playground/utils/xmlstat/XQTSCatalog.xml
File size(bytes): 8904307
-------------------------------------------
Elements:                  69062
Attributes:                170741
Non-Whitespace Text nodes: 32936
Whitespace Text nodes:     80810
Total text-node count:     113746
Comments:                  0
Processing Instructions:   2
Total:                     239805
-------------------------------------------
XQueryXQueryOffsetPath                                                        1
{http://www.w3.org/2005/02/query-test-XQTSCatalog}contextItem                 2
{http://www.w3.org/2005/02/query-test-XQTSCatalog}scenario                    4
{http://www.w3.org/2005/02/query-test-XQTSCatalog}input-document              5
{http://www.w3.org/2005/02/query-test-XQTSCatalog}role                        6
schema                                                                        8
{http://www.w3.org/2005/02/query-test-XQTSCatalog}citation-spec               9
input-document                                                                11
{http://www.w3.org/2005/02/query-test-XQTSCatalog}note                        14
{http://www.w3.org/2005/02/query-test-XQTSCatalog}input-URI                   23
{http://www.w3.org/2005/02/query-test-XQTSCatalog}context-property            30
{http://www.w3.org/2005/02/query-test-XQTSCatalog}implementation-defined-item 37
namespace                                                                     43
{http://www.w3.org/2005/02/query-test-XQTSCatalog}input-query                 44
{http://www.w3.org/2005/02/query-test-XQTSCatalog}source                      49
{http://www.w3.org/2005/02/query-test-XQTSCatalog}module                      58
FileName                                                                      75
ID                                                                            77
featureOwner                                                                  94
last-mod                                                                      138
{http://www.w3.org/2005/02/query-test-XQTSCatalog}test-group                  358
{http://www.w3.org/2005/02/query-test-XQTSCatalog}title                       359
{http://www.w3.org/2005/02/query-test-XQTSCatalog}expected-error              1141
{http://www.w3.org/2005/02/query-test-XQTSCatalog}output-file                 9638
{http://www.w3.org/2005/02/query-test-XQTSCatalog}test-case                   10519
is-XPath2                                                                     10544
date                                                                          10563
Creator                                                                       10585
{http://www.w3.org/2005/02/query-test-XQTSCatalog}input-file                  10636
variable                                                                      10703
{http://www.w3.org/2005/02/query-test-XQTSCatalog}description                 11380
section-title                                                                 13823
spec                                                                          13860
role                                                                          20303
name                                                                          21540
Total name count:                                                             71
-------------------------------------------
{http://www.w3.org/2005/02/query-test-XQTSCatalog}
{http://www.w3.org/2001/XMLSchema-instance}xsi
Total bindings count: 2
-------------------------------------------
Non-Whitespace chars: 1346407
Whitespace chars:     1706529

While raw and with blunt and confusing labeling, there are some things one can see:

• The document has in total 71 different qualified names, but they appear almost a quarter of million, 239803 times(element + attribute count).This is partly why namepools are popular both among relational storage models as well as tree implementors, since they reduce memory usage by an indescribable amount.
• The attribute count is more than twice as big as the element count. That put things in perspective, at least for me. That relationship is a bit off in a broader scale, but not that far, as we shall see.
• Text nodes containing only whitespace(always formatting basically) is larger than actual content(Non-Whitespace chars versus Whitespace chars).

These “conclusions” aren’t that interesting since they’re only based on one document, but I nevertheless state them because they roughly mimmic what The XML Web: a First Study says.

That paper contains statistics over XML documents found on the web, but is for these purposes relatively old(from 2003). I would for that reason not be surprised if the numbers has changed since then, considering the widespread use of feeds, for instance. Another point is that their statistics in some cases are quite heavily influenced by individual sites(such as http://rpmfind.net/ or http://w3.org/), which to me suggests that a too small data sampling is used.

Here’s a very concentrated list of their conclusions:

• “WAP and RDF make up 26% and 17% of all document on the XML Web, respectively.”
• “Our results show that XML documents are in fact relatively shallow: 99% of them have less than 8 levels of element nesting. Also, 15% of the documents we analyzed have recursive content, in which there is much regularity.”
• “Only 75 different DTDs are referenced in our sample”(which is about 200,000 documents). “92% of all DTD references are made to norms 1.1 or 1.2 of the WAP protocol.”
• “Only 0.09% of the document suse either the attribute label SchemaLocation or noNameSpaceSchemaLocation”(but as in the case with DTDs, not referencing a schema from the document is not equivalent to not using a schema for that document).
• “For documents up to 4096 bytes, the number of element nodes dominates the distributions”
• “For documents larger than 4096 bytes, there are proportionally more atttribute nodes than element nodes(51.13% vs. 37.83%).”
• “These observations led us to conclude that the structural information found in XML documents is in fact dominant over the textual content.”
• “It turns out that 782,602 elements(5% in total) have mixed content. Surprisingly, these elements belong to 138,298 documents(72% of all documents).”
• “The prevailing assumption in this community[database community] is that attributes and mixed element content are not as important as element content.”
• “99% of the documents have fewer than 8 levels[level refers to tree depth]. The average depth is 4, and the deepest document has 135 levels.”
• “On average, the second level contains more attributes than any other level. In fact, 89% of all attribute are found in the first 3 levels of the documents.”
• “77% of all element nodes and 6% of all text are found in the first 3 levels of the documents.”
• “28,208 XML document(14.81% of the total) contain recursive elements.”
• “The average document size is 4kb”

I find statistics like these very useful and I believe they can play an important role in discussing implementation approaches.

I would surely not mind a second such study. Perhaps it should have a larger document sampling in order to not be thrown off by individual sites. It would also be nice to see the distribution of encodings used, and the relationship between whitespace-only and regular text nodes.

And of course, xmlstat could be a lot more useful. Essentially that an XHTML page is produced with bar charts describing name distributions, node type distributions, concentrations in relation to depth, and so on. It would help with making decisions for implementations. It wouldn’t surprise if it’s useful for things like XSL-T debugging as well.

In either case, feel free to use or improve this simple and primitive tool. It’s in KDE’s SVN repository, playground/utils/xmlstat, GNU GPL licensed and based on QtCore, QtXml and qmake.

But that’s the positive side of it.

The major problem is that scientists seems to live in somekind of denial.

For example, XMill: an Efficient Compressor for XML Data discusses various techniques for efficiently compressing XML documents. It compares a log of HTTP traffic compressed and uncompressed in a custom text format, to the same data stored in an XML format, and discusses the effectiveness of different compression techniques applied on the custom format, the XML document, and the XML document. Their compression applied on the XML document(which is a lot more verbose than the custom text format) yeilds a smaller file than the custom format compressed, which is impressive — nice numbers.

The problem is that in order to reach those compression ratios, one has to use custom compression algorithms and in addition feed a configuration file that teaches the compressor about the XML format being compressed. In other words, it turns the XML document into a custom, binary format which counteracts the precise reason to why XML was chosen in the first place. So no matter how good those compression ratios are, they can’t be deployed except for within a closed, proprietary network, and hence they will most likely stay on that paper.

Another problem is presentation. Many papers spends a good portion of their text translating terms into formalism and simply other names for things. Sometimes this formalism is necessary, but it always results in requiring quite some effort to decypher a single paper’s home made symbols, conventions and terms. If all I did was reading papers I presume it wouldn’t be problematic, but until that happens it is.

W3C XML Schema perhaps gets criticized the most for being an academic exercise. That’s for example why I (relatively) enjoyed reading XProc: An XML Pipeline Language. Even though its subject is technical, it doesn’t try to sound so or academic. It adapts to me as a reader.

A related topic is how good papers are at explaining their theories. Some papers I grasp after the first read, while others I am still trying with. The same applies to Wikipedia entries. Is my brain having a latent glitch? Can it all be explained as that some theories are inherently more difficult to express? I’d say that a scientist’s success is not only dependent on what he quotes or factual data he or she have managed to consume, it also depends on this person’s pedagogical skills.

I have lost count of how many papers I’ve read that somewhere have sneaked into their discussion about algorithms for axes and storing XML, that processing instructions, comments, namespace instructions, text nodes or something else is ignored. Hello? I’m sure that’s a very comfortable way to do things, but it’s quite difficult to implement that super nifty storing scheme or that super cool axis algorithm for XQuery if the cost is that… you don’t implement XQuery.

The same can be seen with some XML parser “optimizations”. When it turns out that it’s done at the cost of not conforming to the XML specification, it’s not impressive any longer. It’s just wrong. It’s not cool to write an algorithm or implement an optimization that ignore reality. It’s cool to play by the rules and on top of that bring actual improvements, instead of bending the world towards what is practical for the researcher.

I think my effective contributions are modest, although one could say I’ve tried. But I can of course always express my view.

Her response, which made me think, was:

Your view is what matters to me, not some generalized or idealistic view from the usability contributors themselves. I have certainly learned some interesting things from the developers.

One could of course take that as a negative comments towards “the usability contributors” but I think it was to address a certain problem.

Aaron wrote in a recent blog entry:

[...] It punishes developers like Tim for speaking openly about the challenges we face. the free software community relies on our ability to speak openly and honestly to each other; if we start to get punished for it then we have a real problem.

Although the blog entry is in general about a certain article, Aaron is in that paragraph simply pointing out that being able to talk and address issues is dead important.

As reply to one of Celeste’s questions on KDE’s usability, I wrote:

One thing I admire the GNOME project much of, is their ability to change. They manage to get ideas /implemented/ in their main
line, without getting shot down at the proposal-stage. Those ideas might one disagree with or they are perhaps even downright wrong, but the ability to
change, to test new ideas, is a prerequisite for reaching the right ideas. Progress isn’t a linear progression of constantly correct changes, and the
working process must be adapted for that.

I can’t name a particular achievement, but each time a usability idea advances from being a proposal to being tried on the practical level, progress
is happening.

which as well merely says “don’t shoot down ideas just because they’re different or sound bad.” I’m of course only speculating from my view on things, but it wouldn’t surprise me if many nods to that it can be difficult to not have an idea stalled as early as when it is a suggestion.

Open Source and Free Software, at least if we go back in time, was a liberator for sick things in the IT industry, and will continue to be so, as long as those values are withheld. But perhaps the community is too consumed with its achievements on the democratic side, to see the sides of itself that fights its own mindset.

Belief fucks up mankind in spectacular ways. “We just need a revolution from system X to system Y and we will have no more corruption”, “It is ok to reduce the democratic rights for Them because They are not Us”, “We don’t have to listen because we are right”, and other countless examples that demonstrates people thinking there is a difference between people as long as they have a different skin color, operating system, religion, political system, desktop environment, and so on.

My point is simple and well repeated: openess is important. This time, it’s being emphasized for the open source community. Things will stall if ideas from GNOME are on mailing lists tuted as evilness, if less technically minded users are What’s Wrong, if KDE is considered to always be perfect, or if new ideas are shot down for not being what we have. And blogs isn’t the only way ideas are expressed, what ideas that are implemented in software, is another way as well.