Qt + Flex = Problems

GNU Flex, “the fast lexical analyser generator“, is a handy tool for writing compiler-related software, as those who have knows. It saves tremendous amounts of code, allows tokenization rules to be expressed in a concise way, and the generated tokenizer is fast.

For that reason, it is a pity Flex cannot be used in serious projects. Because it doesn’t support Unicode.

I think this is a significant problem for KDE. Not because developers instead chooses hand written tokenizers in order to support Unicode. Because they don’t. They use Flex, despite its severe limitation. Hence, is Unicode ignorant code introduced, and KDE’s brand of multilingual, internationalized software diminishes.

Although it would be better with hand-written tokenizers, they still have the problem of requiring large efforts. For example, Patternist’s tokenizer that handles XQuery/XPath is big and complex, which Flex surely could help simplifying.

I think the solution is to add a QString mode to Flex. When enabled, the tokenizer would consume QString/QChar and therefore in a transparent way take care of Unicode support, while the developer still uses the comforts of Flex. I know nothing about Flex’s internals, but I suspect it would require quite some work.

I think such a thing would mean a lot for KDE, due to the amount of tokenizers in use across the repository. If I were a Troll(of the good kind), I would consider sponsoring getting this infrastructure in place, because it would be an important help for KDE but also a plus for Qt in general.