I have zero knowledge about Baloo, but I can add some comments regarding Unicode.

• the four ranges you used are all adjacent, so you could contract to {0x4E00, 0x9FFF}
• there are more ranges for CJK characters in the BMP, at least {0x3400, 0x4DBF} would be useful (I don't know if CJK users ever use the compatibility characters)
• to be able to fully support the remaining CJK blocks in higher planes, the code would need to handle surrogate pairs
• if Baloo doesn't handle CJK, it maybe also doesn't handle other non-Latin scripts, so I suggest to use QChar::category()

Regarding this - I don't know if it is really chinese look foreign enough to me anyway.
Some lines of text in your test script surely look like Japanese Hiragana to me, especially this one (and tests related to this)

echo "otto东到宛平路anna"> "終末なにしてますか？忙しいですか？救ってもらっていいですか？ EP01 太阳の倾いたこの世界で -broken chronograph-.txt"

Google translate confirms :)
But do your ranges include that characters? This answer on stackoverflow says that there are also other ranges for Hiragana, Katakana, etc... as @cfeck already said. Does it pass the test for you?

link to tables from answer

In D11552#230870, @alexeymin wrote:

Regarding this - I don't know if it is really chinese look foreign enough to me anyway.
Some lines of text in your test script surely look like Japanese Hiragana to me, especially this one (and tests related to this)

echo "otto东到宛平路anna"> "終末なにしてますか？忙しいですか？救ってもらっていいですか？ EP01 太阳の倾いたこの世界で -broken chronograph-.txt"

That's the only thing I was sure of (It was in fact an mkv I just watched). At this stage the actual language does not really matter.

But do your ranges include that characters? This answer on stackoverflow says that there are also other ranges for Hiragana, Katakana, etc... as @cfeck already said.

My rationale was not to throw in every range mentioned on that wikipedia page, but just enough to make this work and illustrate the general approach.

Does it pass the test for you?

All except the last two that is '*ですか？ EP01' (<mixture of Latin/Hiragana) and 'ですか' (<pure Hiragana). I could lie now and say I left out Hiragana character on purpose. I didn't, but for Hiragana the one grapheme = one search term does not apply. So those tests in fact should fail.

@cfeck

if Baloo doesn't handle CJK, it maybe also doesn't handle other non-Latin scripts, so I suggest to use QChar::category()

I wasn't aware of QChar::category() . Thank you.

If you're going to loop over a QString and break it down into QChars anyway, why don't you just use QChar::script?

For the record though - a better way to do this is to use QTextBoundaryFinder which will operate e.g. on grapheme cluster boundaries. This still isn't super great for Chinese though. If you want to really-properly do it you'll end up depending on ICU and using its BreakIterator combined with dict-based support for Chinese, which isn't terribly fast however.

In D11552#231330, @hein wrote:

For the record though - a better way to do this is to use QTextBoundaryFinder which will operate e.g. on grapheme cluster boundaries. This still isn't super great for Chinese though. If you want to really-properly do it you'll end up depending on ICU and using its BreakIterator combined with dict-based support for Chinese, which isn't terribly fast however.

There are a few implications here:

• splitting to much generates to unspecific terms, especially in case of full text indexing (Think of splitting a western language at character level, most texts likely contain almost the full alphabet. Same likely applies to Katakana with its about ~100 graphemes)
• term generation at query and index time have to agree about what a term is, otherwise a search will likely return nothing. Changing the splitting at a later time will require reindexing all affected files
• better splitting will cost some more time at index generation, but likely makes searching faster (additional time for term generation will be neglegible, but the search terms are less complex - e.g. "abc" instead of "a" AND "b" AND "c").

In D11552#231784, @bruns wrote:

In D11552#231330, @hein wrote:

For the record though - a better way to do this is to use QTextBoundaryFinder which will operate e.g. on grapheme cluster boundaries. This still isn't super great for Chinese though. If you want to really-properly do it you'll end up depending on ICU and using its BreakIterator combined with dict-based support for Chinese, which isn't terribly fast however.

There are a few implications here:

• splitting to much generates to unspecific terms, especially in case of full text indexing (Think of splitting a western language at character level, most texts likely contain almost the full alphabet. Same likely applies to Katakana with its about ~100 graphemes)
• term generation at query and index time have to agree about what a term is, otherwise a search will likely return nothing. Changing the splitting at a later time will require reindexing all affected files
• better splitting will cost some more time at index generation, but likely makes searching faster (additional time for term generation will be neglegible, but the search terms are less complex - e.g. "abc" instead of "a" AND "b" AND "c").

Currently termgenerator uses QTextBoundaryFinder bf(QTextBoundaryFinder::Word, text);

@cfeck: Thanks a lot for your help.