<!doctype html>
<html>
<head>
<meta charset="UTF-8">
</head>
<body>
<blockquote type="cite">
<div>
On 01/20/2022 9:20 AM Alessio Cecchi <alessio@skye.it> wrote:
<br>
<br>I'm trying to setup fts-flatcurve with tokenization.
<br>
<br>What are the differences/benefits with "fts_filters = normalizer-icu" vs "fts_filters = lowercase"?
<br>
<br>Reading the Doc I found about normalizer-icu "This is potentially very resource intensive." and about lowercase "Supports UTF8, when compiled with libicu".
<br>
<br>So, using lowercase is almost the same that normalizer-icu but faster?
</div>
</blockquote>
<div>
No, these are 2 different actions.
<br>
</div>
<div class="default-style">
<br>
</div>
<div class="default-style">
Lowercase tries to use language rules to map characters to a "lowercase" equivalent, which is character/language dependent.
</div>
<div class="default-style">
<br>
</div>
<div class="default-style">
Normalization tries to take a string and reduce it to a unique, normalized form, that can be directly compared to other normalized strings. UTF, for example, can have strings that display the same to the user but contain very different byte data. For example, it is possible to create more complicated glyphs by either using a specific code-point (i.e., a 4 byte UTF element) or by using a combination of UTF sequences that, when combined, create an identical display of the character.
<br>
</div>
<div class="default-style">
<br>
</div>
<div class="default-style">
Normalization is a very complicated topic. <a href="https://en.wikipedia.org/wiki/Unicode_equivalence">https://en.wikipedia.org/wiki/Unicode_equivalence</a> might help with further understanding.
<br>
</div>
<div class="default-style">
<br>
</div>
<div class="default-style">
The ICU library deals with general internationalization support, and these two filters are using different parts of that library to do different things. They are not replacements for each other, they are complimentary - you could normalize a string and then lowercase it, for example.
<br>
</div>
<div class="default-style">
<br>
</div>
<div class="default-style">
michael
<br>
</div>
<div class="default-style">
<br>
</div>
<blockquote type="cite">
<p><br>FYI<br><br>for using fts-flatcurve with dovecot RPM packages from repo.dovecot.org you have to rebuild with --with-icu --with-stemmer --with-textcat and related library.<br><br>Thanks<br></p>
<pre class="moz-signature">--
Alessio Cecchi
Postmaster @ <a href="http://www.qboxmail.it" class="moz-txt-link-freetext">http://www.qboxmail.it</a>
<a href="https://www.linkedin.com/in/alessice" class="moz-txt-link-freetext">https://www.linkedin.com/in/alessice</a><br></pre>
</blockquote>
</body>
</html>