You can do it yourself. There is a link below every mail (and also this one) from this list where you can unsubscribe.

On 23 Nov 2018, at 13:49, Yann Shukor yann.shukor@azurtem.com wrote:

Please remove me from your mailing list thanks you

...

---

Dovecot-news mailing list Dovecot-news@dovecot.org https://dovecot.org/mailman/listinfo/dovecot-news

On 11/23/2018 9:39 AM, Odhiambo Washington wrote:

On Fri, 23 Nov 2018 at 17:30, The Doctor mailto:doctor@doctor.nl2k.ab.ca> wrote:

On Fri, Nov 23, 2018 at 04:02:27PM +0300, Odhiambo Washington wrote:
> On Fri, 23 Nov 2018 at 15:29, Timo Sirainen <tss@iki.fi
<mailto:tss@iki.fi>> wrote:
>
> > https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz
> > https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig
> > Binary packages in https://repo.dovecot.org/
> >
> >  * The default postmaster_address is now "postmaster@<user
domain or
> >    server hostname>". If username contains the @domain part,
that's
> >    used. If not, then the server's hostname is used.
> >  * "doveadm stats dump" now returns two decimals for the "avg"
field.
> >
> >  + Added push notification driver that uses a Lua script
> >  + Added new SQL, DNS and connection events.
> >    See https://wiki2.dovecot.org/Events
> >  + Added "doveadm mailbox cache purge" command.
> >  + Added events API support for Lua scripts
> >  + doveadm force-resync -f parameter performs "index fsck"
while opening
> >    the index. This may be useful to fix some types of broken
index files.
> >    This may become the default behavior in a later version.
> >  - director: Kicking a user crashes if login process is very slow
> >  - pop3_no_flag_updates=no: Don't expunge DELEted and RETRed
messages
> >    unless QUIT is sent.
> >  - auth: Fix crypt() segfault with glibc-2.28+
> >  - imap: Running UID FILTER script with errors assert-crashes
> >  - dsync, pop3-migration: POP3 UIDLs weren't added to
> >    dovecot.index.cache while mails were saved.
> >  - dict clients may have been using 100% CPU while waiting for
dict
> >    server to finish commands.
> >  - doveadm user: Fixed user listing via HTTP API
> >  - All levels of Cassandra log messages were logged as Dovecot
errors.
> >  - http/smtp client may have crashed after SSL handshake
> >  - Lua auth converted strings that looked like numbers into
numbers.
> >
> >
>
> FreeBSD 9.3 (i386):
>
>
> Making all in lib-master
> gcc -DHAVE_CONFIG_H -I. -I../..  -I../../src/lib -I../../src/lib-dns
> -I../../src/lib-test  -I../../src/lib-settings
> -I../../src/lib-ssl-iostream
> -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\"
> -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\"
> -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\"
> -DBINDIR=\""/opt/dovecot2.3/bin"\"    -std=gnu99 -g -O2
-fstack-protector
> -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes
> -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2
> -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2
> -I/usr/local/include -MT test-event-stats.o -MD -MP -MF
> .deps/test-event-stats.Tpo -c -o test-event-stats.o
test-event-stats.c
> test-event-stats.c: In function 'kill_stats_child':
> test-event-stats.c:101: warning: implicit declaration of
function 'kill'
> test-event-stats.c:101: error: 'SIGKILL' undeclared (first use
in this
> function)
> test-event-stats.c:101: error: (Each undeclared identifier is
reported only
> once
> test-event-stats.c:101: error: for each function it appears in.)
> test-event-stats.c: In function 'test_no_merging2':
> test-event-stats.c:361: warning: format '%lu' expects type 'long
unsigned
> int', but argument 2 has type 'uint64_t'
> test-event-stats.c: In function 'test_no_merging3':
> test-event-stats.c:387: warning: format '%lu' expects type 'long
unsigned
> int', but argument 2 has type 'uint64_t'
> test-event-stats.c:387: warning: format '%lu' expects type 'long
unsigned
> int', but argument 4 has type 'uint64_t'
> test-event-stats.c:387: warning: format '%lu' expects type 'long
unsigned
> int', but argument 6 has type 'uint64_t'
> test-event-stats.c: In function 'test_merge_events2':
> test-event-stats.c:452: warning: format '%lu' expects type 'long
unsigned
> int', but argument 2 has type 'uint64_t'
> test-event-stats.c: In function 'test_skip_parents':
> test-event-stats.c:484: warning: format '%lu' expects type 'long
unsigned
> int', but argument 2 has type 'uint64_t'
> test-event-stats.c:484: warning: format '%lu' expects type 'long
unsigned
> int', but argument 4 has type 'uint64_t'
> test-event-stats.c:484: warning: format '%lu' expects type 'long
unsigned
> int', but argument 6 has type 'uint64_t'
> test-event-stats.c: In function 'test_merge_events_skip_parents':
> test-event-stats.c:526: warning: format '%lu' expects type 'long
unsigned
> int', but argument 2 has type 'uint64_t'
> test-event-stats.c:526: warning: format '%lu' expects type 'long
unsigned
> int', but argument 4 has type 'uint64_t'
> test-event-stats.c:526: warning: format '%lu' expects type 'long
unsigned
> int', but argument 6 has type 'uint64_t'
> *** Error code 1
>

Are you able to upgrade to FreeBSD 10 or 11 ?

I will be upgrading the servers to 11.2 soon :)

The OS version is irrelevant.

On Sat, 24 Nov 2018 at 10:56, Aki Tuomi aki.tuomi@open-xchange.com wrote:

...

On 23 November 2018 at 17:46 Brad Smith brad@comstyle.com wrote:

On 11/23/2018 9:39 AM, Odhiambo Washington wrote:

...

On Fri, 23 Nov 2018 at 17:30, The Doctor mailto:doctor@doctor.nl2k.ab.ca> wrote:

On Fri, Nov 23, 2018 at 04:02:27PM +0300, Odhiambo Washington

wrote:

...

> On Fri, 23 Nov 2018 at 15:29, Timo Sirainen <tss@iki.fi
<mailto:tss@iki.fi>> wrote:
>
> > https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz
> > https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig
> > Binary packages in https://repo.dovecot.org/
> >
> >  * The default postmaster_address is now "postmaster@<user
domain or
> >    server hostname>". If username contains the @domain part,
that's
> >    used. If not, then the server's hostname is used.
> >  * "doveadm stats dump" now returns two decimals for the "avg"
field.
> >
> >  + Added push notification driver that uses a Lua script
> >  + Added new SQL, DNS and connection events.
> >    See https://wiki2.dovecot.org/Events
> >  + Added "doveadm mailbox cache purge" command.
> >  + Added events API support for Lua scripts
> >  + doveadm force-resync -f parameter performs "index fsck"
while opening
> >    the index. This may be useful to fix some types of broken
index files.
> >    This may become the default behavior in a later version.
> >  - director: Kicking a user crashes if login process is very

slow

...

> >  - pop3_no_flag_updates=no: Don't expunge DELEted and RETRed
messages
> >    unless QUIT is sent.
> >  - auth: Fix crypt() segfault with glibc-2.28+
> >  - imap: Running UID FILTER script with errors assert-crashes
> >  - dsync, pop3-migration: POP3 UIDLs weren't added to
> >    dovecot.index.cache while mails were saved.
> >  - dict clients may have been using 100% CPU while waiting for
dict
> >    server to finish commands.
> >  - doveadm user: Fixed user listing via HTTP API
> >  - All levels of Cassandra log messages were logged as Dovecot
errors.
> >  - http/smtp client may have crashed after SSL handshake
> >  - Lua auth converted strings that looked like numbers into
numbers.
> >
> >
>
> FreeBSD 9.3 (i386):
>
>
> Making all in lib-master
> gcc -DHAVE_CONFIG_H -I. -I../..  -I../../src/lib

-I../../src/lib-dns

...

> -I../../src/lib-test  -I../../src/lib-settings
> -I../../src/lib-ssl-iostream
> -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\"
> -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\"
> -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\"
> -DBINDIR=\""/opt/dovecot2.3/bin"\"    -std=gnu99 -g -O2
-fstack-protector
> -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W

-Wmissing-prototypes

...

> -Wmissing-declarations -Wpointer-arith -Wchar-subscripts

-Wformat=2

...

> -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2
> -I/usr/local/include -MT test-event-stats.o -MD -MP -MF
> .deps/test-event-stats.Tpo -c -o test-event-stats.o
test-event-stats.c
> test-event-stats.c: In function 'kill_stats_child':
> test-event-stats.c:101: warning: implicit declaration of
function 'kill'
> test-event-stats.c:101: error: 'SIGKILL' undeclared (first use
in this
> function)
> test-event-stats.c:101: error: (Each undeclared identifier is
reported only
> once
> test-event-stats.c:101: error: for each function it appears in.)
> test-event-stats.c: In function 'test_no_merging2':
> test-event-stats.c:361: warning: format '%lu' expects type 'long
unsigned
> int', but argument 2 has type 'uint64_t'
> test-event-stats.c: In function 'test_no_merging3':
> test-event-stats.c:387: warning: format '%lu' expects type 'long
unsigned
> int', but argument 2 has type 'uint64_t'
> test-event-stats.c:387: warning: format '%lu' expects type 'long
unsigned
> int', but argument 4 has type 'uint64_t'
> test-event-stats.c:387: warning: format '%lu' expects type 'long
unsigned
> int', but argument 6 has type 'uint64_t'
> test-event-stats.c: In function 'test_merge_events2':
> test-event-stats.c:452: warning: format '%lu' expects type 'long
unsigned
> int', but argument 2 has type 'uint64_t'
> test-event-stats.c: In function 'test_skip_parents':
> test-event-stats.c:484: warning: format '%lu' expects type 'long
unsigned
> int', but argument 2 has type 'uint64_t'
> test-event-stats.c:484: warning: format '%lu' expects type 'long
unsigned
> int', but argument 4 has type 'uint64_t'
> test-event-stats.c:484: warning: format '%lu' expects type 'long
unsigned
> int', but argument 6 has type 'uint64_t'
> test-event-stats.c: In function 'test_merge_events_skip_parents':
> test-event-stats.c:526: warning: format '%lu' expects type 'long
unsigned
> int', but argument 2 has type 'uint64_t'
> test-event-stats.c:526: warning: format '%lu' expects type 'long
unsigned
> int', but argument 4 has type 'uint64_t'
> test-event-stats.c:526: warning: format '%lu' expects type 'long
unsigned
> int', but argument 6 has type 'uint64_t'
> *** Error code 1
>

Are you able to upgrade to FreeBSD 10 or 11 ?

I will be upgrading the servers to 11.2 soon :)

The OS version is irrelevant.

Fix for this format issue and the signal.h issue can be found in master now, and you can download the patch at

https://github.com/dovecot/core/compare/10048229%5E...de42b54a.patch

Aki

The patch doesn't resolve the permissions issue on /var/run/dovecot//stats-writer so I have backed off again - to 2.3.3.

-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."

On Sat, 24 Nov 2018 at 12:04, Aki Tuomi aki.tuomi@open-xchange.com wrote:

On 24 November 2018 at 10:55 Odhiambo Washington < odhiambo@gmail.com> wrote:

On Sat, 24 Nov 2018 at 10:56, Aki Tuomi < aki.tuomi@open-xchange.com> wrote:

...

On 23 November 2018 at 17:46 Brad Smith < brad@comstyle.com> wrote:

...

On 11/23/2018 9:39 AM, Odhiambo Washington wrote:

...

On Fri, 23 Nov 2018 at 17:30, The Doctor < doctor@doctor.nl2k.ab.ca > wrote:

On Fri, Nov 23, 2018 at 04:02:27PM +0300, Odhiambo Washington

wrote:

...

On Fri, 23 Nov 2018 at 15:29, Timo Sirainen < tss@iki.fi > wrote:

...

https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/

• The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
• "doveadm stats dump" now returns two decimals for the "avg" field.

• Added push notification driver that uses a Lua script
• Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
• Added "doveadm mailbox cache purge" command.
• Added events API support for Lua scripts
• doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.

• director: Kicking a user crashes if login process is very

slow

...

...

• pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
• auth: Fix crypt() segfault with glibc-2.28+
• imap: Running UID FILTER script with errors assert-crashes
• dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
• dict clients may have been using 100% CPU while waiting for dict server to finish commands.
• doveadm user: Fixed user listing via HTTP API
• All levels of Cassandra log messages were logged as Dovecot errors.
• http/smtp client may have crashed after SSL handshake
• Lua auth converted strings that looked like numbers into numbers.

FreeBSD 9.3 (i386):

Making all in lib-master gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib

-I../../src/lib-dns

...

-I../../src/lib-test -I../../src/lib-settings -I../../src/lib-ssl-iostream -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\" -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\" -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\" -DBINDIR=\""/opt/dovecot2.3/bin"\" -std=gnu99 -g -O2 -fstack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W

-Wmissing-prototypes

...

-Wmissing-declarations -Wpointer-arith -Wchar-subscripts

-Wformat=2

...

-Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 -I/usr/local/include -MT test-event-stats.o -MD -MP -MF .deps/test-event-stats.Tpo -c -o test-event-stats.o test-event-stats.c test-event-stats.c: In function 'kill_stats_child': test-event-stats.c:101: warning: implicit declaration of function 'kill' test-event-stats.c:101: error: 'SIGKILL' undeclared (first use in this function) test-event-stats.c:101: error: (Each undeclared identifier is reported only once test-event-stats.c:101: error: for each function it appears in.) test-event-stats.c: In function 'test_no_merging2': test-event-stats.c:361: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c: In function 'test_no_merging3': test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'uint64_t' test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned int', but argument 6 has type 'uint64_t' test-event-stats.c: In function 'test_merge_events2': test-event-stats.c:452: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c: In function 'test_skip_parents': test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'uint64_t' test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned int', but argument 6 has type 'uint64_t' test-event-stats.c: In function 'test_merge_events_skip_parents': test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'uint64_t' test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned int', but argument 6 has type 'uint64_t' *** Error code 1

Are you able to upgrade to FreeBSD 10 or 11 ?

...

I will be upgrading the servers to 11.2 soon :)

The OS version is irrelevant.

Fix for this format issue and the signal.h issue can be found in master now, and you can download the patch at

https://github.com/dovecot/core/compare/10048229%5E...de42b54a.patch

Aki

The patch doesn't resolve the permissions issue on /var/run/dovecot//stats-writer so I have backed off again - to 2.3.3.

-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."

Well, to be exact no one claims it does fix it. You can work around it with

service stats { unix_listener stats-writer { mode = 0777 } }

---

Aki Tuomi

I'll wait for the fix instead of the workaround.

Thank you very much for entertaining my noise :-)

-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."

On 11/23/2018 9:31 AM, The Doctor wrote:

On Fri, Nov 23, 2018 at 04:06:53PM +0300, Odhiambo Washington wrote:

...

On Fri, 23 Nov 2018 at 15:29, Timo Sirainen tss@iki.fi wrote:

...

https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/

• The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
• "doveadm stats dump" now returns two decimals for the "avg" field.

• Added push notification driver that uses a Lua script
• Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
• Added "doveadm mailbox cache purge" command.
• Added events API support for Lua scripts
• doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.

• director: Kicking a user crashes if login process is very slow
• pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
• auth: Fix crypt() segfault with glibc-2.28+
• imap: Running UID FILTER script with errors assert-crashes
• dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
• dict clients may have been using 100% CPU while waiting for dict server to finish commands.
• doveadm user: Fixed user listing via HTTP API
• All levels of Cassandra log messages were logged as Dovecot errors.
• http/smtp client may have crashed after SSL handshake
• Lua auth converted strings that looked like numbers into numbers.

FreeBSD 11.2 (amd64):

gmake[2]: Entering directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src/lib-master' gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib -I../../src/lib-dns -I../../src/lib-test -I../../src/lib-settings -I../../src/lib-ssl-iostream -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\" -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\" -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\" -DBINDIR=\""/opt/dovecot2.3/bin"\" -std=gnu99 -g -O2 -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 -I/usr/local/include -MT test-event-stats.o -MD -MP -MF .deps/test-event-stats.Tpo -c -o test-event-stats.o test-event-stats.c test-event-stats.c: In function 'kill_stats_child': test-event-stats.c:101:2: warning: implicit declaration of function 'kill' [-Wimplicit-function-declaration] (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: error: 'SIGKILL' undeclared (first use in this function) (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: note: each undeclared identifier is reported only once for each function it appears in gmake[2]: *** [Makefile:656: test-event-stats.o] Error 1 gmake[2]: Leaving directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src/lib-master' gmake[1]: *** [Makefile:565: install-recursive] Error 1 gmake[1]: Leaving directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src' gmake: *** [Makefile:683: install-recursive] Error 1

Looks like our porters will have their hands full.

Complete over exaggeration.

Hi

THis is the lines I have in my dmesg (see below)

In dovecot log , I see:

Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop)

What do to about it ?

Using lastest 2.3.4 version

Thank you

[132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55 [134031.969596] auth[27031]: segfault at 55e509612c30 ip 000055e509612c30 sp 00007ffeb96dee48 error 15 [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55 [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip 00007f393841013e sp 00007ffe4a16efc0 error 6 in libdovecot.so.0.0.0[7f3938363000+e2000] [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip 00007f4376e32ffb sp 00007fff1cfdc000 error 6 in libdovecot.so.0.0.0[7f4376d86000+e2000] [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83 ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b 00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip 00007ffec43fdcff sp 00007ffec3ba5000 error 6 [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip 00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in libdovecot.so.0.0.0[7fb7c9ba2000+e2000] [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48 [135796.171575] auth[11121]: segfault at 555f8645cc30 ip 0000555f8645cc30 sp 00007ffcbb510868 error 15 [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55 [136710.562003] auth[17828]: segfault at 563443604c30 ip 0000563443604c30 sp 00007ffc1aa8b498 error 15 [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56 [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip 000055b27bc63c30 sp 00007ffd5d5b9298 error 15 [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17 cd 7f 00 00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4 7b b2 55 [138521.070485] auth[32481]: segfault at 556e05197c30 ip 0000556e05197c30 sp 00007ffe87217c08 error 15 [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20 3c 7f 00 00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17 05 6e 55 [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip 00005570d3e46c30 sp 00007ffc9118a5e8 error 15 [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71 11 7f 00 00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2 d3 70 55 [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip 00007ffe49ba7cff sp 00007ffe49317000 error 6 [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip 00007fa68292a13e sp 00007ffcdc00efb0 error 6 in libdovecot.so.0.0.0[7fa68287d000+e2000] [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip 00007f0e09ae259f sp 00007ffc90beb000 error 6 in libdovecot.so.0.0.0[7f0e09a35000+e2000] [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb 2d f5 ff 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8 6a fb ff [139747.003229] auth[9545]: segfault at 558b5e039c30 ip 0000558b5e039c30 sp 00007ffd43633008 error 15 [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e 75 7f 00 00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01 5e 8b 55 [140640.900512] auth[17221]: segfault at 55e952d41c30 ip 000055e952d41c30 sp 00007ffc36fa0908 error 15 [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80 b9 7f 00 00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1 52 e9 55 [140777.927292] auth[18114]: segfault at 55bd72adec30 ip 000055bd72adec30 sp 00007ffca394ba98 error 15 [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67 d7 7f 00 00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab 72 bd 55

Thank you Aki

here the requested data:

File system : ext4

dovecot -n :

# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M

(...)

passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)

On 2018-11-25 08:08, Aki Tuomi wrote:

...

On 25 November 2018 at 06:29 Joan Moreau < jom@grosjo.net> wrote:

Hi

THis is the lines I have in my dmesg (see below)

In dovecot log , I see:

Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop)

What do to about it ?

Using lastest 2.3.4 version

Thank you

---

[132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55 [134031.969596] auth[27031]: segfault at 55e509612c30 ip 000055e509612c30 sp 00007ffeb96dee48 error 15 [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55 [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip 00007f393841013e sp 00007ffe4a16efc0 error 6 in libdovecot.so.0.0.0[7f3938363000+e2000] [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip 00007f4376e32ffb sp 00007fff1cfdc000 error 6 in libdovecot.so.0.0.0[7f4376d86000+e2000] [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83 ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b 00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip 00007ffec43fdcff sp 00007ffec3ba5000 error 6 [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip 00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in libdovecot.so.0.0.0[7fb7c9ba2000+e2000] [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48 [135796.171575] auth[11121]: segfault at 555f8645cc30 ip 0000555f8645cc30 sp 00007ffcbb510868 error 15 [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55 [136710.562003] auth[17828]: segfault at 563443604c30 ip 0000563443604c30 sp 00007ffc1aa8b498 error 15 [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56 [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip 000055b27bc63c30 sp 00007ffd5d5b9298 error 15 [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17 cd 7f 00 00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4 7b b2 55 [138521.070485] auth[32481]: segfault at 556e05197c30 ip 0000556e05197c30 sp 00007ffe87217c08 error 15 [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20 3c 7f 00 00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17 05 6e 55 [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip 00005570d3e46c30 sp 00007ffc9118a5e8 error 15 [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71 11 7f 00 00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2 d3 70 55 [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip 00007ffe49ba7cff sp 00007ffe49317000 error 6 [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip 00007fa68292a13e sp 00007ffcdc00efb0 error 6 in libdovecot.so.0.0.0[7fa68287d000+e2000] [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip 00007f0e09ae259f sp 00007ffc90beb000 error 6 in libdovecot.so.0.0.0[7f0e09a35000+e2000] [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb 2d f5 ff 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8 6a fb ff [139747.003229] auth[9545]: segfault at 558b5e039c30 ip 0000558b5e039c30 sp 00007ffd43633008 error 15 [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e 75 7f 00 00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01 5e 8b 55 [140640.900512] auth[17221]: segfault at 55e952d41c30 ip 000055e952d41c30 sp 00007ffc36fa0908 error 15 [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80 b9 7f 00 00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1 52 e9 55 [140777.927292] auth[18114]: segfault at 55bd72adec30 ip 000055bd72adec30 sp 00007ffca394ba98 error 15 [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67 d7 7f 00 00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab 72 bd 55

Please see https://dovecot.org/bugreport.html

Aki

It's still missing core dump (or bt full from it)

Aki

On 27.11.2018 8.39, Joan Moreau wrote:

Thank you Aki

here the requested data (below)

Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)

Bug appears mostly in auth process and index-worker

dovecot -n :

# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M

(...)

passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)

 

On 2018-11-25 08:08, Aki Tuomi wrote:

 

    On 25 November 2018 at 06:29 Joan Moreau < jom@grosjo.net
    <mailto:jom@grosjo.net>> wrote:
     
     
    Hi
     
    THis is the lines I have in my dmesg (see below)
     
    In dovecot log , I see:
     
    Nov 25 04:26:47 auth-worker: Error: double free or corruption
    (fasttop)
     
    What do to about it ?
     
    Using lastest 2.3.4 version
     
    Thank you
     
    --------
     
    [132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00
    00 00 00 00
    00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5
    7a 7f 00
    00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16
    d6 ee 55
    [134031.969596] auth[27031]: segfault at 55e509612c30 ip
    000055e509612c30 sp 00007ffeb96dee48 error 15
    [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00
    00 00 00 00
    00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4
    ef 7f 00
    00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f
    09 e5 55
    [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip
    00007f393841013e sp 00007ffe4a16efc0 error 6 in
    libdovecot.so.0.0.0[7f3938363000+e2000]
    [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41
    57 41 89 cf
    41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3
    48 83 ec
    58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48
    31 c0 e8
    [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip
    00007f4376e32ffb sp 00007fff1cfdc000 error 6 in
    libdovecot.so.0.0.0[7f4376d86000+e2000]
    [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f
    1f 00 48 83
    ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d
    05 16 0b
    00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69
    dc f6 ff
    [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip
    00007ffec43fdcff sp 00007ffec3ba5000 error 6
    [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff
    48 8b 75 d0
    e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41
    54 49 89
    f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d
    35 69 c3
    [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip
    00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in
    libdovecot.so.0.0.0[7fb7c9ba2000+e2000]
    [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89
    cf 41 56 41
    89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c
    8b 4c 24
    40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00
    00 00 48
    [135796.171575] auth[11121]: segfault at 555f8645cc30 ip
    0000555f8645cc30 sp 00007ffcbb510868 error 15
    [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00
    00 00 00 00
    00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38
    b7 7f 00
    00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43
    86 5f 55
    [136710.562003] auth[17828]: segfault at 563443604c30 ip
    0000563443604c30 sp 00007ffc1aa8b498 error 15
    [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00
    00 00 00 00
    00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da
    d5 7f 00
    00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e
    43 34 56
    [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip
    000055b27bc63c30 sp 00007ffd5d5b9298 error 15
    [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00
    00 00 00 00
    00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17
    cd 7f 00
    00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4
    7b b2 55
    [138521.070485] auth[32481]: segfault at 556e05197c30 ip
    0000556e05197c30 sp 00007ffe87217c08 error 15
    [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00
    00 00 00 00
    00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20
    3c 7f 00
    00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17
    05 6e 55
    [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip
    00005570d3e46c30
    sp 00007ffc9118a5e8 error 15
    [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00
    00 00 00 00
    00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71
    11 7f 00
    00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2
    d3 70 55
    [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip
    00007ffe49ba7cff sp 00007ffe49317000 error 6
    [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff
    48 8b 75 d0
    e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41
    54 49 89
    f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d
    35 69 c3
    [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip
    00007fa68292a13e sp 00007ffcdc00efb0 error 6 in
    libdovecot.so.0.0.0[7fa68287d000+e2000]
    [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41
    57 41 89 cf
    41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3
    48 83 ec
    58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48
    31 c0 e8
    [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip
    00007f0e09ae259f sp 00007ffc90beb000 error 6 in
    libdovecot.so.0.0.0[7f0e09a35000+e2000]
    [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb
    2d f5 ff 66
    66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41
    55 49 89
    f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8
    6a fb ff
    [139747.003229] auth[9545]: segfault at 558b5e039c30 ip
    0000558b5e039c30
    sp 00007ffd43633008 error 15
    [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00
    00 00 00 00
    00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e
    75 7f 00
    00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01
    5e 8b 55
    [140640.900512] auth[17221]: segfault at 55e952d41c30 ip
    000055e952d41c30 sp 00007ffc36fa0908 error 15
    [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00
    00 00 00 00
    00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80
    b9 7f 00
    00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1
    52 e9 55
    [140777.927292] auth[18114]: segfault at 55bd72adec30 ip
    000055bd72adec30 sp 00007ffca394ba98 error 15
    [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00
    00 00 00 00
    00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67
    d7 7f 00
    00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab
    72 bd 55

 
Please see https://dovecot.org/bugreport.html
---
Aki

See https://dovecot.org/bugreport.html#coredumps

Without a backtrace it's not really possible to figure out where it's crashing.

On 28 Nov 2018, at 13.20, Joan Moreau jom@grosjo.net wrote:

Where to get that ?

On 2018-11-27 08:50, Aki Tuomi wrote:

...

It's still missing core dump (or bt full from it)

Aki

On 27.11.2018 8.39, Joan Moreau wrote:

...

Thank you Aki

here the requested data (below)

Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)

Bug appears mostly in auth process and index-worker

dovecot -n :

# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M

(...)

passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)

On 2018-11-25 08:08, Aki Tuomi wrote:

On 25 November 2018 at 06:29 Joan Moreau < jom@grosjo.net mailto:jom@grosjo.net> wrote:

Hi

THis is the lines I have in my dmesg (see below)

In dovecot log , I see:

Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop)

What do to about it ?

Using lastest 2.3.4 version

Thank you

---

[132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55 [134031.969596] auth[27031]: segfault at 55e509612c30 ip 000055e509612c30 sp 00007ffeb96dee48 error 15 [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55 [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip 00007f393841013e sp 00007ffe4a16efc0 error 6 in libdovecot.so.0.0.0[7f3938363000+e2000] [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip 00007f4376e32ffb sp 00007fff1cfdc000 error 6 in libdovecot.so.0.0.0[7f4376d86000+e2000] [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83 ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b 00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip 00007ffec43fdcff sp 00007ffec3ba5000 error 6 [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip 00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in libdovecot.so.0.0.0[7fb7c9ba2000+e2000] [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48 [135796.171575] auth[11121]: segfault at 555f8645cc30 ip 0000555f8645cc30 sp 00007ffcbb510868 error 15 [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55 [136710.562003] auth[17828]: segfault at 563443604c30 ip 0000563443604c30 sp 00007ffc1aa8b498 error 15 [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56 [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip 000055b27bc63c30 sp 00007ffd5d5b9298 error 15 [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17 cd 7f 00 00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4 7b b2 55 [138521.070485] auth[32481]: segfault at 556e05197c30 ip 0000556e05197c30 sp 00007ffe87217c08 error 15 [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20 3c 7f 00 00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17 05 6e 55 [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip 00005570d3e46c30 sp 00007ffc9118a5e8 error 15 [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71 11 7f 00 00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2 d3 70 55 [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip 00007ffe49ba7cff sp 00007ffe49317000 error 6 [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip 00007fa68292a13e sp 00007ffcdc00efb0 error 6 in libdovecot.so.0.0.0[7fa68287d000+e2000] [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip 00007f0e09ae259f sp 00007ffc90beb000 error 6 in libdovecot.so.0.0.0[7f0e09a35000+e2000] [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb 2d f5 ff 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8 6a fb ff [139747.003229] auth[9545]: segfault at 558b5e039c30 ip 0000558b5e039c30 sp 00007ffd43633008 error 15 [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e 75 7f 00 00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01 5e 8b 55 [140640.900512] auth[17221]: segfault at 55e952d41c30 ip 000055e952d41c30 sp 00007ffc36fa0908 error 15 [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80 b9 7f 00 00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1 52 e9 55 [140777.927292] auth[18114]: segfault at 55bd72adec30 ip 000055bd72adec30 sp 00007ffca394ba98 error 15 [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67 d7 7f 00 00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab 72 bd 55

Please see https://dovecot.org/bugreport.html https://dovecot.org/bugreport.html

Aki

finally managed to locate the dump

here the output:

# gdb /usr/libexec/dovecot/auth /var/lib/systemd/coredump/core.auth.0.3a33f56105e043de802a7dfcee265a07.28130.1543516118000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.

For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/auth...done. [New LWP 28130] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/auth wor'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 (gdb) bt full #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007f739c347672 in abort () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007f739c39f878 in __libc_message () from /usr/lib/libc.so.6 No symbol table info available. #3 0x00007f739c3a618a in malloc_printerr () from /usr/lib/libc.so.6 No symbol table info available. #4 0x00007f739c3a7b27 in _int_free () from /usr/lib/libc.so.6 No symbol table info available. #5 0x00007f739cc33585 in mysql_close (mysql=0x5636af7abdc0) at /usr/src/mariadb-10.3.11/libmariadb/libmariadb/mariadb_lib.c:1939 mysql = 0x5636af7abdc0 p = <optimized out> p = <optimized out> #6 0x00005636aef08f7c in driver_sqlpool_disconnect (_db=0x5636af7aaf30) at driver-sqlpool.c:590 conn__foreach_end = 0x5636af7ab570 db = 0x5636af7aaf30 conn = 0x5636af7ab560 #7 0x00005636aeefed65 in db_sql_unref (_conn=0x5636af7b0020) at db-sql.c:128 conn = 0x5636af7ae388 #8 0x00005636aeef7e15 in userdb_deinit (userdb=0x5636af7afff8) at userdb.c:191 idx = 0 __func__ = "userdb_deinit" #9 0x00005636aeede209 in auth_deinit (auth=0x5636af7afd58) at auth.c:335 passdb = <optimized out> userdb = 0x5636af7aff18 passdb = <optimized out> userdb = <optimized out> #10 auths_deinit () at auth.c:433 auth__foreach_end = 0x5636af7ae2f8 auth = 0x5636af7ae2f0 #11 0x00005636aeedcf65 in main_deinit () at main.c:270 l = <optimized out> l = <optimized out> l_end = <optimized out> #12 main (argc=<optimized out>, argv=<optimized out>) at main.c:401 c = <optimized out>

On 2018-11-29 18:07, Joan Moreau wrote:

Can't find any "core" files (updatedb ; locate "core"). Coredump are usually in /var/liv/systemd/coredump for other programs, but nothing for dovecot.

Looks like issue is in 'auth' and 'indexer-worker'. Where can be the coredump files ?

On 2018-11-28 18:13, Timo Sirainen wrote: See https://dovecot.org/bugreport.html#coredumps

Without a backtrace it's not really possible to figure out where it's crashing.

On 28 Nov 2018, at 13.20, Joan Moreau jom@grosjo.net wrote:

Where to get that ?

On 2018-11-27 08:50, Aki Tuomi wrote:

It's still missing core dump (or bt full from it)

Aki

On 27.11.2018 8.39, Joan Moreau wrote:

Thank you Aki

here the requested data (below)

Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)

Bug appears mostly in auth process and index-worker

dovecot -n :

# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M

(...)

passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)

On 2018-11-25 08:08, Aki Tuomi wrote:

On 25 November 2018 at 06:29 Joan Moreau < jom@grosjo.net> wrote:

Hi

THis is the lines I have in my dmesg (see below)

In dovecot log , I see:

Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop)

What do to about it ?

Using lastest 2.3.4 version

Thank you

---

[132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55 [134031.969596] auth[27031]: segfault at 55e509612c30 ip 000055e509612c30 sp 00007ffeb96dee48 error 15 [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55 [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip 00007f393841013e sp 00007ffe4a16efc0 error 6 in libdovecot.so.0.0.0[7f3938363000+e2000] [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip 00007f4376e32ffb sp 00007fff1cfdc000 error 6 in libdovecot.so.0.0.0[7f4376d86000+e2000] [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83 ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b 00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip 00007ffec43fdcff sp 00007ffec3ba5000 error 6 [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip 00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in libdovecot.so.0.0.0[7fb7c9ba2000+e2000] [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48 [135796.171575] auth[11121]: segfault at 555f8645cc30 ip 0000555f8645cc30 sp 00007ffcbb510868 error 15 [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55 [136710.562003] auth[17828]: segfault at 563443604c30 ip 0000563443604c30 sp 00007ffc1aa8b498 error 15 [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56 [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip 000055b27bc63c30 sp 00007ffd5d5b9298 error 15 [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17 cd 7f 00 00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4 7b b2 55 [138521.070485] auth[32481]: segfault at 556e05197c30 ip 0000556e05197c30 sp 00007ffe87217c08 error 15 [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20 3c 7f 00 00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17 05 6e 55 [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip 00005570d3e46c30 sp 00007ffc9118a5e8 error 15 [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71 11 7f 00 00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2 d3 70 55 [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip 00007ffe49ba7cff sp 00007ffe49317000 error 6 [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip 00007fa68292a13e sp 00007ffcdc00efb0 error 6 in libdovecot.so.0.0.0[7fa68287d000+e2000] [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip 00007f0e09ae259f sp 00007ffc90beb000 error 6 in libdovecot.so.0.0.0[7f0e09a35000+e2000] [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb 2d f5 ff 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8 6a fb ff [139747.003229] auth[9545]: segfault at 558b5e039c30 ip 0000558b5e039c30 sp 00007ffd43633008 error 15 [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e 75 7f 00 00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01 5e 8b 55 [140640.900512] auth[17221]: segfault at 55e952d41c30 ip 000055e952d41c30 sp 00007ffc36fa0908 error 15 [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80 b9 7f 00 00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1 52 e9 55 [140777.927292] auth[18114]: segfault at 55bd72adec30 ip 000055bd72adec30 sp 00007ffca394ba98 error 15 [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67 d7 7f 00 00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab 72 bd 55

Please see https://dovecot.org/bugreport.html

Aki

Hi

How to solve this ?

So many similar segfaults

Thank you

On 2018-11-30 06:11, Joan Moreau wrote:

ANother (very very long) example :

# gdb /usr/libexec/dovecot/indexer-worker core.indexer-worker.0.3a33f56105e043de802a7dfcee265a07.21017.1543533424000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.

For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/indexer-worker...done. [New LWP 21017] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/indexer-worker'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 173 { (gdb) bt full #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 lock_type_str = <optimized out> started = <optimized out> ret = <optimized out> __func__ = "file_lock_do" #1 0x00007f768b62b5b6 in file_wait_lock_error (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=<optimized out>, lock_r=0x7fff04501118, error_r=0x7fff045010b0) at file-lock.c:318 lock = <optimized out> ret = <optimized out> #2 0x00007f768b62b660 in file_wait_lock (fd=<optimized out>, path=<optimized out>, lock_type=lock_type@entry=0, lock_method=<optimized out>, timeout_secs=timeout_secs@entry=60, lock_r=lock_r@entry=0x7fff04501118) at file-lock.c:303 error = 0x564540376490 "" ret = <optimized out> #3 0x00007f768a976c87 in squat_trie_lock (trie=0x564540376490, lock_type=0, file_lock_r=0x7fff04501118, dotlock_r=0x7fff04501120) at squat-trie.c:294 ret = <optimized out> dotlock_r = 0x7fff04501120 file_lock_r = 0x7fff04501118 trie = 0x564540376490 ret = <optimized out> __func__ = "squat_trie_lock" lock_type = 0 ret = <optimized out> __func__ = "squat_trie_lock" #4 0x00007f768a978627 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1487 file_lock = 0x0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #5 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #6 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #7 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #8 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #9 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c850 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #10 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #11 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #12 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #13 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #14 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c7b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #15 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #16 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #17 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #18 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #19 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c710 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #20 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #21 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #22 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #23 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #24 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c670 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #25 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #26 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #27 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #28 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #29 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c5d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #30 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #31 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #32 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #33 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #34 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c530 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #35 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #36 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #37 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #38 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #39 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c490 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #40 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #41 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #42 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #43 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #44 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c3f0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #45 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #46 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #47 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #48 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #49 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c350 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #50 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 #51 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #52 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #53 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #54 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c2b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #55 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #56 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #57 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #58 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #59 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c210 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #60 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #61 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #62 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #63 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #64 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c170 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #65 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #66 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #67 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #68 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #69 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c0d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #70 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #71 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #72 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #73 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #74 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c030 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #75 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #76 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #77 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #78 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #79 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bf90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #80 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #81 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #82 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #83 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #84 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bef0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #85 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #86 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #87 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #88 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #89 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210be50 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #90 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #91 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #92 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #93 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #94 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bdb0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #95 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #96 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #97 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #98 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #99 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bd10 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #100 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #101 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #102 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #103 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #104 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bc70 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #105 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #106 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #107 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #108 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #109 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bbd0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #110 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #111 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #112 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #113 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #114 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bb30 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #115 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #116 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #117 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #118 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #119 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210ba90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #120 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #121 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #122 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #123 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #124 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210b9f0 dotlock = 0x0 changed = <optimized out>

(continues for ages)

On 2018-11-29 19:36, Joan Moreau wrote:

finally managed to locate the dump

here the output:

# gdb /usr/libexec/dovecot/auth /var/lib/systemd/coredump/core.auth.0.3a33f56105e043de802a7dfcee265a07.28130.1543516118000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.

For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/auth...done. [New LWP 28130] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/auth wor'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 (gdb) bt full #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007f739c347672 in abort () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007f739c39f878 in __libc_message () from /usr/lib/libc.so.6 No symbol table info available. #3 0x00007f739c3a618a in malloc_printerr () from /usr/lib/libc.so.6 No symbol table info available. #4 0x00007f739c3a7b27 in _int_free () from /usr/lib/libc.so.6 No symbol table info available. #5 0x00007f739cc33585 in mysql_close (mysql=0x5636af7abdc0) at /usr/src/mariadb-10.3.11/libmariadb/libmariadb/mariadb_lib.c:1939 mysql = 0x5636af7abdc0 p = <optimized out> p = <optimized out> #6 0x00005636aef08f7c in driver_sqlpool_disconnect (_db=0x5636af7aaf30) at driver-sqlpool.c:590 conn__foreach_end = 0x5636af7ab570 db = 0x5636af7aaf30 conn = 0x5636af7ab560 #7 0x00005636aeefed65 in db_sql_unref (_conn=0x5636af7b0020) at db-sql.c:128 conn = 0x5636af7ae388 #8 0x00005636aeef7e15 in userdb_deinit (userdb=0x5636af7afff8) at userdb.c:191 idx = 0 __func__ = "userdb_deinit" #9 0x00005636aeede209 in auth_deinit (auth=0x5636af7afd58) at auth.c:335 passdb = <optimized out> userdb = 0x5636af7aff18 passdb = <optimized out> userdb = <optimized out> #10 auths_deinit () at auth.c:433 auth__foreach_end = 0x5636af7ae2f8 auth = 0x5636af7ae2f0 #11 0x00005636aeedcf65 in main_deinit () at main.c:270 l = <optimized out> l = <optimized out> l_end = <optimized out> #12 main (argc=<optimized out>, argv=<optimized out>) at main.c:401 c = <optimized out>

On 2018-11-29 18:07, Joan Moreau wrote:

Can't find any "core" files (updatedb ; locate "core"). Coredump are usually in /var/liv/systemd/coredump for other programs, but nothing for dovecot.

Looks like issue is in 'auth' and 'indexer-worker'. Where can be the coredump files ?

On 2018-11-28 18:13, Timo Sirainen wrote: See https://dovecot.org/bugreport.html#coredumps

Without a backtrace it's not really possible to figure out where it's crashing.

On 28 Nov 2018, at 13.20, Joan Moreau jom@grosjo.net wrote:

Where to get that ?

On 2018-11-27 08:50, Aki Tuomi wrote:

It's still missing core dump (or bt full from it)

Aki

On 27.11.2018 8.39, Joan Moreau wrote:

Thank you Aki

here the requested data (below)

Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)

Bug appears mostly in auth process and index-worker

dovecot -n :

# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M

(...)

passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)

On 2018-11-25 08:08, Aki Tuomi wrote:

On 25 November 2018 at 06:29 Joan Moreau < jom@grosjo.net> wrote:

Hi

THis is the lines I have in my dmesg (see below)

In dovecot log , I see:

Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop)

What do to about it ?

Using lastest 2.3.4 version

Thank you

---

[132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55 [134031.969596] auth[27031]: segfault at 55e509612c30 ip 000055e509612c30 sp 00007ffeb96dee48 error 15 [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55 [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip 00007f393841013e sp 00007ffe4a16efc0 error 6 in libdovecot.so.0.0.0[7f3938363000+e2000] [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip 00007f4376e32ffb sp 00007fff1cfdc000 error 6 in libdovecot.so.0.0.0[7f4376d86000+e2000] [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83 ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b 00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip 00007ffec43fdcff sp 00007ffec3ba5000 error 6 [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip 00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in libdovecot.so.0.0.0[7fb7c9ba2000+e2000] [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48 [135796.171575] auth[11121]: segfault at 555f8645cc30 ip 0000555f8645cc30 sp 00007ffcbb510868 error 15 [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55 [136710.562003] auth[17828]: segfault at 563443604c30 ip 0000563443604c30 sp 00007ffc1aa8b498 error 15 [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56 [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip 000055b27bc63c30 sp 00007ffd5d5b9298 error 15 [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17 cd 7f 00 00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4 7b b2 55 [138521.070485] auth[32481]: segfault at 556e05197c30 ip 0000556e05197c30 sp 00007ffe87217c08 error 15 [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20 3c 7f 00 00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17 05 6e 55 [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip 00005570d3e46c30 sp 00007ffc9118a5e8 error 15 [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71 11 7f 00 00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2 d3 70 55 [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip 00007ffe49ba7cff sp 00007ffe49317000 error 6 [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip 00007fa68292a13e sp 00007ffcdc00efb0 error 6 in libdovecot.so.0.0.0[7fa68287d000+e2000] [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip 00007f0e09ae259f sp 00007ffc90beb000 error 6 in libdovecot.so.0.0.0[7f0e09a35000+e2000] [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb 2d f5 ff 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8 6a fb ff [139747.003229] auth[9545]: segfault at 558b5e039c30 ip 0000558b5e039c30 sp 00007ffd43633008 error 15 [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e 75 7f 00 00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01 5e 8b 55 [140640.900512] auth[17221]: segfault at 55e952d41c30 ip 000055e952d41c30 sp 00007ffc36fa0908 error 15 [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80 b9 7f 00 00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1 52 e9 55 [140777.927292] auth[18114]: segfault at 55bd72adec30 ip 000055bd72adec30 sp 00007ffca394ba98 error 15 [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67 d7 7f 00 00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab 72 bd 55

Please see https://dovecot.org/bugreport.html

Aki

On 04 December 2018 at 16:46 Joan Moreau via dovecot dovecot@dovecot.org wrote:

Hi

How to solve this ?

So many similar segfaults

Thank you

On 2018-11-30 06:11, Joan Moreau wrote:

...

ANother (very very long) example :

# gdb /usr/libexec/dovecot/indexer-worker core.indexer-worker.0.3a33f56105e043de802a7dfcee265a07.21017.1543533424000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.

For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/indexer-worker...done. [New LWP 21017] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/indexer-worker'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 173 { (gdb) bt full #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 lock_type_str = <optimized out> started = <optimized out> ret = <optimized out> __func__ = "file_lock_do" #1 0x00007f768b62b5b6 in file_wait_lock_error (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=<optimized out>, lock_r=0x7fff04501118, error_r=0x7fff045010b0) at file-lock.c:318 lock = <optimized out> ret = <optimized out> #2 0x00007f768b62b660 in file_wait_lock (fd=<optimized out>, path=<optimized out>, lock_type=lock_type@entry=0, lock_method=<optimized out>, timeout_secs=timeout_secs@entry=60, lock_r=lock_r@entry=0x7fff04501118) at file-lock.c:303 error = 0x564540376490 "" ret = <optimized out> #3 0x00007f768a976c87 in squat_trie_lock (trie=0x564540376490, lock_type=0, file_lock_r=0x7fff04501118, dotlock_r=0x7fff04501120) at squat-trie.c:294 ret = <optimized out> dotlock_r = 0x7fff04501120 file_lock_r = 0x7fff04501118 trie = 0x564540376490 ret = <optimized out> __func__ = "squat_trie_lock" lock_type = 0 ret = <optimized out> __func__ = "squat_trie_lock" #4 0x00007f768a978627 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1487 file_lock = 0x0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #5 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #6 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #7 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #8 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #9 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c850 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #10 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #11 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #12 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #13 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #14 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c7b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #15 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #16 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #17 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #18 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #19 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c710 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #20 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #21 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #22 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #23 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #24 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c670 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #25 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #26 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #27 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #28 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #29 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c5d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #30 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #31 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #32 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #33 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #34 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c530 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #35 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #36 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #37 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #38 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #39 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c490 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #40 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #41 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #42 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #43 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #44 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c3f0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #45 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #46 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #47 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #48 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #49 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c350 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #50 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 #51 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #52 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #53 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #54 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c2b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #55 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #56 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #57 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #58 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #59 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c210 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #60 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #61 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #62 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #63 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #64 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c170 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #65 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #66 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #67 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #68 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #69 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c0d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #70 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #71 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #72 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #73 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #74 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c030 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #75 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #76 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #77 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #78 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #79 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bf90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #80 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #81 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #82 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #83 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #84 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bef0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #85 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #86 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #87 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #88 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #89 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210be50 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #90 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #91 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #92 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #93 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #94 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bdb0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #95 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #96 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #97 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #98 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #99 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bd10 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #100 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #101 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #102 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #103 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #104 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bc70 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #105 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #106 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #107 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #108 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #109 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bbd0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #110 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #111 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #112 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #113 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #114 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bb30 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #115 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #116 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #117 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #118 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #119 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210ba90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #120 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #121 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #122 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #123 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #124 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210b9f0 dotlock = 0x0 changed = <optimized out>

(continues for ages)

On 2018-11-29 19:36, Joan Moreau wrote:

...

finally managed to locate the dump

here the output:

# gdb /usr/libexec/dovecot/auth /var/lib/systemd/coredump/core.auth.0.3a33f56105e043de802a7dfcee265a07.28130.1543516118000000 GNU gdb (GDB) 8.2 (gdb) bt full #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007f739c347672 in abort () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007f739c39f878 in __libc_message () from /usr/lib/libc.so.6 No symbol table info available. #3 0x00007f739c3a618a in malloc_printerr () from /usr/lib/libc.so.6 No symbol table info available. #4 0x00007f739c3a7b27 in _int_free () from /usr/lib/libc.so.6 No symbol table info available. #5 0x00007f739cc33585 in mysql_close (mysql=0x5636af7abdc0) at /usr/src/mariadb-10.3.11/libmariadb/libmariadb/mariadb_lib.c:1939 mysql = 0x5636af7abdc0 p = <optimized out> p = <optimized out> #6 0x00005636aef08f7c in driver_sqlpool_disconnect (_db=0x5636af7aaf30) at driver-sqlpool.c:590 conn__foreach_end = 0x5636af7ab570 db = 0x5636af7aaf30 conn = 0x5636af7ab560 #7 0x00005636aeefed65 in db_sql_unref (_conn=0x5636af7b0020) at db-sql.c:128 conn = 0x5636af7ae388 #8 0x00005636aeef7e15 in userdb_deinit (userdb=0x5636af7afff8) at userdb.c:191 idx = 0 __func__ = "userdb_deinit" #9 0x00005636aeede209 in auth_deinit (auth=0x5636af7afd58) at auth.c:335 passdb = <optimized out> userdb = 0x5636af7aff18 passdb = <optimized out> userdb = <optimized out> #10 auths_deinit () at auth.c:433 auth__foreach_end = 0x5636af7ae2f8 auth = 0x5636af7ae2f0 #11 0x00005636aeedcf65 in main_deinit () at main.c:270 l = <optimized out> l = <optimized out> l_end = <optimized out> #12 main (argc=<optimized out>, argv=<optimized out>) at main.c:401 c = <optimized out>

On 27.11.2018 8.39, Joan Moreau wrote:

Thank you Aki

here the requested data (below)

Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)

Bug appears mostly in auth process and index-worker

dovecot -n :

# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M

(...)

passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)

fts squat was already deprecated in 2.2, and it's considered obsolete in 2.3. We hopefully get around removing it from 2.4, but you should move away from it to Solr.

The MySQL crash has been fixed in master, see https://github.com/dovecot/core/commit/3c5101ffdd2a8115e03ed7180d53578765dea...

Aki

We don't consider it as "very early beta". We consider it production ready. It is bit more work to set up though.

Aki

On 04 December 2018 at 17:16 Joan Moreau jom@grosjo.net wrote:

Thanks for mySql

For Squat vs Solr, Solr does not reach Squat by very far in terms of results : If I setup Solr, and search (via the search in Roundbube or Evolution) for a keyword or part of the keyword, the results are complete non sense. The diference between "search in full body" and "search in fields" does not even work.

Solr with Dovecot seems very early beta isnt'it ?

On 2018-12-04 15:57, Aki Tuomi wrote:

...

On 04 December 2018 at 16:46 Joan Moreau via dovecot dovecot@dovecot.org wrote:

Hi

How to solve this ?

So many similar segfaults

Thank you

On 2018-11-30 06:11, Joan Moreau wrote:

ANother (very very long) example :

# gdb /usr/libexec/dovecot/indexer-worker core.indexer-worker.0.3a33f56105e043de802a7dfcee265a07.21017.1543533424000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.

For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/indexer-worker...done. [New LWP 21017] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/indexer-worker'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 173 { (gdb) bt full #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 lock_type_str = <optimized out> started = <optimized out> ret = <optimized out> __func__ = "file_lock_do" #1 0x00007f768b62b5b6 in file_wait_lock_error (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=<optimized out>, lock_r=0x7fff04501118, error_r=0x7fff045010b0) at file-lock.c:318 lock = <optimized out> ret = <optimized out> #2 0x00007f768b62b660 in file_wait_lock (fd=<optimized out>, path=<optimized out>, lock_type=lock_type@entry=0, lock_method=<optimized out>, timeout_secs=timeout_secs@entry=60, lock_r=lock_r@entry=0x7fff04501118) at file-lock.c:303 error = 0x564540376490 "" ret = <optimized out> #3 0x00007f768a976c87 in squat_trie_lock (trie=0x564540376490, lock_type=0, file_lock_r=0x7fff04501118, dotlock_r=0x7fff04501120) at squat-trie.c:294 ret = <optimized out> dotlock_r = 0x7fff04501120 file_lock_r = 0x7fff04501118 trie = 0x564540376490 ret = <optimized out> __func__ = "squat_trie_lock" lock_type = 0 ret = <optimized out> __func__ = "squat_trie_lock" #4 0x00007f768a978627 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1487 file_lock = 0x0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #5 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #6 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #7 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #8 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #9 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c850 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #10 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #11 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #12 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #13 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #14 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c7b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #15 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #16 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #17 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #18 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #19 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c710 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #20 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #21 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #22 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #23 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #24 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c670 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #25 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #26 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #27 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #28 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #29 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c5d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #30 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #31 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #32 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #33 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #34 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c530 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #35 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #36 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #37 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #38 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #39 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c490 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #40 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #41 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #42 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #43 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #44 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c3f0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #45 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #46 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #47 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #48 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #49 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c350 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #50 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 #51 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #52 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #53 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #54 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c2b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #55 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #56 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #57 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #58 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #59 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c210 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #60 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #61 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #62 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #63 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #64 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c170 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #65 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #66 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #67 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #68 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #69 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c0d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #70 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #71 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #72 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #73 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #74 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c030 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #75 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #76 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #77 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #78 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #79 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bf90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #80 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #81 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #82 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #83 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #84 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bef0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #85 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #86 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #87 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #88 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #89 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210be50 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #90 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #91 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #92 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #93 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #94 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bdb0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #95 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #96 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #97 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #98 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #99 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bd10 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #100 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #101 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #102 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #103 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #104 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bc70 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #105 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #106 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #107 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #108 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #109 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bbd0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #110 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #111 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #112 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #113 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #114 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bb30 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #115 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #116 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #117 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #118 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #119 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210ba90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #120 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #121 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #122 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #123 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #124 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210b9f0 dotlock = 0x0 changed = <optimized out>

(continues for ages)

On 2018-11-29 19:36, Joan Moreau wrote:

finally managed to locate the dump

here the output:

# gdb /usr/libexec/dovecot/auth /var/lib/systemd/coredump/core.auth.0.3a33f56105e043de802a7dfcee265a07.28130.1543516118000000 GNU gdb (GDB) 8.2 (gdb) bt full #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007f739c347672 in abort () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007f739c39f878 in __libc_message () from /usr/lib/libc.so.6 No symbol table info available. #3 0x00007f739c3a618a in malloc_printerr () from /usr/lib/libc.so.6 No symbol table info available. #4 0x00007f739c3a7b27 in _int_free () from /usr/lib/libc.so.6 No symbol table info available. #5 0x00007f739cc33585 in mysql_close (mysql=0x5636af7abdc0) at /usr/src/mariadb-10.3.11/libmariadb/libmariadb/mariadb_lib.c:1939 mysql = 0x5636af7abdc0 p = <optimized out> p = <optimized out> #6 0x00005636aef08f7c in driver_sqlpool_disconnect (_db=0x5636af7aaf30) at driver-sqlpool.c:590 conn__foreach_end = 0x5636af7ab570 db = 0x5636af7aaf30 conn = 0x5636af7ab560 #7 0x00005636aeefed65 in db_sql_unref (_conn=0x5636af7b0020) at db-sql.c:128 conn = 0x5636af7ae388 #8 0x00005636aeef7e15 in userdb_deinit (userdb=0x5636af7afff8) at userdb.c:191 idx = 0 __func__ = "userdb_deinit" #9 0x00005636aeede209 in auth_deinit (auth=0x5636af7afd58) at auth.c:335 passdb = <optimized out> userdb = 0x5636af7aff18 passdb = <optimized out> userdb = <optimized out> #10 auths_deinit () at auth.c:433 auth__foreach_end = 0x5636af7ae2f8 auth = 0x5636af7ae2f0 #11 0x00005636aeedcf65 in main_deinit () at main.c:270 l = <optimized out> l = <optimized out> l_end = <optimized out> #12 main (argc=<optimized out>, argv=<optimized out>) at main.c:401 c = <optimized out>

On 27.11.2018 8.39, Joan Moreau wrote:

Thank you Aki

here the requested data (below)

Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)

Bug appears mostly in auth process and index-worker

dovecot -n :

# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M

(...)

passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values) fts squat was already deprecated in 2.2, and it's considered obsolete in 2.3. We hopefully get around removing it from 2.4, but you should move away from it to Solr.

The MySQL crash has been fixed in master, see https://github.com/dovecot/core/commit/3c5101ffdd2a8115e03ed7180d53578765dea...

Aki

On December 4, 2018 at 8:18 AM Aki Tuomi aki.tuomi@open-xchange.com wrote:

We don't consider it as "very early beta". We consider it production ready. It is bit more work to set up though.

FWIW, Squat has been deprecated since 2.0, so none of this should come as a surprise.

https://wiki.dovecot.org/Plugins/FTS

Aki

...

On 04 December 2018 at 17:16 Joan Moreau jom@grosjo.net wrote:

Thanks for mySql

For Squat vs Solr, Solr does not reach Squat by very far in terms of results : If I setup Solr, and search (via the search in Roundbube or Evolution) for a keyword or part of the keyword, the results are complete non sense. The diference between "search in full body" and "search in fields" does not even work.

Solr with Dovecot seems very early beta isnt'it ?

On 2018-12-04 15:57, Aki Tuomi wrote:

...

On 04 December 2018 at 16:46 Joan Moreau via dovecot dovecot@dovecot.org wrote:

Hi

How to solve this ?

So many similar segfaults

Thank you

On 2018-11-30 06:11, Joan Moreau wrote:

ANother (very very long) example :

# gdb /usr/libexec/dovecot/indexer-worker core.indexer-worker.0.3a33f56105e043de802a7dfcee265a07.21017.1543533424000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.

For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/indexer-worker...done. [New LWP 21017] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/indexer-worker'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 173 { (gdb) bt full #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 lock_type_str = <optimized out> started = <optimized out> ret = <optimized out> __func__ = "file_lock_do" #1 0x00007f768b62b5b6 in file_wait_lock_error (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=<optimized out>, lock_r=0x7fff04501118, error_r=0x7fff045010b0) at file-lock.c:318 lock = <optimized out> ret = <optimized out> #2 0x00007f768b62b660 in file_wait_lock (fd=<optimized out>, path=<optimized out>, lock_type=lock_type@entry=0, lock_method=<optimized out>, timeout_secs=timeout_secs@entry=60, lock_r=lock_r@entry=0x7fff04501118) at file-lock.c:303 error = 0x564540376490 "" ret = <optimized out> #3 0x00007f768a976c87 in squat_trie_lock (trie=0x564540376490, lock_type=0, file_lock_r=0x7fff04501118, dotlock_r=0x7fff04501120) at squat-trie.c:294 ret = <optimized out> dotlock_r = 0x7fff04501120 file_lock_r = 0x7fff04501118 trie = 0x564540376490 ret = <optimized out> __func__ = "squat_trie_lock" lock_type = 0 ret = <optimized out> __func__ = "squat_trie_lock" #4 0x00007f768a978627 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1487 file_lock = 0x0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #5 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #6 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #7 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #8 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #9 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c850 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #10 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #11 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #12 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #13 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #14 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c7b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #15 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #16 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #17 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #18 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #19 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c710 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #20 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #21 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #22 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #23 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #24 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c670 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #25 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #26 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #27 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #28 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #29 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c5d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #30 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #31 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #32 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #33 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #34 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c530 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #35 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #36 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #37 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #38 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #39 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c490 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #40 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #41 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #42 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #43 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #44 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c3f0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #45 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #46 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #47 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #48 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #49 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c350 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #50 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 #51 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #52 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #53 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #54 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c2b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #55 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #56 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #57 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #58 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #59 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c210 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #60 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #61 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #62 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #63 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #64 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c170 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #65 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #66 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #67 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #68 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #69 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c0d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #70 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #71 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #72 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #73 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #74 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c030 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #75 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #76 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #77 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #78 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #79 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bf90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #80 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #81 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #82 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #83 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #84 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bef0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #85 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #86 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #87 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #88 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #89 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210be50 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #90 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #91 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #92 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #93 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #94 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bdb0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #95 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #96 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #97 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #98 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #99 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bd10 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #100 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #101 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #102 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #103 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #104 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bc70 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #105 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #106 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #107 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #108 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #109 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bbd0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #110 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #111 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #112 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #113 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #114 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bb30 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #115 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #116 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #117 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #118 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #119 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210ba90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #120 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #121 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #122 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #123 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #124 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210b9f0 dotlock = 0x0 changed = <optimized out>

(continues for ages)

On 2018-11-29 19:36, Joan Moreau wrote:

finally managed to locate the dump

here the output:

# gdb /usr/libexec/dovecot/auth /var/lib/systemd/coredump/core.auth.0.3a33f56105e043de802a7dfcee265a07.28130.1543516118000000 GNU gdb (GDB) 8.2 (gdb) bt full #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007f739c347672 in abort () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007f739c39f878 in __libc_message () from /usr/lib/libc.so.6 No symbol table info available. #3 0x00007f739c3a618a in malloc_printerr () from /usr/lib/libc.so.6 No symbol table info available. #4 0x00007f739c3a7b27 in _int_free () from /usr/lib/libc.so.6 No symbol table info available. #5 0x00007f739cc33585 in mysql_close (mysql=0x5636af7abdc0) at /usr/src/mariadb-10.3.11/libmariadb/libmariadb/mariadb_lib.c:1939 mysql = 0x5636af7abdc0 p = <optimized out> p = <optimized out> #6 0x00005636aef08f7c in driver_sqlpool_disconnect (_db=0x5636af7aaf30) at driver-sqlpool.c:590 conn__foreach_end = 0x5636af7ab570 db = 0x5636af7aaf30 conn = 0x5636af7ab560 #7 0x00005636aeefed65 in db_sql_unref (_conn=0x5636af7b0020) at db-sql.c:128 conn = 0x5636af7ae388 #8 0x00005636aeef7e15 in userdb_deinit (userdb=0x5636af7afff8) at userdb.c:191 idx = 0 __func__ = "userdb_deinit" #9 0x00005636aeede209 in auth_deinit (auth=0x5636af7afd58) at auth.c:335 passdb = <optimized out> userdb = 0x5636af7aff18 passdb = <optimized out> userdb = <optimized out> #10 auths_deinit () at auth.c:433 auth__foreach_end = 0x5636af7ae2f8 auth = 0x5636af7ae2f0 #11 0x00005636aeedcf65 in main_deinit () at main.c:270 l = <optimized out> l = <optimized out> l_end = <optimized out> #12 main (argc=<optimized out>, argv=<optimized out>) at main.c:401 c = <optimized out>

On 27.11.2018 8.39, Joan Moreau wrote:

Thank you Aki

here the requested data (below)

Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)

Bug appears mostly in auth process and index-worker

dovecot -n :

# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M

(...)

passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values) fts squat was already deprecated in 2.2, and it's considered obsolete in 2.3. We hopefully get around removing it from 2.4, but you should move away from it to Solr.

The MySQL crash has been fixed in master, see https://github.com/dovecot/core/commit/3c5101ffdd2a8115e03ed7180d53578765dea...

Aki

Hi

I am giving Solr another try.

Using Solr 7.5 (from Archlinux AUR), I do not see anymore the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr)

Maybe some things have moved ?

Thank you so much

also, the value "break-imap-search" is not supported in fts_solr parameter, contrary to the wiki

and not sure then how to enable full text search in BODY

On 2018-12-04 19:40, Joan Moreau via dovecot wrote:

In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command :

sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance

before updating the schema.xml .

Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users

Additionaly, the url is http://(solr_ [1]server):8983/solr/dovecot/ (error in wiki)

On 2018-12-04 19:01, Joan Moreau via dovecot wrote:

...

Hi

I am giving Solr another try.

Using Solr 7.5 (from Archlinux AUR), I do not see anymore the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr)

Maybe some things have moved ?

Thank you so much

Links:

[1] http://(solr

Well, "break-imap-search" option prevent dovecot from starting.

I used solr on one server since yesterday.

First feedback is that

• BODY search is not working

• FTS is not really working in headers : For instance, if you search "kliinik" but you type "kliini" (missing one k), the search does not return anything , whereas it shall return more results (as the keyword is smaller) that the intended search

On 2018-12-05 07:55, Aki Tuomi wrote:

Full text search gets enabled automatically, although your probably want to set fts_enforced=yes to make sure it's used for all kinds of things automatically. break-imap-search is supported, but does nothing.

Updated the wiki a bit.

Aki

On 4.12.2018 20.50, Joan Moreau via dovecot wrote:

also, the value "break-imap-search" is not supported in fts_solr parameter, contrary to the wiki

and not sure then how to enable full text search in BODY

On 2018-12-04 19:40, Joan Moreau via dovecot wrote:

In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command :

sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance

before updating the schema.xml .

Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users

Additionaly, the url is http://(solr_ [1]server):8983/solr/dovecot/ (error in wiki)

On 2018-12-04 19:01, Joan Moreau via dovecot wrote:

Hi

I am giving Solr another try.

Using Solr 7.5 (from Archlinux AUR), I do not see anymore the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr)

Maybe some things have moved ?

Thank you so much

Links:

[1] http://(solr

It seems we forgot to document that "break-imap-search" was dropped in v2.3. That has now been updated.

Also Solr does not support prefix/substring search unless you configure solr to support it.

Aki

On 5.12.2018 11.50, Joan Moreau wrote:

Well, "break-imap-search" option prevent dovecot from starting.

I used solr on one server since yesterday.

First feedback is that

• BODY search is not working

• FTS is not really working in headers : For instance, if you search "kliinik" but you type "kliini" (missing one k), the search does not return anything , whereas it shall return more results (as the keyword is smaller) that the intended search

 

On 2018-12-05 07:55, Aki Tuomi wrote:

...

Full text search gets enabled automatically, although your probably want to set fts_enforced=yes to make sure it's used for all kinds of things automatically. break-imap-search is supported,  but does nothing.

Updated the wiki a bit.

Aki

On 4.12.2018 20.50, Joan Moreau via dovecot wrote:

...

also, the value "break-imap-search" is not supported in fts_solr parameter, contrary to the wiki

and not sure then how to enable full text search in BODY

 

On 2018-12-04 19:40, Joan Moreau via dovecot wrote:

In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it
would nice to stipulate to the reader  to type the command :

sudo -u solr /opt/solr/bin/solr create -c dovecot # to create
the dovecot instance

before updating the schema.xml .

Also,  schema.xml is in /opt/solr/server/solr/dovecot/conf for
archlinux users

Additionaly, the url is http://(solr_
<http://(solr>server):8983/solr/dovecot/ (error in wiki)


 


On 2018-12-04 19:01, Joan Moreau via dovecot wrote:

    Hi

    I am giving Solr another try.

    Using Solr 7.5 (from Archlinux AUR), I do not see anymore
    the"solr/conf/schema.xml" neither the "managed-schema"
    folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr) 

    Maybe some things have moved ?

    Thank you so much

Squat has not been maintained since 2.2.something, and shall not be maintained. Sorry. If you want, you can become maintainer for squat plugin.

Aki

On 5.12.2018 13.57, Joan Moreau wrote:

THen the Squat shall be maintained until the SOlr plugin is upgraded, as Squat does resolve the problem (fts, partial search, etc...)

 

On 2018-12-05 12:20, Aki Tuomi wrote:

...

It seems we forgot to document that "break-imap-search" was dropped in v2.3. That has now been updated.

Also Solr does not support prefix/substring search unless you configure solr to support it.

Aki

On 5.12.2018 11.50, Joan Moreau wrote:

...

Well, "break-imap-search" option prevent dovecot from starting.

I used solr on one server since yesterday.

First feedback is that

• BODY search is not working

• FTS is not really working in headers : For instance, if you search "kliinik" but you type "kliini" (missing one k), the search does not return anything , whereas it shall return more results (as the keyword is smaller) that the intended search

 

On 2018-12-05 07:55, Aki Tuomi wrote:

Full text search gets enabled automatically, although your
probably want to set fts_enforced=yes to make sure it's used for
all kinds of things automatically. break-imap-search is
supported,  but does nothing.

Updated the wiki a bit.

Aki

On 4.12.2018 20.50, Joan Moreau via dovecot wrote:

    also, the value "break-imap-search" is not supported in
    fts_solr parameter, contrary to the wiki

    and not sure then how to enable full text search in BODY

     


    On 2018-12-04 19:40, Joan Moreau via dovecot wrote:

        In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr
        ), it would nice to stipulate to the reader  to type the
        command :

        sudo -u solr /opt/solr/bin/solr create -c dovecot # to
        create the dovecot instance

        before updating the schema.xml .

        Also,  schema.xml is in
        /opt/solr/server/solr/dovecot/conf for archlinux users

        Additionaly, the url is http://(solr_
        <http://(solr>server):8983/solr/dovecot/ (error in wiki)


         


        On 2018-12-04 19:01, Joan Moreau via dovecot wrote:

            Hi

            I am giving Solr another try.

            Using Solr 7.5 (from Archlinux AUR), I do not see
            anymore the"solr/conf/schema.xml" neither the
            "managed-schema" folder. (as
            per https://wiki.dovecot.org/Plugins/FTS/Solr) 

            Maybe some things have moved ?

            Thank you so much

On Wed Dec 05 2018 07:35:39 GMT-0500 (Eastern Standard Time), Joan Moreau via dovecot dovecot@dovecot.org wrote:

Why making squat obsolete ? It is simple and straightforward

Because no one has stepped up to maintain it?

Hi Daniel,

THere is no need of all this, just the command (on Solr 7.5) "create -c dovecot " is enough

The chema.xml provided on the wiki basically does not work on 7.5

Here the latest one I am working on , but nothing works properly (bad search results, errors in ftp_solr, etc..)

<?xml version="1.0" encoding="UTF-8"?>

<schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="long" class="solr.LongPointField" positionIncrementGap="0" /> <fieldType name="text" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="_version_" type="long" indexed="true" stored="true"/> <field name="bcc" type="text" indexed="true" stored="false"/> <field name="body" type="text" indexed="true" stored="false"/> <field name="box" type="string" indexed="false" required="true" stored="true"/> <field name="hdr" type="text" indexed="true" stored="false"/> <field name="cc" type="text" indexed="true" stored="false"/> <field name="from" type="text" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="text" indexed="true" stored="false"/> <field name="to" type="text" indexed="true" stored="false"/> <field name="uid" type="long" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </fields> </schema>

On 2018-12-10 21:17, Daniel Miller via dovecot wrote:

On 12/4/2018 10:40 AM, Joan Moreau via dovecot wrote:

...

In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command :

sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance

before updating the schema.xml .

Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users

Additionaly, the url is http://(solr_ [1]server):8983/solr/dovecot/ (error in wiki)

After installing Solr, wherever the installation sets up there should a folder similar to:

<your prefix>/solr/server/solr/configsets

If you look there, you'll probably see folders like '_default' and 'sample_techproducts_configs'. I haven't played with the 'techproducts' sample. Copy the '_default' folder, with all its contents, to a 'dovecot' folder. In the new dovecot folder, replace the 'managed-schema' file with the file from the Dovecot Wiki

https://wiki.dovecot.org/Plugins/FTS/Solr?action=AttachFile&do=view&target=solr-7.x-schema.xml

after that, you should be able to run 'solr /opt/solr/bin/solr create -c dovecot' to create the instance. If things still don't work let us know.

The schema is one I've tweaked and updated during my own migrations since Solr 3.3. It's possible there's something else in my config that needs documenting - but having experienced Solr search against my mailstore I never want to be without it.

Daniel

Links:

[1] http://(solr

I shared the errors already so many times (check this mailinling for "solr" in teh title)

Contrary to what you say, with SOlr 7.5 and Dovecot git, I had to remove the "managed-schema" to make solr respond a bit properly. It relies on schema.xml

In order to create the instance, no, it copies the default config in the dovecot instance.

Bottom line, the end results is that the results are either "error : invalid response" (mostly on Android email client) or no results or innacurate results (using Roundcube)

On 2018-12-10 23:30, Daniel Miller via dovecot wrote:

The one on the Wiki is mine...which I'm using now. So it certainly does work - but perhaps there's a setting you have differently from me.

Performing a "create -c dovecot" creates a Solr instance *named* dovecot - that does *not* initialize it with the necessary schema. You need to specify "-d dovecot", with a dovecot configset already setup, to do that.

The other choice is to create the instance as you show, ensure Solr is stopped, delete the "<prefix>/solr/dovecot/data" folder, and copy the managed-schema file to "<prefix>/solr/dovecot/conf". Again, the filename saved in the /conf folder needs to be "managed-schema" - no ".xml" suffix.

If that doesn't work for you - please share the errors.

Daniel

On 12/10/2018 11:40 AM, Joan Moreau wrote:

Hi Daniel,

THere is no need of all this, just the command (on Solr 7.5) "create -c dovecot " is enough

The chema.xml provided on the wiki basically does not work on 7.5

Here the latest one I am working on , but nothing works properly (bad search results, errors in ftp_solr, etc..)

<?xml version="1.0" encoding="UTF-8"?>

<schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="long" class="solr.LongPointField" positionIncrementGap="0" /> <fieldType name="text" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="_version_" type="long" indexed="true" stored="true"/> <field name="bcc" type="text" indexed="true" stored="false"/> <field name="body" type="text" indexed="true" stored="false"/> <field name="box" type="string" indexed="false" required="true" stored="true"/> <field name="hdr" type="text" indexed="true" stored="false"/> <field name="cc" type="text" indexed="true" stored="false"/> <field name="from" type="text" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="text" indexed="true" stored="false"/> <field name="to" type="text" indexed="true" stored="false"/> <field name="uid" type="long" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </fields> </schema>

On 2018-12-10 21:17, Daniel Miller via dovecot wrote: On 12/4/2018 10:40 AM, Joan Moreau via dovecot wrote:

In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command :

sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance

before updating the schema.xml .

Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users

Additionaly, the url is http://(solr_ [1]server):8983/solr/dovecot/ (error in wiki)

After installing Solr, wherever the installation sets up there should a folder similar to:

<your prefix>/solr/server/solr/configsets

If you look there, you'll probably see folders like '_default' and 'sample_techproducts_configs'. I haven't played with the 'techproducts' sample. Copy the '_default' folder, with all its contents, to a 'dovecot' folder. In the new dovecot folder, replace the 'managed-schema' file with the file from the Dovecot Wiki

https://wiki.dovecot.org/Plugins/FTS/Solr?action=AttachFile&do=view&target=solr-7.x-schema.xml

after that, you should be able to run 'solr /opt/solr/bin/solr create -c dovecot' to create the instance. If things still don't work let us know.

The schema is one I've tweaked and updated during my own migrations since Solr 3.3. It's possible there's something else in my config that needs documenting - but having experienced Solr search against my mailstore I never want to be without it.

Daniel

Links:

[1] http://(solr

Daniel,

I have done that so any times (deleteing the data folders, recreating the instance, restarting etc...)

But this is really not the issue

The issue is 1 - fts_solr reports errors in the log file (this is a pure dovecot issue) : how to have much more details on what fts_solr sends to Slor server and what does it returns ? 2 - Solr returns properly for a few hours, then starts crashing or responding non-sense after some time

Additionally, is there a doc of fts-squat in order to adjust the code to new releases of dovect ?

On December 12, 2018 4:44:10 PM Daniel Miller via dovecot dovecot@dovecot.org wrote:

On 12/11/2018 4:46 AM, Joan Moreau via dovecot wrote:

...

I shared the errors already so many times (check this mailinling for "solr" in teh title)

Contrary to what you say, with SOlr 7.5 and Dovecot git, I had to remove the "managed-schema" to make solr respond a bit properly. It relies on schema.xml

In order to create the instance, no, it copies the default config in the dovecot instance. I'm not a Solr expert by any means but I believe you are incorrect. As of Solr 5.x the managed-schema file is the primary method for configuration. The method I detailed previously for setting up a config helps automate creating new Solr instances - but as I stated you can either setup a Solr template and then create the instance from that or create an instance using the default template and then adjust it. The part that you *must* do after creating from the default template is stop the server, delete the entire "<prefix>/solr/dovecot/data" folder, then install the correct managed-schema file, then restart the server. The server will not function with mismatched schema/data. If you'll try that - explicitly "rm -rf <prefix>/solr/dovecot/data", copy the managed-schema file into the conf folder, and restart - things will either work or there's something else that needs correction.

Daniel

...

...

...

Joan,

I understand and sympathize with your frustration - trying to get multiple applications to work together, particularly given the lack of documentation for some of them, can be extremely challenging.  That said, I suggest you consider an alternative viewpoint.  Frequently being misunderstood myself I apologize in advance if I'm reading you wrong - but it appears your view towards the situation is there is a bug in Dovecot related to this problem.  That may well be - but I generally approach these matters from the assumption that *I* made the error in configuration and go from there.  I'm not an official rep for any product nor claim to be any form of expert in these matters - but I do have a working setup and I'd like to help you if I can.  If you're willing to - take a deep breath and let's try starting over.

Looking back through your emails there were two items that stood out - your Dovecot config has two settings I don't use: "fts_decoder" and "fts_enforced".  I also asked you earlier whether or not NFS is involved here and I didn't see an answer - please clarify.

I suggest you try once more: delete Solr completely.  Re-install per the directions and use *my* managed-schema.  Also comment out the Dovecot directives for "fts_decoder" and "fts_enforced" so you're closer to my setup.  Try running again and then post back - I'll do what I can.  Based on the fact that Dovecot+Solr 7.5+my schema is working for me leads me to believe we can get it working for you as well.

Daniel

On 12/15/2018 2:42 PM, Joan Moreau wrote:

here my latest schema.xml (remove the "long" type hich seems to be very deprecated in 7.x)

<?xml version="1.0" encoding="UTF-8"?>

<schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0" /> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="hdr" type="gjtext" indexed="false" stored="false"/> <field name="cc" type="gjtext" indexed="true" stored="false"/> <field name="from" type="gjtext" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjtext" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </fields> </schema>

On 2018-12-15 20:54, Joan Moreau wrote:

...

Daniel, I have done that so any times (deleteing the data folders, recreating the instance, restarting etc...) But this is really not the issue The issue is 1 - fts_solr reports errors in the log file (this is a pure dovecot issue) : how to have much more details on what fts_solr sends to Slor server and what does it returns ? 2 - Solr returns properly for a few hours, then starts crashing or responding non-sense after some time Additionally, is there a doc of fts-squat in order to adjust the code to new releases of dovect ?

On December 12, 2018 4:44:10 PM Daniel Miller via dovecot dovecot@dovecot.org wrote:

On 12/11/2018 4:46 AM, Joan Moreau via dovecot wrote:

...

I shared the errors already so many times (check this mailinling
for "solr" in teh title)

Contrary to what you say, with SOlr 7.5 and Dovecot git,  I had
to remove the "managed-schema" to make solr respond a bit
properly. It relies on schema.xml

In order to create the instance, no, it copies the default
config in the dovecot instance.

I'm not a Solr expert by any means but I believe you are incorrect.

As of Solr 5.x the managed-schema file is the primary method for
configuration.  The method I detailed previously for setting up a
config helps automate creating new Solr instances - but as I
stated you can either setup a Solr template and then create the
instance from that or create an instance using the default
template and then adjust it.

The part that you *must* do after creating from the default
template is stop the server, delete the entire
"<prefix>/solr/dovecot/data" folder, then install the correct
managed-schema file, then restart the server.  The server will
not function with mismatched schema/data.

If you'll try that - explicitly "rm -rf
<prefix>/solr/dovecot/data", copy the managed-schema file into
the conf folder, and restart - things will either work or there's
something else that needs correction.

--
Daniel

Joan,

The reason for dropping squat, I'm assuming, is that Lucene and Solr potentially provide superior features & performance and as they are 3rd-party libraries & apps it reduces the maintenance responsibilities and let's the Dovecot team focus on mail server specific stuff - and let others focus on FTS.  There is a *huge* difference between a functional Solr setup & squat - and if I'm able to get it working we should be able to get you there as well.

I don't recall what OS you're running - I'm on Ubuntu 18.04.  My Java version is OpenJDK 10.0.2.  Attached is my complete Solr config.  Try one more time - stop the server, delete the data folder, unpack the attached into the conf folder - and restart.  I also have

/etc/default/solr.in.sh: SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=3000" SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=60000" SOLR_PID_DIR=/run/solr SOLR_HOME=/usr/local/lib

Adjust the above folders as appropriate - or don't use them at all if you're using the defaults.

/etc/systemd/system/solr.service: # put this file in /etc/systemd/system/ as root # below paths assume solr installed in /opt/solr, SOLR_PID_DIR is /data # and that all configuration exists in /etc/default/solr.in.sh which is the case if previously installed as an init.d service # change port in pid file if differs # note that it is configured to auto restart solr if it fails (Restart=on-faliure) and that's the motivation indeed :) # to switch from systemv (init.d) to systemd, do the following after creating this file: # sudo systemctl daemon-reload # sudo service solr stop # if already running # sudo systemctl enable solr # systemctl start solr # this was inspired by https://confluence.t5.fi/display/~stefan.roos/2015/04/01/Creating+systemd+un... [Unit] Description=Apache SOLR 7.5.0 After=syslog.target network.target remote-fs.target nss-lookup.target systemd-journald-dev-log.socket Before=multi-user.target graphical.target nginx.service dovecot.service Conflicts=shutdown.target [Service] LimitNOFILE=65000 User=vmail Group=mail ExecStartPre=/bin/mkdir -p /run/solr ExecStartPre=/bin/chown -R vmail.mail /run/solr PermissionsStartOnly=true PIDFile=/run/solr/solr-8983.pid Environment=SOLR_INCLUDE=/etc/default/solr.in.sh ExecStart=/opt/solr/bin/solr start ExecStop=/opt/solr/bin/solr stop Restart=on-failure RestartSec=15s TimeoutStopSec=30s [Install] WantedBy=multi-user.target graphical.target dovecot.service

If you don't use systemd disregard - but see if any of the above applies for your setup.

Let me know what happens.  I agree this can be a mortal pain to setup - but it's worth it.

Daniel

On 12/21/2018 4:33 AM, Joan Moreau wrote:

Dear Daniel.

Thank you for your kind reply.

Regarding NFS, no, there is nothing like this in my setup.

Deleteing SOLR and recreating it, I did it so  many times already.

I started with *your* setup in the first place, as FTS_squat (which actually works very well and very straightforward, I have no clue why going for SOlr which is just a pain and not maintaining squat), and it leads to totally funny results (for instance, I type "emirates" in my "Air Companies" subfolder and get a lot of results .. but of competing companies :D )

I added the fts_enforce following AKi advice.

I removed fts_decoder for the time being.

I don't know where to go now. Dovcot still returning errors and SOlr still companinig with "Out of range" and other Java errors.

Bottom line, I am back to squat, but as it is not maintained so crashed also times to times.

I think we should discuss on

(1) Why the damn choice of Solr has been main. As you empahised, maintainend so many independent software is a pain

(2) If there is a real reason why going for SOlr, how to have a working (i.e. getting the right results to the end user) setup ?

(3) If there iare no tangible reason, what about maintaining fts_squat , which did the job nicely for years and no complains about.

On 2018-12-16 08:51, Daniel Miller via dovecot wrote:

...

Joan,

I understand and sympathize with your frustration - trying to get multiple applications to work together, particularly given the lack of documentation for some of them, can be extremely challenging.  That said, I suggest you consider an alternative viewpoint.  Frequently being misunderstood myself I apologize in advance if I'm reading you wrong - but it appears your view towards the situation is there is a bug in Dovecot related to this problem.  That may well be these matters - but I do have a working setup and I'd like to help

• but I generally approach these matters from the assumption that *I* made the error in configuration and go from there.  I'm not an official rep for any product nor claim to be any form of expert in

you if I can.  If you're willing to - take a deep breath and let's try starting over.

Looking back through your emails there were two items that stood out

• your Dovecot config has two settings I don't use: "fts_decoder" and "fts_enforced".  I also asked you earlier whether or not NFS is involved here and I didn't see an answer - please clarify.

I suggest you try once more: delete Solr completely. Re-install per the directions and use *my* managed-schema. Also comment out the Dovecot directives for "fts_decoder" and "fts_enforced" so you're closer to my setup.  Try running again and then post back - I'll do what I can.  Based on the fact that Dovecot+Solr 7.5+my schema is working for me leads me to believe we can get it working for you as well.

Daniel

On 12/15/2018 2:42 PM, Joan Moreau wrote:

...

here my latest schema.xml (remove the "long" type hich seems to be very deprecated in 7.x)

<?xml version="1.0" encoding="UTF-8"?>

<schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0" /> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="hdr" type="gjtext" indexed="false" stored="false"/> <field name="cc" type="gjtext" indexed="true" stored="false"/> <field name="from" type="gjtext" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjtext" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </fields> </schema>

On 2018-12-15 20:54, Joan Moreau wrote:

Daniel,
I have done that so any times (deleteing the data folders,
recreating the instance, restarting etc...)
But this is really not the issue
The issue is
1 - fts_solr reports errors in the log file (this is a pure
dovecot issue) : how to have much more details on what fts_solr
sends to Slor server and what does it returns ?
2 - Solr returns properly for a few hours, then starts crashing
or responding non-sense after some time
Additionally, is there a doc of fts-squat in order to adjust the
code to new releases of dovect ?

On December 12, 2018 4:44:10 PM Daniel Miller via dovecot
<dovecot@dovecot.org> wrote:

    On 12/11/2018 4:46 AM, Joan Moreau via dovecot wrote:

        I shared the errors already so many times (check this
        mailinling for "solr" in teh title)

        Contrary to what you say, with SOlr 7.5 and Dovecot
        git,  I had to remove the "managed-schema" to make solr
        respond a bit properly. It relies on schema.xml

        In order to create the instance, no, it copies  the
        default config in the dovecot instance.

    I'm not a Solr expert by any means but I believe you are
    incorrect.

    As of Solr 5.x the managed-schema file is the primary method
    for configuration.  The method I detailed previously for
    setting up a config helps automate creating new Solr
    instances - but as I stated you can either setup a Solr
    template and then create the instance from that or create an
    instance using the default template and then adjust it.

    The part that you *must* do after creating from the default
    template is stop the server, delete the entire
    "<prefix>/solr/dovecot/data" folder, then install the
    correct managed-schema file, then restart the server.  The
    server will not function with mismatched schema/data.

    If you'll try that - explicitly "rm -rf
    <prefix>/solr/dovecot/data", copy the managed-schema file
    into the conf folder, and restart - things will either work
    or there's something else that needs correction.

    --
    Daniel

Also :

• Java is 10.0.2

• If i delete schema.xml but create only managed-schema, the solr refuses to start with a java error "schema.xml missing"

On 2018-12-30 08:46, Joan Moreau wrote:

Hi Daniel,

I am on Archlinux. Anyway, I adapted the scripts.

2 questions:

1 - It looks like we are not on the same version . I am on 7.5.0. Which version are you running ?

2 - Your conf shows that you let managed-schema but deleted schema.xml. What is the meaning of each ?

Thank you

On 2018-12-22 00:19, Daniel Miller wrote:

Joan,

The reason for dropping squat, I'm assuming, is that Lucene and Solr potentially provide superior features & performance and as they are 3rd-party libraries & apps it reduces the maintenance responsibilities and let's the Dovecot team focus on mail server specific stuff - and let others focus on FTS. There is a *huge* difference between a functional Solr setup & squat - and if I'm able to get it working we should be able to get you there as well.

I don't recall what OS you're running - I'm on Ubuntu 18.04. My Java version is OpenJDK 10.0.2. Attached is my complete Solr config. Try one more time - stop the server, delete the data folder, unpack the attached into the conf folder - and restart. I also have

/etc/default/solr.in.sh: SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=3000" SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=60000" SOLR_PID_DIR=/run/solr SOLR_HOME=/usr/local/lib Adjust the above folders as appropriate - or don't use them at all if you're using the defaults.

/etc/systemd/system/solr.service: # put this file in /etc/systemd/system/ as root # below paths assume solr installed in /opt/solr, SOLR_PID_DIR is /data # and that all configuration exists in /etc/default/solr.in.sh which is the case if previously installed as an init.d service # change port in pid file if differs # note that it is configured to auto restart solr if it fails (Restart=on-faliure) and that's the motivation indeed :) # to switch from systemv (init.d) to systemd, do the following after creating this file: # sudo systemctl daemon-reload # sudo service solr stop # if already running # sudo systemctl enable solr # systemctl start solr # this was inspired by https://confluence.t5.fi/display/~stefan.roos/2015/04/01/Creating+systemd+un... [Unit] Description=Apache SOLR 7.5.0 After=syslog.target network.target remote-fs.target nss-lookup.target systemd-journald-dev-log.socket Before=multi-user.target graphical.target nginx.service dovecot.service Conflicts=shutdown.target [Service] LimitNOFILE=65000 User=vmail Group=mail ExecStartPre=/bin/mkdir -p /run/solr ExecStartPre=/bin/chown -R vmail.mail /run/solr PermissionsStartOnly=true PIDFile=/run/solr/solr-8983.pid Environment=SOLR_INCLUDE=/etc/default/solr.in.sh ExecStart=/opt/solr/bin/solr start ExecStop=/opt/solr/bin/solr stop Restart=on-failure RestartSec=15s TimeoutStopSec=30s [Install] WantedBy=multi-user.target graphical.target dovecot.service If you don't use systemd disregard - but see if any of the above applies for your setup. Let me know what happens. I agree this can be a mortal pain to setup - but it's worth it.

Daniel

On 12/21/2018 4:33 AM, Joan Moreau wrote:

Dear Daniel.

Thank you for your kind reply.

Regarding NFS, no, there is nothing like this in my setup.

Deleteing SOLR and recreating it, I did it so many times already.

I started with *your* setup in the first place, as FTS_squat (which actually works very well and very straightforward, I have no clue why going for SOlr which is just a pain and not maintaining squat), and it leads to totally funny results (for instance, I type "emirates" in my "Air Companies" subfolder and get a lot of results .. but of competing companies :D )

I added the fts_enforce following AKi advice.

I removed fts_decoder for the time being.

I don't know where to go now. Dovcot still returning errors and SOlr still companinig with "Out of range" and other Java errors.

Bottom line, I am back to squat, but as it is not maintained so crashed also times to times.

I think we should discuss on

(1) Why the damn choice of Solr has been main. As you empahised, maintainend so many independent software is a pain

(2) If there is a real reason why going for SOlr, how to have a working (i.e. getting the right results to the end user) setup ?

(3) If there iare no tangible reason, what about maintaining fts_squat , which did the job nicely for years and no complains about.

On 2018-12-16 08:51, Daniel Miller via dovecot wrote:

Joan,

I understand and sympathize with your frustration - trying to get multiple applications to work together, particularly given the lack of documentation for some of them, can be extremely challenging. That said, I suggest you consider an alternative viewpoint. Frequently being misunderstood myself I apologize in advance if I'm reading you wrong - but it appears your view towards the situation is there is a bug in Dovecot related to this problem. That may well be - but I generally approach these matters from the assumption that *I* made the error in configuration and go from there. I'm not an official rep for any product nor claim to be any form of expert in these matters - but I do have a working setup and I'd like to help you if I can. If you're willing to - take a deep breath and let's try starting over.

Looking back through your emails there were two items that stood out - your Dovecot config has two settings I don't use: "fts_decoder" and "fts_enforced". I also asked you earlier whether or not NFS is involved here and I didn't see an answer - please clarify.

I suggest you try once more: delete Solr completely. Re-install per the directions and use *my* managed-schema. Also comment out the Dovecot directives for "fts_decoder" and "fts_enforced" so you're closer to my setup. Try running again and then post back - I'll do what I can. Based on the fact that Dovecot+Solr 7.5+my schema is working for me leads me to believe we can get it working for you as well.

Daniel

On 12/15/2018 2:42 PM, Joan Moreau wrote:

here my latest schema.xml (remove the "long" type hich seems to be very deprecated in 7.x)

<?xml version="1.0" encoding="UTF-8"?>

<schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0" /> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="hdr" type="gjtext" indexed="false" stored="false"/> <field name="cc" type="gjtext" indexed="true" stored="false"/> <field name="from" type="gjtext" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjtext" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </fields> </schema>

On 2018-12-15 20:54, Joan Moreau wrote:

Daniel,

I have done that so any times (deleteing the data folders, recreating the instance, restarting etc...)

But this is really not the issue

The issue is 1 - fts_solr reports errors in the log file (this is a pure dovecot issue) : how to have much more details on what fts_solr sends to Slor server and what does it returns ? 2 - Solr returns properly for a few hours, then starts crashing or responding non-sense after some time

Additionally, is there a doc of fts-squat in order to adjust the code to new releases of dovect ?

On December 12, 2018 4:44:10 PM Daniel Miller via dovecot dovecot@dovecot.org wrote: On 12/11/2018 4:46 AM, Joan Moreau via dovecot wrote:

I shared the errors already so many times (check this mailinling for "solr" in teh title)

Contrary to what you say, with SOlr 7.5 and Dovecot git, I had to remove the "managed-schema" to make solr respond a bit properly. It relies on schema.xml

In order to create the instance, no, it copies the default config in the dovecot instance.

I'm not a Solr expert by any means but I believe you are incorrect.

As of Solr 5.x the managed-schema file is the primary method for configuration. The method I detailed previously for setting up a config helps automate creating new Solr instances - but as I stated you can either setup a Solr template and then create the instance from that or create an instance using the default template and then adjust it.

The part that you *must* do after creating from the default template is stop the server, delete the entire "<prefix>/solr/dovecot/data" folder, then install the correct managed-schema file, then restart the server. The server will not function with mismatched schema/data.

If you'll try that - explicitly "rm -rf <prefix>/solr/dovecot/data", copy the managed-schema file into the conf folder, and restart - things will either work or there's something else that needs correction. -- Daniel

Hi

Solr is a standard package in ArchLinux. ("pacman -S solr") . the systemd installation script is included (and it is launching /opt/solr/bin/solr.in.sh)

Instance : sudo -u solr /opt/solr/bin/solr create -c dovecot -> this creates a separate folder with default solrconfig.xml, schema.xml, etc..

I made a symlink of the data folder to a second drive (ext4) much bigger

On 2018-12-31 14:09, Daniel Miller wrote:

On 12/29/2018 4:49 PM, Joan Moreau wrote:

...

Also :

• Java is 10.0.2 Same as me.

...

• If i delete schema.xml but create only managed-schema, the solr refuses to start with a java error "schema.xml missing" Ok...so we need to do some more digging.

How did you install Solr? (I downloaded a "binary" installation and unpacked it)

How did you create the dovecot instance? (I've provided explicit instructions for how I did it - did you follow those exactly or something different)?

How are you starting Solr? (I use the provided "solr/bin/solr start" command, wrapped inside a systemd service).

-- Daniel

and the first line of the diff is :

< this file, see http://wiki.apache.org/solr/SolrConfigXml.

this file, see http://wiki.apache.org/solr/SolrConfigXml. 38c38 < <luceneMatchVersion>6.4.1</luceneMatchVersion>

<luceneMatchVersion>7.5.0</luceneMatchVersion>

So, are you running 6.4.1 or 7.5.0 ????

On 2019-01-02 08:12, Joan Moreau wrote:

The real main differecne seems coming from "diffconfig.xml"

When I put yours, Solr delete (!) schema.xml and create a "manage-schema" and starts complaining about useless types (tdates, booleans, etc..) that are not needed for Mail fileds

When I put mine (from standard distribution of Arch), it keeps things as they are (yeah !), does not complains about those useless types and startup properly.

I attach my diffconfig

But these are the configurations that one should adjust as per his/her own use.

The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors

On 2019-01-02 07:49, Joan Moreau wrote:

Hi

Solr is a standard package in ArchLinux. ("pacman -S solr") . the systemd installation script is included (and it is launching /opt/solr/bin/solr.in.sh)

Instance : sudo -u solr /opt/solr/bin/solr create -c dovecot -> this creates a separate folder with default solrconfig.xml, schema.xml, etc..

I made a symlink of the data folder to a second drive (ext4) much bigger

On 2018-12-31 14:09, Daniel Miller wrote: On 12/29/2018 4:49 PM, Joan Moreau wrote: Also :

• Java is 10.0.2

Same as me.

• If i delete schema.xml but create only managed-schema, the solr refuses to start with a java error "schema.xml missing"

Ok...so we need to do some more digging.

How did you install Solr? (I downloaded a "binary" installation and unpacked it)

How did you create the dovecot instance? (I've provided explicit instructions for how I did it - did you follow those exactly or something different)?

How are you starting Solr? (I use the provided "solr/bin/solr start" command, wrapped inside a systemd service).

-- Daniel

Other use case :

I type "must" in the search filed-> I have some returns , but very not all, for instance "solarmust" is not in the results

If I type "solarmust" -> then I have the solarmust mail

Honestly, this is highly unstable. Not sure whereas bugs come from Solr or Dovecot

Below my adjusted (corrections from the one of Daniel who is definitely not working) schema.xml

<?xml version="1.0" encoding="UTF-8"?>

<schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="booleans" class="solr.BoolField" sortMissingLast="true" multiValued="true"/> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0"/> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.EdgeNGramTokenizerFactory" maxGramSize="15" minGramSize="3" /> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> <fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="gjtext" indexed="true" stored="false"/> <field name="from" type="gjtext" indexed="true" stored="false"/> <field name="hdr" type="gjtext" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjtext" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>

On 2019-01-02 10:04, Joan Moreau wrote:

The first result show "no results" in dovecot for any search by header (I typed an email add in RoundCube search box, using Dovecot as back end, using Solr as own backend)

So many efforts for crappy results.

Can't we really revive Squat ? It is 2 lines of config, and no single problems

On January 2, 2019 08:16:33 Joan Moreau via dovecot dovecot@dovecot.org wrote:

and the first line of the diff is :

< this file, see http://wiki.apache.org/solr/SolrConfigXml.

...

this file, see http://wiki.apache.org/solr/SolrConfigXml. 38c38 < <luceneMatchVersion>6.4.1</luceneMatchVersion>

---

...

<luceneMatchVersion>7.5.0</luceneMatchVersion>

So, are you running 6.4.1 or 7.5.0 ????

On 2019-01-02 08:12, Joan Moreau wrote:

The real main differecne seems coming from "diffconfig.xml"

When I put yours, Solr delete (!) schema.xml and create a "manage-schema" and starts complaining about useless types (tdates, booleans, etc..) that are not needed for Mail fileds

When I put mine (from standard distribution of Arch), it keeps things as they are (yeah !), does not complains about those useless types and startup properly.

I attach my diffconfig

But these are the configurations that one should adjust as per his/her own use.

The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors

On 2019-01-02 07:49, Joan Moreau wrote:

Hi

Solr is a standard package in ArchLinux. ("pacman -S solr") . the systemd installation script is included (and it is launching /opt/solr/bin/solr.in.sh)

Instance : sudo -u solr /opt/solr/bin/solr create -c dovecot -> this creates a separate folder with default solrconfig.xml, schema.xml, etc..

I made a symlink of the data folder to a second drive (ext4) much bigger

On 2018-12-31 14:09, Daniel Miller wrote: On 12/29/2018 4:49 PM, Joan Moreau wrote: Also :

• Java is 10.0.2

Same as me.

• If i delete schema.xml but create only managed-schema, the solr refuses to start with a java error "schema.xml missing"

Ok...so we need to do some more digging.

How did you install Solr? (I downloaded a "binary" installation and unpacked it)

How did you create the dovecot instance? (I've provided explicit instructions for how I did it - did you follow those exactly or something different)?

How are you starting Solr? (I use the provided "solr/bin/solr start" command, wrapped inside a systemd service).

-- Daniel

Another bug appearing today:

Jan 02 09:59:08 indexer-worker(jom@grosjo.net)<6777>MOgFATKLLFwPHAAA0thIag:oLJjJjKLLFx5GgAA0thIag: Warning: FTS-SOLR(JOM@GROSJO.NET): Mailbox XXXXXX UID=121635 HEADER SIZE IS HUGE, TRUNCATING

header of the said email has nothing of "huge"

On 2019-01-02 15:22, Joan Moreau via dovecot wrote:

Refinement of the schema.xml (below)

THis however does not solve the "no results" and "Out of range" errors in Dovecot and Solr

<?xml version="1.0" encoding="UTF-8"?>

<schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="booleans" class="solr.BoolField" sortMissingLast="true" multiValued="true"/> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0"/> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> </analyzer> </fieldType> <fieldType name="gjfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.NGramTokenizerFactory" maxGramSize="25" minGramSize="3" /> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> </analyzer> </fieldType>

<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="gjfield" indexed="true" stored="false"/> <field name="from" type="gjfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjfield" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>

Hi,

This is the best I can do for having a SCHEMA.XML matching the needs of an email user. Use case on Solr 7.5.0

(note : this does not solve any of the bugs previously mentionned, but at least, the logic of Solr is back on track)

<?xml version="1.0" encoding="UTF-8"?>

<schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> <fieldType name="gjfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="25"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType>

<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="gjfield" indexed="true" stored="false"/> <field name="from" type="gjfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjfield" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>

The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors

I've been watching the progress of this thread with no small concern, mainly because I've been tasked with providing a server-side email search facility with a budget and manpower level that comes down to mainly *1*, i.e., me.

I was expecting, given the strongly worded language about "just use lucene/SOLR" and "ignore squat", that I should invest time + effort into this JAVA nightmare that is SOLR.

I started with squat and another word-indexor system that used out-of-band (not a dovecot plugin) software to provide rapid (sub-second) searches through tens-of-GB-scale mailboxes.

Unlike what I was led to believe, the squat indexes worked surprisingly well, once you sorted out the odd resource size (ulimit-related) issues (vsz & friends) limitations. I did notice the "worst-case" search performance have worryingly high O(x) increases in time, but I'd not seen anything that was a dealbreaker. It goes without saying that various substring searches worked as expected, for the most part.

My experiences with SOLR were similar to Messr. Moreau's: lots of startup errors with provided schemata files. Lots of JAVA nonsense issues. Lots of sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM, version of SOLR, and dovecot to find the "best" working combination, only to find that the searches didn't work out as expected. I expected to be able to do date-ranging based searches. Didn't work. I expected to search CONTENTS of emails, and despite many days of tweaks, I couldn't get it to index even the basics like filenames/types of attachments, so I could exposed attachment-based searching to my users.

So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality:

1. The ability to search for a string within any of the structured fields (from/subject) that returns correct results?

2. The ability to search for any string within the BODY of emails, including the MIME attachment boundaries?

3. The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data?

OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines:

4. The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika?

SOLR seems to have "brand cachet", so presumably it actually works (for somebody).

Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I am no stranger to the "costs" of "free" software, in that you sacrifice your own blood, sweat, and tears just to get these disparate pieces to work together.

I *DO* respect that Timo has to keep the lights (and sauna) on in Finland. Maybe there's a super-secret (no advertised prices, "carrier-only" price list) with _Dovecot, Oy_ wherein the above ARE actually available for something less than 6.022 x 10^23 Euros per centi-second of licencing fees.

But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?)

I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have.

He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them.

Respectfully, =M=

On Wed, 2019-01-02 at 00:59 -0800, M. Balridge wrote:

...

The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors

I've been watching the progress of this thread with no small concern, mainly because I've been tasked with providing a server-side email search facility with a budget and manpower level that comes down to mainly *1*, i.e., me.

I was expecting, given the strongly worded language about "just use lucene/SOLR" and "ignore squat", that I should invest time + effort into this JAVA nightmare that is SOLR.

I started with squat and another word-indexor system that used out-of-band (not a dovecot plugin) software to provide rapid (sub-second) searches through tens-of-GB-scale mailboxes.

Unlike what I was led to believe, the squat indexes worked surprisingly well, once you sorted out the odd resource size (ulimit-related) issues (vsz & friends) limitations. I did notice the "worst-case" search performance have worryingly high O(x) increases in time, but I'd not seen anything that was a dealbreaker. It goes without saying that various substring searches worked as expected, for the most part.

My experiences with SOLR were similar to Messr. Moreau's: lots of startup errors with provided schemata files. Lots of JAVA nonsense issues. Lots of sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM, version of SOLR, and dovecot to find the "best" working combination, only to find that the searches didn't work out as expected. I expected to be able to do date-ranging based searches. Didn't work. I expected to search CONTENTS of emails, and despite many days of tweaks, I couldn't get it to index even the basics like filenames/types of attachments, so I could exposed attachment-based searching to my users.

So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality:

1. The ability to search for a string within any of the structured fields (from/subject) that returns correct results?

2. The ability to search for any string within the BODY of emails, including the MIME attachment boundaries?

3. The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data?

OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines:

4. The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika?

---

SOLR seems to have "brand cachet", so presumably it actually works (for somebody).

Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I am no stranger to the "costs" of "free" software, in that you sacrifice your own blood, sweat, and tears just to get these disparate pieces to work together.

I *DO* respect that Timo has to keep the lights (and sauna) on in Finland. Maybe there's a super-secret (no advertised prices, "carrier-only" price list) with _Dovecot, Oy_ wherein the above ARE actually available for something less than 6.022 x 10^23 Euros per centi-second of licencing fees.

But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?)

I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have.

He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them.

Respectfully, =M=

Fascinating...

SOLR says the following are powered by SOLR...

https://wiki.apache.org/solr/PublicServers

Perhaps if you could find out from that list which of them are using SOLR in conjunction with Dovecot...

food for thought...

I hope you are aware that "linking with Xapian" requires somewhat more work than just -lxapian in linker? If you or someone feels like writing fts_xapian, go for it.

Aki

On 04 January 2019 at 08:20 Joan Moreau via dovecot dovecot@dovecot.org wrote:

What about consedering linking Dovecot with Xapian librairies instead of going to nightmare Solr ?

https://xapian.org/features

On 2019-01-02 17:10, John Tulp wrote:

...

On Wed, 2019-01-02 at 00:59 -0800, M. Balridge wrote: The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors I've been watching the progress of this thread with no small concern, mainly because I've been tasked with providing a server-side email search facility with a budget and manpower level that comes down to mainly *1*, i.e., me.

I was expecting, given the strongly worded language about "just use lucene/SOLR" and "ignore squat", that I should invest time + effort into this JAVA nightmare that is SOLR.

I started with squat and another word-indexor system that used out-of-band (not a dovecot plugin) software to provide rapid (sub-second) searches through tens-of-GB-scale mailboxes.

Unlike what I was led to believe, the squat indexes worked surprisingly well, once you sorted out the odd resource size (ulimit-related) issues (vsz & friends) limitations. I did notice the "worst-case" search performance have worryingly high O(x) increases in time, but I'd not seen anything that was a dealbreaker. It goes without saying that various substring searches worked as expected, for the most part.

My experiences with SOLR were similar to Messr. Moreau's: lots of startup errors with provided schemata files. Lots of JAVA nonsense issues. Lots of sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM, version of SOLR, and dovecot to find the "best" working combination, only to find that the searches didn't work out as expected. I expected to be able to do date-ranging based searches. Didn't work. I expected to search CONTENTS of emails, and despite many days of tweaks, I couldn't get it to index even the basics like filenames/types of attachments, so I could exposed attachment-based searching to my users.

So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality:

1. The ability to search for a string within any of the structured fields (from/subject) that returns correct results?

2. The ability to search for any string within the BODY of emails, including the MIME attachment boundaries?

3. The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data?

OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines:

4. The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika?

---

SOLR seems to have "brand cachet", so presumably it actually works (for somebody).

Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I am no stranger to the "costs" of "free" software, in that you sacrifice your own blood, sweat, and tears just to get these disparate pieces to work together.

I *DO* respect that Timo has to keep the lights (and sauna) on in Finland. Maybe there's a super-secret (no advertised prices, "carrier-only" price list) with _Dovecot, Oy_ wherein the above ARE actually available for something less than 6.022 x 10^23 Euros per centi-second of licencing fees.

But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?)

I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have.

He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them.

Respectfully, =M= Fascinating...

SOLR says the following are powered by SOLR...

https://wiki.apache.org/solr/PublicServers

Perhaps if you could find out from that list which of them are using SOLR in conjunction with Dovecot...

food for thought...

A starting point would be to have a look at the current FTS plugins:

https://github.com/dovecot/core/tree/master/src/plugins/fts-solrandhttps://g... -M

Am Freitag, den 04.01.2019, 18:17 +0800 schrieb Joan Moreau via dovecot:

Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin

On 2019-01-04 17:20, Aki Tuomi wrote:

...

I hope you are aware that "linking with Xapian" requires somewhat more work than just -lxapian in linker? If you or someone feels like writing fts_xapian, go for it.

Aki

...

On 04 January 2019 at 08:20 Joan Moreau via dovecot < dovecot@dovecot.org> wrote:

What about consedering linking Dovecot with Xapian librairies instead of going to nightmare Solr ?

https://xapian.org/features

On 2019-01-02 17:10, John Tulp wrote:

...

On Wed, 2019-01-02 at 00:59 -0800, M. Balridge wrote: The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors I've been watching the progress of this thread with no small concern, mainly because I've been tasked with providing a server-side email search facility with a budget and manpower level that comes down to mainly *1*, i.e., me.

I was expecting, given the strongly worded language about "just use lucene/SOLR" and "ignore squat", that I should invest time + effort into this JAVA nightmare that is SOLR.

I started with squat and another word-indexor system that used out-of-band (not a dovecot plugin) software to provide rapid (sub-second) searches through tens-of-GB-scale mailboxes.

Unlike what I was led to believe, the squat indexes worked surprisingly well, once you sorted out the odd resource size (ulimit-related) issues (vsz & friends) limitations. I did notice the "worst-case" search performance have worryingly high O(x) increases in time, but I'd not seen anything that was a dealbreaker. It goes without saying that various substring searches worked as expected, for the most part.

My experiences with SOLR were similar to Messr. Moreau's: lots of startup errors with provided schemata files. Lots of JAVA nonsense issues. Lots of sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM, version of SOLR, and dovecot to find the "best" working combination, only to find that the searches didn't work out as expected. I expected to be able to do date-ranging based searches. Didn't work. I expected to search CONTENTS of emails, and despite many days of tweaks, I couldn't get it to index even the basics like filenames/types of attachments, so I could exposed attachment-based searching to my users.

So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality:

1. The ability to search for a string within any of the structured fields (from/subject) that returns correct results?

2. The ability to search for any string within the BODY of emails, including the MIME attachment boundaries?

3. The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data?

OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines:

4. The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika?

---

SOLR seems to have "brand cachet", so presumably it actually works (for somebody).

Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I am no stranger to the "costs" of "free" software, in that you sacrifice your own blood, sweat, and tears just to get these disparate pieces to work together.

I *DO* respect that Timo has to keep the lights (and sauna) on in Finland. Maybe there's a super-secret (no advertised prices, "carrier- only" price list) with _Dovecot, Oy_ wherein the above ARE actually available for something less than 6.022 x 10^23 Euros per centi-second of licencing fees.

But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?)

I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have.

He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them.

Respectfully, =M=

Fascinating...

SOLR says the following are powered by SOLR...

https://wiki.apache.org/solr/PublicServers

Perhaps if you could find out from that list which of them are using SOLR in conjunction with Dovecot...

food for thought...

Also, a description of the "to be" functions of the backend:

struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT, -> WHAT OTHER FLAGS ?

{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };

On 2019-01-04 20:33, Joan Moreau via dovecot wrote:

Yes but:

1 - is there a documentation of the main object ? (fts_backend, mail_user, mailbox, etc..)

2 - What are the mandatory functions ?

3 - Search : Supposedly, the FTS shall have several parameters : the keyword(s), the user & mailbox, and the fields (to, from, body, etc..) to be includude in the search. What is the function called in the plugin ?

4 - Indexing : Somehow, what is the logic ? fts core just ask to "index me this email of this mailbox" ? or this is delegated to the plugin to sort out which emails it has indexed yet or not ?

Thank you

On 2019-01-04 18:49, admin wrote: A starting point would be to have a look at the current FTS plugins:

https://github.com/dovecot/core/tree/master/src/plugins/fts-solr and https://github.com/dovecot/core/tree/master/src/plugins/fts-squat

-M

Am Freitag, den 04.01.2019, 18:17 +0800 schrieb Joan Moreau via dovecot:

Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin

On 2019-01-04 17:20, Aki Tuomi wrote: I hope you are aware that "linking with Xapian" requires somewhat more work than just -lxapian in linker? If you or someone feels like writing fts_xapian, go for it.

Aki

On 04 January 2019 at 08:20 Joan Moreau via dovecot dovecot@dovecot.org wrote:

What about consedering linking Dovecot with Xapian librairies instead of going to nightmare Solr ?

https://xapian.org/features

On 2019-01-02 17:10, John Tulp wrote:

On Wed, 2019-01-02 at 00:59 -0800, M. Balridge wrote: The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors I've been watching the progress of this thread with no small concern, mainly because I've been tasked with providing a server-side email search facility with a budget and manpower level that comes down to mainly *1*, i.e., me.

I was expecting, given the strongly worded language about "just use lucene/SOLR" and "ignore squat", that I should invest time + effort into this JAVA nightmare that is SOLR.

I started with squat and another word-indexor system that used out-of-band (not a dovecot plugin) software to provide rapid (sub-second) searches through tens-of-GB-scale mailboxes.

Unlike what I was led to believe, the squat indexes worked surprisingly well, once you sorted out the odd resource size (ulimit-related) issues (vsz & friends) limitations. I did notice the "worst-case" search performance have worryingly high O(x) increases in time, but I'd not seen anything that was a dealbreaker. It goes without saying that various substring searches worked as expected, for the most part.

My experiences with SOLR were similar to Messr. Moreau's: lots of startup errors with provided schemata files. Lots of JAVA nonsense issues. Lots of sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM, version of SOLR, and dovecot to find the "best" working combination, only to find that the searches didn't work out as expected. I expected to be able to do date-ranging based searches. Didn't work. I expected to search CONTENTS of emails, and despite many days of tweaks, I couldn't get it to index even the basics like filenames/types of attachments, so I could exposed attachment-based searching to my users.

So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality:

1. The ability to search for a string within any of the structured fields (from/subject) that returns correct results?

2. The ability to search for any string within the BODY of emails, including the MIME attachment boundaries?

3. The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data?

OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines:

4. The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika?

---

SOLR seems to have "brand cachet", so presumably it actually works (for somebody).

Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I am no stranger to the "costs" of "free" software, in that you sacrifice your own blood, sweat, and tears just to get these disparate pieces to work together.

I *DO* respect that Timo has to keep the lights (and sauna) on in Finland. Maybe there's a super-secret (no advertised prices, "carrier-only" price list) with _Dovecot, Oy_ wherein the above ARE actually available for something less than 6.022 x 10^23 Euros per centi-second of licencing fees.

But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?)

I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have.

He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them.

Respectfully, =M= Fascinating...

SOLR says the following are powered by SOLR...

https://wiki.apache.org/solr/PublicServers

Perhaps if you could find out from that list which of them are using SOLR in conjunction with Dovecot...

food for thought...

Thank Stephan

I basically need to know the role/description of each of the functions of the fts_backend:

struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT, -> WHAT OTHER FLAGS ?

{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };

THank you

On 2019-01-05 08:49, Stephan Bosch wrote:

Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:

...

Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.

I know of one FTS plugin not written by Dovecot developers:

https://github.com/atkinsj/fts-elasticsearch

If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).

Regards,

Stephan.

Anyone willing to explain those functions ?

Most notably " get_last_uid" "set_build_key" "build_more" , what is refresh versus rescan ?

On January 5, 2019 14:23:10 Joan Moreau via dovecot dovecot@dovecot.org wrote:

Thank Stephan I basically need to know the role/description of each of the functions of the fts_backend:

struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT, -> what other flags ? { fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };

THank you On 2019-01-05 08:49, Stephan Bosch wrote:

...

Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:

...

Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin

The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.

I know of one FTS plugin not written by Dovecot developers:

https://github.com/atkinsj/fts-elasticsearch

If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).

Regards,

Stephan.

Thank you

I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?

On 2019-01-06 08:43, Stephan Bosch wrote:

Op 06/01/2019 om 01:00 schreef Joan Moreau:

...

Anyone willing to explain those functions ?

Most notably " get_last_uid"

From src/plugins/fts/fts-api.h:

/* Get the last_uid for the mailbox. */ int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box, uint32_t *last_uid_r);

The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this returns the last UID added to the index for the given mailbox and FTS index.

...

"set_build_key"

From src/plugins/fts/fts-api.h:

/* Switch to building index for specified key. If backend doesn't want to index this key, it can return FALSE and caller will skip to next key. */ bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx, const struct fts_backend_build_key *key);

Same file provides outline of what a build_key is.

...

"build_more" ,

/* Add more content to the index for the currently specified build key. Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters, but it doesn't need to be NUL-terminated. size contains the data size in bytes, not characters. This function may be called many times and the data block sizes may be small. Backend returns 0 if ok, -1 if build should be aborted. */ int fts_backend_update_build_more(struct fts_backend_update_context *ctx, const unsigned char *data, size_t size);

You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing.

...

what is refresh versus rescan ?

From fts-api.h:

/* Refresh index to make sure we see latest changes from lookups. Returns 0 if ok, -1 if error. */ int fts_backend_refresh(struct fts_backend *backend); /* Go through the entire index and make sure all mails are indexed, and delete any extra mails in the index. */ int fts_backend_rescan(struct fts_backend *backend);

Regards,

Stepham

On January 5, 2019 14:23:10 Joan Moreau via dovecot dovecot@dovecot.org wrote:

Thank Stephan

I basically need to know the role/description of each of the functions of the fts_backend:

struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*

{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };

THank you

On 2019-01-05 08:49, Stephan Bosch wrote:

Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot: Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin

The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.

I know of one FTS plugin not written by Dovecot developers:

https://github.com/atkinsj/fts-elasticsearch

If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).

Regards,

Stephan.

also, for fts_backend_solr_update_set_build_key -> where is the data (of the hdr_name or the body) ?

On 2019-01-06 14:10, Joan Moreau wrote:

for the "last uid"-> this is not the last added, but the maximum of the UID in the indexed emails, right ?

On 2019-01-06 11:53, Joan Moreau via dovecot wrote:

Thank you

I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?

On 2019-01-06 08:43, Stephan Bosch wrote:

Op 06/01/2019 om 01:00 schreef Joan Moreau: Anyone willing to explain those functions ?

Most notably " get_last_uid" From src/plugins/fts/fts-api.h:

/* Get the last_uid for the mailbox. */ int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box, uint32_t *last_uid_r);

The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this returns the last UID added to the index for the given mailbox and FTS index.

"set_build_key" From src/plugins/fts/fts-api.h:

/* Switch to building index for specified key. If backend doesn't want to index this key, it can return FALSE and caller will skip to next key. */ bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx, const struct fts_backend_build_key *key);

Same file provides outline of what a build_key is.

"build_more" , /* Add more content to the index for the currently specified build key. Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters, but it doesn't need to be NUL-terminated. size contains the data size in bytes, not characters. This function may be called many times and the data block sizes may be small. Backend returns 0 if ok, -1 if build should be aborted. */ int fts_backend_update_build_more(struct fts_backend_update_context *ctx, const unsigned char *data, size_t size);

You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing.

what is refresh versus rescan ? From fts-api.h:

/* Refresh index to make sure we see latest changes from lookups. Returns 0 if ok, -1 if error. */ int fts_backend_refresh(struct fts_backend *backend); /* Go through the entire index and make sure all mails are indexed, and delete any extra mails in the index. */ int fts_backend_rescan(struct fts_backend *backend);

Regards,

Stepham

On January 5, 2019 14:23:10 Joan Moreau via dovecot dovecot@dovecot.org wrote:

Thank Stephan

I basically need to know the role/description of each of the functions of the fts_backend:

struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*

{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };

THank you

On 2019-01-05 08:49, Stephan Bosch wrote:

Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot: Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin

The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.

I know of one FTS plugin not written by Dovecot developers:

https://github.com/atkinsj/fts-elasticsearch

If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).

Regards,

Stephan.

for fts_backend_xxx_lookup, where is specidifed in which field (to, cc, subject, body, from, all) to lookup ?

On 2019-01-06 16:03, Joan Moreau wrote:

For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ???

There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ?

On 2019-01-06 15:41, Joan Moreau wrote:

also, for fts_backend_solr_update_set_build_key -> where is the data (of the hdr_name or the body) ?

On 2019-01-06 14:10, Joan Moreau wrote:

for the "last uid"-> this is not the last added, but the maximum of the UID in the indexed emails, right ?

On 2019-01-06 11:53, Joan Moreau via dovecot wrote:

Thank you

I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?

On 2019-01-06 08:43, Stephan Bosch wrote:

Op 06/01/2019 om 01:00 schreef Joan Moreau: Anyone willing to explain those functions ?

Most notably " get_last_uid" From src/plugins/fts/fts-api.h:

/* Get the last_uid for the mailbox. */ int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box, uint32_t *last_uid_r);

The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this returns the last UID added to the index for the given mailbox and FTS index.

"set_build_key" From src/plugins/fts/fts-api.h:

/* Switch to building index for specified key. If backend doesn't want to index this key, it can return FALSE and caller will skip to next key. */ bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx, const struct fts_backend_build_key *key);

Same file provides outline of what a build_key is.

"build_more" , /* Add more content to the index for the currently specified build key. Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters, but it doesn't need to be NUL-terminated. size contains the data size in bytes, not characters. This function may be called many times and the data block sizes may be small. Backend returns 0 if ok, -1 if build should be aborted. */ int fts_backend_update_build_more(struct fts_backend_update_context *ctx, const unsigned char *data, size_t size);

You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing.

what is refresh versus rescan ? From fts-api.h:

/* Refresh index to make sure we see latest changes from lookups. Returns 0 if ok, -1 if error. */ int fts_backend_refresh(struct fts_backend *backend); /* Go through the entire index and make sure all mails are indexed, and delete any extra mails in the index. */ int fts_backend_rescan(struct fts_backend *backend);

Regards,

Stepham

On January 5, 2019 14:23:10 Joan Moreau via dovecot dovecot@dovecot.org wrote:

Thank Stephan

I basically need to know the role/description of each of the functions of the fts_backend:

struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*

{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };

THank you

On 2019-01-05 08:49, Stephan Bosch wrote:

Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot: Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin

The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.

I know of one FTS plugin not written by Dovecot developers:

https://github.com/atkinsj/fts-elasticsearch

If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).

Regards,

Stephan.

Hi

ANyone to answer specifically ?

Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance)

Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ?

Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ?

Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ?

Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ?

Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ?

THank you

On 2019-01-06 16:50, Joan Moreau via dovecot wrote:

and finally , for fts_backend_xxxx_lookup_multi, why is that backend dependent ?

Would- nt the below function below be the same for any backend ?

Waiting fro your feedback on all those questions

Thank you

JM

---

static int fts_backend_xapian_lookup_multi(struct fts_backend *_backend, struct mailbox *const boxes[], struct mail_search_arg *args, enum fts_lookup_flags flags, struct fts_multi_result *result) { struct xapian_fts_backend_update_context *ctx = (struct xapian_fts_backend_update_context *)_ctx;

int i=0;

while(boxes[i]!=NULL) { if(fts_backend_xapian_lookup(backend,box[i],args,flags,result->box_results[i])<0) return -1; i++; } return 0; }

On 2019-01-06 16:31, Joan Moreau via dovecot wrote:

for fts_backend_xxx_lookup, where is specidifed in which field (to, cc, subject, body, from, all) to lookup ?

On 2019-01-06 16:03, Joan Moreau wrote:

For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ???

There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ?

On 2019-01-06 15:41, Joan Moreau wrote:

also, for fts_backend_solr_update_set_build_key -> where is the data (of the hdr_name or the body) ?

On 2019-01-06 14:10, Joan Moreau wrote:

for the "last uid"-> this is not the last added, but the maximum of the UID in the indexed emails, right ?

On 2019-01-06 11:53, Joan Moreau via dovecot wrote:

Thank you

I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?

On 2019-01-06 08:43, Stephan Bosch wrote:

Op 06/01/2019 om 01:00 schreef Joan Moreau: Anyone willing to explain those functions ?

Most notably " get_last_uid" From src/plugins/fts/fts-api.h:

/* Get the last_uid for the mailbox. */ int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box, uint32_t *last_uid_r);

The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this returns the last UID added to the index for the given mailbox and FTS index.

"set_build_key" From src/plugins/fts/fts-api.h:

/* Switch to building index for specified key. If backend doesn't want to index this key, it can return FALSE and caller will skip to next key. */ bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx, const struct fts_backend_build_key *key);

Same file provides outline of what a build_key is.

"build_more" , /* Add more content to the index for the currently specified build key. Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters, but it doesn't need to be NUL-terminated. size contains the data size in bytes, not characters. This function may be called many times and the data block sizes may be small. Backend returns 0 if ok, -1 if build should be aborted. */ int fts_backend_update_build_more(struct fts_backend_update_context *ctx, const unsigned char *data, size_t size);

You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing.

what is refresh versus rescan ? From fts-api.h:

/* Refresh index to make sure we see latest changes from lookups. Returns 0 if ok, -1 if error. */ int fts_backend_refresh(struct fts_backend *backend); /* Go through the entire index and make sure all mails are indexed, and delete any extra mails in the index. */ int fts_backend_rescan(struct fts_backend *backend);

Regards,

Stepham

On January 5, 2019 14:23:10 Joan Moreau via dovecot dovecot@dovecot.org wrote:

Thank Stephan

I basically need to know the role/description of each of the functions of the fts_backend:

struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*

{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };

THank you

On 2019-01-05 08:49, Stephan Bosch wrote:

Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot: Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin

The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.

I know of one FTS plugin not written by Dovecot developers:

https://github.com/atkinsj/fts-elasticsearch

If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).

Regards,

Stephan.

because fts_squat is set to be deleted

Xapian and similar libraries offers a very easy interface for FTS

(and basically, I have done it already)

On 2019-01-07 18:31, Michael Slusarz wrote:

Maybe a dumb question (I admit I haven't followed this thread very closely)...

But why are you writing a new FTS driver? If squat allegedly does everything you need it to do, why don't you just take that plugin and fix it up to do what you need? That seems way easier than trying to create a FTS driver from scratch.

michael

...

On January 7, 2019 at 7:05 AM Joan Moreau via dovecot dovecot@dovecot.org wrote:

Hi

ANyone to answer specifically ?

Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance)

Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ?

Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ?

Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ?

Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ?

Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ?

Ok.

Additional question :

• for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)

• for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?

On 2019-01-08 04:24, Timo Sirainen wrote:

On 7 Jan 2019, at 16.05, Joan Moreau via dovecot dovecot@dovecot.org wrote:

...

Hi

ANyone to answer specifically ?

Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance)

All the mails are always supposed to be indexed from the beginning to the last indexed mail. If there's a gap, indexer first indexes all the missing mails. So the latest UID is supposed to be the greatest UID. (Supporting out-of-order indexing would be rather difficult to keep track of.)

...

Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ?

The idea is that it calls something like:

• build_key(type=hdr, hdr_name=From)
• build_more("tss@iki.fi")
• build_key(type=hdr, hdr_name=Subject)
• build_more("Re: Solr -> Xapian ?")
• build_key(type=body_part)
• build_more("message body piece")
• build_more("message body piece2") ...

...

Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ?

lookup() gets struct mail_search_arg *args, which contains the entire IMAP SEARCH query. This could be used for more or less complex query builders.

In case of a single header search, you should have args->args->hdr_field_name contain the header name and args->args->value.str contain the content you're searching for.

...

Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ?

In case of Xapian it might not matter if it automatically refreshes its indexes between each query. But with some other indexes this could happen:

• IMAP session is opened
• IMAP SEARCH is run, which opens and searches the index
• a new mail is delivered to the mailbox and indexed
• IMAP SEARCH is run. Without refresh() it doesn't see the newly indexed mail and doesn't include it in the search results.

...

Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ?

It's run when "doveadm fts rescan" is run manually. Usually that's only run manually to fix up some brokenness. So it's intended to verify that the current mailbox contents match the FTS indexes:

• If there are any mails in FTS index that no longer exist in the actual mailbox, delete those mails from FTS
• If FTS is missing any mails in the middle of the mailbox, make sure that the next mailbox indexing will index those missing mails. I think currently this basically means reindexing all the mails since the first missing mail, even the mails that are already in the index.

fts-lucene implements this, but other FTS backends are lazy and simply rebuild all mails. Actually fts-solr is bad because it doesn't even delete the extra mails.

Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ? and finally , for fts_backend_xxxx_lookup_multi, why is that backend dependent ?

This function is called only when searching in virtual folders. So for example the virtual "All mails" folder, which would contain all mails in all folders. In that case the boxes[] would contain a list of user's all folders, except Trash and Spam. If lookup_multi() isn't implemented (left to NULL), the search is run separately via lookup() for each folder. With lookup_multi() there can be just one lookup, and the backend can filter only the wanted folders and return them directly. So it's an optimization for FTS indexes that support user-global searches rather than only per-folder searches.

...

static int fts_backend_xapian_lookup_multi(struct fts_backend *_backend, struct mailbox *const boxes[], struct mail_search_arg *args, enum fts_lookup_flags flags, struct fts_multi_result *result) { struct xapian_fts_backend_update_context *ctx = (struct xapian_fts_backend_update_context *)_ctx;

int i=0;

while(boxes[i]!=NULL) { if(fts_backend_xapian_lookup(backend,box[i],args,flags,result->box_results[i])<0) return -1; i++; } return 0; }

See fts_backend_lookup_multi() - if you leave lookup_multi=NULL it basically does this.

...

For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ???

rescan() is about fixing up a more or less broken index, or simply to verify that it's all ok. So core doesn't know what messages exist in the FTS index and can't request specific reindexing or expunging. I guess an alternative API could have been to have functions that iterate through all mails in the index, and use that to implement rescan in core. Now thinking about it, that sounds like a simpler and better way.

optimize() is currently done only when explicitly running "doveadm fts optimize", which requests running a slower index optimization. Depends on the FTS backend whether this is useful or not.

...

There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ?

Normally when mails are expunged, update_expunge() is called to notify FTS backend that it should delete the mail also from FTS index.

...

.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*

You probably want to use FTS_BACKEND_FLAG_FUZZY_SEARCH only like Solr. See enum fts_backend_flags in fts-api-private.h

I managed to deal with the namespace issue (updated makefile.am)

However, I reach :

../../../src/lib/compat.h:207:19: error: conflicting declaration of 'ssize_t i_my_pread(int, void*, size_t, __off_t)' with 'C' linkage # define pread i_my_pread ^~~~~~~~~~ ../../../src/lib/compat.h:210:9: note: previous declaration with 'C++' linkage ssize_t i_my_pread(int fd, void *buf, size_t count, off_t offset); ^~~~~~~~~~ ../../../src/lib/compat.h:208:20: error: conflicting declaration of 'ssize_t i_my_pwrite(int, const void*, size_t, __off_t)' with 'C' linkage # define pwrite i_my_pwrite

Any help welcome

Hi,

I figured out the "namespace" issue

Remaining questions are :

1 - WHat does represent "subargs" in mail_search_args

2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)

3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?

4 - How to update configure.ac & additional files to add the "--with-xapian" wichi will test for libxapian presence and add it to the build ?

Thank you

On 2019-01-08 04:24, Timo Sirainen wrote:

On 7 Jan 2019, at 16.05, Joan Moreau via dovecot dovecot@dovecot.org wrote: Hi

ANyone to answer specifically ?

Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance) All the mails are always supposed to be indexed from the beginning to the last indexed mail. If there's a gap, indexer first indexes all the missing mails. So the latest UID is supposed to be the greatest UID. (Supporting out-of-order indexing would be rather difficult to keep track of.)

Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ? The idea is that it calls something like:

• build_key(type=hdr, hdr_name=From)
• build_more("tss@iki.fi")
• build_key(type=hdr, hdr_name=Subject)
• build_more("Re: Solr -> Xapian ?")
• build_key(type=body_part)
• build_more("message body piece")
• build_more("message body piece2") ...

Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ? lookup() gets struct mail_search_arg *args, which contains the entire IMAP SEARCH query. This could be used for more or less complex query builders.

In case of a single header search, you should have args->args->hdr_field_name contain the header name and args->args->value.str contain the content you're searching for.

Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ? In case of Xapian it might not matter if it automatically refreshes its indexes between each query. But with some other indexes this could happen:

• IMAP session is opened
• IMAP SEARCH is run, which opens and searches the index
• a new mail is delivered to the mailbox and indexed
• IMAP SEARCH is run. Without refresh() it doesn't see the newly indexed mail and doesn't include it in the search results.

Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ? It's run when "doveadm fts rescan" is run manually. Usually that's only run manually to fix up some brokenness. So it's intended to verify that the current mailbox contents match the FTS indexes: that the next mailbox indexing will index those missing mails. I think

• If there are any mails in FTS index that no longer exist in the actual mailbox, delete those mails from FTS
• If FTS is missing any mails in the middle of the mailbox, make sure

currently this basically means reindexing all the mails since the first missing mail, even the mails that are already in the index.

fts-lucene implements this, but other FTS backends are lazy and simply rebuild all mails. Actually fts-solr is bad because it doesn't even delete the extra mails.

Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ?and finally , for fts_backend_xxxx_lookup_multi, why is that backend dependent ? This function is called only when searching in virtual folders. So for example the virtual "All mails" folder, which would contain all mails in all folders. In that case the boxes[] would contain a list of user's all folders, except Trash and Spam. If lookup_multi() isn't implemented (left to NULL), the search is run separately via lookup() for each folder. With lookup_multi() there can be just one lookup, and the backend can filter only the wanted folders and return them directly. So it's an optimization for FTS indexes that support user-global searches rather than only per-folder searches.

static int fts_backend_xapian_lookup_multi(struct fts_backend *_backend, struct mailbox *const boxes[], struct mail_search_arg *args, enum fts_lookup_flags flags, struct fts_multi_result *result) { struct xapian_fts_backend_update_context *ctx = (struct xapian_fts_backend_update_context *)_ctx;

int i=0;

while(boxes[i]!=NULL) { if(fts_backend_xapian_lookup(backend,box[i],args,flags,result->box_results[i])<0) return -1; i++; } return 0; } See fts_backend_lookup_multi() - if you leave lookup_multi=NULL it basically does this.

For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ??? rescan() is about fixing up a more or less broken index, or simply to verify that it's all ok. So core doesn't know what messages exist in the FTS index and can't request specific reindexing or expunging. I guess an alternative API could have been to have functions that iterate through all mails in the index, and use that to implement rescan in core. Now thinking about it, that sounds like a simpler and better way.

optimize() is currently done only when explicitly running "doveadm fts optimize", which requests running a slower index optimization. Depends on the FTS backend whether this is useful or not.

There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ? Normally when mails are expunged, update_expunge() is called to notify FTS backend that it should delete the mail also from FTS index.

.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?* You probably want to use FTS_BACKEND_FLAG_FUZZY_SEARCH only like Solr. See enum fts_backend_flags in fts-api-private.h

There is no point into a separate plugin, the purpose is to replace squat as the default fts (solr being a nightmare)

On 2019-01-11 18:23, Aki Tuomi wrote:

I would recommend making this a standalone plugin for now instead of trying to keep it in core fts.

Aki

...

On 11 January 2019 at 18:40 Joan Moreau via dovecot < dovecot@dovecot.org> wrote:

I managed to deal with the namespace issue (updated makefile.am)

However, I reach :

../../../src/lib/compat.h:207:19: error: conflicting declaration of 'ssize_t i_my_pread(int, void*, size_t, __off_t)' with 'C' linkage # define pread i_my_pread ^~~~~~~~~~ ../../../src/lib/compat.h:210:9: note: previous declaration with 'C++' linkage ssize_t i_my_pread(int fd, void *buf, size_t count, off_t offset); ^~~~~~~~~~ ../../../src/lib/compat.h:208:20: error: conflicting declaration of 'ssize_t i_my_pwrite(int, const void*, size_t, __off_t)' with 'C' linkage # define pwrite i_my_pwrite

Any help welcome

Hi,

I figured out the "namespace" issue

Remaining questions are :

1 - WHat does represent "subargs" in mail_search_args

2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)

3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?

4 - How to update configure.ac & additional files to add the "--with-xapian" wichi will test for libxapian presence and add it to the build ?

Thank you

On 2019-01-08 04:24, Timo Sirainen wrote:

On 7 Jan 2019, at 16.05, Joan Moreau via dovecot < dovecot@dovecot.org> wrote: Hi

ANyone to answer specifically ?

Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance) All the mails are always supposed to be indexed from the beginning to the last indexed mail. If there's a gap, indexer first indexes all the missing mails. So the latest UID is supposed to be the greatest UID. (Supporting out-of-order indexing would be rather difficult to keep track of.)

Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ? The idea is that it calls something like:

• build_key(type=hdr, hdr_name=From)
• build_more(" tss@iki.fi")
• build_key(type=hdr, hdr_name=Subject)
• build_more("Re: Solr -> Xapian ?")
• build_key(type=body_part)
• build_more("message body piece")
• build_more("message body piece2") ...

Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ? lookup() gets struct mail_search_arg *args, which contains the entire IMAP SEARCH query. This could be used for more or less complex query builders.

In case of a single header search, you should have args->args->hdr_field_name contain the header name and args->args->value.str contain the content you're searching for.

Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ? In case of Xapian it might not matter if it automatically refreshes its indexes between each query. But with some other indexes this could happen:

• IMAP session is opened
• IMAP SEARCH is run, which opens and searches the index
• a new mail is delivered to the mailbox and indexed
• IMAP SEARCH is run. Without refresh() it doesn't see the newly indexed mail and doesn't include it in the search results.

Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ? It's run when "doveadm fts rescan" is run manually. Usually that's only run manually to fix up some brokenness. So it's intended to verify that the current mailbox contents match the FTS indexes: that the next mailbox indexing will index those missing mails. I think

• If there are any mails in FTS index that no longer exist in the actual mailbox, delete those mails from FTS
• If FTS is missing any mails in the middle of the mailbox, make sure

currently this basically means reindexing all the mails since the first missing mail, even the mails that are already in the index.

fts-lucene implements this, but other FTS backends are lazy and simply rebuild all mails. Actually fts-solr is bad because it doesn't even delete the extra mails.

Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ?and finally , for fts_backend_xxxx_lookup_multi, why is that backend dependent ? This function is called only when searching in virtual folders. So for example the virtual "All mails" folder, which would contain all mails in all folders. In that case the boxes[] would contain a list of user's all folders, except Trash and Spam. If lookup_multi() isn't implemented (left to NULL), the search is run separately via lookup() for each folder. With lookup_multi() there can be just one lookup, and the backend can filter only the wanted folders and return them directly. So it's an optimization for FTS indexes that support user-global searches rather than only per-folder searches.

static int fts_backend_xapian_lookup_multi(struct fts_backend *_backend, struct mailbox *const boxes[], struct mail_search_arg *args, enum fts_lookup_flags flags, struct fts_multi_result *result) { struct xapian_fts_backend_update_context *ctx = (struct xapian_fts_backend_update_context *)_ctx;

int i=0;

while(boxes[i]!=NULL) { if(fts_backend_xapian_lookup(backend,box[i],args,flags,result->box_results[i])<0) return -1; i++; } return 0; } See fts_backend_lookup_multi() - if you leave lookup_multi=NULL it basically does this.

For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ??? rescan() is about fixing up a more or less broken index, or simply to verify that it's all ok. So core doesn't know what messages exist in the FTS index and can't request specific reindexing or expunging. I guess an alternative API could have been to have functions that iterate through all mails in the index, and use that to implement rescan in core. Now thinking about it, that sounds like a simpler and better way.

optimize() is currently done only when explicitly running "doveadm fts optimize", which requests running a slower index optimization. Depends on the FTS backend whether this is useful or not.

There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ? Normally when mails are expunged, update_expunge() is called to notify FTS backend that it should delete the mail also from FTS index.

.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?* You probably want to use FTS_BACKEND_FLAG_FUZZY_SEARCH only like Solr. See enum fts_backend_flags in fts-api-private.h

---

Aki Tuomi

STATUS

• Alpha code is written and compiling now. (attached)

• I would like to start testing. However, there is an error when starting dovecot (git) :

Error: Couldn't load required plugin /usr/lib/dovecot/lib21_fts_xapian_plugin.so: dlopen() failed: /usr/lib/dovecot/lib21_fts_xapian_plugin.so: undefined symbol: _Z30fts_backend_default_can_lookupP11fts_backendPK15mail_search_arg

ldd shows that fts lib is properly linked: # ldd /usr/lib/dovecot/lib21_fts_xapian_plugin.so (...) lib20_fts_plugin.so => /usr/lib/dovecot/lib20_fts_plugin.so (0x00007f25f75e0000) (...) libxapian.so.30 => /usr/lib/libxapian.so.30 (0x00007fe3a51e2000)

Your help very welcome

PENDING QUESTIONS

1 - WHat does represent "subargs" in mail_search_args

2 - for rescan : who is responsible for passing again the new email ? Is

the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)

3 - for get_last_uid : this uncertainity is very unclear. "If there is a

gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?

Thank you

The change of "Extern C" suggested by Timo actually solved the situation

Now, further question :

I put a "i_warning" at each of my functions, and I see in the log :

Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970>ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag: Warning: fts_backend_xapian_alloc Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970>ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag: Warning: fts_backend_xapian_init Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970>ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag: Warning: fts_xapian init (3,25) Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970>ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag: Warning: fts_backend_xapian_get_last_uid Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970>ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag: Error: Mailbox XXX: Status lookup failed: Internal error occurred. Refer to server log for more information. [2019-01-12 10:33:27] Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970>ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag: Warning: fts_backend_xapian_deinit Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970>ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag: Warning: fts_backend_xapian_unset_box Jan 12 10:33:27 lda(jom@grosjo.net)<31367><ms9zEnfCOVyHegAA0thIag>: Warning: fts_backend_xapian_deinit

The error comes from the fact taht the "get_last_uid" is called without selecting a mailbox first, and therefore there is of course no uid.

How come ?

On 2019-01-12 10:50, Aki Tuomi wrote:

Did you remember to load fts first?

mail_plugins =$mail_plugins fts fts_xapian

Aki

...

On 12 January 2019 at 10:37 Joan Moreau via dovecot < dovecot@dovecot.org> wrote:

STATUS

• Alpha code is written and compiling now. (attached)

• I would like to start testing. However, there is an error when starting dovecot (git) :

Error: Couldn't load required plugin /usr/lib/dovecot/lib21_fts_xapian_plugin.so: dlopen() failed: /usr/lib/dovecot/lib21_fts_xapian_plugin.so: undefined symbol: _Z30fts_backend_default_can_lookupP11fts_backendPK15mail_search_arg

ldd shows that fts lib is properly linked: # ldd /usr/lib/dovecot/lib21_fts_xapian_plugin.so (...) lib20_fts_plugin.so => /usr/lib/dovecot/lib20_fts_plugin.so (0x00007f25f75e0000) (...) libxapian.so.30 => /usr/lib/libxapian.so.30 (0x00007fe3a51e2000)

Your help very welcome

PENDING QUESTIONS

1 - WHat does represent "subargs" in mail_search_args

2 - for rescan : who is responsible for passing again the new email ? Is

the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)

3 - for get_last_uid : this uncertainity is very unclear. "If there is a

gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?

Thank you

---

Aki Tuomi

On 11 Jan 2019, at 21.23, Joan Moreau via dovecot dovecot@dovecot.org wrote:

The below patch resolves the compilation error

$ diff -p compat.h compat.h.joan *** compat.h 2019-01-11 20:21:00.726625427 +0100 --- compat.h.joan 2019-01-11 20:14:41.729109919 +0100 *************** struct iovec; *** 202,207 **** --- 202,211 ---- ssize_t i_my_writev(int fd, const struct iovec *iov, int iov_len); #endif

• #ifdef __cplusplus
• extern "C" {
• #endif

You should put this extern "C" into the C++ file you're creating. See for example how fts-lucene/lucene-wrapper.cc does this.

1 - WHat does represent "subargs" in mail_search_args

It's set only for SEARCH_OR and SEARCH_SUB. So for example:

SEARCH TEXT foo TEXT bar TEXT baz

results in:

type=SEARCH_SUB value.subargs = ( { type=SEARCH, value.str="foo" }, { type=SEARCH, value.str="bar" }, { type=SEARCH, value.str="baz" }, )

Or similarly if there's SEARCH OR foo OR TEXT bar TEXT baz or some other combination of OR/ANDs.

2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)

The next indexing run is responsible for it. If you return get_last_uid=0, then indexer starts feeding you all mails. So fts backend doesn't have to know about it.

3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?

I mean if get_last_uid() returns for example 100, it means that UIDs 1..100 have been indexed by the FTS backend. It's possible that at this point there are already mails with UIDs 101..200 in the folder. So when UID=201 is delivered, indexer notices that FTS backend has only UIDs 1..100 indexed so far, and starts feeding it UIDs 101..201 in that order.

You can implement get_last_uid() simply by keeping track of it in dovecot.index* files, similar to how Lucene and Solr already do it with fts_index_get_header() / fts_index_set_header(). They also have a fallback that if the index doesn't have the last_uid value, they do a slower search from the Lucene/Solr index to find the last UID.

additionally, my logic is that the backend stores one databalse per mailox in /xapian-indexes (in the "root" dir of the user), the name od the database is the GUID of the mailbox

For INBOX, that works perfectly, and database is properly createdm and backed starts indexing all emails

For other folder, somehow, the process can not access that (root) folder.

Am I missing something ?

On 2019-01-12 17:37, Joan Moreau wrote:

THank you

Now, for the results

I see the member of fts_result is :

ARRAY_TYPE(seq_range) definite_uids;

I have the UID as a aray of uint32_t *

How to put my UIDs into this "definite_uids" ? Obviously this is not a simple array/pointer. How to say someting similar to result->definite_uids[1]=my_uid ?

On 2019-01-12 10:25, Timo Sirainen wrote: On 11 Jan 2019, at 21.23, Joan Moreau via dovecot dovecot@dovecot.org wrote: The below patch resolves the compilation error

$ diff -p compat.h compat.h.joan *** compat.h 2019-01-11 20:21:00.726625427 +0100 --- compat.h.joan 2019-01-11 20:14:41.729109919 +0100 *************** struct iovec; *** 202,207 **** --- 202,211 ---- ssize_t i_my_writev(int fd, const struct iovec *iov, int iov_len); #endif

• #ifdef __cplusplus
• extern "C" {
• #endif

You should put this extern "C" into the C++ file you're creating. See for example how fts-lucene/lucene-wrapper.cc does this.

1 - WHat does represent "subargs" in mail_search_args It's set only for SEARCH_OR and SEARCH_SUB. So for example:

SEARCH TEXT foo TEXT bar TEXT baz

results in:

type=SEARCH_SUB value.subargs = ( { type=SEARCH, value.str="foo" }, { type=SEARCH, value.str="bar" }, { type=SEARCH, value.str="baz" }, )

Or similarly if there's SEARCH OR foo OR TEXT bar TEXT baz or some other combination of OR/ANDs. 2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large) The next indexing run is responsible for it. If you return get_last_uid=0, then indexer starts feeding you all mails. So fts backend doesn't have to know about it.

3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ? I mean if get_last_uid() returns for example 100, it means that UIDs 1..100 have been indexed by the FTS backend. It's possible that at this point there are already mails with UIDs 101..200 in the folder. So when UID=201 is delivered, indexer notices that FTS backend has only UIDs 1..100 indexed so far, and starts feeding it UIDs 101..201 in that order.

You can implement get_last_uid() simply by keeping track of it in dovecot.index* files, similar to how Lucene and Solr already do it with fts_index_get_header() / fts_index_set_header(). They also have a fallback that if the index doesn't have the last_uid value, they do a slower search from the Lucene/Solr index to find the last UID.