https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
- The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
- "doveadm stats dump" now returns two decimals for the "avg" field.
- Added push notification driver that uses a Lua script
- Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
- Added "doveadm mailbox cache purge" command.
- Added events API support for Lua scripts
- doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.
- director: Kicking a user crashes if login process is very slow
- pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
- auth: Fix crypt() segfault with glibc-2.28+
- imap: Running UID FILTER script with errors assert-crashes
- dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
- dict clients may have been using 100% CPU while waiting for dict server to finish commands.
- doveadm user: Fixed user listing via HTTP API
- All levels of Cassandra log messages were logged as Dovecot errors.
- http/smtp client may have crashed after SSL handshake
- Lua auth converted strings that looked like numbers into numbers.
Associated release for Pigeonhole:
https://pigeonhole.dovecot.org/releases/2.3/dovecot-2.3-pigeonhole-0.5.4.tar... https://pigeonhole.dovecot.org/releases/2.3/dovecot-2.3-pigeonhole-0.5.4.tar... Binary packages included in https://repo.dovecot.org/
- Adjustments to several changes in Dovecot v2.3.4 make this Pigeonhole release dependent on that Dovecot release; it will not compile against older Dovecot versions. And, conversely, you need to upgrade Pigeonhole when upgrading Dovecot to v2.3.4.
- The changes regarding the default postmaster_address in Dovecot v2.3.4 mainly apply to Pigeonhole. The new default should work for all existing installations, thereby fixing several reported v2.3/v0.5 migration problems.
- IMAP FILTER=SIEVE capability: Fix assert crash occurring when running UID FILTER on a Sieve script with errors.
Op 23-11-2018 om 13:29 schreef Timo Sirainen:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
- The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
- "doveadm stats dump" now returns two decimals for the "avg" field.
- Added push notification driver that uses a Lua script
- Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
- Added "doveadm mailbox cache purge" command.
- Added events API support for Lua scripts
- doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.
- director: Kicking a user crashes if login process is very slow
- pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
- auth: Fix crypt() segfault with glibc-2.28+
- imap: Running UID FILTER script with errors assert-crashes
- dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
- dict clients may have been using 100% CPU while waiting for dict server to finish commands.
- doveadm user: Fixed user listing via HTTP API
- All levels of Cassandra log messages were logged as Dovecot errors.
- http/smtp client may have crashed after SSL handshake
- Lua auth converted strings that looked like numbers into numbers.
Please remove me from your mailing list thanks you
On 23/11/2018 13:29, Timo Sirainen wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
- The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
- "doveadm stats dump" now returns two decimals for the "avg" field.
- Added push notification driver that uses a Lua script
- Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
- Added "doveadm mailbox cache purge" command.
- Added events API support for Lua scripts
- doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.
- director: Kicking a user crashes if login process is very slow
- pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
- auth: Fix crypt() segfault with glibc-2.28+
- imap: Running UID FILTER script with errors assert-crashes
- dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
- dict clients may have been using 100% CPU while waiting for dict server to finish commands.
- doveadm user: Fixed user listing via HTTP API
- All levels of Cassandra log messages were logged as Dovecot errors.
- http/smtp client may have crashed after SSL handshake
- Lua auth converted strings that looked like numbers into numbers.
Dovecot-news mailing list Dovecot-news@dovecot.org https://dovecot.org/mailman/listinfo/dovecot-news
You can do it yourself. There is a link below every mail (and also this one) from this list where you can unsubscribe.
On 23 Nov 2018, at 13:49, Yann Shukor <yann.shukor@azurtem.com> wrote:
Please remove me from your mailing list thanks you
Dovecot-news mailing list Dovecot-news@dovecot.org https://dovecot.org/mailman/listinfo/dovecot-news
On Fri, 23 Nov 2018 at 15:29, Timo Sirainen <tss@iki.fi> wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
- The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
- "doveadm stats dump" now returns two decimals for the "avg" field.
- Added push notification driver that uses a Lua script
- Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
- Added "doveadm mailbox cache purge" command.
- Added events API support for Lua scripts
- doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.
- director: Kicking a user crashes if login process is very slow
- pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
- auth: Fix crypt() segfault with glibc-2.28+
- imap: Running UID FILTER script with errors assert-crashes
- dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
- dict clients may have been using 100% CPU while waiting for dict server to finish commands.
- doveadm user: Fixed user listing via HTTP API
- All levels of Cassandra log messages were logged as Dovecot errors.
- http/smtp client may have crashed after SSL handshake
- Lua auth converted strings that looked like numbers into numbers.
FreeBSD 9.3 (i386):
Making all in lib-master gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib -I../../src/lib-dns -I../../src/lib-test -I../../src/lib-settings -I../../src/lib-ssl-iostream -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\" -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\" -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\" -DBINDIR=\""/opt/dovecot2.3/bin"\" -std=gnu99 -g -O2 -fstack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 -I/usr/local/include -MT test-event-stats.o -MD -MP -MF .deps/test-event-stats.Tpo -c -o test-event-stats.o test-event-stats.c test-event-stats.c: In function 'kill_stats_child': test-event-stats.c:101: warning: implicit declaration of function 'kill' test-event-stats.c:101: error: 'SIGKILL' undeclared (first use in this function) test-event-stats.c:101: error: (Each undeclared identifier is reported only once test-event-stats.c:101: error: for each function it appears in.) test-event-stats.c: In function 'test_no_merging2': test-event-stats.c:361: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c: In function 'test_no_merging3': test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'uint64_t' test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned int', but argument 6 has type 'uint64_t' test-event-stats.c: In function 'test_merge_events2': test-event-stats.c:452: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c: In function 'test_skip_parents': test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'uint64_t' test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned int', but argument 6 has type 'uint64_t' test-event-stats.c: In function 'test_merge_events_skip_parents': test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'uint64_t' test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned int', but argument 6 has type 'uint64_t' *** Error code 1
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."
On Fri, Nov 23, 2018 at 04:02:27PM +0300, Odhiambo Washington wrote:
On Fri, 23 Nov 2018 at 15:29, Timo Sirainen <tss@iki.fi> wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
* The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used. * "doveadm stats dump" now returns two decimals for the "avg" field.
+ Added push notification driver that uses a Lua script + Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events + Added "doveadm mailbox cache purge" command. + Added events API support for Lua scripts + doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version. - director: Kicking a user crashes if login process is very slow - pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent. - auth: Fix crypt() segfault with glibc-2.28+ - imap: Running UID FILTER script with errors assert-crashes - dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved. - dict clients may have been using 100% CPU while waiting for dict server to finish commands. - doveadm user: Fixed user listing via HTTP API - All levels of Cassandra log messages were logged as Dovecot errors. - http/smtp client may have crashed after SSL handshake - Lua auth converted strings that looked like numbers into numbers.
FreeBSD 9.3 (i386):
Making all in lib-master gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib -I../../src/lib-dns -I../../src/lib-test -I../../src/lib-settings -I../../src/lib-ssl-iostream -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\" -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\" -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\" -DBINDIR=\""/opt/dovecot2.3/bin"\" -std=gnu99 -g -O2 -fstack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 -I/usr/local/include -MT test-event-stats.o -MD -MP -MF .deps/test-event-stats.Tpo -c -o test-event-stats.o test-event-stats.c test-event-stats.c: In function 'kill_stats_child': test-event-stats.c:101: warning: implicit declaration of function 'kill' test-event-stats.c:101: error: 'SIGKILL' undeclared (first use in this function) test-event-stats.c:101: error: (Each undeclared identifier is reported only once test-event-stats.c:101: error: for each function it appears in.) test-event-stats.c: In function 'test_no_merging2': test-event-stats.c:361: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c: In function 'test_no_merging3': test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'uint64_t' test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned int', but argument 6 has type 'uint64_t' test-event-stats.c: In function 'test_merge_events2': test-event-stats.c:452: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c: In function 'test_skip_parents': test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'uint64_t' test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned int', but argument 6 has type 'uint64_t' test-event-stats.c: In function 'test_merge_events_skip_parents': test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'uint64_t' test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned int', but argument 6 has type 'uint64_t' *** Error code 1
Are you able to upgrade to FreeBSD 10 or 11 ?
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."
-- Member - Liberal International This is doctor@@nl2k.ab.ca Ici doctor@@nl2k.ab.ca Yahweh, Queen & country!Never Satan President Republic!Beware AntiChrist rising! https://www.empire.kred/ROOTNK?t=94a1f39b Look at Psalms 14 and 53 on Atheism sMerry Christmas 2018 and Happy New Year 2019!!
On Fri, 23 Nov 2018 at 17:30, The Doctor <doctor@doctor.nl2k.ab.ca> wrote:
On Fri, Nov 23, 2018 at 04:02:27PM +0300, Odhiambo Washington wrote:
On Fri, 23 Nov 2018 at 15:29, Timo Sirainen <tss@iki.fi> wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
- The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
- "doveadm stats dump" now returns two decimals for the "avg" field.
- Added push notification driver that uses a Lua script
- Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
- Added "doveadm mailbox cache purge" command.
- Added events API support for Lua scripts
- doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.
- director: Kicking a user crashes if login process is very slow
- pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
- auth: Fix crypt() segfault with glibc-2.28+
- imap: Running UID FILTER script with errors assert-crashes
- dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
- dict clients may have been using 100% CPU while waiting for dict server to finish commands.
- doveadm user: Fixed user listing via HTTP API
- All levels of Cassandra log messages were logged as Dovecot errors.
- http/smtp client may have crashed after SSL handshake
- Lua auth converted strings that looked like numbers into numbers.
FreeBSD 9.3 (i386):
Making all in lib-master gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib -I../../src/lib-dns -I../../src/lib-test -I../../src/lib-settings -I../../src/lib-ssl-iostream -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\" -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\" -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\" -DBINDIR=\""/opt/dovecot2.3/bin"\" -std=gnu99 -g -O2 -fstack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 -I/usr/local/include -MT test-event-stats.o -MD -MP -MF .deps/test-event-stats.Tpo -c -o test-event-stats.o test-event-stats.c test-event-stats.c: In function 'kill_stats_child': test-event-stats.c:101: warning: implicit declaration of function 'kill' test-event-stats.c:101: error: 'SIGKILL' undeclared (first use in this function) test-event-stats.c:101: error: (Each undeclared identifier is reported only once test-event-stats.c:101: error: for each function it appears in.) test-event-stats.c: In function 'test_no_merging2': test-event-stats.c:361: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c: In function 'test_no_merging3': test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'uint64_t' test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned int', but argument 6 has type 'uint64_t' test-event-stats.c: In function 'test_merge_events2': test-event-stats.c:452: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c: In function 'test_skip_parents': test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'uint64_t' test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned int', but argument 6 has type 'uint64_t' test-event-stats.c: In function 'test_merge_events_skip_parents': test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'uint64_t' test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned int', but argument 6 has type 'uint64_t' *** Error code 1
Are you able to upgrade to FreeBSD 10 or 11 ?
I will be upgrading the servers to 11.2 soon :)
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."
On 11/23/2018 9:39 AM, Odhiambo Washington wrote:
On Fri, 23 Nov 2018 at 17:30, The Doctor <doctor@doctor.nl2k.ab.ca <mailto:doctor@doctor.nl2k.ab.ca>> wrote:
On Fri, Nov 23, 2018 at 04:02:27PM +0300, Odhiambo Washington wrote: > On Fri, 23 Nov 2018 at 15:29, Timo Sirainen <tss@iki.fi <mailto:tss@iki.fi>> wrote: > > > https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz > > https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig > > Binary packages in https://repo.dovecot.org/ > > > > * The default postmaster_address is now "postmaster@<user domain or > > server hostname>". If username contains the @domain part, that's > > used. If not, then the server's hostname is used. > > * "doveadm stats dump" now returns two decimals for the "avg" field. > > > > + Added push notification driver that uses a Lua script > > + Added new SQL, DNS and connection events. > > See https://wiki2.dovecot.org/Events > > + Added "doveadm mailbox cache purge" command. > > + Added events API support for Lua scripts > > + doveadm force-resync -f parameter performs "index fsck" while opening > > the index. This may be useful to fix some types of broken index files. > > This may become the default behavior in a later version. > > - director: Kicking a user crashes if login process is very slow > > - pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages > > unless QUIT is sent. > > - auth: Fix crypt() segfault with glibc-2.28+ > > - imap: Running UID FILTER script with errors assert-crashes > > - dsync, pop3-migration: POP3 UIDLs weren't added to > > dovecot.index.cache while mails were saved. > > - dict clients may have been using 100% CPU while waiting for dict > > server to finish commands. > > - doveadm user: Fixed user listing via HTTP API > > - All levels of Cassandra log messages were logged as Dovecot errors. > > - http/smtp client may have crashed after SSL handshake > > - Lua auth converted strings that looked like numbers into numbers. > > > > > > FreeBSD 9.3 (i386): > > > Making all in lib-master > gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib -I../../src/lib-dns > -I../../src/lib-test -I../../src/lib-settings > -I../../src/lib-ssl-iostream > -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\" > -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\" > -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\" > -DBINDIR=\""/opt/dovecot2.3/bin"\" -std=gnu99 -g -O2 -fstack-protector > -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes > -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 > -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 > -I/usr/local/include -MT test-event-stats.o -MD -MP -MF > .deps/test-event-stats.Tpo -c -o test-event-stats.o test-event-stats.c > test-event-stats.c: In function 'kill_stats_child': > test-event-stats.c:101: warning: implicit declaration of function 'kill' > test-event-stats.c:101: error: 'SIGKILL' undeclared (first use in this > function) > test-event-stats.c:101: error: (Each undeclared identifier is reported only > once > test-event-stats.c:101: error: for each function it appears in.) > test-event-stats.c: In function 'test_no_merging2': > test-event-stats.c:361: warning: format '%lu' expects type 'long unsigned > int', but argument 2 has type 'uint64_t' > test-event-stats.c: In function 'test_no_merging3': > test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned > int', but argument 2 has type 'uint64_t' > test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned > int', but argument 4 has type 'uint64_t' > test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned > int', but argument 6 has type 'uint64_t' > test-event-stats.c: In function 'test_merge_events2': > test-event-stats.c:452: warning: format '%lu' expects type 'long unsigned > int', but argument 2 has type 'uint64_t' > test-event-stats.c: In function 'test_skip_parents': > test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned > int', but argument 2 has type 'uint64_t' > test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned > int', but argument 4 has type 'uint64_t' > test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned > int', but argument 6 has type 'uint64_t' > test-event-stats.c: In function 'test_merge_events_skip_parents': > test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned > int', but argument 2 has type 'uint64_t' > test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned > int', but argument 4 has type 'uint64_t' > test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned > int', but argument 6 has type 'uint64_t' > *** Error code 1 > Are you able to upgrade to FreeBSD 10 or 11 ?I will be upgrading the servers to 11.2 soon :)
The OS version is irrelevant.
On 23 November 2018 at 17:46 Brad Smith <brad@comstyle.com> wrote:
On 11/23/2018 9:39 AM, Odhiambo Washington wrote:
On Fri, 23 Nov 2018 at 17:30, The Doctor <doctor@doctor.nl2k.ab.ca <mailto:doctor@doctor.nl2k.ab.ca>> wrote:
On Fri, Nov 23, 2018 at 04:02:27PM +0300, Odhiambo Washington wrote: > On Fri, 23 Nov 2018 at 15:29, Timo Sirainen <tss@iki.fi <mailto:tss@iki.fi>> wrote: > > > https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz > > https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig > > Binary packages in https://repo.dovecot.org/ > > > > * The default postmaster_address is now "postmaster@<user domain or > > server hostname>". If username contains the @domain part, that's > > used. If not, then the server's hostname is used. > > * "doveadm stats dump" now returns two decimals for the "avg" field. > > > > + Added push notification driver that uses a Lua script > > + Added new SQL, DNS and connection events. > > See https://wiki2.dovecot.org/Events > > + Added "doveadm mailbox cache purge" command. > > + Added events API support for Lua scripts > > + doveadm force-resync -f parameter performs "index fsck" while opening > > the index. This may be useful to fix some types of broken index files. > > This may become the default behavior in a later version. > > - director: Kicking a user crashes if login process is very slow > > - pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages > > unless QUIT is sent. > > - auth: Fix crypt() segfault with glibc-2.28+ > > - imap: Running UID FILTER script with errors assert-crashes > > - dsync, pop3-migration: POP3 UIDLs weren't added to > > dovecot.index.cache while mails were saved. > > - dict clients may have been using 100% CPU while waiting for dict > > server to finish commands. > > - doveadm user: Fixed user listing via HTTP API > > - All levels of Cassandra log messages were logged as Dovecot errors. > > - http/smtp client may have crashed after SSL handshake > > - Lua auth converted strings that looked like numbers into numbers. > > > > > > FreeBSD 9.3 (i386): > > > Making all in lib-master > gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib -I../../src/lib-dns > -I../../src/lib-test -I../../src/lib-settings > -I../../src/lib-ssl-iostream > -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\" > -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\" > -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\" > -DBINDIR=\""/opt/dovecot2.3/bin"\" -std=gnu99 -g -O2 -fstack-protector > -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes > -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 > -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 > -I/usr/local/include -MT test-event-stats.o -MD -MP -MF > .deps/test-event-stats.Tpo -c -o test-event-stats.o test-event-stats.c > test-event-stats.c: In function 'kill_stats_child': > test-event-stats.c:101: warning: implicit declaration of function 'kill' > test-event-stats.c:101: error: 'SIGKILL' undeclared (first use in this > function) > test-event-stats.c:101: error: (Each undeclared identifier is reported only > once > test-event-stats.c:101: error: for each function it appears in.) > test-event-stats.c: In function 'test_no_merging2': > test-event-stats.c:361: warning: format '%lu' expects type 'long unsigned > int', but argument 2 has type 'uint64_t' > test-event-stats.c: In function 'test_no_merging3': > test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned > int', but argument 2 has type 'uint64_t' > test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned > int', but argument 4 has type 'uint64_t' > test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned > int', but argument 6 has type 'uint64_t' > test-event-stats.c: In function 'test_merge_events2': > test-event-stats.c:452: warning: format '%lu' expects type 'long unsigned > int', but argument 2 has type 'uint64_t' > test-event-stats.c: In function 'test_skip_parents': > test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned > int', but argument 2 has type 'uint64_t' > test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned > int', but argument 4 has type 'uint64_t' > test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned > int', but argument 6 has type 'uint64_t' > test-event-stats.c: In function 'test_merge_events_skip_parents': > test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned > int', but argument 2 has type 'uint64_t' > test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned > int', but argument 4 has type 'uint64_t' > test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned > int', but argument 6 has type 'uint64_t' > *** Error code 1 > Are you able to upgrade to FreeBSD 10 or 11 ?I will be upgrading the servers to 11.2 soon :)
The OS version is irrelevant.
Fix for this format issue and the signal.h issue can be found in master now, and you can download the patch at
https://github.com/dovecot/core/compare/10048229%5E...de42b54a.patch
Aki
On Sat, 24 Nov 2018 at 10:56, Aki Tuomi <aki.tuomi@open-xchange.com> wrote:
On 23 November 2018 at 17:46 Brad Smith <brad@comstyle.com> wrote:
On 11/23/2018 9:39 AM, Odhiambo Washington wrote:
On Fri, 23 Nov 2018 at 17:30, The Doctor <doctor@doctor.nl2k.ab.ca <mailto:doctor@doctor.nl2k.ab.ca>> wrote:
On Fri, Nov 23, 2018 at 04:02:27PM +0300, Odhiambo Washingtonwrote:
> On Fri, 23 Nov 2018 at 15:29, Timo Sirainen <tss@iki.fi <mailto:tss@iki.fi>> wrote: > > > https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz > > https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig > > Binary packages in https://repo.dovecot.org/ > > > > * The default postmaster_address is now "postmaster@<user domain or > > server hostname>". If username contains the @domain part, that's > > used. If not, then the server's hostname is used. > > * "doveadm stats dump" now returns two decimals for the "avg" field. > > > > + Added push notification driver that uses a Lua script > > + Added new SQL, DNS and connection events. > > See https://wiki2.dovecot.org/Events > > + Added "doveadm mailbox cache purge" command. > > + Added events API support for Lua scripts > > + doveadm force-resync -f parameter performs "index fsck" while opening > > the index. This may be useful to fix some types of broken index files. > > This may become the default behavior in a later version. > > - director: Kicking a user crashes if login process is veryslow
> > - pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages > > unless QUIT is sent. > > - auth: Fix crypt() segfault with glibc-2.28+ > > - imap: Running UID FILTER script with errors assert-crashes > > - dsync, pop3-migration: POP3 UIDLs weren't added to > > dovecot.index.cache while mails were saved. > > - dict clients may have been using 100% CPU while waiting for dict > > server to finish commands. > > - doveadm user: Fixed user listing via HTTP API > > - All levels of Cassandra log messages were logged as Dovecot errors. > > - http/smtp client may have crashed after SSL handshake > > - Lua auth converted strings that looked like numbers into numbers. > > > > > > FreeBSD 9.3 (i386): > > > Making all in lib-master > gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib-I../../src/lib-dns
> -I../../src/lib-test -I../../src/lib-settings > -I../../src/lib-ssl-iostream > -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\" > -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\" > -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\" > -DBINDIR=\""/opt/dovecot2.3/bin"\" -std=gnu99 -g -O2 -fstack-protector > -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W-Wmissing-prototypes
> -Wmissing-declarations -Wpointer-arith -Wchar-subscripts-Wformat=2
> -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 > -I/usr/local/include -MT test-event-stats.o -MD -MP -MF > .deps/test-event-stats.Tpo -c -o test-event-stats.o test-event-stats.c > test-event-stats.c: In function 'kill_stats_child': > test-event-stats.c:101: warning: implicit declaration of function 'kill' > test-event-stats.c:101: error: 'SIGKILL' undeclared (first use in this > function) > test-event-stats.c:101: error: (Each undeclared identifier is reported only > once > test-event-stats.c:101: error: for each function it appears in.) > test-event-stats.c: In function 'test_no_merging2': > test-event-stats.c:361: warning: format '%lu' expects type 'long unsigned > int', but argument 2 has type 'uint64_t' > test-event-stats.c: In function 'test_no_merging3': > test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned > int', but argument 2 has type 'uint64_t' > test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned > int', but argument 4 has type 'uint64_t' > test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned > int', but argument 6 has type 'uint64_t' > test-event-stats.c: In function 'test_merge_events2': > test-event-stats.c:452: warning: format '%lu' expects type 'long unsigned > int', but argument 2 has type 'uint64_t' > test-event-stats.c: In function 'test_skip_parents': > test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned > int', but argument 2 has type 'uint64_t' > test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned > int', but argument 4 has type 'uint64_t' > test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned > int', but argument 6 has type 'uint64_t' > test-event-stats.c: In function 'test_merge_events_skip_parents': > test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned > int', but argument 2 has type 'uint64_t' > test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned > int', but argument 4 has type 'uint64_t' > test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned > int', but argument 6 has type 'uint64_t' > *** Error code 1 > Are you able to upgrade to FreeBSD 10 or 11 ?I will be upgrading the servers to 11.2 soon :)
The OS version is irrelevant.
Fix for this format issue and the signal.h issue can be found in master now, and you can download the patch at
https://github.com/dovecot/core/compare/10048229%5E...de42b54a.patch
Aki
The patch doesn't resolve the permissions issue on /var/run/dovecot//stats-writer so I have backed off again - to 2.3.3.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."
On Sat, 24 Nov 2018 at 12:04, Aki Tuomi <aki.tuomi@open-xchange.com> wrote:
On 24 November 2018 at 10:55 Odhiambo Washington < odhiambo@gmail.com> wrote:
On Sat, 24 Nov 2018 at 10:56, Aki Tuomi < aki.tuomi@open-xchange.com> wrote:
On 23 November 2018 at 17:46 Brad Smith < brad@comstyle.com> wrote:
On 11/23/2018 9:39 AM, Odhiambo Washington wrote:
On Fri, 23 Nov 2018 at 17:30, The Doctor < doctor@doctor.nl2k.ab.ca <mailto: doctor@doctor.nl2k.ab.ca>> wrote:
On Fri, Nov 23, 2018 at 04:02:27PM +0300, Odhiambo Washington
wrote:
On Fri, 23 Nov 2018 at 15:29, Timo Sirainen < tss@iki.fi <mailto: tss@iki.fi>> wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
- The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
- "doveadm stats dump" now returns two decimals for the "avg" field.
- Added push notification driver that uses a Lua script
- Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
- Added "doveadm mailbox cache purge" command.
- Added events API support for Lua scripts
- doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.
- director: Kicking a user crashes if login process is very
slow
- pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
- auth: Fix crypt() segfault with glibc-2.28+
- imap: Running UID FILTER script with errors assert-crashes
- dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
- dict clients may have been using 100% CPU while waiting for dict server to finish commands.
- doveadm user: Fixed user listing via HTTP API
- All levels of Cassandra log messages were logged as Dovecot errors.
- http/smtp client may have crashed after SSL handshake
- Lua auth converted strings that looked like numbers into numbers.
FreeBSD 9.3 (i386):
Making all in lib-master gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib
-I../../src/lib-dns
-I../../src/lib-test -I../../src/lib-settings -I../../src/lib-ssl-iostream -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\" -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\" -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\" -DBINDIR=\""/opt/dovecot2.3/bin"\" -std=gnu99 -g -O2 -fstack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W
-Wmissing-prototypes
-Wmissing-declarations -Wpointer-arith -Wchar-subscripts
-Wformat=2
-Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 -I/usr/local/include -MT test-event-stats.o -MD -MP -MF .deps/test-event-stats.Tpo -c -o test-event-stats.o test-event-stats.c test-event-stats.c: In function 'kill_stats_child': test-event-stats.c:101: warning: implicit declaration of function 'kill' test-event-stats.c:101: error: 'SIGKILL' undeclared (first use in this function) test-event-stats.c:101: error: (Each undeclared identifier is reported only once test-event-stats.c:101: error: for each function it appears in.) test-event-stats.c: In function 'test_no_merging2': test-event-stats.c:361: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c: In function 'test_no_merging3': test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'uint64_t' test-event-stats.c:387: warning: format '%lu' expects type 'long unsigned int', but argument 6 has type 'uint64_t' test-event-stats.c: In function 'test_merge_events2': test-event-stats.c:452: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c: In function 'test_skip_parents': test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'uint64_t' test-event-stats.c:484: warning: format '%lu' expects type 'long unsigned int', but argument 6 has type 'uint64_t' test-event-stats.c: In function 'test_merge_events_skip_parents': test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned int', but argument 2 has type 'uint64_t' test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'uint64_t' test-event-stats.c:526: warning: format '%lu' expects type 'long unsigned int', but argument 6 has type 'uint64_t' *** Error code 1
Are you able to upgrade to FreeBSD 10 or 11 ?
I will be upgrading the servers to 11.2 soon :)
The OS version is irrelevant.
Fix for this format issue and the signal.h issue can be found in master now, and you can download the patch at
https://github.com/dovecot/core/compare/10048229%5E...de42b54a.patch
Aki
The patch doesn't resolve the permissions issue on /var/run/dovecot//stats-writer so I have backed off again - to 2.3.3.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."
Well, to be exact no one claims it does fix it. You can work around it with
service stats { unix_listener stats-writer { mode = 0777 } }
Aki Tuomi
I'll wait for the fix instead of the workaround.
Thank you very much for entertaining my noise :-)
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."
On Fri, 23 Nov 2018 at 15:29, Timo Sirainen <tss@iki.fi> wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
- The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
- "doveadm stats dump" now returns two decimals for the "avg" field.
- Added push notification driver that uses a Lua script
- Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
- Added "doveadm mailbox cache purge" command.
- Added events API support for Lua scripts
- doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.
- director: Kicking a user crashes if login process is very slow
- pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
- auth: Fix crypt() segfault with glibc-2.28+
- imap: Running UID FILTER script with errors assert-crashes
- dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
- dict clients may have been using 100% CPU while waiting for dict server to finish commands.
- doveadm user: Fixed user listing via HTTP API
- All levels of Cassandra log messages were logged as Dovecot errors.
- http/smtp client may have crashed after SSL handshake
- Lua auth converted strings that looked like numbers into numbers.
FreeBSD 11.2 (amd64):
gmake[2]: Entering directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src/lib-master' gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib -I../../src/lib-dns -I../../src/lib-test -I../../src/lib-settings -I../../src/lib-ssl-iostream -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\" -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\" -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\" -DBINDIR=\""/opt/dovecot2.3/bin"\" -std=gnu99 -g -O2 -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 -I/usr/local/include -MT test-event-stats.o -MD -MP -MF .deps/test-event-stats.Tpo -c -o test-event-stats.o test-event-stats.c test-event-stats.c: In function 'kill_stats_child': test-event-stats.c:101:2: warning: implicit declaration of function 'kill' [-Wimplicit-function-declaration] (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: error: 'SIGKILL' undeclared (first use in this function) (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: note: each undeclared identifier is reported only once for each function it appears in gmake[2]: *** [Makefile:656: test-event-stats.o] Error 1 gmake[2]: Leaving directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src/lib-master' gmake[1]: *** [Makefile:565: install-recursive] Error 1 gmake[1]: Leaving directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src' gmake: *** [Makefile:683: install-recursive] Error 1
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."
On Fri, Nov 23, 2018 at 04:06:53PM +0300, Odhiambo Washington wrote:
On Fri, 23 Nov 2018 at 15:29, Timo Sirainen <tss@iki.fi> wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
* The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used. * "doveadm stats dump" now returns two decimals for the "avg" field.
+ Added push notification driver that uses a Lua script + Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events + Added "doveadm mailbox cache purge" command. + Added events API support for Lua scripts + doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version. - director: Kicking a user crashes if login process is very slow - pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent. - auth: Fix crypt() segfault with glibc-2.28+ - imap: Running UID FILTER script with errors assert-crashes - dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved. - dict clients may have been using 100% CPU while waiting for dict server to finish commands. - doveadm user: Fixed user listing via HTTP API - All levels of Cassandra log messages were logged as Dovecot errors. - http/smtp client may have crashed after SSL handshake - Lua auth converted strings that looked like numbers into numbers.
FreeBSD 11.2 (amd64):
gmake[2]: Entering directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src/lib-master' gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib -I../../src/lib-dns -I../../src/lib-test -I../../src/lib-settings -I../../src/lib-ssl-iostream -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\" -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\" -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\" -DBINDIR=\""/opt/dovecot2.3/bin"\" -std=gnu99 -g -O2 -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 -I/usr/local/include -MT test-event-stats.o -MD -MP -MF .deps/test-event-stats.Tpo -c -o test-event-stats.o test-event-stats.c test-event-stats.c: In function 'kill_stats_child': test-event-stats.c:101:2: warning: implicit declaration of function 'kill' [-Wimplicit-function-declaration] (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: error: 'SIGKILL' undeclared (first use in this function) (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: note: each undeclared identifier is reported only once for each function it appears in gmake[2]: *** [Makefile:656: test-event-stats.o] Error 1 gmake[2]: Leaving directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src/lib-master' gmake[1]: *** [Makefile:565: install-recursive] Error 1 gmake[1]: Leaving directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src' gmake: *** [Makefile:683: install-recursive] Error 1
Looks like our porters will have their hands full.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."
-- Member - Liberal International This is doctor@@nl2k.ab.ca Ici doctor@@nl2k.ab.ca Yahweh, Queen & country!Never Satan President Republic!Beware AntiChrist rising! https://www.empire.kred/ROOTNK?t=94a1f39b Look at Psalms 14 and 53 on Atheism sMerry Christmas 2018 and Happy New Year 2019!!
On Fri, 23 Nov 2018 at 17:31, The Doctor <doctor@doctor.nl2k.ab.ca> wrote:
On Fri, Nov 23, 2018 at 04:06:53PM +0300, Odhiambo Washington wrote:
On Fri, 23 Nov 2018 at 15:29, Timo Sirainen <tss@iki.fi> wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
- The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
- "doveadm stats dump" now returns two decimals for the "avg" field.
- Added push notification driver that uses a Lua script
- Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
- Added "doveadm mailbox cache purge" command.
- Added events API support for Lua scripts
- doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.
- director: Kicking a user crashes if login process is very slow
- pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
- auth: Fix crypt() segfault with glibc-2.28+
- imap: Running UID FILTER script with errors assert-crashes
- dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
- dict clients may have been using 100% CPU while waiting for dict server to finish commands.
- doveadm user: Fixed user listing via HTTP API
- All levels of Cassandra log messages were logged as Dovecot errors.
- http/smtp client may have crashed after SSL handshake
- Lua auth converted strings that looked like numbers into numbers.
FreeBSD 11.2 (amd64):
gmake[2]: Entering directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src/lib-master' gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib -I../../src/lib-dns -I../../src/lib-test -I../../src/lib-settings -I../../src/lib-ssl-iostream -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\" -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\" -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\" -DBINDIR=\""/opt/dovecot2.3/bin"\" -std=gnu99 -g -O2 -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 -I/usr/local/include -MT test-event-stats.o -MD -MP -MF .deps/test-event-stats.Tpo -c -o test-event-stats.o test-event-stats.c test-event-stats.c: In function 'kill_stats_child': test-event-stats.c:101:2: warning: implicit declaration of function 'kill' [-Wimplicit-function-declaration] (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: error: 'SIGKILL' undeclared (first use in this function) (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: note: each undeclared identifier is reported only once for each function it appears in gmake[2]: *** [Makefile:656: test-event-stats.o] Error 1 gmake[2]: Leaving directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src/lib-master' gmake[1]: *** [Makefile:565: install-recursive] Error 1 gmake[1]: Leaving directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src' gmake: *** [Makefile:683: install-recursive] Error 1
Looks like our porters will have their hands full.
dovecot-2.3.3 has been in the FreeBSD ports already so I don't think it will be much of a problem.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."
On 11/23/2018 9:31 AM, The Doctor wrote:
On Fri, Nov 23, 2018 at 04:06:53PM +0300, Odhiambo Washington wrote:
On Fri, 23 Nov 2018 at 15:29, Timo Sirainen <tss@iki.fi> wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
- The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
- "doveadm stats dump" now returns two decimals for the "avg" field.
- Added push notification driver that uses a Lua script
- Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
- Added "doveadm mailbox cache purge" command.
- Added events API support for Lua scripts
- doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.
- director: Kicking a user crashes if login process is very slow
- pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
- auth: Fix crypt() segfault with glibc-2.28+
- imap: Running UID FILTER script with errors assert-crashes
- dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
- dict clients may have been using 100% CPU while waiting for dict server to finish commands.
- doveadm user: Fixed user listing via HTTP API
- All levels of Cassandra log messages were logged as Dovecot errors.
- http/smtp client may have crashed after SSL handshake
- Lua auth converted strings that looked like numbers into numbers.
FreeBSD 11.2 (amd64):
gmake[2]: Entering directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src/lib-master' gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib -I../../src/lib-dns -I../../src/lib-test -I../../src/lib-settings -I../../src/lib-ssl-iostream -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\" -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\" -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\" -DBINDIR=\""/opt/dovecot2.3/bin"\" -std=gnu99 -g -O2 -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 -I/usr/local/include -MT test-event-stats.o -MD -MP -MF .deps/test-event-stats.Tpo -c -o test-event-stats.o test-event-stats.c test-event-stats.c: In function 'kill_stats_child': test-event-stats.c:101:2: warning: implicit declaration of function 'kill' [-Wimplicit-function-declaration] (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: error: 'SIGKILL' undeclared (first use in this function) (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: note: each undeclared identifier is reported only once for each function it appears in gmake[2]: *** [Makefile:656: test-event-stats.o] Error 1 gmake[2]: Leaving directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src/lib-master' gmake[1]: *** [Makefile:565: install-recursive] Error 1 gmake[1]: Leaving directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src' gmake: *** [Makefile:683: install-recursive] Error 1
Looks like our porters will have their hands full.
Complete over exaggeration.
On Fri, 23 Nov 2018 10:45:56 -0500, Brad Smith stated:
On 11/23/2018 9:31 AM, The Doctor wrote:
On Fri, Nov 23, 2018 at 04:06:53PM +0300, Odhiambo Washington wrote:
On Fri, 23 Nov 2018 at 15:29, Timo Sirainen <tss@iki.fi> wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
- The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
- "doveadm stats dump" now returns two decimals for the "avg" field.
- Added push notification driver that uses a Lua script
- Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
- Added "doveadm mailbox cache purge" command.
- Added events API support for Lua scripts
- doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.
- director: Kicking a user crashes if login process is very slow
- pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
- auth: Fix crypt() segfault with glibc-2.28+
- imap: Running UID FILTER script with errors assert-crashes
- dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
- dict clients may have been using 100% CPU while waiting for dict server to finish commands.
- doveadm user: Fixed user listing via HTTP API
- All levels of Cassandra log messages were logged as Dovecot errors.
- http/smtp client may have crashed after SSL handshake
- Lua auth converted strings that looked like numbers into numbers.
FreeBSD 11.2 (amd64):
gmake[2]: Entering directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src/lib-master' gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib -I../../src/lib-dns -I../../src/lib-test -I../../src/lib-settings -I../../src/lib-ssl-iostream -DPKG_RUNDIR=\""/opt/dovecot2.3/var/run/dovecot"\" -DPKG_STATEDIR=\""/opt/dovecot2.3/var/lib/dovecot"\" -DSYSCONFDIR=\""/opt/dovecot2.3/etc/dovecot"\" -DBINDIR=\""/opt/dovecot2.3/bin"\" -std=gnu99 -g -O2 -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 -I/usr/local/include -MT test-event-stats.o -MD -MP -MF .deps/test-event-stats.Tpo -c -o test-event-stats.o test-event-stats.c test-event-stats.c: In function 'kill_stats_child': test-event-stats.c:101:2: warning: implicit declaration of function 'kill' [-Wimplicit-function-declaration] (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: error: 'SIGKILL' undeclared (first use in this function) (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: note: each undeclared identifier is reported only once for each function it appears in gmake[2]: *** [Makefile:656: test-event-stats.o] Error 1 gmake[2]: Leaving directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src/lib-master' gmake[1]: *** [Makefile:565: install-recursive] Error 1 gmake[1]: Leaving directory '/usr/home/wash/Tools/Dovecot/2.3/dovecot-2.3.4/src' gmake: *** [Makefile:683: install-recursive] Error 1
Looks like our porters will have their hands full.
Complete over exaggeration.
Dovecot 2.3.4 and Dovecot Pigeonhole 0.5.4 are already in the FreeBSD ports system and both install and work fine, at least on my 11.2-RELEASE-p4 amd64 machine. If you are rolling your own, then you have to expect occasional problems.
-- Jerry
Hi
THis is the lines I have in my dmesg (see below)
In dovecot log , I see:
Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop)
What do to about it ?
Using lastest 2.3.4 version
Thank you
[132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55 [134031.969596] auth[27031]: segfault at 55e509612c30 ip 000055e509612c30 sp 00007ffeb96dee48 error 15 [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55 [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip 00007f393841013e sp 00007ffe4a16efc0 error 6 in libdovecot.so.0.0.0[7f3938363000+e2000] [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip 00007f4376e32ffb sp 00007fff1cfdc000 error 6 in libdovecot.so.0.0.0[7f4376d86000+e2000] [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83 ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b 00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip 00007ffec43fdcff sp 00007ffec3ba5000 error 6 [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip 00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in libdovecot.so.0.0.0[7fb7c9ba2000+e2000] [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48 [135796.171575] auth[11121]: segfault at 555f8645cc30 ip 0000555f8645cc30 sp 00007ffcbb510868 error 15 [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55 [136710.562003] auth[17828]: segfault at 563443604c30 ip 0000563443604c30 sp 00007ffc1aa8b498 error 15 [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56 [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip 000055b27bc63c30 sp 00007ffd5d5b9298 error 15 [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17 cd 7f 00 00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4 7b b2 55 [138521.070485] auth[32481]: segfault at 556e05197c30 ip 0000556e05197c30 sp 00007ffe87217c08 error 15 [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20 3c 7f 00 00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17 05 6e 55 [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip 00005570d3e46c30 sp 00007ffc9118a5e8 error 15 [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71 11 7f 00 00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2 d3 70 55 [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip 00007ffe49ba7cff sp 00007ffe49317000 error 6 [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip 00007fa68292a13e sp 00007ffcdc00efb0 error 6 in libdovecot.so.0.0.0[7fa68287d000+e2000] [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip 00007f0e09ae259f sp 00007ffc90beb000 error 6 in libdovecot.so.0.0.0[7f0e09a35000+e2000] [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb 2d f5 ff 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8 6a fb ff [139747.003229] auth[9545]: segfault at 558b5e039c30 ip 0000558b5e039c30 sp 00007ffd43633008 error 15 [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e 75 7f 00 00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01 5e 8b 55 [140640.900512] auth[17221]: segfault at 55e952d41c30 ip 000055e952d41c30 sp 00007ffc36fa0908 error 15 [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80 b9 7f 00 00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1 52 e9 55 [140777.927292] auth[18114]: segfault at 55bd72adec30 ip 000055bd72adec30 sp 00007ffca394ba98 error 15 [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67 d7 7f 00 00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab 72 bd 55
Thank you Aki
here the requested data:
File system : ext4
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)
On 2018-11-25 08:08, Aki Tuomi wrote:
On 25 November 2018 at 06:29 Joan Moreau < jom@grosjo.net> wrote:
Hi
THis is the lines I have in my dmesg (see below)
In dovecot log , I see:
Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop)
What do to about it ?
Using lastest 2.3.4 version
Thank you
[132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55 [134031.969596] auth[27031]: segfault at 55e509612c30 ip 000055e509612c30 sp 00007ffeb96dee48 error 15 [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55 [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip 00007f393841013e sp 00007ffe4a16efc0 error 6 in libdovecot.so.0.0.0[7f3938363000+e2000] [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip 00007f4376e32ffb sp 00007fff1cfdc000 error 6 in libdovecot.so.0.0.0[7f4376d86000+e2000] [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83 ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b 00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip 00007ffec43fdcff sp 00007ffec3ba5000 error 6 [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip 00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in libdovecot.so.0.0.0[7fb7c9ba2000+e2000] [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48 [135796.171575] auth[11121]: segfault at 555f8645cc30 ip 0000555f8645cc30 sp 00007ffcbb510868 error 15 [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55 [136710.562003] auth[17828]: segfault at 563443604c30 ip 0000563443604c30 sp 00007ffc1aa8b498 error 15 [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56 [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip 000055b27bc63c30 sp 00007ffd5d5b9298 error 15 [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17 cd 7f 00 00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4 7b b2 55 [138521.070485] auth[32481]: segfault at 556e05197c30 ip 0000556e05197c30 sp 00007ffe87217c08 error 15 [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20 3c 7f 00 00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17 05 6e 55 [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip 00005570d3e46c30 sp 00007ffc9118a5e8 error 15 [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71 11 7f 00 00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2 d3 70 55 [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip 00007ffe49ba7cff sp 00007ffe49317000 error 6 [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip 00007fa68292a13e sp 00007ffcdc00efb0 error 6 in libdovecot.so.0.0.0[7fa68287d000+e2000] [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip 00007f0e09ae259f sp 00007ffc90beb000 error 6 in libdovecot.so.0.0.0[7f0e09a35000+e2000] [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb 2d f5 ff 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8 6a fb ff [139747.003229] auth[9545]: segfault at 558b5e039c30 ip 0000558b5e039c30 sp 00007ffd43633008 error 15 [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e 75 7f 00 00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01 5e 8b 55 [140640.900512] auth[17221]: segfault at 55e952d41c30 ip 000055e952d41c30 sp 00007ffc36fa0908 error 15 [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80 b9 7f 00 00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1 52 e9 55 [140777.927292] auth[18114]: segfault at 55bd72adec30 ip 000055bd72adec30 sp 00007ffca394ba98 error 15 [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67 d7 7f 00 00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab 72 bd 55
Please see https://dovecot.org/bugreport.html
Aki
Thank you Aki
here the requested data (below)
Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)
Bug appears mostly in auth process and index-worker
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)
On 2018-11-25 08:08, Aki Tuomi wrote:
On 25 November 2018 at 06:29 Joan Moreau < jom@grosjo.net> wrote:
Hi
THis is the lines I have in my dmesg (see below)
In dovecot log , I see:
Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop)
What do to about it ?
Using lastest 2.3.4 version
Thank you
[132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55 [134031.969596] auth[27031]: segfault at 55e509612c30 ip 000055e509612c30 sp 00007ffeb96dee48 error 15 [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55 [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip 00007f393841013e sp 00007ffe4a16efc0 error 6 in libdovecot.so.0.0.0[7f3938363000+e2000] [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip 00007f4376e32ffb sp 00007fff1cfdc000 error 6 in libdovecot.so.0.0.0[7f4376d86000+e2000] [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83 ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b 00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip 00007ffec43fdcff sp 00007ffec3ba5000 error 6 [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip 00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in libdovecot.so.0.0.0[7fb7c9ba2000+e2000] [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48 [135796.171575] auth[11121]: segfault at 555f8645cc30 ip 0000555f8645cc30 sp 00007ffcbb510868 error 15 [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55 [136710.562003] auth[17828]: segfault at 563443604c30 ip 0000563443604c30 sp 00007ffc1aa8b498 error 15 [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56 [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip 000055b27bc63c30 sp 00007ffd5d5b9298 error 15 [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17 cd 7f 00 00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4 7b b2 55 [138521.070485] auth[32481]: segfault at 556e05197c30 ip 0000556e05197c30 sp 00007ffe87217c08 error 15 [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20 3c 7f 00 00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17 05 6e 55 [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip 00005570d3e46c30 sp 00007ffc9118a5e8 error 15 [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71 11 7f 00 00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2 d3 70 55 [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip 00007ffe49ba7cff sp 00007ffe49317000 error 6 [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip 00007fa68292a13e sp 00007ffcdc00efb0 error 6 in libdovecot.so.0.0.0[7fa68287d000+e2000] [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip 00007f0e09ae259f sp 00007ffc90beb000 error 6 in libdovecot.so.0.0.0[7f0e09a35000+e2000] [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb 2d f5 ff 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8 6a fb ff [139747.003229] auth[9545]: segfault at 558b5e039c30 ip 0000558b5e039c30 sp 00007ffd43633008 error 15 [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e 75 7f 00 00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01 5e 8b 55 [140640.900512] auth[17221]: segfault at 55e952d41c30 ip 000055e952d41c30 sp 00007ffc36fa0908 error 15 [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80 b9 7f 00 00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1 52 e9 55 [140777.927292] auth[18114]: segfault at 55bd72adec30 ip 000055bd72adec30 sp 00007ffca394ba98 error 15 [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67 d7 7f 00 00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab 72 bd 55
Please see https://dovecot.org/bugreport.html
Aki
It's still missing core dump (or bt full from it)
Aki
On 27.11.2018 8.39, Joan Moreau wrote:
Thank you Aki
here the requested data (below)
Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)
Bug appears mostly in auth process and index-worker
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)
On 2018-11-25 08:08, Aki Tuomi wrote:
On 25 November 2018 at 06:29 Joan Moreau < jom@grosjo.net <mailto:jom@grosjo.net>> wrote: Hi THis is the lines I have in my dmesg (see below) In dovecot log , I see: Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop) What do to about it ? Using lastest 2.3.4 version Thank you -------- [132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55 [134031.969596] auth[27031]: segfault at 55e509612c30 ip 000055e509612c30 sp 00007ffeb96dee48 error 15 [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55 [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip 00007f393841013e sp 00007ffe4a16efc0 error 6 in libdovecot.so.0.0.0[7f3938363000+e2000] [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip 00007f4376e32ffb sp 00007fff1cfdc000 error 6 in libdovecot.so.0.0.0[7f4376d86000+e2000] [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83 ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b 00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip 00007ffec43fdcff sp 00007ffec3ba5000 error 6 [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip 00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in libdovecot.so.0.0.0[7fb7c9ba2000+e2000] [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48 [135796.171575] auth[11121]: segfault at 555f8645cc30 ip 0000555f8645cc30 sp 00007ffcbb510868 error 15 [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55 [136710.562003] auth[17828]: segfault at 563443604c30 ip 0000563443604c30 sp 00007ffc1aa8b498 error 15 [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56 [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip 000055b27bc63c30 sp 00007ffd5d5b9298 error 15 [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17 cd 7f 00 00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4 7b b2 55 [138521.070485] auth[32481]: segfault at 556e05197c30 ip 0000556e05197c30 sp 00007ffe87217c08 error 15 [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20 3c 7f 00 00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17 05 6e 55 [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip 00005570d3e46c30 sp 00007ffc9118a5e8 error 15 [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71 11 7f 00 00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2 d3 70 55 [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip 00007ffe49ba7cff sp 00007ffe49317000 error 6 [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip 00007fa68292a13e sp 00007ffcdc00efb0 error 6 in libdovecot.so.0.0.0[7fa68287d000+e2000] [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip 00007f0e09ae259f sp 00007ffc90beb000 error 6 in libdovecot.so.0.0.0[7f0e09a35000+e2000] [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb 2d f5 ff 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8 6a fb ff [139747.003229] auth[9545]: segfault at 558b5e039c30 ip 0000558b5e039c30 sp 00007ffd43633008 error 15 [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e 75 7f 00 00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01 5e 8b 55 [140640.900512] auth[17221]: segfault at 55e952d41c30 ip 000055e952d41c30 sp 00007ffc36fa0908 error 15 [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80 b9 7f 00 00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1 52 e9 55 [140777.927292] auth[18114]: segfault at 55bd72adec30 ip 000055bd72adec30 sp 00007ffca394ba98 error 15 [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67 d7 7f 00 00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab 72 bd 55 Please see https://dovecot.org/bugreport.html --- Aki
Where to get that ?
On 2018-11-27 08:50, Aki Tuomi wrote:
It's still missing core dump (or bt full from it)
Aki
On 27.11.2018 8.39, Joan Moreau wrote:
Thank you Aki
here the requested data (below)
Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)
Bug appears mostly in auth process and index-worker
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)
On 2018-11-25 08:08, Aki Tuomi wrote:
On 25 November 2018 at 06:29 Joan Moreau < jom@grosjo.net> wrote:
Hi
THis is the lines I have in my dmesg (see below)
In dovecot log , I see:
Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop)
What do to about it ?
Using lastest 2.3.4 version
Thank you
[132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55 [134031.969596] auth[27031]: segfault at 55e509612c30 ip 000055e509612c30 sp 00007ffeb96dee48 error 15 [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55 [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip 00007f393841013e sp 00007ffe4a16efc0 error 6 in libdovecot.so.0.0.0[7f3938363000+e2000] [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip 00007f4376e32ffb sp 00007fff1cfdc000 error 6 in libdovecot.so.0.0.0[7f4376d86000+e2000] [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83 ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b 00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip 00007ffec43fdcff sp 00007ffec3ba5000 error 6 [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip 00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in libdovecot.so.0.0.0[7fb7c9ba2000+e2000] [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48 [135796.171575] auth[11121]: segfault at 555f8645cc30 ip 0000555f8645cc30 sp 00007ffcbb510868 error 15 [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55 [136710.562003] auth[17828]: segfault at 563443604c30 ip 0000563443604c30 sp 00007ffc1aa8b498 error 15 [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56 [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip 000055b27bc63c30 sp 00007ffd5d5b9298 error 15 [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17 cd 7f 00 00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4 7b b2 55 [138521.070485] auth[32481]: segfault at 556e05197c30 ip 0000556e05197c30 sp 00007ffe87217c08 error 15 [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20 3c 7f 00 00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17 05 6e 55 [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip 00005570d3e46c30 sp 00007ffc9118a5e8 error 15 [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71 11 7f 00 00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2 d3 70 55 [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip 00007ffe49ba7cff sp 00007ffe49317000 error 6 [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip 00007fa68292a13e sp 00007ffcdc00efb0 error 6 in libdovecot.so.0.0.0[7fa68287d000+e2000] [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip 00007f0e09ae259f sp 00007ffc90beb000 error 6 in libdovecot.so.0.0.0[7f0e09a35000+e2000] [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb 2d f5 ff 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8 6a fb ff [139747.003229] auth[9545]: segfault at 558b5e039c30 ip 0000558b5e039c30 sp 00007ffd43633008 error 15 [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e 75 7f 00 00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01 5e 8b 55 [140640.900512] auth[17221]: segfault at 55e952d41c30 ip 000055e952d41c30 sp 00007ffc36fa0908 error 15 [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80 b9 7f 00 00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1 52 e9 55 [140777.927292] auth[18114]: segfault at 55bd72adec30 ip 000055bd72adec30 sp 00007ffca394ba98 error 15 [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67 d7 7f 00 00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab 72 bd 55
Please see https://dovecot.org/bugreport.html
Aki
See https://dovecot.org/bugreport.html#coredumps
Without a backtrace it's not really possible to figure out where it's crashing.
On 28 Nov 2018, at 13.20, Joan Moreau <jom@grosjo.net> wrote:
Where to get that ?
On 2018-11-27 08:50, Aki Tuomi wrote:
It's still missing core dump (or bt full from it)
Aki
On 27.11.2018 8.39, Joan Moreau wrote:
Thank you Aki
here the requested data (below)
Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)
Bug appears mostly in auth process and index-worker
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)
On 2018-11-25 08:08, Aki Tuomi wrote:
On 25 November 2018 at 06:29 Joan Moreau < jom@grosjo.net <mailto:jom@grosjo.net>> wrote:
Hi
THis is the lines I have in my dmesg (see below)
In dovecot log , I see:
Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop)
What do to about it ?
Using lastest 2.3.4 version
Thank you
[132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55 [134031.969596] auth[27031]: segfault at 55e509612c30 ip 000055e509612c30 sp 00007ffeb96dee48 error 15 [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55 [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip 00007f393841013e sp 00007ffe4a16efc0 error 6 in libdovecot.so.0.0.0[7f3938363000+e2000] [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip 00007f4376e32ffb sp 00007fff1cfdc000 error 6 in libdovecot.so.0.0.0[7f4376d86000+e2000] [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83 ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b 00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip 00007ffec43fdcff sp 00007ffec3ba5000 error 6 [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip 00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in libdovecot.so.0.0.0[7fb7c9ba2000+e2000] [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48 [135796.171575] auth[11121]: segfault at 555f8645cc30 ip 0000555f8645cc30 sp 00007ffcbb510868 error 15 [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55 [136710.562003] auth[17828]: segfault at 563443604c30 ip 0000563443604c30 sp 00007ffc1aa8b498 error 15 [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56 [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip 000055b27bc63c30 sp 00007ffd5d5b9298 error 15 [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17 cd 7f 00 00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4 7b b2 55 [138521.070485] auth[32481]: segfault at 556e05197c30 ip 0000556e05197c30 sp 00007ffe87217c08 error 15 [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20 3c 7f 00 00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17 05 6e 55 [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip 00005570d3e46c30 sp 00007ffc9118a5e8 error 15 [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71 11 7f 00 00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2 d3 70 55 [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip 00007ffe49ba7cff sp 00007ffe49317000 error 6 [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip 00007fa68292a13e sp 00007ffcdc00efb0 error 6 in libdovecot.so.0.0.0[7fa68287d000+e2000] [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip 00007f0e09ae259f sp 00007ffc90beb000 error 6 in libdovecot.so.0.0.0[7f0e09a35000+e2000] [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb 2d f5 ff 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8 6a fb ff [139747.003229] auth[9545]: segfault at 558b5e039c30 ip 0000558b5e039c30 sp 00007ffd43633008 error 15 [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e 75 7f 00 00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01 5e 8b 55 [140640.900512] auth[17221]: segfault at 55e952d41c30 ip 000055e952d41c30 sp 00007ffc36fa0908 error 15 [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80 b9 7f 00 00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1 52 e9 55 [140777.927292] auth[18114]: segfault at 55bd72adec30 ip 000055bd72adec30 sp 00007ffca394ba98 error 15 [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67 d7 7f 00 00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab 72 bd 55
Please see https://dovecot.org/bugreport.html <https://dovecot.org/bugreport.html>
Aki
Can't find any "core" files (updatedb ; locate "core"). Coredump are usually in /var/liv/systemd/coredump for other programs, but nothing for dovecot.
Looks like issue is in 'auth' and 'indexer-worker'. Where can be the coredump files ?
On 2018-11-28 18:13, Timo Sirainen wrote:
See https://dovecot.org/bugreport.html#coredumps
Without a backtrace it's not really possible to figure out where it's crashing.
On 28 Nov 2018, at 13.20, Joan Moreau <jom@grosjo.net> wrote:
Where to get that ?
On 2018-11-27 08:50, Aki Tuomi wrote:
It's still missing core dump (or bt full from it)
Aki
On 27.11.2018 8.39, Joan Moreau wrote:
Thank you Aki
here the requested data (below)
Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)
Bug appears mostly in auth process and index-worker
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)
On 2018-11-25 08:08, Aki Tuomi wrote:
On 25 November 2018 at 06:29 Joan Moreau < jom@grosjo.net> wrote:
Hi
THis is the lines I have in my dmesg (see below)
In dovecot log , I see:
Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop)
What do to about it ?
Using lastest 2.3.4 version
Thank you
[132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55 [134031.969596] auth[27031]: segfault at 55e509612c30 ip 000055e509612c30 sp 00007ffeb96dee48 error 15 [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55 [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip 00007f393841013e sp 00007ffe4a16efc0 error 6 in libdovecot.so.0.0.0[7f3938363000+e2000] [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip 00007f4376e32ffb sp 00007fff1cfdc000 error 6 in libdovecot.so.0.0.0[7f4376d86000+e2000] [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83 ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b 00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip 00007ffec43fdcff sp 00007ffec3ba5000 error 6 [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip 00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in libdovecot.so.0.0.0[7fb7c9ba2000+e2000] [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48 [135796.171575] auth[11121]: segfault at 555f8645cc30 ip 0000555f8645cc30 sp 00007ffcbb510868 error 15 [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55 [136710.562003] auth[17828]: segfault at 563443604c30 ip 0000563443604c30 sp 00007ffc1aa8b498 error 15 [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56 [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip 000055b27bc63c30 sp 00007ffd5d5b9298 error 15 [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17 cd 7f 00 00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4 7b b2 55 [138521.070485] auth[32481]: segfault at 556e05197c30 ip 0000556e05197c30 sp 00007ffe87217c08 error 15 [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20 3c 7f 00 00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17 05 6e 55 [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip 00005570d3e46c30 sp 00007ffc9118a5e8 error 15 [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71 11 7f 00 00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2 d3 70 55 [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip 00007ffe49ba7cff sp 00007ffe49317000 error 6 [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip 00007fa68292a13e sp 00007ffcdc00efb0 error 6 in libdovecot.so.0.0.0[7fa68287d000+e2000] [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip 00007f0e09ae259f sp 00007ffc90beb000 error 6 in libdovecot.so.0.0.0[7f0e09a35000+e2000] [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb 2d f5 ff 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8 6a fb ff [139747.003229] auth[9545]: segfault at 558b5e039c30 ip 0000558b5e039c30 sp 00007ffd43633008 error 15 [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e 75 7f 00 00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01 5e 8b 55 [140640.900512] auth[17221]: segfault at 55e952d41c30 ip 000055e952d41c30 sp 00007ffc36fa0908 error 15 [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80 b9 7f 00 00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1 52 e9 55 [140777.927292] auth[18114]: segfault at 55bd72adec30 ip 000055bd72adec30 sp 00007ffca394ba98 error 15 [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67 d7 7f 00 00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab 72 bd 55
Please see https://dovecot.org/bugreport.html
Aki
finally managed to locate the dump
here the output:
# gdb /usr/libexec/dovecot/auth /var/lib/systemd/coredump/core.auth.0.3a33f56105e043de802a7dfcee265a07.28130.1543516118000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/auth...done. [New LWP 28130] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/auth wor'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 (gdb) bt full #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007f739c347672 in abort () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007f739c39f878 in __libc_message () from /usr/lib/libc.so.6 No symbol table info available. #3 0x00007f739c3a618a in malloc_printerr () from /usr/lib/libc.so.6 No symbol table info available. #4 0x00007f739c3a7b27 in _int_free () from /usr/lib/libc.so.6 No symbol table info available. #5 0x00007f739cc33585 in mysql_close (mysql=0x5636af7abdc0) at /usr/src/mariadb-10.3.11/libmariadb/libmariadb/mariadb_lib.c:1939 mysql = 0x5636af7abdc0 p = <optimized out> p = <optimized out> #6 0x00005636aef08f7c in driver_sqlpool_disconnect (_db=0x5636af7aaf30) at driver-sqlpool.c:590 conn__foreach_end = 0x5636af7ab570 db = 0x5636af7aaf30 conn = 0x5636af7ab560 #7 0x00005636aeefed65 in db_sql_unref (_conn=0x5636af7b0020) at db-sql.c:128 conn = 0x5636af7ae388 #8 0x00005636aeef7e15 in userdb_deinit (userdb=0x5636af7afff8) at userdb.c:191 idx = 0 __func__ = "userdb_deinit" #9 0x00005636aeede209 in auth_deinit (auth=0x5636af7afd58) at auth.c:335 passdb = <optimized out> userdb = 0x5636af7aff18 passdb = <optimized out> userdb = <optimized out> #10 auths_deinit () at auth.c:433 auth__foreach_end = 0x5636af7ae2f8 auth = 0x5636af7ae2f0 #11 0x00005636aeedcf65 in main_deinit () at main.c:270 l = <optimized out> l = <optimized out> l_end = <optimized out> #12 main (argc=<optimized out>, argv=<optimized out>) at main.c:401 c = <optimized out>
On 2018-11-29 18:07, Joan Moreau wrote:
Can't find any "core" files (updatedb ; locate "core"). Coredump are usually in /var/liv/systemd/coredump for other programs, but nothing for dovecot.
Looks like issue is in 'auth' and 'indexer-worker'. Where can be the coredump files ?
On 2018-11-28 18:13, Timo Sirainen wrote: See https://dovecot.org/bugreport.html#coredumps
Without a backtrace it's not really possible to figure out where it's crashing.
On 28 Nov 2018, at 13.20, Joan Moreau <jom@grosjo.net> wrote:
Where to get that ?
On 2018-11-27 08:50, Aki Tuomi wrote:
It's still missing core dump (or bt full from it)
Aki
On 27.11.2018 8.39, Joan Moreau wrote:
Thank you Aki
here the requested data (below)
Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)
Bug appears mostly in auth process and index-worker
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)
On 2018-11-25 08:08, Aki Tuomi wrote:
On 25 November 2018 at 06:29 Joan Moreau < jom@grosjo.net> wrote:
Hi
THis is the lines I have in my dmesg (see below)
In dovecot log , I see:
Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop)
What do to about it ?
Using lastest 2.3.4 version
Thank you
[132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55 [134031.969596] auth[27031]: segfault at 55e509612c30 ip 000055e509612c30 sp 00007ffeb96dee48 error 15 [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55 [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip 00007f393841013e sp 00007ffe4a16efc0 error 6 in libdovecot.so.0.0.0[7f3938363000+e2000] [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip 00007f4376e32ffb sp 00007fff1cfdc000 error 6 in libdovecot.so.0.0.0[7f4376d86000+e2000] [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83 ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b 00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip 00007ffec43fdcff sp 00007ffec3ba5000 error 6 [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip 00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in libdovecot.so.0.0.0[7fb7c9ba2000+e2000] [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48 [135796.171575] auth[11121]: segfault at 555f8645cc30 ip 0000555f8645cc30 sp 00007ffcbb510868 error 15 [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55 [136710.562003] auth[17828]: segfault at 563443604c30 ip 0000563443604c30 sp 00007ffc1aa8b498 error 15 [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56 [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip 000055b27bc63c30 sp 00007ffd5d5b9298 error 15 [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17 cd 7f 00 00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4 7b b2 55 [138521.070485] auth[32481]: segfault at 556e05197c30 ip 0000556e05197c30 sp 00007ffe87217c08 error 15 [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20 3c 7f 00 00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17 05 6e 55 [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip 00005570d3e46c30 sp 00007ffc9118a5e8 error 15 [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71 11 7f 00 00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2 d3 70 55 [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip 00007ffe49ba7cff sp 00007ffe49317000 error 6 [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip 00007fa68292a13e sp 00007ffcdc00efb0 error 6 in libdovecot.so.0.0.0[7fa68287d000+e2000] [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip 00007f0e09ae259f sp 00007ffc90beb000 error 6 in libdovecot.so.0.0.0[7f0e09a35000+e2000] [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb 2d f5 ff 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8 6a fb ff [139747.003229] auth[9545]: segfault at 558b5e039c30 ip 0000558b5e039c30 sp 00007ffd43633008 error 15 [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e 75 7f 00 00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01 5e 8b 55 [140640.900512] auth[17221]: segfault at 55e952d41c30 ip 000055e952d41c30 sp 00007ffc36fa0908 error 15 [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80 b9 7f 00 00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1 52 e9 55 [140777.927292] auth[18114]: segfault at 55bd72adec30 ip 000055bd72adec30 sp 00007ffca394ba98 error 15 [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67 d7 7f 00 00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab 72 bd 55
Please see https://dovecot.org/bugreport.html
Aki
ANother (very very long) example :
# gdb /usr/libexec/dovecot/indexer-worker core.indexer-worker.0.3a33f56105e043de802a7dfcee265a07.21017.1543533424000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/indexer-worker...done. [New LWP 21017] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/indexer-worker'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 173 { (gdb) bt full #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 lock_type_str = <optimized out> started = <optimized out> ret = <optimized out> __func__ = "file_lock_do" #1 0x00007f768b62b5b6 in file_wait_lock_error (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=<optimized out>, lock_r=0x7fff04501118, error_r=0x7fff045010b0) at file-lock.c:318 lock = <optimized out> ret = <optimized out> #2 0x00007f768b62b660 in file_wait_lock (fd=<optimized out>, path=<optimized out>, lock_type=lock_type@entry=0, lock_method=<optimized out>, timeout_secs=timeout_secs@entry=60, lock_r=lock_r@entry=0x7fff04501118) at file-lock.c:303 error = 0x564540376490 "" ret = <optimized out> #3 0x00007f768a976c87 in squat_trie_lock (trie=0x564540376490, lock_type=0, file_lock_r=0x7fff04501118, dotlock_r=0x7fff04501120) at squat-trie.c:294 ret = <optimized out> dotlock_r = 0x7fff04501120 file_lock_r = 0x7fff04501118 trie = 0x564540376490 ret = <optimized out> __func__ = "squat_trie_lock" lock_type = 0 ret = <optimized out> __func__ = "squat_trie_lock" #4 0x00007f768a978627 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1487 file_lock = 0x0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #5 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #6 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #7 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #8 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #9 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c850 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #10 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #11 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #12 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #13 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #14 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c7b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #15 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #16 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #17 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #18 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #19 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c710 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #20 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #21 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #22 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #23 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #24 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c670 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #25 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #26 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #27 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #28 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #29 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c5d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #30 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #31 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #32 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #33 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #34 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c530 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #35 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #36 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #37 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #38 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #39 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c490 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #40 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #41 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #42 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #43 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #44 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c3f0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #45 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #46 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #47 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #48 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #49 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c350 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #50 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 #51 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #52 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #53 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #54 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c2b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #55 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #56 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #57 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #58 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #59 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c210 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #60 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #61 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #62 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #63 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #64 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c170 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #65 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #66 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #67 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #68 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #69 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c0d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #70 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #71 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #72 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #73 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #74 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c030 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #75 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #76 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #77 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #78 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #79 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bf90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #80 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #81 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #82 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #83 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #84 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bef0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #85 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #86 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #87 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #88 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #89 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210be50 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #90 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #91 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #92 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #93 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #94 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bdb0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #95 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #96 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #97 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #98 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #99 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bd10 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #100 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #101 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #102 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #103 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #104 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bc70 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #105 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #106 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #107 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #108 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #109 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bbd0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #110 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #111 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #112 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #113 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #114 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bb30 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #115 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #116 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #117 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #118 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #119 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210ba90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #120 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #121 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #122 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #123 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #124 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210b9f0 dotlock = 0x0 changed = <optimized out>
(continues for ages)
On 2018-11-29 19:36, Joan Moreau wrote:
finally managed to locate the dump
here the output:
# gdb /usr/libexec/dovecot/auth /var/lib/systemd/coredump/core.auth.0.3a33f56105e043de802a7dfcee265a07.28130.1543516118000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/auth...done. [New LWP 28130] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/auth wor'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 (gdb) bt full #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007f739c347672 in abort () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007f739c39f878 in __libc_message () from /usr/lib/libc.so.6 No symbol table info available. #3 0x00007f739c3a618a in malloc_printerr () from /usr/lib/libc.so.6 No symbol table info available. #4 0x00007f739c3a7b27 in _int_free () from /usr/lib/libc.so.6 No symbol table info available. #5 0x00007f739cc33585 in mysql_close (mysql=0x5636af7abdc0) at /usr/src/mariadb-10.3.11/libmariadb/libmariadb/mariadb_lib.c:1939 mysql = 0x5636af7abdc0 p = <optimized out> p = <optimized out> #6 0x00005636aef08f7c in driver_sqlpool_disconnect (_db=0x5636af7aaf30) at driver-sqlpool.c:590 conn__foreach_end = 0x5636af7ab570 db = 0x5636af7aaf30 conn = 0x5636af7ab560 #7 0x00005636aeefed65 in db_sql_unref (_conn=0x5636af7b0020) at db-sql.c:128 conn = 0x5636af7ae388 #8 0x00005636aeef7e15 in userdb_deinit (userdb=0x5636af7afff8) at userdb.c:191 idx = 0 __func__ = "userdb_deinit" #9 0x00005636aeede209 in auth_deinit (auth=0x5636af7afd58) at auth.c:335 passdb = <optimized out> userdb = 0x5636af7aff18 passdb = <optimized out> userdb = <optimized out> #10 auths_deinit () at auth.c:433 auth__foreach_end = 0x5636af7ae2f8 auth = 0x5636af7ae2f0 #11 0x00005636aeedcf65 in main_deinit () at main.c:270 l = <optimized out> l = <optimized out> l_end = <optimized out> #12 main (argc=<optimized out>, argv=<optimized out>) at main.c:401 c = <optimized out>
On 2018-11-29 18:07, Joan Moreau wrote:
Can't find any "core" files (updatedb ; locate "core"). Coredump are usually in /var/liv/systemd/coredump for other programs, but nothing for dovecot.
Looks like issue is in 'auth' and 'indexer-worker'. Where can be the coredump files ?
On 2018-11-28 18:13, Timo Sirainen wrote: See https://dovecot.org/bugreport.html#coredumps
Without a backtrace it's not really possible to figure out where it's crashing.
On 28 Nov 2018, at 13.20, Joan Moreau <jom@grosjo.net> wrote:
Where to get that ?
On 2018-11-27 08:50, Aki Tuomi wrote:
It's still missing core dump (or bt full from it)
Aki
On 27.11.2018 8.39, Joan Moreau wrote:
Thank you Aki
here the requested data (below)
Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)
Bug appears mostly in auth process and index-worker
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)
On 2018-11-25 08:08, Aki Tuomi wrote:
On 25 November 2018 at 06:29 Joan Moreau < jom@grosjo.net> wrote:
Hi
THis is the lines I have in my dmesg (see below)
In dovecot log , I see:
Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop)
What do to about it ?
Using lastest 2.3.4 version
Thank you
[132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55 [134031.969596] auth[27031]: segfault at 55e509612c30 ip 000055e509612c30 sp 00007ffeb96dee48 error 15 [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55 [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip 00007f393841013e sp 00007ffe4a16efc0 error 6 in libdovecot.so.0.0.0[7f3938363000+e2000] [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip 00007f4376e32ffb sp 00007fff1cfdc000 error 6 in libdovecot.so.0.0.0[7f4376d86000+e2000] [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83 ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b 00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip 00007ffec43fdcff sp 00007ffec3ba5000 error 6 [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip 00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in libdovecot.so.0.0.0[7fb7c9ba2000+e2000] [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48 [135796.171575] auth[11121]: segfault at 555f8645cc30 ip 0000555f8645cc30 sp 00007ffcbb510868 error 15 [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55 [136710.562003] auth[17828]: segfault at 563443604c30 ip 0000563443604c30 sp 00007ffc1aa8b498 error 15 [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56 [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip 000055b27bc63c30 sp 00007ffd5d5b9298 error 15 [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17 cd 7f 00 00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4 7b b2 55 [138521.070485] auth[32481]: segfault at 556e05197c30 ip 0000556e05197c30 sp 00007ffe87217c08 error 15 [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20 3c 7f 00 00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17 05 6e 55 [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip 00005570d3e46c30 sp 00007ffc9118a5e8 error 15 [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71 11 7f 00 00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2 d3 70 55 [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip 00007ffe49ba7cff sp 00007ffe49317000 error 6 [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip 00007fa68292a13e sp 00007ffcdc00efb0 error 6 in libdovecot.so.0.0.0[7fa68287d000+e2000] [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip 00007f0e09ae259f sp 00007ffc90beb000 error 6 in libdovecot.so.0.0.0[7f0e09a35000+e2000] [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb 2d f5 ff 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8 6a fb ff [139747.003229] auth[9545]: segfault at 558b5e039c30 ip 0000558b5e039c30 sp 00007ffd43633008 error 15 [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e 75 7f 00 00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01 5e 8b 55 [140640.900512] auth[17221]: segfault at 55e952d41c30 ip 000055e952d41c30 sp 00007ffc36fa0908 error 15 [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80 b9 7f 00 00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1 52 e9 55 [140777.927292] auth[18114]: segfault at 55bd72adec30 ip 000055bd72adec30 sp 00007ffca394ba98 error 15 [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67 d7 7f 00 00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab 72 bd 55
Please see https://dovecot.org/bugreport.html
Aki
Hi
How to solve this ?
So many similar segfaults
Thank you
On 2018-11-30 06:11, Joan Moreau wrote:
ANother (very very long) example :
# gdb /usr/libexec/dovecot/indexer-worker core.indexer-worker.0.3a33f56105e043de802a7dfcee265a07.21017.1543533424000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/indexer-worker...done. [New LWP 21017] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/indexer-worker'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 173 { (gdb) bt full #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 lock_type_str = <optimized out> started = <optimized out> ret = <optimized out> __func__ = "file_lock_do" #1 0x00007f768b62b5b6 in file_wait_lock_error (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=<optimized out>, lock_r=0x7fff04501118, error_r=0x7fff045010b0) at file-lock.c:318 lock = <optimized out> ret = <optimized out> #2 0x00007f768b62b660 in file_wait_lock (fd=<optimized out>, path=<optimized out>, lock_type=lock_type@entry=0, lock_method=<optimized out>, timeout_secs=timeout_secs@entry=60, lock_r=lock_r@entry=0x7fff04501118) at file-lock.c:303 error = 0x564540376490 "" ret = <optimized out> #3 0x00007f768a976c87 in squat_trie_lock (trie=0x564540376490, lock_type=0, file_lock_r=0x7fff04501118, dotlock_r=0x7fff04501120) at squat-trie.c:294 ret = <optimized out> dotlock_r = 0x7fff04501120 file_lock_r = 0x7fff04501118 trie = 0x564540376490 ret = <optimized out> __func__ = "squat_trie_lock" lock_type = 0 ret = <optimized out> __func__ = "squat_trie_lock" #4 0x00007f768a978627 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1487 file_lock = 0x0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #5 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #6 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #7 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #8 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #9 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c850 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #10 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #11 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #12 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #13 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #14 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c7b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #15 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #16 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #17 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #18 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #19 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c710 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #20 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #21 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #22 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #23 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #24 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c670 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #25 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #26 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #27 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #28 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #29 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c5d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #30 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #31 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #32 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #33 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #34 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c530 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #35 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #36 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #37 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #38 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #39 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c490 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #40 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #41 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #42 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #43 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #44 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c3f0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #45 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #46 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #47 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #48 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #49 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c350 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #50 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 #51 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #52 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #53 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #54 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c2b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #55 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #56 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #57 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #58 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #59 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c210 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #60 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #61 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #62 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #63 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #64 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c170 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #65 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #66 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #67 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #68 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #69 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c0d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #70 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #71 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #72 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #73 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #74 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c030 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #75 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #76 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #77 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #78 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #79 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bf90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #80 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #81 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #82 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #83 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #84 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bef0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #85 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #86 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #87 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #88 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #89 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210be50 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #90 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #91 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #92 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #93 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #94 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bdb0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #95 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #96 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #97 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #98 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #99 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bd10 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #100 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #101 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #102 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #103 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #104 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bc70 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #105 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #106 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #107 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #108 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #109 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bbd0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #110 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #111 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #112 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #113 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #114 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bb30 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #115 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #116 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #117 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #118 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #119 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210ba90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #120 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #121 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #122 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #123 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #124 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210b9f0 dotlock = 0x0 changed = <optimized out>
(continues for ages)
On 2018-11-29 19:36, Joan Moreau wrote:
finally managed to locate the dump
here the output:
# gdb /usr/libexec/dovecot/auth /var/lib/systemd/coredump/core.auth.0.3a33f56105e043de802a7dfcee265a07.28130.1543516118000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/auth...done. [New LWP 28130] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/auth wor'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 (gdb) bt full #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007f739c347672 in abort () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007f739c39f878 in __libc_message () from /usr/lib/libc.so.6 No symbol table info available. #3 0x00007f739c3a618a in malloc_printerr () from /usr/lib/libc.so.6 No symbol table info available. #4 0x00007f739c3a7b27 in _int_free () from /usr/lib/libc.so.6 No symbol table info available. #5 0x00007f739cc33585 in mysql_close (mysql=0x5636af7abdc0) at /usr/src/mariadb-10.3.11/libmariadb/libmariadb/mariadb_lib.c:1939 mysql = 0x5636af7abdc0 p = <optimized out> p = <optimized out> #6 0x00005636aef08f7c in driver_sqlpool_disconnect (_db=0x5636af7aaf30) at driver-sqlpool.c:590 conn__foreach_end = 0x5636af7ab570 db = 0x5636af7aaf30 conn = 0x5636af7ab560 #7 0x00005636aeefed65 in db_sql_unref (_conn=0x5636af7b0020) at db-sql.c:128 conn = 0x5636af7ae388 #8 0x00005636aeef7e15 in userdb_deinit (userdb=0x5636af7afff8) at userdb.c:191 idx = 0 __func__ = "userdb_deinit" #9 0x00005636aeede209 in auth_deinit (auth=0x5636af7afd58) at auth.c:335 passdb = <optimized out> userdb = 0x5636af7aff18 passdb = <optimized out> userdb = <optimized out> #10 auths_deinit () at auth.c:433 auth__foreach_end = 0x5636af7ae2f8 auth = 0x5636af7ae2f0 #11 0x00005636aeedcf65 in main_deinit () at main.c:270 l = <optimized out> l = <optimized out> l_end = <optimized out> #12 main (argc=<optimized out>, argv=<optimized out>) at main.c:401 c = <optimized out>
On 2018-11-29 18:07, Joan Moreau wrote:
Can't find any "core" files (updatedb ; locate "core"). Coredump are usually in /var/liv/systemd/coredump for other programs, but nothing for dovecot.
Looks like issue is in 'auth' and 'indexer-worker'. Where can be the coredump files ?
On 2018-11-28 18:13, Timo Sirainen wrote: See https://dovecot.org/bugreport.html#coredumps
Without a backtrace it's not really possible to figure out where it's crashing.
On 28 Nov 2018, at 13.20, Joan Moreau <jom@grosjo.net> wrote:
Where to get that ?
On 2018-11-27 08:50, Aki Tuomi wrote:
It's still missing core dump (or bt full from it)
Aki
On 27.11.2018 8.39, Joan Moreau wrote:
Thank you Aki
here the requested data (below)
Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)
Bug appears mostly in auth process and index-worker
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)
On 2018-11-25 08:08, Aki Tuomi wrote:
On 25 November 2018 at 06:29 Joan Moreau < jom@grosjo.net> wrote:
Hi
THis is the lines I have in my dmesg (see below)
In dovecot log , I see:
Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop)
What do to about it ?
Using lastest 2.3.4 version
Thank you
[132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55 [134031.969596] auth[27031]: segfault at 55e509612c30 ip 000055e509612c30 sp 00007ffeb96dee48 error 15 [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55 [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip 00007f393841013e sp 00007ffe4a16efc0 error 6 in libdovecot.so.0.0.0[7f3938363000+e2000] [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip 00007f4376e32ffb sp 00007fff1cfdc000 error 6 in libdovecot.so.0.0.0[7f4376d86000+e2000] [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83 ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b 00 <e8> 00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip 00007ffec43fdcff sp 00007ffec3ba5000 error 6 [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip 00007fb7c9c4f5b1 sp 00007fffed922000 error 6 in libdovecot.so.0.0.0[7fb7c9ba2000+e2000] [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 <e8> 6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48 [135796.171575] auth[11121]: segfault at 555f8645cc30 ip 0000555f8645cc30 sp 00007ffcbb510868 error 15 [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55 [136710.562003] auth[17828]: segfault at 563443604c30 ip 0000563443604c30 sp 00007ffc1aa8b498 error 15 [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56 [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip 000055b27bc63c30 sp 00007ffd5d5b9298 error 15 [138331.686721] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 9a 08 17 cd 7f 00 00 <40> 14 0f 17 cd 7f 00 00 21 80 00 00 00 00 00 00 00 14 c4 7b b2 55 [138521.070485] auth[32481]: segfault at 556e05197c30 ip 0000556e05197c30 sp 00007ffe87217c08 error 15 [138521.070487] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa a7 20 3c 7f 00 00 <40> 24 ae 20 3c 7f 00 00 21 80 00 00 00 00 00 00 00 54 17 05 6e 55 [138782.544700] auth[2408]: segfault at 5570d3e46c30 ip 00005570d3e46c30 sp 00007ffc9118a5e8 error 15 [138782.544709] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 da fc 71 11 7f 00 00 <40> 54 03 72 11 7f 00 00 21 80 00 00 00 00 00 00 00 44 e2 d3 70 55 [139113.225511] indexer-worker[3785]: segfault at 7ffe49316ff8 ip 00007ffe49ba7cff sp 00007ffe49317000 error 6 [139113.225514] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0 e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3 [139113.733885] indexer-worker[5483]: segfault at 7ffcdc00efb8 ip 00007fa68292a13e sp 00007ffcdc00efb0 error 6 in libdovecot.so.0.0.0[7fa68287d000+e2000] [139113.733890] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8 [139120.612228] indexer-worker[5498]: segfault at 7ffc90beaff8 ip 00007f0e09ae259f sp 00007ffc90beb000 error 6 in libdovecot.so.0.0.0[7f0e09a35000+e2000] [139120.612232] Code: c2 48 8b 44 24 08 48 8b 30 31 c0 e8 cb 2d f5 ff 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41 89 fe 41 55 49 89 f5 <41> 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24 40 e8 6a fb ff [139747.003229] auth[9545]: segfault at 558b5e039c30 ip 0000558b5e039c30 sp 00007ffd43633008 error 15 [139747.003239] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 2a ef 0e 75 7f 00 00 <40> a4 f5 0e 75 7f 00 00 21 80 00 00 00 00 00 00 00 74 01 5e 8b 55 [140640.900512] auth[17221]: segfault at 55e952d41c30 ip 000055e952d41c30 sp 00007ffc36fa0908 error 15 [140640.900523] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 5a ec 80 b9 7f 00 00 <40> d4 f2 80 b9 7f 00 00 21 80 00 00 00 00 00 00 00 f4 d1 52 e9 55 [140777.927292] auth[18114]: segfault at 55bd72adec30 ip 000055bd72adec30 sp 00007ffca394ba98 error 15 [140777.927302] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba cb 67 d7 7f 00 00 <40> 34 d2 67 d7 7f 00 00 21 80 00 00 00 00 00 00 00 c4 ab 72 bd 55
Please see https://dovecot.org/bugreport.html
Aki
Hi
How to solve this ?
So many similar segfaults
Thank you
On 2018-11-30 06:11, Joan Moreau wrote:
ANother (very very long) example :
# gdb /usr/libexec/dovecot/indexer-worker core.indexer-worker.0.3a33f56105e043de802a7dfcee265a07.21017.1543533424000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/indexer-worker...done. [New LWP 21017] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/indexer-worker'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 173 { (gdb) bt full #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 lock_type_str = <optimized out> started = <optimized out> ret = <optimized out> __func__ = "file_lock_do" #1 0x00007f768b62b5b6 in file_wait_lock_error (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=<optimized out>, lock_r=0x7fff04501118, error_r=0x7fff045010b0) at file-lock.c:318 lock = <optimized out> ret = <optimized out> #2 0x00007f768b62b660 in file_wait_lock (fd=<optimized out>, path=<optimized out>, lock_type=lock_type@entry=0, lock_method=<optimized out>, timeout_secs=timeout_secs@entry=60, lock_r=lock_r@entry=0x7fff04501118) at file-lock.c:303 error = 0x564540376490 "" ret = <optimized out> #3 0x00007f768a976c87 in squat_trie_lock (trie=0x564540376490, lock_type=0, file_lock_r=0x7fff04501118, dotlock_r=0x7fff04501120) at squat-trie.c:294 ret = <optimized out> dotlock_r = 0x7fff04501120 file_lock_r = 0x7fff04501118 trie = 0x564540376490 ret = <optimized out> __func__ = "squat_trie_lock" lock_type = 0 ret = <optimized out> __func__ = "squat_trie_lock" #4 0x00007f768a978627 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1487 file_lock = 0x0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #5 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #6 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #7 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #8 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #9 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c850 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #10 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #11 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #12 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #13 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #14 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c7b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #15 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #16 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #17 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #18 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #19 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c710 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #20 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #21 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #22 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #23 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #24 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c670 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #25 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #26 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #27 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #28 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #29 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c5d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #30 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #31 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #32 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #33 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #34 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c530 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #35 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #36 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #37 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #38 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #39 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c490 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #40 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #41 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #42 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #43 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #44 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c3f0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #45 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #46 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #47 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #48 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #49 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c350 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #50 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 #51 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #52 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #53 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #54 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c2b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #55 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #56 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #57 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #58 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #59 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c210 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #60 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #61 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #62 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #63 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #64 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c170 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #65 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #66 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #67 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #68 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #69 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c0d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #70 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #71 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #72 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #73 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #74 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c030 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #75 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #76 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #77 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #78 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #79 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bf90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #80 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #81 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #82 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #83 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #84 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bef0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #85 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #86 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #87 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #88 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #89 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210be50 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #90 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #91 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #92 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #93 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #94 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bdb0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #95 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #96 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #97 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #98 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #99 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bd10 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #100 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #101 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #102 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #103 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #104 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bc70 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #105 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #106 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #107 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #108 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #109 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bbd0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #110 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #111 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #112 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #113 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #114 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bb30 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #115 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #116 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #117 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #118 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #119 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210ba90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #120 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #121 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #122 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #123 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #124 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210b9f0 dotlock = 0x0 changed = <optimized out>
(continues for ages)
On 2018-11-29 19:36, Joan Moreau wrote:
finally managed to locate the dump
here the output:
# gdb /usr/libexec/dovecot/auth /var/lib/systemd/coredump/core.auth.0.3a33f56105e043de802a7dfcee265a07.28130.1543516118000000 GNU gdb (GDB) 8.2 (gdb) bt full #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007f739c347672 in abort () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007f739c39f878 in __libc_message () from /usr/lib/libc.so.6 No symbol table info available. #3 0x00007f739c3a618a in malloc_printerr () from /usr/lib/libc.so.6 No symbol table info available. #4 0x00007f739c3a7b27 in _int_free () from /usr/lib/libc.so.6 No symbol table info available. #5 0x00007f739cc33585 in mysql_close (mysql=0x5636af7abdc0) at /usr/src/mariadb-10.3.11/libmariadb/libmariadb/mariadb_lib.c:1939 mysql = 0x5636af7abdc0 p = <optimized out> p = <optimized out> #6 0x00005636aef08f7c in driver_sqlpool_disconnect (_db=0x5636af7aaf30) at driver-sqlpool.c:590 conn__foreach_end = 0x5636af7ab570 db = 0x5636af7aaf30 conn = 0x5636af7ab560 #7 0x00005636aeefed65 in db_sql_unref (_conn=0x5636af7b0020) at db-sql.c:128 conn = 0x5636af7ae388 #8 0x00005636aeef7e15 in userdb_deinit (userdb=0x5636af7afff8) at userdb.c:191 idx = 0 __func__ = "userdb_deinit" #9 0x00005636aeede209 in auth_deinit (auth=0x5636af7afd58) at auth.c:335 passdb = <optimized out> userdb = 0x5636af7aff18 passdb = <optimized out> userdb = <optimized out> #10 auths_deinit () at auth.c:433 auth__foreach_end = 0x5636af7ae2f8 auth = 0x5636af7ae2f0 #11 0x00005636aeedcf65 in main_deinit () at main.c:270 l = <optimized out> l = <optimized out> l_end = <optimized out> #12 main (argc=<optimized out>, argv=<optimized out>) at main.c:401 c = <optimized out>
On 27.11.2018 8.39, Joan Moreau wrote:
Thank you Aki
here the requested data (below)
Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)
Bug appears mostly in auth process and index-worker
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)
On 04 December 2018 at 16:46 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Hi
How to solve this ?
So many similar segfaults
Thank you
On 2018-11-30 06:11, Joan Moreau wrote:
ANother (very very long) example :
# gdb /usr/libexec/dovecot/indexer-worker core.indexer-worker.0.3a33f56105e043de802a7dfcee265a07.21017.1543533424000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/indexer-worker...done. [New LWP 21017] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/indexer-worker'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 173 { (gdb) bt full #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 lock_type_str = <optimized out> started = <optimized out> ret = <optimized out> __func__ = "file_lock_do" #1 0x00007f768b62b5b6 in file_wait_lock_error (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=<optimized out>, lock_r=0x7fff04501118, error_r=0x7fff045010b0) at file-lock.c:318 lock = <optimized out> ret = <optimized out> #2 0x00007f768b62b660 in file_wait_lock (fd=<optimized out>, path=<optimized out>, lock_type=lock_type@entry=0, lock_method=<optimized out>, timeout_secs=timeout_secs@entry=60, lock_r=lock_r@entry=0x7fff04501118) at file-lock.c:303 error = 0x564540376490 "" ret = <optimized out> #3 0x00007f768a976c87 in squat_trie_lock (trie=0x564540376490, lock_type=0, file_lock_r=0x7fff04501118, dotlock_r=0x7fff04501120) at squat-trie.c:294 ret = <optimized out> dotlock_r = 0x7fff04501120 file_lock_r = 0x7fff04501118 trie = 0x564540376490 ret = <optimized out> __func__ = "squat_trie_lock" lock_type = 0 ret = <optimized out> __func__ = "squat_trie_lock" #4 0x00007f768a978627 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1487 file_lock = 0x0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #5 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #6 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #7 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #8 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #9 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c850 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #10 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #11 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #12 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #13 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #14 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c7b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #15 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #16 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #17 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #18 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #19 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c710 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #20 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #21 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #22 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #23 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #24 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c670 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #25 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #26 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #27 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #28 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #29 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c5d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #30 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #31 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #32 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #33 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #34 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c530 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #35 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #36 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #37 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #38 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #39 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c490 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #40 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #41 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #42 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #43 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #44 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c3f0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #45 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #46 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #47 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #48 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #49 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c350 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #50 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 #51 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #52 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #53 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #54 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c2b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #55 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #56 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #57 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #58 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #59 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c210 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #60 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #61 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #62 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #63 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #64 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c170 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #65 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #66 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #67 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #68 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #69 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c0d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #70 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #71 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #72 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #73 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #74 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c030 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #75 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #76 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #77 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #78 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #79 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bf90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #80 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #81 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #82 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #83 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #84 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bef0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #85 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #86 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #87 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #88 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #89 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210be50 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #90 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #91 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #92 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #93 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #94 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bdb0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #95 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #96 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #97 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #98 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #99 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bd10 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #100 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #101 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #102 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #103 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #104 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bc70 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #105 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #106 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #107 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #108 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #109 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bbd0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #110 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #111 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #112 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #113 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #114 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bb30 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #115 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #116 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #117 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #118 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #119 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210ba90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #120 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #121 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #122 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #123 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #124 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210b9f0 dotlock = 0x0 changed = <optimized out>
(continues for ages)
On 2018-11-29 19:36, Joan Moreau wrote:
finally managed to locate the dump
here the output:
# gdb /usr/libexec/dovecot/auth /var/lib/systemd/coredump/core.auth.0.3a33f56105e043de802a7dfcee265a07.28130.1543516118000000 GNU gdb (GDB) 8.2 (gdb) bt full #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007f739c347672 in abort () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007f739c39f878 in __libc_message () from /usr/lib/libc.so.6 No symbol table info available. #3 0x00007f739c3a618a in malloc_printerr () from /usr/lib/libc.so.6 No symbol table info available. #4 0x00007f739c3a7b27 in _int_free () from /usr/lib/libc.so.6 No symbol table info available. #5 0x00007f739cc33585 in mysql_close (mysql=0x5636af7abdc0) at /usr/src/mariadb-10.3.11/libmariadb/libmariadb/mariadb_lib.c:1939 mysql = 0x5636af7abdc0 p = <optimized out> p = <optimized out> #6 0x00005636aef08f7c in driver_sqlpool_disconnect (_db=0x5636af7aaf30) at driver-sqlpool.c:590 conn__foreach_end = 0x5636af7ab570 db = 0x5636af7aaf30 conn = 0x5636af7ab560 #7 0x00005636aeefed65 in db_sql_unref (_conn=0x5636af7b0020) at db-sql.c:128 conn = 0x5636af7ae388 #8 0x00005636aeef7e15 in userdb_deinit (userdb=0x5636af7afff8) at userdb.c:191 idx = 0 __func__ = "userdb_deinit" #9 0x00005636aeede209 in auth_deinit (auth=0x5636af7afd58) at auth.c:335 passdb = <optimized out> userdb = 0x5636af7aff18 passdb = <optimized out> userdb = <optimized out> #10 auths_deinit () at auth.c:433 auth__foreach_end = 0x5636af7ae2f8 auth = 0x5636af7ae2f0 #11 0x00005636aeedcf65 in main_deinit () at main.c:270 l = <optimized out> l = <optimized out> l_end = <optimized out> #12 main (argc=<optimized out>, argv=<optimized out>) at main.c:401 c = <optimized out>
On 27.11.2018 8.39, Joan Moreau wrote:
Thank you Aki
here the requested data (below)
Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)
Bug appears mostly in auth process and index-worker
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values)
fts squat was already deprecated in 2.2, and it's considered obsolete in 2.3. We hopefully get around removing it from 2.4, but you should move away from it to Solr.
The MySQL crash has been fixed in master, see https://github.com/dovecot/core/commit/3c5101ffdd2a8115e03ed7180d53578765dea...
Aki
Thanks for mySql
For Squat vs Solr, Solr does not reach Squat by very far in terms of results : If I setup Solr, and search (via the search in Roundbube or Evolution) for a keyword or part of the keyword, the results are complete non sense. The diference between "search in full body" and "search in fields" does not even work.
Solr with Dovecot seems very early beta isnt'it ?
On 2018-12-04 15:57, Aki Tuomi wrote:
On 04 December 2018 at 16:46 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Hi
How to solve this ?
So many similar segfaults
Thank you
On 2018-11-30 06:11, Joan Moreau wrote:
ANother (very very long) example :
# gdb /usr/libexec/dovecot/indexer-worker core.indexer-worker.0.3a33f56105e043de802a7dfcee265a07.21017.1543533424000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/indexer-worker...done. [New LWP 21017] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/indexer-worker'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 173 { (gdb) bt full #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 lock_type_str = <optimized out> started = <optimized out> ret = <optimized out> __func__ = "file_lock_do" #1 0x00007f768b62b5b6 in file_wait_lock_error (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=<optimized out>, lock_r=0x7fff04501118, error_r=0x7fff045010b0) at file-lock.c:318 lock = <optimized out> ret = <optimized out> #2 0x00007f768b62b660 in file_wait_lock (fd=<optimized out>, path=<optimized out>, lock_type=lock_type@entry=0, lock_method=<optimized out>, timeout_secs=timeout_secs@entry=60, lock_r=lock_r@entry=0x7fff04501118) at file-lock.c:303 error = 0x564540376490 "" ret = <optimized out> #3 0x00007f768a976c87 in squat_trie_lock (trie=0x564540376490, lock_type=0, file_lock_r=0x7fff04501118, dotlock_r=0x7fff04501120) at squat-trie.c:294 ret = <optimized out> dotlock_r = 0x7fff04501120 file_lock_r = 0x7fff04501118 trie = 0x564540376490 ret = <optimized out> __func__ = "squat_trie_lock" lock_type = 0 ret = <optimized out> __func__ = "squat_trie_lock" #4 0x00007f768a978627 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1487 file_lock = 0x0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #5 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #6 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #7 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #8 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #9 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c850 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #10 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #11 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #12 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #13 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #14 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c7b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #15 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #16 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #17 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #18 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #19 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c710 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #20 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #21 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #22 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #23 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #24 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c670 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #25 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #26 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #27 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #28 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #29 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c5d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #30 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #31 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #32 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #33 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #34 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c530 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #35 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #36 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #37 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #38 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #39 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c490 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #40 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #41 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #42 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #43 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #44 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c3f0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #45 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #46 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #47 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #48 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #49 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c350 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #50 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 #51 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #52 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #53 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #54 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c2b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #55 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #56 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #57 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #58 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #59 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c210 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #60 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #61 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #62 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #63 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #64 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c170 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #65 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #66 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #67 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #68 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #69 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c0d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #70 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #71 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #72 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #73 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #74 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c030 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #75 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #76 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #77 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #78 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #79 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bf90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #80 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #81 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #82 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #83 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #84 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bef0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #85 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #86 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #87 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #88 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #89 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210be50 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #90 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #91 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #92 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #93 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #94 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bdb0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #95 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #96 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #97 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #98 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #99 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bd10 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #100 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #101 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #102 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #103 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #104 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bc70 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #105 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #106 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #107 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #108 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #109 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bbd0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #110 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #111 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #112 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #113 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #114 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bb30 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #115 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #116 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #117 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #118 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #119 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210ba90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #120 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #121 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #122 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #123 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #124 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210b9f0 dotlock = 0x0 changed = <optimized out>
(continues for ages)
On 2018-11-29 19:36, Joan Moreau wrote:
finally managed to locate the dump
here the output:
# gdb /usr/libexec/dovecot/auth /var/lib/systemd/coredump/core.auth.0.3a33f56105e043de802a7dfcee265a07.28130.1543516118000000 GNU gdb (GDB) 8.2 (gdb) bt full #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007f739c347672 in abort () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007f739c39f878 in __libc_message () from /usr/lib/libc.so.6 No symbol table info available. #3 0x00007f739c3a618a in malloc_printerr () from /usr/lib/libc.so.6 No symbol table info available. #4 0x00007f739c3a7b27 in _int_free () from /usr/lib/libc.so.6 No symbol table info available. #5 0x00007f739cc33585 in mysql_close (mysql=0x5636af7abdc0) at /usr/src/mariadb-10.3.11/libmariadb/libmariadb/mariadb_lib.c:1939 mysql = 0x5636af7abdc0 p = <optimized out> p = <optimized out> #6 0x00005636aef08f7c in driver_sqlpool_disconnect (_db=0x5636af7aaf30) at driver-sqlpool.c:590 conn__foreach_end = 0x5636af7ab570 db = 0x5636af7aaf30 conn = 0x5636af7ab560 #7 0x00005636aeefed65 in db_sql_unref (_conn=0x5636af7b0020) at db-sql.c:128 conn = 0x5636af7ae388 #8 0x00005636aeef7e15 in userdb_deinit (userdb=0x5636af7afff8) at userdb.c:191 idx = 0 __func__ = "userdb_deinit" #9 0x00005636aeede209 in auth_deinit (auth=0x5636af7afd58) at auth.c:335 passdb = <optimized out> userdb = 0x5636af7aff18 passdb = <optimized out> userdb = <optimized out> #10 auths_deinit () at auth.c:433 auth__foreach_end = 0x5636af7ae2f8 auth = 0x5636af7ae2f0 #11 0x00005636aeedcf65 in main_deinit () at main.c:270 l = <optimized out> l = <optimized out> l_end = <optimized out> #12 main (argc=<optimized out>, argv=<optimized out>) at main.c:401 c = <optimized out>
On 27.11.2018 8.39, Joan Moreau wrote:
Thank you Aki
here the requested data (below)
Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)
Bug appears mostly in auth process and index-worker
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values) fts squat was already deprecated in 2.2, and it's considered obsolete in 2.3. We hopefully get around removing it from 2.4, but you should move away from it to Solr.
The MySQL crash has been fixed in master, see https://github.com/dovecot/core/commit/3c5101ffdd2a8115e03ed7180d53578765dea...
Aki
We don't consider it as "very early beta". We consider it production ready. It is bit more work to set up though.
Aki
On 04 December 2018 at 17:16 Joan Moreau <jom@grosjo.net> wrote:
Thanks for mySql
For Squat vs Solr, Solr does not reach Squat by very far in terms of results : If I setup Solr, and search (via the search in Roundbube or Evolution) for a keyword or part of the keyword, the results are complete non sense. The diference between "search in full body" and "search in fields" does not even work.
Solr with Dovecot seems very early beta isnt'it ?
On 2018-12-04 15:57, Aki Tuomi wrote:
On 04 December 2018 at 16:46 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Hi
How to solve this ?
So many similar segfaults
Thank you
On 2018-11-30 06:11, Joan Moreau wrote:
ANother (very very long) example :
# gdb /usr/libexec/dovecot/indexer-worker core.indexer-worker.0.3a33f56105e043de802a7dfcee265a07.21017.1543533424000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/indexer-worker...done. [New LWP 21017] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/indexer-worker'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 173 { (gdb) bt full #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 lock_type_str = <optimized out> started = <optimized out> ret = <optimized out> __func__ = "file_lock_do" #1 0x00007f768b62b5b6 in file_wait_lock_error (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=<optimized out>, lock_r=0x7fff04501118, error_r=0x7fff045010b0) at file-lock.c:318 lock = <optimized out> ret = <optimized out> #2 0x00007f768b62b660 in file_wait_lock (fd=<optimized out>, path=<optimized out>, lock_type=lock_type@entry=0, lock_method=<optimized out>, timeout_secs=timeout_secs@entry=60, lock_r=lock_r@entry=0x7fff04501118) at file-lock.c:303 error = 0x564540376490 "" ret = <optimized out> #3 0x00007f768a976c87 in squat_trie_lock (trie=0x564540376490, lock_type=0, file_lock_r=0x7fff04501118, dotlock_r=0x7fff04501120) at squat-trie.c:294 ret = <optimized out> dotlock_r = 0x7fff04501120 file_lock_r = 0x7fff04501118 trie = 0x564540376490 ret = <optimized out> __func__ = "squat_trie_lock" lock_type = 0 ret = <optimized out> __func__ = "squat_trie_lock" #4 0x00007f768a978627 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1487 file_lock = 0x0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #5 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #6 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #7 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #8 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #9 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c850 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #10 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #11 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #12 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #13 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #14 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c7b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #15 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #16 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #17 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #18 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #19 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c710 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #20 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #21 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #22 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #23 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #24 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c670 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #25 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #26 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #27 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #28 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #29 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c5d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #30 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #31 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #32 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #33 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #34 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c530 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #35 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #36 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #37 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #38 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #39 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c490 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #40 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #41 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #42 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #43 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #44 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c3f0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #45 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #46 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #47 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #48 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #49 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c350 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #50 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 #51 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #52 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #53 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #54 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c2b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #55 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #56 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #57 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #58 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #59 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c210 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #60 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #61 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #62 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #63 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #64 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c170 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #65 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #66 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #67 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #68 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #69 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c0d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #70 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #71 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #72 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #73 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #74 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c030 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #75 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #76 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #77 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #78 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #79 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bf90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #80 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #81 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #82 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #83 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #84 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bef0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #85 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #86 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #87 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #88 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #89 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210be50 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #90 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #91 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #92 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #93 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #94 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bdb0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #95 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #96 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #97 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #98 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #99 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bd10 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #100 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #101 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #102 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #103 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #104 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bc70 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #105 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #106 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #107 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #108 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #109 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bbd0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #110 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #111 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #112 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #113 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #114 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bb30 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #115 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #116 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #117 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #118 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #119 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210ba90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #120 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #121 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #122 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #123 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #124 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210b9f0 dotlock = 0x0 changed = <optimized out>
(continues for ages)
On 2018-11-29 19:36, Joan Moreau wrote:
finally managed to locate the dump
here the output:
# gdb /usr/libexec/dovecot/auth /var/lib/systemd/coredump/core.auth.0.3a33f56105e043de802a7dfcee265a07.28130.1543516118000000 GNU gdb (GDB) 8.2 (gdb) bt full #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007f739c347672 in abort () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007f739c39f878 in __libc_message () from /usr/lib/libc.so.6 No symbol table info available. #3 0x00007f739c3a618a in malloc_printerr () from /usr/lib/libc.so.6 No symbol table info available. #4 0x00007f739c3a7b27 in _int_free () from /usr/lib/libc.so.6 No symbol table info available. #5 0x00007f739cc33585 in mysql_close (mysql=0x5636af7abdc0) at /usr/src/mariadb-10.3.11/libmariadb/libmariadb/mariadb_lib.c:1939 mysql = 0x5636af7abdc0 p = <optimized out> p = <optimized out> #6 0x00005636aef08f7c in driver_sqlpool_disconnect (_db=0x5636af7aaf30) at driver-sqlpool.c:590 conn__foreach_end = 0x5636af7ab570 db = 0x5636af7aaf30 conn = 0x5636af7ab560 #7 0x00005636aeefed65 in db_sql_unref (_conn=0x5636af7b0020) at db-sql.c:128 conn = 0x5636af7ae388 #8 0x00005636aeef7e15 in userdb_deinit (userdb=0x5636af7afff8) at userdb.c:191 idx = 0 __func__ = "userdb_deinit" #9 0x00005636aeede209 in auth_deinit (auth=0x5636af7afd58) at auth.c:335 passdb = <optimized out> userdb = 0x5636af7aff18 passdb = <optimized out> userdb = <optimized out> #10 auths_deinit () at auth.c:433 auth__foreach_end = 0x5636af7ae2f8 auth = 0x5636af7ae2f0 #11 0x00005636aeedcf65 in main_deinit () at main.c:270 l = <optimized out> l = <optimized out> l_end = <optimized out> #12 main (argc=<optimized out>, argv=<optimized out>) at main.c:401 c = <optimized out>
On 27.11.2018 8.39, Joan Moreau wrote:
Thank you Aki
here the requested data (below)
Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)
Bug appears mostly in auth process and index-worker
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values) fts squat was already deprecated in 2.2, and it's considered obsolete in 2.3. We hopefully get around removing it from 2.4, but you should move away from it to Solr.
The MySQL crash has been fixed in master, see https://github.com/dovecot/core/commit/3c5101ffdd2a8115e03ed7180d53578765dea...
Aki
So far, not getting any results with Solr. Squat however exactly meet the need.
On 2018-12-04 16:18, Aki Tuomi wrote:
We don't consider it as "very early beta". We consider it production ready. It is bit more work to set up though.
Aki
On 04 December 2018 at 17:16 Joan Moreau <jom@grosjo.net> wrote:
Thanks for mySql
For Squat vs Solr, Solr does not reach Squat by very far in terms of results : If I setup Solr, and search (via the search in Roundbube or Evolution) for a keyword or part of the keyword, the results are complete non sense. The diference between "search in full body" and "search in fields" does not even work.
Solr with Dovecot seems very early beta isnt'it ?
On 2018-12-04 15:57, Aki Tuomi wrote:
On 04 December 2018 at 16:46 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Hi
How to solve this ?
So many similar segfaults
Thank you
On 2018-11-30 06:11, Joan Moreau wrote:
ANother (very very long) example :
# gdb /usr/libexec/dovecot/indexer-worker core.indexer-worker.0.3a33f56105e043de802a7dfcee265a07.21017.1543533424000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/indexer-worker...done. [New LWP 21017] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/indexer-worker'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 173 { (gdb) bt full #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 lock_type_str = <optimized out> started = <optimized out> ret = <optimized out> __func__ = "file_lock_do" #1 0x00007f768b62b5b6 in file_wait_lock_error (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=<optimized out>, lock_r=0x7fff04501118, error_r=0x7fff045010b0) at file-lock.c:318 lock = <optimized out> ret = <optimized out> #2 0x00007f768b62b660 in file_wait_lock (fd=<optimized out>, path=<optimized out>, lock_type=lock_type@entry=0, lock_method=<optimized out>, timeout_secs=timeout_secs@entry=60, lock_r=lock_r@entry=0x7fff04501118) at file-lock.c:303 error = 0x564540376490 "" ret = <optimized out> #3 0x00007f768a976c87 in squat_trie_lock (trie=0x564540376490, lock_type=0, file_lock_r=0x7fff04501118, dotlock_r=0x7fff04501120) at squat-trie.c:294 ret = <optimized out> dotlock_r = 0x7fff04501120 file_lock_r = 0x7fff04501118 trie = 0x564540376490 ret = <optimized out> __func__ = "squat_trie_lock" lock_type = 0 ret = <optimized out> __func__ = "squat_trie_lock" #4 0x00007f768a978627 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1487 file_lock = 0x0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #5 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #6 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #7 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #8 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #9 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c850 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #10 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #11 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #12 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #13 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #14 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c7b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #15 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #16 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #17 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #18 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #19 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c710 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #20 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #21 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #22 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #23 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #24 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c670 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #25 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #26 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #27 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #28 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #29 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c5d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #30 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #31 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #32 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #33 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #34 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c530 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #35 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #36 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #37 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #38 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #39 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c490 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #40 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #41 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #42 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #43 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #44 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c3f0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #45 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #46 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #47 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #48 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #49 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c350 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #50 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 #51 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #52 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #53 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #54 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c2b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #55 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #56 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #57 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #58 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #59 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c210 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #60 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #61 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #62 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #63 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #64 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c170 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #65 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #66 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #67 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #68 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #69 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c0d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #70 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #71 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #72 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #73 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #74 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c030 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #75 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #76 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #77 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #78 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #79 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bf90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #80 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #81 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #82 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #83 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #84 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bef0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #85 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #86 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #87 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #88 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #89 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210be50 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #90 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #91 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #92 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #93 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #94 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bdb0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #95 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #96 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #97 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #98 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #99 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bd10 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #100 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #101 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #102 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #103 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #104 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bc70 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #105 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #106 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #107 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #108 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #109 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bbd0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #110 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #111 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #112 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #113 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #114 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bb30 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #115 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #116 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #117 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #118 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #119 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210ba90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #120 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #121 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #122 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #123 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #124 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210b9f0 dotlock = 0x0 changed = <optimized out>
(continues for ages)
On 2018-11-29 19:36, Joan Moreau wrote:
finally managed to locate the dump
here the output:
# gdb /usr/libexec/dovecot/auth /var/lib/systemd/coredump/core.auth.0.3a33f56105e043de802a7dfcee265a07.28130.1543516118000000 GNU gdb (GDB) 8.2 (gdb) bt full #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007f739c347672 in abort () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007f739c39f878 in __libc_message () from /usr/lib/libc.so.6 No symbol table info available. #3 0x00007f739c3a618a in malloc_printerr () from /usr/lib/libc.so.6 No symbol table info available. #4 0x00007f739c3a7b27 in _int_free () from /usr/lib/libc.so.6 No symbol table info available. #5 0x00007f739cc33585 in mysql_close (mysql=0x5636af7abdc0) at /usr/src/mariadb-10.3.11/libmariadb/libmariadb/mariadb_lib.c:1939 mysql = 0x5636af7abdc0 p = <optimized out> p = <optimized out> #6 0x00005636aef08f7c in driver_sqlpool_disconnect (_db=0x5636af7aaf30) at driver-sqlpool.c:590 conn__foreach_end = 0x5636af7ab570 db = 0x5636af7aaf30 conn = 0x5636af7ab560 #7 0x00005636aeefed65 in db_sql_unref (_conn=0x5636af7b0020) at db-sql.c:128 conn = 0x5636af7ae388 #8 0x00005636aeef7e15 in userdb_deinit (userdb=0x5636af7afff8) at userdb.c:191 idx = 0 __func__ = "userdb_deinit" #9 0x00005636aeede209 in auth_deinit (auth=0x5636af7afd58) at auth.c:335 passdb = <optimized out> userdb = 0x5636af7aff18 passdb = <optimized out> userdb = <optimized out> #10 auths_deinit () at auth.c:433 auth__foreach_end = 0x5636af7ae2f8 auth = 0x5636af7ae2f0 #11 0x00005636aeedcf65 in main_deinit () at main.c:270 l = <optimized out> l = <optimized out> l_end = <optimized out> #12 main (argc=<optimized out>, argv=<optimized out>) at main.c:401 c = <optimized out> On 27.11.2018 8.39, Joan Moreau wrote:
Thank you Aki
here the requested data (below)
Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)
Bug appears mostly in auth process and index-worker
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values) fts squat was already deprecated in 2.2, and it's considered obsolete in 2.3. We hopefully get around removing it from 2.4, but you should move away from it to Solr.
The MySQL crash has been fixed in master, see https://github.com/dovecot/core/commit/3c5101ffdd2a8115e03ed7180d53578765dea...
Aki
On December 4, 2018 at 8:18 AM Aki Tuomi <aki.tuomi@open-xchange.com> wrote:
We don't consider it as "very early beta". We consider it production ready. It is bit more work to set up though.
FWIW, Squat has been deprecated since 2.0, so none of this should come as a surprise.
https://wiki.dovecot.org/Plugins/FTS
Aki
On 04 December 2018 at 17:16 Joan Moreau <jom@grosjo.net> wrote:
Thanks for mySql
For Squat vs Solr, Solr does not reach Squat by very far in terms of results : If I setup Solr, and search (via the search in Roundbube or Evolution) for a keyword or part of the keyword, the results are complete non sense. The diference between "search in full body" and "search in fields" does not even work.
Solr with Dovecot seems very early beta isnt'it ?
On 2018-12-04 15:57, Aki Tuomi wrote:
On 04 December 2018 at 16:46 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Hi
How to solve this ?
So many similar segfaults
Thank you
On 2018-11-30 06:11, Joan Moreau wrote:
ANother (very very long) example :
# gdb /usr/libexec/dovecot/indexer-worker core.indexer-worker.0.3a33f56105e043de802a7dfcee265a07.21017.1543533424000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/indexer-worker...done. [New LWP 21017] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/indexer-worker'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 173 { (gdb) bt full #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 lock_type_str = <optimized out> started = <optimized out> ret = <optimized out> __func__ = "file_lock_do" #1 0x00007f768b62b5b6 in file_wait_lock_error (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=<optimized out>, lock_r=0x7fff04501118, error_r=0x7fff045010b0) at file-lock.c:318 lock = <optimized out> ret = <optimized out> #2 0x00007f768b62b660 in file_wait_lock (fd=<optimized out>, path=<optimized out>, lock_type=lock_type@entry=0, lock_method=<optimized out>, timeout_secs=timeout_secs@entry=60, lock_r=lock_r@entry=0x7fff04501118) at file-lock.c:303 error = 0x564540376490 "" ret = <optimized out> #3 0x00007f768a976c87 in squat_trie_lock (trie=0x564540376490, lock_type=0, file_lock_r=0x7fff04501118, dotlock_r=0x7fff04501120) at squat-trie.c:294 ret = <optimized out> dotlock_r = 0x7fff04501120 file_lock_r = 0x7fff04501118 trie = 0x564540376490 ret = <optimized out> __func__ = "squat_trie_lock" lock_type = 0 ret = <optimized out> __func__ = "squat_trie_lock" #4 0x00007f768a978627 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1487 file_lock = 0x0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #5 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #6 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #7 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #8 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #9 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c850 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #10 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #11 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #12 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #13 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #14 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c7b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #15 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #16 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #17 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #18 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #19 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c710 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #20 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #21 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #22 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #23 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #24 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c670 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #25 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #26 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #27 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #28 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #29 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c5d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #30 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #31 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #32 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #33 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #34 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c530 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #35 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #36 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #37 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #38 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #39 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c490 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #40 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #41 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #42 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #43 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #44 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c3f0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #45 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #46 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #47 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #48 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #49 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c350 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #50 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 #51 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #52 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #53 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #54 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c2b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #55 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #56 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #57 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #58 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #59 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c210 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #60 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #61 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #62 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #63 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #64 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c170 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #65 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #66 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #67 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #68 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #69 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c0d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #70 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #71 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #72 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #73 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #74 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c030 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #75 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #76 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #77 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #78 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #79 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bf90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #80 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #81 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #82 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #83 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #84 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bef0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #85 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #86 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #87 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #88 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #89 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210be50 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #90 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #91 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #92 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #93 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #94 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bdb0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #95 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #96 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #97 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #98 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #99 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bd10 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #100 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #101 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #102 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #103 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #104 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bc70 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #105 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #106 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #107 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #108 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #109 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bbd0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #110 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #111 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #112 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #113 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #114 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bb30 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #115 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #116 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #117 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #118 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #119 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210ba90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #120 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #121 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #122 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #123 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #124 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210b9f0 dotlock = 0x0 changed = <optimized out>
(continues for ages)
On 2018-11-29 19:36, Joan Moreau wrote:
finally managed to locate the dump
here the output:
# gdb /usr/libexec/dovecot/auth /var/lib/systemd/coredump/core.auth.0.3a33f56105e043de802a7dfcee265a07.28130.1543516118000000 GNU gdb (GDB) 8.2 (gdb) bt full #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007f739c347672 in abort () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007f739c39f878 in __libc_message () from /usr/lib/libc.so.6 No symbol table info available. #3 0x00007f739c3a618a in malloc_printerr () from /usr/lib/libc.so.6 No symbol table info available. #4 0x00007f739c3a7b27 in _int_free () from /usr/lib/libc.so.6 No symbol table info available. #5 0x00007f739cc33585 in mysql_close (mysql=0x5636af7abdc0) at /usr/src/mariadb-10.3.11/libmariadb/libmariadb/mariadb_lib.c:1939 mysql = 0x5636af7abdc0 p = <optimized out> p = <optimized out> #6 0x00005636aef08f7c in driver_sqlpool_disconnect (_db=0x5636af7aaf30) at driver-sqlpool.c:590 conn__foreach_end = 0x5636af7ab570 db = 0x5636af7aaf30 conn = 0x5636af7ab560 #7 0x00005636aeefed65 in db_sql_unref (_conn=0x5636af7b0020) at db-sql.c:128 conn = 0x5636af7ae388 #8 0x00005636aeef7e15 in userdb_deinit (userdb=0x5636af7afff8) at userdb.c:191 idx = 0 __func__ = "userdb_deinit" #9 0x00005636aeede209 in auth_deinit (auth=0x5636af7afd58) at auth.c:335 passdb = <optimized out> userdb = 0x5636af7aff18 passdb = <optimized out> userdb = <optimized out> #10 auths_deinit () at auth.c:433 auth__foreach_end = 0x5636af7ae2f8 auth = 0x5636af7ae2f0 #11 0x00005636aeedcf65 in main_deinit () at main.c:270 l = <optimized out> l = <optimized out> l_end = <optimized out> #12 main (argc=<optimized out>, argv=<optimized out>) at main.c:401 c = <optimized out>
On 27.11.2018 8.39, Joan Moreau wrote:
Thank you Aki
here the requested data (below)
Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)
Bug appears mostly in auth process and index-worker
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values) fts squat was already deprecated in 2.2, and it's considered obsolete in 2.3. We hopefully get around removing it from 2.4, but you should move away from it to Solr.
The MySQL crash has been fixed in master, see https://github.com/dovecot/core/commit/3c5101ffdd2a8115e03ed7180d53578765dea...
Aki
Yes, but the bottom line is that Squat does the job needed for end users, Solr does not
On 2018-12-04 16:53, Michael Slusarz wrote:
On December 4, 2018 at 8:18 AM Aki Tuomi <aki.tuomi@open-xchange.com> wrote:
We don't consider it as "very early beta". We consider it production ready. It is bit more work to set up though.
FWIW, Squat has been deprecated since 2.0, so none of this should come as a surprise.
https://wiki.dovecot.org/Plugins/FTS
Aki
On 04 December 2018 at 17:16 Joan Moreau <jom@grosjo.net> wrote:
Thanks for mySql
For Squat vs Solr, Solr does not reach Squat by very far in terms of results : If I setup Solr, and search (via the search in Roundbube or Evolution) for a keyword or part of the keyword, the results are complete non sense. The diference between "search in full body" and "search in fields" does not even work.
Solr with Dovecot seems very early beta isnt'it ?
On 2018-12-04 15:57, Aki Tuomi wrote:
On 04 December 2018 at 16:46 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Hi
How to solve this ?
So many similar segfaults
Thank you
On 2018-11-30 06:11, Joan Moreau wrote:
ANother (very very long) example :
# gdb /usr/libexec/dovecot/indexer-worker core.indexer-worker.0.3a33f56105e043de802a7dfcee265a07.21017.1543533424000000 GNU gdb (GDB) 8.2 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/indexer-worker...done. [New LWP 21017] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `dovecot/indexer-worker'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 173 { (gdb) bt full #0 0x00007f768b62b13e in file_lock_do (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, error_r=0x7fff045010b0) at file-lock.c:173 lock_type_str = <optimized out> started = <optimized out> ret = <optimized out> __func__ = "file_lock_do" #1 0x00007f768b62b5b6 in file_wait_lock_error (fd=18, path=0x564540376790 "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search", lock_type=0, lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=<optimized out>, lock_r=0x7fff04501118, error_r=0x7fff045010b0) at file-lock.c:318 lock = <optimized out> ret = <optimized out> #2 0x00007f768b62b660 in file_wait_lock (fd=<optimized out>, path=<optimized out>, lock_type=lock_type@entry=0, lock_method=<optimized out>, timeout_secs=timeout_secs@entry=60, lock_r=lock_r@entry=0x7fff04501118) at file-lock.c:303 error = 0x564540376490 "" ret = <optimized out> #3 0x00007f768a976c87 in squat_trie_lock (trie=0x564540376490, lock_type=0, file_lock_r=0x7fff04501118, dotlock_r=0x7fff04501120) at squat-trie.c:294 ret = <optimized out> dotlock_r = 0x7fff04501120 file_lock_r = 0x7fff04501118 trie = 0x564540376490 ret = <optimized out> __func__ = "squat_trie_lock" lock_type = 0 ret = <optimized out> __func__ = "squat_trie_lock" #4 0x00007f768a978627 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1487 file_lock = 0x0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #5 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #6 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #7 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #8 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #9 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c850 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #10 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #11 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #12 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #13 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #14 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c7b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #15 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #16 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #17 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #18 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #19 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c710 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #20 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #21 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #22 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #23 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #24 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c670 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #25 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #26 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #27 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #28 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #29 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c5d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #30 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #31 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #32 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #33 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #34 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c530 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #35 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #36 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #37 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #38 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #39 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c490 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #40 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #41 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #42 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #43 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #44 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c3f0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #45 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #46 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #47 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #48 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #49 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c350 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #50 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 #51 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #52 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #53 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #54 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c2b0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #55 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #56 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #57 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #58 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #59 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c210 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #60 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #61 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #62 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #63 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #64 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c170 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #65 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #66 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #67 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #68 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #69 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c0d0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #70 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #71 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #72 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #73 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #74 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210c030 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #75 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #76 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #77 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #78 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #79 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bf90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #80 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #81 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #82 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #83 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #84 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bef0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #85 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #86 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #87 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #88 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #89 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210be50 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #90 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #91 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #92 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #93 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #94 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bdb0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #95 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #96 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #97 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #98 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #99 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bd10 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #100 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #101 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #102 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #103 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #104 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bc70 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #105 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #106 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #107 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #108 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #109 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bbd0 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #110 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #111 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #112 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #113 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #114 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210bb30 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #115 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #116 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #117 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #118 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #119 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210ba90 dotlock = 0x0 changed = <optimized out> ret = <optimized out> #120 0x00007f768a97b19d in squat_uidlist_map_header (uidlist=0x5645403767f0) at squat-uidlist.c:378 No locals. #121 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477 mmap_hdr = <optimized out> ret = <optimized out> #122 0x00007f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at squat-uidlist.c:546 No locals. #123 0x00007f768a97b5aa in squat_uidlist_refresh (uidlist=<optimized out>) at squat-uidlist.c:569 No locals. #124 0x00007f768a9787c2 in squat_trie_map (trie=0x564540376490, building=<optimized out>) at squat-trie.c:1526 file_lock = 0x56454210b9f0 dotlock = 0x0 changed = <optimized out>
(continues for ages)
On 2018-11-29 19:36, Joan Moreau wrote:
finally managed to locate the dump
here the output:
# gdb /usr/libexec/dovecot/auth /var/lib/systemd/coredump/core.auth.0.3a33f56105e043de802a7dfcee265a07.28130.1543516118000000 GNU gdb (GDB) 8.2 (gdb) bt full #0 0x00007f739c35cd7f in raise () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007f739c347672 in abort () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007f739c39f878 in __libc_message () from /usr/lib/libc.so.6 No symbol table info available. #3 0x00007f739c3a618a in malloc_printerr () from /usr/lib/libc.so.6 No symbol table info available. #4 0x00007f739c3a7b27 in _int_free () from /usr/lib/libc.so.6 No symbol table info available. #5 0x00007f739cc33585 in mysql_close (mysql=0x5636af7abdc0) at /usr/src/mariadb-10.3.11/libmariadb/libmariadb/mariadb_lib.c:1939 mysql = 0x5636af7abdc0 p = <optimized out> p = <optimized out> #6 0x00005636aef08f7c in driver_sqlpool_disconnect (_db=0x5636af7aaf30) at driver-sqlpool.c:590 conn__foreach_end = 0x5636af7ab570 db = 0x5636af7aaf30 conn = 0x5636af7ab560 #7 0x00005636aeefed65 in db_sql_unref (_conn=0x5636af7b0020) at db-sql.c:128 conn = 0x5636af7ae388 #8 0x00005636aeef7e15 in userdb_deinit (userdb=0x5636af7afff8) at userdb.c:191 idx = 0 __func__ = "userdb_deinit" #9 0x00005636aeede209 in auth_deinit (auth=0x5636af7afd58) at auth.c:335 passdb = <optimized out> userdb = 0x5636af7aff18 passdb = <optimized out> userdb = <optimized out> #10 auths_deinit () at auth.c:433 auth__foreach_end = 0x5636af7ae2f8 auth = 0x5636af7ae2f0 #11 0x00005636aeedcf65 in main_deinit () at main.c:270 l = <optimized out> l = <optimized out> l_end = <optimized out> #12 main (argc=<optimized out>, argv=<optimized out>) at main.c:401 c = <optimized out> On 27.11.2018 8.39, Joan Moreau wrote:
Thank you Aki
here the requested data (below)
Please not as well that we have numerous subfolders (>50) and pretty big mailbox sizes (>20G)
Bug appears mostly in auth process and index-worker
dovecot -n :
# 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf # Pigeonhole version 0.6.devel (65909cfa) # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4 # Hostname: gjserver base_dir = /run/dovecot default_login_user = dovecot default_vsz_limit = 16 G disable_plaintext_auth = no listen = * log_path = /var/log/mail/dovecot.log mail_gid = mail mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n mail_plugins = fts fts_squat mail_uid = mailusers managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 24 M
(...)
passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } (the rest default values) fts squat was already deprecated in 2.2, and it's considered obsolete in 2.3. We hopefully get around removing it from 2.4, but you should move away from it to Solr.
The MySQL crash has been fixed in master, see https://github.com/dovecot/core/commit/3c5101ffdd2a8115e03ed7180d53578765dea...
Aki
Hi
I am giving Solr another try.
Using Solr 7.5 (from Archlinux AUR), I do not see anymore the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr)
Maybe some things have moved ?
Thank you so much
In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command :
sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance
before updating the schema.xml .
Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users
Additionaly, the url is http://(solr_ [1]server):8983/solr/dovecot/ (error in wiki)
On 2018-12-04 19:01, Joan Moreau via dovecot wrote:
Hi
I am giving Solr another try.
Using Solr 7.5 (from Archlinux AUR), I do not see anymore the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr)
Maybe some things have moved ?
Thank you so much
Links:
[1] http://(solr
also, the value "break-imap-search" is not supported in fts_solr parameter, contrary to the wiki
and not sure then how to enable full text search in BODY
On 2018-12-04 19:40, Joan Moreau via dovecot wrote:
In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command :
sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance
before updating the schema.xml .
Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users
Additionaly, the url is http://(solr_ [1]server):8983/solr/dovecot/ (error in wiki)
On 2018-12-04 19:01, Joan Moreau via dovecot wrote:
Hi
I am giving Solr another try.
Using Solr 7.5 (from Archlinux AUR), I do not see anymore the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr)
Maybe some things have moved ?
Thank you so much
Links:
[1] http://(solr
Full text search gets enabled automatically, although your probably want to set fts_enforced=yes to make sure it's used for all kinds of things automatically. break-imap-search is supported, but does nothing.
Updated the wiki a bit.
Aki
On 4.12.2018 20.50, Joan Moreau via dovecot wrote:
also, the value "break-imap-search" is not supported in fts_solr parameter, contrary to the wiki
and not sure then how to enable full text search in BODY
On 2018-12-04 19:40, Joan Moreau via dovecot wrote:
In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command :
sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance
before updating the schema.xml .
Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users
Additionaly, the url is http://(solr_ <http://(solr>server):8983/solr/dovecot/ (error in wiki)
On 2018-12-04 19:01, Joan Moreau via dovecot wrote:
Hi I am giving Solr another try. Using Solr 7.5 (from Archlinux AUR), I do not see anymore the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr) Maybe some things have moved ? Thank you so much
Well, "break-imap-search" option prevent dovecot from starting.
I used solr on one server since yesterday.
First feedback is that
BODY search is not working
FTS is not really working in headers : For instance, if you search "kliinik" but you type "kliini" (missing one k), the search does not return anything , whereas it shall return more results (as the keyword is smaller) that the intended search
On 2018-12-05 07:55, Aki Tuomi wrote:
Full text search gets enabled automatically, although your probably want to set fts_enforced=yes to make sure it's used for all kinds of things automatically. break-imap-search is supported, but does nothing.
Updated the wiki a bit.
Aki
On 4.12.2018 20.50, Joan Moreau via dovecot wrote:
also, the value "break-imap-search" is not supported in fts_solr parameter, contrary to the wiki
and not sure then how to enable full text search in BODY
On 2018-12-04 19:40, Joan Moreau via dovecot wrote:
In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command :
sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance
before updating the schema.xml .
Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users
Additionaly, the url is http://(solr_ [1]server):8983/solr/dovecot/ (error in wiki)
On 2018-12-04 19:01, Joan Moreau via dovecot wrote:
Hi
I am giving Solr another try.
Using Solr 7.5 (from Archlinux AUR), I do not see anymore the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr)
Maybe some things have moved ?
Thank you so much
Links:
[1] http://(solr
Can you provide doveconf -n
did you set plugin { fts_enforced=yes }
Aki
On 5.12.2018 11.50, Joan Moreau wrote:
Well, "break-imap-search" option prevent dovecot from starting.
I used solr on one server since yesterday.
First feedback is that
BODY search is not working
FTS is not really working in headers : For instance, if you search "kliinik" but you type "kliini" (missing one k), the search does not return anything , whereas it shall return more results (as the keyword is smaller) that the intended search
On 2018-12-05 07:55, Aki Tuomi wrote:
Full text search gets enabled automatically, although your probably want to set fts_enforced=yes to make sure it's used for all kinds of things automatically. break-imap-search is supported, but does nothing.
Updated the wiki a bit.
Aki
On 4.12.2018 20.50, Joan Moreau via dovecot wrote:
also, the value "break-imap-search" is not supported in fts_solr parameter, contrary to the wiki
and not sure then how to enable full text search in BODY
On 2018-12-04 19:40, Joan Moreau via dovecot wrote:
In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command : sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance before updating the schema.xml . Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users Additionaly, the url is http://(solr_ <http://(solr>server):8983/solr/dovecot/ (error in wiki) On 2018-12-04 19:01, Joan Moreau via dovecot wrote: Hi I am giving Solr another try. Using Solr 7.5 (from Archlinux AUR), I do not see anymore the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr) Maybe some things have moved ? Thank you so much
It seems we forgot to document that "break-imap-search" was dropped in v2.3. That has now been updated.
Also Solr does not support prefix/substring search unless you configure solr to support it.
Aki
On 5.12.2018 11.50, Joan Moreau wrote:
Well, "break-imap-search" option prevent dovecot from starting.
I used solr on one server since yesterday.
First feedback is that
BODY search is not working
FTS is not really working in headers : For instance, if you search "kliinik" but you type "kliini" (missing one k), the search does not return anything , whereas it shall return more results (as the keyword is smaller) that the intended search
On 2018-12-05 07:55, Aki Tuomi wrote:
Full text search gets enabled automatically, although your probably want to set fts_enforced=yes to make sure it's used for all kinds of things automatically. break-imap-search is supported, but does nothing.
Updated the wiki a bit.
Aki
On 4.12.2018 20.50, Joan Moreau via dovecot wrote:
also, the value "break-imap-search" is not supported in fts_solr parameter, contrary to the wiki
and not sure then how to enable full text search in BODY
On 2018-12-04 19:40, Joan Moreau via dovecot wrote:
In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command : sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance before updating the schema.xml . Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users Additionaly, the url is http://(solr_ <http://(solr>server):8983/solr/dovecot/ (error in wiki) On 2018-12-04 19:01, Joan Moreau via dovecot wrote: Hi I am giving Solr another try. Using Solr 7.5 (from Archlinux AUR), I do not see anymore the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr) Maybe some things have moved ? Thank you so much
THen the Squat shall be maintained until the SOlr plugin is upgraded, as Squat does resolve the problem (fts, partial search, etc...)
On 2018-12-05 12:20, Aki Tuomi wrote:
It seems we forgot to document that "break-imap-search" was dropped in v2.3. That has now been updated.
Also Solr does not support prefix/substring search unless you configure solr to support it.
Aki
On 5.12.2018 11.50, Joan Moreau wrote:
Well, "break-imap-search" option prevent dovecot from starting.
I used solr on one server since yesterday.
First feedback is that
BODY search is not working
FTS is not really working in headers : For instance, if you search "kliinik" but you type "kliini" (missing one k), the search does not return anything , whereas it shall return more results (as the keyword is smaller) that the intended search
On 2018-12-05 07:55, Aki Tuomi wrote:
Full text search gets enabled automatically, although your probably want to set fts_enforced=yes to make sure it's used for all kinds of things automatically. break-imap-search is supported, but does nothing.
Updated the wiki a bit.
Aki
On 4.12.2018 20.50, Joan Moreau via dovecot wrote:
also, the value "break-imap-search" is not supported in fts_solr parameter, contrary to the wiki
and not sure then how to enable full text search in BODY
On 2018-12-04 19:40, Joan Moreau via dovecot wrote:
In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command :
sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance
before updating the schema.xml .
Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users
Additionaly, the url is http://(solr_ [1]server):8983/solr/dovecot/ (error in wiki)
On 2018-12-04 19:01, Joan Moreau via dovecot wrote:
Hi
I am giving Solr another try.
Using Solr 7.5 (from Archlinux AUR), I do not see anymore the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr)
Maybe some things have moved ?
Thank you so much
Links:
[1] http://(solr
Squat has not been maintained since 2.2.something, and shall not be maintained. Sorry. If you want, you can become maintainer for squat plugin.
Aki
On 5.12.2018 13.57, Joan Moreau wrote:
THen the Squat shall be maintained until the SOlr plugin is upgraded, as Squat does resolve the problem (fts, partial search, etc...)
On 2018-12-05 12:20, Aki Tuomi wrote:
It seems we forgot to document that "break-imap-search" was dropped in v2.3. That has now been updated.
Also Solr does not support prefix/substring search unless you configure solr to support it.
Aki
On 5.12.2018 11.50, Joan Moreau wrote:
Well, "break-imap-search" option prevent dovecot from starting.
I used solr on one server since yesterday.
First feedback is that
BODY search is not working
FTS is not really working in headers : For instance, if you search "kliinik" but you type "kliini" (missing one k), the search does not return anything , whereas it shall return more results (as the keyword is smaller) that the intended search
On 2018-12-05 07:55, Aki Tuomi wrote:
Full text search gets enabled automatically, although your probably want to set fts_enforced=yes to make sure it's used for all kinds of things automatically. break-imap-search is supported, but does nothing. Updated the wiki a bit. Aki On 4.12.2018 20.50, Joan Moreau via dovecot wrote: also, the value "break-imap-search" is not supported in fts_solr parameter, contrary to the wiki and not sure then how to enable full text search in BODY On 2018-12-04 19:40, Joan Moreau via dovecot wrote: In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command : sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance before updating the schema.xml . Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users Additionaly, the url is http://(solr_ <http://(solr>server):8983/solr/dovecot/ (error in wiki) On 2018-12-04 19:01, Joan Moreau via dovecot wrote: Hi I am giving Solr another try. Using Solr 7.5 (from Archlinux AUR), I do not see anymore the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr) Maybe some things have moved ? Thank you so much
Additionally, I get "Invalid search response" on my Android , using dovecot-solr, whereas squat works fine
Why making squat obsolete ? It is simple and straightforward
On December 5, 2018 12:57:22 PM Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
THen the Squat shall be maintained until the SOlr plugin is upgraded, as Squat does resolve the problem (fts, partial search, etc...)
On 2018-12-05 12:20, Aki Tuomi wrote:
It seems we forgot to document that "break-imap-search" was dropped in v2.3. That has now been updated. Also Solr does not support prefix/substring search unless you configure solr to support it. Aki On 5.12.2018 11.50, Joan Moreau wrote:
Well, "break-imap-search" option prevent dovecot from starting.
I used solr on one server since yesterday.
First feedback is that that the intended search
- BODY search is not working
- FTS is not really working in headers : For instance, if you search "kliinik" but you type "kliini" (missing one k), the search does not return anything , whereas it shall return more results (as the keyword is smaller)
On 2018-12-05 07:55, Aki Tuomi wrote: Full text search gets enabled automatically, although your probably want to set fts_enforced=yes to make sure it's used for all kinds of things automatically. break-imap-search is supported, but does nothing. Updated the wiki a bit. Aki On 4.12.2018 20.50, Joan Moreau via dovecot wrote: also, the value "break-imap-search" is not supported in fts_solr parameter, contrary to the wiki and not sure then how to enable full text search in BODY
On 2018-12-04 19:40, Joan Moreau via dovecot wrote: In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command : sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance before updating the schema.xml . Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users Additionaly, the url is http://(solr_server):8983/solr/dovecot/ (error in wiki)
On 2018-12-04 19:01, Joan Moreau via dovecot wrote: Hi I am giving Solr another try. Using Solr 7.5 (from Archlinux AUR), I do not see anymore the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr) Maybe some things have moved ? Thank you so much
After some testsing, I managed to get proper functionning - The schema.xml is attached below (quite different from the one provided on teh wiki) (in bold the core differences) (NGramFilterFactory is the class that replace the fts_squat "partial=3 full=15", everything else is just a big hammer to smash a tiny fly) - One need to remove the "managed-schema" file in the {prefix}/server/solr/dovecot/conf. - One need to remove everything under {prefix}/server/solr/dovecot/data/ - The {prefix}/server/solr/dovecot/conf/solrconfig.xml is as below (see diff) - my dovecot.conf is : fts = solr fts_autoindex = yes fts_enforced = yes fts_decoder = decode2text fts_solr = url=http://(SOLR SERVER):8983/solr/dovecot/ --- schema.xml <?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <FIELDTYPE NAME="STRING" CLASS="SOLR.STRFIELD" /> <FIELDTYPE NAME="LONG" CLASS="SOLR.TRIELONGFIELD" /> <FIELDTYPE NAME="BOOLEAN" CLASS="SOLR.BOOLFIELD" /> <fieldType name="text" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <FILTER CLASS="SOLR.NGRAMFILTERFACTORY" MINGRAMSIZE="3" MAXGRAMSIZE="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <FILTER CLASS="SOLR.NGRAMFILTERFACTORY" MINGRAMSIZE="3" MAXGRAMSIZE="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> <field name="_version_" type="long" indexed="true" stored="true"/> <field name="bcc" type="text" indexed="true" stored="false"/> <field name="body" type="text" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="text" indexed="true" stored="false"/> <field name="from" type="text" indexed="true" stored="false"/> <field name="hdr" type="text" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="text" indexed="true" stored="false"/> <field name="to" type="text" indexed="true" stored="false"/> <field name="uid" type="long" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema> ------ diff solrconfig --- /data/backup/solr/solrconfig.xml.joan 2018-12-08 14:31:47.716344505 +0000 +++ solrconfig.xml 2018-12-08 15:36:28.948267225 +0000 @@ -1128,6 +1128,7 @@ See http://wiki.apache.org/solr/GuessingFieldTypes --> + <schemaFactory class="ClassicIndexSchemaFactory"></schemaFactory> <updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/> <updateProcessor class="solr.RemoveBlankFieldUpdateProcessorFactory" name="remove-blank"/> <updateProcessor class="solr.FieldNameMutatingUpdateProcessorFactory" name="field-name-mutating"> @@ -1158,43 +1159,8 @@ <str>yyyy-MM-dd</str> </arr> </updateProcessor> - <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields"> - <lst name="typeMapping"> - <str name="valueClass">java.lang.String</str> - <str name="fieldType">text_general</str> - <lst name="copyField"> - <str name="dest">*_str</str> - <int name="maxChars">256</int> - </lst> - <!-- Use as default mapping instead of defaultFieldType --> - <bool name="default">true</bool> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Boolean</str> - <str name="fieldType">booleans</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.util.Date</str> - <str name="fieldType">pdates</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Long</str> - <str name="valueClass">java.lang.Integer</str> - <str name="fieldType">plongs</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Number</str> - <str name="fieldType">pdoubles</str> - </lst> - </updateProcessor> <!-- The update.autoCreateFields property can be turned to false to disable schemaless mode --> - <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:true}" - processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields"> - <processor class="solr.LogUpdateProcessorFactory"/> - <processor class="solr.DistributedUpdateProcessorFactory"/> - <processor class="solr.RunUpdateProcessorFactory"/> - </updateRequestProcessorChain> <!-- Deduplication @@ -1273,7 +1239,6 @@ <!-- The following response writers are implicitly configured unless overridden... --> - <!-- <queryResponseWriter name="xml" default="true" class="solr.XMLResponseWriter" /> @@ -1284,7 +1249,6 @@ <queryResponseWriter name="phps" class="solr.PHPSerializedResponseWriter"/> <queryResponseWriter name="csv" class="solr.CSVResponseWriter"/> <queryResponseWriter name="schema.xml" class="solr.SchemaXmlResponseWriter"/> - --> <queryResponseWriter name="json" class="solr.JSONResponseWriter"> <!-- For the purposes of the tutorial, JSON responses are written as @@ -1293,7 +1257,7 @@ --> <str name="content-type">text/plain; charset=UTF-8</str> </queryResponseWriter> - + <!-- Custom response writers can be declared as needed... -->
After some testsing, I managed to get proper functionning - The schema.xml is attached below (quite different from the one provided on teh wiki) (in bold the core differences) (NGramFilterFactory is the class that replace the fts_squat "partial=3 full=15", everything else is just a big hammer to smash a tiny fly) - One need to remove the "managed-schema" file in the {prefix}/server/solr/dovecot/conf. - One need to remove everything under {prefix}/server/solr/dovecot/data/ - The {prefix}/server/solr/dovecot/conf/solrconfig.xml is as below (see diff) - Restart Solr - my dovecot.conf is : fts = solr fts_autoindex = yes fts_enforced = yes fts_decoder = decode2text fts_solr = url=http://(SOLR SERVER):8983/solr/dovecot/ --- schema.xml <?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <FIELDTYPE NAME="STRING" CLASS="SOLR.STRFIELD" /> <FIELDTYPE NAME="LONG" CLASS="SOLR.TRIELONGFIELD" /> <FIELDTYPE NAME="BOOLEAN" CLASS="SOLR.BOOLFIELD" /> <fieldType name="text" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <FILTER CLASS="SOLR.NGRAMFILTERFACTORY" MINGRAMSIZE="3" MAXGRAMSIZE="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <FILTER CLASS="SOLR.NGRAMFILTERFACTORY" MINGRAMSIZE="3" MAXGRAMSIZE="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> <field name="_version_" type="long" indexed="true" stored="true"/> <field name="bcc" type="text" indexed="true" stored="false"/> <field name="body" type="text" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="text" indexed="true" stored="false"/> <field name="from" type="text" indexed="true" stored="false"/> <field name="hdr" type="text" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="text" indexed="true" stored="false"/> <field name="to" type="text" indexed="true" stored="false"/> <field name="uid" type="long" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema> ------ diff solrconfig --- /data/backup/solr/solrconfig.xml.joan 2018-12-08 14:31:47.716344505 +0000 +++ solrconfig.xml 2018-12-08 15:36:28.948267225 +0000 @@ -1128,6 +1128,7 @@ See http://wiki.apache.org/solr/GuessingFieldTypes --> + <schemaFactory class="ClassicIndexSchemaFactory"></schemaFactory> <updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/> <updateProcessor class="solr.RemoveBlankFieldUpdateProcessorFactory" name="remove-blank"/> <updateProcessor class="solr.FieldNameMutatingUpdateProcessorFactory" name="field-name-mutating"> @@ -1158,43 +1159,8 @@ <str>yyyy-MM-dd</str> </arr> </updateProcessor> - <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields"> - <lst name="typeMapping"> - <str name="valueClass">java.lang.String</str> - <str name="fieldType">text_general</str> - <lst name="copyField"> - <str name="dest">*_str</str> - <int name="maxChars">256</int> - </lst> - <!-- Use as default mapping instead of defaultFieldType --> - <bool name="default">true</bool> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Boolean</str> - <str name="fieldType">booleans</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.util.Date</str> - <str name="fieldType">pdates</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Long</str> - <str name="valueClass">java.lang.Integer</str> - <str name="fieldType">plongs</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Number</str> - <str name="fieldType">pdoubles</str> - </lst> - </updateProcessor> <!-- The update.autoCreateFields property can be turned to false to disable schemaless mode --> - <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:true}" - processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields"> - <processor class="solr.LogUpdateProcessorFactory"/> - <processor class="solr.DistributedUpdateProcessorFactory"/> - <processor class="solr.RunUpdateProcessorFactory"/> - </updateRequestProcessorChain> <!-- Deduplication @@ -1273,7 +1239,6 @@ <!-- The following response writers are implicitly configured unless overridden... --> - <!-- <queryResponseWriter name="xml" default="true" class="solr.XMLResponseWriter" /> @@ -1284,7 +1249,6 @@ <queryResponseWriter name="phps" class="solr.PHPSerializedResponseWriter"/> <queryResponseWriter name="csv" class="solr.CSVResponseWriter"/> <queryResponseWriter name="schema.xml" class="solr.SchemaXmlResponseWriter"/> - --> <queryResponseWriter name="json" class="solr.JSONResponseWriter"> <!-- For the purposes of the tutorial, JSON responses are written as @@ -1293,7 +1257,7 @@ --> <str name="content-type">text/plain; charset=UTF-8</str> </queryResponseWriter> - + <!-- Custom response writers can be declared as needed... -->
However, Solr crashes and Dovecot plugin ftS_solr returns multitude of errors Dec 09 09:21:09 imap(jom@grosjo.net)<3349><DiRnXpN8Lux/AAAB>: Error: fts_solr: received invalid uid '0' Dec 09 09:21:10 imap(jom@grosjo.net)<3349><DiRnXpN8Lux/AAAB>: Error: fts_solr: received invalid uid '0' And returns are totaly funny (keywords not presentin teh results) I am back to fs_squat.... On 2018-12-08 18:28, Joan Moreau via dovecot wrote:
After some testsing, I managed to get proper functionning
- The schema.xml is attached below (quite different from the one provided on teh wiki) (in bold the core differences) (NGramFilterFactory is the class that replace the fts_squat "partial=3 full=15", everything else is just a big hammer to smash a tiny fly)
- One need to remove the "managed-schema" file in the {prefix}/server/solr/dovecot/conf.
- One need to remove everything under {prefix}/server/solr/dovecot/data/
- The {prefix}/server/solr/dovecot/conf/solrconfig.xml is as below (see diff)
- Restart Solr
- my dovecot.conf is :
fts = solr fts_autoindex = yes fts_enforced = yes fts_decoder = decode2text
fts_solr = url=http://(SOLR SERVER):8983/solr/dovecot/
--- schema.xml
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey>
<FIELDTYPE NAME="STRING" CLASS="SOLR.STRFIELD" /> <FIELDTYPE NAME="LONG" CLASS="SOLR.TRIELONGFIELD" /> <FIELDTYPE NAME="BOOLEAN" CLASS="SOLR.BOOLFIELD" /> <fieldType name="text" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <FILTER CLASS="SOLR.NGRAMFILTERFACTORY" MINGRAMSIZE="3" MAXGRAMSIZE="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <FILTER CLASS="SOLR.NGRAMFILTERFACTORY" MINGRAMSIZE="3" MAXGRAMSIZE="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType>
<field name="_version_" type="long" indexed="true" stored="true"/> <field name="bcc" type="text" indexed="true" stored="false"/> <field name="body" type="text" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="text" indexed="true" stored="false"/> <field name="from" type="text" indexed="true" stored="false"/> <field name="hdr" type="text" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="text" indexed="true" stored="false"/> <field name="to" type="text" indexed="true" stored="false"/> <field name="uid" type="long" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
------ diff solrconfig
--- /data/backup/solr/solrconfig.xml.joan 2018-12-08 14:31:47.716344505 +0000 +++ solrconfig.xml 2018-12-08 15:36:28.948267225 +0000 @@ -1128,6 +1128,7 @@
See http://wiki.apache.org/solr/GuessingFieldTypes --> + <schemaFactory class="ClassicIndexSchemaFactory"></schemaFactory> <updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/> <updateProcessor class="solr.RemoveBlankFieldUpdateProcessorFactory" name="remove-blank"/> <updateProcessor class="solr.FieldNameMutatingUpdateProcessorFactory" name="field-name-mutating"> @@ -1158,43 +1159,8 @@ <str>yyyy-MM-dd</str> </arr> </updateProcessor> - <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields"> - <lst name="typeMapping"> - <str name="valueClass">java.lang.String</str> - <str name="fieldType">text_general</str> - <lst name="copyField"> - <str name="dest">*_str</str> - <int name="maxChars">256</int> - </lst> - <!-- Use as default mapping instead of defaultFieldType --> - <bool name="default">true</bool> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Boolean</str> - <str name="fieldType">booleans</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.util.Date</str> - <str name="fieldType">pdates</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Long</str> - <str name="valueClass">java.lang.Integer</str> - <str name="fieldType">plongs</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Number</str> - <str name="fieldType">pdoubles</str> - </lst> - </updateProcessor>
<!-- The update.autoCreateFields property can be turned to false to disable schemaless mode --> - <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:true}" - processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields"> - <processor class="solr.LogUpdateProcessorFactory"/> - <processor class="solr.DistributedUpdateProcessorFactory"/> - <processor class="solr.RunUpdateProcessorFactory"/> - </updateRequestProcessorChain>
<!-- Deduplication
@@ -1273,7 +1239,6 @@ <!-- The following response writers are implicitly configured unless overridden... --> - <!-- <queryResponseWriter name="xml" default="true" class="solr.XMLResponseWriter" /> @@ -1284,7 +1249,6 @@ <queryResponseWriter name="phps" class="solr.PHPSerializedResponseWriter"/> <queryResponseWriter name="csv" class="solr.CSVResponseWriter"/> <queryResponseWriter name="schema.xml" class="solr.SchemaXmlResponseWriter"/> - -->
<queryResponseWriter name="json" class="solr.JSONResponseWriter"> <!-- For the purposes of the tutorial, JSON responses are written as @@ -1293,7 +1257,7 @@ --> <str name="content-type">text/plain; charset=UTF-8</str> </queryResponseWriter> - + <!-- Custom response writers can be declared as needed... -->
Hi! Thank you for your report, we'll look thru it. Aki On 9.12.2018 11.24, Joan Moreau via dovecot wrote:
However, Solr crashes and Dovecot plugin ftS_solr returns multitude of errors
Dec 09 09:21:09 imap(jom@grosjo.net)<3349><DiRnXpN8Lux/AAAB>: Error: fts_solr: received invalid uid '0' Dec 09 09:21:10 imap(jom@grosjo.net)<3349><DiRnXpN8Lux/AAAB>: Error: fts_solr: received invalid uid '0'
And returns are totaly funny (keywords not presentin teh results)
I am back to fs_squat....
On 2018-12-08 18:28, Joan Moreau via dovecot wrote:
After some testsing, I managed to get proper functionning
- The schema.xml is attached below (quite different from the one provided on teh wiki) (in bold the core differences) (NGramFilterFactory is the class that replace the fts_squat "partial=3 full=15", everything else is just a big hammer to smash a tiny fly)
- One need to remove the "managed-schema" file in the {prefix}/server/solr/dovecot/conf.
- One need to remove everything under {prefix}/server/solr/dovecot/data/
- The {prefix}/server/solr/dovecot/conf/solrconfig.xml is as below (see diff) - Restart Solr - my dovecot.conf is :
fts = solr fts_autoindex = yes fts_enforced = yes fts_decoder = decode2text
fts_solr = url=http://(SOLR SERVER):8983/solr/dovecot/
--- schema.xml
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey>
*<fieldType name="string" class="solr.StrField" />* *<fieldType name="long" class="solr.TrieLongField" />* *<fieldType name="boolean" class="solr.BoolField" />* <fieldType name="text" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> *<filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" />* <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> *<filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" />* <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType>
<field name="_version_" type="long" indexed="true" stored="true"/> <field name="bcc" type="text" indexed="true" stored="false"/> <field name="body" type="text" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="text" indexed="true" stored="false"/> <field name="from" type="text" indexed="true" stored="false"/> <field name="hdr" type="text" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="text" indexed="true" stored="false"/> <field name="to" type="text" indexed="true" stored="false"/> <field name="uid" type="long" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
------ diff solrconfig
--- /data/backup/solr/solrconfig.xml.joan 2018-12-08 14:31:47.716344505 +0000 +++ solrconfig.xml 2018-12-08 15:36:28.948267225 +0000 @@ -1128,6 +1128,7 @@
See http://wiki.apache.org/solr/GuessingFieldTypes --> + <schemaFactory class="ClassicIndexSchemaFactory"></schemaFactory> <updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/> <updateProcessor class="solr.RemoveBlankFieldUpdateProcessorFactory" name="remove-blank"/> <updateProcessor class="solr.FieldNameMutatingUpdateProcessorFactory" name="field-name-mutating"> @@ -1158,43 +1159,8 @@ <str>yyyy-MM-dd</str> </arr> </updateProcessor> - <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields"> - <lst name="typeMapping"> - <str name="valueClass">java.lang.String</str> - <str name="fieldType">text_general</str> - <lst name="copyField"> - <str name="dest">*_str</str> - <int name="maxChars">256</int> - </lst> - <!-- Use as default mapping instead of defaultFieldType --> - <bool name="default">true</bool> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Boolean</str> - <str name="fieldType">booleans</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.util.Date</str> - <str name="fieldType">pdates</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Long</str> - <str name="valueClass">java.lang.Integer</str> - <str name="fieldType">plongs</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Number</str> - <str name="fieldType">pdoubles</str> - </lst> - </updateProcessor>
<!-- The update.autoCreateFields property can be turned to false to disable schemaless mode --> - <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:true}" - processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields"> - <processor class="solr.LogUpdateProcessorFactory"/> - <processor class="solr.DistributedUpdateProcessorFactory"/> - <processor class="solr.RunUpdateProcessorFactory"/> - </updateRequestProcessorChain>
<!-- Deduplication
@@ -1273,7 +1239,6 @@ <!-- The following response writers are implicitly configured unless overridden... --> - <!-- <queryResponseWriter name="xml" default="true" class="solr.XMLResponseWriter" /> @@ -1284,7 +1249,6 @@ <queryResponseWriter name="phps" class="solr.PHPSerializedResponseWriter"/> <queryResponseWriter name="csv" class="solr.CSVResponseWriter"/> <queryResponseWriter name="schema.xml" class="solr.SchemaXmlResponseWriter"/> - -->
<queryResponseWriter name="json" class="solr.JSONResponseWriter"> <!-- For the purposes of the tutorial, JSON responses are written as @@ -1293,7 +1257,7 @@ --> <str name="content-type">text/plain; charset=UTF-8</str> </queryResponseWriter> - + <!-- Custom response writers can be declared as needed... -->
On 12/4/2018 10:40 AM, Joan Moreau via dovecot wrote:
In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command :
sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance
before updating the schema.xml .
Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users
Additionaly, the url is http://(solr_ <http://(solr>server):8983/solr/dovecot/ (error in wiki)
After installing Solr, wherever the installation sets up there should a folder similar to:
<your prefix>/solr/server/solr/configsets
If you look there, you'll probably see folders like '_default' and 'sample_techproducts_configs'. I haven't played with the 'techproducts' sample. Copy the '_default' folder, with all its contents, to a 'dovecot' folder. In the new dovecot folder, replace the 'managed-schema' file with the file from the Dovecot Wiki
https://wiki.dovecot.org/Plugins/FTS/Solr?action=AttachFile&do=view&target=solr-7.x-schema.xml
after that, you should be able to run 'solr /opt/solr/bin/solr create -c dovecot' to create the instance. If things still don't work let us know.
The schema is one I've tweaked and updated during my own migrations since Solr 3.3. It's possible there's something else in my config that needs documenting - but having experienced Solr search against my mailstore I never want to be without it.
Daniel
Hi Daniel,
THere is no need of all this, just the command (on Solr 7.5) "create -c dovecot " is enough
The chema.xml provided on the wiki basically does not work on 7.5
Here the latest one I am working on , but nothing works properly (bad search results, errors in ftp_solr, etc..)
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="long" class="solr.LongPointField" positionIncrementGap="0" /> <fieldType name="text" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="_version_" type="long" indexed="true" stored="true"/> <field name="bcc" type="text" indexed="true" stored="false"/> <field name="body" type="text" indexed="true" stored="false"/> <field name="box" type="string" indexed="false" required="true" stored="true"/> <field name="hdr" type="text" indexed="true" stored="false"/> <field name="cc" type="text" indexed="true" stored="false"/> <field name="from" type="text" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="text" indexed="true" stored="false"/> <field name="to" type="text" indexed="true" stored="false"/> <field name="uid" type="long" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </fields> </schema>
On 2018-12-10 21:17, Daniel Miller via dovecot wrote:
On 12/4/2018 10:40 AM, Joan Moreau via dovecot wrote:
In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command :
sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance
before updating the schema.xml .
Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users
Additionaly, the url is http://(solr_ [1]server):8983/solr/dovecot/ (error in wiki)
After installing Solr, wherever the installation sets up there should a folder similar to:
<your prefix>/solr/server/solr/configsets
If you look there, you'll probably see folders like '_default' and 'sample_techproducts_configs'. I haven't played with the 'techproducts' sample. Copy the '_default' folder, with all its contents, to a 'dovecot' folder. In the new dovecot folder, replace the 'managed-schema' file with the file from the Dovecot Wiki
https://wiki.dovecot.org/Plugins/FTS/Solr?action=AttachFile&do=view&target=solr-7.x-schema.xml
after that, you should be able to run 'solr /opt/solr/bin/solr create -c dovecot' to create the instance. If things still don't work let us know.
The schema is one I've tweaked and updated during my own migrations since Solr 3.3. It's possible there's something else in my config that needs documenting - but having experienced Solr search against my mailstore I never want to be without it.
Daniel
Links:
[1] http://(solr
The one on the Wiki is mine...which I'm using now. So it certainly does work - but perhaps there's a setting you have differently from me.
Performing a "create -c dovecot" creates a Solr instance *named* dovecot
- that does *not* initialize it with the necessary schema. You need to specify "-d dovecot", with a dovecot configset already setup, to do that.
The other choice is to create the instance as you show, ensure Solr is stopped, delete the "<prefix>/solr/dovecot/data" folder, and copy the managed-schema file to "<prefix>/solr/dovecot/conf". Again, the filename saved in the /conf folder needs to be "managed-schema" - no ".xml" suffix.
If that doesn't work for you - please share the errors.
Daniel
On 12/10/2018 11:40 AM, Joan Moreau wrote:
Hi Daniel,
THere is no need of all this, just the command (on Solr 7.5) "create -c dovecot " is enough
The chema.xml provided on the wiki basically does not work on 7.5
Here the latest one I am working on , but nothing works properly (bad search results, errors in ftp_solr, etc..)
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="long" class="solr.LongPointField" positionIncrementGap="0" /> <fieldType name="text" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="_version_" type="long" indexed="true" stored="true"/> <field name="bcc" type="text" indexed="true" stored="false"/> <field name="body" type="text" indexed="true" stored="false"/> <field name="box" type="string" indexed="false" required="true" stored="true"/> <field name="hdr" type="text" indexed="true" stored="false"/> <field name="cc" type="text" indexed="true" stored="false"/> <field name="from" type="text" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="text" indexed="true" stored="false"/> <field name="to" type="text" indexed="true" stored="false"/> <field name="uid" type="long" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </fields> </schema>
On 2018-12-10 21:17, Daniel Miller via dovecot wrote:
On 12/4/2018 10:40 AM, Joan Moreau via dovecot wrote:
In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command :
sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance
before updating the schema.xml .
Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users
Additionaly, the url is http://(solr_ <http://(solr>server):8983/solr/dovecot/ (error in wiki)
After installing Solr, wherever the installation sets up there should a folder similar to:
<your prefix>/solr/server/solr/configsets
If you look there, you'll probably see folders like '_default' and 'sample_techproducts_configs'. I haven't played with the 'techproducts' sample. Copy the '_default' folder, with all its contents, to a 'dovecot' folder. In the new dovecot folder, replace the 'managed-schema' file with the file from the Dovecot Wiki
https://wiki.dovecot.org/Plugins/FTS/Solr?action=AttachFile&do=view&target=solr-7.x-schema.xml
after that, you should be able to run 'solr /opt/solr/bin/solr create -c dovecot' to create the instance. If things still don't work let us know.
The schema is one I've tweaked and updated during my own migrations since Solr 3.3. It's possible there's something else in my config that needs documenting - but having experienced Solr search against my mailstore I never want to be without it.
Daniel
I shared the errors already so many times (check this mailinling for "solr" in teh title)
Contrary to what you say, with SOlr 7.5 and Dovecot git, I had to remove the "managed-schema" to make solr respond a bit properly. It relies on schema.xml
In order to create the instance, no, it copies the default config in the dovecot instance.
Bottom line, the end results is that the results are either "error : invalid response" (mostly on Android email client) or no results or innacurate results (using Roundcube)
On 2018-12-10 23:30, Daniel Miller via dovecot wrote:
The one on the Wiki is mine...which I'm using now. So it certainly does work - but perhaps there's a setting you have differently from me.
Performing a "create -c dovecot" creates a Solr instance *named* dovecot - that does *not* initialize it with the necessary schema. You need to specify "-d dovecot", with a dovecot configset already setup, to do that.
The other choice is to create the instance as you show, ensure Solr is stopped, delete the "<prefix>/solr/dovecot/data" folder, and copy the managed-schema file to "<prefix>/solr/dovecot/conf". Again, the filename saved in the /conf folder needs to be "managed-schema" - no ".xml" suffix.
If that doesn't work for you - please share the errors.
Daniel
On 12/10/2018 11:40 AM, Joan Moreau wrote:
Hi Daniel,
THere is no need of all this, just the command (on Solr 7.5) "create -c dovecot " is enough
The chema.xml provided on the wiki basically does not work on 7.5
Here the latest one I am working on , but nothing works properly (bad search results, errors in ftp_solr, etc..)
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="long" class="solr.LongPointField" positionIncrementGap="0" /> <fieldType name="text" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="_version_" type="long" indexed="true" stored="true"/> <field name="bcc" type="text" indexed="true" stored="false"/> <field name="body" type="text" indexed="true" stored="false"/> <field name="box" type="string" indexed="false" required="true" stored="true"/> <field name="hdr" type="text" indexed="true" stored="false"/> <field name="cc" type="text" indexed="true" stored="false"/> <field name="from" type="text" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="text" indexed="true" stored="false"/> <field name="to" type="text" indexed="true" stored="false"/> <field name="uid" type="long" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </fields> </schema>
On 2018-12-10 21:17, Daniel Miller via dovecot wrote: On 12/4/2018 10:40 AM, Joan Moreau via dovecot wrote:
In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command :
sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance
before updating the schema.xml .
Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users
Additionaly, the url is http://(solr_ [1]server):8983/solr/dovecot/ (error in wiki)
After installing Solr, wherever the installation sets up there should a folder similar to:
<your prefix>/solr/server/solr/configsets
If you look there, you'll probably see folders like '_default' and 'sample_techproducts_configs'. I haven't played with the 'techproducts' sample. Copy the '_default' folder, with all its contents, to a 'dovecot' folder. In the new dovecot folder, replace the 'managed-schema' file with the file from the Dovecot Wiki
https://wiki.dovecot.org/Plugins/FTS/Solr?action=AttachFile&do=view&target=solr-7.x-schema.xml
after that, you should be able to run 'solr /opt/solr/bin/solr create -c dovecot' to create the instance. If things still don't work let us know.
The schema is one I've tweaked and updated during my own migrations since Solr 3.3. It's possible there's something else in my config that needs documenting - but having experienced Solr search against my mailstore I never want to be without it.
Daniel
Links:
[1] http://(solr
On 12/11/2018 4:46 AM, Joan Moreau via dovecot wrote:
I shared the errors already so many times (check this mailinling for "solr" in teh title)
Contrary to what you say, with SOlr 7.5 and Dovecot git, I had to remove the "managed-schema" to make solr respond a bit properly. It relies on schema.xml
In order to create the instance, no, it copies the default config in the dovecot instance.
I'm not a Solr expert by any means but I believe you are incorrect.
As of Solr 5.x the managed-schema file is the primary method for configuration. The method I detailed previously for setting up a config helps automate creating new Solr instances - but as I stated you can either setup a Solr template and then create the instance from that or create an instance using the default template and then adjust it.
The part that you *must* do after creating from the default template is stop the server, delete the entire "<prefix>/solr/dovecot/data" folder, then install the correct managed-schema file, then restart the server. The server will not function with mismatched schema/data.
If you'll try that - explicitly "rm -rf <prefix>/solr/dovecot/data", copy the managed-schema file into the conf folder, and restart - things will either work or there's something else that needs correction.
-- Daniel
Daniel,
I have done that so any times (deleteing the data folders, recreating the instance, restarting etc...)
But this is really not the issue
The issue is 1 - fts_solr reports errors in the log file (this is a pure dovecot issue) : how to have much more details on what fts_solr sends to Slor server and what does it returns ? 2 - Solr returns properly for a few hours, then starts crashing or responding non-sense after some time
Additionally, is there a doc of fts-squat in order to adjust the code to new releases of dovect ?
On December 12, 2018 4:44:10 PM Daniel Miller via dovecot <dovecot@dovecot.org> wrote:
On 12/11/2018 4:46 AM, Joan Moreau via dovecot wrote:
I shared the errors already so many times (check this mailinling for "solr" in teh title)
Contrary to what you say, with SOlr 7.5 and Dovecot git, I had to remove the "managed-schema" to make solr respond a bit properly. It relies on schema.xml
In order to create the instance, no, it copies the default config in the dovecot instance. I'm not a Solr expert by any means but I believe you are incorrect. As of Solr 5.x the managed-schema file is the primary method for configuration. The method I detailed previously for setting up a config helps automate creating new Solr instances - but as I stated you can either setup a Solr template and then create the instance from that or create an instance using the default template and then adjust it. The part that you *must* do after creating from the default template is stop the server, delete the entire "<prefix>/solr/dovecot/data" folder, then install the correct managed-schema file, then restart the server. The server will not function with mismatched schema/data. If you'll try that - explicitly "rm -rf <prefix>/solr/dovecot/data", copy the managed-schema file into the conf folder, and restart - things will either work or there's something else that needs correction.
Daniel
here my latest schema.xml (remove the "long" type hich seems to be very deprecated in 7.x)
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0" /> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="hdr" type="gjtext" indexed="false" stored="false"/> <field name="cc" type="gjtext" indexed="true" stored="false"/> <field name="from" type="gjtext" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjtext" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </fields> </schema>
On 2018-12-15 20:54, Joan Moreau wrote:
Daniel,
I have done that so any times (deleteing the data folders, recreating the instance, restarting etc...)
But this is really not the issue
The issue is 1 - fts_solr reports errors in the log file (this is a pure dovecot issue) : how to have much more details on what fts_solr sends to Slor server and what does it returns ? 2 - Solr returns properly for a few hours, then starts crashing or responding non-sense after some time
Additionally, is there a doc of fts-squat in order to adjust the code to new releases of dovect ?
On December 12, 2018 4:44:10 PM Daniel Miller via dovecot <dovecot@dovecot.org> wrote: On 12/11/2018 4:46 AM, Joan Moreau via dovecot wrote:
I shared the errors already so many times (check this mailinling for "solr" in teh title)
Contrary to what you say, with SOlr 7.5 and Dovecot git, I had to remove the "managed-schema" to make solr respond a bit properly. It relies on schema.xml
In order to create the instance, no, it copies the default config in the dovecot instance.
I'm not a Solr expert by any means but I believe you are incorrect.
As of Solr 5.x the managed-schema file is the primary method for configuration. The method I detailed previously for setting up a config helps automate creating new Solr instances - but as I stated you can either setup a Solr template and then create the instance from that or create an instance using the default template and then adjust it.
The part that you *must* do after creating from the default template is stop the server, delete the entire "<prefix>/solr/dovecot/data" folder, then install the correct managed-schema file, then restart the server. The server will not function with mismatched schema/data.
If you'll try that - explicitly "rm -rf <prefix>/solr/dovecot/data", copy the managed-schema file into the conf folder, and restart - things will either work or there's something else that needs correction. -- Daniel
Joan,
I understand and sympathize with your frustration - trying to get multiple applications to work together, particularly given the lack of documentation for some of them, can be extremely challenging. That said, I suggest you consider an alternative viewpoint. Frequently being misunderstood myself I apologize in advance if I'm reading you wrong - but it appears your view towards the situation is there is a bug in Dovecot related to this problem. That may well be - but I generally approach these matters from the assumption that *I* made the error in configuration and go from there. I'm not an official rep for any product nor claim to be any form of expert in these matters - but I do have a working setup and I'd like to help you if I can. If you're willing to - take a deep breath and let's try starting over.
Looking back through your emails there were two items that stood out - your Dovecot config has two settings I don't use: "fts_decoder" and "fts_enforced". I also asked you earlier whether or not NFS is involved here and I didn't see an answer - please clarify.
I suggest you try once more: delete Solr completely. Re-install per the directions and use *my* managed-schema. Also comment out the Dovecot directives for "fts_decoder" and "fts_enforced" so you're closer to my setup. Try running again and then post back - I'll do what I can. Based on the fact that Dovecot+Solr 7.5+my schema is working for me leads me to believe we can get it working for you as well.
Daniel
On 12/15/2018 2:42 PM, Joan Moreau wrote:
here my latest schema.xml (remove the "long" type hich seems to be very deprecated in 7.x)
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0" /> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="hdr" type="gjtext" indexed="false" stored="false"/> <field name="cc" type="gjtext" indexed="true" stored="false"/> <field name="from" type="gjtext" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjtext" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </fields> </schema>
On 2018-12-15 20:54, Joan Moreau wrote:
Daniel, I have done that so any times (deleteing the data folders, recreating the instance, restarting etc...) But this is really not the issue The issue is 1 - fts_solr reports errors in the log file (this is a pure dovecot issue) : how to have much more details on what fts_solr sends to Slor server and what does it returns ? 2 - Solr returns properly for a few hours, then starts crashing or responding non-sense after some time Additionally, is there a doc of fts-squat in order to adjust the code to new releases of dovect ?
On December 12, 2018 4:44:10 PM Daniel Miller via dovecot <dovecot@dovecot.org> wrote:
On 12/11/2018 4:46 AM, Joan Moreau via dovecot wrote:I shared the errors already so many times (check this mailinling for "solr" in teh title) Contrary to what you say, with SOlr 7.5 and Dovecot git, I had to remove the "managed-schema" to make solr respond a bit properly. It relies on schema.xml In order to create the instance, no, it copies the default config in the dovecot instance.I'm not a Solr expert by any means but I believe you are incorrect. As of Solr 5.x the managed-schema file is the primary method for configuration. The method I detailed previously for setting up a config helps automate creating new Solr instances - but as I stated you can either setup a Solr template and then create the instance from that or create an instance using the default template and then adjust it. The part that you *must* do after creating from the default template is stop the server, delete the entire "<prefix>/solr/dovecot/data" folder, then install the correct managed-schema file, then restart the server. The server will not function with mismatched schema/data. If you'll try that - explicitly "rm -rf <prefix>/solr/dovecot/data", copy the managed-schema file into the conf folder, and restart - things will either work or there's something else that needs correction. -- Daniel
Dear Daniel.
Thank you for your kind reply.
Regarding NFS, no, there is nothing like this in my setup.
Deleteing SOLR and recreating it, I did it so many times already.
I started with *your* setup in the first place, as FTS_squat (which actually works very well and very straightforward, I have no clue why going for SOlr which is just a pain and not maintaining squat), and it leads to totally funny results (for instance, I type "emirates" in my "Air Companies" subfolder and get a lot of results .. but of competing companies :D )
I added the fts_enforce following AKi advice.
I removed fts_decoder for the time being.
I don't know where to go now. Dovcot still returning errors and SOlr still companinig with "Out of range" and other Java errors.
Bottom line, I am back to squat, but as it is not maintained so crashed also times to times.
I think we should discuss on
(1) Why the damn choice of Solr has been main. As you empahised, maintainend so many independent software is a pain
(2) If there is a real reason why going for SOlr, how to have a working (i.e. getting the right results to the end user) setup ?
(3) If there iare no tangible reason, what about maintaining fts_squat , which did the job nicely for years and no complains about.
On 2018-12-16 08:51, Daniel Miller via dovecot wrote:
Joan,
I understand and sympathize with your frustration - trying to get multiple applications to work together, particularly given the lack of documentation for some of them, can be extremely challenging. That said, I suggest you consider an alternative viewpoint. Frequently being misunderstood myself I apologize in advance if I'm reading you wrong - but it appears your view towards the situation is there is a bug in Dovecot related to this problem. That may well be - but I generally approach these matters from the assumption that *I* made the error in configuration and go from there. I'm not an official rep for any product nor claim to be any form of expert in these matters - but I do have a working setup and I'd like to help you if I can. If you're willing to - take a deep breath and let's try starting over.
Looking back through your emails there were two items that stood out - your Dovecot config has two settings I don't use: "fts_decoder" and "fts_enforced". I also asked you earlier whether or not NFS is involved here and I didn't see an answer - please clarify.
I suggest you try once more: delete Solr completely. Re-install per the directions and use *my* managed-schema. Also comment out the Dovecot directives for "fts_decoder" and "fts_enforced" so you're closer to my setup. Try running again and then post back - I'll do what I can. Based on the fact that Dovecot+Solr 7.5+my schema is working for me leads me to believe we can get it working for you as well.
Daniel
On 12/15/2018 2:42 PM, Joan Moreau wrote:
here my latest schema.xml (remove the "long" type hich seems to be very deprecated in 7.x)
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0" /> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="hdr" type="gjtext" indexed="false" stored="false"/> <field name="cc" type="gjtext" indexed="true" stored="false"/> <field name="from" type="gjtext" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjtext" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </fields> </schema>
On 2018-12-15 20:54, Joan Moreau wrote:
Daniel,
I have done that so any times (deleteing the data folders, recreating the instance, restarting etc...)
But this is really not the issue
The issue is 1 - fts_solr reports errors in the log file (this is a pure dovecot issue) : how to have much more details on what fts_solr sends to Slor server and what does it returns ? 2 - Solr returns properly for a few hours, then starts crashing or responding non-sense after some time
Additionally, is there a doc of fts-squat in order to adjust the code to new releases of dovect ?
On December 12, 2018 4:44:10 PM Daniel Miller via dovecot <dovecot@dovecot.org> wrote: On 12/11/2018 4:46 AM, Joan Moreau via dovecot wrote:
I shared the errors already so many times (check this mailinling for "solr" in teh title)
Contrary to what you say, with SOlr 7.5 and Dovecot git, I had to remove the "managed-schema" to make solr respond a bit properly. It relies on schema.xml
In order to create the instance, no, it copies the default config in the dovecot instance.
I'm not a Solr expert by any means but I believe you are incorrect.
As of Solr 5.x the managed-schema file is the primary method for configuration. The method I detailed previously for setting up a config helps automate creating new Solr instances - but as I stated you can either setup a Solr template and then create the instance from that or create an instance using the default template and then adjust it.
The part that you *must* do after creating from the default template is stop the server, delete the entire "<prefix>/solr/dovecot/data" folder, then install the correct managed-schema file, then restart the server. The server will not function with mismatched schema/data.
If you'll try that - explicitly "rm -rf <prefix>/solr/dovecot/data", copy the managed-schema file into the conf folder, and restart - things will either work or there's something else that needs correction. -- Daniel
Joan,
The reason for dropping squat, I'm assuming, is that Lucene and Solr potentially provide superior features & performance and as they are 3rd-party libraries & apps it reduces the maintenance responsibilities and let's the Dovecot team focus on mail server specific stuff - and let others focus on FTS. There is a *huge* difference between a functional Solr setup & squat - and if I'm able to get it working we should be able to get you there as well.
I don't recall what OS you're running - I'm on Ubuntu 18.04. My Java version is OpenJDK 10.0.2. Attached is my complete Solr config. Try one more time - stop the server, delete the data folder, unpack the attached into the conf folder - and restart. I also have
/etc/default/solr.in.sh: SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=3000" SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=60000" SOLR_PID_DIR=/run/solr SOLR_HOME=/usr/local/lib
Adjust the above folders as appropriate - or don't use them at all if you're using the defaults.
/etc/systemd/system/solr.service: # put this file in /etc/systemd/system/ as root # below paths assume solr installed in /opt/solr, SOLR_PID_DIR is /data # and that all configuration exists in /etc/default/solr.in.sh which is the case if previously installed as an init.d service # change port in pid file if differs # note that it is configured to auto restart solr if it fails (Restart=on-faliure) and that's the motivation indeed :) # to switch from systemv (init.d) to systemd, do the following after creating this file: # sudo systemctl daemon-reload # sudo service solr stop # if already running # sudo systemctl enable solr # systemctl start solr # this was inspired by https://confluence.t5.fi/display/~stefan.roos/2015/04/01/Creating+systemd+un... [Unit] Description=Apache SOLR 7.5.0 After=syslog.target network.target remote-fs.target nss-lookup.target systemd-journald-dev-log.socket Before=multi-user.target graphical.target nginx.service dovecot.service Conflicts=shutdown.target [Service] LimitNOFILE=65000 User=vmail Group=mail ExecStartPre=/bin/mkdir -p /run/solr ExecStartPre=/bin/chown -R vmail.mail /run/solr PermissionsStartOnly=true PIDFile=/run/solr/solr-8983.pid Environment=SOLR_INCLUDE=/etc/default/solr.in.sh ExecStart=/opt/solr/bin/solr start ExecStop=/opt/solr/bin/solr stop Restart=on-failure RestartSec=15s TimeoutStopSec=30s [Install] WantedBy=multi-user.target graphical.target dovecot.service
If you don't use systemd disregard - but see if any of the above applies for your setup.
Let me know what happens. I agree this can be a mortal pain to setup - but it's worth it.
Daniel
On 12/21/2018 4:33 AM, Joan Moreau wrote:
Dear Daniel.
Thank you for your kind reply.
Regarding NFS, no, there is nothing like this in my setup.
Deleteing SOLR and recreating it, I did it so many times already.
I started with *your* setup in the first place, as FTS_squat (which actually works very well and very straightforward, I have no clue why going for SOlr which is just a pain and not maintaining squat), and it leads to totally funny results (for instance, I type "emirates" in my "Air Companies" subfolder and get a lot of results .. but of competing companies :D )
I added the fts_enforce following AKi advice.
I removed fts_decoder for the time being.
I don't know where to go now. Dovcot still returning errors and SOlr still companinig with "Out of range" and other Java errors.
Bottom line, I am back to squat, but as it is not maintained so crashed also times to times.
I think we should discuss on
(1) Why the damn choice of Solr has been main. As you empahised, maintainend so many independent software is a pain
(2) If there is a real reason why going for SOlr, how to have a working (i.e. getting the right results to the end user) setup ?
(3) If there iare no tangible reason, what about maintaining fts_squat , which did the job nicely for years and no complains about.
On 2018-12-16 08:51, Daniel Miller via dovecot wrote:
Joan,
I understand and sympathize with your frustration - trying to get multiple applications to work together, particularly given the lack of documentation for some of them, can be extremely challenging. That said, I suggest you consider an alternative viewpoint. Frequently being misunderstood myself I apologize in advance if I'm reading you wrong - but it appears your view towards the situation is there is a bug in Dovecot related to this problem. That may well be these matters - but I do have a working setup and I'd like to help
- but I generally approach these matters from the assumption that *I* made the error in configuration and go from there. I'm not an official rep for any product nor claim to be any form of expert in
you if I can. If you're willing to - take a deep breath and let's try starting over.
Looking back through your emails there were two items that stood out
- your Dovecot config has two settings I don't use: "fts_decoder" and "fts_enforced". I also asked you earlier whether or not NFS is involved here and I didn't see an answer - please clarify.
I suggest you try once more: delete Solr completely. Re-install per the directions and use *my* managed-schema. Also comment out the Dovecot directives for "fts_decoder" and "fts_enforced" so you're closer to my setup. Try running again and then post back - I'll do what I can. Based on the fact that Dovecot+Solr 7.5+my schema is working for me leads me to believe we can get it working for you as well.
Daniel
On 12/15/2018 2:42 PM, Joan Moreau wrote:
here my latest schema.xml (remove the "long" type hich seems to be very deprecated in 7.x)
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0" /> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="hdr" type="gjtext" indexed="false" stored="false"/> <field name="cc" type="gjtext" indexed="true" stored="false"/> <field name="from" type="gjtext" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjtext" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </fields> </schema>
On 2018-12-15 20:54, Joan Moreau wrote:
Daniel, I have done that so any times (deleteing the data folders, recreating the instance, restarting etc...) But this is really not the issue The issue is 1 - fts_solr reports errors in the log file (this is a pure dovecot issue) : how to have much more details on what fts_solr sends to Slor server and what does it returns ? 2 - Solr returns properly for a few hours, then starts crashing or responding non-sense after some time Additionally, is there a doc of fts-squat in order to adjust the code to new releases of dovect ? On December 12, 2018 4:44:10 PM Daniel Miller via dovecot <dovecot@dovecot.org> wrote: On 12/11/2018 4:46 AM, Joan Moreau via dovecot wrote: I shared the errors already so many times (check this mailinling for "solr" in teh title) Contrary to what you say, with SOlr 7.5 and Dovecot git, I had to remove the "managed-schema" to make solr respond a bit properly. It relies on schema.xml In order to create the instance, no, it copies the default config in the dovecot instance. I'm not a Solr expert by any means but I believe you are incorrect. As of Solr 5.x the managed-schema file is the primary method for configuration. The method I detailed previously for setting up a config helps automate creating new Solr instances - but as I stated you can either setup a Solr template and then create the instance from that or create an instance using the default template and then adjust it. The part that you *must* do after creating from the default template is stop the server, delete the entire "<prefix>/solr/dovecot/data" folder, then install the correct managed-schema file, then restart the server. The server will not function with mismatched schema/data. If you'll try that - explicitly "rm -rf <prefix>/solr/dovecot/data", copy the managed-schema file into the conf folder, and restart - things will either work or there's something else that needs correction. -- Daniel
Hi Daniel,
I am on Archlinux. Anyway, I adapted the scripts.
2 questions:
1 - It looks like we are not on the same version . I am on 7.5.0. Which version are you running ?
2 - Your conf shows that you let managed-schema but deleted schema.xml. What is the meaning of each ?
Thank you
On 2018-12-22 00:19, Daniel Miller wrote:
Joan,
The reason for dropping squat, I'm assuming, is that Lucene and Solr potentially provide superior features & performance and as they are 3rd-party libraries & apps it reduces the maintenance responsibilities and let's the Dovecot team focus on mail server specific stuff - and let others focus on FTS. There is a *huge* difference between a functional Solr setup & squat - and if I'm able to get it working we should be able to get you there as well.
I don't recall what OS you're running - I'm on Ubuntu 18.04. My Java version is OpenJDK 10.0.2. Attached is my complete Solr config. Try one more time - stop the server, delete the data folder, unpack the attached into the conf folder - and restart. I also have
/etc/default/solr.in.sh: SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=3000" SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=60000" SOLR_PID_DIR=/run/solr SOLR_HOME=/usr/local/lib Adjust the above folders as appropriate - or don't use them at all if you're using the defaults.
/etc/systemd/system/solr.service: # put this file in /etc/systemd/system/ as root # below paths assume solr installed in /opt/solr, SOLR_PID_DIR is /data # and that all configuration exists in /etc/default/solr.in.sh which is the case if previously installed as an init.d service # change port in pid file if differs # note that it is configured to auto restart solr if it fails (Restart=on-faliure) and that's the motivation indeed :) # to switch from systemv (init.d) to systemd, do the following after creating this file: # sudo systemctl daemon-reload # sudo service solr stop # if already running # sudo systemctl enable solr # systemctl start solr # this was inspired by https://confluence.t5.fi/display/~stefan.roos/2015/04/01/Creating+systemd+un... [Unit] Description=Apache SOLR 7.5.0 After=syslog.target network.target remote-fs.target nss-lookup.target systemd-journald-dev-log.socket Before=multi-user.target graphical.target nginx.service dovecot.service Conflicts=shutdown.target [Service] LimitNOFILE=65000 User=vmail Group=mail ExecStartPre=/bin/mkdir -p /run/solr ExecStartPre=/bin/chown -R vmail.mail /run/solr PermissionsStartOnly=true PIDFile=/run/solr/solr-8983.pid Environment=SOLR_INCLUDE=/etc/default/solr.in.sh ExecStart=/opt/solr/bin/solr start ExecStop=/opt/solr/bin/solr stop Restart=on-failure RestartSec=15s TimeoutStopSec=30s [Install] WantedBy=multi-user.target graphical.target dovecot.service If you don't use systemd disregard - but see if any of the above applies for your setup. Let me know what happens. I agree this can be a mortal pain to setup - but it's worth it.
Daniel
On 12/21/2018 4:33 AM, Joan Moreau wrote:
Dear Daniel.
Thank you for your kind reply.
Regarding NFS, no, there is nothing like this in my setup.
Deleteing SOLR and recreating it, I did it so many times already.
I started with *your* setup in the first place, as FTS_squat (which actually works very well and very straightforward, I have no clue why going for SOlr which is just a pain and not maintaining squat), and it leads to totally funny results (for instance, I type "emirates" in my "Air Companies" subfolder and get a lot of results .. but of competing companies :D )
I added the fts_enforce following AKi advice.
I removed fts_decoder for the time being.
I don't know where to go now. Dovcot still returning errors and SOlr still companinig with "Out of range" and other Java errors.
Bottom line, I am back to squat, but as it is not maintained so crashed also times to times.
I think we should discuss on
(1) Why the damn choice of Solr has been main. As you empahised, maintainend so many independent software is a pain
(2) If there is a real reason why going for SOlr, how to have a working (i.e. getting the right results to the end user) setup ?
(3) If there iare no tangible reason, what about maintaining fts_squat , which did the job nicely for years and no complains about.
On 2018-12-16 08:51, Daniel Miller via dovecot wrote:
Joan,
I understand and sympathize with your frustration - trying to get multiple applications to work together, particularly given the lack of documentation for some of them, can be extremely challenging. That said, I suggest you consider an alternative viewpoint. Frequently being misunderstood myself I apologize in advance if I'm reading you wrong - but it appears your view towards the situation is there is a bug in Dovecot related to this problem. That may well be - but I generally approach these matters from the assumption that *I* made the error in configuration and go from there. I'm not an official rep for any product nor claim to be any form of expert in these matters - but I do have a working setup and I'd like to help you if I can. If you're willing to - take a deep breath and let's try starting over.
Looking back through your emails there were two items that stood out - your Dovecot config has two settings I don't use: "fts_decoder" and "fts_enforced". I also asked you earlier whether or not NFS is involved here and I didn't see an answer - please clarify.
I suggest you try once more: delete Solr completely. Re-install per the directions and use *my* managed-schema. Also comment out the Dovecot directives for "fts_decoder" and "fts_enforced" so you're closer to my setup. Try running again and then post back - I'll do what I can. Based on the fact that Dovecot+Solr 7.5+my schema is working for me leads me to believe we can get it working for you as well.
Daniel
On 12/15/2018 2:42 PM, Joan Moreau wrote:
here my latest schema.xml (remove the "long" type hich seems to be very deprecated in 7.x)
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0" /> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="hdr" type="gjtext" indexed="false" stored="false"/> <field name="cc" type="gjtext" indexed="true" stored="false"/> <field name="from" type="gjtext" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjtext" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </fields> </schema>
On 2018-12-15 20:54, Joan Moreau wrote:
Daniel,
I have done that so any times (deleteing the data folders, recreating the instance, restarting etc...)
But this is really not the issue
The issue is 1 - fts_solr reports errors in the log file (this is a pure dovecot issue) : how to have much more details on what fts_solr sends to Slor server and what does it returns ? 2 - Solr returns properly for a few hours, then starts crashing or responding non-sense after some time
Additionally, is there a doc of fts-squat in order to adjust the code to new releases of dovect ?
On December 12, 2018 4:44:10 PM Daniel Miller via dovecot <dovecot@dovecot.org> wrote: On 12/11/2018 4:46 AM, Joan Moreau via dovecot wrote:
I shared the errors already so many times (check this mailinling for "solr" in teh title)
Contrary to what you say, with SOlr 7.5 and Dovecot git, I had to remove the "managed-schema" to make solr respond a bit properly. It relies on schema.xml
In order to create the instance, no, it copies the default config in the dovecot instance.
I'm not a Solr expert by any means but I believe you are incorrect.
As of Solr 5.x the managed-schema file is the primary method for configuration. The method I detailed previously for setting up a config helps automate creating new Solr instances - but as I stated you can either setup a Solr template and then create the instance from that or create an instance using the default template and then adjust it.
The part that you *must* do after creating from the default template is stop the server, delete the entire "<prefix>/solr/dovecot/data" folder, then install the correct managed-schema file, then restart the server. The server will not function with mismatched schema/data.
If you'll try that - explicitly "rm -rf <prefix>/solr/dovecot/data", copy the managed-schema file into the conf folder, and restart - things will either work or there's something else that needs correction. -- Daniel
Also :
Java is 10.0.2
If i delete schema.xml but create only managed-schema, the solr refuses to start with a java error "schema.xml missing"
On 2018-12-30 08:46, Joan Moreau wrote:
Hi Daniel,
I am on Archlinux. Anyway, I adapted the scripts.
2 questions:
1 - It looks like we are not on the same version . I am on 7.5.0. Which version are you running ?
2 - Your conf shows that you let managed-schema but deleted schema.xml. What is the meaning of each ?
Thank you
On 2018-12-22 00:19, Daniel Miller wrote:
Joan,
The reason for dropping squat, I'm assuming, is that Lucene and Solr potentially provide superior features & performance and as they are 3rd-party libraries & apps it reduces the maintenance responsibilities and let's the Dovecot team focus on mail server specific stuff - and let others focus on FTS. There is a *huge* difference between a functional Solr setup & squat - and if I'm able to get it working we should be able to get you there as well.
I don't recall what OS you're running - I'm on Ubuntu 18.04. My Java version is OpenJDK 10.0.2. Attached is my complete Solr config. Try one more time - stop the server, delete the data folder, unpack the attached into the conf folder - and restart. I also have
/etc/default/solr.in.sh: SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=3000" SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=60000" SOLR_PID_DIR=/run/solr SOLR_HOME=/usr/local/lib Adjust the above folders as appropriate - or don't use them at all if you're using the defaults.
/etc/systemd/system/solr.service: # put this file in /etc/systemd/system/ as root # below paths assume solr installed in /opt/solr, SOLR_PID_DIR is /data # and that all configuration exists in /etc/default/solr.in.sh which is the case if previously installed as an init.d service # change port in pid file if differs # note that it is configured to auto restart solr if it fails (Restart=on-faliure) and that's the motivation indeed :) # to switch from systemv (init.d) to systemd, do the following after creating this file: # sudo systemctl daemon-reload # sudo service solr stop # if already running # sudo systemctl enable solr # systemctl start solr # this was inspired by https://confluence.t5.fi/display/~stefan.roos/2015/04/01/Creating+systemd+un... [Unit] Description=Apache SOLR 7.5.0 After=syslog.target network.target remote-fs.target nss-lookup.target systemd-journald-dev-log.socket Before=multi-user.target graphical.target nginx.service dovecot.service Conflicts=shutdown.target [Service] LimitNOFILE=65000 User=vmail Group=mail ExecStartPre=/bin/mkdir -p /run/solr ExecStartPre=/bin/chown -R vmail.mail /run/solr PermissionsStartOnly=true PIDFile=/run/solr/solr-8983.pid Environment=SOLR_INCLUDE=/etc/default/solr.in.sh ExecStart=/opt/solr/bin/solr start ExecStop=/opt/solr/bin/solr stop Restart=on-failure RestartSec=15s TimeoutStopSec=30s [Install] WantedBy=multi-user.target graphical.target dovecot.service If you don't use systemd disregard - but see if any of the above applies for your setup. Let me know what happens. I agree this can be a mortal pain to setup - but it's worth it.
Daniel
On 12/21/2018 4:33 AM, Joan Moreau wrote:
Dear Daniel.
Thank you for your kind reply.
Regarding NFS, no, there is nothing like this in my setup.
Deleteing SOLR and recreating it, I did it so many times already.
I started with *your* setup in the first place, as FTS_squat (which actually works very well and very straightforward, I have no clue why going for SOlr which is just a pain and not maintaining squat), and it leads to totally funny results (for instance, I type "emirates" in my "Air Companies" subfolder and get a lot of results .. but of competing companies :D )
I added the fts_enforce following AKi advice.
I removed fts_decoder for the time being.
I don't know where to go now. Dovcot still returning errors and SOlr still companinig with "Out of range" and other Java errors.
Bottom line, I am back to squat, but as it is not maintained so crashed also times to times.
I think we should discuss on
(1) Why the damn choice of Solr has been main. As you empahised, maintainend so many independent software is a pain
(2) If there is a real reason why going for SOlr, how to have a working (i.e. getting the right results to the end user) setup ?
(3) If there iare no tangible reason, what about maintaining fts_squat , which did the job nicely for years and no complains about.
On 2018-12-16 08:51, Daniel Miller via dovecot wrote:
Joan,
I understand and sympathize with your frustration - trying to get multiple applications to work together, particularly given the lack of documentation for some of them, can be extremely challenging. That said, I suggest you consider an alternative viewpoint. Frequently being misunderstood myself I apologize in advance if I'm reading you wrong - but it appears your view towards the situation is there is a bug in Dovecot related to this problem. That may well be - but I generally approach these matters from the assumption that *I* made the error in configuration and go from there. I'm not an official rep for any product nor claim to be any form of expert in these matters - but I do have a working setup and I'd like to help you if I can. If you're willing to - take a deep breath and let's try starting over.
Looking back through your emails there were two items that stood out - your Dovecot config has two settings I don't use: "fts_decoder" and "fts_enforced". I also asked you earlier whether or not NFS is involved here and I didn't see an answer - please clarify.
I suggest you try once more: delete Solr completely. Re-install per the directions and use *my* managed-schema. Also comment out the Dovecot directives for "fts_decoder" and "fts_enforced" so you're closer to my setup. Try running again and then post back - I'll do what I can. Based on the fact that Dovecot+Solr 7.5+my schema is working for me leads me to believe we can get it working for you as well.
Daniel
On 12/15/2018 2:42 PM, Joan Moreau wrote:
here my latest schema.xml (remove the "long" type hich seems to be very deprecated in 7.x)
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0" /> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="hdr" type="gjtext" indexed="false" stored="false"/> <field name="cc" type="gjtext" indexed="true" stored="false"/> <field name="from" type="gjtext" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjtext" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </fields> </schema>
On 2018-12-15 20:54, Joan Moreau wrote:
Daniel,
I have done that so any times (deleteing the data folders, recreating the instance, restarting etc...)
But this is really not the issue
The issue is 1 - fts_solr reports errors in the log file (this is a pure dovecot issue) : how to have much more details on what fts_solr sends to Slor server and what does it returns ? 2 - Solr returns properly for a few hours, then starts crashing or responding non-sense after some time
Additionally, is there a doc of fts-squat in order to adjust the code to new releases of dovect ?
On December 12, 2018 4:44:10 PM Daniel Miller via dovecot <dovecot@dovecot.org> wrote: On 12/11/2018 4:46 AM, Joan Moreau via dovecot wrote:
I shared the errors already so many times (check this mailinling for "solr" in teh title)
Contrary to what you say, with SOlr 7.5 and Dovecot git, I had to remove the "managed-schema" to make solr respond a bit properly. It relies on schema.xml
In order to create the instance, no, it copies the default config in the dovecot instance.
I'm not a Solr expert by any means but I believe you are incorrect.
As of Solr 5.x the managed-schema file is the primary method for configuration. The method I detailed previously for setting up a config helps automate creating new Solr instances - but as I stated you can either setup a Solr template and then create the instance from that or create an instance using the default template and then adjust it.
The part that you *must* do after creating from the default template is stop the server, delete the entire "<prefix>/solr/dovecot/data" folder, then install the correct managed-schema file, then restart the server. The server will not function with mismatched schema/data.
If you'll try that - explicitly "rm -rf <prefix>/solr/dovecot/data", copy the managed-schema file into the conf folder, and restart - things will either work or there's something else that needs correction. -- Daniel
On 12/29/2018 4:49 PM, Joan Moreau wrote:
Also :
- Java is 10.0.2
Same as me.
- If i delete schema.xml but create only managed-schema, the solr refuses to start with a java error "schema.xml missing"
Ok...so we need to do some more digging.
How did you install Solr? (I downloaded a "binary" installation and unpacked it)
How did you create the dovecot instance? (I've provided explicit instructions for how I did it - did you follow those exactly or something different)?
How are you starting Solr? (I use the provided "solr/bin/solr start" command, wrapped inside a systemd service).
-- Daniel
Hi
Solr is a standard package in ArchLinux. ("pacman -S solr") . the systemd installation script is included (and it is launching /opt/solr/bin/solr.in.sh)
Instance : sudo -u solr /opt/solr/bin/solr create -c dovecot -> this creates a separate folder with default solrconfig.xml, schema.xml, etc..
I made a symlink of the data folder to a second drive (ext4) much bigger
On 2018-12-31 14:09, Daniel Miller wrote:
On 12/29/2018 4:49 PM, Joan Moreau wrote:
Also :
- Java is 10.0.2 Same as me.
- If i delete schema.xml but create only managed-schema, the solr refuses to start with a java error "schema.xml missing" Ok...so we need to do some more digging.
How did you install Solr? (I downloaded a "binary" installation and unpacked it)
How did you create the dovecot instance? (I've provided explicit instructions for how I did it - did you follow those exactly or something different)?
How are you starting Solr? (I use the provided "solr/bin/solr start" command, wrapped inside a systemd service).
-- Daniel
The real main differecne seems coming from "diffconfig.xml"
When I put yours, Solr delete (!) schema.xml and create a "manage-schema" and starts complaining about useless types (tdates, booleans, etc..) that are not needed for Mail fileds
When I put mine (from standard distribution of Arch), it keeps things as they are (yeah !), does not complains about those useless types and startup properly.
I attach my diffconfig
But these are the configurations that one should adjust as per his/her own use.
The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors
On 2019-01-02 07:49, Joan Moreau wrote:
Hi
Solr is a standard package in ArchLinux. ("pacman -S solr") . the systemd installation script is included (and it is launching /opt/solr/bin/solr.in.sh)
Instance : sudo -u solr /opt/solr/bin/solr create -c dovecot -> this creates a separate folder with default solrconfig.xml, schema.xml, etc..
I made a symlink of the data folder to a second drive (ext4) much bigger
On 2018-12-31 14:09, Daniel Miller wrote: On 12/29/2018 4:49 PM, Joan Moreau wrote: Also :
- Java is 10.0.2
Same as me.
- If i delete schema.xml but create only managed-schema, the solr refuses to start with a java error "schema.xml missing"
Ok...so we need to do some more digging.
How did you install Solr? (I downloaded a "binary" installation and unpacked it)
How did you create the dovecot instance? (I've provided explicit instructions for how I did it - did you follow those exactly or something different)?
How are you starting Solr? (I use the provided "solr/bin/solr start" command, wrapped inside a systemd service).
-- Daniel
and the first line of the diff is :
< this file, see http://wiki.apache.org/solr/SolrConfigXml.
this file, see http://wiki.apache.org/solr/SolrConfigXml. 38c38 < <luceneMatchVersion>6.4.1</luceneMatchVersion>
<luceneMatchVersion>7.5.0</luceneMatchVersion>
So, are you running 6.4.1 or 7.5.0 ????
On 2019-01-02 08:12, Joan Moreau wrote:
The real main differecne seems coming from "diffconfig.xml"
When I put yours, Solr delete (!) schema.xml and create a "manage-schema" and starts complaining about useless types (tdates, booleans, etc..) that are not needed for Mail fileds
When I put mine (from standard distribution of Arch), it keeps things as they are (yeah !), does not complains about those useless types and startup properly.
I attach my diffconfig
But these are the configurations that one should adjust as per his/her own use.
The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors
On 2019-01-02 07:49, Joan Moreau wrote:
Hi
Solr is a standard package in ArchLinux. ("pacman -S solr") . the systemd installation script is included (and it is launching /opt/solr/bin/solr.in.sh)
Instance : sudo -u solr /opt/solr/bin/solr create -c dovecot -> this creates a separate folder with default solrconfig.xml, schema.xml, etc..
I made a symlink of the data folder to a second drive (ext4) much bigger
On 2018-12-31 14:09, Daniel Miller wrote: On 12/29/2018 4:49 PM, Joan Moreau wrote: Also :
- Java is 10.0.2
Same as me.
- If i delete schema.xml but create only managed-schema, the solr refuses to start with a java error "schema.xml missing"
Ok...so we need to do some more digging.
How did you install Solr? (I downloaded a "binary" installation and unpacked it)
How did you create the dovecot instance? (I've provided explicit instructions for how I did it - did you follow those exactly or something different)?
How are you starting Solr? (I use the provided "solr/bin/solr start" command, wrapped inside a systemd service).
-- Daniel
The first result show "no results" in dovecot for any search by header (I typed an email add in RoundCube search box, using Dovecot as back end, using Solr as own backend)
So many efforts for crappy results.
Can't we really revive Squat ? It is 2 lines of config, and no single problems
On January 2, 2019 08:16:33 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
and the first line of the diff is : < this file, see http://wiki.apache.org/solr/SolrConfigXml.
this file, see http://wiki.apache.org/solr/SolrConfigXml. 38c38 < <luceneMatchVersion>6.4.1</luceneMatchVersion>
<luceneMatchVersion>7.5.0</luceneMatchVersion>
So, are you running 6.4.1 or 7.5.0 ????
On 2019-01-02 08:12, Joan Moreau wrote:
The real main differecne seems coming from "diffconfig.xml"
When I put yours, Solr delete (!) schema.xml and create a "manage-schema" and starts complaining about useless types (tdates, booleans, etc..) that are not needed for Mail fileds
When I put mine (from standard distribution of Arch), it keeps things as they are (yeah !), does not complains about those useless types and startup properly.
I attach my diffconfig
But these are the configurations that one should adjust as per his/her own use.
The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors
On 2019-01-02 07:49, Joan Moreau wrote:
Hi
Solr is a standard package in ArchLinux. ("pacman -S solr") . the systemd installation script is included (and it is launching /opt/solr/bin/solr.in.sh)
Instance : sudo -u solr /opt/solr/bin/solr create -c dovecot -> this creates a separate folder with default solrconfig.xml, schema.xml, etc..
I made a symlink of the data folder to a second drive (ext4) much bigger
On 2018-12-31 14:09, Daniel Miller wrote:
On 12/29/2018 4:49 PM, Joan Moreau wrote:
Also :
- Java is 10.0.2
Same as me.
- If i delete schema.xml but create only managed-schema, the solr refuses to start with a java error "schema.xml missing"
Ok...so we need to do some more digging.
How did you install Solr? (I downloaded a "binary" installation and unpacked it)
How did you create the dovecot instance? (I've provided explicit instructions for how I did it - did you follow those exactly or something different)?
How are you starting Solr? (I use the provided "solr/bin/solr start" command, wrapped inside a systemd service).
-- Daniel
Other use case :
I type "must" in the search filed-> I have some returns , but very not all, for instance "solarmust" is not in the results
If I type "solarmust" -> then I have the solarmust mail
Honestly, this is highly unstable. Not sure whereas bugs come from Solr or Dovecot
Below my adjusted (corrections from the one of Daniel who is definitely not working) schema.xml
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="booleans" class="solr.BoolField" sortMissingLast="true" multiValued="true"/> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0"/> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.EdgeNGramTokenizerFactory" maxGramSize="15" minGramSize="3" /> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> <fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="gjtext" indexed="true" stored="false"/> <field name="from" type="gjtext" indexed="true" stored="false"/> <field name="hdr" type="gjtext" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjtext" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
On 2019-01-02 10:04, Joan Moreau wrote:
The first result show "no results" in dovecot for any search by header (I typed an email add in RoundCube search box, using Dovecot as back end, using Solr as own backend)
So many efforts for crappy results.
Can't we really revive Squat ? It is 2 lines of config, and no single problems
On January 2, 2019 08:16:33 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
and the first line of the diff is :
< this file, see http://wiki.apache.org/solr/SolrConfigXml.
this file, see http://wiki.apache.org/solr/SolrConfigXml. 38c38 < <luceneMatchVersion>6.4.1</luceneMatchVersion>
<luceneMatchVersion>7.5.0</luceneMatchVersion>
So, are you running 6.4.1 or 7.5.0 ????
On 2019-01-02 08:12, Joan Moreau wrote:
The real main differecne seems coming from "diffconfig.xml"
When I put yours, Solr delete (!) schema.xml and create a "manage-schema" and starts complaining about useless types (tdates, booleans, etc..) that are not needed for Mail fileds
When I put mine (from standard distribution of Arch), it keeps things as they are (yeah !), does not complains about those useless types and startup properly.
I attach my diffconfig
But these are the configurations that one should adjust as per his/her own use.
The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors
On 2019-01-02 07:49, Joan Moreau wrote:
Hi
Solr is a standard package in ArchLinux. ("pacman -S solr") . the systemd installation script is included (and it is launching /opt/solr/bin/solr.in.sh)
Instance : sudo -u solr /opt/solr/bin/solr create -c dovecot -> this creates a separate folder with default solrconfig.xml, schema.xml, etc..
I made a symlink of the data folder to a second drive (ext4) much bigger
On 2018-12-31 14:09, Daniel Miller wrote: On 12/29/2018 4:49 PM, Joan Moreau wrote: Also :
- Java is 10.0.2
Same as me.
- If i delete schema.xml but create only managed-schema, the solr refuses to start with a java error "schema.xml missing"
Ok...so we need to do some more digging.
How did you install Solr? (I downloaded a "binary" installation and unpacked it)
How did you create the dovecot instance? (I've provided explicit instructions for how I did it - did you follow those exactly or something different)?
How are you starting Solr? (I use the provided "solr/bin/solr start" command, wrapped inside a systemd service).
-- Daniel
Refinement of the schema.xml (below)
THis however does not solve the "no results" and "Out of range" errors in Dovecot and Solr
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="booleans" class="solr.BoolField" sortMissingLast="true" multiValued="true"/> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0"/> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> </analyzer> </fieldType> <fieldType name="gjfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.NGramTokenizerFactory" maxGramSize="25" minGramSize="3" /> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> </analyzer> </fieldType>
<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="gjfield" indexed="true" stored="false"/> <field name="from" type="gjfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjfield" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
Another bug appearing today:
Jan 02 09:59:08 indexer-worker(jom@grosjo.net)<6777><MOgFATKLLFwPHAAA0thIag:oLJjJjKLLFx5GgAA0thIag>: Warning: FTS-SOLR(JOM@GROSJO.NET): Mailbox XXXXXX UID=121635 HEADER SIZE IS HUGE, TRUNCATING
header of the said email has nothing of "huge"
On 2019-01-02 15:22, Joan Moreau via dovecot wrote:
Refinement of the schema.xml (below)
THis however does not solve the "no results" and "Out of range" errors in Dovecot and Solr
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="booleans" class="solr.BoolField" sortMissingLast="true" multiValued="true"/> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0"/> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> </analyzer> </fieldType> <fieldType name="gjfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.NGramTokenizerFactory" maxGramSize="25" minGramSize="3" /> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> </analyzer> </fieldType>
<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="gjfield" indexed="true" stored="false"/> <field name="from" type="gjfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjfield" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
In the code, it says
#define SOLR_HEADER_MAX_SIZE (1024*1024)
and line 620
if (!ctx->truncate_header && str_len(ctx->cur_value) >= SOLR_HEADER_MAX_SIZE) { /* a large header */
so this is a 1Mo header. This is of course completely wrong.
Maybe this is not the root cause of the errors of fts_solr, but maybe this will help
On 2019-01-02 18:00, Joan Moreau via dovecot wrote:
Another bug appearing today:
Jan 02 09:59:08 indexer-worker(jom@grosjo.net)<6777><MOgFATKLLFwPHAAA0thIag:oLJjJjKLLFx5GgAA0thIag>: Warning: FTS-SOLR(JOM@GROSJO.NET): Mailbox XXXXXX UID=121635 HEADER SIZE IS HUGE, TRUNCATING
header of the said email has nothing of "huge"
On 2019-01-02 15:22, Joan Moreau via dovecot wrote:
Refinement of the schema.xml (below)
THis however does not solve the "no results" and "Out of range" errors in Dovecot and Solr
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="booleans" class="solr.BoolField" sortMissingLast="true" multiValued="true"/> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0"/> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> </analyzer> </fieldType> <fieldType name="gjfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.NGramTokenizerFactory" maxGramSize="25" minGramSize="3" /> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> </analyzer> </fieldType>
<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="gjfield" indexed="true" stored="false"/> <field name="from" type="gjfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjfield" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
Hi,
This is the best I can do for having a SCHEMA.XML matching the needs of an email user. Use case on Solr 7.5.0
(note : this does not solve any of the bugs previously mentionned, but at least, the logic of Solr is back on track)
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> <fieldType name="gjfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="25"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType>
<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="gjfield" indexed="true" stored="false"/> <field name="from" type="gjfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjfield" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
Op 02/01/2019 om 11:00 schreef Joan Moreau via dovecot:
Another bug appearing today:
Jan 02 09:59:08 indexer-worker(jom@grosjo.net)<6777><MOgFATKLLFwPHAAA0thIag:oLJjJjKLLFx5GgAA0thIag>: Warning:*fts-solr(jom@grosjo.net)*: Mailbox XXXXXX UID=121635*header size is huge, truncating*
header of the said email has nothing of "huge"
Does this happen consistently with that same message? Can you show it to me?
Regards,
Stephan.
On 2019-01-02 15:22, Joan Moreau via dovecot wrote:
Refinement of the schema.xml (below)
THis however does not solve the "no results" and "Out of range" errors in Dovecot and Solr
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="booleans" class="solr.BoolField" sortMissingLast="true" multiValued="true"/> <fieldType name="gjlong" class="solr.LongPointField" positionIncrementGap="0"/> <fieldType name="gjtext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> </analyzer> </fieldType> <fieldType name="gjfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.NGramTokenizerFactory" maxGramSize="25" minGramSize="3" /> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> </analyzer> </fieldType>
<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="gjtext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="gjfield" indexed="true" stored="false"/> <field name="from" type="gjfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="gjtext" indexed="true" stored="false"/> <field name="to" type="gjfield" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors
I've been watching the progress of this thread with no small concern, mainly because I've been tasked with providing a server-side email search facility with a budget and manpower level that comes down to mainly *1*, i.e., me.
I was expecting, given the strongly worded language about "just use lucene/SOLR" and "ignore squat", that I should invest time + effort into this JAVA nightmare that is SOLR.
I started with squat and another word-indexor system that used out-of-band (not a dovecot plugin) software to provide rapid (sub-second) searches through tens-of-GB-scale mailboxes.
Unlike what I was led to believe, the squat indexes worked surprisingly well, once you sorted out the odd resource size (ulimit-related) issues (vsz & friends) limitations. I did notice the "worst-case" search performance have worryingly high O(x) increases in time, but I'd not seen anything that was a dealbreaker. It goes without saying that various substring searches worked as expected, for the most part.
My experiences with SOLR were similar to Messr. Moreau's: lots of startup errors with provided schemata files. Lots of JAVA nonsense issues. Lots of sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM, version of SOLR, and dovecot to find the "best" working combination, only to find that the searches didn't work out as expected. I expected to be able to do date-ranging based searches. Didn't work. I expected to search CONTENTS of emails, and despite many days of tweaks, I couldn't get it to index even the basics like filenames/types of attachments, so I could exposed attachment-based searching to my users.
So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality:
The ability to search for a string within any of the structured fields (from/subject) that returns correct results?
The ability to search for any string within the BODY of emails, including the MIME attachment boundaries?
The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data?
OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines:
- The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika?
SOLR seems to have "brand cachet", so presumably it actually works (for somebody).
Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I am no stranger to the "costs" of "free" software, in that you sacrifice your own blood, sweat, and tears just to get these disparate pieces to work together.
I *DO* respect that Timo has to keep the lights (and sauna) on in Finland. Maybe there's a super-secret (no advertised prices, "carrier-only" price list) with _Dovecot, Oy_ wherein the above ARE actually available for something less than 6.022 x 10^23 Euros per centi-second of licencing fees.
But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?)
I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have.
He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them.
Respectfully, =M=
On 02 January 2019 at 10:59 "M. Balridge" <dovecot@r.paypc.com> wrote:
The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors
I've been watching the progress of this thread with no small concern, mainly because I've been tasked with providing a server-side email search facility with a budget and manpower level that comes down to mainly *1*, i.e., me.
I was expecting, given the strongly worded language about "just use lucene/SOLR" and "ignore squat", that I should invest time + effort into this JAVA nightmare that is SOLR.
I started with squat and another word-indexor system that used out-of-band (not a dovecot plugin) software to provide rapid (sub-second) searches through tens-of-GB-scale mailboxes.
Unlike what I was led to believe, the squat indexes worked surprisingly well, once you sorted out the odd resource size (ulimit-related) issues (vsz & friends) limitations. I did notice the "worst-case" search performance have worryingly high O(x) increases in time, but I'd not seen anything that was a dealbreaker. It goes without saying that various substring searches worked as expected, for the most part.
My experiences with SOLR were similar to Messr. Moreau's: lots of startup errors with provided schemata files. Lots of JAVA nonsense issues. Lots of sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM, version of SOLR, and dovecot to find the "best" working combination, only to find that the searches didn't work out as expected. I expected to be able to do date-ranging based searches. Didn't work. I expected to search CONTENTS of emails, and despite many days of tweaks, I couldn't get it to index even the basics like filenames/types of attachments, so I could exposed attachment-based searching to my users.
So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality:
The ability to search for a string within any of the structured fields (from/subject) that returns correct results?
The ability to search for any string within the BODY of emails, including the MIME attachment boundaries?
The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data?
OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines:
- The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika?
SOLR seems to have "brand cachet", so presumably it actually works (for somebody).
Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I am no stranger to the "costs" of "free" software, in that you sacrifice your own blood, sweat, and tears just to get these disparate pieces to work together.
I *DO* respect that Timo has to keep the lights (and sauna) on in Finland. Maybe there's a super-secret (no advertised prices, "carrier-only" price list) with _Dovecot, Oy_ wherein the above ARE actually available for something less than 6.022 x 10^23 Euros per centi-second of licencing fees.
But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?)
I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have.
He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them.
Respectfully, =M=
We do intend to polish fts-solr before we drop fts-squat. And even then, anyone is free to pick it up and continue the work, as it works as plugin just fine, so it's not a matter of us just flushing it to oblivion.
fts-squat is not really worth pursuing for us since it would eat away effort from our current dovecot fts plugin, which unfortunately is not currently open-sourced.
Aki
On Wed, 2019-01-02 at 00:59 -0800, M. Balridge wrote:
The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors
I've been watching the progress of this thread with no small concern, mainly because I've been tasked with providing a server-side email search facility with a budget and manpower level that comes down to mainly *1*, i.e., me.
I was expecting, given the strongly worded language about "just use lucene/SOLR" and "ignore squat", that I should invest time + effort into this JAVA nightmare that is SOLR.
I started with squat and another word-indexor system that used out-of-band (not a dovecot plugin) software to provide rapid (sub-second) searches through tens-of-GB-scale mailboxes.
Unlike what I was led to believe, the squat indexes worked surprisingly well, once you sorted out the odd resource size (ulimit-related) issues (vsz & friends) limitations. I did notice the "worst-case" search performance have worryingly high O(x) increases in time, but I'd not seen anything that was a dealbreaker. It goes without saying that various substring searches worked as expected, for the most part.
My experiences with SOLR were similar to Messr. Moreau's: lots of startup errors with provided schemata files. Lots of JAVA nonsense issues. Lots of sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM, version of SOLR, and dovecot to find the "best" working combination, only to find that the searches didn't work out as expected. I expected to be able to do date-ranging based searches. Didn't work. I expected to search CONTENTS of emails, and despite many days of tweaks, I couldn't get it to index even the basics like filenames/types of attachments, so I could exposed attachment-based searching to my users.
So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality:
The ability to search for a string within any of the structured fields (from/subject) that returns correct results?
The ability to search for any string within the BODY of emails, including the MIME attachment boundaries?
The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data?
OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines:
- The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika?
SOLR seems to have "brand cachet", so presumably it actually works (for somebody).
Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I am no stranger to the "costs" of "free" software, in that you sacrifice your own blood, sweat, and tears just to get these disparate pieces to work together.
I *DO* respect that Timo has to keep the lights (and sauna) on in Finland. Maybe there's a super-secret (no advertised prices, "carrier-only" price list) with _Dovecot, Oy_ wherein the above ARE actually available for something less than 6.022 x 10^23 Euros per centi-second of licencing fees.
But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?)
I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have.
He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them.
Respectfully, =M=
Fascinating...
SOLR says the following are powered by SOLR...
https://wiki.apache.org/solr/PublicServers
Perhaps if you could find out from that list which of them are using SOLR in conjunction with Dovecot...
food for thought...
What about consedering linking Dovecot with Xapian librairies instead of going to nightmare Solr ?
On 2019-01-02 17:10, John Tulp wrote:
On Wed, 2019-01-02 at 00:59 -0800, M. Balridge wrote: The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors I've been watching the progress of this thread with no small concern, mainly because I've been tasked with providing a server-side email search facility with a budget and manpower level that comes down to mainly *1*, i.e., me.
I was expecting, given the strongly worded language about "just use lucene/SOLR" and "ignore squat", that I should invest time + effort into this JAVA nightmare that is SOLR.
I started with squat and another word-indexor system that used out-of-band (not a dovecot plugin) software to provide rapid (sub-second) searches through tens-of-GB-scale mailboxes.
Unlike what I was led to believe, the squat indexes worked surprisingly well, once you sorted out the odd resource size (ulimit-related) issues (vsz & friends) limitations. I did notice the "worst-case" search performance have worryingly high O(x) increases in time, but I'd not seen anything that was a dealbreaker. It goes without saying that various substring searches worked as expected, for the most part.
My experiences with SOLR were similar to Messr. Moreau's: lots of startup errors with provided schemata files. Lots of JAVA nonsense issues. Lots of sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM, version of SOLR, and dovecot to find the "best" working combination, only to find that the searches didn't work out as expected. I expected to be able to do date-ranging based searches. Didn't work. I expected to search CONTENTS of emails, and despite many days of tweaks, I couldn't get it to index even the basics like filenames/types of attachments, so I could exposed attachment-based searching to my users.
So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality:
The ability to search for a string within any of the structured fields (from/subject) that returns correct results?
The ability to search for any string within the BODY of emails, including the MIME attachment boundaries?
The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data?
OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines:
- The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika?
SOLR seems to have "brand cachet", so presumably it actually works (for somebody).
Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I am no stranger to the "costs" of "free" software, in that you sacrifice your own blood, sweat, and tears just to get these disparate pieces to work together.
I *DO* respect that Timo has to keep the lights (and sauna) on in Finland. Maybe there's a super-secret (no advertised prices, "carrier-only" price list) with _Dovecot, Oy_ wherein the above ARE actually available for something less than 6.022 x 10^23 Euros per centi-second of licencing fees.
But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?)
I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have.
He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them.
Respectfully, =M= Fascinating...
SOLR says the following are powered by SOLR...
https://wiki.apache.org/solr/PublicServers
Perhaps if you could find out from that list which of them are using SOLR in conjunction with Dovecot...
food for thought...
I hope you are aware that "linking with Xapian" requires somewhat more work than just -lxapian in linker? If you or someone feels like writing fts_xapian, go for it.
Aki
On 04 January 2019 at 08:20 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
What about consedering linking Dovecot with Xapian librairies instead of going to nightmare Solr ?
On 2019-01-02 17:10, John Tulp wrote:
On Wed, 2019-01-02 at 00:59 -0800, M. Balridge wrote: The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors I've been watching the progress of this thread with no small concern, mainly because I've been tasked with providing a server-side email search facility with a budget and manpower level that comes down to mainly *1*, i.e., me.
I was expecting, given the strongly worded language about "just use lucene/SOLR" and "ignore squat", that I should invest time + effort into this JAVA nightmare that is SOLR.
I started with squat and another word-indexor system that used out-of-band (not a dovecot plugin) software to provide rapid (sub-second) searches through tens-of-GB-scale mailboxes.
Unlike what I was led to believe, the squat indexes worked surprisingly well, once you sorted out the odd resource size (ulimit-related) issues (vsz & friends) limitations. I did notice the "worst-case" search performance have worryingly high O(x) increases in time, but I'd not seen anything that was a dealbreaker. It goes without saying that various substring searches worked as expected, for the most part.
My experiences with SOLR were similar to Messr. Moreau's: lots of startup errors with provided schemata files. Lots of JAVA nonsense issues. Lots of sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM, version of SOLR, and dovecot to find the "best" working combination, only to find that the searches didn't work out as expected. I expected to be able to do date-ranging based searches. Didn't work. I expected to search CONTENTS of emails, and despite many days of tweaks, I couldn't get it to index even the basics like filenames/types of attachments, so I could exposed attachment-based searching to my users.
So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality:
The ability to search for a string within any of the structured fields (from/subject) that returns correct results?
The ability to search for any string within the BODY of emails, including the MIME attachment boundaries?
The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data?
OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines:
- The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika?
SOLR seems to have "brand cachet", so presumably it actually works (for somebody).
Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I am no stranger to the "costs" of "free" software, in that you sacrifice your own blood, sweat, and tears just to get these disparate pieces to work together.
I *DO* respect that Timo has to keep the lights (and sauna) on in Finland. Maybe there's a super-secret (no advertised prices, "carrier-only" price list) with _Dovecot, Oy_ wherein the above ARE actually available for something less than 6.022 x 10^23 Euros per centi-second of licencing fees.
But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?)
I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have.
He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them.
Respectfully, =M= Fascinating...
SOLR says the following are powered by SOLR...
https://wiki.apache.org/solr/PublicServers
Perhaps if you could find out from that list which of them are using SOLR in conjunction with Dovecot...
food for thought...
Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
On 2019-01-04 17:20, Aki Tuomi wrote:
I hope you are aware that "linking with Xapian" requires somewhat more work than just -lxapian in linker? If you or someone feels like writing fts_xapian, go for it.
Aki
On 04 January 2019 at 08:20 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
What about consedering linking Dovecot with Xapian librairies instead of going to nightmare Solr ?
On 2019-01-02 17:10, John Tulp wrote:
On Wed, 2019-01-02 at 00:59 -0800, M. Balridge wrote: The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors I've been watching the progress of this thread with no small concern, mainly because I've been tasked with providing a server-side email search facility with a budget and manpower level that comes down to mainly *1*, i.e., me.
I was expecting, given the strongly worded language about "just use lucene/SOLR" and "ignore squat", that I should invest time + effort into this JAVA nightmare that is SOLR.
I started with squat and another word-indexor system that used out-of-band (not a dovecot plugin) software to provide rapid (sub-second) searches through tens-of-GB-scale mailboxes.
Unlike what I was led to believe, the squat indexes worked surprisingly well, once you sorted out the odd resource size (ulimit-related) issues (vsz & friends) limitations. I did notice the "worst-case" search performance have worryingly high O(x) increases in time, but I'd not seen anything that was a dealbreaker. It goes without saying that various substring searches worked as expected, for the most part.
My experiences with SOLR were similar to Messr. Moreau's: lots of startup errors with provided schemata files. Lots of JAVA nonsense issues. Lots of sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM, version of SOLR, and dovecot to find the "best" working combination, only to find that the searches didn't work out as expected. I expected to be able to do date-ranging based searches. Didn't work. I expected to search CONTENTS of emails, and despite many days of tweaks, I couldn't get it to index even the basics like filenames/types of attachments, so I could exposed attachment-based searching to my users.
So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality:
The ability to search for a string within any of the structured fields (from/subject) that returns correct results?
The ability to search for any string within the BODY of emails, including the MIME attachment boundaries?
The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data?
OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines:
- The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika?
SOLR seems to have "brand cachet", so presumably it actually works (for somebody).
Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I am no stranger to the "costs" of "free" software, in that you sacrifice your own blood, sweat, and tears just to get these disparate pieces to work together.
I *DO* respect that Timo has to keep the lights (and sauna) on in Finland. Maybe there's a super-secret (no advertised prices, "carrier-only" price list) with _Dovecot, Oy_ wherein the above ARE actually available for something less than 6.022 x 10^23 Euros per centi-second of licencing fees.
But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?)
I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have.
He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them.
Respectfully, =M= Fascinating...
SOLR says the following are powered by SOLR...
https://wiki.apache.org/solr/PublicServers
Perhaps if you could find out from that list which of them are using SOLR in conjunction with Dovecot...
food for thought...
A starting point would be to have a look at the current FTS plugins:
https://github.com/dovecot/core/tree/master/src/plugins/fts-solrandhttps://g... -M
Am Freitag, den 04.01.2019, 18:17 +0800 schrieb Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
On 2019-01-04 17:20, Aki Tuomi wrote:
I hope you are aware that "linking with Xapian" requires somewhat more work than just -lxapian in linker? If you or someone feels like writing fts_xapian, go for it.
Aki
On 04 January 2019 at 08:20 Joan Moreau via dovecot < dovecot@dovecot.org> wrote:
What about consedering linking Dovecot with Xapian librairies instead of going to nightmare Solr ?
On 2019-01-02 17:10, John Tulp wrote:
On Wed, 2019-01-02 at 00:59 -0800, M. Balridge wrote: The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors I've been watching the progress of this thread with no small concern, mainly because I've been tasked with providing a server-side email search facility with a budget and manpower level that comes down to mainly *1*, i.e., me.
I was expecting, given the strongly worded language about "just use lucene/SOLR" and "ignore squat", that I should invest time + effort into this JAVA nightmare that is SOLR.
I started with squat and another word-indexor system that used out-of-band (not a dovecot plugin) software to provide rapid (sub-second) searches through tens-of-GB-scale mailboxes.
Unlike what I was led to believe, the squat indexes worked surprisingly well, once you sorted out the odd resource size (ulimit-related) issues (vsz & friends) limitations. I did notice the "worst-case" search performance have worryingly high O(x) increases in time, but I'd not seen anything that was a dealbreaker. It goes without saying that various substring searches worked as expected, for the most part.
My experiences with SOLR were similar to Messr. Moreau's: lots of startup errors with provided schemata files. Lots of JAVA nonsense issues. Lots of sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM, version of SOLR, and dovecot to find the "best" working combination, only to find that the searches didn't work out as expected. I expected to be able to do date-ranging based searches. Didn't work. I expected to search CONTENTS of emails, and despite many days of tweaks, I couldn't get it to index even the basics like filenames/types of attachments, so I could exposed attachment-based searching to my users.
So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality:
The ability to search for a string within any of the structured fields (from/subject) that returns correct results?
The ability to search for any string within the BODY of emails, including the MIME attachment boundaries?
The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data?
OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines:
- The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika?
SOLR seems to have "brand cachet", so presumably it actually works (for somebody).
Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I am no stranger to the "costs" of "free" software, in that you sacrifice your own blood, sweat, and tears just to get these disparate pieces to work together.
I *DO* respect that Timo has to keep the lights (and sauna) on in Finland. Maybe there's a super-secret (no advertised prices, "carrier- only" price list) with _Dovecot, Oy_ wherein the above ARE actually available for something less than 6.022 x 10^23 Euros per centi-second of licencing fees.
But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?)
I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have.
He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them.
Respectfully, =M=
Fascinating...
SOLR says the following are powered by SOLR...
https://wiki.apache.org/solr/PublicServers
Perhaps if you could find out from that list which of them are using SOLR in conjunction with Dovecot...
food for thought...
Yes but:
1 - is there a documentation of the main object ? (fts_backend, mail_user, mailbox, etc..)
2 - What are the mandatory functions ?
3 - Search : Supposedly, the FTS shall have several parameters : the keyword(s), the user & mailbox, and the fields (to, from, body, etc..) to be includude in the search. What is the function called in the plugin ?
4 - Indexing : Somehow, what is the logic ? fts core just ask to "index me this email of this mailbox" ? or this is delegated to the plugin to sort out which emails it has indexed yet or not ?
Thank you
On 2019-01-04 18:49, admin wrote:
A starting point would be to have a look at the current FTS plugins:
https://github.com/dovecot/core/tree/master/src/plugins/fts-solr and https://github.com/dovecot/core/tree/master/src/plugins/fts-squat
-M
Am Freitag, den 04.01.2019, 18:17 +0800 schrieb Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
On 2019-01-04 17:20, Aki Tuomi wrote: I hope you are aware that "linking with Xapian" requires somewhat more work than just -lxapian in linker? If you or someone feels like writing fts_xapian, go for it.
Aki
On 04 January 2019 at 08:20 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
What about consedering linking Dovecot with Xapian librairies instead of going to nightmare Solr ?
On 2019-01-02 17:10, John Tulp wrote:
On Wed, 2019-01-02 at 00:59 -0800, M. Balridge wrote: The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors I've been watching the progress of this thread with no small concern, mainly because I've been tasked with providing a server-side email search facility with a budget and manpower level that comes down to mainly *1*, i.e., me.
I was expecting, given the strongly worded language about "just use lucene/SOLR" and "ignore squat", that I should invest time + effort into this JAVA nightmare that is SOLR.
I started with squat and another word-indexor system that used out-of-band (not a dovecot plugin) software to provide rapid (sub-second) searches through tens-of-GB-scale mailboxes.
Unlike what I was led to believe, the squat indexes worked surprisingly well, once you sorted out the odd resource size (ulimit-related) issues (vsz & friends) limitations. I did notice the "worst-case" search performance have worryingly high O(x) increases in time, but I'd not seen anything that was a dealbreaker. It goes without saying that various substring searches worked as expected, for the most part.
My experiences with SOLR were similar to Messr. Moreau's: lots of startup errors with provided schemata files. Lots of JAVA nonsense issues. Lots of sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM, version of SOLR, and dovecot to find the "best" working combination, only to find that the searches didn't work out as expected. I expected to be able to do date-ranging based searches. Didn't work. I expected to search CONTENTS of emails, and despite many days of tweaks, I couldn't get it to index even the basics like filenames/types of attachments, so I could exposed attachment-based searching to my users.
So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality:
The ability to search for a string within any of the structured fields (from/subject) that returns correct results?
The ability to search for any string within the BODY of emails, including the MIME attachment boundaries?
The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data?
OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines:
- The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika?
SOLR seems to have "brand cachet", so presumably it actually works (for somebody).
Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I am no stranger to the "costs" of "free" software, in that you sacrifice your own blood, sweat, and tears just to get these disparate pieces to work together.
I *DO* respect that Timo has to keep the lights (and sauna) on in Finland. Maybe there's a super-secret (no advertised prices, "carrier-only" price list) with _Dovecot, Oy_ wherein the above ARE actually available for something less than 6.022 x 10^23 Euros per centi-second of licencing fees.
But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?)
I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have.
He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them.
Respectfully, =M= Fascinating...
SOLR says the following are powered by SOLR...
https://wiki.apache.org/solr/PublicServers
Perhaps if you could find out from that list which of them are using SOLR in conjunction with Dovecot...
food for thought...
Also, a description of the "to be" functions of the backend:
struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT, -> WHAT OTHER FLAGS ?
{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };
On 2019-01-04 20:33, Joan Moreau via dovecot wrote:
Yes but:
1 - is there a documentation of the main object ? (fts_backend, mail_user, mailbox, etc..)
2 - What are the mandatory functions ?
3 - Search : Supposedly, the FTS shall have several parameters : the keyword(s), the user & mailbox, and the fields (to, from, body, etc..) to be includude in the search. What is the function called in the plugin ?
4 - Indexing : Somehow, what is the logic ? fts core just ask to "index me this email of this mailbox" ? or this is delegated to the plugin to sort out which emails it has indexed yet or not ?
Thank you
On 2019-01-04 18:49, admin wrote: A starting point would be to have a look at the current FTS plugins:
https://github.com/dovecot/core/tree/master/src/plugins/fts-solr and https://github.com/dovecot/core/tree/master/src/plugins/fts-squat
-M
Am Freitag, den 04.01.2019, 18:17 +0800 schrieb Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
On 2019-01-04 17:20, Aki Tuomi wrote: I hope you are aware that "linking with Xapian" requires somewhat more work than just -lxapian in linker? If you or someone feels like writing fts_xapian, go for it.
Aki
On 04 January 2019 at 08:20 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
What about consedering linking Dovecot with Xapian librairies instead of going to nightmare Solr ?
On 2019-01-02 17:10, John Tulp wrote:
On Wed, 2019-01-02 at 00:59 -0800, M. Balridge wrote: The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors I've been watching the progress of this thread with no small concern, mainly because I've been tasked with providing a server-side email search facility with a budget and manpower level that comes down to mainly *1*, i.e., me.
I was expecting, given the strongly worded language about "just use lucene/SOLR" and "ignore squat", that I should invest time + effort into this JAVA nightmare that is SOLR.
I started with squat and another word-indexor system that used out-of-band (not a dovecot plugin) software to provide rapid (sub-second) searches through tens-of-GB-scale mailboxes.
Unlike what I was led to believe, the squat indexes worked surprisingly well, once you sorted out the odd resource size (ulimit-related) issues (vsz & friends) limitations. I did notice the "worst-case" search performance have worryingly high O(x) increases in time, but I'd not seen anything that was a dealbreaker. It goes without saying that various substring searches worked as expected, for the most part.
My experiences with SOLR were similar to Messr. Moreau's: lots of startup errors with provided schemata files. Lots of JAVA nonsense issues. Lots of sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM, version of SOLR, and dovecot to find the "best" working combination, only to find that the searches didn't work out as expected. I expected to be able to do date-ranging based searches. Didn't work. I expected to search CONTENTS of emails, and despite many days of tweaks, I couldn't get it to index even the basics like filenames/types of attachments, so I could exposed attachment-based searching to my users.
So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality:
The ability to search for a string within any of the structured fields (from/subject) that returns correct results?
The ability to search for any string within the BODY of emails, including the MIME attachment boundaries?
The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data?
OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines:
- The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika?
SOLR seems to have "brand cachet", so presumably it actually works (for somebody).
Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I am no stranger to the "costs" of "free" software, in that you sacrifice your own blood, sweat, and tears just to get these disparate pieces to work together.
I *DO* respect that Timo has to keep the lights (and sauna) on in Finland. Maybe there's a super-secret (no advertised prices, "carrier-only" price list) with _Dovecot, Oy_ wherein the above ARE actually available for something less than 6.022 x 10^23 Euros per centi-second of licencing fees.
But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?)
I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have.
He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them.
Respectfully, =M= Fascinating...
SOLR says the following are powered by SOLR...
https://wiki.apache.org/solr/PublicServers
Perhaps if you could find out from that list which of them are using SOLR in conjunction with Dovecot...
food for thought...
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).
Regards,
Stephan.
Thank Stephan
I basically need to know the role/description of each of the functions of the fts_backend:
struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT, -> WHAT OTHER FLAGS ?
{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).
Regards,
Stephan.
Anyone willing to explain those functions ?
On January 5, 2019 14:23:10 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Thank Stephan I basically need to know the role/description of each of the functions of the fts_backend:
struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT, -> what other flags ? { fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };
THank you On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).
Regards,
Stephan.
Anyone willing to explain those functions ?
Most notably " get_last_uid" "set_build_key" "build_more" , what is refresh versus rescan ?
On January 5, 2019 14:23:10 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Thank Stephan I basically need to know the role/description of each of the functions of the fts_backend:
struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT, -> what other flags ? { fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };
THank you On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).
Regards,
Stephan.
Op 06/01/2019 om 01:00 schreef Joan Moreau:
Anyone willing to explain those functions ?
Most notably " get_last_uid"
From src/plugins/fts/fts-api.h:
/* Get the last_uid for the mailbox. */ int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box, uint32_t *last_uid_r);
The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this returns the last UID added to the index for the given mailbox and FTS index.
"set_build_key"
From src/plugins/fts/fts-api.h:
/* Switch to building index for specified key. If backend doesn't want to index this key, it can return FALSE and caller will skip to next key. */ bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx, const struct fts_backend_build_key *key);
Same file provides outline of what a build_key is.
"build_more" ,
/* Add more content to the index for the currently specified build key. Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters, but it doesn't need to be NUL-terminated. size contains the data size in bytes, not characters. This function may be called many times and the data block sizes may be small. Backend returns 0 if ok, -1 if build should be aborted. */ int fts_backend_update_build_more(struct fts_backend_update_context *ctx, const unsigned char *data, size_t size);
You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing.
what is refresh versus rescan ?
From fts-api.h:
/* Refresh index to make sure we see latest changes from lookups. Returns 0 if ok, -1 if error. */ int fts_backend_refresh(struct fts_backend *backend); /* Go through the entire index and make sure all mails are indexed, and delete any extra mails in the index. */ int fts_backend_rescan(struct fts_backend *backend);
Regards,
Stepham
On January 5, 2019 14:23:10 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of the fts_backend:
struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).
Regards,
Stephan.
Thank you
I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?
On 2019-01-06 08:43, Stephan Bosch wrote:
Op 06/01/2019 om 01:00 schreef Joan Moreau:
Anyone willing to explain those functions ?
Most notably " get_last_uid"
From src/plugins/fts/fts-api.h:
/* Get the last_uid for the mailbox. */ int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box, uint32_t *last_uid_r);
The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this returns the last UID added to the index for the given mailbox and FTS index.
"set_build_key"
From src/plugins/fts/fts-api.h:
/* Switch to building index for specified key. If backend doesn't want to index this key, it can return FALSE and caller will skip to next key. */ bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx, const struct fts_backend_build_key *key);
Same file provides outline of what a build_key is.
"build_more" ,
/* Add more content to the index for the currently specified build key. Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters, but it doesn't need to be NUL-terminated. size contains the data size in bytes, not characters. This function may be called many times and the data block sizes may be small. Backend returns 0 if ok, -1 if build should be aborted. */ int fts_backend_update_build_more(struct fts_backend_update_context *ctx, const unsigned char *data, size_t size);
You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing.
what is refresh versus rescan ?
From fts-api.h:
/* Refresh index to make sure we see latest changes from lookups. Returns 0 if ok, -1 if error. */ int fts_backend_refresh(struct fts_backend *backend); /* Go through the entire index and make sure all mails are indexed, and delete any extra mails in the index. */ int fts_backend_rescan(struct fts_backend *backend);
Regards,
Stepham
On January 5, 2019 14:23:10 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of the fts_backend:
struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot: Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).
Regards,
Stephan.
for the "last uid"-> this is not the last added, but the maximum of the UID in the indexed emails, right ?
On 2019-01-06 11:53, Joan Moreau via dovecot wrote:
Thank you
I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?
On 2019-01-06 08:43, Stephan Bosch wrote:
Op 06/01/2019 om 01:00 schreef Joan Moreau: Anyone willing to explain those functions ?
Most notably " get_last_uid" From src/plugins/fts/fts-api.h:
/* Get the last_uid for the mailbox. */ int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box, uint32_t *last_uid_r);
The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this returns the last UID added to the index for the given mailbox and FTS index.
"set_build_key" From src/plugins/fts/fts-api.h:
/* Switch to building index for specified key. If backend doesn't want to index this key, it can return FALSE and caller will skip to next key. */ bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx, const struct fts_backend_build_key *key);
Same file provides outline of what a build_key is.
"build_more" , /* Add more content to the index for the currently specified build key. Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters, but it doesn't need to be NUL-terminated. size contains the data size in bytes, not characters. This function may be called many times and the data block sizes may be small. Backend returns 0 if ok, -1 if build should be aborted. */ int fts_backend_update_build_more(struct fts_backend_update_context *ctx, const unsigned char *data, size_t size);
You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing.
what is refresh versus rescan ? From fts-api.h:
/* Refresh index to make sure we see latest changes from lookups. Returns 0 if ok, -1 if error. */ int fts_backend_refresh(struct fts_backend *backend); /* Go through the entire index and make sure all mails are indexed, and delete any extra mails in the index. */ int fts_backend_rescan(struct fts_backend *backend);
Regards,
Stepham
On January 5, 2019 14:23:10 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of the fts_backend:
struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot: Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).
Regards,
Stephan.
also, for fts_backend_solr_update_set_build_key -> where is the data (of the hdr_name or the body) ?
On 2019-01-06 14:10, Joan Moreau wrote:
for the "last uid"-> this is not the last added, but the maximum of the UID in the indexed emails, right ?
On 2019-01-06 11:53, Joan Moreau via dovecot wrote:
Thank you
I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?
On 2019-01-06 08:43, Stephan Bosch wrote:
Op 06/01/2019 om 01:00 schreef Joan Moreau: Anyone willing to explain those functions ?
Most notably " get_last_uid" From src/plugins/fts/fts-api.h:
/* Get the last_uid for the mailbox. */ int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box, uint32_t *last_uid_r);
The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this returns the last UID added to the index for the given mailbox and FTS index.
"set_build_key" From src/plugins/fts/fts-api.h:
/* Switch to building index for specified key. If backend doesn't want to index this key, it can return FALSE and caller will skip to next key. */ bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx, const struct fts_backend_build_key *key);
Same file provides outline of what a build_key is.
"build_more" , /* Add more content to the index for the currently specified build key. Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters, but it doesn't need to be NUL-terminated. size contains the data size in bytes, not characters. This function may be called many times and the data block sizes may be small. Backend returns 0 if ok, -1 if build should be aborted. */ int fts_backend_update_build_more(struct fts_backend_update_context *ctx, const unsigned char *data, size_t size);
You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing.
what is refresh versus rescan ? From fts-api.h:
/* Refresh index to make sure we see latest changes from lookups. Returns 0 if ok, -1 if error. */ int fts_backend_refresh(struct fts_backend *backend); /* Go through the entire index and make sure all mails are indexed, and delete any extra mails in the index. */ int fts_backend_rescan(struct fts_backend *backend);
Regards,
Stepham
On January 5, 2019 14:23:10 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of the fts_backend:
struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot: Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).
Regards,
Stephan.
For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ???
There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ?
On 2019-01-06 15:41, Joan Moreau wrote:
also, for fts_backend_solr_update_set_build_key -> where is the data (of the hdr_name or the body) ?
On 2019-01-06 14:10, Joan Moreau wrote:
for the "last uid"-> this is not the last added, but the maximum of the UID in the indexed emails, right ?
On 2019-01-06 11:53, Joan Moreau via dovecot wrote:
Thank you
I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?
On 2019-01-06 08:43, Stephan Bosch wrote:
Op 06/01/2019 om 01:00 schreef Joan Moreau: Anyone willing to explain those functions ?
Most notably " get_last_uid" From src/plugins/fts/fts-api.h:
/* Get the last_uid for the mailbox. */ int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box, uint32_t *last_uid_r);
The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this returns the last UID added to the index for the given mailbox and FTS index.
"set_build_key" From src/plugins/fts/fts-api.h:
/* Switch to building index for specified key. If backend doesn't want to index this key, it can return FALSE and caller will skip to next key. */ bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx, const struct fts_backend_build_key *key);
Same file provides outline of what a build_key is.
"build_more" , /* Add more content to the index for the currently specified build key. Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters, but it doesn't need to be NUL-terminated. size contains the data size in bytes, not characters. This function may be called many times and the data block sizes may be small. Backend returns 0 if ok, -1 if build should be aborted. */ int fts_backend_update_build_more(struct fts_backend_update_context *ctx, const unsigned char *data, size_t size);
You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing.
what is refresh versus rescan ? From fts-api.h:
/* Refresh index to make sure we see latest changes from lookups. Returns 0 if ok, -1 if error. */ int fts_backend_refresh(struct fts_backend *backend); /* Go through the entire index and make sure all mails are indexed, and delete any extra mails in the index. */ int fts_backend_rescan(struct fts_backend *backend);
Regards,
Stepham
On January 5, 2019 14:23:10 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of the fts_backend:
struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot: Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).
Regards,
Stephan.
for fts_backend_xxx_lookup, where is specidifed in which field (to, cc, subject, body, from, all) to lookup ?
On 2019-01-06 16:03, Joan Moreau wrote:
For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ???
There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ?
On 2019-01-06 15:41, Joan Moreau wrote:
also, for fts_backend_solr_update_set_build_key -> where is the data (of the hdr_name or the body) ?
On 2019-01-06 14:10, Joan Moreau wrote:
for the "last uid"-> this is not the last added, but the maximum of the UID in the indexed emails, right ?
On 2019-01-06 11:53, Joan Moreau via dovecot wrote:
Thank you
I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?
On 2019-01-06 08:43, Stephan Bosch wrote:
Op 06/01/2019 om 01:00 schreef Joan Moreau: Anyone willing to explain those functions ?
Most notably " get_last_uid" From src/plugins/fts/fts-api.h:
/* Get the last_uid for the mailbox. */ int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box, uint32_t *last_uid_r);
The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this returns the last UID added to the index for the given mailbox and FTS index.
"set_build_key" From src/plugins/fts/fts-api.h:
/* Switch to building index for specified key. If backend doesn't want to index this key, it can return FALSE and caller will skip to next key. */ bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx, const struct fts_backend_build_key *key);
Same file provides outline of what a build_key is.
"build_more" , /* Add more content to the index for the currently specified build key. Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters, but it doesn't need to be NUL-terminated. size contains the data size in bytes, not characters. This function may be called many times and the data block sizes may be small. Backend returns 0 if ok, -1 if build should be aborted. */ int fts_backend_update_build_more(struct fts_backend_update_context *ctx, const unsigned char *data, size_t size);
You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing.
what is refresh versus rescan ? From fts-api.h:
/* Refresh index to make sure we see latest changes from lookups. Returns 0 if ok, -1 if error. */ int fts_backend_refresh(struct fts_backend *backend); /* Go through the entire index and make sure all mails are indexed, and delete any extra mails in the index. */ int fts_backend_rescan(struct fts_backend *backend);
Regards,
Stepham
On January 5, 2019 14:23:10 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of the fts_backend:
struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot: Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).
Regards,
Stephan.
and finally , for fts_backend_xxxx_lookup_multi, why is that backend dependent ?
Would- nt the below function below be the same for any backend ?
Waiting fro your feedback on all those questions
Thank you
JM
static int fts_backend_xapian_lookup_multi(struct fts_backend *_backend, struct mailbox *const boxes[], struct mail_search_arg *args, enum fts_lookup_flags flags, struct fts_multi_result *result) { struct xapian_fts_backend_update_context *ctx = (struct xapian_fts_backend_update_context *)_ctx;
int i=0;
while(boxes[i]!=NULL) { if(fts_backend_xapian_lookup(backend,box[i],args,flags,result->box_results[i])<0) return -1; i++; } return 0; }
On 2019-01-06 16:31, Joan Moreau via dovecot wrote:
for fts_backend_xxx_lookup, where is specidifed in which field (to, cc, subject, body, from, all) to lookup ?
On 2019-01-06 16:03, Joan Moreau wrote:
For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ???
There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ?
On 2019-01-06 15:41, Joan Moreau wrote:
also, for fts_backend_solr_update_set_build_key -> where is the data (of the hdr_name or the body) ?
On 2019-01-06 14:10, Joan Moreau wrote:
for the "last uid"-> this is not the last added, but the maximum of the UID in the indexed emails, right ?
On 2019-01-06 11:53, Joan Moreau via dovecot wrote:
Thank you
I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?
On 2019-01-06 08:43, Stephan Bosch wrote:
Op 06/01/2019 om 01:00 schreef Joan Moreau: Anyone willing to explain those functions ?
Most notably " get_last_uid" From src/plugins/fts/fts-api.h:
/* Get the last_uid for the mailbox. */ int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box, uint32_t *last_uid_r);
The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this returns the last UID added to the index for the given mailbox and FTS index.
"set_build_key" From src/plugins/fts/fts-api.h:
/* Switch to building index for specified key. If backend doesn't want to index this key, it can return FALSE and caller will skip to next key. */ bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx, const struct fts_backend_build_key *key);
Same file provides outline of what a build_key is.
"build_more" , /* Add more content to the index for the currently specified build key. Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters, but it doesn't need to be NUL-terminated. size contains the data size in bytes, not characters. This function may be called many times and the data block sizes may be small. Backend returns 0 if ok, -1 if build should be aborted. */ int fts_backend_update_build_more(struct fts_backend_update_context *ctx, const unsigned char *data, size_t size);
You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing.
what is refresh versus rescan ? From fts-api.h:
/* Refresh index to make sure we see latest changes from lookups. Returns 0 if ok, -1 if error. */ int fts_backend_refresh(struct fts_backend *backend); /* Go through the entire index and make sure all mails are indexed, and delete any extra mails in the index. */ int fts_backend_rescan(struct fts_backend *backend);
Regards,
Stepham
On January 5, 2019 14:23:10 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of the fts_backend:
struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot: Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).
Regards,
Stephan.
Hi
ANyone to answer specifically ?
Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance)
Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ?
Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ?
Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ?
Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ?
Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ?
THank you
On 2019-01-06 16:50, Joan Moreau via dovecot wrote:
and finally , for fts_backend_xxxx_lookup_multi, why is that backend dependent ?
Would- nt the below function below be the same for any backend ?
Waiting fro your feedback on all those questions
Thank you
JM
static int fts_backend_xapian_lookup_multi(struct fts_backend *_backend, struct mailbox *const boxes[], struct mail_search_arg *args, enum fts_lookup_flags flags, struct fts_multi_result *result) { struct xapian_fts_backend_update_context *ctx = (struct xapian_fts_backend_update_context *)_ctx;
int i=0;
while(boxes[i]!=NULL) { if(fts_backend_xapian_lookup(backend,box[i],args,flags,result->box_results[i])<0) return -1; i++; } return 0; }
On 2019-01-06 16:31, Joan Moreau via dovecot wrote:
for fts_backend_xxx_lookup, where is specidifed in which field (to, cc, subject, body, from, all) to lookup ?
On 2019-01-06 16:03, Joan Moreau wrote:
For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ???
There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ?
On 2019-01-06 15:41, Joan Moreau wrote:
also, for fts_backend_solr_update_set_build_key -> where is the data (of the hdr_name or the body) ?
On 2019-01-06 14:10, Joan Moreau wrote:
for the "last uid"-> this is not the last added, but the maximum of the UID in the indexed emails, right ?
On 2019-01-06 11:53, Joan Moreau via dovecot wrote:
Thank you
I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?
On 2019-01-06 08:43, Stephan Bosch wrote:
Op 06/01/2019 om 01:00 schreef Joan Moreau: Anyone willing to explain those functions ?
Most notably " get_last_uid" From src/plugins/fts/fts-api.h:
/* Get the last_uid for the mailbox. */ int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box, uint32_t *last_uid_r);
The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this returns the last UID added to the index for the given mailbox and FTS index.
"set_build_key" From src/plugins/fts/fts-api.h:
/* Switch to building index for specified key. If backend doesn't want to index this key, it can return FALSE and caller will skip to next key. */ bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx, const struct fts_backend_build_key *key);
Same file provides outline of what a build_key is.
"build_more" , /* Add more content to the index for the currently specified build key. Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters, but it doesn't need to be NUL-terminated. size contains the data size in bytes, not characters. This function may be called many times and the data block sizes may be small. Backend returns 0 if ok, -1 if build should be aborted. */ int fts_backend_update_build_more(struct fts_backend_update_context *ctx, const unsigned char *data, size_t size);
You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing.
what is refresh versus rescan ? From fts-api.h:
/* Refresh index to make sure we see latest changes from lookups. Returns 0 if ok, -1 if error. */ int fts_backend_refresh(struct fts_backend *backend); /* Go through the entire index and make sure all mails are indexed, and delete any extra mails in the index. */ int fts_backend_rescan(struct fts_backend *backend);
Regards,
Stepham
On January 5, 2019 14:23:10 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of the fts_backend:
struct fts_backend fts_backend_xapian = { .name = "xapian", .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
{ fts_backend_xapian_alloc, fts_backend_xapian_init, fts_backend_xapian_deinit, fts_backend_xapian_get_last_uid, fts_backend_xapian_update_init, fts_backend_xapian_update_deinit, fts_backend_xapian_update_set_mailbox, fts_backend_xapian_update_expunge, fts_backend_xapian_update_set_build_key, fts_backend_xapian_update_unset_build_key, fts_backend_xapian_update_build_more, fts_backend_xapian_refresh, fts_backend_xapian_rescan, fts_backend_xapian_optimize, fts_backend_default_can_lookup, fts_backend_xapian_lookup, fts_backend_xapian_lookup_multi, fts_backend_xapian_lookup_done } };
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot: Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).
Regards,
Stephan.
Maybe a dumb question (I admit I haven't followed this thread very closely)...
But why are you writing a new FTS driver? If squat allegedly does everything you need it to do, why don't you just take that plugin and fix it up to do what you need? That seems way easier than trying to create a FTS driver from scratch.
michael
On January 7, 2019 at 7:05 AM Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Hi ANyone to answer specifically ? Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance) Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ? Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ? Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ? Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ? Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ?
because fts_squat is set to be deleted
Xapian and similar libraries offers a very easy interface for FTS
(and basically, I have done it already)
On 2019-01-07 18:31, Michael Slusarz wrote:
Maybe a dumb question (I admit I haven't followed this thread very closely)...
But why are you writing a new FTS driver? If squat allegedly does everything you need it to do, why don't you just take that plugin and fix it up to do what you need? That seems way easier than trying to create a FTS driver from scratch.
michael
On January 7, 2019 at 7:05 AM Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Hi
ANyone to answer specifically ?
Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance)
Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ?
Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ?
Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ?
Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ?
Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ?
On 7 Jan 2019, at 16.05, Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Hi
ANyone to answer specifically ?
Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance)
All the mails are always supposed to be indexed from the beginning to the last indexed mail. If there's a gap, indexer first indexes all the missing mails. So the latest UID is supposed to be the greatest UID. (Supporting out-of-order indexing would be rather difficult to keep track of.)
Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ?
The idea is that it calls something like:
- build_key(type=hdr, hdr_name=From)
- build_more("tss@iki.fi")
- build_key(type=hdr, hdr_name=Subject)
- build_more("Re: Solr -> Xapian ?")
- build_key(type=body_part)
- build_more("message body piece")
- build_more("message body piece2") ...
Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ?
lookup() gets struct mail_search_arg *args, which contains the entire IMAP SEARCH query. This could be used for more or less complex query builders.
In case of a single header search, you should have args->args->hdr_field_name contain the header name and args->args->value.str contain the content you're searching for.
Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ?
In case of Xapian it might not matter if it automatically refreshes its indexes between each query. But with some other indexes this could happen:
- IMAP session is opened
- IMAP SEARCH is run, which opens and searches the index
- a new mail is delivered to the mailbox and indexed
- IMAP SEARCH is run. Without refresh() it doesn't see the newly indexed mail and doesn't include it in the search results.
Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ?
It's run when "doveadm fts rescan" is run manually. Usually that's only run manually to fix up some brokenness. So it's intended to verify that the current mailbox contents match the FTS indexes:
- If there are any mails in FTS index that no longer exist in the actual mailbox, delete those mails from FTS
- If FTS is missing any mails in the middle of the mailbox, make sure that the next mailbox indexing will index those missing mails. I think currently this basically means reindexing all the mails since the first missing mail, even the mails that are already in the index.
fts-lucene implements this, but other FTS backends are lazy and simply rebuild all mails. Actually fts-solr is bad because it doesn't even delete the extra mails.
Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ?
and finally , for fts_backend_xxxx_lookup_multi, why is that backend dependent ?
This function is called only when searching in virtual folders. So for example the virtual "All mails" folder, which would contain all mails in all folders. In that case the boxes[] would contain a list of user's all folders, except Trash and Spam. If lookup_multi() isn't implemented (left to NULL), the search is run separately via lookup() for each folder. With lookup_multi() there can be just one lookup, and the backend can filter only the wanted folders and return them directly. So it's an optimization for FTS indexes that support user-global searches rather than only per-folder searches.
static int fts_backend_xapian_lookup_multi(struct fts_backend *_backend, struct mailbox *const boxes[], struct mail_search_arg *args, enum fts_lookup_flags flags, struct fts_multi_result *result) { struct xapian_fts_backend_update_context *ctx = (struct xapian_fts_backend_update_context *)_ctx;
int i=0;
while(boxes[i]!=NULL) { if(fts_backend_xapian_lookup(backend,box[i],args,flags,result->box_results[i])<0) return -1; i++; } return 0; }
See fts_backend_lookup_multi() - if you leave lookup_multi=NULL it basically does this.
For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ???
rescan() is about fixing up a more or less broken index, or simply to verify that it's all ok. So core doesn't know what messages exist in the FTS index and can't request specific reindexing or expunging. I guess an alternative API could have been to have functions that iterate through all mails in the index, and use that to implement rescan in core. Now thinking about it, that sounds like a simpler and better way.
optimize() is currently done only when explicitly running "doveadm fts optimize", which requests running a slower index optimization. Depends on the FTS backend whether this is useful or not.
There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ?
Normally when mails are expunged, update_expunge() is called to notify FTS backend that it should delete the mail also from FTS index.
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
You probably want to use FTS_BACKEND_FLAG_FUZZY_SEARCH only like Solr. See enum fts_backend_flags in fts-api-private.h
Ok.
Additional question :
for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)
for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?
On 2019-01-08 04:24, Timo Sirainen wrote:
On 7 Jan 2019, at 16.05, Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Hi
ANyone to answer specifically ?
Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance)
All the mails are always supposed to be indexed from the beginning to the last indexed mail. If there's a gap, indexer first indexes all the missing mails. So the latest UID is supposed to be the greatest UID. (Supporting out-of-order indexing would be rather difficult to keep track of.)
Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ?
The idea is that it calls something like:
- build_key(type=hdr, hdr_name=From)
- build_more("tss@iki.fi")
- build_key(type=hdr, hdr_name=Subject)
- build_more("Re: Solr -> Xapian ?")
- build_key(type=body_part)
- build_more("message body piece")
- build_more("message body piece2") ...
Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ?
lookup() gets struct mail_search_arg *args, which contains the entire IMAP SEARCH query. This could be used for more or less complex query builders.
In case of a single header search, you should have args->args->hdr_field_name contain the header name and args->args->value.str contain the content you're searching for.
Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ?
In case of Xapian it might not matter if it automatically refreshes its indexes between each query. But with some other indexes this could happen:
- IMAP session is opened
- IMAP SEARCH is run, which opens and searches the index
- a new mail is delivered to the mailbox and indexed
- IMAP SEARCH is run. Without refresh() it doesn't see the newly indexed mail and doesn't include it in the search results.
Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ?
It's run when "doveadm fts rescan" is run manually. Usually that's only run manually to fix up some brokenness. So it's intended to verify that the current mailbox contents match the FTS indexes:
- If there are any mails in FTS index that no longer exist in the actual mailbox, delete those mails from FTS
- If FTS is missing any mails in the middle of the mailbox, make sure that the next mailbox indexing will index those missing mails. I think currently this basically means reindexing all the mails since the first missing mail, even the mails that are already in the index.
fts-lucene implements this, but other FTS backends are lazy and simply rebuild all mails. Actually fts-solr is bad because it doesn't even delete the extra mails.
Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ? and finally , for fts_backend_xxxx_lookup_multi, why is that backend dependent ?
This function is called only when searching in virtual folders. So for example the virtual "All mails" folder, which would contain all mails in all folders. In that case the boxes[] would contain a list of user's all folders, except Trash and Spam. If lookup_multi() isn't implemented (left to NULL), the search is run separately via lookup() for each folder. With lookup_multi() there can be just one lookup, and the backend can filter only the wanted folders and return them directly. So it's an optimization for FTS indexes that support user-global searches rather than only per-folder searches.
static int fts_backend_xapian_lookup_multi(struct fts_backend *_backend, struct mailbox *const boxes[], struct mail_search_arg *args, enum fts_lookup_flags flags, struct fts_multi_result *result) { struct xapian_fts_backend_update_context *ctx = (struct xapian_fts_backend_update_context *)_ctx;
int i=0;
while(boxes[i]!=NULL) { if(fts_backend_xapian_lookup(backend,box[i],args,flags,result->box_results[i])<0) return -1; i++; } return 0; }
See fts_backend_lookup_multi() - if you leave lookup_multi=NULL it basically does this.
For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ???
rescan() is about fixing up a more or less broken index, or simply to verify that it's all ok. So core doesn't know what messages exist in the FTS index and can't request specific reindexing or expunging. I guess an alternative API could have been to have functions that iterate through all mails in the index, and use that to implement rescan in core. Now thinking about it, that sounds like a simpler and better way.
optimize() is currently done only when explicitly running "doveadm fts optimize", which requests running a slower index optimization. Depends on the FTS backend whether this is useful or not.
There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ?
Normally when mails are expunged, update_expunge() is called to notify FTS backend that it should delete the mail also from FTS index.
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
You probably want to use FTS_BACKEND_FLAG_FUZZY_SEARCH only like Solr. See enum fts_backend_flags in fts-api-private.h
Also,
1 - WHat does represent "subargs" in mail_search_args
2 - I made my first code, and the error I get compiling within the dovecot architecture is
"In file included from fts-xapian-plugin.c:4: fts-xapian-plugin.h:6:1: error: unknown type name 'using'; did you mean 'uint'? using namespace std;"
if I remove this, the Xapian library is also complaining about "namespace" keyword
In file included from /usr/include/xapian.h:47, from fts-backend-xapian.c:11: /usr/include/xapian/types.h:31:1: error: unknown type name 'namespace'; did you mean 'i_isspace'? namespace Xapian {
Someone can bring me some light ?
Thanks
On 2019-01-09 09:58, Joan Moreau via dovecot wrote:
Ok.
Additional question :
for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)
for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?
On 2019-01-08 04:24, Timo Sirainen wrote: On 7 Jan 2019, at 16.05, Joan Moreau via dovecot <dovecot@dovecot.org> wrote: Hi
ANyone to answer specifically ?
Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance) All the mails are always supposed to be indexed from the beginning to the last indexed mail. If there's a gap, indexer first indexes all the missing mails. So the latest UID is supposed to be the greatest UID. (Supporting out-of-order indexing would be rather difficult to keep track of.)
Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ? The idea is that it calls something like:
- build_key(type=hdr, hdr_name=From)
- build_more("tss@iki.fi")
- build_key(type=hdr, hdr_name=Subject)
- build_more("Re: Solr -> Xapian ?")
- build_key(type=body_part)
- build_more("message body piece")
- build_more("message body piece2") ...
Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ? lookup() gets struct mail_search_arg *args, which contains the entire IMAP SEARCH query. This could be used for more or less complex query builders.
In case of a single header search, you should have args->args->hdr_field_name contain the header name and args->args->value.str contain the content you're searching for.
Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ? In case of Xapian it might not matter if it automatically refreshes its indexes between each query. But with some other indexes this could happen:
- IMAP session is opened
- IMAP SEARCH is run, which opens and searches the index
- a new mail is delivered to the mailbox and indexed
- IMAP SEARCH is run. Without refresh() it doesn't see the newly indexed mail and doesn't include it in the search results.
Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ? It's run when "doveadm fts rescan" is run manually. Usually that's only run manually to fix up some brokenness. So it's intended to verify that the current mailbox contents match the FTS indexes:
- If there are any mails in FTS index that no longer exist in the actual mailbox, delete those mails from FTS
- If FTS is missing any mails in the middle of the mailbox, make sure that the next mailbox indexing will index those missing mails. I think currently this basically means reindexing all the mails since the first missing mail, even the mails that are already in the index.
fts-lucene implements this, but other FTS backends are lazy and simply rebuild all mails. Actually fts-solr is bad because it doesn't even delete the extra mails.
Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ? and finally , for fts_backend_xxxx_lookup_multi, why is that backend dependent ?
This function is called only when searching in virtual folders. So for example the virtual "All mails" folder, which would contain all mails in all folders. In that case the boxes[] would contain a list of user's all folders, except Trash and Spam. If lookup_multi() isn't implemented (left to NULL), the search is run separately via lookup() for each folder. With lookup_multi() there can be just one lookup, and the backend can filter only the wanted folders and return them directly. So it's an optimization for FTS indexes that support user-global searches rather than only per-folder searches.
static int fts_backend_xapian_lookup_multi(struct fts_backend *_backend, struct mailbox *const boxes[], struct mail_search_arg *args, enum fts_lookup_flags flags, struct fts_multi_result *result) { struct xapian_fts_backend_update_context *ctx = (struct xapian_fts_backend_update_context *)_ctx;
int i=0;
while(boxes[i]!=NULL) { if(fts_backend_xapian_lookup(backend,box[i],args,flags,result->box_results[i])<0) return -1; i++; } return 0; }
See fts_backend_lookup_multi() - if you leave lookup_multi=NULL it basically does this.
For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ???
rescan() is about fixing up a more or less broken index, or simply to verify that it's all ok. So core doesn't know what messages exist in the FTS index and can't request specific reindexing or expunging. I guess an alternative API could have been to have functions that iterate through all mails in the index, and use that to implement rescan in core. Now thinking about it, that sounds like a simpler and better way.
optimize() is currently done only when explicitly running "doveadm fts optimize", which requests running a slower index optimization. Depends on the FTS backend whether this is useful or not.
There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ?
Normally when mails are expunged, update_expunge() is called to notify FTS backend that it should delete the mail also from FTS index.
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
You probably want to use FTS_BACKEND_FLAG_FUZZY_SEARCH only like Solr. See enum fts_backend_flags in fts-api-private.h
I managed to deal with the namespace issue (updated makefile.am)
However, I reach :
../../../src/lib/compat.h:207:19: error: conflicting declaration of 'ssize_t i_my_pread(int, void*, size_t, __off_t)' with 'C' linkage # define pread i_my_pread ^~~~~~~~~~ ../../../src/lib/compat.h:210:9: note: previous declaration with 'C++' linkage ssize_t i_my_pread(int fd, void *buf, size_t count, off_t offset); ^~~~~~~~~~ ../../../src/lib/compat.h:208:20: error: conflicting declaration of 'ssize_t i_my_pwrite(int, const void*, size_t, __off_t)' with 'C' linkage # define pwrite i_my_pwrite
Any help welcome
Hi,
I figured out the "namespace" issue
Remaining questions are :
1 - WHat does represent "subargs" in mail_search_args
2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)
3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?
4 - How to update configure.ac & additional files to add the "--with-xapian" wichi will test for libxapian presence and add it to the build ?
Thank you
On 2019-01-08 04:24, Timo Sirainen wrote:
On 7 Jan 2019, at 16.05, Joan Moreau via dovecot <dovecot@dovecot.org> wrote: Hi
ANyone to answer specifically ?
Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance) All the mails are always supposed to be indexed from the beginning to the last indexed mail. If there's a gap, indexer first indexes all the missing mails. So the latest UID is supposed to be the greatest UID. (Supporting out-of-order indexing would be rather difficult to keep track of.)
Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ? The idea is that it calls something like:
- build_key(type=hdr, hdr_name=From)
- build_more("tss@iki.fi")
- build_key(type=hdr, hdr_name=Subject)
- build_more("Re: Solr -> Xapian ?")
- build_key(type=body_part)
- build_more("message body piece")
- build_more("message body piece2") ...
Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ? lookup() gets struct mail_search_arg *args, which contains the entire IMAP SEARCH query. This could be used for more or less complex query builders.
In case of a single header search, you should have args->args->hdr_field_name contain the header name and args->args->value.str contain the content you're searching for.
Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ? In case of Xapian it might not matter if it automatically refreshes its indexes between each query. But with some other indexes this could happen:
- IMAP session is opened
- IMAP SEARCH is run, which opens and searches the index
- a new mail is delivered to the mailbox and indexed
- IMAP SEARCH is run. Without refresh() it doesn't see the newly indexed mail and doesn't include it in the search results.
Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ? It's run when "doveadm fts rescan" is run manually. Usually that's only run manually to fix up some brokenness. So it's intended to verify that the current mailbox contents match the FTS indexes: that the next mailbox indexing will index those missing mails. I think
- If there are any mails in FTS index that no longer exist in the actual mailbox, delete those mails from FTS
- If FTS is missing any mails in the middle of the mailbox, make sure
currently this basically means reindexing all the mails since the first missing mail, even the mails that are already in the index.
fts-lucene implements this, but other FTS backends are lazy and simply rebuild all mails. Actually fts-solr is bad because it doesn't even delete the extra mails.
Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ?and finally , for fts_backend_xxxx_lookup_multi, why is that backend dependent ? This function is called only when searching in virtual folders. So for example the virtual "All mails" folder, which would contain all mails in all folders. In that case the boxes[] would contain a list of user's all folders, except Trash and Spam. If lookup_multi() isn't implemented (left to NULL), the search is run separately via lookup() for each folder. With lookup_multi() there can be just one lookup, and the backend can filter only the wanted folders and return them directly. So it's an optimization for FTS indexes that support user-global searches rather than only per-folder searches.
static int fts_backend_xapian_lookup_multi(struct fts_backend *_backend, struct mailbox *const boxes[], struct mail_search_arg *args, enum fts_lookup_flags flags, struct fts_multi_result *result) { struct xapian_fts_backend_update_context *ctx = (struct xapian_fts_backend_update_context *)_ctx;
int i=0;
while(boxes[i]!=NULL) { if(fts_backend_xapian_lookup(backend,box[i],args,flags,result->box_results[i])<0) return -1; i++; } return 0; } See fts_backend_lookup_multi() - if you leave lookup_multi=NULL it basically does this.
For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ??? rescan() is about fixing up a more or less broken index, or simply to verify that it's all ok. So core doesn't know what messages exist in the FTS index and can't request specific reindexing or expunging. I guess an alternative API could have been to have functions that iterate through all mails in the index, and use that to implement rescan in core. Now thinking about it, that sounds like a simpler and better way.
optimize() is currently done only when explicitly running "doveadm fts optimize", which requests running a slower index optimization. Depends on the FTS backend whether this is useful or not.
There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ? Normally when mails are expunged, update_expunge() is called to notify FTS backend that it should delete the mail also from FTS index.
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?* You probably want to use FTS_BACKEND_FLAG_FUZZY_SEARCH only like Solr. See enum fts_backend_flags in fts-api-private.h
There is no point into a separate plugin, the purpose is to replace squat as the default fts (solr being a nightmare)
On 2019-01-11 18:23, Aki Tuomi wrote:
I would recommend making this a standalone plugin for now instead of trying to keep it in core fts.
Aki
On 11 January 2019 at 18:40 Joan Moreau via dovecot < dovecot@dovecot.org> wrote:
I managed to deal with the namespace issue (updated makefile.am)
However, I reach :
../../../src/lib/compat.h:207:19: error: conflicting declaration of 'ssize_t i_my_pread(int, void*, size_t, __off_t)' with 'C' linkage # define pread i_my_pread ^~~~~~~~~~ ../../../src/lib/compat.h:210:9: note: previous declaration with 'C++' linkage ssize_t i_my_pread(int fd, void *buf, size_t count, off_t offset); ^~~~~~~~~~ ../../../src/lib/compat.h:208:20: error: conflicting declaration of 'ssize_t i_my_pwrite(int, const void*, size_t, __off_t)' with 'C' linkage # define pwrite i_my_pwrite
Any help welcome
Hi,
I figured out the "namespace" issue
Remaining questions are :
1 - WHat does represent "subargs" in mail_search_args
2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)
3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?
4 - How to update configure.ac & additional files to add the "--with-xapian" wichi will test for libxapian presence and add it to the build ?
Thank you
On 2019-01-08 04:24, Timo Sirainen wrote:
On 7 Jan 2019, at 16.05, Joan Moreau via dovecot < dovecot@dovecot.org> wrote: Hi
ANyone to answer specifically ?
Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance) All the mails are always supposed to be indexed from the beginning to the last indexed mail. If there's a gap, indexer first indexes all the missing mails. So the latest UID is supposed to be the greatest UID. (Supporting out-of-order indexing would be rather difficult to keep track of.)
Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ? The idea is that it calls something like:
- build_key(type=hdr, hdr_name=From)
- build_more(" tss@iki.fi")
- build_key(type=hdr, hdr_name=Subject)
- build_more("Re: Solr -> Xapian ?")
- build_key(type=body_part)
- build_more("message body piece")
- build_more("message body piece2") ...
Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ? lookup() gets struct mail_search_arg *args, which contains the entire IMAP SEARCH query. This could be used for more or less complex query builders.
In case of a single header search, you should have args->args->hdr_field_name contain the header name and args->args->value.str contain the content you're searching for.
Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ? In case of Xapian it might not matter if it automatically refreshes its indexes between each query. But with some other indexes this could happen:
- IMAP session is opened
- IMAP SEARCH is run, which opens and searches the index
- a new mail is delivered to the mailbox and indexed
- IMAP SEARCH is run. Without refresh() it doesn't see the newly indexed mail and doesn't include it in the search results.
Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ? It's run when "doveadm fts rescan" is run manually. Usually that's only run manually to fix up some brokenness. So it's intended to verify that the current mailbox contents match the FTS indexes: that the next mailbox indexing will index those missing mails. I think
- If there are any mails in FTS index that no longer exist in the actual mailbox, delete those mails from FTS
- If FTS is missing any mails in the middle of the mailbox, make sure
currently this basically means reindexing all the mails since the first missing mail, even the mails that are already in the index.
fts-lucene implements this, but other FTS backends are lazy and simply rebuild all mails. Actually fts-solr is bad because it doesn't even delete the extra mails.
Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ?and finally , for fts_backend_xxxx_lookup_multi, why is that backend dependent ? This function is called only when searching in virtual folders. So for example the virtual "All mails" folder, which would contain all mails in all folders. In that case the boxes[] would contain a list of user's all folders, except Trash and Spam. If lookup_multi() isn't implemented (left to NULL), the search is run separately via lookup() for each folder. With lookup_multi() there can be just one lookup, and the backend can filter only the wanted folders and return them directly. So it's an optimization for FTS indexes that support user-global searches rather than only per-folder searches.
static int fts_backend_xapian_lookup_multi(struct fts_backend *_backend, struct mailbox *const boxes[], struct mail_search_arg *args, enum fts_lookup_flags flags, struct fts_multi_result *result) { struct xapian_fts_backend_update_context *ctx = (struct xapian_fts_backend_update_context *)_ctx;
int i=0;
while(boxes[i]!=NULL) { if(fts_backend_xapian_lookup(backend,box[i],args,flags,result->box_results[i])<0) return -1; i++; } return 0; } See fts_backend_lookup_multi() - if you leave lookup_multi=NULL it basically does this.
For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ??? rescan() is about fixing up a more or less broken index, or simply to verify that it's all ok. So core doesn't know what messages exist in the FTS index and can't request specific reindexing or expunging. I guess an alternative API could have been to have functions that iterate through all mails in the index, and use that to implement rescan in core. Now thinking about it, that sounds like a simpler and better way.
optimize() is currently done only when explicitly running "doveadm fts optimize", which requests running a slower index optimization. Depends on the FTS backend whether this is useful or not.
There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ? Normally when mails are expunged, update_expunge() is called to notify FTS backend that it should delete the mail also from FTS index.
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?* You probably want to use FTS_BACKEND_FLAG_FUZZY_SEARCH only like Solr. See enum fts_backend_flags in fts-api-private.h
Aki Tuomi
The below patch resolves the compilation error
$ DIFF -P COMPAT.H COMPAT.H.JOAN *** compat.h 2019-01-11 20:21:00.726625427 +0100 --- compat.h.joan 2019-01-11 20:14:41.729109919 +0100 *************** struct iovec; *** 202,207 **** --- 202,211 ---- ssize_t i_my_writev(int fd, const struct iovec *iov, int iov_len); #endif
- #ifdef __cplusplus
- extern "C" {
- #endif
- #if !defined(HAVE_PREAD) || defined(PREAD_WRAPPERS) || defined(PREAD_BROKEN)
ifndef IN_COMPAT_C
# define pread i_my_pread *************** ssize_t i_my_pread(int fd, void *buf, si *** 211,216 **** --- 215,225 ---- ssize_t i_my_pwrite(int fd, const void *buf, size_t count, off_t offset); #endif
- #ifdef __cplusplus
- }
- #endif
- #ifndef HAVE_SETEUID
define seteuid i_my_seteuid
int i_my_seteuid(uid_t euid);
To resolve integration in source tree, the following diff resolve the case:
$ DIFF -P CONFIGURE.AC CONFIGURE.AC.JOAN *** configure.ac 2019-01-11 20:19:47.905942264 +0100 --- configure.ac.joan 2019-01-11 17:54:58.433381828 +0100 *************** AS_HELP_STRING([--with-solr], [Build wit *** 172,177 **** --- 172,184 ---- TEST_WITH(solr, $withval), want_solr=no)
- AC_ARG_WITH(xapian,
- AS_HELP_STRING([--with-xapian], [Build with Xapian full text search support]),
- TEST_WITH(xapian, $withval),
- want_xapian=auto)
- AM_CONDITIONAL(BUILD_XAPIAN, test "$want_xapian" = "yes")
- AC_ARG_WITH(sodium, AS_HELP_STRING([--with-sodium], [Build with libsodium support (enables argon2, default: auto)]), TEST_WITH(sodium, $withval), *************** DOVECOT_WANT_SOLR *** 746,751 **** --- 753,759 ---- DOVECOT_WANT_CLUCENE DOVECOT_WANT_STEMMER DOVECOT_WANT_TEXTCAT
- DOVECOT_WANT_XAPIAN
DOVECOT_WANT_ICU
*************** fi *** 757,762 **** --- 765,774 ---- if test $have_solr = no; then not_fts="$not_fts solr" fi
if test $have_xapian = no; then
not_fts="$not_fts xapian"
fi
dnl ** dnl ** Settings *************** src/plugins/fs-compress/Makefile *** 899,904 **** --- 911,917 ---- src/plugins/fts/Makefile src/plugins/fts-lucene/Makefile src/plugins/fts-solr/Makefile
- src/plugins/fts-xapian/Makefile src/plugins/fts-squat/Makefile src/plugins/last-login/Makefile src/plugins/lazy-expunge/Makefile
$ DIFF -P MAKEFILE.AM MAKEFILE.AM.JOAN
*** Makefile.am 2019-01-11 20:22:23.910740574 +0100
--- Makefile.am.joan 2019-01-11 17:51:19.051153270 +0100
*************** DISTCLEANFILES =
*** 99,105 ****
distcheck-hook:
if which scan-build > /dev/null; then
cd $(distdir)/_build;
! scan-build -o scan-reports ../configure --with-ldap=auto
--with-pgsql=auto --with-mysql=auto --with-sqlite=auto --with-solr=auto
--with-gssapi=auto --with-libwrap=auto;
rm -rf scan-reports;
scan-build -o scan-reports make 2>&1 || exit 1;
if ! rmdir scan-reports 2>/dev/null; then
--- 99,105 ----
distcheck-hook:
if which scan-build > /dev/null; then
cd $(distdir)/_build;
! scan-build -o scan-reports ../configure --with-ldap=auto
--with-pgsql=auto --with-mysql=auto --with-sqlite=auto --with-solr=auto
--with-xapian=auto --with-gssapi=auto --with-libwrap=auto;
rm -rf scan-reports;
scan-build -o scan-reports make 2>&1 || exit 1;
if ! rmdir scan-reports 2>/dev/null; then \
WHAT ABOUT THE OTHER QUESTIONS ?
1 - WHat does represent "subargs" in mail_search_args
2 - for rescan : who is responsible for passing again the new email ? Is
the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)
3 - for get_last_uid : this uncertainity is very unclear. "If there is a
gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?
Thank you
On 2019-01-11 18:27, Joan Moreau wrote:
There is no point into a separate plugin, the purpose is to replace squat as the default fts (solr being a nightmare)
On 2019-01-11 18:23, Aki Tuomi wrote: I would recommend making this a standalone plugin for now instead of trying to keep it in core fts.
Aki On 11 January 2019 at 18:40 Joan Moreau via dovecot < dovecot@dovecot.org> wrote:
I managed to deal with the namespace issue (updated makefile.am)
However, I reach :
../../../src/lib/compat.h:207:19: error: conflicting declaration of 'ssize_t i_my_pread(int, void*, size_t, __off_t)' with 'C' linkage # define pread i_my_pread ^~~~~~~~~~ ../../../src/lib/compat.h:210:9: note: previous declaration with 'C++' linkage ssize_t i_my_pread(int fd, void *buf, size_t count, off_t offset); ^~~~~~~~~~ ../../../src/lib/compat.h:208:20: error: conflicting declaration of 'ssize_t i_my_pwrite(int, const void*, size_t, __off_t)' with 'C' linkage # define pwrite i_my_pwrite
Any help welcome
Hi,
I figured out the "namespace" issue
Remaining questions are :
1 - WHat does represent "subargs" in mail_search_args
2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)
3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?
4 - How to update configure.ac & additional files to add the "--with-xapian" wichi will test for libxapian presence and add it to the build ?
Thank you
On 2019-01-08 04:24, Timo Sirainen wrote:
On 7 Jan 2019, at 16.05, Joan Moreau via dovecot < dovecot@dovecot.org> wrote: Hi
ANyone to answer specifically ?
Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance) All the mails are always supposed to be indexed from the beginning to the last indexed mail. If there's a gap, indexer first indexes all the missing mails. So the latest UID is supposed to be the greatest UID. (Supporting out-of-order indexing would be rather difficult to keep track of.)
Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ? The idea is that it calls something like:
- build_key(type=hdr, hdr_name=From)
- build_more(" tss@iki.fi")
- build_key(type=hdr, hdr_name=Subject)
- build_more("Re: Solr -> Xapian ?")
- build_key(type=body_part)
- build_more("message body piece")
- build_more("message body piece2") ...
Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ? lookup() gets struct mail_search_arg *args, which contains the entire IMAP SEARCH query. This could be used for more or less complex query builders.
In case of a single header search, you should have args->args->hdr_field_name contain the header name and args->args->value.str contain the content you're searching for.
Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ? In case of Xapian it might not matter if it automatically refreshes its indexes between each query. But with some other indexes this could happen:
- IMAP session is opened
- IMAP SEARCH is run, which opens and searches the index
- a new mail is delivered to the mailbox and indexed
- IMAP SEARCH is run. Without refresh() it doesn't see the newly indexed mail and doesn't include it in the search results.
Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ? It's run when "doveadm fts rescan" is run manually. Usually that's only run manually to fix up some brokenness. So it's intended to verify that the current mailbox contents match the FTS indexes: that the next mailbox indexing will index those missing mails. I think
- If there are any mails in FTS index that no longer exist in the actual mailbox, delete those mails from FTS
- If FTS is missing any mails in the middle of the mailbox, make sure
currently this basically means reindexing all the mails since the first missing mail, even the mails that are already in the index.
fts-lucene implements this, but other FTS backends are lazy and simply rebuild all mails. Actually fts-solr is bad because it doesn't even delete the extra mails.
Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ?and finally , for fts_backend_xxxx_lookup_multi, why is that backend dependent ? This function is called only when searching in virtual folders. So for example the virtual "All mails" folder, which would contain all mails in all folders. In that case the boxes[] would contain a list of user's all folders, except Trash and Spam. If lookup_multi() isn't implemented (left to NULL), the search is run separately via lookup() for each folder. With lookup_multi() there can be just one lookup, and the backend can filter only the wanted folders and return them directly. So it's an optimization for FTS indexes that support user-global searches rather than only per-folder searches.
static int fts_backend_xapian_lookup_multi(struct fts_backend *_backend, struct mailbox *const boxes[], struct mail_search_arg *args, enum fts_lookup_flags flags, struct fts_multi_result *result) { struct xapian_fts_backend_update_context *ctx = (struct xapian_fts_backend_update_context *)_ctx;
int i=0;
while(boxes[i]!=NULL) { if(fts_backend_xapian_lookup(backend,box[i],args,flags,result->box_results[i])<0) return -1; i++; } return 0; } See fts_backend_lookup_multi() - if you leave lookup_multi=NULL it basically does this.
For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ??? rescan() is about fixing up a more or less broken index, or simply to verify that it's all ok. So core doesn't know what messages exist in the FTS index and can't request specific reindexing or expunging. I guess an alternative API could have been to have functions that iterate through all mails in the index, and use that to implement rescan in core. Now thinking about it, that sounds like a simpler and better way.
optimize() is currently done only when explicitly running "doveadm fts optimize", which requests running a slower index optimization. Depends on the FTS backend whether this is useful or not.
There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ? Normally when mails are expunged, update_expunge() is called to notify FTS backend that it should delete the mail also from FTS index.
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?* You probably want to use FTS_BACKEND_FLAG_FUZZY_SEARCH only like Solr. See enum fts_backend_flags in fts-api-private.h
Aki Tuomi
STATUS
Alpha code is written and compiling now. (attached)
I would like to start testing. However, there is an error when starting dovecot (git) :
Error: Couldn't load required plugin /usr/lib/dovecot/lib21_fts_xapian_plugin.so: dlopen() failed: /usr/lib/dovecot/lib21_fts_xapian_plugin.so: undefined symbol: _Z30fts_backend_default_can_lookupP11fts_backendPK15mail_search_arg
ldd shows that fts lib is properly linked: # ldd /usr/lib/dovecot/lib21_fts_xapian_plugin.so (...) lib20_fts_plugin.so => /usr/lib/dovecot/lib20_fts_plugin.so (0x00007f25f75e0000) (...) libxapian.so.30 => /usr/lib/libxapian.so.30 (0x00007fe3a51e2000)
Your help very welcome
PENDING QUESTIONS
1 - WHat does represent "subargs" in mail_search_args
2 - for rescan : who is responsible for passing again the new email ? Is
the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)
3 - for get_last_uid : this uncertainity is very unclear. "If there is a
gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?
Thank you
The change of "Extern C" suggested by Timo actually solved the situation
Now, further question :
I put a "i_warning" at each of my functions, and I see in the log :
Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970><ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag>: Warning: fts_backend_xapian_alloc Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970><ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag>: Warning: fts_backend_xapian_init Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970><ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag>: Warning: fts_xapian init (3,25) Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970><ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag>: Warning: fts_backend_xapian_get_last_uid Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970><ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag>: Error: Mailbox XXX: Status lookup failed: Internal error occurred. Refer to server log for more information. [2019-01-12 10:33:27] Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970><ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag>: Warning: fts_backend_xapian_deinit Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970><ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag>: Warning: fts_backend_xapian_unset_box Jan 12 10:33:27 lda(jom@grosjo.net)<31367><ms9zEnfCOVyHegAA0thIag>: Warning: fts_backend_xapian_deinit
The error comes from the fact taht the "get_last_uid" is called without selecting a mailbox first, and therefore there is of course no uid.
How come ?
On 2019-01-12 10:50, Aki Tuomi wrote:
Did you remember to load fts first?
mail_plugins =$mail_plugins fts fts_xapian
Aki
On 12 January 2019 at 10:37 Joan Moreau via dovecot < dovecot@dovecot.org> wrote:
STATUS
Alpha code is written and compiling now. (attached)
I would like to start testing. However, there is an error when starting dovecot (git) :
Error: Couldn't load required plugin /usr/lib/dovecot/lib21_fts_xapian_plugin.so: dlopen() failed: /usr/lib/dovecot/lib21_fts_xapian_plugin.so: undefined symbol: _Z30fts_backend_default_can_lookupP11fts_backendPK15mail_search_arg
ldd shows that fts lib is properly linked: # ldd /usr/lib/dovecot/lib21_fts_xapian_plugin.so (...) lib20_fts_plugin.so => /usr/lib/dovecot/lib20_fts_plugin.so (0x00007f25f75e0000) (...) libxapian.so.30 => /usr/lib/libxapian.so.30 (0x00007fe3a51e2000)
Your help very welcome
PENDING QUESTIONS
1 - WHat does represent "subargs" in mail_search_args
2 - for rescan : who is responsible for passing again the new email ? Is
the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)
3 - for get_last_uid : this uncertainity is very unclear. "If there is a
gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?
Thank you
Aki Tuomi
Sorted this out. sorry for noise
On 2019-01-12 11:39, Joan Moreau wrote:
The change of "Extern C" suggested by Timo actually solved the situation
Now, further question :
I put a "i_warning" at each of my functions, and I see in the log :
Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970><ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag>: Warning: fts_backend_xapian_alloc Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970><ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag>: Warning: fts_backend_xapian_init Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970><ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag>: Warning: fts_xapian init (3,25) Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970><ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag>: Warning: fts_backend_xapian_get_last_uid Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970><ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag>: Error: Mailbox XXX: Status lookup failed: Internal error occurred. Refer to server log for more information. [2019-01-12 10:33:27] Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970><ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag>: Warning: fts_backend_xapian_deinit Jan 12 10:33:27 indexer-worker(jom@grosjo.net)<30970><ms9zEnfCOVyHegAA0thIag:8IhwIHfCOVz6eAAA0thIag>: Warning: fts_backend_xapian_unset_box Jan 12 10:33:27 lda(jom@grosjo.net)<31367><ms9zEnfCOVyHegAA0thIag>: Warning: fts_backend_xapian_deinit
The error comes from the fact taht the "get_last_uid" is called without selecting a mailbox first, and therefore there is of course no uid.
How come ?
On 2019-01-12 10:50, Aki Tuomi wrote: Did you remember to load fts first?
mail_plugins =$mail_plugins fts fts_xapian
Aki On 12 January 2019 at 10:37 Joan Moreau via dovecot < dovecot@dovecot.org> wrote:
STATUS
Alpha code is written and compiling now. (attached)
I would like to start testing. However, there is an error when starting dovecot (git) :
Error: Couldn't load required plugin /usr/lib/dovecot/lib21_fts_xapian_plugin.so: dlopen() failed: /usr/lib/dovecot/lib21_fts_xapian_plugin.so: undefined symbol: _Z30fts_backend_default_can_lookupP11fts_backendPK15mail_search_arg
ldd shows that fts lib is properly linked: # ldd /usr/lib/dovecot/lib21_fts_xapian_plugin.so (...) lib20_fts_plugin.so => /usr/lib/dovecot/lib20_fts_plugin.so (0x00007f25f75e0000) (...) libxapian.so.30 => /usr/lib/libxapian.so.30 (0x00007fe3a51e2000)
Your help very welcome
PENDING QUESTIONS
1 - WHat does represent "subargs" in mail_search_args
2 - for rescan : who is responsible for passing again the new email ? Is
the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)
3 - for get_last_uid : this uncertainity is very unclear. "If there is a
gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?
Thank you
Aki Tuomi
On 11 Jan 2019, at 21.23, Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
The below patch resolves the compilation error
$ diff -p compat.h compat.h.joan *** compat.h 2019-01-11 20:21:00.726625427 +0100 --- compat.h.joan 2019-01-11 20:14:41.729109919 +0100 *************** struct iovec; *** 202,207 **** --- 202,211 ---- ssize_t i_my_writev(int fd, const struct iovec *iov, int iov_len); #endif
- #ifdef __cplusplus
- extern "C" {
- #endif
You should put this extern "C" into the C++ file you're creating. See for example how fts-lucene/lucene-wrapper.cc does this.
1 - WHat does represent "subargs" in mail_search_args
It's set only for SEARCH_OR and SEARCH_SUB. So for example:
SEARCH TEXT foo TEXT bar TEXT baz
results in:
type=SEARCH_SUB value.subargs = ( { type=SEARCH, value.str="foo" }, { type=SEARCH, value.str="bar" }, { type=SEARCH, value.str="baz" }, )
Or similarly if there's SEARCH OR foo OR TEXT bar TEXT baz or some other combination of OR/ANDs.
2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)
The next indexing run is responsible for it. If you return get_last_uid=0, then indexer starts feeding you all mails. So fts backend doesn't have to know about it.
3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?
I mean if get_last_uid() returns for example 100, it means that UIDs 1..100 have been indexed by the FTS backend. It's possible that at this point there are already mails with UIDs 101..200 in the folder. So when UID=201 is delivered, indexer notices that FTS backend has only UIDs 1..100 indexed so far, and starts feeding it UIDs 101..201 in that order.
You can implement get_last_uid() simply by keeping track of it in dovecot.index* files, similar to how Lucene and Solr already do it with fts_index_get_header() / fts_index_set_header(). They also have a fallback that if the index doesn't have the last_uid value, they do a slower search from the Lucene/Solr index to find the last UID.
THank you
Now, for the results
I see the member of fts_result is :
ARRAY_TYPE(seq_range) definite_uids;
I have the UID as a aray of uint32_t *
How to put my UIDs into this "definite_uids" ? Obviously this is not a simple array/pointer. How to say someting similar to result->definite_uids[1]=my_uid ?
On 2019-01-12 10:25, Timo Sirainen wrote:
On 11 Jan 2019, at 21.23, Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
The below patch resolves the compilation error
$ diff -p compat.h compat.h.joan *** compat.h 2019-01-11 20:21:00.726625427 +0100 --- compat.h.joan 2019-01-11 20:14:41.729109919 +0100 *************** struct iovec; *** 202,207 **** --- 202,211 ---- ssize_t i_my_writev(int fd, const struct iovec *iov, int iov_len); #endif
- #ifdef __cplusplus
- extern "C" {
- #endif
You should put this extern "C" into the C++ file you're creating. See for example how fts-lucene/lucene-wrapper.cc does this.
1 - WHat does represent "subargs" in mail_search_args
It's set only for SEARCH_OR and SEARCH_SUB. So for example:
SEARCH TEXT foo TEXT bar TEXT baz
results in:
type=SEARCH_SUB value.subargs = ( { type=SEARCH, value.str="foo" }, { type=SEARCH, value.str="bar" }, { type=SEARCH, value.str="baz" }, )
Or similarly if there's SEARCH OR foo OR TEXT bar TEXT baz or some other combination of OR/ANDs.
2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large)
The next indexing run is responsible for it. If you return get_last_uid=0, then indexer starts feeding you all mails. So fts backend doesn't have to know about it.
3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ?
I mean if get_last_uid() returns for example 100, it means that UIDs 1..100 have been indexed by the FTS backend. It's possible that at this point there are already mails with UIDs 101..200 in the folder. So when UID=201 is delivered, indexer notices that FTS backend has only UIDs 1..100 indexed so far, and starts feeding it UIDs 101..201 in that order.
You can implement get_last_uid() simply by keeping track of it in dovecot.index* files, similar to how Lucene and Solr already do it with fts_index_get_header() / fts_index_set_header(). They also have a fallback that if the index doesn't have the last_uid value, they do a slower search from the Lucene/Solr index to find the last UID.
additionally, my logic is that the backend stores one databalse per mailox in /xapian-indexes (in the "root" dir of the user), the name od the database is the GUID of the mailbox
For INBOX, that works perfectly, and database is properly createdm and backed starts indexing all emails
For other folder, somehow, the process can not access that (root) folder.
Am I missing something ?
On 2019-01-12 17:37, Joan Moreau wrote:
THank you
Now, for the results
I see the member of fts_result is :
ARRAY_TYPE(seq_range) definite_uids;
I have the UID as a aray of uint32_t *
How to put my UIDs into this "definite_uids" ? Obviously this is not a simple array/pointer. How to say someting similar to result->definite_uids[1]=my_uid ?
On 2019-01-12 10:25, Timo Sirainen wrote: On 11 Jan 2019, at 21.23, Joan Moreau via dovecot <dovecot@dovecot.org> wrote: The below patch resolves the compilation error
$ diff -p compat.h compat.h.joan *** compat.h 2019-01-11 20:21:00.726625427 +0100 --- compat.h.joan 2019-01-11 20:14:41.729109919 +0100 *************** struct iovec; *** 202,207 **** --- 202,211 ---- ssize_t i_my_writev(int fd, const struct iovec *iov, int iov_len); #endif
- #ifdef __cplusplus
- extern "C" {
- #endif
You should put this extern "C" into the C++ file you're creating. See for example how fts-lucene/lucene-wrapper.cc does this.
1 - WHat does represent "subargs" in mail_search_args It's set only for SEARCH_OR and SEARCH_SUB. So for example:
SEARCH TEXT foo TEXT bar TEXT baz
results in:
type=SEARCH_SUB value.subargs = ( { type=SEARCH, value.str="foo" }, { type=SEARCH, value.str="bar" }, { type=SEARCH, value.str="baz" }, )
Or similarly if there's SEARCH OR foo OR TEXT bar TEXT baz or some other combination of OR/ANDs. 2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large) The next indexing run is responsible for it. If you return get_last_uid=0, then indexer starts feeding you all mails. So fts backend doesn't have to know about it.
3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ? I mean if get_last_uid() returns for example 100, it means that UIDs 1..100 have been indexed by the FTS backend. It's possible that at this point there are already mails with UIDs 101..200 in the folder. So when UID=201 is delivered, indexer notices that FTS backend has only UIDs 1..100 indexed so far, and starts feeding it UIDs 101..201 in that order.
You can implement get_last_uid() simply by keeping track of it in dovecot.index* files, similar to how Lucene and Solr already do it with fts_index_get_header() / fts_index_set_header(). They also have a fallback that if the index doesn't have the last_uid value, they do a slower search from the Lucene/Solr index to find the last UID.
I somehow fixed the folder issue. (seems some unix rights after too many tests)
Getting back on the "fts_results" structure:
I am trying:
I_ARRAY_INIT(&(RESULT->DEFINITE_UIDS),R->SIZE); I_ARRAY_INIT(&(RESULT->MAYBE_UIDS),0);
uint32_t uid; for(i=0;i<r->size;i++) { try { uid=atol(backend->dbr->get_document(r->data[i]).get_value(1).c_str()); i_warning("Rresult UID=%d",uid); ARRAY_IDX_SET(&(RESULT->DEFINITE_UIDS),I,&UID); } catch(Xapian::Error e) { i_warning(e.get_msg().c_str()); } }
I can see in hte log that UID are properly found on Xapian database, but no results are transmitted to dovecot and to the imap client (roundcube in my case)
Help please :)
On 2019-01-12 18:15, Joan Moreau wrote:
additionally, my logic is that the backend stores one databalse per mailox in /xapian-indexes (in the "root" dir of the user), the name od the database is the GUID of the mailbox
For INBOX, that works perfectly, and database is properly createdm and backed starts indexing all emails
For other folder, somehow, the process can not access that (root) folder.
Am I missing something ?
On 2019-01-12 17:37, Joan Moreau wrote:
THank you
Now, for the results
I see the member of fts_result is :
ARRAY_TYPE(seq_range) definite_uids;
I have the UID as a aray of uint32_t *
How to put my UIDs into this "definite_uids" ? Obviously this is not a simple array/pointer. How to say someting similar to result->definite_uids[1]=my_uid ?
On 2019-01-12 10:25, Timo Sirainen wrote: On 11 Jan 2019, at 21.23, Joan Moreau via dovecot <dovecot@dovecot.org> wrote: The below patch resolves the compilation error
$ diff -p compat.h compat.h.joan *** compat.h 2019-01-11 20:21:00.726625427 +0100 --- compat.h.joan 2019-01-11 20:14:41.729109919 +0100 *************** struct iovec; *** 202,207 **** --- 202,211 ---- ssize_t i_my_writev(int fd, const struct iovec *iov, int iov_len); #endif
- #ifdef __cplusplus
- extern "C" {
- #endif
You should put this extern "C" into the C++ file you're creating. See for example how fts-lucene/lucene-wrapper.cc does this.
1 - WHat does represent "subargs" in mail_search_args It's set only for SEARCH_OR and SEARCH_SUB. So for example:
SEARCH TEXT foo TEXT bar TEXT baz
results in:
type=SEARCH_SUB value.subargs = ( { type=SEARCH, value.str="foo" }, { type=SEARCH, value.str="bar" }, { type=SEARCH, value.str="baz" }, )
Or similarly if there's SEARCH OR foo OR TEXT bar TEXT baz or some other combination of OR/ANDs. 2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large) The next indexing run is responsible for it. If you return get_last_uid=0, then indexer starts feeding you all mails. So fts backend doesn't have to know about it.
3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ? I mean if get_last_uid() returns for example 100, it means that UIDs 1..100 have been indexed by the FTS backend. It's possible that at this point there are already mails with UIDs 101..200 in the folder. So when UID=201 is delivered, indexer notices that FTS backend has only UIDs 1..100 indexed so far, and starts feeding it UIDs 101..201 in that order.
You can implement get_last_uid() simply by keeping track of it in dovecot.index* files, similar to how Lucene and Solr already do it with fts_index_get_header() / fts_index_set_header(). They also have a fallback that if the index doesn't have the last_uid value, they do a slower search from the Lucene/Solr index to find the last UID.
I found the solution o this using SEQ_RANGE_ARRAY_ADD(&RESULT->DEFINITE_UIDS, UID);
Now, I can see in the logs that several times, the dovecot calls the fts_backend_xapian_update_set_mailbox with box == NULL. WHy so ?
THank you
On 2019-01-12 21:40, Joan Moreau via dovecot wrote:
I somehow fixed the folder issue. (seems some unix rights after too many tests)
Getting back on the "fts_results" structure:
I am trying:
I_ARRAY_INIT(&(RESULT->DEFINITE_UIDS),R->SIZE); I_ARRAY_INIT(&(RESULT->MAYBE_UIDS),0);
uint32_t uid; for(i=0;i<r->size;i++) { try { uid=atol(backend->dbr->get_document(r->data[i]).get_value(1).c_str()); i_warning("Rresult UID=%d",uid); ARRAY_IDX_SET(&(RESULT->DEFINITE_UIDS),I,&UID); } catch(Xapian::Error e) { i_warning(e.get_msg().c_str()); } }
I can see in hte log that UID are properly found on Xapian database, but no results are transmitted to dovecot and to the imap client (roundcube in my case)
Help please :)
On 2019-01-12 18:15, Joan Moreau wrote:
additionally, my logic is that the backend stores one databalse per mailox in /xapian-indexes (in the "root" dir of the user), the name od the database is the GUID of the mailbox
For INBOX, that works perfectly, and database is properly createdm and backed starts indexing all emails
For other folder, somehow, the process can not access that (root) folder.
Am I missing something ?
On 2019-01-12 17:37, Joan Moreau wrote:
THank you
Now, for the results
I see the member of fts_result is :
ARRAY_TYPE(seq_range) definite_uids;
I have the UID as a aray of uint32_t *
How to put my UIDs into this "definite_uids" ? Obviously this is not a simple array/pointer. How to say someting similar to result->definite_uids[1]=my_uid ?
On 2019-01-12 10:25, Timo Sirainen wrote: On 11 Jan 2019, at 21.23, Joan Moreau via dovecot <dovecot@dovecot.org> wrote: The below patch resolves the compilation error
$ diff -p compat.h compat.h.joan *** compat.h 2019-01-11 20:21:00.726625427 +0100 --- compat.h.joan 2019-01-11 20:14:41.729109919 +0100 *************** struct iovec; *** 202,207 **** --- 202,211 ---- ssize_t i_my_writev(int fd, const struct iovec *iov, int iov_len); #endif
- #ifdef __cplusplus
- extern "C" {
- #endif
You should put this extern "C" into the C++ file you're creating. See for example how fts-lucene/lucene-wrapper.cc does this.
1 - WHat does represent "subargs" in mail_search_args It's set only for SEARCH_OR and SEARCH_SUB. So for example:
SEARCH TEXT foo TEXT bar TEXT baz
results in:
type=SEARCH_SUB value.subargs = ( { type=SEARCH, value.str="foo" }, { type=SEARCH, value.str="bar" }, { type=SEARCH, value.str="baz" }, )
Or similarly if there's SEARCH OR foo OR TEXT bar TEXT baz or some other combination of OR/ANDs. 2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large) The next indexing run is responsible for it. If you return get_last_uid=0, then indexer starts feeding you all mails. So fts backend doesn't have to know about it.
3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ? I mean if get_last_uid() returns for example 100, it means that UIDs 1..100 have been indexed by the FTS backend. It's possible that at this point there are already mails with UIDs 101..200 in the folder. So when UID=201 is delivered, indexer notices that FTS backend has only UIDs 1..100 indexed so far, and starts feeding it UIDs 101..201 in that order.
You can implement get_last_uid() simply by keeping track of it in dovecot.index* files, similar to how Lucene and Solr already do it with fts_index_get_header() / fts_index_set_header(). They also have a fallback that if the index doesn't have the last_uid value, they do a slower search from the Lucene/Solr index to find the last UID.
On 13 Jan 2019, at 10.45, Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Now, I can see in the logs that several times, the dovecot calls the fts_backend_xapian_update_set_mailbox with box == NULL. WHy so ?
fts-api.h says:
/* Switch to updating the specified mailbox. box may also be set to NULL to make sure the previous mailbox won't tried to be accessed anymore. */ void fts_backend_update_set_mailbox(struct fts_backend_update_context *ctx, struct mailbox *box);
So it's just telling you that you can close/free any stuff related to that mailbox.
additionally, my logic is that the backend stores one databalse per mailox in /xapian-indexes (in the "root" dir of the user), the name od the database is the GUID of the mailbox
For INBOX, that works perfectly, and database is properly createdm and backed starts indexing all emails
For other folder, somehow, the process can not access that (root) folder.
Am I missing something ?
This is a bit ambiguous, because some people mean mailbox=folder and others mean mailbox=user account, and GUID can also be the internal Dovecot folder GUID, or a GUID of the user.
I'd recommend using a single database per user anyway.
To get back on thi "build_more" function:
this is what I receive:
(see below)
2 poitns : the header name seems to be added at the end of the *data. not always, why so ?
where is the body ?
Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA(mime-version)=1.0MIME-VERSION Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA2(mime-version)=1.0MIME-VERSION Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: Start indexing 'Sent' (/data/mail/grosjo.net/jom/xapian-indexes/db_49fdf110ec9bc14c375b0000d6a3092d) Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA(content-type)=MULTIPART/ALTERNATIVE; BOUNDARY="=_87A48D791CC8B262204294719234352F"CONTENT-TYPE Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA2(content-type)=MULTIPART/ALTERNATIVE; BOUNDARY="=_87A48D791CC8B262204294719234352F"CONTENT-TYPE Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA(date)=TUE, 22 JAN 2019 09:25:49 +0100DATE Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA2(date)=TUE, 22 JAN 2019 09:25:49 +0100DATE Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA(from)="JOAN MOREAU" <JOM@GROSJO.NET> Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA2(from)="JOAN MOREAU" <JOM@GROSJO.NET> Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA(to)="JOAN MOREAU" <JOAN.MOREAU@M4X.ORG> Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA2(to)="JOAN MOREAU" <JOAN.MOREAU@M4X.ORG> Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA(subject)=TESTSUBJECT Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA2(subject)=TESTSUBJECT Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA(user-agent)=ROUNDCUBE WEBMAIL/1.4-GITUSER-AGENT Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA2(user-agent)=ROUNDCUBE WEBMAIL/1.4-GITUSER-AGENT Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA(message-id)=<1C18523A5A00849C8BE7970F44276F1B@GROSJO.NET>MESSAGE-ID Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA2(message-id)=<1C18523A5A00849C8BE7970F44276F1B@GROSJO.NET>MESSAGE-ID Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA(x-sender)=JOM@GROSJO.NETX-SENDER Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA2(x-sender)=JOM@GROSJO.NETX-SENDER Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA(content-transfer-encoding)=7BITCONTENT-TRANSFER-ENCODING Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA2(content-transfer-encoding)=7BITCONTENT-TRANSFER-ENCODING Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA(content-type)=TEXT/PLAIN; CHARSET=UTF-8; FORMAT=FLOWEDCONTENT-TYPE Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA2(content-type)=TEXT/PLAIN; CHARSET=UTF-8; FORMAT=FLOWEDCONTENT-TYPE Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA(content-transfer-encoding)=QUOTED-PRINTABLECONTENT-TRANSFER-ENCODING Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA2(content-transfer-encoding)=QUOTED-PRINTABLECONTENT-TRANSFER-ENCODING Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA(content-type)=TEXT/HTML; CHARSET=UTF-8CONTENT-TYPE Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: DATA2(content-type)=TEXT/HTML; CHARSET=UTF-8CONTENT-TYPE Jan 22 08:25:50 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: Done indexing 'Sent' (1 msgs in 3 ms, rate: 333.3) Jan 22 08:25:50 gjserver dovecot[20984]: imap-login: Login: user=<jom@grosjo.net>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=21699, secured, session=<99GeuQeANlt/AAAB> Jan 22 08:25:51 gjserver dovecot[20984]: imap(jom@grosjo.net)<21699><99GeuQeANlt/AAAB>: Logged out in=20201 out=567147 deleted=0 expunged=0 trashed=0 hdr_count=200 hdr_bytes=62139 body_count=0 body_bytes=0 Jan 22 08:25:51 gjserver dovecot[20984]: indexer-worker(jom@grosjo.net)<20998><jyqauQeAMlt/AAAB:qHyjLY7TRlwGUgAA0thIag>: Indexed 1 messages in Sent (UIDs 60585..60585)
On 2019-01-08 04:24, Timo Sirainen wrote:
On 7 Jan 2019, at 16.05, Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Hi
ANyone to answer specifically ?
Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance)
All the mails are always supposed to be indexed from the beginning to the last indexed mail. If there's a gap, indexer first indexes all the missing mails. So the latest UID is supposed to be the greatest UID. (Supporting out-of-order indexing would be rather difficult to keep track of.)
Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ?
The idea is that it calls something like:
- build_key(type=hdr, hdr_name=From)
- build_more("tss@iki.fi")
- build_key(type=hdr, hdr_name=Subject)
- build_more("Re: Solr -> Xapian ?")
- build_key(type=body_part)
- build_more("message body piece")
- build_more("message body piece2") ...
Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ?
lookup() gets struct mail_search_arg *args, which contains the entire IMAP SEARCH query. This could be used for more or less complex query builders.
In case of a single header search, you should have args->args->hdr_field_name contain the header name and args->args->value.str contain the content you're searching for.
Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ?
In case of Xapian it might not matter if it automatically refreshes its indexes between each query. But with some other indexes this could happen:
- IMAP session is opened
- IMAP SEARCH is run, which opens and searches the index
- a new mail is delivered to the mailbox and indexed
- IMAP SEARCH is run. Without refresh() it doesn't see the newly indexed mail and doesn't include it in the search results.
Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ?
It's run when "doveadm fts rescan" is run manually. Usually that's only run manually to fix up some brokenness. So it's intended to verify that the current mailbox contents match the FTS indexes:
- If there are any mails in FTS index that no longer exist in the actual mailbox, delete those mails from FTS
- If FTS is missing any mails in the middle of the mailbox, make sure that the next mailbox indexing will index those missing mails. I think currently this basically means reindexing all the mails since the first missing mail, even the mails that are already in the index.
fts-lucene implements this, but other FTS backends are lazy and simply rebuild all mails. Actually fts-solr is bad because it doesn't even delete the extra mails.
Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ? and finally , for fts_backend_xxxx_lookup_multi, why is that backend dependent ?
This function is called only when searching in virtual folders. So for example the virtual "All mails" folder, which would contain all mails in all folders. In that case the boxes[] would contain a list of user's all folders, except Trash and Spam. If lookup_multi() isn't implemented (left to NULL), the search is run separately via lookup() for each folder. With lookup_multi() there can be just one lookup, and the backend can filter only the wanted folders and return them directly. So it's an optimization for FTS indexes that support user-global searches rather than only per-folder searches.
static int fts_backend_xapian_lookup_multi(struct fts_backend *_backend, struct mailbox *const boxes[], struct mail_search_arg *args, enum fts_lookup_flags flags, struct fts_multi_result *result) { struct xapian_fts_backend_update_context *ctx = (struct xapian_fts_backend_update_context *)_ctx;
int i=0;
while(boxes[i]!=NULL) { if(fts_backend_xapian_lookup(backend,box[i],args,flags,result->box_results[i])<0) return -1; i++; } return 0; }
See fts_backend_lookup_multi() - if you leave lookup_multi=NULL it basically does this.
For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ???
rescan() is about fixing up a more or less broken index, or simply to verify that it's all ok. So core doesn't know what messages exist in the FTS index and can't request specific reindexing or expunging. I guess an alternative API could have been to have functions that iterate through all mails in the index, and use that to implement rescan in core. Now thinking about it, that sounds like a simpler and better way.
optimize() is currently done only when explicitly running "doveadm fts optimize", which requests running a slower index optimization. Depends on the FTS backend whether this is useful or not.
There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ?
Normally when mails are expunged, update_expunge() is called to notify FTS backend that it should delete the mail also from FTS index.
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
You probably want to use FTS_BACKEND_FLAG_FUZZY_SEARCH only like Solr. See enum fts_backend_flags in fts-api-private.h
El 04/01/19 a las 03:20, Joan Moreau via dovecot escribió:
What about consedering linking Dovecot with Xapian librairies instead of going to nightmare Solr ? https://xapian.org/features
given that notmuch already does a good job at indexing email (although only supports maildirs afaik), wouldn't it be simpler to write a plugin for running notmuch searches from dovecot?
On 1/2/2019 12:59 AM, M. Balridge wrote:
So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality:
- The ability to search for a string within any of the structured fields (from/subject) that returns correct results?
Yes.
- The ability to search for any string within the BODY of emails, including the MIME attachment boundaries?
Yes.
- The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data?
Dunno - I don't think I've needed that and I'm not sure how to do it. My mail clients are Thunderbird and AquaMail (on Android). If you'll give me either the desired Thunderbird steps or telnet-based IMAP command I'm happy to test.
OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines:
- The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika?
Haven't tried.
SOLR seems to have "brand cachet", so presumably it actually works (for somebody).
It works - just sometimes needs more effort to setup than it should.
Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software.
I think we're all in agreement here.
But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?)
I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have.
He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them.
Short answer - it actually works. Longer answer - I've gone through a hate/love/hate/like relationship with Solr myself. The transition from v3 to v4 was a major headache - and I gave up for a while. But versions 6 & 7 have been pretty good for me. I'm neither a Dovecot nor a Solr developer - just enough of a fiddler to get them working to fulfill my own needs.
If my unreliable memory serves I believe the Dovecot fts-solr plugin hasn't needed to change much (I recall one significant change required when Solr changed it's protocol - I think an XML/JSON thing). So having a stable interface let's Timo & Co. forget about on-going FTS development and continue focusing on things not provided by other tools. Hopefully they'll revisit SIS...
I recall reading something about the Lucene library (which Squat & Solr are based on) and again my memory is the C version(s) weren't getting maintained as well as might be desired. I think having the Solr/Lucene team focusing on Java development was another point of consideration for Dovecot's squat - but I could be totally off here.
Based on the errors reported by Joan I believe that system's problems are due to configuration - either Solr, Dovecot, or both. They don't sound like Java related issues (which are a *major* pain to deal with!). I've provided a copy of what is a working configuration *for me*. I'm happy to continue helping as best I can - and if Joan, you, or anyone else would like my aid I'll do my best. If you're crazy I-mean-trusting enough to have me SSH or remote view to your system I'm willing to take a look. I've had enough people help me over the years for various packages that I'd like to pay it forward where I can.
-- Daniel
I'm running 7.5.0. The solrconfig.xml file is what I've modified over time - I haven't started one from scratch for a while but perhaps I'll try.
Have you tried using the complete config that I sent you? With *all* the files I included - and *none* of yours?
--
Daniel
On 1/1/2019 4:12 PM, Joan Moreau wrote:
The real main differecne seems coming from "diffconfig.xml"
When I put yours, Solr delete (!) schema.xml and create a "manage-schema" and starts complaining about useless types (tdates, booleans, etc..) that are not needed for Mail fileds
When I put mine (from standard distribution of Arch), it keeps things as they are (yeah !), does not complains about those useless types and startup properly.
I attach my diffconfig
But these are the configurations that one should adjust as per his/her own use.
The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors
On 2019-01-02 07:49, Joan Moreau wrote:
Hi
Solr is a standard package in ArchLinux. ("pacman -S solr") . the systemd installation script is included (and it is launching /opt/solr/bin/solr.in.sh)
Instance : sudo -u solr /opt/solr/bin/solr create -c dovecot -> this creates a separate folder with default solrconfig.xml, schema.xml, etc..
I made a symlink of the data folder to a second drive (ext4) much bigger
On 2018-12-31 14:09, Daniel Miller wrote:
On 12/29/2018 4:49 PM, Joan Moreau wrote: Also : - Java is 10.0.2 Same as me. - If i delete schema.xml but create only managed-schema, the solr refuses to start with a java error "schema.xml missing" Ok...so we need to do some more digging. How did you install Solr? (I downloaded a "binary" installation and unpacked it) How did you create the dovecot instance? (I've provided explicit instructions for how I did it - did you follow those exactly or something different)? How are you starting Solr? (I use the provided "solr/bin/solr start" command, wrapped inside a systemd service). -- Daniel
--
Daniel
On 1/1/2019 3:49 PM, Joan Moreau via dovecot wrote:
Hi
Solr is a standard package in ArchLinux. ("pacman -S solr") . the systemd installation script is included (and it is launching /opt/solr/bin/solr.in.sh)
Instance : sudo -u solr /opt/solr/bin/solr create -c dovecot -> this creates a separate folder with default solrconfig.xml, schema.xml, etc..
I made a symlink of the data folder to a second drive (ext4) much bigger
I'm using that nasty word *should*...in that the above installation *should* yield working results. But...since I don't use Arch and have no insight into it I suggest downloading a binary tarball from the Solr site and do a clean install. It may behave identically...or maybe something will be different.
-- Daniel
On 12/29/2018 4:46 PM, Joan Moreau wrote:
Hi Daniel,
I am on Archlinux. Anyway, I adapted the scripts.
2 questions:
1 - It looks like we are not on the same version . I am on 7.5.0. Which version are you running ?
Solr 7.5.0.
2 - Your conf shows that you let managed-schema but deleted schema.xml. What is the meaning of each ?
schema.xml is the legacy configuration file. managed-schema is the config file used by current Solr versions.
-- Daniel
On 1/3/2019 10:56 AM, Tanstaafl wrote:
On 12/21/2018, 11:19:42 AM, Daniel Miller via dovecot <dovecot@dovecot.org> wrote:
There is a *huge* difference between a functional Solr setup & squat Interesting. Care to elaborate?
This is one of those things that has to be experienced to be understood. When you can perform an FTS search across (pause while I check current stats...):
du -c -h /var/mail 136G
Solr numDocs: 520102
and using any IMAP client that supports server-side searches (like Thunderbird & AquaMail) the results are basically instantaneous...it's worth the effort. And that's searching a Dovecot virtual folder defined as "* all", including all my archives, all my list subscriptions, and all the shared Inbox/Sent folders from my other users.
But I certainly wish it was easier to setup.
-- Daniel
Hi
This is the summary of my work with SOLR-Dovecot, in my QUEST TO REPRODUCE THE PREVIOULSY EXCELLENT WORK OF FTS_SQUAT
@Aki : Based on the time I have spent on this, I would love to see you updating the Wiki with those improvements, and adding my name somewhere
@All : Hope it helps
- INSTALLATION:
-> Create a clean install using the default, (at least in the Archlinux package), and do a "sudo -u solr solr create -c dovecot ". The config files are then in /opt/solr/server/solr/dovecot/conf and datafiles in /opt/solr/server/solr/dovecot/data
-> In /opt/solr/server/solr/dovecot/conf/solrconfig.xml:
* around line 313, change <openSearcher>false</openSearcher> to<openSearcher>true</openSearcher>
* around line 147, set <writeLockTimeout>2000</writeLockTimeout>(or above)
* around line 1127, before <updateProcessorclass="solr.UUIDUpdateProcessorFactory" name="uuid"/>, add <schemaFactory class="ClassicIndexSchemaFactory"></schemaFactory>
* around line 1161, delete the whole <updateProcessorclass="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields">
* around line 1192, remove the whole <updateRequestProcessorChainname="add-unknown-fields-to-the-schema" ... />
-> Remove /opt/solr/server/solr/dovecot/conf/managed-schema
-> Change "schema.xml" by the one below to reproduce fts_squat behavior (equivalent to " fts_squat = partial=3 full=25" in dovecot.conf) (note : such a huge trouble to replace a single line setup, anyway...)
-> Move /opt/solr/server/solr (or the subfolder data) to a partition with *space*, ideally ext4 or faster file system (it looks like Solr is not considering using a simple mysql database, which would make sense to avoid all the fuzz and let it transit to a non-java state, but that is another story)
-> Config of dovecot.conf is as below
-> The systemd unit shall specify high ulimit for files and proc (see below)
-> Increase the memory available for the JavaVM (I put 12Gb as I have quite a space on my server, but you may adapt it as per your specs) : in /opt/solr/bin/solr.in.sh, set SOLR_HEAP="12288m"
-> As Solr is complaining a lot, you may consider a filter for it in your syslog-ng or journald as it pollutes greatly your audit files
-> (re)Start solr (first) and dovecot by systemctl
-> Launch redindex ( doveadm fts rescan -u <username> )
-> wait for a big while to let the system re-index all your mail boxes
- BUGS SO FAR
-> Line 620 of fts_solr dovecot plugin : the size oof header is improperly calculated ("huge header" warning for a simple email, which kilss the index of that considered email, so basically MOST emails as the calculation is wrong)
-> The UID returned by SOlr is to be considered as a STRING (and that is maybe the source of problem of the "out of bound" errors in fts_solr dovecot, as "long" is not enough)
-> Java errors : A lot of non sense for me, I am not expert in Java. But, with increased memory, it seems not crashing, even if complaining quite a lot in the logs
-------SCHEMA.XML IN /OPT/SOLR/SERVER/SOLR/DOVECOT/CONF
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="dovecottext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> <fieldType name="dovecotfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="25"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType>
<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="dovecottext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="dovecotfield" indexed="true" stored="false"/> <field name="from" type="dovecotfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="dovecottext" indexed="true" stored="false"/> <field name="to" type="dovecotfield" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
-- DOVECOT.CONF
mail_plugins = fts fts_solr
plugin { plugin = fts fts_solr managesieve sieve
fts = solr fts_autoindex = yes fts_enforced = yes fts_solr = url=http://127.0.0.1:8983/solr/dovecot/
(replace 127.0.0.1 by your solr server if you want to use an external server) (...)
}
-- /ETC/SYSTEMD/SYSTEM/MULTI-USER.TARGET.WANTS/SOLR.SERVICE
[Unit] Description=Solr full text search engine After=network.target
[Service] Type=simple User=solr Group=solr PrivateTmp=yes WorkingDirectory=/opt/solr LIMITNOFILE=65000 LIMITNPROC=65000 ExecStart=/opt/solr/bin/solr start -f
[Install] WantedBy=multi-user.target
We'll go thru all your updates, and try to update the wiki with what there is. Your effort is really appreciated here.
Aki
On 04 January 2019 at 02:38 Joan Moreau via dovecot <dovecot@dovecot.org> wrote:
Hi
This is the summary of my work with SOLR-Dovecot, in my QUEST TO REPRODUCE THE PREVIOULSY EXCELLENT WORK OF FTS_SQUAT
@Aki : Based on the time I have spent on this, I would love to see you updating the Wiki with those improvements, and adding my name somewhere
@All : Hope it helps
- INSTALLATION:
-> Create a clean install using the default, (at least in the Archlinux package), and do a "sudo -u solr solr create -c dovecot ". The config files are then in /opt/solr/server/solr/dovecot/conf and datafiles in /opt/solr/server/solr/dovecot/data
-> In /opt/solr/server/solr/dovecot/conf/solrconfig.xml:
* around line 313, change <openSearcher>false</openSearcher> to<openSearcher>true</openSearcher>
* around line 147, set <writeLockTimeout>2000</writeLockTimeout>(or above)
* around line 1127, before <updateProcessorclass="solr.UUIDUpdateProcessorFactory" name="uuid"/>, add <schemaFactory class="ClassicIndexSchemaFactory"></schemaFactory>
* around line 1161, delete the whole <updateProcessorclass="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields">
* around line 1192, remove the whole <updateRequestProcessorChainname="add-unknown-fields-to-the-schema" ... />
-> Remove /opt/solr/server/solr/dovecot/conf/managed-schema
-> Change "schema.xml" by the one below to reproduce fts_squat behavior (equivalent to " fts_squat = partial=3 full=25" in dovecot.conf) (note : such a huge trouble to replace a single line setup, anyway...)
-> Move /opt/solr/server/solr (or the subfolder data) to a partition with *space*, ideally ext4 or faster file system (it looks like Solr is not considering using a simple mysql database, which would make sense to avoid all the fuzz and let it transit to a non-java state, but that is another story)
-> Config of dovecot.conf is as below
-> The systemd unit shall specify high ulimit for files and proc (see below)
-> Increase the memory available for the JavaVM (I put 12Gb as I have quite a space on my server, but you may adapt it as per your specs) : in /opt/solr/bin/solr.in.sh, set SOLR_HEAP="12288m"
-> As Solr is complaining a lot, you may consider a filter for it in your syslog-ng or journald as it pollutes greatly your audit files
-> (re)Start solr (first) and dovecot by systemctl
-> Launch redindex ( doveadm fts rescan -u <username> )
-> wait for a big while to let the system re-index all your mail boxes
- BUGS SO FAR
-> Line 620 of fts_solr dovecot plugin : the size oof header is improperly calculated ("huge header" warning for a simple email, which kilss the index of that considered email, so basically MOST emails as the calculation is wrong)
-> The UID returned by SOlr is to be considered as a STRING (and that is maybe the source of problem of the "out of bound" errors in fts_solr dovecot, as "long" is not enough)
-> Java errors : A lot of non sense for me, I am not expert in Java. But, with increased memory, it seems not crashing, even if complaining quite a lot in the logs
-------SCHEMA.XML IN /OPT/SOLR/SERVER/SOLR/DOVECOT/CONF
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="dovecottext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> <fieldType name="dovecotfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="25"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType>
<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="dovecottext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="dovecotfield" indexed="true" stored="false"/> <field name="from" type="dovecotfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="dovecottext" indexed="true" stored="false"/> <field name="to" type="dovecotfield" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
-- DOVECOT.CONF
mail_plugins = fts fts_solr
plugin { plugin = fts fts_solr managesieve sieve
fts = solr fts_autoindex = yes fts_enforced = yes fts_solr = url=http://127.0.0.1:8983/solr/dovecot/
(replace 127.0.0.1 by your solr server if you want to use an external server) (...)
}
-- /ETC/SYSTEMD/SYSTEM/MULTI-USER.TARGET.WANTS/SOLR.SERVICE
[Unit] Description=Solr full text search engine After=network.target
[Service] Type=simple User=solr Group=solr PrivateTmp=yes WorkingDirectory=/opt/solr LIMITNOFILE=65000 LIMITNPROC=65000 ExecStart=/opt/solr/bin/solr start -f
[Install] WantedBy=multi-user.target
Hi
This is the summary of my work with SOLR-Dovecot, in my QUEST TO REPRODUCE THE PREVIOULSY EXCELLENT WORK OF FTS_SQUAT
@Aki : Based on the time I have spent on this, I would love to see you updating the Wiki with those improvements, and adding my name somewhere
@All : Hope it helps
- INSTALLATION:
-> Create a clean install using the default, (at least in the Archlinux package), and do a "sudo -u solr solr create -c dovecot ". The config files are then in /opt/solr/server/solr/dovecot/conf and datafiles in /opt/solr/server/solr/dovecot/data
-> In /opt/solr/server/solr/dovecot/conf/solrconfig.xml:
* around line 313, change <openSearcher>false</openSearcher> to<openSearcher>true</openSearcher>
* around line 147, set <writeLockTimeout>2000</writeLockTimeout>(or above)
* around line 696 : uncomment <str name="df">hdr</str>
* around line 1127, before <updateProcessorclass="solr.UUIDUpdateProcessorFactory" name="uuid"/>, add <schemaFactory class="ClassicIndexSchemaFactory"></schemaFactory>
* around line 1161, delete the whole <updateProcessorclass="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields">
* around line 1192, remove the whole <updateRequestProcessorChainname="add-unknown-fields-to-the-schema" ... />
-> Remove /opt/solr/server/solr/dovecot/conf/managed-schema
-> Change "schema.xml" by the one below to reproduce fts_squat behavior (equivalent to " fts_squat = partial=3 full=25" in dovecot.conf) (note : such a huge trouble to replace a single line setup, anyway...)
-> Move /opt/solr/server/solr (or the subfolder data) to a partition with *space*, ideally ext4 or faster file system (it looks like Solr is not considering using a simple mysql database, which would make sense to avoid all the fuzz and let it transit to a non-java state, but that is another story)
-> Config of dovecot.conf is as below
-> The systemd unit shall specify high ulimit for files and proc (see below)
-> Increase the memory available for the JavaVM (I put 12Gb as I have quite a space on my server, but you may adapt it as per your specs) : in /opt/solr/bin/solr.in.sh, set SOLR_HEAP="12288m"
-> As Solr is complaining a lot, you may consider a filter for it in your syslog-ng or journald as it pollutes greatly your audit files
-> (re)Start solr (first) and dovecot by systemctl
-> Launch redindex ( doveadm fts rescan -u <username> )
-> wait for a big while to let the system re-index all your mail boxes
- BUGS SO FAR
-> Line 620 of fts_solr dovecot plugin : the size oof header is improperly calculated ("huge header" warning for a simple email, which kilss the index of that considered email, so basically MOST emails as the calculation is wrong)
-> The UID returned by SOlr is to be considered as a STRING (and that is maybe the source of problem of the "out of bound" errors in fts_solr dovecot, as "long" is not enough)
-> Java errors : A lot of non sense for me, I am not expert in Java. But, with increased memory, it seems not crashing, even if complaining quite a lot in the logs
-------SCHEMA.XML IN /OPT/SOLR/SERVER/SOLR/DOVECOT/CONF
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="dovecottext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> <fieldType name="dovecotfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="25"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType>
<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="dovecottext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="dovecotfield" indexed="true" stored="false"/> <field name="from" type="dovecotfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="dovecottext" indexed="true" stored="false"/> <field name="to" type="dovecotfield" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
-- DOVECOT.CONF
mail_plugins = fts fts_solr
plugin { plugin = fts fts_solr managesieve sieve
fts = solr fts_autoindex = yes fts_enforced = yes fts_solr = url=http://127.0.0.1:8983/solr/dovecot/
(replace 127.0.0.1 by your solr server if you want to use an external server) (...)
}
-- /ETC/SYSTEMD/SYSTEM/MULTI-USER.TARGET.WANTS/SOLR.SERVICE
[Unit] Description=Solr full text search engine After=network.target
[Service] Type=simple User=solr Group=solr PrivateTmp=yes WorkingDirectory=/opt/solr LIMITNOFILE=65000 LIMITNPROC=65000 ExecStart=/opt/solr/bin/solr start -f
[Install] WantedBy=multi-user.target
Hi,
Op 04/01/2019 om 05:36 schreef Joan Moreau via dovecot:
Hi
This is the summary of my work with SOLR-Dovecot, in my *quest to reproduce the previoulsy excellent work of fts_squat*
@Aki : Based on the time I have spent on this, I would love to see you updating the Wiki with those improvements, and adding my name somewhere
@All : Hope it helps
I'll be going through the description below soon. I've recently independently installed fts-solr from scratch. Although this wasn't a flawless effort, I managed to get some basic indexing going. From this mail thread I understand that there are quite a few more problems than I've seen myself so far. Then again, I didn't perform extensive tests with actual searches.
Maybe we can turn all this into a test suite that we can run internally here at Dovecot. At the very least, the described Dovecot bugs need to be addressed and the wiki needs to be updated.
I'll get back to you.
Regards,
Stephan.
*- Installation:*
-> Create a clean install using the default, (at least in the Archlinux package), and do a "sudo -u solr solr create -c dovecot ". The config files are then in /opt/solr/server/solr/dovecot/conf and datafiles in /opt/solr/server/solr/dovecot/data
-> In /opt/solr/server/solr/dovecot/conf/solrconfig.xml:
* around line 313, change <openSearcher>false</openSearcher> to <openSearcher>true</openSearcher>
* around line 147, set <writeLockTimeout>2000</writeLockTimeout> (or above)
* around line 696 : uncomment <str name="df">hdr</str>
* around line 1127, before <updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/>, add <schemaFactory class="ClassicIndexSchemaFactory"></schemaFactory>
* around line 1161, delete the whole <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields">
* around line 1192, remove the whole <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" ... />
-> Remove /opt/solr/server/solr/dovecot/conf/managed-schema
-> Change "schema.xml" by the one below to reproduce fts_squat behavior (equivalent to " fts_squat = partial=3 full=25" in dovecot.conf) (note : such a huge trouble to replace a single line setup, anyway...)
-> Move /opt/solr/server/solr (or the subfolder data) to a partition with *space*, ideally ext4 or faster file system (it looks like Solr is not considering using a simple mysql database, which would make sense to avoid all the fuzz and let it transit to a non-java state, but that is another story)
-> Config of dovecot.conf is as below
-> The systemd unit shall specify high ulimit for files and proc (see below)
-> Increase the memory available for the JavaVM (I put 12Gb as I have quite a space on my server, but you may adapt it as per your specs) : in /opt/solr/bin/solr.in.sh, set SOLR_HEAP="12288m"
-> As Solr is complaining a lot, you may consider a filter for it in your syslog-ng or journald as it pollutes greatly your audit files
-> (re)Start solr (first) and dovecot by systemctl
-> Launch redindex ( doveadm fts rescan -u <username> )
-> wait for a big while to let the system re-index all your mail boxes
*- Bugs so far*
-> Line 620 of fts_solr dovecot plugin : the size oof header is improperly calculated ("huge header" warning for a simple email, which kilss the index of that considered email, so basically MOST emails as the calculation is wrong)
-> The UID returned by SOlr is to be considered as a STRING (and that is maybe the source of problem of the "out of bound" errors in fts_solr dovecot, as "long" is not enough)
-> Java errors : A lot of non sense for me, I am not expert in Java. But, with increased memory, it seems not crashing, even if complaining quite a lot in the logs
*-------SCHEMA.XML in /opt/solr/server/solr/dovecot/conf*
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="dovecottext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> <fieldType name="dovecotfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="25"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType>
<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="dovecottext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="dovecotfield" indexed="true" stored="false"/> <field name="from" type="dovecotfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="dovecottext" indexed="true" stored="false"/> <field name="to" type="dovecotfield" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
*-- DOVECOT.CONF*
mail_plugins = fts fts_solr
plugin { plugin = fts fts_solr managesieve sieve
fts = solr fts_autoindex = yes fts_enforced = yes fts_solr = url=http://127.0.0.1:8983/solr/dovecot/
(replace 127.0.0.1 by your solr server if you want to use an external server) (...)
}
*-- /etc/systemd/system/multi-user.target.wants/solr.service*
[Unit] Description=Solr full text search engine After=network.target
[Service] Type=simple User=solr Group=solr PrivateTmp=yes WorkingDirectory=/opt/solr *LimitNOFILE=65000* *LimitNPROC=65000* ExecStart=/opt/solr/bin/solr start -f
[Install] WantedBy=multi-user.target
Hi Stephan,
What's up with that ?
Thank you so much
On 2019-01-05 02:04, Stephan Bosch wrote:
Hi,
Op 04/01/2019 om 05:36 schreef Joan Moreau via dovecot:
Hi
This is the summary of my work with SOLR-Dovecot, in my *quest to reproduce the previoulsy excellent work of fts_squat*
@Aki : Based on the time I have spent on this, I would love to see you updating the Wiki with those improvements, and adding my name somewhere
@All : Hope it helps I'll be going through the description below soon. I've recently independently installed fts-solr from scratch. Although this wasn't a flawless effort, I managed to get some basic indexing going. From this mail thread I understand that there are quite a few more problems than I've seen myself so far. Then again, I didn't perform extensive tests with actual searches.
Maybe we can turn all this into a test suite that we can run internally here at Dovecot. At the very least, the described Dovecot bugs need to be addressed and the wiki needs to be updated.
I'll get back to you.
Regards,
Stephan.
*- Installation:*
-> Create a clean install using the default, (at least in the Archlinux package), and do a "sudo -u solr solr create -c dovecot ". The config files are then in /opt/solr/server/solr/dovecot/conf and datafiles in /opt/solr/server/solr/dovecot/data
-> In /opt/solr/server/solr/dovecot/conf/solrconfig.xml:
around line 313, change <openSearcher>false</openSearcher> to <openSearcher>true</openSearcher>
around line 147, set <writeLockTimeout>2000</writeLockTimeout> (or above)
around line 696 : uncomment <str name="df">hdr</str>
around line 1127, before <updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/>, add <schemaFactory class="ClassicIndexSchemaFactory"></schemaFactory>
around line 1161, delete the whole <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields">
around line 1192, remove the whole <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" ... />
-> Remove /opt/solr/server/solr/dovecot/conf/managed-schema
-> Change "schema.xml" by the one below to reproduce fts_squat behavior (equivalent to " fts_squat = partial=3 full=25" in dovecot.conf) (note : such a huge trouble to replace a single line setup, anyway...)
-> Move /opt/solr/server/solr (or the subfolder data) to a partition with *space*, ideally ext4 or faster file system (it looks like Solr is not considering using a simple mysql database, which would make sense to avoid all the fuzz and let it transit to a non-java state, but that is another story)
-> Config of dovecot.conf is as below
-> The systemd unit shall specify high ulimit for files and proc (see below)
-> Increase the memory available for the JavaVM (I put 12Gb as I have quite a space on my server, but you may adapt it as per your specs) : in /opt/solr/bin/solr.in.sh, set SOLR_HEAP="12288m"
-> As Solr is complaining a lot, you may consider a filter for it in your syslog-ng or journald as it pollutes greatly your audit files
-> (re)Start solr (first) and dovecot by systemctl
-> Launch redindex ( doveadm fts rescan -u <username> )
-> wait for a big while to let the system re-index all your mail boxes
*- Bugs so far*
-> Line 620 of fts_solr dovecot plugin : the size oof header is improperly calculated ("huge header" warning for a simple email, which kilss the index of that considered email, so basically MOST emails as the calculation is wrong)
-> The UID returned by SOlr is to be considered as a STRING (and that is maybe the source of problem of the "out of bound" errors in fts_solr dovecot, as "long" is not enough)
-> Java errors : A lot of non sense for me, I am not expert in Java. But, with increased memory, it seems not crashing, even if complaining quite a lot in the logs
*-------SCHEMA.XML in /opt/solr/server/solr/dovecot/conf*
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="dovecottext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> <fieldType name="dovecotfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="25"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType>
<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="dovecottext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="dovecotfield" indexed="true" stored="false"/> <field name="from" type="dovecotfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="dovecottext" indexed="true" stored="false"/> <field name="to" type="dovecotfield" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
*-- DOVECOT.CONF*
mail_plugins = fts fts_solr
plugin { plugin = fts fts_solr managesieve sieve
fts = solr fts_autoindex = yes fts_enforced = yes fts_solr = url=http://127.0.0.1:8983/solr/dovecot/
(replace 127.0.0.1 by your solr server if you want to use an external server) (...)
}
*-- /etc/systemd/system/multi-user.target.wants/solr.service*
[Unit] Description=Solr full text search engine After=network.target
[Service] Type=simple User=solr Group=solr PrivateTmp=yes WorkingDirectory=/opt/solr *LimitNOFILE=65000* *LimitNPROC=65000* ExecStart=/opt/solr/bin/solr start -f
[Install] WantedBy=multi-user.target
Op 14/01/2019 om 07:44 schreef Joan Moreau via dovecot:
Hi Stephan,
What's up with that ?
Thank you so much
Working on it, somewhat anyway.
BTW, did you see this ? :
""" $ sudo -u solr /opt/solr/bin/solr create -c dovecot WARNING: Using _default configset with data driven schema functionality. NOT RECOMMENDED for production use. To turn off: bin/solr config -c dovecot -p 8983 -action set-user-property -property update.autoCreateFields -value false INFO - 2019-01-14 23:19:56.831; org.apache.solr.util.configuration.SSLCredentialProviderFactory; Processing SSL Credential Provider chain: env;sysprop
Created new core 'dovecot' """
I'll be trying your steps first, but the mentioned command might at least get rid of some of the cruft in the default config file.
Regards,
Stephan.
On 2019-01-05 02:04, Stephan Bosch wrote:
Hi,
Op 04/01/2019 om 05:36 schreef Joan Moreau via dovecot:
Hi
This is the summary of my work with SOLR-Dovecot, in my *quest to reproduce the previoulsy excellent work of fts_squat*
@Aki : Based on the time I have spent on this, I would love to see you updating the Wiki with those improvements, and adding my name somewhere
@All : Hope it helps
I'll be going through the description below soon. I've recently independently installed fts-solr from scratch. Although this wasn't a flawless effort, I managed to get some basic indexing going. From this mail thread I understand that there are quite a few more problems than I've seen myself so far. Then again, I didn't perform extensive tests with actual searches.
Maybe we can turn all this into a test suite that we can run internally here at Dovecot. At the very least, the described Dovecot bugs need to be addressed and the wiki needs to be updated.
I'll get back to you.
Regards,
Stephan.
*- Installation:*
-> Create a clean install using the default, (at least in the Archlinux package), and do a "sudo -u solr solr create -c dovecot ". The config files are then in /opt/solr/server/solr/dovecot/conf and datafiles in /opt/solr/server/solr/dovecot/data
-> In /opt/solr/server/solr/dovecot/conf/solrconfig.xml:
* around line 313, change <openSearcher>false</openSearcher> to <openSearcher>true</openSearcher>
* around line 147, set <writeLockTimeout>2000</writeLockTimeout> (or above)
* around line 696 : uncomment <str name="df">hdr</str>
* around line 1127, before <updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/>, add <schemaFactory class="ClassicIndexSchemaFactory"></schemaFactory>
* around line 1161, delete the whole <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields">
* around line 1192, remove the whole <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" ... />
-> Remove /opt/solr/server/solr/dovecot/conf/managed-schema
-> Change "schema.xml" by the one below to reproduce fts_squat behavior (equivalent to " fts_squat = partial=3 full=25" in dovecot.conf) (note : such a huge trouble to replace a single line setup, anyway...)
-> Move /opt/solr/server/solr (or the subfolder data) to a partition with *space*, ideally ext4 or faster file system (it looks like Solr is not considering using a simple mysql database, which would make sense to avoid all the fuzz and let it transit to a non-java state, but that is another story)
-> Config of dovecot.conf is as below
-> The systemd unit shall specify high ulimit for files and proc (see below)
-> Increase the memory available for the JavaVM (I put 12Gb as I have quite a space on my server, but you may adapt it as per your specs) : in /opt/solr/bin/solr.in.sh, set SOLR_HEAP="12288m"
-> As Solr is complaining a lot, you may consider a filter for it in your syslog-ng or journald as it pollutes greatly your audit files
-> (re)Start solr (first) and dovecot by systemctl
-> Launch redindex ( doveadm fts rescan -u <username> )
-> wait for a big while to let the system re-index all your mail boxes
*- Bugs so far*
-> Line 620 of fts_solr dovecot plugin : the size oof header is improperly calculated ("huge header" warning for a simple email, which kilss the index of that considered email, so basically MOST emails as the calculation is wrong)
-> The UID returned by SOlr is to be considered as a STRING (and that is maybe the source of problem of the "out of bound" errors in fts_solr dovecot, as "long" is not enough)
-> Java errors : A lot of non sense for me, I am not expert in Java. But, with increased memory, it seems not crashing, even if complaining quite a lot in the logs
*-------SCHEMA.XML in /opt/solr/server/solr/dovecot/conf*
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="dovecottext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> <fieldType name="dovecotfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="25"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType>
<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="dovecottext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="dovecotfield" indexed="true" stored="false"/> <field name="from" type="dovecotfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="dovecottext" indexed="true" stored="false"/> <field name="to" type="dovecotfield" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
*-- DOVECOT.CONF*
mail_plugins = fts fts_solr
plugin { plugin = fts fts_solr managesieve sieve
fts = solr fts_autoindex = yes fts_enforced = yes fts_solr = url=http://127.0.0.1:8983/solr/dovecot/
(replace 127.0.0.1 by your solr server if you want to use an external server) (...)
}
*-- /etc/systemd/system/multi-user.target.wants/solr.service*
[Unit] Description=Solr full text search engine After=network.target
[Service] Type=simple User=solr Group=solr PrivateTmp=yes WorkingDirectory=/opt/solr *LimitNOFILE=65000* *LimitNPROC=65000* ExecStart=/opt/solr/bin/solr start -f
[Install] WantedBy=multi-user.target
Yes, the " -property update.autoCreateFields -value false " seems interesting
However, we smash the created schema just after
On 2019-01-14 23:25, Stephan Bosch wrote:
Op 14/01/2019 om 07:44 schreef Joan Moreau via dovecot:
Hi Stephan,
What's up with that ?
Thank you so much
Working on it, somewhat anyway.
BTW, did you see this ? :
""" $ sudo -u solr /opt/solr/bin/solr create -c dovecot WARNING: Using _default configset with data driven schema functionality. NOT RECOMMENDED for production use. To turn off: bin/solr config -c dovecot -p 8983 -action set-user-property -property update.autoCreateFields -value false INFO - 2019-01-14 23:19:56.831; org.apache.solr.util.configuration.SSLCredentialProviderFactory; Processing SSL Credential Provider chain: env;sysprop
Created new core 'dovecot' """
I'll be trying your steps first, but the mentioned command might at least get rid of some of the cruft in the default config file.
Regards,
Stephan.
On 2019-01-05 02:04, Stephan Bosch wrote:
Hi,
Op 04/01/2019 om 05:36 schreef Joan Moreau via dovecot: Hi
This is the summary of my work with SOLR-Dovecot, in my *quest to reproduce the previoulsy excellent work of fts_squat*
@Aki : Based on the time I have spent on this, I would love to see you updating the Wiki with those improvements, and adding my name somewhere
@All : Hope it helps
I'll be going through the description below soon. I've recently independently installed fts-solr from scratch. Although this wasn't a flawless effort, I managed to get some basic indexing going. From this mail thread I understand that there are quite a few more problems than I've seen myself so far. Then again, I didn't perform extensive tests with actual searches.
Maybe we can turn all this into a test suite that we can run internally here at Dovecot. At the very least, the described Dovecot bugs need to be addressed and the wiki needs to be updated.
I'll get back to you.
Regards,
Stephan.
*- Installation:*
-> Create a clean install using the default, (at least in the Archlinux package), and do a "sudo -u solr solr create -c dovecot ". The config files are then in /opt/solr/server/solr/dovecot/conf and datafiles in /opt/solr/server/solr/dovecot/data
-> In /opt/solr/server/solr/dovecot/conf/solrconfig.xml:
around line 313, change <openSearcher>false</openSearcher> to <openSearcher>true</openSearcher>
around line 147, set <writeLockTimeout>2000</writeLockTimeout> (or above)
around line 696 : uncomment <str name="df">hdr</str>
around line 1127, before <updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/>, add <schemaFactory class="ClassicIndexSchemaFactory"></schemaFactory>
around line 1161, delete the whole <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields">
around line 1192, remove the whole <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" ... />
-> Remove /opt/solr/server/solr/dovecot/conf/managed-schema
-> Change "schema.xml" by the one below to reproduce fts_squat behavior (equivalent to " fts_squat = partial=3 full=25" in dovecot.conf) (note : such a huge trouble to replace a single line setup, anyway...)
-> Move /opt/solr/server/solr (or the subfolder data) to a partition with *space*, ideally ext4 or faster file system (it looks like Solr is not considering using a simple mysql database, which would make sense to avoid all the fuzz and let it transit to a non-java state, but that is another story)
-> Config of dovecot.conf is as below
-> The systemd unit shall specify high ulimit for files and proc (see below)
-> Increase the memory available for the JavaVM (I put 12Gb as I have quite a space on my server, but you may adapt it as per your specs) : in /opt/solr/bin/solr.in.sh, set SOLR_HEAP="12288m"
-> As Solr is complaining a lot, you may consider a filter for it in your syslog-ng or journald as it pollutes greatly your audit files
-> (re)Start solr (first) and dovecot by systemctl
-> Launch redindex ( doveadm fts rescan -u <username> )
-> wait for a big while to let the system re-index all your mail boxes
*- Bugs so far*
-> Line 620 of fts_solr dovecot plugin : the size oof header is improperly calculated ("huge header" warning for a simple email, which kilss the index of that considered email, so basically MOST emails as the calculation is wrong)
-> The UID returned by SOlr is to be considered as a STRING (and that is maybe the source of problem of the "out of bound" errors in fts_solr dovecot, as "long" is not enough)
-> Java errors : A lot of non sense for me, I am not expert in Java. But, with increased memory, it seems not crashing, even if complaining quite a lot in the logs
*-------SCHEMA.XML in /opt/solr/server/solr/dovecot/conf*
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="dovecottext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> <fieldType name="dovecotfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="25"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType>
<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="dovecottext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="dovecotfield" indexed="true" stored="false"/> <field name="from" type="dovecotfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="dovecottext" indexed="true" stored="false"/> <field name="to" type="dovecotfield" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
*-- DOVECOT.CONF*
mail_plugins = fts fts_solr
plugin { plugin = fts fts_solr managesieve sieve
fts = solr fts_autoindex = yes fts_enforced = yes fts_solr = url=http://127.0.0.1:8983/solr/dovecot/
(replace 127.0.0.1 by your solr server if you want to use an external server) (...)
}
*-- /etc/systemd/system/multi-user.target.wants/solr.service*
[Unit] Description=Solr full text search engine After=network.target
[Service] Type=simple User=solr Group=solr PrivateTmp=yes WorkingDirectory=/opt/solr *LimitNOFILE=65000* *LimitNPROC=65000* ExecStart=/opt/solr/bin/solr start -f
[Install] WantedBy=multi-user.target
Hi Joan,
Op 14/01/2019 om 07:44 schreef Joan Moreau via dovecot:
Hi Stephan,
What's up with that ?
Thank you so much
On 2019-01-05 02:04, Stephan Bosch wrote:
Hi,
Op 04/01/2019 om 05:36 schreef Joan Moreau via dovecot:
Hi
This is the summary of my work with SOLR-Dovecot, in my *quest to reproduce the previoulsy excellent work of fts_squat*
@Aki : Based on the time I have spent on this, I would love to see you updating the Wiki with those improvements, and adding my name somewhere
@All : Hope it helps
*- Installation:*
-> Create a clean install using the default, (at least in the Archlinux package), and do a "sudo -u solr solr create -c dovecot ". The config files are then in /opt/solr/server/solr/dovecot/conf and datafiles in /opt/solr/server/solr/dovecot/data
On my system (Debian) these directories are wildly different (e.g. data is under /var), but other than that, this information is OK.
Used this as a side-reference for Debian installation: https://tecadmin.net/install-apache-solr-on-debian/
Accessed http://solr-host.tld:8983/solr/ to check whether all is OK.
-> In /opt/solr/server/solr/dovecot/conf/solrconfig.xml:
* around line 313, change <openSearcher>false</openSearcher> to <openSearcher>true</openSearcher>
* around line 147, set <writeLockTimeout>2000</writeLockTimeout> (or above)
* around line 696 : uncomment <str name="df">hdr</str>
* around line 1127, before <updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/>, add <schemaFactory class="ClassicIndexSchemaFactory"></schemaFactory>
* around line 1161, delete the whole <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields">
* around line 1192, remove the whole <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" ... />
Applied these changes. We should probably provide an example config file on the Wiki that incorporates all this.. or maybe a diff.
We also need to evaluate what the merit of all of this is. I did something similar in my previous effort, but it was all based on getting an error from Solr and then removing that section of the config file with the assumption it wasn't needed. So far, I have little clue what these things are and why these things are enabled by default. As I said in an earlier mail, there is an option to leave some of this cruft out at backend initialization, but I haven't tried that yet.
-> Remove /opt/solr/server/solr/dovecot/conf/managed-schema
-> Change "schema.xml" by the one below to reproduce fts_squat behavior (equivalent to " fts_squat = partial=3 full=25" in dovecot.conf) (note : such a huge trouble to replace a single line setup, anyway...)
Did that too.
-> Move /opt/solr/server/solr (or the subfolder data) to a partition with *space*, ideally ext4 or faster file system (it looks like Solr is not considering using a simple mysql database, which would make sense to avoid all the fuzz and let it transit to a non-java state, but that is another story)
Skipped that.
-> Config of dovecot.conf is as below
I also enabled debug for fts_solr.
-> The systemd unit shall specify high ulimit for files and proc (see below)
Debian does something weird here. It doesn't use an explicit systemd unit. It is generated from the SysV init file. I ended up setting the ulimits in /etc/security/limits.conf for user solr.
-> Increase the memory available for the JavaVM (I put 12Gb as I have quite a space on my server, but you may adapt it as per your specs) : in /opt/solr/bin/solr.in.sh, set SOLR_HEAP="12288m"
Skipped that.
-> As Solr is complaining a lot, you may consider a filter for it in your syslog-ng or journald as it pollutes greatly your audit files
What does it complain about and when does it happen? I haven't seen much logging from Solr so far.
-> (re)Start solr (first) and dovecot by systemctl
-> Launch redindex ( doveadm fts rescan -u <username> )
-> wait for a big while to let the system re-index all your mail boxes
Weirdly, rescan returns immediately here. When I perform doveadm index INBOX for my test user, I do see a lot of fts and HTTP activity.
*- Bugs so far*
-> Line 620 of fts_solr dovecot plugin : the size oof header is improperly calculated ("huge header" warning for a simple email, which kilss the index of that considered email, so basically MOST emails as the calculation is wrong)
-> The UID returned by SOlr is to be considered as a STRING (and that is maybe the source of problem of the "out of bound" errors in fts_solr dovecot, as "long" is not enough)
-> Java errors : A lot of non sense for me, I am not expert in Java. But, with increased memory, it seems not crashing, even if complaining quite a lot in the logs
Can you elaborate on the errors you have seen so far? When do these happen? How can I reproduce them?
Regards,
Stephan.
*-------SCHEMA.XML in /opt/solr/server/solr/dovecot/conf*
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="dovecottext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> <fieldType name="dovecotfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="25"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType>
<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="dovecottext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="dovecotfield" indexed="true" stored="false"/> <field name="from" type="dovecotfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="dovecottext" indexed="true" stored="false"/> <field name="to" type="dovecotfield" indexed="true" stored="false"/> <field name="uid" type="string" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
*-- DOVECOT.CONF*
mail_plugins = fts fts_solr
plugin { plugin = fts fts_solr managesieve sieve
fts = solr fts_autoindex = yes fts_enforced = yes fts_solr = url=http://127.0.0.1:8983/solr/dovecot/
(replace 127.0.0.1 by your solr server if you want to use an external server) (...)
}
*-- /etc/systemd/system/multi-user.target.wants/solr.service*
[Unit] Description=Solr full text search engine After=network.target
[Service] Type=simple User=solr Group=solr PrivateTmp=yes WorkingDirectory=/opt/solr *LimitNOFILE=65000* *LimitNPROC=65000* ExecStart=/opt/solr/bin/solr start -f
[Install] WantedBy=multi-user.target
On Sat, Jan 26, 2019 at 01:44:16PM +0100, Stephan Bosch wrote:
Hi Joan,
Op 14/01/2019 om 07:44 schreef Joan Moreau via dovecot:
Hi Stephan,
What's up with that ?
Thank you so much
On 2019-01-05 02:04, Stephan Bosch wrote:
Hi,
Op 04/01/2019 om 05:36 schreef Joan Moreau via dovecot:
... ...
-> The systemd unit shall specify high ulimit for files and proc (see below)
Debian does something weird here. It doesn't use an explicit systemd unit. It is generated from the SysV init file. I ended up setting the ulimits in /etc/security/limits.conf for user solr.
Please make sure the changes you make don't make your Debian package *require* systemd. There are Debian-derived distros that avoid systemd.
-- hendrik
Op 26/01/2019 om 15:24 schreef Hendrik Boom:
On Sat, Jan 26, 2019 at 01:44:16PM +0100, Stephan Bosch wrote:
Hi Joan,
Op 14/01/2019 om 07:44 schreef Joan Moreau via dovecot:
Hi Stephan,
What's up with that ?
Thank you so much
On 2019-01-05 02:04, Stephan Bosch wrote:
Debian does something weird here. It doesn't use an explicit systemd unit. It is generated from the SysV init file. I ended up setting the ulimits in /etc/security/limits.conf for user solr. Please make sure the changes you make don't make your Debian package *require* systemd. There are Debian-derived distros that avoid systemd.
Don't worry, I am not working on packaging this. I just want to know what the problems are and how these can be solved, so that we can update the wiki.
Regards,
Stephan.
*- Installation:*
-> Create a clean install using the default, (at least in the Archlinux package), and do a "sudo -u solr solr create -c dovecot ". The config files are then in /opt/solr/server/solr/dovecot/conf and datafiles in /opt/solr/server/solr/dovecot/data
On my system (Debian) these directories are wildly different (e.g. data is under /var), but other than that, this information is OK.
Used this as a side-reference for Debian installation: https://tecadmin.net/install-apache-solr-on-debian/
Accessed http://solr-host.tld:8983/solr/ to check whether all is OK.
MAKE SURE YOU HAVE A DOVECOT INSTANCE (NOT THE DEFAULT INSTANCE) , WITH THE FUNCTION BELOW:
SOLR CREATE -C DOVECOT (OR WHATEVER NAME)
Weirdly, rescan returns immediately here. When I perform
doveadm index INBOXfor my test user, I do see a lot of fts and HTTP activity.
THE SOLR PLUGIN IS NOT CODED ENTIRELY, REFRESH AND RESCAN FUNCTIONS ARE MISSING :
https://github.com/dovecot/core/blob/master/src/plugins/fts-solr/fts-backend...
static int fts_backend_solr_refresh(struct fts_backend *backend ATTR_UNUSED) { return 0; }
static int fts_backend_solr_rescan(struct fts_backend *backend) { /* FIXME: proper rescan needed. for now we'll just reset the last-uids */ return fts_backend_reset_last_uids(backend); }
*- Bugs so far*
-> Line 620 of fts_solr dovecot plugin : the size oof header is improperly calculated ("huge header" warning for a simple email, which kilss the index of that considered email, so basically MOST emails as the calculation is wrong)
YOU CAN CHECK THAT REGULARLY IN DOVECOT LOG FILE. MY GUESS IS THE MIX OF UNICODE WHICH IS NOT PROPERLY ADDRESSED HERE.
-> The UID returned by SOlr is to be considered as a STRING (and that is maybe the source of problem of the "out of bound" errors in fts_solr dovecot, as "long" is not enough)
THIS IS JUST HIGHLY VISIBLE IN SOLR SCHEMA.XML. SWITHCING IT TO "LONG" IN SCHEMA.XML RETURNS PLENTY OF ERRORS.
-> Java errors : A lot of non sense for me, I am not expert in Java. But, with increased memory, it seems not crashing, even if complaining quite a lot in the logs
Can you elaborate on the errors you have seen so far? When do these happen? How can I reproduce them?
HONESTLY, I HAVE NO CLUE WHAT THE PROBLEMS ARE. I JUST INCREASED THE MEMORY OF THE JVM AND THE SYSTEMS STOPPED CRASHING. LOG FILES ARE HUGE ANYWAY.
(forgot to CC mailing list)
Op 26/01/2019 om 20:07 schreef Joan Moreau via dovecot:
*- Bugs so far*
-> Line 620 of fts_solr dovecot plugin : the size oof header is improperly calculated ("huge header" warning for a simple email, which kilss the index of that considered email, so basically MOST emails as the calculation is wrong)
*You can check that regularly in dovecot log file. My guess is the mix of Unicode which is not properly addressed here.*
Does this happen with specific messages? Do you have a sample message for me? I don't see how Unicode could cause this.
-> The UID returned by SOlr is to be considered as a STRING (and that is maybe the source of problem of the "out of bound" errors in fts_solr dovecot, as "long" is not enough)
*This is just highly visible in Solr schema.xml. Swithcing it to "long" in schema.xml returns plenty of errors.*
I cannot reproduce this so far (see modified schema below). In a simple test I just get the desired results and no errors logged.
-> Java errors : A lot of non sense for me, I am not expert in Java. But, with increased memory, it seems not crashing, even if complaining quite a lot in the logs
Can you elaborate on the errors you have seen so far? When do these happen? How can I reproduce them?
*Honestly, I have no clue what the problems are. I just increased the memory of the JVM and the systems stopped crashing. Log files are huge anyway.*
What errors do you see? I see only INFO entries in my /var/solr/logs/solr.log. Looks like Solr is pretty verbose by default (lots of INFO output), but there must be a way to reduce that.
Regards,
Stephan.
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="long" class="solr.LongPointField" positionIncrementGap="0"/> <fieldType name="dovecottext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> <fieldType name="dovecotfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="25"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType>
<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="dovecottext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="dovecotfield" indexed="true" stored="false"/> <field name="from" type="dovecotfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="dovecottext" indexed="true" stored="false"/> <field name="to" type="dovecotfield" indexed="true" stored="false"/> <field name="uid" type="long" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
On 2019-01-30 07:33, Stephan Bosch wrote:
(forgot to CC mailing list)
Op 26/01/2019 om 20:07 schreef Joan Moreau via dovecot:
*- Bugs so far*
-> Line 620 of fts_solr dovecot plugin : the size oof header is improperly calculated ("huge header" warning for a simple email, which kilss the index of that considered email, so basically MOST emails as the calculation is wrong) *You can check that regularly in dovecot log file. My guess is the mix of Unicode which is not properly addressed here.*
Does this happen with specific messages? Do you have a sample message for me? I don't see how Unicode could cause this.
MY ONLY GUESS IS THAT IT REFERS TO SOME 'STRLEN', WHICH IS WRONG OF COURSE IN CASE OF UNICODE EMAILS. THIS IS JUST A GUESS.
BUT DO A GREP FOR "HUGE" IN THE DOVECOT LOG OF A BUSY SERVER TO FIND EXAMPLES.
(SORRY, I SWITCHED TO XAPIAN, AS SOLR IS CREATING TOO MUCH TROUBLES FOR MY SERVER, SO NO MORE CONCRETE EXAMPLE)
-> The UID returned by SOlr is to be considered as a STRING (and that is maybe the source of problem of the "out of bound" errors in fts_solr dovecot, as "long" is not enough) *This is just highly visible in Solr schema.xml. Swithcing it to "long" in schema.xml returns plenty of errors.*
I cannot reproduce this so far (see modified schema below). In a simple test I just get the desired results and no errors logged.
I got this with large mailboxes (where UID seems not acceptable for Solr ). The fault is not on Dovecot side but Solr, and the returned UID(s) for a search is garbage instead of a proper value -> Putting it as string solves this
-> Java errors : A lot of non sense for me, I am not expert in Java. But, with increased memory, it seems not crashing, even if complaining quite a lot in the logs
Can you elaborate on the errors you have seen so far? When do these happen? How can I reproduce them? *Honestly, I have no clue what the problems are. I just increased the memory of the JVM and the systems stopped crashing. Log files are huge anyway.*
What errors do you see? I see only INFO entries in my /var/solr/logs/solr.log. Looks like Solr is pretty verbose by default (lots of INFO output), but there must be a way to reduce that.
I DELETED SOLR. NO MORE LOGS. MAYBE SOMEONE ELSE CAN TELL.
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <fieldType name="long" class="solr.LongPointField" positionIncrementGap="0"/> <fieldType name="dovecottext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> <fieldType name="dovecotfield" class="solr.TextField" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.ClassicTokenizerFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="25"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.TrimFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType>
<fieldType name="string" class="solr.StrField"/> <field name="_version_" type="string" indexed="true" stored="true"/> <field name="bcc" type="string" indexed="false" stored="false"/> <field name="body" type="dovecottext" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="dovecotfield" indexed="true" stored="false"/> <field name="from" type="dovecotfield" indexed="true" stored="false"/> <field name="hdr" type="string" indexed="false" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="dovecottext" indexed="true" stored="false"/> <field name="to" type="dovecotfield" indexed="true" stored="false"/> <field name="uid" type="long" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema>
On 1/3/2019, 3:07:18 PM, Daniel Miller via dovecot <dovecot@dovecot.org> wrote:
On 1/3/2019 10:56 AM, Tanstaafl wrote:
On 12/21/2018, 11:19:42 AM, Daniel Miller via dovecot <dovecot@dovecot.org> wrote:
There is a *huge* difference between a functional Solr setup & squat Interesting. Care to elaborate?
This is one of those things that has to be experienced to be understood. When you can perform an FTS search across (pause while I check current stats...):
du -c -h /var/mail 136G
Solr numDocs: 520102
and using any IMAP client that supports server-side searches (like Thunderbird & AquaMail) the results are basically instantaneous...it's worth the effort. And that's searching a Dovecot virtual folder defined as "* all", including all my archives, all my list subscriptions, and all the shared Inbox/Sent folders from my other users.
But I certainly wish it was easier to setup.
Thanks Daniel...
So, as one who has no experience of the benefit of either...
How does this compare with Squat? Meaning, Is it exponentially faster? Twice as fast?
On 1/5/2019 9:58 AM, Tanstaafl wrote:
Thanks Daniel...
So, as one who has no experience of the benefit of either...
How does this compare with Squat? Meaning, Is it exponentially faster? Twice as fast?
It's been many years since I last had a Squat setup - but that's my memory.
-- Daniel
Additionally, here the errors I get in logs:
Dovecot:
Dec 09 09:21:09 imap(jom@grosjo.net)<3349><DiRnXpN8Lux/AAAB>: Error: fts_solr: received invalid uid '0' Dec 09 09:21:10 imap(jom@grosjo.net)<3349><DiRnXpN8Lux/AAAB>: Error: fts_solr: received invalid uid '0'
or
11 03:36:03 indexer-worker(jom@grosjo.net)<2093><icWMJaIwD1znEgAA0thIag:GPBOCKMwD1wtCAAA0thIag>: Error: fts_solr: Indexing failed: 500 Server Error
Solr:
CAUSED BY: ORG.APACHE.LUCENE.STORE.ALREADYCLOSEDEXCEPTION: THIS INDEXWRITER IS CLOSED Dec 11 06:00:14 gjserver solr[16761]: at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:679) Dec 11 06:00:14 gjserver solr[16761]: at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:693) Dec 11 06:00:14 gjserver solr[16761]: at org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:4929) Dec 11 06:00:14 gjserver solr[16761]: at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:290) Dec 11 06:00:14 gjserver solr[16761]: at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:279) Dec 11 06:00:14 gjserver solr[16761]: at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:235) Dec 11 06:00:14 gjserver solr[16761]: at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2039) Dec 11 06:00:14 gjserver solr[16761]: ... 51 more DEC 11 06:00:14 GJSERVER SOLR[16761]: CAUSED BY: JAVA.LANG.ARRAYINDEXOUTOFBOUNDSEXCEPTION: -65536 Dec 11 06:00:14 gjserver solr[16761]: at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:198) Dec 11 06:00:14 gjserver solr[16761]: at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:224) Dec 11 06:00:14 gjserver solr[16761]: at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:80) Dec 11 06:00:14 gjserver solr[16761]: at org.apache.lucene.index.FreqProxTermsWriterPerField.addTerm(FreqProxTermsWriterPerField.java:184) Dec 11 06:00:14 gjserver solr[16761]: at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:185)
On 2018-12-10 21:40, Joan Moreau wrote:
Hi Daniel,
THere is no need of all this, just the command (on Solr 7.5) "create -c dovecot " is enough
The chema.xml provided on the wiki basically does not work on 7.5
Here the latest one I am working on , but nothing works properly (bad search results, errors in ftp_solr, etc..)
<?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="long" class="solr.LongPointField" positionIncrementGap="0" /> <fieldType name="text" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="_version_" type="long" indexed="true" stored="true"/> <field name="bcc" type="text" indexed="true" stored="false"/> <field name="body" type="text" indexed="true" stored="false"/> <field name="box" type="string" indexed="false" required="true" stored="true"/> <field name="hdr" type="text" indexed="true" stored="false"/> <field name="cc" type="text" indexed="true" stored="false"/> <field name="from" type="text" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="text" indexed="true" stored="false"/> <field name="to" type="text" indexed="true" stored="false"/> <field name="uid" type="long" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </fields> </schema>
On 2018-12-10 21:17, Daniel Miller via dovecot wrote: On 12/4/2018 10:40 AM, Joan Moreau via dovecot wrote:
In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command :
sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance
before updating the schema.xml .
Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users
Additionaly, the url is http://(solr_ [1]server):8983/solr/dovecot/ (error in wiki)
After installing Solr, wherever the installation sets up there should a folder similar to:
<your prefix>/solr/server/solr/configsets
If you look there, you'll probably see folders like '_default' and 'sample_techproducts_configs'. I haven't played with the 'techproducts' sample. Copy the '_default' folder, with all its contents, to a 'dovecot' folder. In the new dovecot folder, replace the 'managed-schema' file with the file from the Dovecot Wiki
https://wiki.dovecot.org/Plugins/FTS/Solr?action=AttachFile&do=view&target=solr-7.x-schema.xml
after that, you should be able to run 'solr /opt/solr/bin/solr create -c dovecot' to create the instance. If things still don't work let us know.
The schema is one I've tweaked and updated during my own migrations since Solr 3.3. It's possible there's something else in my config that needs documenting - but having experienced Solr search against my mailstore I never want to be without it.
Daniel
Links:
[1] http://(solr
On 12/10/2018 10:02 PM, Joan Moreau wrote:
Additionally, here the errors I get in logs:
Dovecot:
Dec 09 09:21:09 imap(jom@grosjo.net)<3349><DiRnXpN8Lux/AAAB>: Error: fts_solr: received invalid uid '0' Dec 09 09:21:10 imap(jom@grosjo.net)<3349><DiRnXpN8Lux/AAAB>: Error: fts_solr: received invalid uid '0'
or
11 03:36:03 indexer-worker(jom@grosjo.net)<2093><icWMJaIwD1znEgAA0thIag:GPBOCKMwD1wtCAAA0thIag>: Error: fts_solr: Indexing failed: 500 Server Error
This looks like a permissions issue. Are you using NFS?
-- Daniel
On Fri, Nov 23, 2018 at 02:29:22PM +0200, Timo Sirainen wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
* The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used. * "doveadm stats dump" now returns two decimals for the "avg" field.
+ Added push notification driver that uses a Lua script + Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events + Added "doveadm mailbox cache purge" command. + Added events API support for Lua scripts + doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version. - director: Kicking a user crashes if login process is very slow - pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent. - auth: Fix crypt() segfault with glibc-2.28+ - imap: Running UID FILTER script with errors assert-crashes - dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved. - dict clients may have been using 100% CPU while waiting for dict server to finish commands. - doveadm user: Fixed user listing via HTTP API - All levels of Cassandra log messages were logged as Dovecot errors. - http/smtp client may have crashed after SSL handshake - Lua auth converted strings that looked like numbers into numbers.
The release does not build. Here is a patch to fix the build. test-event-stats.c:101:8: warning: implicit declaration of function 'kill' is invalid in C99 [-Wimplicit-function-declaration] (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: error: use of undeclared identifier 'SIGKILL' (void)kill(stats_pid, SIGKILL); ^ Index: src/lib-master/test-event-stats.c --- src/lib-master/test-event-stats.c.orig +++ src/lib-master/test-event-stats.c @@ -12,6 +12,7 @@ #include "stats-client.h" #include "test-common.h" #include <fcntl.h> +#include <signal.h> #include <unistd.h> #include <sys/socket.h> #include <sys/un.h>
On 23.11.2018 15.20, Brad Smith wrote:
On Fri, Nov 23, 2018 at 02:29:22PM +0200, Timo Sirainen wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
- The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
- "doveadm stats dump" now returns two decimals for the "avg" field.
- Added push notification driver that uses a Lua script
- Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
- Added "doveadm mailbox cache purge" command.
- Added events API support for Lua scripts
- doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.
- director: Kicking a user crashes if login process is very slow
- pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
- auth: Fix crypt() segfault with glibc-2.28+
- imap: Running UID FILTER script with errors assert-crashes
- dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
- dict clients may have been using 100% CPU while waiting for dict server to finish commands.
- doveadm user: Fixed user listing via HTTP API
- All levels of Cassandra log messages were logged as Dovecot errors.
- http/smtp client may have crashed after SSL handshake
- Lua auth converted strings that looked like numbers into numbers. The release does not build. Here is a patch to fix the build.
test-event-stats.c:101:8: warning: implicit declaration of function 'kill' is invalid in C99 [-Wimplicit-function-declaration] (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: error: use of undeclared identifier 'SIGKILL' (void)kill(stats_pid, SIGKILL); ^
On *BSD I assume? It would be useful to mention. But we'll think what we will do with this.
Aki
On 11/23/2018 8:23 AM, Aki Tuomi wrote:
On 23.11.2018 15.20, Brad Smith wrote:
On Fri, Nov 23, 2018 at 02:29:22PM +0200, Timo Sirainen wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
- The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
- "doveadm stats dump" now returns two decimals for the "avg" field.
- Added push notification driver that uses a Lua script
- Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
- Added "doveadm mailbox cache purge" command.
- Added events API support for Lua scripts
- doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.
- director: Kicking a user crashes if login process is very slow
- pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
- auth: Fix crypt() segfault with glibc-2.28+
- imap: Running UID FILTER script with errors assert-crashes
- dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
- dict clients may have been using 100% CPU while waiting for dict server to finish commands.
- doveadm user: Fixed user listing via HTTP API
- All levels of Cassandra log messages were logged as Dovecot errors.
- http/smtp client may have crashed after SSL handshake
- Lua auth converted strings that looked like numbers into numbers. The release does not build. Here is a patch to fix the build.
test-event-stats.c:101:8: warning: implicit declaration of function 'kill' is invalid in C99 [-Wimplicit-function-declaration] (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: error: use of undeclared identifier 'SIGKILL' (void)kill(stats_pid, SIGKILL); ^ On *BSD I assume? It would be useful to mention. But we'll think what we will do with this.
Yes, OpenBSD.
On 23.11.2018 15.51, Brad Smith wrote:
On 11/23/2018 8:23 AM, Aki Tuomi wrote:
On 23.11.2018 15.20, Brad Smith wrote:
On Fri, Nov 23, 2018 at 02:29:22PM +0200, Timo Sirainen wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
* The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used. * "doveadm stats dump" now returns two decimals for the "avg" field.
+ Added push notification driver that uses a Lua script + Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events + Added "doveadm mailbox cache purge" command. + Added events API support for Lua scripts + doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version. - director: Kicking a user crashes if login process is very slow - pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent. - auth: Fix crypt() segfault with glibc-2.28+ - imap: Running UID FILTER script with errors assert-crashes - dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved. - dict clients may have been using 100% CPU while waiting for dict server to finish commands. - doveadm user: Fixed user listing via HTTP API - All levels of Cassandra log messages were logged as Dovecot errors. - http/smtp client may have crashed after SSL handshake - Lua auth converted strings that looked like numbers into numbers. The release does not build. Here is a patch to fix the build.
test-event-stats.c:101:8: warning: implicit declaration of function 'kill' is invalid in C99 [-Wimplicit-function-declaration] (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: error: use of undeclared identifier 'SIGKILL' (void)kill(stats_pid, SIGKILL); ^ On *BSD I assume? It would be useful to mention. But we'll think what we will do with this.
Yes, OpenBSD.
There is a fix pending now for master, and will be released on next release.
Aki
I installed 2.3.4 and just used it with the config files for 2.3.3 without changing anything in the configuration. I then realized that the LDA was throwing errors.
2018-11-24 00:02:51 1gQIaw-000AZS-Bc </var/spool/virtual/ crownkenya.com/john.doe/Maildir>: dovecot_virtual_delivery transport output: lda(john.doe@our.domain.name)Error: net_connect_unix(/var/run/dovecot//stats-writer) failed: Permission denied
I checked on the presence of the sockets in /var/run/dovecot:
srw------- 1 root wheel 0 Nov 24 09:07 stats-reader srw-rw---- 1 root dovecot 0 Nov 24 09:07 stats-writer
I have tried to find any mention of stats-{writer|reader} in the example configs shipped with 2.3.4 and found nothing. I have backed-off 2.3.4 for now till I can figure out how to assign proper permissions to these sockets
- or just to figure out why by default, permission is being denied.
On Fri, 23 Nov 2018 at 16:53, Aki Tuomi <aki.tuomi@open-xchange.com> wrote:
On 23.11.2018 15.51, Brad Smith wrote:
On 11/23/2018 8:23 AM, Aki Tuomi wrote:
On 23.11.2018 15.20, Brad Smith wrote:
On Fri, Nov 23, 2018 at 02:29:22PM +0200, Timo Sirainen wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
- The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
- "doveadm stats dump" now returns two decimals for the "avg" field.
- Added push notification driver that uses a Lua script
- Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
- Added "doveadm mailbox cache purge" command.
- Added events API support for Lua scripts
- doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.
- director: Kicking a user crashes if login process is very slow
- pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
- auth: Fix crypt() segfault with glibc-2.28+
- imap: Running UID FILTER script with errors assert-crashes
- dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
- dict clients may have been using 100% CPU while waiting for dict server to finish commands.
- doveadm user: Fixed user listing via HTTP API
- All levels of Cassandra log messages were logged as Dovecot errors.
- http/smtp client may have crashed after SSL handshake
- Lua auth converted strings that looked like numbers into numbers. The release does not build. Here is a patch to fix the build.
test-event-stats.c:101:8: warning: implicit declaration of function 'kill' is invalid in C99 [-Wimplicit-function-declaration] (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: error: use of undeclared identifier 'SIGKILL' (void)kill(stats_pid, SIGKILL); ^ On *BSD I assume? It would be useful to mention. But we'll think what we will do with this.
Yes, OpenBSD.
There is a fix pending now for master, and will be released on next release.
Aki
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."
On 24 Nov 2018, at 8.33, Odhiambo Washington <odhiambo@gmail.com> wrote:
I installed 2.3.4 and just used it with the config files for 2.3.3 without changing anything in the configuration. I then realized that the LDA was throwing errors.
2018-11-24 00:02:51 1gQIaw-000AZS-Bc </var/spool/virtual/crownkenya.com/john.doe/Maildir <http://crownkenya.com/john.doe/Maildir>>: dovecot_virtual_delivery transport output: lda(john.doe@our.domain.name <mailto:john.doe@our.domain.name>)Error: net_connect_unix(/var/run/dovecot//stats-writer) failed: Permission denied
I checked on the presence of the sockets in /var/run/dovecot:
srw------- 1 root wheel 0 Nov 24 09:07 stats-reader srw-rw---- 1 root dovecot 0 Nov 24 09:07 stats-writer
What user/group does dovecot_virtual_delivery run as? Change the stats-writer socket's owner to be that user. For example:
service stats { unix_listener stats-writer { user = vmail } }
Or alternatively change dovecot_virtual_delivery to use dovecot group.
I have tried to find any mention of stats-{writer|reader} in the example configs shipped with 2.3.4 and found nothing. I have backed-off 2.3.4 for now till I can figure out how to assign proper permissions to these sockets - or just to figure out why by default, permission is being denied.
Looks like this is happening now because in earlier versions the dovecot-lda process wasn't sending any statistics.
On Sat, 24 Nov 2018 at 13:29, Timo Sirainen <tss@iki.fi> wrote:
On 24 Nov 2018, at 8.33, Odhiambo Washington <odhiambo@gmail.com> wrote:
I installed 2.3.4 and just used it with the config files for 2.3.3 without changing anything in the configuration. I then realized that the LDA was throwing errors.
2018-11-24 00:02:51 1gQIaw-000AZS-Bc </var/spool/virtual/ crownkenya.com/john.doe/Maildir>: dovecot_virtual_delivery transport output: lda(john.doe@our.domain.name)Error: net_connect_unix(/var/run/dovecot//stats-writer) failed: Permission denied
I checked on the presence of the sockets in /var/run/dovecot:
srw------- 1 root wheel 0 Nov 24 09:07 stats-reader srw-rw---- 1 root dovecot 0 Nov 24 09:07 stats-writer
What user/group does dovecot_virtual_delivery run as?
It runs as the MTA user - mailnull.
Change the stats-writer socket's owner to be that user. For example:
service stats { unix_listener stats-writer { user = vmail } }
That makes sense.
Or alternatively change dovecot_virtual_delivery to use dovecot group.
That would not be possible because it will not have permissions to write the mail files.
I have tried to find any mention of stats-{writer|reader} in the example configs shipped with 2.3.4 and found nothing. I have backed-off 2.3.4 for now till I can figure out how to assign proper permissions to these sockets
- or just to figure out why by default, permission is being denied.
Looks like this is happening now because in earlier versions the dovecot-lda process wasn't sending any statistics.
BTW, was it an oversight that this config snippet is not documented with the example-config files? Or on the Wiki? Or my eyes are failing me? :-)
Thank you very much.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."
On Fri, 23 Nov 2018 at 16:23, Aki Tuomi <aki.tuomi@open-xchange.com> wrote:
On 23.11.2018 15.20, Brad Smith wrote:
On Fri, Nov 23, 2018 at 02:29:22PM +0200, Timo Sirainen wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig Binary packages in https://repo.dovecot.org/
- The default postmaster_address is now "postmaster@<user domain or server hostname>". If username contains the @domain part, that's used. If not, then the server's hostname is used.
- "doveadm stats dump" now returns two decimals for the "avg" field.
- Added push notification driver that uses a Lua script
- Added new SQL, DNS and connection events. See https://wiki2.dovecot.org/Events
- Added "doveadm mailbox cache purge" command.
- Added events API support for Lua scripts
- doveadm force-resync -f parameter performs "index fsck" while opening the index. This may be useful to fix some types of broken index files. This may become the default behavior in a later version.
- director: Kicking a user crashes if login process is very slow
- pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages unless QUIT is sent.
- auth: Fix crypt() segfault with glibc-2.28+
- imap: Running UID FILTER script with errors assert-crashes
- dsync, pop3-migration: POP3 UIDLs weren't added to dovecot.index.cache while mails were saved.
- dict clients may have been using 100% CPU while waiting for dict server to finish commands.
- doveadm user: Fixed user listing via HTTP API
- All levels of Cassandra log messages were logged as Dovecot errors.
- http/smtp client may have crashed after SSL handshake
- Lua auth converted strings that looked like numbers into numbers. The release does not build. Here is a patch to fix the build.
test-event-stats.c:101:8: warning: implicit declaration of function 'kill' is invalid in C99 [-Wimplicit-function-declaration] (void)kill(stats_pid, SIGKILL); ^ test-event-stats.c:101:24: error: use of undeclared identifier 'SIGKILL' (void)kill(stats_pid, SIGKILL); ^
On *BSD I assume? It would be useful to mention. But we'll think what we will do with this.
Aki
I can confirm that this fix works for FreeBSD (8.4, 9.3, 11.2).
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."
participants (18)
-
admin
-
Aki Tuomi
-
Brad Smith
-
Daniel Miller
-
fauno
-
Hendrik Boom
-
Jeroen de Meijer
-
Jerry
-
Joan Moreau
-
John Tulp
-
M. Balridge
-
Michael Slusarz
-
Odhiambo Washington
-
Stephan Bosch
-
Tanstaafl
-
The Doctor
-
Timo Sirainen
-
Yann Shukor