- dovecot - dovecot.org

large search indexer tasks, submitted to flatcurve+tika+tesseract backend for attachment scanning, timeout even with "fts_index_timeout = 0"; how to increase/remove timeouts?
by PGNet Dev 28 Jul '22

28 Jul '22

i'm running dovecot 2.3.19.1 search indexing is enabled, via fts_flatcurve (dovecot23_fts_flatcurve) attachment processing is via fts_tika, with tika-server-standard-2.4.2 backend & tesseract5 OCR i'm working getting this optimized on a smaller, 4-core server that opt includes capping tesseract @ single-thread, Environment=OMP_THREAD_LIMIT=1 still, some attachments are OCR-scanned, and can take some time. when i exec large reindex jobs, i get occassional timeout errors on dovecot's indexer-worker connection to tiks backend, e.g., 2022-07-23 09:54:43 indexer-worker(postmaster(a)example.com)<DIcjEWb922JhXAAA+IOfAw>: Error: fts_tika: PUT http://127.0.0.1:9998/tika/ failed: Request timed out (Request queued 61.031 secs ago, 1 send attempts in 60.103 secs, 60.080 in http ioloop, 0.000 in other ioloops, connected 60.103 secs ago) 2022-07-23 09:54:43 indexer-worker(postmaster(a)example.com)<DIcjEWb922JhXAAA+IOfAw>: Error: Mailbox Sent: Precache for UID=90782 failed: Internal error occurred. Refer to server log for more information. [2022-07-23 09:54:43] (attempted to index 2 messages between UIDs 90778..90782) i don't see any fts timeout info @ https://wiki.dovecot.org/Timeouts here https://doc.dovecot.org/settings/plugin/fts-plugin/#plugin_setting-fts-fts_… " fts_index_timeout Default: 0 Values: Unsigned integer When the full text search backend detects that the index isn’t up-to-date, the indexer is told to index the messages and is given this much time to do so. If this time limit is reached, an error is returned, indicating that the search timed out during waiting for the indexing to complete: NO [INUSE] Timeout while waiting for indexing to finish A value of 0 means no timeout. " double-checking my config, /usr/sbin/dovecot -c /etc/dovecot/dovecot.conf -n submission_relay_connect_timeout = 10 secs submission_timeout = 10 secs there's no tika/fts timeout override, so should be "0" by default. i.e., no timeout @ Patch: enhancements for solr/tika integration https://dovecot.org/list/dovecot/2022-January/123828.html a patch was proposed for similar issue with fts-solr indexer (not sure if it's in current release AND effective for my config), but that *limits* the size of attachments submitted. i _do_ want to index all/large attachments, so, iiuc, need to ensure no, or longer, timeout. where do I set that timeout to not fail, as above, on large index tasks?

2 2

RHEL9 Latest Repo?
by dovecot＠ptld.com 28 Jul '22

28 Jul '22

Any plans or timeline for when there will be a latest repo for RHEL9? [dovecot-2.3-latest] name=Dovecot 2.3 RHEL $releasever - $basearch baseurl=http://repo.dovecot.org/ce-2.3-latest/rhel/$releasever/RPMS/$basear… gpgkey=https://repo.dovecot.org/DOVECOT-REPO-GPG gpgcheck=1 enabled=1

3 2

test-crypto.c - Assert failed
by Tamsy 27 Jul '22

27 Jul '22

Dear List, Please pardon me if this has been already discussed before. I couldn't find the matter with a quick search. On a new standard Ubuntu 22.04 LTS installation Dovecot's "configure && make" runs through but "make check" fails. Is dovecot-2.3.19.1 not yet compatible with openSSL 3.0.2 (openssl 3.0.2-0ubuntu1.6) or is this just happening here? Distributor ID: Ubuntu Description: Ubuntu 22.04 LTS Release: 22.04 Codename: jammy Linux 5.15.0-41-generic #44-Ubuntu SMP Wed Jun 22 14:20:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux $ make check <SNIP> test_cipher_test_vectors ............................................. : ok test_cipher_aead_test_vectors ........................................ : ok test_hmac_test_vectors ............................................... : ok test_load_v1_keys .................................................... : ok test_load_v1_key ..................................................... : ok test_load_v1_public_key .............................................. : ok test_load_v2_key ..................................................... : ok test_load_v2_public_key .............................................. : ok test_get_info_v2_key ................................................. : ok test_gen_and_get_info_rsa_pem ........................................ : ok test_get_info_rsa_private_key ........................................ : ok test_get_info_invalid_keys ........................................... : ok test_get_info_key_encrypted .......................................... : ok test_get_info_pw_encrypted ........................................... : ok test-crypto.c:827: Assert failed: ret == TRUE Panic: file dcrypt-openssl.c: line 2639 (dcrypt_openssl_private_to_public_key): assertion failed: (priv_key != NULL && pub_key_r != NULL) Error: Raw backtrace: ./test-crypto(+0x60704) [0x168704] -> ./test-crypto(backtrace_append+0x1c) [0x168893] -> ./test-crypto(backtrace_get+0x2a) [0x1688bf] -> ./test-crypto(+0x28ef4) [0x130ef4] -> ./test-crypto(default_fatal_handler+0) [0x130fc6] -> ./test-crypto(default_error_handler+0) [0x131014] -> ./test-crypto(i_fatal+0) [0x1312ae] -> .libs/libdcrypt_openssl.so(+0xe795) [0x4ea3795] -> ./test-crypto(dcrypt_key_convert_private_to_public+0x7b) [0x11eeb1] -> ./test-crypto(+0x21655) [0x129655] -> ./test-crypto(+0x23d66) [0x12bd66] -> ./test-crypto(test_run+0x21) [0x12c13a] -> ./test-crypto(main+0x83) [0x12af16] -> /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x4893d90] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x4893e40] -> ./test-crypto(_start+0x25) [0x11c1a5] ../../run-test.sh: line 39: 164971 Aborted (core dumped) /usr/bin/valgrind -q $trace_children --error-exitcode=213 --leak-check=full --gen-suppressions=all --suppressions="$supp_path" --log-file=$test_out $noundef $* ==164971== Conditional jump or move depends on uninitialised value(s) ==164971== at 0x514E234: ??? (in /usr/lib/x86_64-linux-gnu/libcrypto.so.3) ==164971== by 0x514E511: ??? (in /usr/lib/x86_64-linux-gnu/libcrypto.so.3) ==164971== by 0x504F0F4: EVP_DecryptFinal_ex (in /usr/lib/x86_64-linux-gnu/libcrypto.so.3) ==164971== by 0x4E9CD3F: dcrypt_openssl_ctx_sym_final (in /usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt/.libs/libdcrypt_openssl.so) ==164971== by 0x11E3DB: dcrypt_ctx_sym_final (in /usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt/test-crypto) ==164971== by 0x1270DE: test_cipher_aead_test_vectors (in /usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt/test-crypto) ==164971== by 0x12BD65: test_run_funcs (in /usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt/test-crypto) ==164971== by 0x12C139: test_run (in /usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt/test-crypto) ==164971== by 0x12AF15: main (in /usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt/test-crypto) ==164971== { <insert_a_suppression_name_here> Memcheck:Cond obj:/usr/lib/x86_64-linux-gnu/libcrypto.so.3 obj:/usr/lib/x86_64-linux-gnu/libcrypto.so.3 fun:EVP_DecryptFinal_ex fun:dcrypt_openssl_ctx_sym_final fun:dcrypt_ctx_sym_final fun:test_cipher_aead_test_vectors fun:test_run_funcs fun:test_run fun:main } ==164971== 2,304 bytes in 1 blocks are possibly lost in loss record 911 of 947 ==164971== at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==164971== by 0x4005D97: malloc (rtld-malloc.h:56) ==164971== by 0x4005D97: _dlfo_mappings_segment_allocate (dl-find_object.c:217) ==164971== by 0x4005D97: _dl_find_object_update_1 (dl-find_object.c:671) ==164971== by 0x4005D97: _dl_find_object_update (dl-find_object.c:804) ==164971== by 0x400ECCF: dl_open_worker_begin (dl-open.c:735) ==164971== by 0x49DEC27: _dl_catch_exception (dl-error-skeleton.c:208) ==164971== by 0x400DF99: dl_open_worker (dl-open.c:782) ==164971== by 0x49DEC27: _dl_catch_exception (dl-error-skeleton.c:208) ==164971== by 0x400E34D: _dl_open (dl-open.c:883) ==164971== by 0x48FA6BB: dlopen_doit (dlopen.c:56) ==164971== by 0x49DEC27: _dl_catch_exception (dl-error-skeleton.c:208) ==164971== by 0x49DECF2: _dl_catch_error (dl-error-skeleton.c:227) ==164971== by 0x48FA1AD: _dlerror_run (dlerror.c:138) ==164971== by 0x48FA747: dlopen_implementation (dlopen.c:71) ==164971== by 0x48FA747: dlopen@@GLIBC_2.34 (dlopen.c:81) ==164971== { <insert_a_suppression_name_here> Memcheck:Leak match-leak-kinds: possible fun:malloc fun:malloc fun:_dlfo_mappings_segment_allocate fun:_dl_find_object_update_1 fun:_dl_find_object_update fun:dl_open_worker_begin fun:_dl_catch_exception fun:dl_open_worker fun:_dl_catch_exception fun:_dl_open fun:dlopen_doit fun:_dl_catch_exception fun:_dl_catch_error fun:_dlerror_run fun:dlopen_implementation fun:dlopen@@GLIBC_2.34 } ==164971== 2,304 bytes in 1 blocks are possibly lost in loss record 912 of 947 ==164971== at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==164971== by 0x4005D97: malloc (rtld-malloc.h:56) ==164971== by 0x4005D97: _dlfo_mappings_segment_allocate (dl-find_object.c:217) ==164971== by 0x4005D97: _dl_find_object_update_1 (dl-find_object.c:671) ==164971== by 0x4005D97: _dl_find_object_update (dl-find_object.c:804) ==164971== by 0x400ECCF: dl_open_worker_begin (dl-open.c:735) ==164971== by 0x49DEC27: _dl_catch_exception (dl-error-skeleton.c:208) ==164971== by 0x400DF99: dl_open_worker (dl-open.c:782) ==164971== by 0x49DEC27: _dl_catch_exception (dl-error-skeleton.c:208) ==164971== by 0x400E34D: _dl_open (dl-open.c:883) ==164971== by 0x49DEF90: do_dlopen (dl-libc.c:95) ==164971== by 0x49DEC27: _dl_catch_exception (dl-error-skeleton.c:208) ==164971== by 0x49DECF2: _dl_catch_error (dl-error-skeleton.c:227) ==164971== by 0x49DF0C6: dlerror_run (dl-libc.c:45) ==164971== by 0x49DF0C6: __libc_dlopen_mode (dl-libc.c:162) ==164971== by 0x498F7C0: __libc_unwind_link_get (unwind-link.c:50) ==164971== by 0x498F7C0: __libc_unwind_link_get (unwind-link.c:40) ==164971== { <insert_a_suppression_name_here> Memcheck:Leak match-leak-kinds: possible fun:malloc fun:malloc fun:_dlfo_mappings_segment_allocate fun:_dl_find_object_update_1 fun:_dl_find_object_update fun:dl_open_worker_begin fun:_dl_catch_exception fun:dl_open_worker fun:_dl_catch_exception fun:_dl_open fun:do_dlopen fun:_dl_catch_exception fun:_dl_catch_error fun:dlerror_run fun:__libc_dlopen_mode fun:__libc_unwind_link_get fun:__libc_unwind_link_get } Failed to run: ./test-crypto make[3]: *** [Makefile:1136: check-local] Error 1 make[3]: Leaving directory '/usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt' make[2]: *** [Makefile:983: check-am] Error 2 make[2]: Leaving directory '/usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt' make[1]: *** [Makefile:573: check-recursive] Error 1 make[1]: Leaving directory '/usr/local/src/dovecot-2.3.19.1/src' make: *** [Makefile:702: check-recursive] Error 1

2 2

Ubuntu 22.04/Jammy repo coming soon ?
by Dean Carpenter 25 Jul '22

25 Jul '22

Is there any idea of when a repo for Ubuntu 22.04/Jammy will be built ? I've tried a couple of test installs onto a 22.04 system using the 20.04/Focal repo, but no luck yet. libssl issues etc. -- Dean Carpenter

2 2

Re: How to correctly use readonly=yes with the sqlite driver?
by oddlama 24 Jul '22

24 Jul '22

On 24/07/2022 10:52, Eduardo M KALINOWSKI wrote: > The documentation mentions that "Prior to v2.3.18, Dovecot uses the > whole value as filename to connect, whitespace included.". Are you > running a new enough version? > > -- > You may get an opportunity for advancement today. Watch it! > > Eduardo M KALINOWSKI > eduardo at kalinowski.com.br I am running the newest available version from the main gentoo repository, which is 2.3.19.1 - unfortunately, nobody else on the internet seems to have written about this feature before, so I cannot easily check if I made a mistake. Do you require any further information? Best regards, oddlama

2 1

How to correctly use readonly=yes with the sqlite driver?
by oddlama 24 Jul '22

24 Jul '22

Hi! I want to use a sqlite database in readonly mode, which I found mentionend in the docs [1]. Unfortunately no examples were given, so I tried to infer the usage by inspecting core/driver-sqlite.c. If I interpret it correctly, this should have worked: /etc/dovecot/dovecot-sql.conf.ext: driver = sqlite connect = readonly=yes /var/vmail/virtual.sqlite user_query = #... Unfortunately, dovecot tries to open a file named "readonly=yes /var/vmail/virtual.sqlite" instead. I have no idea why the option is not parsed. Is this a bug or am I using the feature incorrectly? Best regards, oddlama [1]: https://doc.dovecot.org/settings/plugin/sql-sqlite/

2 1

Dovecot sync stopped working since 2.3.18-r1 when .maildir has 700 permissions
by Martin Kuchta 23 Jul '22

23 Jul '22

Hello, Since upgrading to dovecot 2.3.18-r1 my sync setup using replicator plugin stopped working. It seems there is a problem accessing a .maildir with 700 permissions, only accessible by the owner. Everything worked fine prior to this version and I made no configuration changes. # 2.3.19.1 (9b53102964): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.19 (4eae2f79) # OS: Linux 5.10.74-gentoo x86_64 Gentoo Base System release 2.8 # Hostname: www.example.com auth_mechanisms = plain login auth_username_format = %Ln doveadm_password = # hidden, use -P to show it hostname = www.example.xom listen = * login_greeting = Dovecot ready. mail_location = maildir:~/.maildir mail_plugins = notify replication managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext imapflags notify vnd.dovecot.pipe namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = * driver = pam } plugin { mail_replica = tcps:www.example.com:8000 sieve = file:~/sieve;active=~/.dovecot.sieve sieve_extensions = +notify +imapflags +vnd.dovecot.pipe sieve_pipe_bin_dir = /usr/lib/dovecot/sieve-pipe sieve_plugins = sieve_extprograms } postmaster_address = postmaster(a)example.com protocols = imap lmtp sieve service aggregator { fifo_listener replication-notify-fifo { mode = 0666 } unix_listener replication-notify { mode = 0666 } } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } } service doveadm { inet_listener { port = 8000 ssl = yes } } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } ssl_cert = </etc/letsencrypt/live/example.com/fullchain.pem ssl_cipher_list = ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = passwd } protocol lmtp { mail_plugins = notify replication sieve postmaster_address = postmaster(a)example.com } protocol lda { mail_plugins = notify replication sieve } local_name mail.example.com { ssl_cert = </etc/letsencrypt/live/example.com/fullchain.pem ssl_key = # hidden, use -P to show it } local_name example.com { ssl_cert = </etc/letsencrypt/live/example.com/fullchain.pem ssl_key = # hidden, use -P to show it } -- roughgrain.com - Mastering Mentoring +447780565902

1 1

Tools to get a report of which folders have new mail?
by Steve Litt 20 Jul '22

20 Jul '22

Hi all, I use a Dovecot IMAP server on my Linux desktop computer, and I'm pretty good at writing shellscripts. Is there any way I could use doveadm or other tools to create a report that shows all my folders in a hierarchy? Also, is there a way to show only those with new mail? Thanks, SteveT Steve Litt Summer 2022 featured book: Thriving in Tough Times http://www.troubleshooters.com/bookstore/thrive.htm

7 10

Tracing Sieve actions
by Doug Hardie 20 Jul '22

20 Jul '22

I encountered an interesting problem that one originator was being dumped into the Deleted file directly by my sieve. The sieve file was quite large and it was not obvious which entry was causing the issue. I recall there was a way to get sieve-test to show what is going on and which lines it used, but I could not replicate it tonight for anything. I ended up having to change all the deliver to the Deleted files to something else and test one at a time to find the offending entry. It took a long time. How do you get sieve-test to show the actual path it took through the file? -- Doug

3 4

Re: dovecot Digest, Vol 231, Issue 31
by Joseph Tam 20 Jul '22

20 Jul '22

> > doveadm -fjson mailbox status -u user unseen "*" > > Very nice Aki! I can pass that JSON to a Python program I make to parse > JSON, and then just report the ones not having "unseen":"0" . Thank Or use format "-ftab" and grep non-zero entries. Simpler than parsing JSON. Joseph Tam <jtam.home(a)gmail.com>

1 0