- dovecot - dovecot.org

mdbox import error from read-only filesystem
by hby 14 Jan '19

14 Jan '19

Dovecot version: 2.2.34 doveadm import tries to call open() on the source indexes/logs of mdbox data, but even if it should work as it is just a read-related call, it fails on read-only filesystems. The main use case for read-only filesystem is restoring from backup: the massive deduplicated backup data is exposed using FUSE. For deduplication to work efficiently, we have to backup the whole mail storage and thus the only way to handle the restore to mount the whole data. I would like to ask for some proper solution for this problem. Suggestions: first, the doveadm import should never modify the indexes, and second: the open() calls should respect the read-only filesystem and should fall back to "r" flag when using open. The problem goes further when using SIS (which goes beyond multiple terrabytes). The only workaround currently is to export the whole mdbox data to a writeable storage, and parametrize the import command to use that and the mounted SIS data separately for import...which is just problematic, ugly and error prune. Thank you for the continous work on dovecot, I hope you get this feedback in a good way.

2 2

Freebsd: Fatal error - Support not compiled in for passdb driver 'sql'
by Graham S. Jarvis 14 Jan '19

14 Jan '19

Hello, I have the following fatal error when I uncomment #!include auth-sql.conf.ext in .../conf.d/10-auth.conf dovecot: auth: Fatal: Support not compiled in for passdb driver 'sql' I have installed via 'pkg'. Do I have to install the ports tree to get sql support ? Thanks, -Graham-

4 9

managesieve configuration
by Dominik Menke 14 Jan '19

14 Jan '19

Hello list, I'm trying to get pidgeonhole/managesieve running, and I'm stuck connecting clients to the server (Dovecot 2.2.33.2-1ubuntu4.1 on an Ubnutu 18.04 machine). So far, my config looks like this: protocols = imap lmtp sieve disable_plaintext_auth = yes auth_mechanisms = plain login scram-sha-1 service managesieve-login { inet_listener sieve { port = 4190 ssl = yes } service_count = 1 } service managesieve { process_limit = 256 } protocol sieve { managesieve_max_line_length = 65536 } (please let me know if you need more details). When I connect via openssl s_client -connect $myserver:4190 I get the following prompt (after the usual certificate prompt): "IMPLEMENTATION" "Dovecot (Ubuntu) Pigeonhole" "SIEVE" "fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext vacation-seconds imapsieve vnd.dovecot.imapsieve" "NOTIFY" "mailto" "SASL" "PLAIN LOGIN SCRAM-SHA-1" "VERSION" "1.0" OK "Dovecot (Ubuntu) ready." and I can login successfully: AUTHENTICATE "PLAIN" "base64(0x00 $user 0x00 $password)" OK "Logged In." Note how this is different from the troubleshooting guide [1], which suggests gnutls-bin and waiting for the STARTTLS capability before hitting Ctrl-D. This is what I get with gnutls-bin: $ gnutls-cli --starttls --insecure -p 4190 $myserver Processed 0 CA certificate(s). Resolving '$myserver:4190'... Connecting to '$myserverip:4190'... - Simple Client Mode: _ where "_" denotes the waiting prompt. When I hit Ctrl-D here, I get an output similar that of s_client. Now, my problem are the clients: neither Thunderbird's sieve extenstion [2], nor the Ruby ManageSieve class [3], nor Roundcube's managesieve plugin [4] (via Net_Sieve module [5]) are able to communicate with my server. They all wait for a "STARTTLS" line, before they attempt to perform a TLS handshake. This leads me to my question: How do I force Dovecot to print at least a STARTTLS line after a client connects to port 4190? Looking Kind regards, Dominik [1]: https://wiki.dovecot.org/Pigeonhole/ManageSieve/Troubleshooting#Manual_TLS_… [2]: https://github.com/thsmi/sieve [3]: https://www.rubydoc.info/gems/ruby-managesieve/0.4.3/ManageSieve [4]: https://github.com/roundcube/roundcubemail/tree/1.3.8/plugins/managesieve [5]: http://pear.php.net/package/Net_Sieve/docs/1.3.4/Net_Sieve/Net_Sieve.html

4 11

degfault imaptest
by Marc Roos 14 Jan '19

14 Jan '19

imaptest host=mail04.local port=143 user=xxx pass=xxx mbox=test.mbox [Fri Jan 11 15:09:14 2019] imaptest[7634]: segfault at 1354ff0 ip 0000000001354ff0 sp 00007ffed5d8f558 error 15 [Fri Jan 11 15:09:22 2019] imaptest[7635]: segfault at 2267ff0 ip 0000000002267ff0 sp 00007ffee5890308 error 15 [Fri Jan 11 15:10:47 2019] imaptest[7648]: segfault at 10e6ff0 ip 00000000010e6ff0 sp 00007ffedbb19f98 error 15 CentOS Linux release 7.6.1810 (Core) imaptest-20151005-1.el7.x86_64 Linux test2 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

3 4

Indexing paralelism
by Joan Moreau 14 Jan '19

14 Jan '19

Hi Observing the processes of FTS, I observe the following: 1 - For one user, indexer-wroker does not start several threads for each request. On teh contrary, it waits for the first request to finish before starting the second. How to make sure all requests (or a limited number of it, for instance linked to the CPU number on the machine) are started asap ? 2 - If a IMAP query is received, the dovecot checks the last UID from FTS and launch a request of indexing to finish the index *before* running the search query . THis creates timeouts (and can take a while if many request are pending - see point 1) How to prevent that (i.e. the serach request is launched (read only) no matter what ? THe completeion of missing UIDs is launched in a separate thread ? Thank you

2 1

[FTS Xapian] Beta release
by Joan Moreau 13 Jan '19

13 Jan '19

Hi Please find attached the beta release of FTS Xapian, with the objective to replace fts_squat that is being deprecated. Configuration is exactly the same as for fts_squat: plugin { plugin = fts fts_xapian (...) fts = xapian fts_autoindex = yes fts_enforced = yes fts_xapian = partial=2 full=20 This is installed on my production server (>120Gb of mailboxes), and I will observe it during the coming days. I will definitely appreciate that this is added in the core git of docevot, in order to have a versionning of it, to remove squat and let basic users able to avoid Solr alternative as much as possible. Thanks JM

2 9

[FTS Xapian] Beta release
by Joan Moreau 13 Jan '19

13 Jan '19

Hi Please find attached the beta release of FTS Xapian, with the objective to replace fts_squat that is being deprecated. Configuration is exactly the same as for fts_squat: plugin { plugin = fts fts_xapian (...) fts = xapian fts_autoindex = yes fts_enforced = yes fts_xapian = partial=2 full=20 This is installed on my production server, and I will observe it during the coming days. I will definitely appreciate that this is added in the core git of docevot, in order to have a versionning of it, to remove squat and let basic users able to avoid Solr alternative as much as possible. THanks JM

1 0

Issue with LMTP proxying and port number
by Laz C. Peterson 13 Jan '19

13 Jan '19

Hello there, Everything was working fine on Dovecot 2.2.10 (on CentOS 7), but after updating to version 2.2.36, our director servers are not able to proxy LMTP. We are sending mail from exim to the local Dovecot LMTP socket, which then used to send it to our internal mail servers on port 24. Besides the update, nothing has changed in the config files. The configuration has been working great for months … Then we had a great idea of doing holiday maintenance. (Currently, we are working around this issue by having exim send mail to the IP and port, instead of the socket.) Doing a tcpdump capture shows no TCP traffic either. The logs seem to say that it is trying to go to port 0, so my guess is that it’s not getting the port directive from either static or sql — since the docs state that if these settings aren't specified, it will always use the same connection type for the proxy that it received the connection on. Any thoughts? I can share config if necessary. ~ Laz Peterson Paravis, LLC

3 5

Dovecot Submission Proxy Auth
by Paul Hecker 12 Jan '19

12 Jan '19

Hi, Dovecot 2.3.1 (8e2f634). Could not get Dovecot to forward the (plain) authentication to the SMTP server using submission. Reason why I need it is sender spoofing (do not want my employees to send messages in behalf of me). In exim I can disable sender spoofing with the authenticated user. When sending through dovecot, exim either does not accept the email (need auth) or relay every sender address (because relaying from localhost). Am I missing a setting or do I need any additional field in the (MySQL) user_query/password_query to forward the password? You can find my config here: https://gist.github.com/lluuaapp/7daddf761131da47237b0f45e6bab5a8 Thanks, Paul

5 12

Fatal: master: service(indexer-worker): child 493 killed with signal 11 (core dumped)
by Tim Mohlmann 12 Jan '19

12 Jan '19

Dear list, We been having some issues where the indexer-worker is crashing. This happens on production servers which are handling a slight amount of mail, but is also reproducible by moving messages. Also, users on my server are complaining about "Trashed" items coming back etc. Some details: - Dovecot 2.3.2.1 in alpine:3.8 based Docker container. As part of the Mailu distribution. (https://github.com/Mailu/Mailu) - I've first seen this issues on my production server, which stores mail on GlusterFS - I've been able to reproduce running the Docker container on a Virtual machine, using local storage. - There is another Mailu user reporting the same problem on a different VM provider / disk infrastructure: https://github.com/Mailu/Mailu/issues/751 - Libc: musl-1.1.19-r10 Output of dovecot -n: # dovecot -n # 2.3.2.1 (0719df592): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.2 (7704de5e) # OS: Linux 4.16.3-301.fc28.x86_64 x86_64 ext4 # Hostname: 98a2726271d3 auth_mechanisms = plain login disable_plaintext_auth = no first_valid_gid = 8 first_valid_uid = 8 hostname = mail.usrpro.io log_path = /dev/stderr mail_access_groups = mail mail_gid = mail mail_home = /mail/%u mail_location = maildir:/mail/%u mail_plugins = " fts fts_lucene quota quota_clone zlib" mail_privileged_group = mail mail_uid = mail mail_vsize_bg_after_count = 100 maildir_stat_dirs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext spamtest spamtestplus editheader imapsieve vnd.dovecot.imapsieve namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = } passdb { args = /etc/dovecot/auth.conf driver = dict } plugin { fts = lucene fts_autoindex = yes fts_autoindex_exclude = \Junk fts_lucene = whitespace_chars=@. imapsieve_mailbox1_before = file:/conf/report-spam.sieve imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_name = Junk imapsieve_mailbox2_before = file:/conf/report-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = Junk imapsieve_mailbox2_name = * quota = count:User quota quota_clone_dict = proxy:/tmp/podop.socket:quota quota_vsizes = yes sieve = file:~/sieve;active=~/.dovecot.sieve sieve_before = dict:proxy:/tmp/podop.socket:sieve sieve_execute_bin_dir = /conf/bin sieve_extensions = +spamtest +spamtestplus +editheader sieve_global_extensions = +vnd.dovecot.execute sieve_plugins = sieve_imapsieve sieve_extprograms sieve_spamtest_max_value = 15 sieve_spamtest_status_header = X-Spam-Level sieve_spamtest_status_type = strlen sieve_vacation_dont_check_recipient = yes sieve_vacation_send_from_recipient = yes } postmaster_address = admin(a)usrpro.io protocols = imap pop3 lmtp sieve service auth-worker { unix_listener auth-worker { group = mail mode = 0660 user = dovecot } user = mail } service auth { user = dovecot } service imap-login { inet_listener imap { port = 143 } } service lmtp { inet_listener lmtp { port = 2525 } } service managesieve-login { inet_listener sieve { port = 4190 } } submission_host = 192.168.203.6 userdb { args = /etc/dovecot/auth.conf driver = dict } protocol imap { mail_plugins = " fts fts_lucene quota quota_clone zlib imap_quota imap_sieve" } protocol lmtp { mail_plugins = " fts fts_lucene quota quota_clone zlib sieve" recipient_delimiter = + } And the actual error log: imap_1 | Dec 19 16:31:08 indexer-worker(admin(a)usrpro.io)<490><m+t5VmJ93K7AqMsG:grc9HUxyGlzqAQAANEhNiw>: Fatal: master: service(indexer-worker): child 490 killed with signal 11 (core dumped) imap_1 | Dec 19 16:31:09 indexer-worker(admin(a)usrpro.io)<493><m+t5VmJ93K7AqMsG:HRLEK0xyGlztAQAANEhNiw>: Error: lucene index /mail/admin(a)usrpro.io/lucene-indexes: IndexWriter() failed (#1): Lock obtain timed out imap_1 | Dec 19 16:31:09 indexer-worker(admin(a)usrpro.io)<493><m+t5VmJ93K7AqMsG:HRLEK0xyGlztAQAANEhNiw>: Error: Mailbox INBOX: Mail search failed: Internal error occurred. Refer to server log for more information. [2018-12-19 16:31:08] imap_1 | Dec 19 16:31:09 indexer-worker(admin(a)usrpro.io)<493><m+t5VmJ93K7AqMsG:HRLEK0xyGlztAQAANEhNiw>: Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: backend deinit (attempted to index 1 messages (UIDs 1299..1299)) imap_1 | Dec 19 16:31:10 indexer-worker(admin(a)usrpro.io)<493><m+t5VmJ93K7AqMsG:GKWdMU1yGlztAQAANEhNiw>: Fatal: master: service(indexer-worker): child 493 killed with signal 11 (core dumped) imap_1 | Dec 19 16:31:10 indexer: Error: Indexer worker disconnected, discarding 1 requests for admin(a)usrpro.io imap_1 | Dec 19 16:31:11 indexer-worker(admin(a)usrpro.io)<494><m+t5VmJ93K7AqMsG:MRCzBE5yGlzuAQAANEhNiw>: Error: lucene index /mail/admin(a)usrpro.io/lucene-indexes: IndexWriter() failed (#1): Lock obtain timed out imap_1 | Dec 19 16:31:11 indexer-worker(admin(a)usrpro.io)<494><m+t5VmJ93K7AqMsG:MRCzBE5yGlzuAQAANEhNiw>: Error: Mailbox INBOX: Mail search failed: Internal error occurred. Refer to server log for more information. [2018-12-19 16:31:10] imap_1 | Dec 19 16:31:11 indexer-worker(admin(a)usrpro.io)<494><m+t5VmJ93K7AqMsG:MRCzBE5yGlzuAQAANEhNiw>: Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: backend deinit (attempted to index 1 messages (UIDs 1310..1310)) imap_1 | Dec 19 16:31:11 indexer: Error: Indexer worker disconnected, discarding 1 requests for admin(a)usrpro.io I managed to find a core dump file, which appeared outside of the container. So I copied it back in, installed and ran gdb: GNU gdb (GDB) 8.0.1 Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-alpine-linux-musl". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/indexer-worker...(no debugging symbols found)...done. [New LWP 1075] warning: Can't read pathname for load map: No error information. Core was generated by `dovecot/indexer-worker'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007fbd9a31c11a in free () from /lib/ld-musl-x86_64.so.1 So this seems musl related. I installed musl-dbg and ran again: GNU gdb (GDB) 8.0.1 Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-alpine-linux-musl". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/indexer-worker...(no debugging symbols found)...done. [New LWP 1075] warning: Can't read pathname for load map: No error information. Core was generated by `dovecot/indexer-worker'. Program terminated with signal SIGSEGV, Segmentation fault. #0 a_crash () at ./arch/x86_64/atomic_arch.h:108 108 ./arch/x86_64/atomic_arch.h: No such file or directory. Now I'm kinda lost in space. I don't know where that header file is. Tried running "find" on the filesystem and a google search. But nothing specific showed up. I am starting to feel this bug is more musl related than Dovecot. Since this has bitten our project more in the past, I'm considering to move the project to Debian based images. But I want to be 100% sure this is not a dovecot bug. Note, earlier I created an image with Alpine:edge with musl 1.1.20-r2 and Dovecot 2.3.3. Running that image I saw the same error symptoms (Was just a hopeful trail and error). I did not do any debugging on that one. Thanks in advance! Tim

5 11