November 2020 - dovecot

[Dovecot] Doveadm sometimes failed "to iterate through some users" after upgrade to 2.2
by Alessio Cecchi 04 May '21

04 May '21

Hi, I'm running every week in cron "doveadm expunge -A mailbox Spam before 30d" and monthly "doveadm quota recalc -A". After the upgrade to dovecot 2.2.12 (from 2.1.17), "doveadm -A" sometimes exit with this error: doveadm(user(a)domain.com): Error: User listing returned failure doveadm: Error: Failed to iterate through some users # echo $? 75 If I re-run the same command the user where doveadm stops changeing (always in alphabetical order): info@ mario@ pippo@ zazza@ and finally, after 3 or 4 run, ends with success. I have enable debug (via doveadm -D and auth/mail debub in 10-logging.con) but there are no errors or difference in the log when doveadm exit with the error (error code is 75). If I run # doveadm expunge -A mailbox Spam savedbefore 30d doveadm works fine Can somebody help me? Here is my configuration: # dovecot -n # 2.2.12: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.9 auth_debug = yes auth_master_user_separator = * auth_verbose = yes deliver_log_format = msgid=%m, from=%f, subject="%s": %$ dict { expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext sqlquota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no dotlock_use_excl = no first_valid_gid = 89 first_valid_uid = 89 imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags imap_idle_notify_interval = 29 mins imap_logout_format = in=%i out=%o session=<%{session}> last_valid_gid = 89 last_valid_uid = 89 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes listen = * mail_debug = yes mail_fsync = always mail_location = maildir:~/Maildir mail_nfs_index = yes mail_nfs_storage = yes mail_plugins = quota expire maildir_very_dirty_syncs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave vnd.dovecot.duplicate mmap_disable = yes namespace { list = children location = maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u prefix = shared/%%n/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { auto = subscribe special_use = \Junk } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / } passdb { args = username_format=%Ld /etc/dovecot/extra/alias-domains-denylogin.txt deny = yes driver = passwd-file } passdb { args = /etc/dovecot/extra/master-users driver = passwd-file master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { expire = Trash expire2 = Spam expire_dict = proxy::expire quota = maildir:UserQuota quota2 = dict:Quota Usage::noenforcing:proxy::sqlquota quota_grace = 10M quota_rule2 = Trash:storage=+100M quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = ~/.dovecot.sieve sieve_before = /etc/dovecot/sieve/before.sieve sieve_dir = ~/sieve sieve_extensions = +vnd.dovecot.duplicate -vacation } pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_logout_format = top=%t/%p, retr=%r/%b, del=%d/%m, size=%s, bytes=%i/%o, session=<%{session}> protocols = imap pop3 sieve service auth-worker { process_limit = 1000 } service auth { client_limit = 5500 unix_listener auth-userdb { group = vchkpw mode = 0660 user = vpopmail } } service dict { process_limit = 300 unix_listener dict { group = vchkpw mode = 0660 user = vpopmail } } service imap-login { process_min_avail = 4 service_count = 0 } service imap { process_limit = 4000 service_count = 100 } service managesieve-login { inet_listener sieve { port = 4190 } } service pop3-login { service_count = 0 } service pop3 { process_limit = 1023 service_count = 100 } service quota-warning { executable = script /etc/dovecot/scripts/quota-warning.sh unix_listener quota-warning { user = vpopmail } user = vpopmail } ssl_cert = </etc/dovecot/ssl/wildcard-myemail.net.pem ssl_key = </etc/dovecot/ssl/wildcard-myemail.net.key submission_host = localhost userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lda { mail_location = maildir:~/Maildir:INDEX=MEMORY mail_plugins = quota expire sieve } protocol imap { mail_max_userip_connections = 15 mail_plugins = quota expire imap_quota } protocol sieve { mail_max_userip_connections = 2 } protocol pop3 { mail_max_userip_connections = 5 } -- Alessio Cecchi is: @ ILS -> http://www.linux.it/~alessice/ on LinkedIn -> http://www.linkedin.com/in/alessice Assistenza Sistemi GNU/Linux -> http://www.cecchi.biz Cloud Email Hosting -> http://www.qboxmail.com @ PLUG -> ex-Presidente, adesso senatore a vita, http://www.prato.linux.it

3 3

IMAP session hangs on 8k-long commands if COMPRESS=DEFLATE is enabled
by Guilhem Moulin 21 Apr '21

21 Apr '21

Hi there, Dovecot 2.3.7 appears to hang when the client sends a long command after enabling the IMAP COMPRESS extension [RFC 4978]. PoC script attached along with the doveconf(1) output. Without COMPRESS=DEFLATE, and with the default ‘imap_max_line_length’ value (64k) I'm able send commands up to 65539 bytes long (that's 3 bytes more than 2¹⁶, so maybe the leading tag and the trailing CRLF aren't counted), after which Dovecot rightfully replies with tagged BAD responses. $ COMPRESS=n /tmp/many-fetch.pl 65539 […] C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [65539 bytes] S: x OK Fetch completed (0.005 + 0.001 + 0.004 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [65540 bytes] S: x BAD Error in IMAP command UID FETCH: IMAP command line too large (0.001 + 0.001 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [65541 bytes] S: x BAD Error in IMAP command UID FETCH: IMAP command line too large (0.001 + 0.001 secs). With COMPRESS=DEFLATE however, the server never sends anything back when the command size exceeds 8192 bytes: $ COMPRESS=y /tmp/many-fetch.pl 8192 […] S: x OK [READ-ONLY] Examine completed (0.001 + 0.000 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [8191 bytes] S: x OK Fetch completed (0.001 + 0.000 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [8192 bytes] S: x OK Fetch completed (0.001 + 0.000 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [8193 bytes] [ _hangs until the server terminates the session for inactivity_ ] S: * BYE Disconnected for inactivity. strace(1) show that Dovecot didn't send the response, even though the server log indicates that it's able to fully read the command. Interestingly, when setting ‘imap_max_line_length’ to 4096, the server also hangs after receiving a command of size >4096 bytes (instead of replying with a tagged BAD response). $ COMPRESS=y /tmp/many-fetch.pl 4096 C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [4095 bytes] S: x OK Fetch completed (0.001 + 0.000 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [4096 bytes] S: x OK Fetch completed (0.001 + 0.000 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [4097 bytes] [ _hangs until the server terminates the session for inactivity_ ] I'm also able to reproduce the buggy behavior against Dovecot versions found in other Debian releases. It's identical with 2.3.4 (Buster), the server also hang past 8192 bytes. 2.2.27 (Stretch) and 2.2.13 (Jessie) are able to send replies to 8k-long commands, but hang past 65536 bytes. For all 3 releases, without COMPRESS=DEFLATE Dovecot doesn't hang and rightfully sends tagged BAD responses to commands of size >65539 bytes. So while the hanging behavior appears to be a long-standing bug, the fact that's it's now hit at min($imap_max_line_length, 8192) instead of $imap_max_line_length seems to be a regression from somewhere between 2.2.27 and 2.3.4. Cheers, -- Guilhem.

3 2

BUG report: dbox movealt doesn't move reverse (to default location)
by Vitalii 05 Feb '21

05 Feb '21

Hi, Here is the problem that occured: #sudo doveadm -D altmove -u vv(a)vito.pp.ua all works properly and moves all storage to alt dir. However, to move it back to default storage I add '-r' by manual: #sudo doveadm -D altmove -u vv(a)vito.pp.ua -r all That seems behave like the original command ignoring '-r'. my dovecot conf: # 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.4 () # OS: Linux 4.19.0-9-amd64 x86_64 Debian 10.4 # Hostname: qemu7.internetguru.ca auth_mechanisms = plain login disable_plaintext_auth = no imapc_features = rfc822.size fetch-headers imapc_host = imap.gmail.com imapc_port = 993 imapc_ssl = imaps imapc_user = %u listen = *,[::] log_timestamp = "%Y-%m-%d %H:%M:%S " mail_attachment_dir = /wd640/mdbox-att/%d/%n mail_max_userip_connections = 100 mail_prefetch_count = 20 mail_privileged_group = vmail passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } plugin { quota = dict:user::file:/var/vmail/%d/%n/.quotausage sieve = /var/vmail/%d/%n/.sieve sieve_max_redirects = 25 } protocols = imap pop3 service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service imap-login { client_limit = 1000 process_limit = 512 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service stats { unix_listener stats-reader { group = vmail mode = 0660 user = vmail } unix_listener stats-writer { group = vmail mode = 0660 user = vmail } } ssl_cert = </etc/postfix/smtpd.cert ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } protocol imap { mail_plugins = quota imap_quota } protocol pop3 { mail_plugins = quota pop3_uidl_format = %08Xu%08Xv } protocol lda { mail_plugins = sieve quota postmaster_address = postmaster(a)some.ca } protocol lmtp { mail_plugins = quota sieve postmaster_address = postmaster(a)some.ca } I am using debian 10. P.S. I have not found much information about dbox attachments commands. Seems none needs them.

3 3

Antispam plugin
by Maciej Milaszewski 17 Jan '21

17 Jan '21

Hi System centos8 + dovecot-2.3.8 from repo # 2.3.8 (9df20d2db): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.8 (b7b03ba2) # OS: Linux 4.18.0-193.19.1.el8_2.x86_64 x86_64 CentOS Linux release 8.2.2004 (Core) I need "Antispam plugin". What antispam-plugin I must use ? In older version dovecot-2.2.36.4 i use "dovecot-antispam-plugin" but in centos I had a problem with configure - probably toold wersion "./configure: line 3193: DC_DOVECOT: command not found" In wiki dovect is "antispam-plugin" I try this http://hg.dovecot.org/dovecot-antispam-plugin but get 404 Backand spamassassin

4 3

Dovecot Replication Errors (only) when using tcps: as the mail_replica Protocol
by Reuben Farrelly 18 Dec '20

18 Dec '20

Hi, I've been seeing errors logged for some time with replication processes, whereby replication sessions seem to be timing out periodically. This is with dovecot version 2.3.10.1 (a3d0e1171) and both are Gentoo x86_64. After some investigation I've determined that these timeouts are only ever occurring with tcps as the replication connection type. These errors never occur if non-encrypted tcp is configured. I've been able to validate this by changing only the replica_type on both ends of the replication configuration to tcp, and with no other changes and after a few days of operation there is not a single error logged. mail_replica = tcps:lightning.reub.net:4813 <<< periodic timeouts mail_replica = tcp:lightning.reub.net:4814 <<< works perfectly Example of the error: Jun 12 15:45:44 thunderstorm.reub.net dovecot[21149]: dsync-local(kaylene)<zx+WKTAU416UMwAAzkCIew>: Error: dsync(lightning.reub.net): I/O has stalled, no activity for 600 seconds (last sent=mailbox_delete, last recv=handshake) Jun 12 15:45:44 thunderstorm.reub.net dovecot[21149]: dsync-local(kaylene)<zx+WKTAU416UMwAAzkCIew>: Error: Timeout during state=recv_mailbox_tree doveadm: Error: Timeout during state=slave_recv_mailbox: 6 Time(s) doveadm: Error: Timeout during state=sync_mails (send=mail_requests recv=attributes): 31 Time(s) doveadm: Error: dsync(thunderstorm.reub.net): I/O has stalled, no activity for 600 seconds (last sent=mail_change (EOL), last recv=mailbox): 31 Time(s) doveadm: Error: dsync(thunderstorm.reub.net): I/O has stalled, no activity for 600 seconds (last sent=mailbox_delete, last recv=mailbox_delete): 6 Time(s) It is seen on both sides of the replication setup. The replica is offsite but only a few ms of latency away and there is no packet loss. The replication is happening over IPv6, and the local firewall is logging that sessions are always permitted, and only ever finishing due to tcp-fin or tcp-rst-from-client . SSL appears to be correctly configured, and it seems that the replication itself is for the most part working. Clients are able to use imaps just fine so I don't think there's anything much wrong from an SSL perspective else I'd be seeing complete replication failure and/or client devices unable to connect. Can anyone suggest how we can further debug this problem? Thanks, Reuben

4 3

Migrate mailbox from NFS folder to new NFS folder
by Claudio Corvino 14 Dec '20

14 Dec '20

Hi everyone, I need to migrate many user mailboxes from an NFS folder to a new NFS folder, both mounted on the same Dovecot server (2.2.27). Once migrated, users will connect to a new Dovecot server (2.3.4.1). I would like to know how to use /doveadm backup/ (or sync?) instead of /rsync/ as I noted that after migration with rsync I have duplicates and some e-mails marked as unread. What is the correct way to achieve this? Thanks Cheers

2 1

Files in tmp directory
by David Morsberger 05 Dec '20

05 Dec '20

I had to restore many thousands of dovecot emails from a crashed Mac Server to a new Ubuntu server. After completing I am noticing a few things that may be okay or may need to be fixed. 1) a couple of mailboxes have a lot (one has 102462) emails in the tmp directory. The files do not have the standard Mairdir++ filename format. An example: 1581262582.M397740P72267.mini.mmpcrofton.com <http://1581262582.m397740p72267.mini.mmpcrofton.com/> (missing the stuff at the end, ,S=,W=:2,S) 2) I see files in the cur directory that do not have the Maildir++ format. An example: 1601287326.V810Ie80087M240104.mmp-mail.mmpcrofton.com:2,S. (do not have the ,S=,W= values and the second value, V810Ie80087M240104, is longer that the other file names) How do I get the emails in (1) out of tmp? Do I need to convert the emails in (2)? David

2 3

Error: Mailbox INBOX: Sync failed
by Jos Chrispijn 02 Dec '20

02 Dec '20

Just saw in my logfile this line appeared: imap(xxxxxx)<56614><CMMJw9i0aoBSsH9H>: Error: Mailbox INBOX: Sync failed for mbox: Expunged message reappeared to mailbox (UID 174550 < 174552, seq=11, idx_msgs=9) Looks like messages' UIDs are mixed up? I thought that if Dovecot sees broken X-UID headers for "new" messages, they're silently fixed? Can you tell how I can correct this error/warning? Thanks! BR, Jos

1 1

Dovecot+Solr+Tika segmentation fault
by Manuel Josupeit-Walter 02 Dec '20

02 Dec '20

3 2

director_username_hash = %d and doveadm director map
by Ralf Becker 30 Nov '20

30 Nov '20

Our directors hash by domain (director_username_hash = %d), as some of our users share folders with other users of the same domain. We now started using director tags to map domains to their backends. Unfortunately doveadm director map seems no to work with director_username_hash = %d user hash mail server ip expire time <unknown> 432784257 10.44.88.1 2020-11-23 13:10:55 <unknown> 4244233328 10.44.88.1 2020-11-23 13:13:55 <unknown> 1913982503 10.44.88.1 2020-11-23 13:15:40 How can I check to which backend / IP a domain is maped, aka how is that hash calculated? doveadm director move seems also not to do anything meaningful for %d, or at least I have not found out how to use it to move a domain to a different backend. Hoping for some insight :) Ralf -- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0

3 5