December 2020 - dovecot

IMAP session hangs on 8k-long commands if COMPRESS=DEFLATE is enabled
by Guilhem Moulin 21 Apr '21

21 Apr '21

Hi there, Dovecot 2.3.7 appears to hang when the client sends a long command after enabling the IMAP COMPRESS extension [RFC 4978]. PoC script attached along with the doveconf(1) output. Without COMPRESS=DEFLATE, and with the default ‘imap_max_line_length’ value (64k) I'm able send commands up to 65539 bytes long (that's 3 bytes more than 2¹⁶, so maybe the leading tag and the trailing CRLF aren't counted), after which Dovecot rightfully replies with tagged BAD responses. $ COMPRESS=n /tmp/many-fetch.pl 65539 […] C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [65539 bytes] S: x OK Fetch completed (0.005 + 0.001 + 0.004 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [65540 bytes] S: x BAD Error in IMAP command UID FETCH: IMAP command line too large (0.001 + 0.001 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [65541 bytes] S: x BAD Error in IMAP command UID FETCH: IMAP command line too large (0.001 + 0.001 secs). With COMPRESS=DEFLATE however, the server never sends anything back when the command size exceeds 8192 bytes: $ COMPRESS=y /tmp/many-fetch.pl 8192 […] S: x OK [READ-ONLY] Examine completed (0.001 + 0.000 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [8191 bytes] S: x OK Fetch completed (0.001 + 0.000 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [8192 bytes] S: x OK Fetch completed (0.001 + 0.000 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [8193 bytes] [ _hangs until the server terminates the session for inactivity_ ] S: * BYE Disconnected for inactivity. strace(1) show that Dovecot didn't send the response, even though the server log indicates that it's able to fully read the command. Interestingly, when setting ‘imap_max_line_length’ to 4096, the server also hangs after receiving a command of size >4096 bytes (instead of replying with a tagged BAD response). $ COMPRESS=y /tmp/many-fetch.pl 4096 C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [4095 bytes] S: x OK Fetch completed (0.001 + 0.000 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [4096 bytes] S: x OK Fetch completed (0.001 + 0.000 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [4097 bytes] [ _hangs until the server terminates the session for inactivity_ ] I'm also able to reproduce the buggy behavior against Dovecot versions found in other Debian releases. It's identical with 2.3.4 (Buster), the server also hang past 8192 bytes. 2.2.27 (Stretch) and 2.2.13 (Jessie) are able to send replies to 8k-long commands, but hang past 65536 bytes. For all 3 releases, without COMPRESS=DEFLATE Dovecot doesn't hang and rightfully sends tagged BAD responses to commands of size >65539 bytes. So while the hanging behavior appears to be a long-standing bug, the fact that's it's now hit at min($imap_max_line_length, 8192) instead of $imap_max_line_length seems to be a regression from somewhere between 2.2.27 and 2.3.4. Cheers, -- Guilhem.

3 2

BUG report: dbox movealt doesn't move reverse (to default location)
by Vitalii 05 Feb '21

05 Feb '21

Hi, Here is the problem that occured: #sudo doveadm -D altmove -u vv(a)vito.pp.ua all works properly and moves all storage to alt dir. However, to move it back to default storage I add '-r' by manual: #sudo doveadm -D altmove -u vv(a)vito.pp.ua -r all That seems behave like the original command ignoring '-r'. my dovecot conf: # 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.4 () # OS: Linux 4.19.0-9-amd64 x86_64 Debian 10.4 # Hostname: qemu7.internetguru.ca auth_mechanisms = plain login disable_plaintext_auth = no imapc_features = rfc822.size fetch-headers imapc_host = imap.gmail.com imapc_port = 993 imapc_ssl = imaps imapc_user = %u listen = *,[::] log_timestamp = "%Y-%m-%d %H:%M:%S " mail_attachment_dir = /wd640/mdbox-att/%d/%n mail_max_userip_connections = 100 mail_prefetch_count = 20 mail_privileged_group = vmail passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } plugin { quota = dict:user::file:/var/vmail/%d/%n/.quotausage sieve = /var/vmail/%d/%n/.sieve sieve_max_redirects = 25 } protocols = imap pop3 service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service imap-login { client_limit = 1000 process_limit = 512 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service stats { unix_listener stats-reader { group = vmail mode = 0660 user = vmail } unix_listener stats-writer { group = vmail mode = 0660 user = vmail } } ssl_cert = </etc/postfix/smtpd.cert ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } protocol imap { mail_plugins = quota imap_quota } protocol pop3 { mail_plugins = quota pop3_uidl_format = %08Xu%08Xv } protocol lda { mail_plugins = sieve quota postmaster_address = postmaster(a)some.ca } protocol lmtp { mail_plugins = quota sieve postmaster_address = postmaster(a)some.ca } I am using debian 10. P.S. I have not found much information about dbox attachments commands. Seems none needs them.

3 3

Antispam plugin
by Maciej Milaszewski 17 Jan '21

17 Jan '21

Hi System centos8 + dovecot-2.3.8 from repo # 2.3.8 (9df20d2db): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.8 (b7b03ba2) # OS: Linux 4.18.0-193.19.1.el8_2.x86_64 x86_64 CentOS Linux release 8.2.2004 (Core) I need "Antispam plugin". What antispam-plugin I must use ? In older version dovecot-2.2.36.4 i use "dovecot-antispam-plugin" but in centos I had a problem with configure - probably toold wersion "./configure: line 3193: DC_DOVECOT: command not found" In wiki dovect is "antispam-plugin" I try this http://hg.dovecot.org/dovecot-antispam-plugin but get 404 Backand spamassassin

4 3

migration with doveadm backup to new cluster running dovecot 2.2.36 and replicator
by Thomas Winterstein 12 Jan '21

12 Jan '21

Hello everyone, we are working on migrating from dovecot 2.0.9 (maildir) to 2.2.36 (mdbox). The new cluster has two backend mail servers which replicate through doveadm replicator. To move the data initially we use doveadm backup (imapc). Our migration command doveadm -o mail_fsync=never backup -R -u $user imapc: To test the replication of new and purge of old mails with live data changes we ran imapc on a daily basis but encountered the problem that some mailboxes multiplied in size. We then made sure that imapc and replication don't run at the same time but after the first incremental imapc process, we still had the same problems. The doveadm-backup man-page states that it's possible to run it multiple times during migration. But is it also possible to have the replicator running in between? From our understanding the doveadm backup should just work as an imap connection between the servers, synchronizing all changes made on the source to the destination. Or does the conversion from maildir to mdbox format in our case produce the problems? If you're not supposed to run the replicator before having fully migrated, how can we shorten the downtime? rsync? And how can we be sure that similar problems don't occur after the migration if we can't test all mechanisms together with live data? thanks -- Thomas Winterstein

3 7

Imapsieve FLAG cause executes a pipe call twice
by DocMAX 06 Jan '21

06 Jan '21

I have this setup: imapsieve_mailbox1_name = * imapsieve_mailbox1_causes = FLAG imapsieve_mailbox1_before = file:~/sieve/learn-flag.sieve Now if i change the SPAM flag i can see: Dez 24 14:49:08 server dovecot[219886]: imap(docmax)<221469><BStCETa3fMzAqAEB>: Debug: imapsieve: mailbox Junk-E-Mail/dziennik.pl: FLAG event (changed flags: NonJunk) Dez 24 14:49:08 server dovecot[219886]: imap(docmax)<221469><BStCETa3fMzAqAEB>: Debug: imapsieve: Matched static mailbox rule [1] Dez 24 14:49:08 server dovecot[219886]: imap(docmax)<221469><BStCETa3fMzAqAEB>: Debug: imapsieve: mailbox Junk-E-Mail/dziennik.pl: FLAG event (changed flags: Junk) Dez 24 14:49:08 server dovecot[219886]: imap(docmax)<221469><BStCETa3fMzAqAEB>: Debug: imapsieve: Matched static mailbox rule [1] Dez 24 14:49:11 server dovecot[219886]: imap(docmax)<221469><BStCETa3fMzAqAEB>: Debug: imapsieve: mailbox Junk-E-Mail/dziennik.pl: FLAG event (changed flags: Junk) Dez 24 14:49:11 server dovecot[219886]: imap(docmax)<221469><BStCETa3fMzAqAEB>: Debug: imapsieve: Matched static mailbox rule [1] Dez 24 14:49:11 server dovecot[219886]: imap(docmax)<221469><BStCETa3fMzAqAEB>: Debug: imapsieve: mailbox Junk-E-Mail/dziennik.pl: FLAG event (changed flags: NonJunk) And my script is: require [ "variables" , "imap4flags", "fileinto", "vnd.dovecot.pipe", "copy", "imapsieve", "vnd.dovecot.execute" ]; if hasflag :is "Junk" { pipe :copy "rspam-spam"; stop; } if hasflag :is "NonJunk" { pipe :copy "rspam-ham"; stop; } Dez 24 15:05:24 server dovecot[222484]: imap(docmax)<224089></EW5RDa3UMPAqAEB>: Debug: sieve: action pipe: running program: rspam-spam Dez 24 15:05:24 server dovecot[222484]: imap(docmax)<224089></EW5RDa3UMPAqAEB>: Debug: sieve: action pipe: running program: rspam-ham Dez 24 15:05:27 server dovecot[222484]: imap(docmax)<224089></EW5RDa3UMPAqAEB>: Debug: sieve: action pipe: running program: rspam-spam Dez 24 15:05:27 server dovecot[222484]: imap(docmax)<224089></EW5RDa3UMPAqAEB>: Debug: sieve: action pipe: running program: rspam-spam Dez 24 15:05:32 server dovecot[222484]: imap(docmax)<224089></EW5RDa3UMPAqAEB>: Debug: sieve: action pipe: running program: rspam-spam Dez 24 15:05:32 server dovecot[222484]: imap(docmax)<224089></EW5RDa3UMPAqAEB>: Debug: sieve: action pipe: running program: rspam-ham Dez 24 15:05:36 server dovecot[222484]: imap(docmax)<224089></EW5RDa3UMPAqAEB>: Debug: sieve: action pipe: running program: rspam-spam Dez 24 15:05:36 server dovecot[222484]: imap(docmax)<224089></EW5RDa3UMPAqAEB>: Debug: sieve: action pipe: running program: rspam-spam I espected to have only ONE pipe when i change flag (in Thunderbird). Instead i get multiple calls. Why is that and how can i fix it? Regards, DocMAX

3 2

LDA ignores virtual mailbox settings
by Toni Mueller 01 Jan '21

01 Jan '21

Hi, I have a Debian/Buster system with Postfix and Dovecot from the Debian repo, and with virtual users only (ie, no system users). These virtual users are having all of their uids, gids, homes and mail quota in an PostgreSQL database. The intent is to have postfix deliver the email via dovecot's LDA, so I can set quota on a per-user basis. But for some odd reason, Dovecot's LDA can't find the user data from the userdb, and then complains about not being able to write to /var/mail. The following example shows how things fail when delivering a message from Postfix's queue (therefore, the passdb failure is expected): In /etc/dovecot.conf, I have this, amongst other things: mail_location = maildir:~/Maildir:INBOX=~/Maildir passdb { driver = sql args = /etc/dovecot/dovecot-sql.conf.ext } userdb { driver = prefetch } userdb { driver = sql args = /etc/dovecot/dovecot-sql.conf.ext } protocol lda { mail_plugins = autocreate quota mail_log trash virtual notify } 16:04:16 postfix/qmgr[4970]: 8CD6CE072E: from=<user(a)example.com>, size=880, nrcpt=1 (queue active) 16:04:16 dovecot: auth: Debug: master in: USER#0111#011user(a)example.com#011service=lda 16:04:16 dovecot: auth: Debug: prefetch(user(a)example.com): passdb didn't return userdb entries, trying the next userdb 16:04:16 dovecot: auth: Debug: sql(user(a)example.com): SELECT '/path-to-mailboxen/' || virtual_users.home AS home, uid , gid , quota as quota_rule FROM virtual_users WHERE email = 'user(a)example.com' AND status = 'A' 16:04:16 dovecot: auth: Debug: userdb out: USER#0111#011user@example.com#011home=/path-to-mailboxen/example.com/user#011uid=12345#011gid=12345#011quota_rule=*:storage=0 ^^^^^^^^^^^^^^^^^^^^^^ This shows that the database lookup works. The intended effect should be that the message is delivered to /path-to-mailboxen/example.com/user/Maidir/new 16:04:16 dovecot: lda(user(a)example.com)<5291><DDxBHYCw6F+rFAAApiCoHg>: Error: setegid(privileged) failed: Operation not permitted 16:04:16 dovecot: lda(user(a)example.com)<5291><DDxBHYCw6F+rFAAApiCoHg>: Error: Mailbox INBOX: open(/var/mail/user(a)example.com) failed: Permission denied (euid=12345(mailbox) egid=12345(mailbox) missing +w perm: /var/mail, we're not in group 8(mail), dir owned by 0:8 mode=0775) ^^^^^^^^^^^^^^^^^^^^^^ And this shows that dovecot-lda just ignores the result. 16:04:16 dovecot: lda(user(a)example.com)<5291><DDxBHYCw6F+rFAAApiCoHg>: Error: Mailbox INBOX: Failed to autocreate mailbox: Mailbox INBOX: open(/var/mail/user(a)example.com) failed: Permission denied (euid=12345(mailbox) egid=12345(mailbox) missing +w perm: /var/mail, we're not in group 8(mail), dir owned by 0:8 mode=0775) 16:04:16 dovecot: lda(user(a)example.com)<5291><DDxBHYCw6F+rFAAApiCoHg>: msgid=<20201226224933.014608(a)laptop.example.com>: save failed to open mailbox INBOX: Mailbox INBOX: Failed to autocreate mailbox: Mailbox INBOX: open(/var/mail/user(a)example.com) failed: Permission denied (euid=12345(mailbox) egid=12345(mailbox) missing +w perm: /var/mail, we're not in group 8(mail), dir owned by 0:8 mode=0775) 16:04:16 postfix/pipe[5284]: 8CD6CE072E: to=<user(a)example.com>, orig_to=<userA(a)example2.com>, relay=dovecot, delay=62083, delays=62083/0.04/0/0.04, dsn=4.3.0, status=deferred (temporary failure) In /etc/postfix/master.cf, I have this to call it: dovecot unix - n n - - pipe flags=DRhu user=_mailbox argv=/usr/lib/dovecot/deliver -f ${sender} -d ${user}@${domain} -a ${recipient} I've tried strace-ing dovecot-lda, but it didn't really help me to understand why it discards the result of the userdb lookup. Can anyone please provide a cluebat, please? Thanks, Toni

5 15

Re: Using dovecot with RoundCubeMail - where is the information for new mail in (blue coloured) directories?
by Scott Ruckh 01 Jan '21

01 Jan '21

On 2020-12-20 17:37, Philip Rhoades wrote: > @lbutlr, > > > On 2020-12-21 00:43, @lbutlr wrote: >> On 20 Dec 2020, at 04:05, Philip Rhoades <phil(a)pricom.com.au> wrote: >>> I guess that is what it seems to look like to me as well - but what >>> does that mean for what I am trying to do? - ie reset the blue folder >>> highlight arbitrarily? >> >> Sound like the Roundcube style choices are outside th purview of >> dovecot. > > > Hmm . . the first response I got here was that it was definitely an > IMAP thing . . I think he was unclear about the actual problem . . > > I will try the RCM forum . . > > Thanks, > > P. I am not sure I have understood this thread completely, but I am running dovecot/RoundCube. If a folder has new messages, then it has a badge that is dark blue (with the theme I am using) with the number of un-read messages. Otherwise it has a badge that is light-blue to indicate un-read messages only. Folders with no badges have zero un-read messages. So there is definitely a distinction for "new" messages.

2 1

migration from 2.0.16
by Barbara M. 01 Jan '21

01 Jan '21

I have an old server with CentOS 6.x and dovecot 2.0.16 (postfix-2.6.6 and roundcube), that was an update from a 1.x many years ago ... Users in /etc/passwd Mailbox format mbox some filtering via procmail About 5.000 users, 1 TB data (/var/mail + /home/users) Obviously I am searching for a smooth upgrade path (with no or minimal downtime and users problems) ;-) Until few days ago my idea was a CentOS 8.x new box with the standard default packages (dovecot-2.3.8, postfix-3.3, ...). Now this can be reconsidered. I suppose there will be a RockyLinux or something equivalent but if there is a good reason I can consider Debian or other OS if they have a decent EOL or some advantages. Anyway, the more relevant problem at the moment is collect info for the best approch to have a smooth dovecot upgrade. My dream is the possibility to configure a new server and rsync the data (/var/mail + /home/users), and, when the tests are satisfiable do the final sync and swap the IP, but I suppose deleting the .imap folders isn't a simple complete solution to compatibility problems. Any hints, links, experiences are appreciated. Thanks, B.

6 9

Initial Replication
by David Morsberger 01 Jan '21

01 Jan '21

I searched the archives and do not see a question/answer for this. Should I perform a set of ordered steps after setting up replication? I have setup dovecot on both ends with identical config except for the remote hostname. Main concern is the existing server gets replicated to the new server. I want nothing from new server replicated to the old server. Risking "Trial and error" and "crossing fingers” doesn’t work with production emails Specifically: Do I need to setup the mail directories for the various virtual users on the remote? If so, do the directories need to empty? Should I initially do a manually one-way synch using doveadm from the existing server? If so, is there a way of turning off automatic replication while the manual one runs? Anything else? T David

2 2

Imap client with ~7500 imap cmds ~250/~500 read() syscall
by Marc Roos 31 Dec '20

31 Dec '20

I am playing a bit with the monitoring, and I have noticed there is one client of the chart with ~7500, while the others are between ~17 and ~60 (dovecot_user_num_cmds). I assume this is related to the client, or is it possible something is wrong on the server side? Is someone having also such numbers or is this really strange. Is it possible to limit these cmd's?

3 8