- dovecot - dovecot.org

Dovecot discards mail in a certain situation
by Aleš Pečnik 21 Apr '21

21 Apr '21

Hi all: I'm using Dovecot with Postfix (via LMTP). Because Postfix doesn't de-duplicate recipients, I'm using sieve to drop duplicates, following this recipe: https://serverfault.com/questions/112958/postfix-aliases-and-duplicate-e-ma… A "sieve_before" script contains: require ["duplicate"]; if duplicate { discard; stop; } It works well, except when the recipient is over quota. Dovecot is configured for tempfail in this case: quota_full_tempfail = yes When a mail arrives for a user who's over quota, Postfix logs this: AF4481700032: to=<?@?>, relay=?[private/dovecot-lmtp], delay=0.05, delays=0.05/0/0/0, dsn=4.2.2, status=deferred (host ?[private/dovecot-lmtp] said: 452 4.2.2 <?@?> Quota exceeded (mailbox for user is full) (in reply to end of DATA command)) Approximately 10 minutes later, Postfix retries, and log shows: dovecot: lmtp(?)<1742912><+EYHBXjidmBAmBoAcAY70w>: sieve: msgid=<489132c2-7b77-cd2f-ce60-439f6558ddcf@?>: marked message to be discarded if not explicitly delivered (discard action) postfix/lmtp[1742691]: AF4481700032: to=<?@?>, relay=?[private/dovecot-lmtp], delay=431, delays=431/0/0/0, dsn=2.0.0, status=sent (250 2.0.0 <?@?> +EYHBXjidmBAmBoAcAY70w Saved) postfix/qmgr[421343]: AF4481700032: removed The message is then nowhere to be found: it's not delivered to mailbox, bounce isn't sent to the sender, mail is not in postfix queue. Apparently it's been discarded. I'd say that at first LMTP delivery attempt, sieve records message as seen, even though it is eventually rejected (452 over quota). During second attempt sieve "recognizes" the message as seen before and discards it. (This is just my interpretation of what could be happening.) Is this a bug or have I misconfigured something? My dovecot -n output is below. Thanks for help, Aleš Pečnik # dovecot -n # 2.3.7.2 (3c910f64b): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.7.2 () # OS: Linux 5.4.0-62-generic x86_64 Ubuntu 20.04.1 LTS auth_mechanisms = plain login auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@: auth_username_format = %Ln auth_worker_max_count = 3 default_vsz_limit = 1 G first_valid_gid = 12 first_valid_uid = 8 hostname = something.org last_valid_gid = 12 last_valid_uid = 8 listen = * mail_access_groups = mail mail_location = maildir:/var/dovecot/%u mail_plugins = " quota mailbox_alias mail_log notify acl" mail_privileged_group = mail namespace { location = maildir:/var/dovecot/public:INDEXPVT=/var/dovecot/%n/public prefix = Public. separator = . subscriptions = no type = public } namespace inbox { inbox = yes location = mailbox Drafts { auto = no special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Trash { auto = no special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-sql-master.conf.ext driver = sql master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { acl = vfile mailbox_alias_new = Sent Messages mailbox_alias_new2 = Sent Items mailbox_alias_new3 = Deleted Items mailbox_alias_new4 = Deleted Messages mailbox_alias_old = Sent mailbox_alias_old2 = Sent mailbox_alias_old3 = Trash mailbox_alias_old4 = Trash quota = maildir:User quota quota_rule = *:storage=200M quota_rule2 = Trash:storage=+50M quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = file:/var/dovecot/%n/sieve;active=/var/dovecot/%n/sieve/active.sieve sieve_before = /var/lib/dovecot/sieve.d/ sieve_extensions = +notify +imapflags +editheader +duplicate } postmaster_address = postmaster(a)something.org protocols = " imap lmtp" quota_full_tempfail = yes service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 0 } process_min_avail = 12 service_count = 0 } service imap-postlogin { executable = script-login /etc/dovecot/postlogin.sh user = $default_internal_user } service imap { executable = imap imap-postlogin } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service pop3-login { inet_listener pop3 { port = 0 } inet_listener pop3s { port = 0 } } ssl_cert = </etc/ssl/my.crt ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lmtp { mail_plugins = " quota mailbox_alias mail_log notify acl sieve" } protocol lda { mail_plugins = " quota mailbox_alias mail_log notify acl sieve" } protocol imap { mail_max_userip_connections = 30 mail_plugins = " quota mailbox_alias mail_log notify acl imap_quota imap_acl" }

2 2

IMAP session hangs on 8k-long commands if COMPRESS=DEFLATE is enabled
by Guilhem Moulin 21 Apr '21

21 Apr '21

Hi there, Dovecot 2.3.7 appears to hang when the client sends a long command after enabling the IMAP COMPRESS extension [RFC 4978]. PoC script attached along with the doveconf(1) output. Without COMPRESS=DEFLATE, and with the default ‘imap_max_line_length’ value (64k) I'm able send commands up to 65539 bytes long (that's 3 bytes more than 2¹⁶, so maybe the leading tag and the trailing CRLF aren't counted), after which Dovecot rightfully replies with tagged BAD responses. $ COMPRESS=n /tmp/many-fetch.pl 65539 […] C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [65539 bytes] S: x OK Fetch completed (0.005 + 0.001 + 0.004 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [65540 bytes] S: x BAD Error in IMAP command UID FETCH: IMAP command line too large (0.001 + 0.001 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [65541 bytes] S: x BAD Error in IMAP command UID FETCH: IMAP command line too large (0.001 + 0.001 secs). With COMPRESS=DEFLATE however, the server never sends anything back when the command size exceeds 8192 bytes: $ COMPRESS=y /tmp/many-fetch.pl 8192 […] S: x OK [READ-ONLY] Examine completed (0.001 + 0.000 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [8191 bytes] S: x OK Fetch completed (0.001 + 0.000 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [8192 bytes] S: x OK Fetch completed (0.001 + 0.000 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [8193 bytes] [ _hangs until the server terminates the session for inactivity_ ] S: * BYE Disconnected for inactivity. strace(1) show that Dovecot didn't send the response, even though the server log indicates that it's able to fully read the command. Interestingly, when setting ‘imap_max_line_length’ to 4096, the server also hangs after receiving a command of size >4096 bytes (instead of replying with a tagged BAD response). $ COMPRESS=y /tmp/many-fetch.pl 4096 C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [4095 bytes] S: x OK Fetch completed (0.001 + 0.000 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [4096 bytes] S: x OK Fetch completed (0.001 + 0.000 secs). C: x UID FETCH 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,… [4097 bytes] [ _hangs until the server terminates the session for inactivity_ ] I'm also able to reproduce the buggy behavior against Dovecot versions found in other Debian releases. It's identical with 2.3.4 (Buster), the server also hang past 8192 bytes. 2.2.27 (Stretch) and 2.2.13 (Jessie) are able to send replies to 8k-long commands, but hang past 65536 bytes. For all 3 releases, without COMPRESS=DEFLATE Dovecot doesn't hang and rightfully sends tagged BAD responses to commands of size >65539 bytes. So while the hanging behavior appears to be a long-standing bug, the fact that's it's now hit at min($imap_max_line_length, 8192) instead of $imap_max_line_length seems to be a regression from somewhere between 2.2.27 and 2.3.4. Cheers, -- Guilhem.

3 2

Quota warning invoked twice when moving messages to Trash folder
by Dan Conway 20 Apr '21

20 Apr '21

Hello all, I currently have quota warnings enabled through the following configuration on Dovecot 2.2.19: plugin { quota = dict:User quota::file:%h/Maildir/dovecot-quota quota_warning = storage=100%% quota-warning 100 %u %{userdb:quota_rule} %i quota_warning2 = storage=95%% quota-warning 95 %u %{userdb:quota_rule} %i quota_warning3 = storage=90%% quota-warning 90 %u %{userdb:quota_rule} %i } service quota-warning { executable = script /usr/local/bin/quota-warning.pl group = mailusers unix_listener quota-warning { group = mailusers mode = 0775 } } When using Thunderbird as a client for testing, I fill up an inbox until one of the warnings is invoked. (90%). The script runs, and the mailbox sits at about 93%. When moving a handful of messages to the Trash folder, the 100% quota-warning service is invoked. But getting the quota through 'doveadm quota get -u user' yields that same 93% used. I ran strace on the main process that is taking in IMAP commands from the Thunderbird client. Two lines are worth noting from that output: 134 uid store 2311:2312,2314:2317 +Flags (\\Seen)\r\n", 3619 137 uid move 2311:2312,2314:2317 \"INBOX.Trash\"\r\n", 3553 The first line sets the "Seen" flag on a list of messages, while the second line moves the messages to the "INBOX.Trash" folder. My guess as to what is happening is when the "uid move" command is received, The messages are copied first, then removed from the server, causing the quota to spike for a very short amount of time. Are those suspicions correct? After looking at RFC 6851, Section 3.3 (Semantics of MOVE and UID MOVE), it states: "Each message included in the set is moved, rather than copied ... ". Any idea why the spike in quota usage occurs? Thanks.

1 0

POP3 server fails to replicate deletions
by Stephen Gildea 20 Apr '21

20 Apr '21

When I delete messages with POP3, the change doesn't get replicated. To reproduce: run the pop3 server from the command line, type this to it: DELE 1 QUIT This correctly deletes the first message on the local server, but it does not send a replication notify message. Poking around a bit with gdb, it appears that the pop3 server decides incorrectly that the changes it is making are not worth notifying about. It is hard to find something that doesn't happen, and I would appreciate suggestions for further debugging this case. I do get the expected replication when new mail arrives and also when I manipulate my mailbox with IMAP. I have tested this with Dovecot 2.3.7.2 (on Ubuntu 20.04) and with the git HEAD, 2.3.14+. What may be the same bug was reported on this list last August against Dovecot 2.3.8, but it got no reply then. My mail store is on a local ext4 file system. I have replication enabled in my configuration: $ doveconf mail_plugins mail_plugins = mail_crypt notify replication $ doveconf -f protocol=pop3 mail_plugins mail_plugins = mail_crypt notify replication

2 1

fts_enforced=yes or body return NO [SERVERBUG] Internal error occurred when SEARCH
by Alessio Cecchi 20 Apr '21

20 Apr '21

Hi, we have some issues related to the fts_enforced option in dovecot config. Following the table in the docs (https://doc.dovecot.org/settings/plugin/fts-plugin/#plugin-fts-setting-fts-…) we cannot figure out the current behavior of the flag. In our case we have dovecot up and running, dovecot indexes updated using doveadm command (fts rescan/index), but solr instance offline for testing. We try to search using header subject. We tried "yes" and "body" fts_enforced flag's options but we get in telnet session: C: a search header subject "test" S: a NO [SERVERBUG] Internal error occurred. Refer to server log for more information. [2021-04-20 15:52:57] (0.005 + 0.000 + 0.004 secs). But dovecot-log don't include any specific error, except for a generic: Apr 20 15:52:57 imap dovecot: imap(test(a)emailtest.net) session=<t7RIwxyaxMxbj8Qu>: Error: fts_solr: Lookup failed: connect(10.0.0.2:8983) failed: Connection refused Is there any flag or method to check if a solr instance is up and running and, eventually, switching to the internal dovecot indexes instead ? Using the "no" option, for enforced flag, for us is unavailable because the related search could be too long and we could have timeout waiting the results. We aim to achieve the following result: if our solr instance is unavailable, we should switch the imap searches from full-text to headers only. Could you suggest something in this direction? We running doveoct 2.3.14. Thanks -- Alessio Cecchi Postmaster @ http://www.qboxmail.it https://www.linkedin.com/in/alessice

1 0

Replication between three servers
by Miro Igov 20 Apr '21

20 Apr '21

Hello, I have dovecot replication working fine between 2 servers with LDA configured to sync mail coming from Postfix. These 2 servers are used in master/slave mode - only the one server is used by IMAP clients and the second server is just a backup in case of failure. Second server is also MX backup and it happens that incoming mails come from it via Postfix and get synchronized with the first server. I would like to include third server as backup. How could be done while mail_replica accepts only one host?

2 4

What is the proper <luceneMatchVersion> value in solrconfig.xml for dovecot?
by Steve Dondley 19 Apr '21

19 Apr '21

I'm looking at config documentation for solr on dovecot: https://doc.dovecot.org/configuration_manual/fts/solr/ In the suggested solrconfig.xml file (https://raw.githubusercontent.com/dovecot/core/master/doc/solr-config-7.7.0…) it has the following line: <luceneMatchVersion>7.7.0</luceneMatchVersion> I'm running solr version 8.8.1, however. I'm wondering if I should change this line to: <luceneMatchVersion>8.8.1</luceneMatchVersion> Things seems to work fine with the 7.7.0 value but there is a comment in the config file that says:  I'm not familiar with Lucene or Solr so I'm uncertain as to what to set this to. Thanks.

1 0

Sending alert to client after quota has reached 100%
by Dan Conway 19 Apr '21

19 Apr '21

Hello all, I hope you are doing well. In regards to the quota_warning configuration, more specifically when an email is sent to the user when quota has passed a certain percentage, i.e.: quota_warning = <limit configuration> <quota-warning socket name> <parameters> #quota_warning2 = ... #quota_warning3 = ..etc.. Is it possible to communicate with the client by sending an ALERT IMAP message to stdout much like post-login scripting, rather than sending an email to the user? i.e: #!/bin/sh # Message goes to stdout printf "* OK [ALERT] This is an alert\r\n" exec "$@" Thank you.

2 1

Only INBOX is searched when using gmail with dovecot FTS and solr
by hi-dovecot＠mcarle.io 16 Apr '21

16 Apr '21

Hi, I am using https://github.com/docker-mailserver/docker-mailserver to setup my mailserver and added solr for full-text search. Nearly everything is working as expected. But I have a problem with the full-text search, as I have a lot of folders in my mailbox and I just can't find any mail in any of these folders except INBOX when I search with the Gmail android app. After analyzing the logs, I assume, that the bug is in the Gmail app, as it is only searching in the INBOX folder. I searched a bit and found some reports of other users confirming that. As this "design decision" (or bug) is quite old, I assume that Google won't fix it. But as many people are using Gmail, I assume that there must be a way to search all folders with Gmail. Sadly, I haven't found any possibility yet. As a workaround one may put a proxy in between and simply remove the "+box:123" filter from the path (e.g. with a small regex). With that, solr will return all found mails from that user, regardless of the folder. This may be a quite simple, but very dirty hack. Hence, I hope that you guys know a better way, a good way to overcome this issue and enable all Gmail users to search for email in all folders. Best regards, Marcel

3 2

mail_max_userip_connections per remote IP not working
by Alessio Cecchi 16 Apr '21

16 Apr '21

Hi, I'm tryng to set a specific mail_max_userip_connections for a remote IP (webmail IMAP software), but it seems not working: remote 1.2.3.4 { protocol imap { mail_max_userip_connections = 100 } } and also this isn't working remote 1.2.3.4 { mail_max_userip_connections = 100 } I insert it at the end of 20-imap.conf file. Is something wrong or is not supported? I'm running dovecot 2.3.14. Thanks -- Alessio Cecchi Postmaster @http://www.qboxmail.it https://www.linkedin.com/in/alessice

2 1