May 2015 - dovecot - dovecot.org

[Dovecot] Replication and public namespaces
by Markus Weippert 11 Apr '16

11 Apr '16

Hi, I'm having some issues with replicating public namespaces. Everything seems to work fine for private namespaces, but while importing some huge mailboxes (many small mails) into a public namespace via imapsync, something goes wrong. The expected mail flow is: old-server (imapsync)> new-server1 (replication)> new-server2 But then, dovecot seems to run into race conditions when the replications process tries to sync the same public mailbox under two or more different users at the same time. As a result, messages get duplicated, new-server2 sends those back to new-server1 which then starts to produce duplicates too. If I don't kill the processes in time and delete the faulty mailbox, they start to produce thousands of mails. In fact, server2 should not export messages at all, since it's not productive yet and does not get any mail except from the replication. The only thing getting logged (only few compared to the huge amount of duplicates produced): "dsync-server(user(a)example.com): Warning: Maildir /...: Expunged message reappeared, giving a new UID" Is there any way to fix this? Regards, Markus doveadm_password = *********** doveadm_port = ***** mail_home = /var/vmail/private/%d/%n mail_location = maildir:~/mail mail_plugins = acl virtual listescape notify replication namespace inbox { inbox = yes location = prefix = separator = / type = private } namespace legacy { alias_for = # ... } namespace ns_public { list = children location = maildir:/var/vmail/public/%d prefix = Public/ separator = / subscriptions = no type = public } namespace virtual { # ... } plugin { mail_replica = tcps:************** } protocols = imap lmtp sieve replication_dsync_parameters = -d -l 60 -N -x virtual -x ns_public -U replication_full_sync_interval = 4 hours replication_max_conns = 20 service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } } service doveadm { inet_listener { port = 8143 ssl = yes } process_min_avail = 5 user = vmail vsz_limit = 4 G } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 user = vmail } }

3 7

basic configuratio error
by soko.tica 01 Feb '16

01 Feb '16

Hello list, After many hours I still fail to configure properly dovecot as the LDA with opensmtpd, for system users only. I don't want pop at all, jus lmtp and imap, but still haven't stripped it. Here is the info, thanks in advance for your response: ===== # doveconf -n # 2.2.14: /etc/dovecot/dovecot.conf # OS: OpenBSD 5.6 i386 auth_verbose = yes auth_verbose_passwords = yes disable_plaintext_auth = no first_valid_uid = 1000 hostname = imap.home.lan imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes lda_original_recipient_header = X-Original-To mail_debug = yes mail_location = maildir:~/Maildir managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave duplicate mbox_write_locks = fcntl mmap_disable = yes namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = bsdauth } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } pop3_client_workarounds = outlook-no-nuls oe-ns-eoh postmaster_address = postmaster(a)mileva.home.lan protocols = imap pop3 lmtp sieve service auth-worker { user = root } service auth { unix_listener auth-userdb { group = _dovenull mode = 0666 user = _dovenull } } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } process_min_avail = 0 service_count = 1 } service imap { process_limit = 1024 } service lmtp { unix_listener lmtp { mode = 0666 } } service managesieve-login { inet_listener sieve { port = 4190 } inet_listener sieve_deprecated { port = 2000 } } ssl_cert = </etc/ssl/dovecotcert.pem ssl_key = </etc/ssl/private/dovecot.pem userdb { driver = passwd } verbose_ssl = yes protocol lmtp { mail_plugins = } protocol lda { mail_plugins = } # /usr/local/sbin/dovecot -F Oct 28 13:00:01 mileva newsyslog[1110]: logfile turned over Oct 28 14:30:46 mileva dovecot: master: Dovecot v2.2.14 starting up for imap, pop3, lmtp, sieve Oct 28 14:30:46 mileva dovecot: master: Error: service(aggregator): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(pop3): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: open(/var/dovecot/login-master-notifya57a2b8821460734) failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(pop3-login): safe_mkstemp(/var/dovecot/login-master-notify) failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(log): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(lmtp): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(ipc): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(indexer-worker): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(indexer): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(imap): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(imap-urlauth-worker): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(imap-urlauth): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: open(/var/dovecot/login-master-notify5dd2a3a0591effcc) failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(imap-urlauth-login): safe_mkstemp(/var/dovecot/login-master-notify) failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: open(/var/dovecot/login-master-notify152a1bb891f6f764) failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(imap-login): safe_mkstemp(/var/dovecot/login-master-notify) failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(doveadm): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(dns_client): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(director): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(dict): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(config): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(auth): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(auth-worker): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(anvil): command startup failed, throttling for 2 secs Oct 28 14:30:56 mileva dovecot: master: Error: service(ssl-params): command startup failed, throttling for 2 secs [snipped 18 pages of /var/log/maillog] Oct 28 20:47:44 mileva dovecot: master: Dovecot v2.2.14 starting up for imap, pop3, lmtp, sieve Oct 28 20:47:44 mileva dovecot: master: Error: service(aggregator): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(pop3): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: open(/var/dovecot/login-master-notifydd522320d99e7f4c) failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(pop3-login): safe_mkstemp(/var/dovecot/login-master-notify) failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(log): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(lmtp): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(ipc): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(indexer-worker): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(indexer): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(imap): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(imap-urlauth-worker): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(imap-urlauth): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: open(/var/dovecot/login-master-notify95aa9b38117677e4) failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(imap-urlauth-login): safe_mkstemp(/var/dovecot/login-master-notify) failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: open(/var/dovecot/login-master-notify4d021350494e6f7c) failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(imap-login): safe_mkstemp(/var/dovecot/login-master-notify) failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(doveadm): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(dns_client): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(director): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(dict): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(config): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(auth): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(auth-worker): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(anvil): command startup failed, throttling for 2 secs Oct 28 20:47:46 mileva dovecot: master: Warning: Killed with signal 2 (by pid=0 uid=0 code=kill) Oct 28 20:47:46 mileva dovecot: master: Error: open(/var/dovecot/.temp.mileva.home.lan.17422.055a8b6881266714) failed: Too many open files Oct 28 20:47:46 mileva dovecot: master: Error: file_dotlock_open(/var/dovecot/instances) failed: Too many open files Oct 28 20:47:46 mileva dovecot: master: Error: service(ssl-params): command startup failed, throttling for 2 secs # /usr/sbin/smtpd -n configuration OK

2 3

userdb username change ignored when using (My)SQL was: Re: userdb username changed
by Péter Márton 17 Dec '15

17 Dec '15

Hi! It works when using LDAP. I've duplicated the "username change" debug line, just to see that the variables are really updated: Apr 20 14:30:27 imap21 dovecot: auth-worker(27127): Debug: sql(ppp(a)example.net): username changed ppp(a)example.net -> uppp Apr 20 14:30:27 imap21 dovecot: auth-worker(27127): Debug: sql(uppp): username changed uppp -> uppp Apr 20 14:30:27 imap21 dovecot: auth: Debug: userdb out: USER#0111#011ppp@example.net#011home=/home/ppp#011uid=500#011gid=500#011quota_rule=*:storage=3100b:messages=1024 And the answer is of course yes. Just the userdb out string has the wrong value. The right value is lost somewhere. But where? Regards, Peter Log of the (for me unusable) LDAP query: Apr 20 14:28:07 imap21 dovecot: auth: Debug: master in: USER#0111#011m2500j6#011service=doveadm Apr 20 14:28:07 imap21 dovecot: auth: Debug: ldap(m2500j6): user search: base=ou=Users,ou=Mail,dc=bnet,dc=at scope=subtree filter=(uid=m2500j6) fields=mailHost Apr 20 14:28:07 imap21 dovecot: auth: Debug: ldap(m2500j6): result: mailHost=localhost; mailHost unused Apr 20 14:28:07 imap21 dovecot: auth: Debug: ldap(m2500j6): username changed m2500j6 -> localhost Apr 20 14:28:07 imap21 dovecot: auth: Debug: ldap(localhost): username changed localhost -> localhost Apr 20 14:28:07 imap21 dovecot: auth: Debug: ldap(localhost): result: mailHost=localhost Apr 20 14:28:07 imap21 dovecot: auth: Debug: userdb out: USER#0111#011localhost#011

2 4

doveadm-server does NOT know fts commands
by SATOH Fumiyasu 16 Dec '15

16 Dec '15

Hi, I'm using Dovecot 2.2.13 with Dovecot Director, Doveadm server and FTS. I want to run doveadm on Director hosts, but Dovecot server on backend hosts reject "fts *" commands. On Dovecot Director hosts: # doveadm fts optimize -u u0001(a)example.jp # echo $? 75 # doveadm fts rescan -u u0001(a)example.jp # echo $? 75 doveadm index -u u0001(a)example.jp INBOX # echo $? 0 On Dovecot backend hosts: # grep doveadm /var/log/maillog Jul 14 10:25:40 backend dovecot: doveadm: Error: doveadm: Client sent unknown command: fts optimize Jul 14 10:25:51 backend dovecot: doveadm: Error: doveadm: Client sent unknown command: fts rescan Is this doveadm-server restriction? On Dovecot backend hosts (2): # doveadm fts optimize -u u0001(a)example.jp -S /var/run/dovecot/doveadm-server # echo $? 75 # doveadm fts rescan -u u0001(a)example.jp -S /var/run/dovecot/doveadm-server # echo $? 75 # doveadm index -u u0001(a)example.jp -S /var/run/dovecot/doveadm-server INBOX # echo $? 0 # doveadm fts optimize -u u0001(a)example.jp # echo $? 0 # doveadm fts rescan -u u0001(a)example.jp # echo $? 0 # doveadm index -u u0001(a)example.jp INBOX # echo $? 0 -- -- Name: SATOH Fumiyasu @ OSS Technology Corp. (fumiyas @ osstech co jp) -- Business Home: http://www.OSSTech.co.jp/ -- GitHub Home: https://GitHub.com/fumiyas/ -- PGP Fingerprint: BBE1 A1C9 525A 292E 6729 CDEC ADC2 9DCA 5E1C CBCA

2 3

mmap_disable=yes not honored always
by Peter Benko 06 Dec '15

06 Dec '15

<Sorry for the resend, it seems that the formatting of my original mail got garbled> Hi all, I'm experimenting with having the mail store on a 9p file system that lacks mmap() functionality. So I disabled it in dovecot: mmap_disable = yes However, I keep getting the following error messages in my log: Oct 15 16:55:00 computer-name dovecot: imap user(a)domain.com[192.168.1.3] Error: mmap() failed with file /mail/mailboxes/domain.com/user/indexpvt-accounts/other-user/mailboxes/INBOX/dovecot.index.pvt.log: Invalid argument I only get those messages for the shared folder's private indexes. This setup works perfectly on an ext2/3/4 fs. Any opinions, is this a bug? Is mmap_disable not mandatory for all operations? Best regards, Peter

2 1

[PATCH] [dovecot 2.2.9] Quota warnings ignored with FS quotas
by Grzegorz Nosek 06 Dec '15

06 Dec '15

Hi all, I noticed that in some circumstances quota warnings are ignored. The bug arises when both of the following are used: 1. percentage-based quota warnings, i.e.: quota_warning = storage=1%% quota-warning 1 %u 2. filesystem quota backend (and probably others, except for quotas configured directly in dovecot config) Percentage-based quota warnings have rule.bytes_limit recalculated based on root_set->default_rule.bytes_limit, however this value is zero when FS quotas are in use. Real quota values (from quotactl) are fetched very late, in quota_warnings_execute() but at that point no recalculation happens. As the warning rules have bytes_limit==0, they're effectively ignored. The patch below enables quota warnings to be sent when using filesystem (and possibly maildirsize-based) quotas. Based and tested on Ubuntu 14.04's dovecot 2.2.9. Best regards, Grzegorz Nosek diff --git a/src/plugins/quota/quota.c b/src/plugins/quota/quota.c index adbd70d..8e4d7e0 100644 --- a/src/plugins/quota/quota.c +++ b/src/plugins/quota/quota.c @@ -1163,6 +1163,8 @@ static void quota_warnings_execute(struct quota_transaction_context *ctx, &count_current, &count_limit) < 0) return; + quota_root_recalculate_relative_rules(root->set, bytes_limit, count_limit); + bytes_before = bytes_current - ctx->bytes_used; count_before = count_current - ctx->count_used; for (i = 0; i < count; i++) {

3 3

[Dovecot] lmtp_rcpt_check_quota not working
by Bernhard Schmidt 06 Dec '15

06 Dec '15

Hi, Dovecot 2.2.9 (Debian package from backports) does not seem to respect lmtp_rcpt_check_quota # 2.2.9: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.4 deliver_log_format = from=<%e>, size=%p, message-id=<%m>, status=%$ disable_plaintext_auth = no lmtp_rcpt_check_quota = yes mail_debug = yes mail_gid = mstore mail_location = maildir:~/Maildir:INDEX=/var/cache/dovecot/index/%-1.1u/%u mail_plugins = quota mail_uid = mstore managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave mmap_disable = yes namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = INBOX. separator = . type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { quota = maildir quota_rule = ?:storage=512M quota_rule2 = INBOX.Trash:ignore quota_status_nouser = DUNNO quota_status_overquota = 452 4.2.2 Mailbox is full quota_status_success = DUNNO quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=85%% quota-warning 85 %u sieve = ~/currently-active-script.sieve sieve_dir = ~/sieve } pop3_uidl_format = %08Xv%08Xu protocols = imap lmtp sieve pop3 quota_full_tempfail = yes service anvil { client_limit = 3000 unix_listener anvil { group = sudo mode = 0660 } } service auth { client_limit = 3000 unix_listener auth-userdb { group = mstore mode = 0660 user = mstore } } service imap-login { client_limit = 1024 inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } process_limit = 2500 process_min_avail = 4 service_count = 0 } service lmtp { inet_listener lmtp { address = xxx port = 24 } } service managesieve-login { inet_listener sieve { port = 2000 } service_count = 1 } service managesieve { process_limit = 1024 } service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 ssl = yes } } service quota-status { client_limit = 5 executable = quota-status -p postfix inet_listener { address = xxx port = 12340 } } service quota-warning { executable = script /etc/dovecot/quotawarnmsg.sh unix_listener quota-warning { user = mstore } user = mstore } ssl_cert = </etc/dovecot/dovecot.pem ssl_key = </etc/dovecot/private/dovecot.pem userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } verbose_proctitle = yes protocol imap { mail_plugins = quota imap_quota imap_zlib } protocol lda { mail_plugins = quota sieve } protocol lmtp { mail_plugins = quota sieve } so Dovecot should reject in RCPT TO phase, but according to the logs of the upstream Postfix it does so after END-OF-DATA (host xxx[2001:xxx] said: 452 4.2.2 <user(a)domain.de> Quota exceeded (mailbox for user is full) (in reply to end of DATA command)) I haven't seen any commits regarding to this in 2.2.10 or tip. Regards, Bernhard

3 6

imap_logout_format improvements / ideas
by Timo Sirainen 03 Oct '15

03 Oct '15

http://hg.dovecot.org/dovecot-2.2/rev/93bba97afb2a http://hg.dovecot.org/dovecot-2.2/rev/e0a17714f0c9 So all in all, you can now add to imap_logout_format: # %i - total number of bytes read from client # %o - total number of bytes sent to client # %{fetch_hdr_count} - Number of mails with mail header data sent to client # %{fetch_hdr_bytes} - Number of bytes with mail header data sent to client # %{fetch_body_count} - Number of mails with mail body data sent to client # %{fetch_body_bytes} - Number of bytes with mail body data sent to client # %{deleted} - Number of mails where client added \Deleted flag # %{expunged} - Number of mails that client expunged # %{trashed} - Number of mails that client copied/moved to the # special_use=\Trash mailbox. Any thoughts on what else would be useful? I was also thinking about separating (deletes and) expunges from Trash vs non-Trash, but I'm not sure how useful that would be (and what would it be called?) It's also a bit annoying that a copy+expunge to a Trash will return expunged=1 trashed=1, but a move to Trash (via IMAP MOVE) will return expunged=0 trashed=1. But trying to fix that would be too much trouble.

4 4

Multiple passwords for a user (SQL)
by Marco Fretz 22 Sep '15

22 Sep '15

Hi everyone, I'm not sure if this has been discuessed already as I coun't find anything in the archives - maybe I'm looking for the wrong thing. I want to build something similar to googles "app passwords" where your use a different password (strong, auto generated) for each device / app. In general I think this is pretty easy on the database side as I just need a password table and link say the passwords with an user_id to the corresponding user in the users table - one use has many passwords (1:n). for dovecot this means that it will get multiple rows with passwords back from the "password_query". is there a way to tell dovecot to check all those returned passwords and "pass" the request if one of those passwords match? thank you, best regards Marco

2 3

[Dovecot] Pigeonhole sieve re-filter extension?
by T.B. 22 Sep '15

22 Sep '15

Dear Stephan Bosch, dear Dovecot community, "sieve-filter" re-filtering of existing mails is already a great progress for everyone using sieve! But as far as I know this functionality is just accessible by the server administrator. In the meanwhile everyone knows the comfort of re-executing filter scripts on his own existing mailboxes - triggered by himself! This service is already very widespread. Examples are MUA's like Outlook or Thunderbird which provide this service just on the client side. Or almost every web-mail provider offers this. Is there already an managesieve protocol extension that allows client triggered server side re-filtering? If not, maybe you could design a RFC for this? What do you think? Best regards. T.B.

4 8