November 2015 - dovecot

[Dovecot] Replication and public namespaces
by Markus Weippert 11 Apr '16

11 Apr '16

Hi, I'm having some issues with replicating public namespaces. Everything seems to work fine for private namespaces, but while importing some huge mailboxes (many small mails) into a public namespace via imapsync, something goes wrong. The expected mail flow is: old-server (imapsync)> new-server1 (replication)> new-server2 But then, dovecot seems to run into race conditions when the replications process tries to sync the same public mailbox under two or more different users at the same time. As a result, messages get duplicated, new-server2 sends those back to new-server1 which then starts to produce duplicates too. If I don't kill the processes in time and delete the faulty mailbox, they start to produce thousands of mails. In fact, server2 should not export messages at all, since it's not productive yet and does not get any mail except from the replication. The only thing getting logged (only few compared to the huge amount of duplicates produced): "dsync-server(user(a)example.com): Warning: Maildir /...: Expunged message reappeared, giving a new UID" Is there any way to fix this? Regards, Markus doveadm_password = *********** doveadm_port = ***** mail_home = /var/vmail/private/%d/%n mail_location = maildir:~/mail mail_plugins = acl virtual listescape notify replication namespace inbox { inbox = yes location = prefix = separator = / type = private } namespace legacy { alias_for = # ... } namespace ns_public { list = children location = maildir:/var/vmail/public/%d prefix = Public/ separator = / subscriptions = no type = public } namespace virtual { # ... } plugin { mail_replica = tcps:************** } protocols = imap lmtp sieve replication_dsync_parameters = -d -l 60 -N -x virtual -x ns_public -U replication_full_sync_interval = 4 hours replication_max_conns = 20 service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } } service doveadm { inet_listener { port = 8143 ssl = yes } process_min_avail = 5 user = vmail vsz_limit = 4 G } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 user = vmail } }

3 7

basic configuratio error
by soko.tica 01 Feb '16

01 Feb '16

Hello list, After many hours I still fail to configure properly dovecot as the LDA with opensmtpd, for system users only. I don't want pop at all, jus lmtp and imap, but still haven't stripped it. Here is the info, thanks in advance for your response: ===== # doveconf -n # 2.2.14: /etc/dovecot/dovecot.conf # OS: OpenBSD 5.6 i386 auth_verbose = yes auth_verbose_passwords = yes disable_plaintext_auth = no first_valid_uid = 1000 hostname = imap.home.lan imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes lda_original_recipient_header = X-Original-To mail_debug = yes mail_location = maildir:~/Maildir managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave duplicate mbox_write_locks = fcntl mmap_disable = yes namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = bsdauth } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } pop3_client_workarounds = outlook-no-nuls oe-ns-eoh postmaster_address = postmaster(a)mileva.home.lan protocols = imap pop3 lmtp sieve service auth-worker { user = root } service auth { unix_listener auth-userdb { group = _dovenull mode = 0666 user = _dovenull } } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } process_min_avail = 0 service_count = 1 } service imap { process_limit = 1024 } service lmtp { unix_listener lmtp { mode = 0666 } } service managesieve-login { inet_listener sieve { port = 4190 } inet_listener sieve_deprecated { port = 2000 } } ssl_cert = </etc/ssl/dovecotcert.pem ssl_key = </etc/ssl/private/dovecot.pem userdb { driver = passwd } verbose_ssl = yes protocol lmtp { mail_plugins = } protocol lda { mail_plugins = } # /usr/local/sbin/dovecot -F Oct 28 13:00:01 mileva newsyslog[1110]: logfile turned over Oct 28 14:30:46 mileva dovecot: master: Dovecot v2.2.14 starting up for imap, pop3, lmtp, sieve Oct 28 14:30:46 mileva dovecot: master: Error: service(aggregator): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(pop3): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: open(/var/dovecot/login-master-notifya57a2b8821460734) failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(pop3-login): safe_mkstemp(/var/dovecot/login-master-notify) failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(log): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(lmtp): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(ipc): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(indexer-worker): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(indexer): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(imap): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(imap-urlauth-worker): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(imap-urlauth): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: open(/var/dovecot/login-master-notify5dd2a3a0591effcc) failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(imap-urlauth-login): safe_mkstemp(/var/dovecot/login-master-notify) failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: open(/var/dovecot/login-master-notify152a1bb891f6f764) failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(imap-login): safe_mkstemp(/var/dovecot/login-master-notify) failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(doveadm): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(dns_client): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(director): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(dict): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(config): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(auth): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(auth-worker): pipe() failed: Too many open files Oct 28 14:30:46 mileva dovecot: master: Error: service(anvil): command startup failed, throttling for 2 secs Oct 28 14:30:56 mileva dovecot: master: Error: service(ssl-params): command startup failed, throttling for 2 secs [snipped 18 pages of /var/log/maillog] Oct 28 20:47:44 mileva dovecot: master: Dovecot v2.2.14 starting up for imap, pop3, lmtp, sieve Oct 28 20:47:44 mileva dovecot: master: Error: service(aggregator): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(pop3): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: open(/var/dovecot/login-master-notifydd522320d99e7f4c) failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(pop3-login): safe_mkstemp(/var/dovecot/login-master-notify) failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(log): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(lmtp): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(ipc): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(indexer-worker): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(indexer): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(imap): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(imap-urlauth-worker): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(imap-urlauth): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: open(/var/dovecot/login-master-notify95aa9b38117677e4) failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(imap-urlauth-login): safe_mkstemp(/var/dovecot/login-master-notify) failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: open(/var/dovecot/login-master-notify4d021350494e6f7c) failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(imap-login): safe_mkstemp(/var/dovecot/login-master-notify) failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(doveadm): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(dns_client): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(director): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(dict): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(config): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(auth): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(auth-worker): pipe() failed: Too many open files Oct 28 20:47:44 mileva dovecot: master: Error: service(anvil): command startup failed, throttling for 2 secs Oct 28 20:47:46 mileva dovecot: master: Warning: Killed with signal 2 (by pid=0 uid=0 code=kill) Oct 28 20:47:46 mileva dovecot: master: Error: open(/var/dovecot/.temp.mileva.home.lan.17422.055a8b6881266714) failed: Too many open files Oct 28 20:47:46 mileva dovecot: master: Error: file_dotlock_open(/var/dovecot/instances) failed: Too many open files Oct 28 20:47:46 mileva dovecot: master: Error: service(ssl-params): command startup failed, throttling for 2 secs # /usr/sbin/smtpd -n configuration OK

2 3

Sieve "foreverypart" and "mime" extensions (RFC 5703) now implemented
by Stephan Bosch 24 Jan '16

24 Jan '16

Hi, I have finally managed to implement the Sieve "foreverypart" and "mime" extensions (RFC 5703). These are now included in the main Mercurial repository and will be included in the next release. I will be fixing a few minor details in the coming week; e.g., interaction with "include" and "editheader" extensions needs more work. Still, in the mean time, those of you who are interested can already test this feature a bit. It is not a small feature, so knowing bugs early is helpful. You can install it from the Mercurial repository or get the latest Debian packages from Xi. When I finish the last details (probably next week), I will release a first RC version of Pigeonhole v0.4.10 right away. Regards, Stephan.

2 2

doveadm import - strange warning
by Ralf Hildebrandt 08 Jan '16

08 Jan '16

I'm using the doveadm import command to extract mails from my backup: % doveadm import -u restore(a)backup.invalid mdbox:/home/copymail/mdbox "" \ mailbox INBOX header Delivered-To backup+$SEARCHPATTERN(a)backup.invalid This works like a charm, I'm getting all the mails, and I can access the newly generated mailbox. But when executing the command I'm getting an annoying warning: doveadm(restore(a)backup.invalid): Error: chdir(/root/) failed: Permission denied (euid=1001(copymail) egid=1001(copymail) missing +x perm: /root, dir owned by 0:0 mode=0700) doveadm(restore(a)backup.invalid): Error: chdir(/root) failed: Permission denied I'm using a file based authentication: # cat passwd restore@backup.invalid:{SHA}+/...the.hash...:1001:1001:(gecos):/home/restore:: So why is "restore(a)backup.invalid" being associated with /root? I'm using 2:2.2.18-1~auto+91, but the warning has been around for ages :) -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein

5 7

userdb username change ignored when using (My)SQL was: Re: userdb username changed
by Péter Márton 17 Dec '15

17 Dec '15

Hi! It works when using LDAP. I've duplicated the "username change" debug line, just to see that the variables are really updated: Apr 20 14:30:27 imap21 dovecot: auth-worker(27127): Debug: sql(ppp(a)example.net): username changed ppp(a)example.net -> uppp Apr 20 14:30:27 imap21 dovecot: auth-worker(27127): Debug: sql(uppp): username changed uppp -> uppp Apr 20 14:30:27 imap21 dovecot: auth: Debug: userdb out: USER#0111#011ppp@example.net#011home=/home/ppp#011uid=500#011gid=500#011quota_rule=*:storage=3100b:messages=1024 And the answer is of course yes. Just the userdb out string has the wrong value. The right value is lost somewhere. But where? Regards, Peter Log of the (for me unusable) LDAP query: Apr 20 14:28:07 imap21 dovecot: auth: Debug: master in: USER#0111#011m2500j6#011service=doveadm Apr 20 14:28:07 imap21 dovecot: auth: Debug: ldap(m2500j6): user search: base=ou=Users,ou=Mail,dc=bnet,dc=at scope=subtree filter=(uid=m2500j6) fields=mailHost Apr 20 14:28:07 imap21 dovecot: auth: Debug: ldap(m2500j6): result: mailHost=localhost; mailHost unused Apr 20 14:28:07 imap21 dovecot: auth: Debug: ldap(m2500j6): username changed m2500j6 -> localhost Apr 20 14:28:07 imap21 dovecot: auth: Debug: ldap(localhost): username changed localhost -> localhost Apr 20 14:28:07 imap21 dovecot: auth: Debug: ldap(localhost): result: mailHost=localhost Apr 20 14:28:07 imap21 dovecot: auth: Debug: userdb out: USER#0111#011localhost#011

2 4

doveadm-server does NOT know fts commands
by SATOH Fumiyasu 16 Dec '15

16 Dec '15

Hi, I'm using Dovecot 2.2.13 with Dovecot Director, Doveadm server and FTS. I want to run doveadm on Director hosts, but Dovecot server on backend hosts reject "fts *" commands. On Dovecot Director hosts: # doveadm fts optimize -u u0001(a)example.jp # echo $? 75 # doveadm fts rescan -u u0001(a)example.jp # echo $? 75 doveadm index -u u0001(a)example.jp INBOX # echo $? 0 On Dovecot backend hosts: # grep doveadm /var/log/maillog Jul 14 10:25:40 backend dovecot: doveadm: Error: doveadm: Client sent unknown command: fts optimize Jul 14 10:25:51 backend dovecot: doveadm: Error: doveadm: Client sent unknown command: fts rescan Is this doveadm-server restriction? On Dovecot backend hosts (2): # doveadm fts optimize -u u0001(a)example.jp -S /var/run/dovecot/doveadm-server # echo $? 75 # doveadm fts rescan -u u0001(a)example.jp -S /var/run/dovecot/doveadm-server # echo $? 75 # doveadm index -u u0001(a)example.jp -S /var/run/dovecot/doveadm-server INBOX # echo $? 0 # doveadm fts optimize -u u0001(a)example.jp # echo $? 0 # doveadm fts rescan -u u0001(a)example.jp # echo $? 0 # doveadm index -u u0001(a)example.jp INBOX # echo $? 0 -- -- Name: SATOH Fumiyasu @ OSS Technology Corp. (fumiyas @ osstech co jp) -- Business Home: http://www.OSSTech.co.jp/ -- GitHub Home: https://GitHub.com/fumiyas/ -- PGP Fingerprint: BBE1 A1C9 525A 292E 6729 CDEC ADC2 9DCA 5E1C CBCA

2 3

mmap_disable=yes not honored always
by Peter Benko 06 Dec '15

06 Dec '15

<Sorry for the resend, it seems that the formatting of my original mail got garbled> Hi all, I'm experimenting with having the mail store on a 9p file system that lacks mmap() functionality. So I disabled it in dovecot: mmap_disable = yes However, I keep getting the following error messages in my log: Oct 15 16:55:00 computer-name dovecot: imap user(a)domain.com[192.168.1.3] Error: mmap() failed with file /mail/mailboxes/domain.com/user/indexpvt-accounts/other-user/mailboxes/INBOX/dovecot.index.pvt.log: Invalid argument I only get those messages for the shared folder's private indexes. This setup works perfectly on an ext2/3/4 fs. Any opinions, is this a bug? Is mmap_disable not mandatory for all operations? Best regards, Peter

2 1

[PATCH] [dovecot 2.2.9] Quota warnings ignored with FS quotas
by Grzegorz Nosek 06 Dec '15

06 Dec '15

Hi all, I noticed that in some circumstances quota warnings are ignored. The bug arises when both of the following are used: 1. percentage-based quota warnings, i.e.: quota_warning = storage=1%% quota-warning 1 %u 2. filesystem quota backend (and probably others, except for quotas configured directly in dovecot config) Percentage-based quota warnings have rule.bytes_limit recalculated based on root_set->default_rule.bytes_limit, however this value is zero when FS quotas are in use. Real quota values (from quotactl) are fetched very late, in quota_warnings_execute() but at that point no recalculation happens. As the warning rules have bytes_limit==0, they're effectively ignored. The patch below enables quota warnings to be sent when using filesystem (and possibly maildirsize-based) quotas. Based and tested on Ubuntu 14.04's dovecot 2.2.9. Best regards, Grzegorz Nosek diff --git a/src/plugins/quota/quota.c b/src/plugins/quota/quota.c index adbd70d..8e4d7e0 100644 --- a/src/plugins/quota/quota.c +++ b/src/plugins/quota/quota.c @@ -1163,6 +1163,8 @@ static void quota_warnings_execute(struct quota_transaction_context *ctx, &count_current, &count_limit) < 0) return; + quota_root_recalculate_relative_rules(root->set, bytes_limit, count_limit); + bytes_before = bytes_current - ctx->bytes_used; count_before = count_current - ctx->count_used; for (i = 0; i < count; i++) {

3 3

SIS attachments compressed
by Juan Carlos Sanchez 06 Dec '15

06 Dec '15

Hello: In case it is defined to use zlib compression, does SIS save attachments compressed? I have found this question replied in the past (2012) and the answer was NO. I ask again just to know if there have been changes or if it is in the roadmap to add compression. I have been testing to migrate from maildir+zlib to mdbox+zlib+sis and the results, in terms of space used, and worst than using just mdbox+zlib (without SIS), and suppose the reason in the lack of compression in attachments. Thanks in advance. -- ------------------------------------------------------ Juan Carlos Sanchez Hernandez Responsable de Seguridad y Correo Electronico Servicio de Planificacion Informatica y Comunicaciones Universidad Politecnica de Madrid Rectorado Avda. Ramiro de Maeztu 7 28040 Madrid ------------------------------------------------------

2 1

[Dovecot] lmtp_rcpt_check_quota not working
by Bernhard Schmidt 06 Dec '15

06 Dec '15

Hi, Dovecot 2.2.9 (Debian package from backports) does not seem to respect lmtp_rcpt_check_quota # 2.2.9: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.4 deliver_log_format = from=<%e>, size=%p, message-id=<%m>, status=%$ disable_plaintext_auth = no lmtp_rcpt_check_quota = yes mail_debug = yes mail_gid = mstore mail_location = maildir:~/Maildir:INDEX=/var/cache/dovecot/index/%-1.1u/%u mail_plugins = quota mail_uid = mstore managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave mmap_disable = yes namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = INBOX. separator = . type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { quota = maildir quota_rule = ?:storage=512M quota_rule2 = INBOX.Trash:ignore quota_status_nouser = DUNNO quota_status_overquota = 452 4.2.2 Mailbox is full quota_status_success = DUNNO quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=85%% quota-warning 85 %u sieve = ~/currently-active-script.sieve sieve_dir = ~/sieve } pop3_uidl_format = %08Xv%08Xu protocols = imap lmtp sieve pop3 quota_full_tempfail = yes service anvil { client_limit = 3000 unix_listener anvil { group = sudo mode = 0660 } } service auth { client_limit = 3000 unix_listener auth-userdb { group = mstore mode = 0660 user = mstore } } service imap-login { client_limit = 1024 inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } process_limit = 2500 process_min_avail = 4 service_count = 0 } service lmtp { inet_listener lmtp { address = xxx port = 24 } } service managesieve-login { inet_listener sieve { port = 2000 } service_count = 1 } service managesieve { process_limit = 1024 } service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 ssl = yes } } service quota-status { client_limit = 5 executable = quota-status -p postfix inet_listener { address = xxx port = 12340 } } service quota-warning { executable = script /etc/dovecot/quotawarnmsg.sh unix_listener quota-warning { user = mstore } user = mstore } ssl_cert = </etc/dovecot/dovecot.pem ssl_key = </etc/dovecot/private/dovecot.pem userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } verbose_proctitle = yes protocol imap { mail_plugins = quota imap_quota imap_zlib } protocol lda { mail_plugins = quota sieve } protocol lmtp { mail_plugins = quota sieve } so Dovecot should reject in RCPT TO phase, but according to the logs of the upstream Postfix it does so after END-OF-DATA (host xxx[2001:xxx] said: 452 4.2.2 <user(a)domain.de> Quota exceeded (mailbox for user is full) (in reply to end of DATA command)) I haven't seen any commits regarding to this in 2.2.10 or tip. Regards, Bernhard

3 6