- dovecot - dovecot.org

Filesystem level backups?
by Steve Atkins 09 Apr '18

09 Apr '18

Up until now I've been backing up my IMAP server by taking an atomic snapshot of the entire VM, so I've not really had to worry about it. I'm moving to hosting where I have neither VM snapshots nor a filesystem that can take snapshots, so I need to think about consistent backups a bit more. Will a simple backup of all the files of an multi-dbox give me a consistent state when I recover, or do I need to do something (e.g. lock writes to the mailbox) while I run a backup? (I could use doveadm backup if I were backing up to local storage I controlled, but I'm pushing it to offsite buckets so I'd prefer to avoid backing up the entire thing locally, then copying that offsite). Cheers, Steve

4 5

Raw Backtrace Error when backing up specific user
by Marius Fassbender 09 Apr '18

09 Apr '18

Hi, probably since the update to 2.3.1 I can't backup a specific users mailbox. For backup I use the command 'dsync -o plugin/quota= -f -u user@domain backup dbox:/backup/dir' When I execute this command with the -D option, I get to see (above only Debug messages) dsync(user@domain): Panic: file mailbox-attribute.c: line 362 (mailbox_attribute_get_stream): assertion failed: (value_r->value != NULL || value_r->value_stream != NULL) dsync(user@domain): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0xcc724) [0x7f0f15364724] -> /usr/lib64/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7f0f1536476a] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f0f152d524b] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x566ac) [0x7f0f156686ac] -> doveadm(dsync_mailbox_import_attribute+0x4d) [0x5581a663bf4d] -> doveadm(dsync_brain_sync_mails+0x2ef) [0x5581a66377ff] -> doveadm(dsync_brain_run+0x2b0) [0x5581a6632e90] -> doveadm(+0x2960c) [0x5581a661a60c] -> doveadm(+0x2b4f7) [0x5581a661c4f7] -> doveadm(+0x2c0dc) [0x5581a661d0dc] -> doveadm(doveadm_mail_try_run+0x205) [0x5581a661da75] -> doveadm(main+0x475) [0x5581a660cce5] -> /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f0f14ca3c05] -> doveadm(+0x1be65) [0x5581a660ce65] cancelled my dovecot -n output: # 2.3.1 (0580a21bf): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.1 (d9bc6dfe) # OS: Linux 3.10.0-693.21.1.el7.x86_64 x86_64 CentOS Linux release 7.4.1708 (Core) xfs doveconf: Warning: please set ssl_dh=</etc/dovecot/dh.pem doveconf: Warning: You can generate it with: dd if=/var/lib/dovecot/ssl-parameters.dat bs=1 skip=88 | openssl dhparam -inform der > /etc/dovecot/dh.pem auth_cache_negative_ttl = 0 auth_cache_size = 100 M auth_master_user_separator = * auth_mechanisms = login plain auth_verbose = yes auth_verbose_passwords = sha1:16 deliver_log_format = msgid=%m from=%f subject=%s psize=%p status=%$ first_valid_gid = 5000 first_valid_uid = 5000 hostname = imap_capability = +XLIST imap_client_workarounds = tb-extra-mailbox-sep tb-lsub-flags last_valid_gid = 5000 last_valid_uid = 5000 lmtp_rcpt_check_quota = yes lmtp_save_to_detail_mailbox = yes mail_attachment_dir = /storage/mail/attachments mail_attachment_hash = %{sha512} mail_gid = 5000 mail_home = /storage/mail/settings/%Ld/%Ln mail_location = sdbox:/storage/mail/mails/%Ld/%Ln mail_plugins = zlib quota acl listescape fts fts_lucene mail_server_admin = mailto:postmaster@domain mail_uid = 5000 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve namespace { list = children location = sdbox:/storage/mail/mails/%%d/%%n prefix = shared/%%u/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes location = mailbox Archive { special_use = \Archive } mailbox Archives { special_use = \Archive } mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox "Gesendete Objekte" { special_use = \Sent } mailbox Junk { special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / subscriptions = yes type = private } passdb { args = /etc/dovecot/passwd.masterusers driver = passwd-file master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile:/storage/mail/settings/global-acls:cache_secs=300 acl_shared_dict = file:/storage/mail/settings/shared-mailboxes fts = lucene fts_autoindex = yes fts_lucene = whitespace_chars=@. imapsieve_mailbox1_before = file:/usr/lib64/dovecot/sieve/report-spam.sieve imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_name = Spam imapsieve_mailbox2_before = file:/usr/lib64/dovecot/sieve/report-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = Spam imapsieve_mailbox2_name = * mail_log_fields = uid box msgid size subject quota = dict:User quota::file:%h/dovecot-quota quota_grace = 5%% quota_rule = Trash:storage=+5%% quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = file:%h/sieve;active=%h/.dovecot.sieve sieve_execute_bin_dir = /usr/lib64/dovecot/sieve sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.execute sieve_pipe_bin_dir = /usr/lib64/dovecot/sieve sieve_plugins = sieve_imapsieve sieve_extprograms zlib_save = gz zlib_save_level = 6 } postmaster_address = postmaster@domain protocols = imap lmtp sieve service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0600 user = postfix } unix_listener auth-userdb { group = dovemail mode = 0600 user = dovemail } } service lmtp { inet_listener lmtp { address = 127.0.0.1 port = 24 } unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service managesieve-login { inet_listener sieve { port = 4190 } } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = dovemail } user = dovecot } ssl = required ssl_cert = </etc/pki/tls/certs/cert.cer ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { args = /etc/dovecot/dovecot-ldap.conf.ext default_fields = master_user=%u driver = ldap } protocol lmtp { mail_plugins = zlib quota acl listescape fts fts_lucene sieve postmaster_address = postmaster@domain } protocol imap { mail_plugins = zlib quota acl listescape fts fts_lucene imap_acl imap_quota imap_sieve imap_zlib } I tried to rebuild the settings files of this mailbox by deleting them and restart the service but this didnt help. A reboot did neither. I dont know where to start fixing this problem, does anyone has an advice? Thanks in advance

1 0

Re: 2.3.1 Replication is throwing scary errors
by Reuben Farrelly 08 Apr '18

08 Apr '18

Hi, [Formatting is a bit rough, replying from a trimmed digest email] > Message: 1 > Date: Fri, 6 Apr 2018 15:04:35 +0200 > From: Michael Grimm <trashcan(a)ellael.org> > To: Dovecot Mailing List <dovecot(a)dovecot.org> > Subject: Re: 2.3.1 Replication is throwing scary errors > Message-ID: <E7E7A927-68F8-443F-BA59-E66CED8FE878(a)ellael.org> > Content-Type: text/plain; charset=utf-8 > > Reuben Farrelly wrote: >> From: Michael Grimm <trashcan(a)ellael.org> > >>> [This is Dovecot 2.3.1 at FreeBSD STABLE-11.1 running in two jails at distinct servers.] >>> I did upgrade from 2.2.35 to 2.3.1 today, and I do become pounded by error messages at server1 (and vice versa at server2) as follows: >>> | Apr 2 17:12:18 <mail.err> server1.lan dovecot: doveadm: Error: dsync(server2.lan): I/O has stalled, \ >>> no activity for 600 seconds (last sent=mail_change, last recv=mail_change (EOL)) >>> | Apr 2 17:12:18 <mail.err> server1.lan dovecot: doveadm: Error: Timeout during state=sync_mails \ >>> (send=changes recv=mail_requests) >>> [?] >>> | Apr 2 18:59:03 <mail.err> server1.lan dovecot: doveadm: Error: dsync(server2.lan): I/O has stalled, \ >>> no activity for 600 seconds (last sent=mail, last recv=mail (EOL)) >>> | Apr 2 18:59:03 <mail.err> server1.lan dovecot: doveadm: Error: Timeout during state=sync_mails \ >>> (send=mails recv=recv_last_common) >>> I cannot see in my personal account any missing replications, *but* I haven't tested this thoroughly enough. I do have customers being serviced at these productive servers, *thus* I'm back to 2.2.35 until I do understand or have learned what is going on. > > In my reply to this statement of mine I mentioned that I have seen those timeouts quite some times during the past year. Thus, I upgraded to 2.3.1 again, and boom: after some hours I ended up in hanging processes [1] like (see Remko's mail in addition) ... > > doveadm-server: [IP4/6 <USER1> SOME/MAILBOX import:0/0] (doveadm-server) > > ? at server2 paired with a file like ? > > -rw------- 1 vmail dovecot uarch 0 Apr 3 16:52 /home/to/USER1/.dovecot-sync.lock > > Corresponding logfile entries at server2 are like ? > > Apr 3 17:10:49 <mail.err> server2.lan dovecot: doveadm: Error: Couldn't lock /home/to/USER1/.dovecot-sync.lock: \ > fcntl(/home/to/USER1/.dovecot-sync.lock, write-lock, F_SETLKW) locking failed: Timed out after 30 seconds \ > (WRITE lock held by pid 51110) > > [1] Even stopping dovecot will not end those processes. One has to manually kill those before restarting dovecot. > > After one day of testing 2.3.1 with a couple of those episodes of locking/timeout, and now missing mails depending with server your MUA will connect to, I went back to 2.2.35. After two days at that version I never had such an episode again. > >> It's not just you. This issue hit me recently, and it was impacting >> replication noticeably. I am following git master-2.3 . > [...] >> There is also a second issue of a long standing race with replication >> occurring somewhere whereby if a mail comes in, is written to disk, is >> replicated and then deleted in short succession, it will reappear >> again to the MUA. I suspect the mail is being replicated back from >> the remote. A few people have reported it over the years but it's not >> reliable or consistent, so it has never been fixed. >> And lastly there has been an ongoing but seemingly minor issue >> relating to locking timing out after 30s particularly on the remote >> host that is being replicated to. I rarely see the problem on my >> local disk where almost all of the mail comes in, it's almost always >> occurring on the replicate/remote system. > > It might be time to describe our setups in order to possibly find common grounds that might trigger this issue you describe and Rimko and myself ran into as well. > > Servers: Cloud Instances (both identical), around 25ms latency apart. > Intel Core Processor (Haswell, no TSX) (3092.91-MHz K8-class CPU) > Both servers are connected via IPsec/racoon tunnels > OS: FreeBSD 11.1-STABLE (both servers) > Filesystem: ZFS > MTA: postfix 3.4-20180401 (postfix delivers via dovecot's LMTP) > IMAP: dovecot running in FreeBSD jails (issues with 2.3.1, fine with 2.2.35) > Replication: unsecured tcp / master-master > MUA: mainly iOS or macOS mail.app, rarely roundcube For me: Servers: Main: VM on a VMWare ESXi local system (light load), local SSD disks (no NFS) Redundant: Linode VM in Singapore, around 250ms away Also no NFS. Linode use SSDs for IO. There is native IPv6 connectivity between both VMs. As I am using TCPs I don't have a VPN between them - just raw IPv6 end to end. OS: Gentoo Linux x86_64 kept well up to date Filesystem: EXT4 for both MTA: Postfix 3.4-x (same as you) IMAP: Dovecot running natively on the machine (no jail/chroot) Replication: Secured TCP / master-master (but only one master is actively used for SMTP and MUA access) MUA: Thunderbird, Maildroid, MS Mail, Gmail client, Roundcube, LG Mail client on handset etc > I believe it is worthwhile to mention here that I run a poor man's fail-over approach (round-robin DNS) as follows: > > DNS: mail.server.tld resolves to one IP4 and one IP6 address of each server, thus 4 IP addresses in total For me I have a simple pair of servers that replicate mail to each other. Clients connect to the main server only, which is where mail comes in. The second one that experiences the problem with locking is only a redundant server which is a secondary MX, that only gets occasional SMTP deliveries from time to time. So the amount of replication back from the "secondary" is very minimal; the vast bulk of the replication is the primary replicating changes out. Reuben

1 0

changed behavior for dovecot-lda in 2.3.x
by Martin Waschbüsch 08 Apr '18

08 Apr '18

Hi all, Hey all, I upgraded to dovecot 2.3.1 (from 2.2.34) and noticed that the behavior for dovecot-lda changed. Apparently it no longer accepts -f "" or -f "<>? With 2.2.34, both were accepted now I get: root@mail:~# /usr/local/libexec/dovecot/dovecot-lda -f "<>" lda(root): Fatal: Invalid -f parameter: Null path not allowed or root@mail:~# /usr/local/libexec/dovecot/dovecot-lda -f "" lda(root): Fatal: Invalid -f parameter: Path is empty string I guess this must be a bug? I mean, envelope sender *must* be empty for bounces. Thanks, Martin

2 2

Can’t authenticate any users after upgrade.
by Kevin Cummings 08 Apr '18

08 Apr '18

I’m in the process of upgrading an old server from Fedora 21 to something more modern. Now, Dovecot won’t let any client login to get their email. PAM audit_log_acct_message() failed: Operation not permitted imap-login: Disconnected (AUTH failed, 2 attempts in 10 secs): user=<username>, method=PLAIN, rip=192.168.1.94 lip=192.168.1.94, TLS, session=<sessionid> # 2.3.1 (8e2f634): /etc/dovecot/dovecot.conf # OS: Linux 4.4.14-200.fc22.x86_64 x86_64 Fedora release 22 (Twenty Two) # Hostname: kjchome.homeip.net mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } ssl = required ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_cipher_list = PROFILE=SYSTEM ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = passwd } -- Kevin J. Cummings cummings(a)kjchome.homeip.net cummings(a)kjc386.framingham.ma.us kjchome(a)icloud.com Registered Linux User #1232 (http://www.linuxcounter.net/)

4 8

sharing a folder with all users in a domain
by David Mehler 08 Apr '18

08 Apr '18

Hello, I'm wanting to make a shared folder that anyone in a domain can get to, but not anyone in any other domain. Here's my dovecot shared namespace: namespace { type = shared separator = / prefix = shared/%%u/ location = maildir:%%h:INDEX=~/shared/%%u subscriptions = no list = yes } The dovecot-dict-sql.conf.ext file has: map { pattern = shared/shared-boxes/user/$to/$from table = user_shares value_field = dummy fields { from_user = $from to_user = $to } } map { pattern = shared/shared-boxes/anyone/$from table = anyone_shares value_field = dummy fields { from_user = $from } } and in my global acl file I have: shared/office@ user(a)domain.com lrwstipekxa shared/office@ user2(a)domain.com lrwstipekxa Thanks. Dave.

1 0

Warning message about quota.conf
by forums＠mehl-family.fr 06 Apr '18

06 Apr '18

Hi, In my maillog I see this warning message at every reception of e-mail: _STATUS=SENT (DELIVERED VIA DOVECOT SERVICE (DOVECONF: WARNING: /ETC/DOVECOT/CONF.D/90-QUOTA.CONF LINE 39: GLOBAL SETTING MAIL_PLUGINS WON'T CHAN....._ It is just a warning, the e-mail was delivered. I don't find where is the problem. Thank's.

1 0

Dsync from multiple IMAPs -- personal/local dovecot setup
by Kim ALLAMANDOLA 06 Apr '18

06 Apr '18

Hello list, new aspiring user here, if I'm not in the wright place sorry, I do not fund any other option for now... As from subject I'm looking for a *personal* *standalone* dovecot setup, especially I'm looking for a DSync from multiple IMAP accounts to a local maildir (to being compatible with notmuch). My actual mail setup is: OfflineIMAP that sync different mailboxes in a personal taxonomy ~/mail/$mailboxName/... and notmuch-emacs + various scripts as MUA. Local postfix just to avoid emacs freeze during send, muchsync to support It does work but is slow, buggy and sometime complicated to kept up to date (I have many homegrow spaghetti-code scripts and snippets in zsh, elisp, python etc stratified in years). My dream is switch to a local Dovecot that DSync from multiple IMAPs instead of OfflineIMAP, with sieve filters instead of homegrow scripts that wrap notmuch search --output=files | xargs ... For the MUA part I'd like to stick with notmuch. So my questions are: is possible to dsync from multiple IMAPS to a maildir? if yes does dovecot accept physical messages moves if I refile (archive etc) them via notmuch? Thanks! -- Kim ALLAMANDOLA

1 0

Re: 2.3.1 Replication is throwing scary errors
by Reuben Farrelly 06 Apr '18

06 Apr '18

Hi, > ------------------------------ > > Message: 2 > Date: Mon, 2 Apr 2018 22:06:07 +0200 > From: Michael Grimm <trashcan(a)ellael.org> > To: Dovecot Mailing List <dovecot(a)dovecot.org> > Subject: 2.3.1 Replication is throwing scary errors > Message-ID: <29998016-D62F-4348-93D1-613B13DA90DB(a)ellael.org> > Content-Type: text/plain; charset=utf-8 > > Hi > > [This is Dovecot 2.3.1 at FreeBSD STABLE-11.1 running in two jails at distinct servers.] > > I did upgrade from 2.2.35 to 2.3.1 today, and I do become pounded by error messages at server1 (and vice versa at server2) as follows: > > | Apr 2 17:12:18 <mail.err> server1.lan dovecot: doveadm: Error: dsync(server2.lan): I/O has stalled, \ > no activity for 600 seconds (last sent=mail_change, last recv=mail_change (EOL)) > | Apr 2 17:12:18 <mail.err> server1.lan dovecot: doveadm: Error: Timeout during state=sync_mails \ > (send=changes recv=mail_requests) > [?] > | Apr 2 18:59:03 <mail.err> server1.lan dovecot: doveadm: Error: dsync(server2.lan): I/O has stalled, \ > no activity for 600 seconds (last sent=mail, last recv=mail (EOL)) > | Apr 2 18:59:03 <mail.err> server1.lan dovecot: doveadm: Error: Timeout during state=sync_mails \ > (send=mails recv=recv_last_common) > > I cannot see in my personal account any missing replications, *but* I haven't tested this thoroughly enough. I do have customers being serviced at these productive servers, *thus* I'm back to 2.2.35 until I do understand or have learned what is going on. > > Any ideas/feedback? > > FYI: I haven't seen such errors before. Replication has been working for years now, without any glitches at all. > > Regards, > Michael It's not just you. This issue hit me recently, and it was impacting replication noticeably. I am following git master-2.3 . Here's a last known reasonably good point in the tree where things worked quite well: EGIT_REPO_URI="https://github.com/dovecot/core.git" EGIT_BRANCH="master-2.3" EGIT_COMMIT="d9a1a7cbec19f4c6a47add47688351f8c3a0e372" So something after that (which could have gone into 2.3.1) has caused this. There is also a second issue of a long standing race with replication occurring somewhere whereby if a mail comes in, is written to disk, is replicated and then deleted in short succession, it will reappear again to the MUA. I suspect the mail is being replicated back from the remote. A few people have reported it over the years but it's not reliable or consistent, so it has never been fixed. And lastly there has been an ongoing but seemingly minor issue relating to locking timing out after 30s particularly on the remote host that is being replicated to. I rarely see the problem on my local disk where almost all of the mail comes in, it's almost always occurring on the replicate/remote system. For me it seems very unlikely that on an unloaded/idle VPS there are legitimate problems obtaining a lock in under 30s. This is with the default locking configuration. This problem started happening a lot more after the breakage in (1) above. These replication issues are similar, and could possibly be related. My system is Gentoo Linux keeping up with the latest kernels, and on an EXT4 FS. I am using TCPS based replication. My remote replicate is also on Gentoo Linux with EXT4 but on a Linode VPS (around 250ms latency away). I know in a later post you've said that you don't think it has anything to do with dovecot-2.3.1, so I'd be interested to know what really is the cause in that case. Reuben

4 3

Compatibility of Submission in 2.3.1
by Paul Hecker 06 Apr '18

06 Apr '18

Hi, Dovecot 2.3.1 (8e2f634). When using submission one client of mine can not send messages. Directly sending through exim works. Seems that dovecot is too picky or follow the RFC too precise here to be compatible. Here is the debug log (note the space between “From:" and "<user2(a)host.com>”): Apr 6 12:02:08 mail dovecot: submission(user(a)host.com)<14972><ko4tJitpt9JYSSSC>: Debug: smtp-server: conn 22.73.xx.144:53943 [0]: Received new command: MAIL From: <user(a)host.com> Apr 6 12:02:08 mail dovecot: submission(user(a)host.com)<14972><ko4tJitpt9JYSSSC>: Debug: smtp-server: conn 22.73.xx.144:53943 [0]: command MAIL; 501 reply: Submitted Apr 6 12:02:08 mail dovecot: submission(user(a)host.com)<14972><ko4tJitpt9JYSSSC>: Debug: smtp-server: conn 22.73.xx.144:53943 [0]: command MAIL: Ready to reply Apr 6 12:02:08 mail dovecot: submission(user(a)host.com)<14972><ko4tJitpt9JYSSSC>: Debug: smtp-server: conn 22.73.xx.144:53943 [0]: Trigger output Apr 6 12:02:08 mail dovecot: submission(user(a)host.com)<14972><ko4tJitpt9JYSSSC>: Debug: smtp-server: conn 22.73.xx.144:53943 [0]: Timeout stop Apr 6 12:02:08 mail dovecot: submission(user(a)host.com)<14972><ko4tJitpt9JYSSSC>: Debug: smtp-server: conn 22.73.xx.144:53943 [0]: Sending replies Apr 6 12:02:08 mail dovecot: submission(user(a)host.com)<14972><ko4tJitpt9JYSSSC>: Debug: smtp-server: conn 22.73.xx.144:53943 [0]: command MAIL; 501 reply: Sent: 501 5.5.4 Invalid FROM: Missing '<' at beginning of path Apr 6 12:02:08 mail dovecot: submission(user(a)host.com)<14972><ko4tJitpt9JYSSSC>: Debug: smtp-server: conn 22.73.xx.144:53943 [0]: command MAIL: Destroy Apr 6 12:02:08 mail dovecot: submission(user(a)host.com)<14972><ko4tJitpt9JYSSSC>: Debug: smtp-server: conn 22.73.xx.144:53943 [0]: command MAIL; 501 reply: Destroy Is there a chance that you will incorporate a fix (be more compatible but less stringent)? Thanks, Paul

3 3