August 2019 - dovecot

intermittant passdb error
by NeuralNET 23 Jan '20

23 Jan '20

I'm having great difficulty debugging this problem.. I'm having intermittant problems connecting to dovecot. This started a couple of days ago following an apt-get update on the server. I'm running Ubuntu 14.04, dovecot 2.2.9. Here's my dovecot -n --- # 2.2.9: /etc/dovecot/dovecot.conf # OS: Linux 3.19.0-59-generic x86_64 Ubuntu 14.04.4 LTS auth_mechanisms = plain login mail_location = maildir:~/Maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } plugin { sieve_dir = ~/sieve } protocols = imap service auth { unix_listener /var/spool/postfix/private/auth { mode = 0666 } user = $default_internal_user } ssl = required ssl_cert = </etc/dovecot/box.neuralnet.org.uk_crt.pem ssl_key = </etc/dovecot/box.neuralnet.org.uk_key.pem ssl_protocols = !SSLv2 !SSLv3 userdb { driver = passwd } --- Pretty standard stuff. In addition disable_plaintext_auth = yes is set in 10-auth.conf This showed up in the mail.err after the reboot; --- May 17 09:04:51 server dovecot: imap-login: Error: read(anvil) failed: EOF May 17 09:04:51 server dovecot: auth: Error: read(anvil-auth-penalty) failed: EOF May 17 09:04:51 server dovecot: auth: Error: net_connect_unix(anvil-auth-penalty) failed: Permission denied --- At that point The following commands were run on the server a few minutes apart with no changes made in the interim; --- username@box /etc/dovecot $ doveadm user username field valueuserdb lookup: user username doesn't exist username@box /etc/dovecot $ doveadm user username field value uid 1000 gid 1000 home /home/username mail maildir:~/Maildir system_groups_user username --- so, passdb wasn't accessible, but now it is ? Imap logins are showing in mail.log as failing, but thunderbird is currently connected; --- May 18 12:00:58 box dovecot: imap-login: Disconnected (auth failed, 3 attempts in 19 secs): user=<username>, method=PLAIN, rip=10.0.0.3, lip=10.0.0.99, TLS, session=<********> May 18 12:08:27 box dovecot: imap-login: Disconnected (auth failed, 3 attempts in 20 secs): user=<username>, method=PLAIN, rip=10.0.0.3, lip=10.0.0.99, TLS, session=<********> May 18 12:08:50 box dovecot: imap-login: Disconnected (auth failed, 3 attempts in 12 secs): user=<username>, method=PLAIN, rip=10.0.0.3, lip=10.0.0.99, TLS, session=<********> --- do I conclude that method=plain is failing, whereas method=login succeeds ? I tried setting auth_debug_password=yes and get the following; --- May 18 12:26:42 box dovecot: auth: Debug: client in: CONT#0111#011A********== (previous base64 data may contain sensitive data) May 18 12:26:42 box dovecot: auth-worker(12214): Debug: Loading modules from directory: /usr/lib/dovecot/modules/auth May 18 12:26:42 box dovecot: auth-worker(12214): Debug: pam(expert,10.0.0.3): lookup service=dovecot May 18 12:26:42 box dovecot: auth-worker(12214): Debug: pam(expert,10.0.0.3): #1/1 style=1 msg=Password: May 18 12:26:42 box dovecot: auth-worker(12211): pam(username,10.0.0.3): pam_authenticate() failed: Authentication failure (password mismatch?) (given password: ******) May 18 12:26:44 box dovecot: auth: Debug: client passdb out: FAIL#0112#011user=username --- Any assistance debugging this further is greatly recieved! Thanks.

2 2

CentOS x64 compilation fails.
by Mart Pirita 16 Dec '19

16 Dec '19

Hello. I have problems to compile Dovecot2.2.13 with rpmbuild using options: ./configure \ --prefix=/usr \ --with-ssl=openssl \ --with-ssldir=/etc/ssl \ --sysconfdir=/etc \ --without-vpopmail \ --with-pam \ --without-bsdauth \ --without-sql \ --without-nss \ --without-ldap \ --without-pgsql \ --without-mysql \ --without-sqlite \ --with-rundir=/var/run/dovecot \ --without-deliver \ --without-gssapi make Using CentOS 5.11, option - rpmbuild -ba dovecot2.2.spec fails with error: /bin/mkdir -p '/var/tmp/dovecot-2.2.13-mp-root/usr/lib64/dovecot' /bin/sh ../../libtool --mode=install /usr/bin/install -c libdovecot-storage.la '/var/tmp/dovecot-2.2.13-mp-root/usr/lib64/dovecot' libtool: install: error: cannot install `libdovecot-storage.la' to a directory not ending in /usr/lib/dovecot make[4]: *** [install-pkglibLTLIBRARIES] Error 1 make[4]: Leaving directory `/usr/src/redhat/BUILD/dovecot-2.2.13/src/lib-storage' make[3]: *** [install-am] Error 2 make[3]: Leaving directory `/usr/src/redhat/BUILD/dovecot-2.2.13/src/lib-storage' make[2]: *** [install-recursive] Error 1 make[2]: Leaving directory `/usr/src/redhat/BUILD/dovecot-2.2.13/src/lib-storage' make[1]: *** [install-recursive] Error 1 make[1]: Leaving directory `/usr/src/redhat/BUILD/dovecot-2.2.13/src' make: *** [install-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.14076 (%install) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.14076 (%install) But using CentOS 5.11with option- rpmbuild --target=i686 -ba dovecot2.2.spec works fine. However, using CentOS 6.5, with option - rpmbuild -ba dovecot2.2.spec fails with error: /bin/mkdir -p '/root/rpmbuild/BUILDROOT/dovecot-2.2.13-mp.x86_64/usr/lib64/dovecot' /bin/sh ../../libtool --mode=install /usr/bin/install -c libdovecot-storage.la '/root/rpmbuild/BUILDROOT/dovecot-2.2.13-mp.x86_64/usr/lib64/dovecot' libtool: install: error: cannot install `libdovecot-storage.la' to a directory not ending in /usr/lib/dovecot make[4]: *** [install-pkglibLTLIBRARIES] Error 1 make[4]: Leaving directory `/root/rpmbuild/BUILD/dovecot-2.2.13/src/lib-storage' make[3]: *** [install-am] Error 2 make[3]: Leaving directory `/root/rpmbuild/BUILD/dovecot-2.2.13/src/lib-storage' make[2]: *** [install-recursive] Error 1 make[2]: Leaving directory `/root/rpmbuild/BUILD/dovecot-2.2.13/src/lib-storage' make[1]: *** [install-recursive] Error 1 make[1]: Leaving directory `/root/rpmbuild/BUILD/dovecot-2.2.13/src' make: *** [install-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.HP3meN (%install) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.HP3meN (%install) But when CentOS 5.11with option rpmbuild --target=i686 -ba dovecot2.2.spec suceeds, CentOS 6.5still fails: cpio: dovecot-2.2.13/src/lib-compression/abspath.c: Cannot stat: No such file or directory cpio: dovecot-2.2.13/src/lib-compression/compat.h: Cannot stat: No such file or directory cpio: dovecot-2.2.13/src/lib-compression/lib.h: Cannot stat: No such file or directory 18096 blocks + /usr/lib/rpm/check-buildroot /root/rpmbuild/BUILDROOT/dovecot-2.2.13-mp.i386/usr/lib/dovecot/dovecot-config:LIBDOVECOT='-L/root/rpmbuild/BUILDROOT/dovecot-2.2.13-mp.i386/usr/lib/dovecot -ldovecot' /root/rpmbuild/BUILDROOT/dovecot-2.2.13-mp.i386/usr/lib/dovecot/dovecot-config:LIBDOVECOT_INCLUDE=-I/root/rpmbuild/BUILDROOT/dovecot-2.2.13-mp.i386/usr/include/dovecot Found '/root/rpmbuild/BUILDROOT/dovecot-2.2.13-mp.i386' in installed files; aborting error: Bad exit status from /var/tmp/rpm-tmp.69mGY0 (%install) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.69mGY0 (%install) How to fix these errors? -- Mart

5 8

BUG: panic when using fs:posix as dict for acl_shared_dict
by Marco Giunta 13 Dec '19

13 Dec '19

Hi, I'm using Dovecot 2.2.33.2 on a RHEL 7, new installation. When I use fs:posix as dict for acl_shared_dict, like in Dovecot wiki (https://wiki.dovecot.org/SharedMailboxes/ClusterSetup) doveadm-server crash with error: # doveadm acl set -u USERNAME FOLDER user=DEST_USERNAME lookup read write-seen doveadm(USERNAME): Panic: file dict-fs.c: line 127 (fs_dict_iterate_init): assertion failed: ((flags & DICT_ITERATE_FLAG_RECURSE) == 0) doveadm(USERNAME): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x9f3de) [0x7f0e4a4b23de] -> /usr/lib64/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7f0e4a4b244a] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f0e4a44377c] -> /usr/lib64/dovecot/libdovecot.so.0(+0x31688) [0x7f0e4a444688] -> /usr/lib64/dovecot/libdovecot.so.0(dict_iterate_init_multiple+0x4d) [0x7f0e4a47cadd] -> /usr/lib64/dovecot/libdovecot.so.0(dict_iterate_init+0x29) [0x7f0e4a47cb89] -> /usr/lib64/dovecot/lib01_acl_plugin.so(acl_lookup_dict_rebuild+0x3e1) [0x7f0e49a40371] -> /usr/lib64/dovecot/lib01_acl_plugin.so(acl_backend_vfile_acllist_rebuild+0x488) [0x7f0e49a3dd18] -> /usr/lib64/dovecot/lib01_acl_plugin.so(acl_backend_vfile_object_update+0x3c7) [0x7f0e49a3e867] -> /usr/lib64/dovecot/lib01_acl_plugin.so(acl_mailbox_update_acl+0x68) [0x7f0e49a41e28] -> /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so(+0x2c11) [0x7f0e48da1c11] -> /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so(+0x3060) [0x7f0e48da2060] -> doveadm(+0x2b41c) [0x556f1280b41c] -> doveadm(+0x2c01a) [0x556f1280c01a] -> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x23b) [0x556f1280ce7b] -> doveadm(doveadm_cmd_run_ver2+0x50c) [0x556f1281c73c] -> doveadm(doveadm_cmd_try_run_ver2+0x37) [0x556f1281c7d7] -> doveadm(main+0x1e4) [0x556f127fb944] -> /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f0e4a071c05] -> doveadm(+0x1bd35) [0x556f127fbd35] Aborted Attached coredump and configuration. I'm not a programmer, but seems that assert is raised by 'fs_dict_iterate_init' function inside 'src/lib-dict-extra/dict-fs.c': static struct dict_iterate_context * fs_dict_iterate_init(struct dict *_dict, const char *const *paths, enum dict_iterate_flags flags) { ... /* these flags are not supported for now */ i_assert((flags & DICT_ITERATE_FLAG_RECURSE) == 0); ... because it is called by 'acl_lookup_dict_iterate_read' function in file 'src/plugins/acl/acl-lookup-dict.c' static void acl_lookup_dict_iterate_read(struct acl_lookup_dict_iter *iter) { ... dict_iter = dict_iterate_init(iter->dict->dict, prefix, DICT_ITERATE_FLAG_RECURSE); ... with DICT_ITERATE_FLAG_RECURSE set. Same problem also with Dovecot 2.3.0. Thanks, Marco -- Marco Giunta - ITCS SysAdmin Via Bonomea, 265 34136 - Trieste, Italy Tel: +39-040-3787-503 Fax: +39-040-3787-244

4 6

Supporting RFC 5466 (IMAP4 Extension for Named Searches (Filters))
by Jesus Cea 06 Dec '19

06 Dec '19

I wonder if Dovecot supports RFC 5466 (IMAP4 Extension for Named Searches (Filters)) or if there is any plan about it. -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

4 6

dovecot sieve duplicates detection
by André Rodier 04 Dec '19

04 Dec '19

Hello, I have tested the sieve duplicate script with success so far, but I have a question. I would like to know if the "duplicate" sieve flag in Dovecot is global to all folders, or specific to one folder only. For instance, if I copy an email from one folder to another, and I have a discard action on duplicate email, is this action will be applied (in this case, discard) or not. If the duplicate is global to all folders, is there a way to restrict the search in one folder only. Thanks for your help. André

3 10

User found but password failure
by Mauricio Tavares 11 Nov '19

11 Nov '19

Trying to figure out which step is causing me not to be able to login. I am using a password file, passdb { driver = passwd-file args = scheme=SHA512-CRYPT username_format=%u /etc/dovecot/passwd } We will assume that the pw I created using 'doveadm pw -s SHA512-CRYPT' matches the password I will be using to login below. I crank the debugging mode, auth_debug_passwords = yes auth_debug = yes And then try to login [root@mail ~]# nc -t localhost 143 * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN AUTH=GSSAPI] Howdy a login raub(a)example.com password [blank] which from what I gathered from /var/log/dovecot means it found matching username but did not match password: secured session=lyJttvyQ9I0AAAAAAAAAAAAAAAAAAAAB lip=::1 rip=::1 lport=143 rport=36340 resp=base64-reply (previous base64 data may contain sensitive data) Aug 26 03:25:52 auth: Debug: passwd-file(raub@example.com,::1,<lyJttvyQ9I0AAAAAAAAAAAAAAAAAAAAB>): lookup: user=raub(a)example.com file=/etc/dovecot/passwd Aug 26 03:25:52 auth: Debug: client passdb out: OK 1 user=raub(a)example.com Aug 26 03:25:52 auth: Debug: master in: REQUEST 1520959489 26560 1 63ee7f45236f85fd39573a5c8a2eb46a session_pid=26563 request_auth_token Aug 26 03:25:52 auth-worker(26562): Debug: passwd(raub@example.com,::1,<lyJttvyQ9I0AAAAAAAAAAAAAAAAAAAAB>): lookup Aug 26 03:25:52 auth-worker(26562): Info: passwd(raub@example.com,::1,<lyJttvyQ9I0AAAAAAAAAAAAAAAAAAAAB>): unknown user Aug 26 03:25:52 auth: Debug: passwd-file(raub@example.com,::1,<lyJttvyQ9I0AAAAAAAAAAAAAAAAAAAAB>): lookup: user=raub(a)example.com file=/etc/dovecot/passwd Aug 26 03:25:52 auth: Error: plain(raub@example.com,::1,<lyJttvyQ9I0AAAAAAAAAAAAAAAAAAAAB>): user not found from any userdbs Aug 26 03:25:52 auth: Debug: master userdb out: NOTFOUND 1520959489 Aug 26 03:25:52 imap: Error: Authenticated user not found from userdb, auth lookup id=1520959489 (auth connected 1 msecs ago, handshake 0 msecs ago, request took 1 msecs, client-pid=26560 client-id=1) Aug 26 03:25:52 imap-login: Info: Internal login failure (pid=26560 id=1) (internal failure, 1 successful auths): user=<raub(a)example.com>, method=PLAIN, rip=::1, lip=::1, mpid=26563, secured, session=<lyJttvyQ9I0AAAAAAAAAAAAAAAAAAAAB> Per [1] I decided to see what the response (base64-reply) I am submitting to dovecot looks like: echo 'base64-reply' | base64 -d raub(a)example.compassword Which has the right username and password but follows the userpassword format, not useruserpassword as I was led to believe by [1]. Is that to be expected? If it is then the issue is further down the line. [1] https://wiki.dovecot.org/Debugging/Authentication

3 5

Migration
by Hans Brage 01 Oct '19

01 Oct '19

Hi! I'm currently running an small imap-server on Dovecot 2.2.4 but will retire that server. I've set up an new server with Dovecot 2.3.4 and will migrate the mailboxes (maildir-format) from the old to the new server. I'm planning to restructure the accounts a bit when migrating so I need to move them one by one. Its only a few so thats not a big issue. Both servers are using virtual accounts (users file). I've understood that best method for migration is to run doveadm backup -R from the new server. But I really cant figure out what other parameters or settings that I need to perform that task and if it should be run as user vmail or root. Would be greatful for any tips, hints, links or similiar. Regards, Hans

6 9

Best way to move mail from one server to another
by James Brown 01 Oct '19

01 Oct '19

I’d like to migrate to a new server. I only need to transfer the mail store, have already done the users. I’ve seen different recommendations: IMAP-Sync Rsync Doveadmin backup and dsync What are the pros and cons of each? What would be best for me? I’m thinking that Doveadmin backup is the way to go. Both old and new servers are on same network. New server is running Dovecot 2.3.2.1, old server is 2.2.32. macOS X. Does anyone have any advice or examples? Thanks, James.

13 29

doveadm / vsize
by Paul Macdonald 27 Sep '19

27 Sep '19

Hi, i want to keep track of user mailbox size, I'm using doveadm mailbox status -u <user> vsize INBOX for a given mailbox this is under reporting ( by a lot) is this recursive? what shoud i be using to get an accurate disk usage metric? thanks Paul. -- ------------------------- Paul Macdonald IFDNRG Ltd Web and video hosting ------------------------- t: 0131 5548070 m: 07970339546 e: paul(a)ifdnrg.com w: http://www.ifdnrg.com ------------------------- IFDNRG 40 Maritime Street Edinburgh EH6 6SA ---------------------------------------------------- Virtual Servers from £50.00pm High specification Dedicated Servers from £150.00pm ----------------------------------------------------

3 4

WARNING: using attachment_dir with plugin zlib can corrupt mails
by Patrick Cernko 12 Sep '19

12 Sep '19

Hello list, hello Dovecot developers, this week, I discovered a serious bug in Dovecot, that lead to several broken mails on our servers. The bug corrupts the first few characters of the mail header during saving. On our setup, it was almost always only the very first line of text, that was corrupted. Depending on the IMAP client (they seem to request different header fields, ... during mail access), the bug causes the imap process to hang up the TCP connection and log errors like this: > imap(USERNAME)<4767><TeQP4ASOTK5/AAAB>: Error: Corrupted record in index cache file /IMAP/mail/mailboxes/USERNAME/mdbox/mailboxes/Trash/dbox-Mails/dovecot.index.cache: UID 489113: Broken fields in mailbox Trash: read(attachments-connector(zlib(/IMAP/mail/mailboxes/USERNAME/mdbox/storage/m.813))): FETCH BODY[HEADER.FIELDS (RETURN-PATH SUBJECT)] got too little data: 2 vs 122 In our case that finally grabbed my attention, the client was the users iphone that did not display any new messages but his Thunderbird did. The bug seems to be triggered by a bad "interaction" of attachment_dir option and zlib plugin. If you use both, you most likely are affected, too, except you only use zlib plugin for reading previously compressed stored mails. That's also the workaround we use now: zlib plugin only enabled in mail_plugins but no plugin/zlib_save set. The bug occurs on very specific mails. Due to privacy reasons I could not provide sample mails here. Storing such mails seems to trigger the bug reproducible. I attached a very minimal doveconf -n config, that can be used to trigger the bug. If one of the developers is interested, I can try to generate an "anonymized" version of such a specific mail that still causes the issue. I discovered the bug on our productive systems, running latest Dovecot 2.2 release, but the latest 2.3 I used during debugging is affected, too. During debugging, I also found one hint, that might help find the bug: If you store a problematic mail with zlib_save=gz (or zlib_save=bz2) and then disable the zlib plugin in mail_plugins, you can call doveadm fetch -u test hdr all | grep -v ^hdr: | gzip --decompress on test's mailbox with only that one broken mail. This will display the beginning of the rfc822 mail text until gzip terminates with "gzip: stdin: unexpected end of file", approximately after twice the length of the mail HEADER. This might indicate, that dovecot stores the uncompressed size of the header in it's data structures although the mail is stored compressed. I also found a very efficient way to find all affected mails in our setup: doveadm -f flow fetch -A 'user guid mailbox uid seq flags hdr' all | \ grep -a "^[^ ]+ user=" | \ grep -avF ' hdr=Return-path: ' | \ grep -av '.* hdr=[[:print:][:space:]]*$' (runtime for ~6M mails on our servers was 20-30min) This can be even more optimized if you have a powerful storage system with GNU parallel: > doveadm user '*' | parallel "doveadm -f flow fetch -u '{}' 'user guid mailbox uid seq flags hdr' all | grep -a '^user=' | grep -avF ' hdr=Return-path: ' | grep -av '.* hdr=[[:print:][:space:]]*$' || true" (runtime for ~6M mails on our servers was ~4min) The command will give you a list of mails that possibly are affected, check the full output of doveadm fetch -u USERNAME hdr guid GUID | less to verify that the header is really broken. On our systems I found 39 mails within ~12M mails. I was able to recover these mails "manually" by reconstructing the Return-Path header line, importing the fixed mails and expunging the corrupt ones. Before importing, I had to disable zlib_save option obviously. Best regards, -- Patrick Cernko <pcernko(a)mpi-klsb.mpg.de> +49 681 9325 5815 Joint Administration: Information Services and Technology Max-Planck-Institute fuer Informatik & Softwaresysteme

4 4