September 2016 - dovecot

Panic: file mail-transaction-log-file.c: line 104 (mail_transaction_log_file_free): assertion failed: (!file->locked)
by Arkadiusz Miśkiewicz 01 Nov '16

01 Nov '16

2.2.25 (also happens on 2.2.24). Happens every time I try to make deliver and only for this user: Jul 14 09:52:02 mbox dovecot: lmtp(25601): Connect from local Jul 14 09:52:02 mbox dovecot: lmtp(powiadomienia): session=<CRU2EaJEh1cBZAAALZVUYQ>, Error: Index /var/mail/powiadomienia/dovecot.index: Lost log for seq=1009 offset=40: Missing middle file seq=1009 (between 1009..4294967295) Jul 14 09:52:02 mbox dovecot: lmtp(powiadomienia): session=<CRU2EaJEh1cBZAAALZVUYQ>, Warning: fscking index file /var/mail/powiadomienia/dovecot.index Jul 14 09:52:02 mbox dovecot: lmtp(powiadomienia): session=<CRU2EaJEh1cBZAAALZVUYQ>, Error: Fixed index file /var/mail/powiadomienia/dovecot.index: log_file_seq 1009 -> 1011 Jul 14 09:52:02 mbox dovecot: lmtp(powiadomienia): session=<CRU2EaJEh1cBZAAALZVUYQ>, Panic: file mail-transaction-log-file.c: line 104 (mail_transaction_log_file_free): assertion failed: (!file->locked) Jul 14 09:52:02 mbox dovecot: lmtp(powiadomienia): session=<CRU2EaJEh1cBZAAALZVUYQ>, Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x8d7d2) [0x7feb89fc97d2] -> /usr/lib64/dovecot/libdovecot.so.0(+0x8d8bd) [0x7feb89fc98bd] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7feb89f67e31] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_transaction_log_file_free+0x160) [0x7feb8a331fa0] -> /usr/lib64/dovecot/libdovecot- storage.so.0(mail_transaction_logs_clean+0x4d) [0x7feb8a3360ed] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_transaction_log_close+0x30) [0x7feb8a336230] -> /usr/lib64/dovecot/libdovecot- storage.so.0(mail_transaction_log_move_to_memory+0xd5) [0x7feb8a3363e5] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_move_to_memory+0xa0) [0x7feb8a330440] -> /usr/lib64/dovecot/libdovecot- storage.so.0(mail_index_write+0x183) [0x7feb8a32e9d3] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_fsck+0xc1f) [0x7feb8a3186ff] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_sync_map+0x49b) [0x7feb8a322eab] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_map+0x71) [0x7feb8a31a231] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0xe0fed) [0x7feb8a32ffed] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0xe15f3) [0x7feb8a3305f3] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_open+0x78) [0x7feb8a3306d8] -> /usr/lib64/dovecot/libdovecot-storage.so.0(index_storage_mailbox_open+0x92) [0x7feb8a309202] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x6c0e2) [0x7feb8a2bb0e2] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x6c1c8) [0x7feb8a2bb1c8] -> /usr/lib64/dovecot/plugins/lib20_zlib_plugin.so(+0x2fdc) [0x7feb85697fdc] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x450c6) [0x7feb8a2940c6] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_open+0x20) [0x7feb8a294240] -> /usr/lib64/dovecot/libdovecot-lda.so.0(mail_deliver_save_open+0xad) [0x7feb8a58d1ad] -> /usr/lib64/dovecot/libdovecot-lda.so.0(mail_deliver_save+0xbb) [0x7feb8a58d48b] -> /usr/lib64/dovecot/libdovecot-lda.so.0(mail_deliver+0x123) [0x7feb8a58d9e3] -> dovecot/lmtp [DATA powiadomienia]() [0x406bc8] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x4c) [0x7feb89fdd67c] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x101) [0x7feb89fdeb01] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x25) [0x7feb89fdd705] Jul 14 09:52:02 mbox dovecot: lmtp(powiadomienia): session=<CRU2EaJEh1cBZAAALZVUYQ>, Fatal: master: service(lmtp): child 25601 killed with signal 6 (core dumps disabled) -- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )

2 4

Maildir Expunged GUID mismatch for UID
by Steven Xu 14 Oct '16

14 Oct '16

Dovecot version:2.2.25 Since we used to keep our email files on widows server, I made the following changes in maildir-storage.h #define MAILDIR_EXTRA_SEP ',' #define MAILDIR_INFO_SEP_S ":" to "+". Everything seems working except EXPUNG, The dovecot log is flooded by messages like following: imap(xxxxx): Error: Mailbox INBOX: Expunged GUID mismatch for UID 7039 Then I read the source code, and found the following lines in maildir-sync-index.c T_BEGIN { guid = maildir_uidlist_lookup_ext(ctx->mbox->uidlist, uid, MAILDIR_UIDLIST_REC_EXT_GUID); if (guid == NULL) guid = t_strcut(filename, ':'); mail_generate_guid_128_hash(guid, guid_128); } T_END; I have to change the code to guid = t_strcut(filename, '+'); So, should MAILDIR_EXTRA_SEP be used here instead of ':'? Thanks, Steven

2 1

doveadm backup fails (compromised single attachment storage)
by Webert de Souza Lima 03 Oct '16

03 Oct '16

Hi, A couple of months ago I had a problem with Single Attachment Storage after infrastructure migration; All mailboxes were rsynced to another filesystem, and that may have broken Single Attachment Storage. Many, many (if not all) mailboxes show the below logs on dovecot: imap(foo(a)bar.com): Error: read(attachments-connector(zlib(/dovecotdir/mail/ bar.com/foo/mailboxes/INBOX/dbox-Mails/u.26426))) failed: read(/dovecotdir/attach/ bar.com/de/86/de8673894d6fb3f4460e3c26436eefa9a73517fa0f000452f553822367220… b/l]) failed: open(/dovecotdir/attach/ bar.com/de/86/de8673894d6fb3f4460e3c26436eefa9a73517fa0f000452f553822367220…) failed: No such file or directory When that happens, the MUA keeps syncing forever. Now, I need to migrate all mailboxes (again) to another dovecot instance (with no SAS), which works perfectly for new users but when I try to migrate users from my current dovecot server for this new server, I get such errors again, and I can't migrate: 2016-09-29T12:20:50.995934059Z Sep 29 12:20:50 dsync-server(foo(a)bar.com): Error: dsync(cf7d091311eb): read(attachments-connector(zlib(/dovecotdir/mdbox/bar.com/foo/storage/m.1))) failed: read(/dovecotdir/attach/ bar.com/0c/df/0cdf86b1920938fe3a043f87e2ee9e63dda276bd5b9fba687e4a0c63d181c… b/l]) failed: open(/dovecotdir/attach/ bar.com/0c/df/0cdf86b1920938fe3a043f87e2ee9e63dda276bd5b9fba687e4a0c63d181c…) failed: No such file or directory (last sent=mail, last recv=mail_request (EOL)) Is there a way to fix the attachments problem? (I know I can't recover such files, that's Ok) Is there a way to migrate (dsync backup) ignoring such problems? Thanks in advance.

1 2

Shared folder in a sharded cluster setup
by Peer Heinlein 03 Oct '16

03 Oct '16

Hi! With Dovecot Director and Proxy or the new (great!) TAG-feature from Dovecot it's easy to set up a shared IMAP-Cluster with individual local filesystems. But I'm unsure if it's possible to build a setup where shared mailboxes still can work. If user A is on Cluster (1) and user B is on (2), and Cluster (1) does not have access to the mail-home from B on (2), then user A can not reach the shared folders provided from User B on (2). I hope that there is a kind of backend-proxy-mechanism, so that the imap process of A on (1) can imap-proxy the requests for the shared folder to a node from cluster shard (2). And: To be exact, the imap process on (1) should forward the request to cluster (2) by the director system to make sure, that the connection will terminate on the right active backend of User B. This sounds like a special problem if local filesystems with mdbox are used and I now the great features of using Dovecot on Object Store, where every node can check out all mail-locations from all users. But especially on obox-systems it is very important that requests for a user are always terminated on the same backend. So how can shared folders work there?! Node (1) can not checkout the shared folders from User B if his obox storage is already active on another host (2)! Peer -- Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-42 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 1

Re: NFSv4 and Maildir
by Joseph Tam 01 Oct '16

01 Oct '16

> we have a setup with (CentOS 6) Director+Dovecot, Maildir as storage on > NetApp NFS v3. Every time I try to switch to NFS v4 I found issue with > lock (and others). So for me NFSv4 with Maildir is "unstable" or need a > fine tuning that I don't know. I found the same thing, and turning off write delegation seemed to have solved the problem. I still don't know why, though. Joseph Tam <jtam.home(a)gmail.com>

2 1

Bug: Shared Mailbox - Case Sensitivity
by Leander Schäfer 01 Oct '16

01 Oct '16

Hi, unfortunately I found a bug in Dovecot's ACL handling for shared mailboxes. It turns out Dovecot doesn't enforce lower casing the privileged username to whom the mailbox should be shared to. This results in a invalid configuration. Users get confused, since they passed on a valid email address in their ACL setup. /usr/local/www/default/mail/test(a)mydomain.localdomain/maildir/.Spam/dovecot-acl user=leander(a)mydomain.localdomain eilrwts ^^ works /usr/local/www/default/mail/leander(a)mydomain.localdomain/maildir/dovecot-acl user=test(a)mydomain.localdomain eilrwts ^^ works /usr/local/www/default/mail/test(a)mydomain.localdomain/maildir/.Drafts/dovecot-acl user=Leander(a)MyDomain.LocalDomain eilrwts ^^ Doesn't work Best regards Leander Schäfer

3 4

acl_group not working not working correctly
by Leander Schäfer 01 Oct '16

01 Oct '16

Hi, I'm trying to setup group based ACLs coming from OpenLDAP. My setup doesn't require a POSIX Group match. In the Dovecot configuration file I have this: "user_attrs = [...], mailAclGroups=acl_groups" as well as "acl = vfile:/usr/local/etc/dovecot/global-acls:cache_secs=300". The user has "public" in the LDAP attribute "mailAclGroups". It seems to get everything right. I checked with doveadm - and I see public ist listed as expected: cat /var/log/debug.log [...] Sep 16 23:39:04 WM-01 dovecot: auth: Debug: client passdb out: OK 1 user=leander(a)mydomain.localdomain acl_groups=public [...] cat /usr/local/etc/dovecot/global-acls INBOX owner lrwstipekxa Drafts owner lrwstipeka Sent owner lrwstipeka Spam owner lrwstipeka Trash owner lrwstipeka Public authenticated l Public group-override=public lrwstipekx Public/* group-override=public lrwstipekx doveadm mailbox list -u leander(a)mydomain.localdomain Drafts Sent Trash Spam Shared Public Public/Service Center Shared/test(a)mydomain.localdomain Shared/test(a)mydomain.localdomain/Drafts Shared/test(a)mydomain.localdomain/Sent Shared/test(a)mydomain.localdomain/Trash Shared/test(a)mydomain.localdomain/Spam INBOX But here comes the strange thing: telnet equal to Thunderbird: . LIST "" "*" * LIST (\HasNoChildren \Drafts) "/" Drafts * LIST (\HasNoChildren \Sent) "/" Sent * LIST (\HasNoChildren \Trash) "/" Trash * LIST (\HasNoChildren \Junk) "/" Spam * LIST (\Noselect \HasChildren) "/" Shared * LIST (\HasChildren) "/" Shared/test(a)mydomain.localdomain * LIST (\HasNoChildren) "/" Shared/test(a)mydomain.localdomain/Drafts * LIST (\HasNoChildren) "/" Shared/test(a)mydomain.localdomain/Sent * LIST (\HasNoChildren) "/" Shared/test(a)mydomain.localdomain/Trash * LIST (\HasNoChildren) "/" Shared/test(a)mydomain.localdomain/Spam * LIST (\HasNoChildren) "/" INBOX . OK List completed (0.000 + 0.000 + 0.092 secs). Public and Public/* shoul be listed as well, but it isn't. Any idea why it is behaving like this? Thanks Best regards Leander Schäfer

1 1

[patch] Improved error checking for the dovecot-antispam-plugin
by Robert Munteanu 30 Sep '16

30 Sep '16

Hi, Hopefully this is the right channel for such a patch. I have a minor enhancement to submit for the antispam plugin http://hg.dovecot.org/dovecot-antispam-plugin It adds minimal error checking for the sendmail_binary, otherwise the reported error in case of a missing binary or one with missing permissions is generic and not useful. Thanks, Robert -- http://robert.muntea.nu/

3 6

NFSv4 and Maildir
by Robert Blayzor 30 Sep '16

30 Sep '16

Recently moving to newer storage platforms for mailbox storage so looking at moving mounts from NFSv3 with lots of issues with locking and caching to NFSv4. There seems to be a lot of benefits to v4 along with some other new features, namely “delegation”. So the question boils down to, to delegate or not delegate on Maildir storage. There may be many reasons based on actual platform why to do (or not to do this), but I want to get the general opinion from others that may have more experience with this. Our setup is several FreeBSD 10.x clients running Dovecot/Exim, NetApp NFS mail storage (probably moving to TrueNAS) and using F5 load balancers for client side connections/SSL offload. From what I’ve found (and what i’ve read in the RFC) is that delegation seems to work best when there is NOT a lot of file contention from clients accessing the same files. I realize that in some situations many people are using director to try and keep users on the same client; in our case we’re doing it with F5 iRules. The F5 iRules work great for POP3 and IMAP session persistence, but unfortunately that doesn’t work for SMTP and Dovecot LDA, so we still have possible race conditions from the MTA’s delivering into “INBOX”. (mostly dovecot indexes updating at the same time). So the big question is, who is using Dovecot with maildirs with NFSv4 mounts. What has your experience been? Are you using delegation? By choice and why did you come to that decision. I’m drawing up the conclusion that if you can *mostly* control client control to specific files (ie: directing access to a mailbox to come from one client), then delegation might be ok. However, if you’re not using director and have several NFS mail clients racing to access mailboxes, then delegation might turn into chaos. Your comments welcome and appreciated. -- Robert inoc.net!rblayzor XMPP: rblayzor.AT.inoc.net PGP Key: 78BEDCE1 @ pgp.mit.edu

2 1

Re: Dovecot 2.2.25 fails on SSL
by Andreas M. Kirchwitz 30 Sep '16

30 Sep '16

Joseph Tam <jtam.home(a)gmail.com> wrote: >>> OK, the origin of your problem becomes clearer. You can hardcode these >>> paths into the executables by doing something like >>> >>> env CFLAGS='-I/my'ssl/include' \ >>> LDFLAGS='-L/your/ssl/lib -Wl,-rpath,/my/ssl/lib' \ >>> configure ... >> >> Based on your mail I've tried CFLAGS/LDFLAGS again, and >> now Dovecot didn't even compile any longer. > > I don't use the same OS as you, but what errors dis you get? To be exact here, it's not the compiler but the linker failing (of course, the whole problem is about the linking process). With "--as-needed", the crypto/ssl libraries are not linked at all with the object files. I don't quite understand why it doesn't fall back to the system crypto/ssl libraries because they are in the default pathes with all other libraries. (That's basically what most other software packages do if my custom pathes for "-L" "-Wl,-R" somehow get ignored in the building process.) IMHO, the unusual option "--as-needed" should be removed. There seems to be no benefit but it basically keeps Dovecot to be linked against any custom-specified library. Maybe it's just a problem with RHEL/CentOS 6 and the GCC that ships with it. I'm compiling a lot of software myself and link it against my custom OpenSSL. Never had this problem before, otherwise I would have known to specify "-Wl,--no-as-needed" to reverse ld's behavior to the default. Well, at least I've learned something new. :-) Regards, Andreas

2 1