February 2019 - dovecot

doveadm import with subfolder oddity
by Mark Moseley 12 Feb '19

12 Feb '19

This has got to be something weird in my config. And the standard disclaimer of '"happy to post doveconf -n, but wanted to see if this is normal first" :) Background: Ubuntu Xenial, running 2.2.36. Mailbox type is mdbox and I've got a period separator in my inbox namespace: namespace { hidden = no inbox = yes list = yes location = mailbox Spam { auto = no autoexpunge = 1 weeks special_use = \Junk } mailbox Trash { auto = no special_use = \Trash } prefix = INBOX. separator = . subscriptions = yes type = private } If I do a import for a regular folder under INBOX, it works just fine: doveadm import -u testbox2(a)testing.local -U testbox1(a)testing.local mdbox:~/mdbox INBOX all mailbox Sent ... returns happily, message count gets incremented If I try to do the same with a subfolder (and a subfolder that most definitely exists on both source and destination side), I get an error: doveadm import -u testbox2(a)testing.local -U testbox1(a)testing.local mdbox:~/mdbox INBOX all mailbox Sub.Sub1 doveadm(testbox2(a)testing.local): Error: remote(10.1.17.98:4000): Mailbox Sub.Sub1: Mailbox sync failed: Mailbox doesn't exist: Sub.Sub1 If I use / instead of . in my query, it works: doveadm import -u testbox2(a)testing.local -U testbox1(a)testing.local mdbox:~/mdbox INBOX all mailbox Sub/Sub1 ... returns happily and message count gets incremented. Since we're using '.' as our separator, that was a bit unexpected :) Ironically, if I'm doing a IMAPc 'import', it works just fine with a query of 'all mailbox Sub.Sub1'. It's only when importing from a local src and local dest (i.e. source_location == mdbox:~/mdbox) that it fails. With source_location set to 'imapc:', it works. I imagine that's due to using straight IMAP on the source side. Likely a misconfig on my part? Expected behavior? I can see in the strace that the error is triggered when doveadm is looking at the source mailbox. It looks for mdbox/mailboxes/Sub.Sub1/dbox-Mails first, then falls back to mdbox/mailboxes/Sub/Sub1/dbox-Mails (which it finds). Then a little bit later in the strace, it again looks for mdbox/mailboxes/Sub.Sub1/dbox-Mails (which it doesn't find) but doesn't try mdbox/mailboxes/Sub/Sub1/dbox-Mails this time, and then spits out 'Mailbox Sub.Sub1: Mailbox sync failed: Mailbox doesn't exist: Sub.Sub1'. With a query of 'all mailbox Sub/Sub1', the stat() is for mdbox/mailboxes/Sub/Sub1/dbox-Mails which it finds and uses happily. Having to substitute the '.'s for '/'s in the 'mailbox' part of the query isn't an awful workaround, but it very much feels like I'm doing something wrong. This is a production setup, so everything else is otherwise working fine. But I've only just begun working with 'doveadm import', so I might be turning up some issues with my config. Thanks! Sorry I'm so verbose :)

1 1

Doveadm service as non-root user
by Mark Moseley 12 Feb '19

12 Feb '19

Running: Ubuntu xenial, dovecot 2.2.36 I've been working on moving our user base from maildir to mdbox and trying to come up with solutions for things like moving emails around. In the past, with maildir, our support guys could just mv the files around and done. For mdbox, I've been working on getting things set up to use doveadm. One weirdness I've seen is that in imports (i.e. doveadm import), mail gets copied correctly but the resulting files are left with root ownership (I don't have 'service doveadm' 'user' set, so I guess it defaults to root). It's typically new m.* files as well as the dovecot.list.index and dovecot.list.index.log files. Looking at strace, no chown is done on them, nor was there setuid. The import had no trouble finding the correct user in the db, so I know that it knows the correct UID (I can see it just fine in debug logs too). And it will happily import to existing m.* files with no permissions issues (but considering it's running as root, I wouldn't expect it to). I've seen this using 'import' via IMAPc as well as with both src and dest on the same server. I can see this behavior in both scenarios. We have a single shared UID for mail, so especially in that "src/dest on same server" case, it's not a matter of UID-mismatch. It's a director setup, so all doveadm commands are coming through the director. If I run the import directly on the backend (which obviously would be a bad idea in real life), the ownership of new m.* files seems to be correct (I can see it setuid'ing to the correct UID from userdb in strace). If I run the import on the director, I can get a new root-owned file every time it rolls over to the next m.* file. Two questions: * Is that a bug? Is this expected behavior? Seems like the expected thing would be to use the UID from userdb and either do a setuid (just like running 'doveadm import' locally did) or chown'ing any new files to the correct UID. I always always assume misconfiguration (vs bug, since it's almost never a bug) but I'm baffled on this one. * I see that it's possible to set a user for service doveadm and the wiki even suggests that it's a good idea in a single UID setup. If there are no mailboxes with any other UIDs, *will setting 'service doveadm' to the same UID possibly break anything*? I can't think of why it would, but I want to be duly diligent. Plus I'm a little leery about closing the door to ever having additional UIDs for mailboxes. Happy to provide 'doveconf -n' but wanted to check first, before spending 15 minutes gently obfuscating it :)

2 3

expunge not removing attachments?
by Ralf Hildebrandt 12 Feb '19

12 Feb '19

I have a large mail backup folder backup(a)backup.invalid; I'm cleaning up daily like this: infimum=`date -d "-4 day" +"%Y-%m-%d"` doveadm expunge -u backup(a)backup.invalid mailbox INBOX SAVEDBEFORE $infimum doveadm purge -u backup(a)backup.invalid yet I see this: # find attachments/ -type f -ctime +5 | wc -l 7522 # find attachments/ -type f | wc -l 127579 # find attachments/ -type f -mtime +5 | wc -l 14361 # find attachments/ -type f | wc -l 127793 About 5.9% of the files in attachments and below are older than 5 days. Why? Is that normal? using dovecot 2:2.3.1-1 from the official repos. -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | https://www.charite.de

2 3

APPEND Error in dovecot.log
by Alessandro Battaglini 12 Feb '19

12 Feb '19

Hi dovecot --version => 2.2.22 (fe789d2) Scenario: Sometimes imap client send email correctly but there's is any message stored in sent mail folder in that scenario I have found an error in my dovecot.log : 2019-02-11 09:04:52 imap(-----@------------): Info: Disconnected in APPEND (1 msgs, 125 secs, 2818048/4426899 bytes) in=3365352 out=1522206 Question: what about this? Is there, in dovecot configuration, a parameter that insist to any kind of APPEND like value.? I have not find that. Have you any idea to solve this? Thanks

1 0

assertion failed: (srcleft <= CHARSET_MAX_PENDING_BUF_SIZE)
by Giacomo 11 Feb '19

11 Feb '19

I'm running the following: # 2.3.4 (0ecbaf23d): /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 11.2-RELEASE-p4 amd64 # FS: ZFS auth_mechanisms = plain login auth_username_format = %Ln listen = * mail_location = maildir:~/Maildir namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = } passdb { driver = pam } protocols = imap lmtp service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } user = dovecot } service imap-login { inet_listener imap { port = 0 } } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service stats { unix_listener stats-reader { group = mode = 0600 user = } unix_listener stats-writer { group = $default_internal_group mode = 0666 user = } user = $default_internal_user } ssl_cert = </root/cert.pem ssl_key = # hidden, use -P to show it userdb { driver = passwd } dmesg shows: pid 74441 (imap), uid 1002: exited on signal 6 pid 74784 (imap), uid 1002: exited on signal 6 pid 21471 (imap), uid 1002: exited on signal 6 pid 21470 (imap), uid 1002: exited on signal 6 pid 21464 (imap), uid 1002: exited on signal 6 pid 21731 (imap), uid 1002: exited on signal 6 Sometimes the dovecot process serving an user crashes. He runs mostly airmail on macOS and samsung mail on Android. Happens only to that particular user. The log shows the following: (maillog) Dec 16 14:52:35 mail dovecot: imap-login: Login: user=<USER>, method=PLAIN, rip=79.12.201.157, lip=192.168.1.28, mpid=74779, TLS, session=<wMQI+iN9VtRPDMmd> Dec 16 14:52:39 mail dovecot: imap(USER)<74441><nLa+7SN9+fhPDMmd>: Panic: file charset-iconv.c: line 83 (charset_to_utf8_try): assertion failed: (srcleft <= CHARSET_MAX_PENDING_BUF_SIZE) Dec 16 14:52:39 mail dovecot: imap(USER)<74441><nLa+7SN9+fhPDMmd>: Fatal: master: service(imap): child 74441 killed with signal 6 (core not dumped - https://dovecot.org/bugreport.html#coredumps - set service imap { drop_priv_before_exec=yes }) I'm here if more info is needed. Thanks for taking a look.

4 7

Bounces?
by ＠lbutlr 11 Feb '19

11 Feb '19

I just got this On 9 Feb 2019, at 00:00, dovecot-request(a)dovecot.org wrote: > Your membership in the mailing list dovecot has been disabled due to > excessive bounces The last bounce received from you was dated > 02-Feb-2019. You will not get any more messages from this list until > you re-enable your membership. You will receive 1 more reminders like > this before your membership in the list is deleted. > > To re-enable your membership, you can simply respond to this message > (leaving the Subject: line intact), or visit the confirmation page at > > https://dovecot.org/mailman/confirm/dovecot/41881d… When clicking the link I get "Bad confirmation sting" and something about maybe the link expired. I see no connection attempts from dovecot.org over the last few days other than this message. -- If at first you don't succeed, destroy all evidence that you tried.

8 18

Use sieve to move all mail automatically based on List-Id
by Matthias Fechner 09 Feb '19

09 Feb '19

Dear all, I currently use the following rule to automatically sort email into folders based on mailing lists: # split out the various list forms # Mailman & other lists using list-id if exists "list-id" { if header :regex "list-id" "<([a-z_0-9-]+)[.@]" { set :lower "listname" "${1}"; fileinto :create "${listname}"; } else { if header :regex "list-id" "^\\s*<?([a-z_0-9-]+)[.@]" { set :lower "listname" "${1}"; fileinto :create "${listname}"; } else { keep; } } stop; } This works really nice. The only problem I have here is that it does use the ID, but not the name and this can lead to strange folder names. If an email has e.g. the following header: List-Id: mfechner/ci-test <142.ci-test.mfechner.gitlab.fechner.net> it moves the email to the folder 142 as the regex matches inside the <> string till the none-letter/none-number/_/- is matching. I do not understand the expresion [.@] so it is nearly impossible for my to extend that query. What I would like to do is using the ID as folder name as fallback. So in this case I would like that the email is moved to folder `mfechner/ci-test`. I'm not sure if sieve can handle the `/` correctly. Thanks a lot for any tip. Gruß Matthias -- "Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to produce bigger and better idiots. So far, the universe is winning." -- Rich Cook

1 0

lib-master test failure i686
by Eric Broch 08 Feb '19

08 Feb '19

Hello List, I've built and checked successfully dovecot and dovecot-pigeonhole on CentOS 6 x86_64 Immediately below are the successful make check (ok) tests of dovecot 'lib-master' (x86_64). Below that are the unsuccessful (fails) tests of dovecot 'lib-master' on i686. Does anyone have a clue as to why these test would fail on i686? (x86_64): Making check in lib-master make[2]: Entering directory `/builddir/build/BUILD/dovecot-2.3.4/src/lib-master' make check-local make[3]: Entering directory `/builddir/build/BUILD/dovecot-2.3.4/src/lib-master' for bin in test-master-service-settings-cache test-event-stats; do \ if ! ./$bin; then exit 1; fi; \ done 0 / 0 tests failed no merging parent is NULL ............................................ : ok no merging parent sent to stats ...................................... : ok no merging parent timestamp differs .................................. : ok merge events parent NULL ............................................. : ok merge events parent sent to stats .................................... : ok skip empty parents ................................................... : ok merge events and skip empty parents .................................. : ok 0 / 7 tests failed make[3]: Leaving directory `/builddir/build/BUILD/dovecot-2.3.4/src/lib-master' make[2]: Leaving directory `/builddir/build/BUILD/dovecot-2.3.4/src/lib-master' (i686): Making check in lib-master make[2]: Entering directory `/builddir/build/BUILD/dovecot-2.3.4/src/lib-master' make check-local make[3]: Entering directory `/builddir/build/BUILD/dovecot-2.3.4/src/lib-master' for bin in test-master-service-settings-cache test-event-stats; do \ if ! ./$bin; then exit 1; fi; \ done 0 / 0 tests failed no merging parent is NULL ............................................ : ok test-event-stats.c:365: Assert failed: compare_test_stats_to( "EVENT %lu 1 0 0" " stest-event-stats.c %d" " l0 0 ctest2\n", id, l) no merging parent sent to stats ...................................... : FAILED test-event-stats.c:394: Assert failed: compare_test_stats_to( "BEGIN %lu 0 1 0 0" " stest-event-stats.c %d ctest1\n" "EVENT %lu 1 1 0" " stest-event-stats.c %d" " l1 0 ctest2\n" "END\t%lu\n", idp, lp, idp, l, idp) no merging parent timestamp differs .................................. : FAILED merge events parent NULL ............................................. : ok test-event-stats.c:458: Assert failed: compare_test_stats_to( "EVENT %lu 1 0 0" " stest-event-stats.c %d l0 0" " ctest3 ctest2 ctest1 Tkey3" " 10 0 Ikey2 20" " Skey1 str1\n", id, l) merge events parent sent to stats .................................... : FAILED test-event-stats.c:490: Assert failed: compare_test_stats_to( "BEGIN %lu 0 1 0 0" " stest-event-stats.c %d ctest1\n" "EVENT %lu 1 3 0 " "stest-event-stats.c %d l3 0" " ctest2\nEND\t%lu\n", id, lp, id, l, id) skip empty parents ................................................... : FAILED test-event-stats.c:533: Assert failed: compare_test_stats_to( "BEGIN %lu 0 1 0 0" " stest-event-stats.c %d ctest1\n" "EVENT %lu 1 3 0 " "stest-event-stats.c %d l3 0 " "ctest4 ctest5 Tkey3 10 0 Skey4" " str4\nEND\t%lu\n", id, lp, id, l, id) merge events and skip empty parents .................................. : FAILED 5 / 7 tests failed make[3]: Leaving directory `/builddir/build/BUILD/dovecot-2.3.4/src/lib-master' make[3]: *** [check-local] Error 1 make[2]: *** [check-am] Error 2 make[2]: Leaving directory `/builddir/build/BUILD/dovecot-2.3.4/src/lib-master' -- Eric Broch White Horse Technical Consulting (WHTC)

2 4

mbox 2 Maildir
by Tamas Hegedus 08 Feb '19

08 Feb '19

Hi, I am not an professional system admin, but I am running a small debian mail server for many years. It is time to migrate from mbox to Maildir. There is a nice and clean description for doing this without service downtime at https://wiki.dovecot.org/Tools/Doveadm/Sync I have to set per-user config for mailbox type using userdb extra fields. However, I miss some information and I can not find useful info by google. 1. The main question: dovecot will know that for some users the mailbox format is still mbox, but others already have Maildir. I do not see any hint that I should setup postfix/procmail for handling the delivery per-user basis. But I think I have to set this, too. 2. I have not learned much about userdb and it seems that dovecot was written for and used by experts. I mean that the docs/wiki is not enough for me: from the wiki page of userdb it will be for me difficult to setup one. Can you suggest some docs for dummies? Thanks for your help, Tamas dovecot --version 2.2.27 (c0f36b0) postfix 3.1.8 procmail v3.23pre 2001/09/13 -- Tamas Hegedus, PhD Senior Research Fellow MTA-SE Molecular Biophysics Research Group Hungarian Academy of Sciences | phone: (36) 1-459 1500/60233 Semmelweis University | fax: (36) 1-266 6656 Tuzolto utca 37-47 | mailto:tamas@hegelab.org Budapest, 1094, Hungary | http://www.hegelab.org --- This email has been checked for viruses by AVG. https://www.avg.com

4 8

system wakeup caused by write operations to /var/lib/dovecot/instances
by Tijl 07 Feb '19

07 Feb '19

Hi, I hope someone can help me with this. Problem Dovecot writes to _/var/lib/dovecot/instances_ 1 up to 3 times a day, causing the system to wake up and disks to spin up. Details content of /var/lib/dovecot/instances : 1549073010 dovecot /var/run/dovecot /etc/dovecot/dovecot.conf --> only the timestamp is changed (to current time) when dovecot writes to this file --> during write the file is temporary renamed to _instances.lock_ dovecot running processes: dovecot (main process) --> it's this process that writes to /var/lib/instances dovecot/anvil dovecot/log dovecot/config --> sometimes this process is not running, but doesn't change anything note: no clients connected problem also occur in the middle of the night (no human activity, client devices powered off, all external traffic blocked) the writings occur around every 8 hours, but vary, sometimes it's 6 hours up to 13 hours. Configuration See attachment: doveconfig-short-[platform].txt (output of _"/usr/sbin/dovecot -n"_) doveconfig-long-[platform].txt (output of _"/usr/bin/doveconf"_) --> _3-docker_ config is the most simple setup of the 3 Environment Tested on 3 platforms Platform 1: Intel Atom C2538 (Synology DiskStation 415+) Linux version 3.10.105 Dovecot version 2.2.19 (Synology MailPlus Server) Platform 2: ARMv7 rev 1 (Synology DiskStation 215j) Linux version 3.2.40 Dovecot version 2.2.19 (Synology Mail Server) Platform 3: Intel Atom C2538 (Docker container) Linux version 3.10.105 Dovecot version 2.3.2.1 (0719df592) How can dovecot be run without writing to /var/lib/dovecot/instances everyday? Is there a configuration setting for this? Thanks already! Tijl

3 4