December 2015 - dovecot

mail_log plugin and uid=error in log file
by Alessio Cecchi 02 Dec '15

02 Dec '15

Hi, I enabled notify and mail_log plugins to log the message uid for new emails delivered via LMTP. All works fine except that the uid is not always logged (see uid=error instead of uid=NUMBER) from dovecot.log: Dec 01 10:03:08 lmtp(alessio(a)domain.com): Info: copy from <lmtp DATA>: box=INBOX, uid=error, msgid=<14489594(a)domain.com>, from="WordPress" <wordpress(a)domain.com>, subject=Enquiry from Joy Dec 01 10:03:09 lmtp(alessio(a)domain.com): Info: copy from <lmtp DATA>: box=INBOX, uid=92, msgid=<03c4e16d(a)www.domain.com>, from="WordPress" <wordpress(a)domain.com>, subject=Enquiry from Joy When the uid number is missing the message have is uid in dovecot-uidlist file but is represented in different format: 92 G1448960589.M630964P41853.qb-dev :1448960589.M630964P41853.qb-dev,S=35341,W=35856 93 :1448960586.M423202P41802.qb-dev,S=35378,W=35890 It seems that the uid is determined after the log is written ... I'm running Dovecot 2.2.19, Maildir as storage format and LMTP for delivery. Is a bug or a "feature"? Can be fixed? Thanks this is my configuration: # dovecot -n # 2.2.19: /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.9 # OS: Linux 2.6.32-573.7.1.el6.x86_64 x86_64 CentOS release 6.7 (Final) auth_master_user_separator = * dict { expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext sqlquota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } first_valid_gid = 89 first_valid_uid = 89 last_valid_gid = 89 last_valid_uid = 89 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes log_path = /var/log/dovecot/dovecot.log mail_fsync = always mail_location = maildir:~/Maildir mail_plugins = quota zlib fts fts_lucene acl mail_log notify managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = / } passdb { args = /etc/dovecot/dovecot-deny-sql.conf.ext deny = yes driver = sql } passdb { args = /etc/dovecot/extra/master-users driver = passwd-file master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { acl = vfile:/etc/dovecot/global-acls:cache_secs=300 fts = lucene fts_lucene = whitespace_chars=@. mail_log_events = copy save mail_log_fields = uid box msgid from subject quota = dict:User quota::noenforcing:proxy::sqlquota sieve_default = /etc/dovecot/sieve/default.sieve zlib_save = gz zlib_save_level = 6 } protocols = imap sieve lmtp service auth { unix_listener auth-userdb { group = vchkpw mode = 0660 user = vmail } } service dict { unix_listener dict { group = vchkpw mode = 0660 user = vmail } } service imap-postlogin { executable = script-login /etc/dovecot/scripts/imap-postlogin.sh unix_listener imap-postlogin { group = vchkpw mode = 0660 user = vmail } user = vmail } service imap { executable = imap imap-postlogin } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } } ssl_cert = </etc/dovecot/ssl/wildcard.pem ssl_key = </etc/dovecot/ssl/wildcard.key ssl_protocols = !SSLv2 !SSLv3 userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } verbose_proctitle = yes protocol lmtp { mail_plugins = quota zlib fts fts_lucene acl mail_log notify sieve } protocol imap { mail_plugins = quota zlib fts fts_lucene acl mail_log notify imap_quota imap_zlib imap_acl } -- Alessio Cecchi Postmaster @ http://www.qboxmail.it https://www.linkedin.com/in/alessice

4 5

doveadm backup -R
by Peer Heinlein 02 Dec '15

02 Dec '15

We're using doveadm -o imapc_user="$USER" -o imapc_password="$PASSWORD" -o imapc_host=$SERVER1 -o pop3c_user="$USER" -o pop3c_password="$PASSWORD" -o pop3c_host=$SERVER1 -D -v backup -R -u "$USER" imapc: to migrate Mails from Courier to Dovecot. We have some complaints on the old system that there are changes on the old system where delete mails are appearing again (IMAP) and/or mails are downloaded several times (POP3). AFAIK there can't be changes on the old system if "backup -R" is used. Or is there any situation where changes on the old system (where we connect with imapc) may happen? Peer -- Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-42 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 1

Logstash pattern (GROK, KV, ...) to parse dovecot logs anyone?
by Christian Rohmann 02 Dec '15

02 Dec '15

Hello dovecot-users, I am currently playing with Elastics ELK stack and was kind of surprised to NOT yet find a good set of GROK or KV pattern to parse dovecots lush and information rich logs. The last post regarding this endeavor was in 2014 (http://www.dovecot.org/list/dovecot/2014-June/096589.html) which "only" extracts the key->value pairs but not other parts of the log lines. One finds the occasional attempt here and there on GitHub, like https://github.com/PCextreme/logstash-grok-patterns/blob/master/mail . But nothing in comparison to the simply amazingly good patterns there are for Postfix from whyscream (https://github.com/whyscream/postfix-grok-patterns) He even added some "I don't understand this yet" rule to learn where the parsing lags. I was wondering if anyone here is running logstash and does already have a set of GROK or KV configuration and is willing to share that with the world? A joint effort might get us to a complete extraction of key->values and all other interesting fields for dovecot quickly I hope. Regards Christian

1 0

mail_log & doveadm
by Larry Rosenman 01 Dec '15

01 Dec '15

Is there any way to have doveadm NOT log using mail_log, but have the daemons do it? (or have doveadm's log go to syslogd)? Thanks! ex: doveadm(ler): Info: copy from SA/FN: box=#ARCHIVE/2015/11/SA/FN, uid=1, msgid=<cm.021005.htukhul.cdhuutux.j(a)cmail2.com>, size=107835, vsize=110376, from="Alley Boost" <alleyboost1(a)gmail.com> , subject=Alley Boost: VIP Mixer TONIGHT, Startup Expo TOMORROW, & More! :), flags=(\Seen) -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 7011 W Parmer Ln, Apt 1115, Austin, TX 78729-6961

2 1

lmtp panic
by Sergey Schwartz 01 Dec '15

01 Dec '15

Gents, I've just upgraded to the latest build of dovecot , now lmtp delivery process panics for just one user Nov 10 15:36:49 mx10 dovecot: lmtp(oleg.vasilyev(a)bgoperator.com): Panic: file str.c: line 22 (str_new_const): assertion failed: (str[len] == '\0') Nov 10 15:36:49 mx10 dovecot: lmtp(oleg.vasilyev(a)bgoperator.com): Panic: file str.c: line 22 (str_new_const): assertion failed: (str[len] == '\0') Nov 10 15:36:49 mx10 dovecot: lmtp(oleg.vasilyev(a)bgoperator.com): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x820de) [0x7f50e596b0de] -> /usr/lib/dovecot/libdovecot.so.0(+0x821cc) [0x7f50e596b1cc] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f50e59128de] -> /usr/lib/dovecot/libdovecot.so.0(+0xa8bf8) [0x7f50e5991bf8] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x5ddbd) [0x7f50e380ddbd] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_match+0xf1) [0x7f50e37f43b1] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x5f555) [0x7f50e380f555] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_interpreter_continue+0xe7) [0x7f50e37eb2e7] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_interpreter_run+0x2b) [0x7f50e37eb46b] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x4e6ea) [0x7f50e37fe6ea] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_execute+0x47) [0x7f50e37ff277] -> /usr/lib/dovecot/modules/lib90_sieve_plugin.so(+0x3b81) [0x7f50e3a62b81] -> /usr/lib/dovecot/libdovecot-lda.so.0(mail_deliver+0x49) [0x7f50e5f26899] -> dovecot/lmtp(+0x6a04) [0x7f50e6357a04] -> dovecot/lmtp(+0x72d7) [0x7f50e63582d7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x4c) [0x7f50e597ebbc] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x101) [0x7f50e597ffb1] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x25) [0x7f50e597ec45] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f50e597ede8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f50e59182e3] -> dovecot/lmtp(main+0x165) [0x7f50e6356135] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f50e5545ec5] -> dovecot/lmtp(+0x5223) [0x7f50e6356223] Nov 10 15:36:49 mx10 dovecot: lmtp(oleg.vasilyev(a)bgoperator.com): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x820de) [0x7f887f40e0de] -> /usr/lib/dovecot/libdovecot.so.0(+0x821cc) [0x7f887f40e1cc] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f887f3b58de] -> /usr/lib/dovecot/libdovecot.so.0(+0xa8bf8) [0x7f887f434bf8] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x5ddbd) [0x7f887d2b0dbd] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_match+0xf1) [0x7f887d2973b1] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x5f555) [0x7f887d2b2555] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_interpreter_continue+0xe7) [0x7f887d28e2e7] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_interpreter_run+0x2b) [0x7f887d28e46b] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x4e6ea) [0x7f887d2a16ea] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_execute+0x47) [0x7f887d2a2277] -> /usr/lib/dovecot/modules/lib90_sieve_plugin.so(+0x3b81) [0x7f887d505b81] -> /usr/lib/dovecot/libdovecot-lda.so.0(mail_deliver+0x49) [0x7f887f9c9899] -> dovecot/lmtp(+0x6a04) [0x7f887fdfaa04] -> dovecot/lmtp(+0x72d7) [0x7f887fdfb2d7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x4c) [0x7f887f421bbc] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x101) [0x7f887f422fb1] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x25) [0x7f887f421c45] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f887f421de8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f887f3bb2e3] -> dovecot/lmtp(main+0x165) [0x7f887fdf9135] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f887efe8ec5] -> dovecot/lmtp(+0x5223) [0x7f887fdf9223] Nov 10 15:36:49 mx10 dovecot: lmtp(7502): Connect from local Nov 10 15:36:49 mx10 dovecot: lmtp(7509): Connect from local Nov 10 15:36:49 mx10 dovecot: lmtp(oleg.vasilyev(a)bgoperator.com): Panic: file str.c: line 22 (str_new_const): assertion failed: (str[len] == '\0') Nov 10 15:36:49 mx10 dovecot: lmtp(oleg.vasilyev(a)bgoperator.com): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x820de) [0x7fd26d0530de] -> /usr/lib/dovecot/libdovecot.so.0(+0x821cc) [0x7fd26d0531cc] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fd26cffa8de] -> /usr/lib/dovecot/libdovecot.so.0(+0xa8bf8) [0x7fd26d079bf8] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x5ddbd) [0x7fd26aef5dbd] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_match+0xf1) [0x7fd26aedc3b1] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x5f555) [0x7fd26aef7555] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_interpreter_continue+0xe7) [0x7fd26aed32e7] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_interpreter_run+0x2b) [0x7fd26aed346b] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x4e6ea) [0x7fd26aee66ea] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_execute+0x47) [0x7fd26aee7277] -> /usr/lib/dovecot/modules/lib90_sieve_plugin.so(+0x3b81) [0x7fd26b14ab81] -> /usr/lib/dovecot/libdovecot-lda.so.0(mail_deliver+0x49) [0x7fd26d60e899] -> dovecot/lmtp(+0x6a04) [0x7fd26da3fa04] -> dovecot/lmtp(+0x72d7) [0x7fd26da402d7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x4c) [0x7fd26d066bbc] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x101) [0x7fd26d067fb1] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x25) [0x7fd26d066c45] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7fd26d066de8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7fd26d0002e3] -> dovecot/lmtp(main+0x165) [0x7fd26da3e135] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fd26cc2dec5] -> dovecot/lmtp(+0x5223) [0x7fd26da3e223] Nov 10 15:36:49 mx10 dovecot: lmtp(oleg.vasilyev(a)bgoperator.com): Fatal: master: service(lmtp): child 7494 killed with signal 6 (core dumped) Nov 10 15:36:49 mx10 dovecot: lmtp(oleg.vasilyev(a)bgoperator.com): Fatal: master: service(lmtp): child 7497 killed with signal 6 (core dumped) Nov 10 15:36:49 mx10 dovecot: lmtp(oleg.vasilyev(a)bgoperator.com): Fatal: master: service(lmtp): child 7499 killed with signal 6 (core dumped) Nov 10 15:36:49 mx10 dovecot: lmtp(oleg.vasilyev(a)bgoperator.com): Fatal: master: service(lmtp): child 6719 killed with signal 6 (core dumped) Nov 10 15:36:49 mx10 dovecot: lmtp(oleg.vasilyev(a)bgoperator.com): Fatal: master: service(lmtp): child 7502 killed with signal 6 (core dumped) Nov 10 15:36:51 mx10 dovecot: lmtp(7509): Disconnect from local: Connection closed (in DATA finished) Oleg has lots and lots of small emails :) I'm running ubuntu server 14.04 x86_64 dovecot 2.2.19 (0b1c73b01a5a) build from http://xi.rename-it.nl/debian/ jessie-auto/dovecot-2.2 main -- Best regards, Sergey Schwartz Senior System Administrator Biblio Globus Tour Operator www.bgoperator.ru T: +7 495 5042500 ext 1532 E: sergey.schwartz(a)bgoperator.com

3 7

Let lmtp create target directories
by martin f krafft 01 Dec '15

01 Dec '15

Hello, we're using vmm¹ to manage our postfix+dovecot virtual mail setup, which allows us to give every virtual user a separate EUID and every domain a separate EGID for additional security (vs. handling all virtual mail with a single "vmail" user). As a consequence, however, vmm must itself create the user directories with the appropriate owners, and to do so, it requires root rights. I am trying to investigate getting rid of this need³. Since Dovecot quite happily creates ~/Maildir when necessary, couldn't it also create parents? The home directory should be trivial (same EUID/EGID), but grandparents etc. might need a different policy (e.g. 0/EGID for the grandparent, 0/0 for great-grandparents, etc.). Is this something that could fall within the realm of Dovecot's lmtp? Or is the lmtp invoked as the user and doesn't actually drop root? If so, might there be another way? ¹) http://vmm.localdomain.org/² ²) Hallo Pascal ³) http://bugs.debian.org/804382 Thanks, -- @martinkrafft | http://madduck.net/ | http://two.sentenc.es/ "perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." -- antoine de saint-exupéry spamtraps: madduck.bogus(a)madduck.net

2 1

Looks like a bug to me: Dovecot ignores Maildir/new timestamp
by Fredrik Roubert 01 Dec '15

01 Dec '15

Hello! I run Debian jessie and have an issue with the interaction between Dovecot (2.2.13), Mutt (1.5.23) and the Android Gmail (5.8.105868218) IMAP client, when using Maildir: I use Dovecot LMTP for local delivery into Maildir and access these files both through Dovecot IMAP from the Android Gmail IMAP client and also directly through the filesystem by running Mutt locally on the mail server. When a new mail arrives, Dovecot LMTP will add a new file in Maildir/new and also update the Dovecot index cache. If I then open Mutt and just delete this new message, then Mutt will just delete this file from Maildir/new and the Dovecot index cache will be out-of-sync. (For testing, Mutt can be cut out of the equation entirely by instead just running "rm Maildir/new/*" from the command line to mimic what Mutt does when deleting a new message, but I mention the use of Mutt here to show why this is a real use-case for me.) If I after this open the Gmail app and pull down the message index to refresh from the Dovecot IMAP server, it'll get the cached version of the index, in which the new message is still present (but only the headers, no body will be loaded and the app will show no body snippet). Pulling down the message index again to force a second refresh will cause the new/deleted message to disappear again. This seems to be caused by Dovecot not looking at the Maildir/new timestamp to determine whether the index cache is up-to-date, but instead only looking at the Maildir/cur timestamp. If I instead repeat all these testing steps from the beginning, sending a new mail message, deleting it with "rm Maildir/new/*" but then also running "touch Maildir/cur" before opening the Gmail app and doing the refresh, then Dovecot does update the cache and does not send any ghost messages to the IMAP client. It does seem reasonable to me that Mutt doesn't touch Maildir/cur when just deleting new messages and I suspect that it might be a bug in Dovecot that it timestamp of Maildir/new isn't checked and the index cache updated if that timestamp is newer than the last index update done by the Dovecot LMTP service. Does this analysis seem sound? Did I miss anything? (Output of "doveconf -n" attached.) Cheers // Fredrik Roubert -- Forsterstrasse 64 | +41 78 8170377 CH-8044 Zürich | https://roubert.name/fredrik/

3 2

Sieve, shared folders, different daemons
by Frido Otten 01 Dec '15

01 Dec '15

Hi, In our dovecot setup we use 3 different config files. One for maildropping(sieve), one for dovecot/imap on standard ports with ssl config and one for dovecot/imap on standard ports+1 with a different ssl certificate. Currently the client using the standard port+1 config wants to make use of shared folders, but the thing is that the current namespace separator in all configs is '.'. This is conflicting with shared folders. Can we simply run one of the dovecot/imap daemons configured with a different separator and shared folders without having impact on sieve and the current connected users to the standard ports? Regards, Frido

2 2