- dovecot - dovecot.org

Error using dsync to convert really old mboxes to Maildir and workaround
by faginbagin 24 Sep '18

24 Sep '18

I ran across what I think may be a bug in dsync when trying to convert from mbox to Maildir format. My first attempt failed with a runaway dsync process spewing out thousands of identical lines of the form: dsync(buus): Error: Duplicate mailbox GUID 00fe691ff60da85b090d0000c8119523 for mailboxes Archive/Personal and Work/VendorInfo - giving a new GUID 00fe691ff60da85b090d0000c8119523 to Archive/Personal These two mail folders happen to be among a number of folders I've had for many years, but also haven't looked at in many years. I used to use UW-IMAP, but switched to dovecot about two years ago, and I'm sure I've never used dovecot to look at those folders. So I suspect the root cause of the above error message was that dovecot had never generated GUIDs for these and some other really old folders. After killing dsync with SIGKILL, which couldn't be killed with SIGTERM, I restored my mbox format Mail folder from backup, restored the mail_location and then coerced dovecot to generate unique GUIDs for all my mail folders with this command: doveadm mailbox status guid `doveadm mailbox list` After the above command, I was able to convert my mail from mbox to Maildir format. Maybe this will help someone else somewhere down the road.

2 2

Released Pigeonhole v0.5.3.rc1 for Dovecot v2.3.3.rc1.
by Stephan Bosch 23 Sep '18

23 Sep '18

Hello Dovecot users, Here is the Pigeonhole release candidate that goes with the Dovecot v2.3.3 release candidate. It consists only of bugfixes. Changelog v0.5.3: - Fix assertion panic occurring when managesieve service fails to open INBOX while saving a Sieve script. This was caused by a lack of cleanup after failure. - Fix specific messages causing an assert panic with actions that compose a reply (e.g. vacation). With some rather weird input from the original message, the header folding algorithm (as used for composing the References header for the reply) got confused, causing the panic. - IMAP FILTER=SIEVE capability: Fix FILTER SIEVE SCRIPT command parsing. After finishing reading the Sieve script, the command parsing sometimes didn't continue with the search arguments. This is a time- critical bug that likely only occurs when the Sieve script is sent in the next TCP frame. The release is available as follows: http://pigeonhole.dovecot.org/releases/2.3/rc/dovecot-2.3-pigeonhole-0.5.3.… http://pigeonhole.dovecot.org/releases/2.3/rc/dovecot-2.3-pigeonhole-0.5.3.… Refer to http://pigeonhole.dovecot.org and the Dovecot wiki for more information. Have fun testing this release candidate and don't hesitate to notify me when there are any problems. Regards, -- Stephan Bosch stephan(a)rename-it.nl

1 0

Global sieve doesn't run and I don't know why
by Ranbir 21 Sep '18

21 Sep '18

Hi Everyone, I have dovecot, postfix, roundcube and piegeonhole managesieve all working nicely together. Personal scripts defined in roundcube run without issue. I have configured the global sieve dir defined in 90-sieve.conf: sieve_global_dir = /var/lib/dovecot/sieve/global/ I pre-compiled the script in there: # ls -l /var/lib/dovecot/sieve/global/ total 8 -rw-r--r--. 1 root root 118 Sep 19 16:52 global.sieve -rw-r--r--. 1 root root 197 Sep 19 16:56 global.svbin But, in the logs for dovecot, I never see the script being run: https://www.pastiebin.com/5ba3faa8628c2 Users are from freeipa. Their home directories are symlinks to where the mail really lives. For example: lrwxrwxrwx. 1 ranbir virtual 42 May 25 2017 ranbir -> /var/spool/mail/domain.tld/ranbir/ And the contents of that dir: [root@mailhost conf.d]# ls -al /home/DOMAIN/ranbir/ total 64 drwxrwx---. 4 ranbir virtual 4096 Sep 18 17:40 . drwxrwx---+ 13 virtual virtual 4096 Jul 8 12:06 .. -rw-------. 1 ranbir ranbir 17808 Sep 20 01:13 .bash_history -rw-------. 1 ranbir ranbir 18 Dec 14 2015 .bash_logout -rw-------. 1 ranbir ranbir 193 Dec 14 2015 .bash_profile -rw-------. 1 ranbir ranbir 231 Dec 14 2015 .bashrc lrwxrwxrwx. 1 virtual virtual 23 Feb 11 2018 .dovecot.sieve -> sieve/managesieve.sieve lrwxrwxrwx. 1 virtual virtual 23 Sep 18 16:53 .dovecot.sieve.log -> sieve/dovecot.sieve.log -rw-rw----. 1 virtual virtual 322 Sep 18 17:40 .dovecot.svbin -rw-------. 1 ranbir ranbir 59 Jun 7 2016 .lesshst drwx------. 56 virtual virtual 4096 Sep 20 15:02 Maildir drwxrwx---. 3 ranbir virtual 89 Sep 18 17:16 sieve -rw-------. 1 ranbir ranbir 6415 Feb 11 2018 .viminfo Did I miss something in the configuration? Dovecot isn't throwing any errors. I can only conclude I haven't completed the setup for global sieve to work or that I've misunderstood what the "sieve_global_dir" paramater is for. Thanks! -- Ranbir

4 11

Re: v2.3.3 release candidate released
by Tuomi, Aki 21 Sep '18

21 Sep '18

It should work if you add it to apt. Directory listing is denied, which causes 403 if you access with browser. Aki -------- Original message --------From: Michael Marley <michael(a)michaelmarley.com> Date: 21/09/2018 18:00 (GMT+02:00) To: Dovecot Mailing List <dovecot(a)dovecot.org> Subject: Re: v2.3.3 release candidate released On 2018-09-21 09:45, Timo Sirainen wrote: > https://dovecot.org/releases/2.3/rc/dovecot-2.3.3.rc1.tar.gz > https://dovecot.org/releases/2.3/rc/dovecot-2.3.3.rc1.tar.gz.sig > > Binary packages are also available in https://repo.dovecot.org/ in > ce-2.3.3 repository (not ce-2.3-latest). > > * doveconf hides more secrets now in the default output. > * ssl_dh setting is no longer enforced at startup. If it's not set and > non-ECC DH key exchange happens, error is logged and client is > disconnected. > > + Added log_debug=<filter> setting. > + Added log_core_filter=<log filter> setting. > + quota-clone: Write to dict asynchronously > + --enable-hardening attempts to use retpoline Spectre 2 mitigations > + lmtp proxy: Support source_ip passdb extra field. > + doveadm stats dump: Support more fields and output stddev by > default. > + push-notification: Add SSL support for OX backend. > - NUL bytes in mail headers can cause truncated replies when fetched. > - director: Conflicting host up/down state changes may in some rare > situations ended up in a loop of two directors constantly > overwriting > each others' changes. > - virtual plugin: Some searches used 100% CPU for many seconds > - dsync assert-crashed with acl plugin in some situations. > - mail_attachment_detection_options=add-flags-on-save assert-crashed > with some specific Sieve scripts. > - Mail snippet generation crashed with mails containing invalid > Content-Type:multipart header. > - Log prefix ordering was different for some log lines. > - quota: With noenforcing option current quota usage wasn't updated. > - auth: Kerberos authentication against Samba assert-crashed. > - stats clients were unnecessarily chatty with the stats server. > - imapc: Fixed various assert-crashes when reconnecting to server. > - lmtp, submission: Fix potential crash if client disconnects while > handling a command. > - quota: Fixed compiling with glibc-2.26 / support libtirpc. > - fts-solr: Empty search values resulted in 400 Bad Request errors > - fts-solr: default_ns parameter couldn't be used > - submission server crashed if relay server returned over 7 lines in > a reply (e.g. to EHLO) It looks like the ce-2.3.3 repository (or at least https://repo.dovecot.org/ce-2.3.3/ubuntu/xenial/) is returning a 403. Michael

1 0

v2.3.3 release candidate released
by Timo Sirainen 21 Sep '18

21 Sep '18

https://dovecot.org/releases/2.3/rc/dovecot-2.3.3.rc1.tar.gz https://dovecot.org/releases/2.3/rc/dovecot-2.3.3.rc1.tar.gz.sig Binary packages are also available in https://repo.dovecot.org/ in ce-2.3.3 repository (not ce-2.3-latest). * doveconf hides more secrets now in the default output. * ssl_dh setting is no longer enforced at startup. If it's not set and non-ECC DH key exchange happens, error is logged and client is disconnected. + Added log_debug=<filter> setting. + Added log_core_filter=<log filter> setting. + quota-clone: Write to dict asynchronously + --enable-hardening attempts to use retpoline Spectre 2 mitigations + lmtp proxy: Support source_ip passdb extra field. + doveadm stats dump: Support more fields and output stddev by default. + push-notification: Add SSL support for OX backend. - NUL bytes in mail headers can cause truncated replies when fetched. - director: Conflicting host up/down state changes may in some rare situations ended up in a loop of two directors constantly overwriting each others' changes. - virtual plugin: Some searches used 100% CPU for many seconds - dsync assert-crashed with acl plugin in some situations. - mail_attachment_detection_options=add-flags-on-save assert-crashed with some specific Sieve scripts. - Mail snippet generation crashed with mails containing invalid Content-Type:multipart header. - Log prefix ordering was different for some log lines. - quota: With noenforcing option current quota usage wasn't updated. - auth: Kerberos authentication against Samba assert-crashed. - stats clients were unnecessarily chatty with the stats server. - imapc: Fixed various assert-crashes when reconnecting to server. - lmtp, submission: Fix potential crash if client disconnects while handling a command. - quota: Fixed compiling with glibc-2.26 / support libtirpc. - fts-solr: Empty search values resulted in 400 Bad Request errors - fts-solr: default_ns parameter couldn't be used - submission server crashed if relay server returned over 7 lines in a reply (e.g. to EHLO)

2 1

Re: Duplicate mails on pop3 expunge with dsync replication on 2.2.35 (2.2.33.2 works)
by Gerald Galster 21 Sep '18

21 Sep '18

Hi Tim, > Do you have any new insights on the problem with reappearing mail using dovecot replication + pop3? > > It's driving me mad. I'm running dovecot 2.2.34 (874deae) on OpenBSD and it looks like I have the same problem as you have. unfortunately there has been no response, I'm stuck with 2.2.33.2 for the time being. I can only suspect it has something to do with mailbox locking which was introduced in 2.2.34 (int dsync_mailbox_lock) or maybe with some cached values (drop_older_timestamp) as it just happens when the mail is deleted on the node that it has been synced to but not if it's deleted on the node where it has originally been received. Best regards, Gerald diff -Nru dovecot-2.2.33.2/src/doveadm/dsync/dsync-brain-mailbox.c dovecot-2.2.34/src/doveadm/dsync/dsync-brain-mailbox.c --- dovecot-2.2.33.2/src/doveadm/dsync/dsync-brain-mailbox.c 2017-10-05 19:10:44.000000000 +0200 +++ dovecot-2.2.34/src/doveadm/dsync/dsync-brain-mailbox.c 2018-02-28 15:45:34.000000000 +0100 @@ -522,25 +529,33 @@ } /* mailbox appears to have changed. do a full sync here and get the - state again */ + state again. Lock before syncing. */ + if (dsync_mailbox_lock(brain, box, &lock) < 0) { + brain->failed = TRUE; + mailbox_free(&box); + return -1; + } if (mailbox_sync(box, MAILBOX_SYNC_FLAG_FULL_READ) < 0) { i_error("Can't sync mailbox %s: %s", mailbox_get_vname(box), mailbox_get_last_internal_error(box, &brain->mail_error)); ... @@ -599,6 +615,7 @@ static void dsync_cache_fields_update(const struct dsync_mailbox *local_box, const struct dsync_mailbox *remote_box, + struct mailbox *box, struct mailbox_update *update) { ARRAY_TYPE(mailbox_cache_field) local_sorted, remote_sorted, changes; @@ -630,7 +647,8 @@ local_fields = array_get(&local_sorted, &local_count); remote_fields = array_get(&remote_sorted, &remote_count); t_array_init(&changes, local_count + remote_count); - drop_older_timestamp = ioloop_time - MAIL_CACHE_FIELD_DROP_SECS; + drop_older_timestamp = ioloop_time - + box->index->optimization_set.cache.unaccessed_field_drop_secs; diff -Nru dovecot-2.2.33.2/src/doveadm/dsync/dsync-mailbox.c dovecot-2.2.34/src/doveadm/dsync/dsync-mailbox.c --- dovecot-2.2.33.2/src/doveadm/dsync/dsync-mailbox.c 2017-06-23 13:18:28.000000000 +0200 +++ dovecot-2.2.34/src/doveadm/dsync/dsync-mailbox.c 2018-02-28 15:45:34.000000000 +0100 @@ -1,7 +1,9 @@ -/* Copyright (c) 2013-2017 Dovecot authors, see the included COPYING file */ +/* Copyright (c) 2013-2018 Dovecot authors, see the included COPYING file */ #include "lib.h" #include "istream.h" +#include "mail-storage-private.h" +#include "dsync-brain-private.h" #include "dsync-mailbox.h" void dsync_mailbox_attribute_dup(pool_t pool, @@ -20,3 +22,40 @@ dest_r->last_change = src->last_change; dest_r->modseq = src->modseq; } + +int dsync_mailbox_lock(struct dsync_brain *brain, struct mailbox *box, + struct file_lock **lock_r) +{ + const char *path, *error; + int ret; + + /* Make sure the mailbox is open - locking requires it */ + if (mailbox_open(box) < 0) { + i_error("Can't open mailbox %s: %s", mailbox_get_vname(box), + mailbox_get_last_internal_error(box, &brain->mail_error)); + return -1; + } + + ret = mailbox_get_path_to(box, MAILBOX_LIST_PATH_TYPE_INDEX, &path); + if (ret < 0) { + i_error("Can't get mailbox %s path: %s", mailbox_get_vname(box), + mailbox_get_last_internal_error(box, &brain->mail_error)); + return -1; + } + if (ret == 0) { + /* No index files - don't do any locking. In theory we still + could, but this lock is mainly meant to prevent replication + problems, and replication wouldn't work without indexes. */ + *lock_r = NULL; + return 0; + } + + if (mailbox_lock_file_create(box, DSYNC_MAILBOX_LOCK_FILENAME, + brain->mailbox_lock_timeout_secs, + lock_r, &error) <= 0) { + i_error("Failed to lock mailbox %s for dsyncing: %s", + box->vname, error); + return -1; + } + return 0; +}

2 2

Sieve and output variable for execute
by Adam Raszkiewicz 20 Sep '18

20 Sep '18

I'm trying to get blow Sieve filter to work require ["fileinto", "imap4flags", "mailbox", "body", "envelope", "vnd.dovecot.pipe", "variables", "vnd.dovecot.execute"]; if envelope :matches "To" "*@*" { set "recipient" "${0}"; set "user" "${1}"; set "recip_domain" "${2}"; } if envelope :matches "From" "*" { set "sender" "${0}"; } #Check if recipient is valid user if execute :output "valid_user" "user-verification" "${recipient}" { if string :matches "${valid_user}" "True" { if body :raw :contains ["message/notification"] { setflag "\\Seen"; fileinto :create "Notifications"; stop; } } } Where user-verification is an extprogram which calls API and verify user based on email address then returns boolean (as an output to the console). Everything works fine when I will remove if string :matches "${valid_user}" "True"statement other way it looks like is not recognizing valid_uservariable. When I pipe valid_user to some script just to capture value for that variable it throws an error: error: specified :args item `True?' is invalid. Why question mark was added to the variable in this case? Thoughts?

2 1

[Sieve] Matches on body content - looking for working example
by Adam Raszkiewicz 20 Sep '18

20 Sep '18

I have tried to do something like if body :content ["multipart"] :matches ["Original-Message-ID" “*”] { set "Original_Message_ID" "${0}"; } but instead getting Original Message ID I’m getting value from previous match which was if envelope :matches "From" "*" { set "sender" "${0}"; } Is there any example of working :matches matching-type with body? Thanks

3 3

Fatal: Support not compiled in for passdb driver 'pam'
by Evandro Holderbach 20 Sep '18

20 Sep '18

Hi Guys, Can anyone help me, I'm having trouble halting caching authentication for the PAM module. I follow this procedure: https://wiki.dovecot.org/PasswordDatabase/PAM I got this error Sep 20 10:05:58 pop3-login: Info: Disconnected: Auth process broken (disconnected before auth was ready, waited 0 secs): user=<>, rip=, lip=, session=<QmW2BU52TACsFBTt> Sep 20 10:06:00 auth: Fatal: Support not compiled in for passdb driver 'pam' Sep 20 10:06:00 master: Error: service(auth): command startup failed, throttling for 4 secs Sep 20 10:06:00 auth: Fatal: Support not compiled in for passdb driver 'pam' dovecot -n # 2.2.4: /usr/local/etc/dovecot/dovecot.conf # OS: Linux 2.6.32-754.3.5.el6.x86_64 x86_64 CentOS release 6.10 (Final) ext4 auth_cache_negative_ttl = 1 days auth_cache_size = 10 M auth_cache_ttl = 1 days auth_verbose = yes disable_plaintext_auth = no instance_name = Mail Popper 1 listen = 172.20.20.222 login_greeting = Mail Popper 1 Ready mail_location = mbox:/var/empty:INBOX=/var/mail/%u:INDEX=MEMORY mail_privileged_group = mail namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = session=yes cache_key=%n dovecot driver = pam } passdb { driver = shadow } protocols = pop3 ssl = no userdb { args = blocking=yes driver = passwd } protocol pop3 { pop3_uidl_format = %08Xu%08Xv } Thank you -- Evandro

1 0

dovecot.index.pvt reset
by Chris 20 Sep '18

20 Sep '18

All, maybe that's a similar issue like the one mentioned in [1], nevertheless I started a new thread. Using public folders: location = maildir:/var/vmail/public/:LAYOUT=fs:INDEXPVT=%h/.public-indexes When deleting a public maildir in filesystem (rm -rf ...) and copying back from backup, it's older than the users' dovecot.index.pvt files. (%h is in another directory). Then the following error appears: Sep 19 07:27:00 mail1 dovecot: imap(dr): Error: /var/vmail/users/dr/.public-indexes/folder1/folder2/dovecot.index.pvt reset, view is now inconsistent Sometimes even a panic: Sep 19 07:27:00 mail1 dovecot: imap(dr): Panic: file mail-index-map.c: line 548 (mail_index_map_lookup _seq_range): assertion failed: (first_uid <= last_uid) When deleting such a maildir via IMAP or doveadm, the error isn't appearing. Is there anything else, one has to consider, when restoring public maildirs from rsync backup? TIA! Chris [1] 5e26fce3-5f9f-219e-55fb-48981027dca5(a)shift.agency

2 1