- dovecot - dovecot.org

Re: Duplicate mails on pop3 expunge with dsync replication on 2.2.35 (2.2.33.2 works)
by Gerald Galster 21 Sep '18

21 Sep '18

Hi Tim, > Do you have any new insights on the problem with reappearing mail using dovecot replication + pop3? > > It's driving me mad. I'm running dovecot 2.2.34 (874deae) on OpenBSD and it looks like I have the same problem as you have. unfortunately there has been no response, I'm stuck with 2.2.33.2 for the time being. I can only suspect it has something to do with mailbox locking which was introduced in 2.2.34 (int dsync_mailbox_lock) or maybe with some cached values (drop_older_timestamp) as it just happens when the mail is deleted on the node that it has been synced to but not if it's deleted on the node where it has originally been received. Best regards, Gerald diff -Nru dovecot-2.2.33.2/src/doveadm/dsync/dsync-brain-mailbox.c dovecot-2.2.34/src/doveadm/dsync/dsync-brain-mailbox.c --- dovecot-2.2.33.2/src/doveadm/dsync/dsync-brain-mailbox.c 2017-10-05 19:10:44.000000000 +0200 +++ dovecot-2.2.34/src/doveadm/dsync/dsync-brain-mailbox.c 2018-02-28 15:45:34.000000000 +0100 @@ -522,25 +529,33 @@ } /* mailbox appears to have changed. do a full sync here and get the - state again */ + state again. Lock before syncing. */ + if (dsync_mailbox_lock(brain, box, &lock) < 0) { + brain->failed = TRUE; + mailbox_free(&box); + return -1; + } if (mailbox_sync(box, MAILBOX_SYNC_FLAG_FULL_READ) < 0) { i_error("Can't sync mailbox %s: %s", mailbox_get_vname(box), mailbox_get_last_internal_error(box, &brain->mail_error)); ... @@ -599,6 +615,7 @@ static void dsync_cache_fields_update(const struct dsync_mailbox *local_box, const struct dsync_mailbox *remote_box, + struct mailbox *box, struct mailbox_update *update) { ARRAY_TYPE(mailbox_cache_field) local_sorted, remote_sorted, changes; @@ -630,7 +647,8 @@ local_fields = array_get(&local_sorted, &local_count); remote_fields = array_get(&remote_sorted, &remote_count); t_array_init(&changes, local_count + remote_count); - drop_older_timestamp = ioloop_time - MAIL_CACHE_FIELD_DROP_SECS; + drop_older_timestamp = ioloop_time - + box->index->optimization_set.cache.unaccessed_field_drop_secs; diff -Nru dovecot-2.2.33.2/src/doveadm/dsync/dsync-mailbox.c dovecot-2.2.34/src/doveadm/dsync/dsync-mailbox.c --- dovecot-2.2.33.2/src/doveadm/dsync/dsync-mailbox.c 2017-06-23 13:18:28.000000000 +0200 +++ dovecot-2.2.34/src/doveadm/dsync/dsync-mailbox.c 2018-02-28 15:45:34.000000000 +0100 @@ -1,7 +1,9 @@ -/* Copyright (c) 2013-2017 Dovecot authors, see the included COPYING file */ +/* Copyright (c) 2013-2018 Dovecot authors, see the included COPYING file */ #include "lib.h" #include "istream.h" +#include "mail-storage-private.h" +#include "dsync-brain-private.h" #include "dsync-mailbox.h" void dsync_mailbox_attribute_dup(pool_t pool, @@ -20,3 +22,40 @@ dest_r->last_change = src->last_change; dest_r->modseq = src->modseq; } + +int dsync_mailbox_lock(struct dsync_brain *brain, struct mailbox *box, + struct file_lock **lock_r) +{ + const char *path, *error; + int ret; + + /* Make sure the mailbox is open - locking requires it */ + if (mailbox_open(box) < 0) { + i_error("Can't open mailbox %s: %s", mailbox_get_vname(box), + mailbox_get_last_internal_error(box, &brain->mail_error)); + return -1; + } + + ret = mailbox_get_path_to(box, MAILBOX_LIST_PATH_TYPE_INDEX, &path); + if (ret < 0) { + i_error("Can't get mailbox %s path: %s", mailbox_get_vname(box), + mailbox_get_last_internal_error(box, &brain->mail_error)); + return -1; + } + if (ret == 0) { + /* No index files - don't do any locking. In theory we still + could, but this lock is mainly meant to prevent replication + problems, and replication wouldn't work without indexes. */ + *lock_r = NULL; + return 0; + } + + if (mailbox_lock_file_create(box, DSYNC_MAILBOX_LOCK_FILENAME, + brain->mailbox_lock_timeout_secs, + lock_r, &error) <= 0) { + i_error("Failed to lock mailbox %s for dsyncing: %s", + box->vname, error); + return -1; + } + return 0; +}

2 2

Sieve and output variable for execute
by Adam Raszkiewicz 20 Sep '18

20 Sep '18

I'm trying to get blow Sieve filter to work require ["fileinto", "imap4flags", "mailbox", "body", "envelope", "vnd.dovecot.pipe", "variables", "vnd.dovecot.execute"]; if envelope :matches "To" "*@*" { set "recipient" "${0}"; set "user" "${1}"; set "recip_domain" "${2}"; } if envelope :matches "From" "*" { set "sender" "${0}"; } #Check if recipient is valid user if execute :output "valid_user" "user-verification" "${recipient}" { if string :matches "${valid_user}" "True" { if body :raw :contains ["message/notification"] { setflag "\\Seen"; fileinto :create "Notifications"; stop; } } } Where user-verification is an extprogram which calls API and verify user based on email address then returns boolean (as an output to the console). Everything works fine when I will remove if string :matches "${valid_user}" "True"statement other way it looks like is not recognizing valid_uservariable. When I pipe valid_user to some script just to capture value for that variable it throws an error: error: specified :args item `True?' is invalid. Why question mark was added to the variable in this case? Thoughts?

2 1

[Sieve] Matches on body content - looking for working example
by Adam Raszkiewicz 20 Sep '18

20 Sep '18

I have tried to do something like if body :content ["multipart"] :matches ["Original-Message-ID" “*”] { set "Original_Message_ID" "${0}"; } but instead getting Original Message ID I’m getting value from previous match which was if envelope :matches "From" "*" { set "sender" "${0}"; } Is there any example of working :matches matching-type with body? Thanks

3 3

Fatal: Support not compiled in for passdb driver 'pam'
by Evandro Holderbach 20 Sep '18

20 Sep '18

Hi Guys, Can anyone help me, I'm having trouble halting caching authentication for the PAM module. I follow this procedure: https://wiki.dovecot.org/PasswordDatabase/PAM I got this error Sep 20 10:05:58 pop3-login: Info: Disconnected: Auth process broken (disconnected before auth was ready, waited 0 secs): user=<>, rip=, lip=, session=<QmW2BU52TACsFBTt> Sep 20 10:06:00 auth: Fatal: Support not compiled in for passdb driver 'pam' Sep 20 10:06:00 master: Error: service(auth): command startup failed, throttling for 4 secs Sep 20 10:06:00 auth: Fatal: Support not compiled in for passdb driver 'pam' dovecot -n # 2.2.4: /usr/local/etc/dovecot/dovecot.conf # OS: Linux 2.6.32-754.3.5.el6.x86_64 x86_64 CentOS release 6.10 (Final) ext4 auth_cache_negative_ttl = 1 days auth_cache_size = 10 M auth_cache_ttl = 1 days auth_verbose = yes disable_plaintext_auth = no instance_name = Mail Popper 1 listen = 172.20.20.222 login_greeting = Mail Popper 1 Ready mail_location = mbox:/var/empty:INBOX=/var/mail/%u:INDEX=MEMORY mail_privileged_group = mail namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = session=yes cache_key=%n dovecot driver = pam } passdb { driver = shadow } protocols = pop3 ssl = no userdb { args = blocking=yes driver = passwd } protocol pop3 { pop3_uidl_format = %08Xu%08Xv } Thank you -- Evandro

1 0

dovecot.index.pvt reset
by Chris 20 Sep '18

20 Sep '18

All, maybe that's a similar issue like the one mentioned in [1], nevertheless I started a new thread. Using public folders: location = maildir:/var/vmail/public/:LAYOUT=fs:INDEXPVT=%h/.public-indexes When deleting a public maildir in filesystem (rm -rf ...) and copying back from backup, it's older than the users' dovecot.index.pvt files. (%h is in another directory). Then the following error appears: Sep 19 07:27:00 mail1 dovecot: imap(dr): Error: /var/vmail/users/dr/.public-indexes/folder1/folder2/dovecot.index.pvt reset, view is now inconsistent Sometimes even a panic: Sep 19 07:27:00 mail1 dovecot: imap(dr): Panic: file mail-index-map.c: line 548 (mail_index_map_lookup _seq_range): assertion failed: (first_uid <= last_uid) When deleting such a maildir via IMAP or doveadm, the error isn't appearing. Is there anything else, one has to consider, when restoring public maildirs from rsync backup? TIA! Chris [1] 5e26fce3-5f9f-219e-55fb-48981027dca5(a)shift.agency

2 1

dovecot.index.pvt view is inconsistent
by collect＠shift.agency 20 Sep '18

20 Sep '18

Hi, we currently work on a fresh setup coming from 2.2.13 going to 2.3.2.1 (0719df592) (Debian Dovecot Repo). We make heavily use of Public Folder with Per-user \Seen flag and Subscription. namespace { type = public separator = / prefix = Verteiler/ location = maildir:/var/vmail/public/example.com/verteiler:SUBSCRIPTIONS=subscriptions… subscriptions = yes hidden = no list = children } With our current version 2.2.13 this setup works. With version 2.3.2.1 we experience errors in the logfile: Sep 8 23:12:27 mail dovecot: Error: imap(user(a)example.com)<11328><e6jPlGJ1IxBOXnhS>: /var/vmail/mailboxes/example.com/user/public/example.com/verteiler/.info/dovecot.index.pvt view is inconsistent dovecot.index.pvt.log is beeing created. We found some appearance of this issue in forums but without any sufficient solution. Is the information provided in this email enough? What else would be relevant? Thank you! Tobias

2 1

blacklistd support for dovecot
by Kurt Lidl 20 Sep '18

20 Sep '18

Greetings all! I ported the NetBSD blacklistd support into FreeBSD a while ago (FreeBSD 11.x timeframe). I also added support to several of the system daemons there, so they could all talk to a centralized daemon (blacklistd). One of the third party apps that I added support to was sendmail - that works pretty well. So I'm now finally turning my attention adding blacklistd support to dovecot (which is the imap daemon that I use). I did an implementation to an earlier version, and that worked OK, at least for non-encrypted access to the imap server. However, the encrypted access (TLS) doesn't work right. To authenticate a network connection, the daemon must pass the fd of the remote connection over a unix domain socket to the actual blacklistd daemon. The blacklistd daemon will then call getpeername() on the connection, so it can determine the remote connection endpoint, rather than just having to believe some made-up set of data that is passed to it from process on the local machine. Christos Zoulas' talk on blacklistd is here: https://www.youtube.com/watch?v=fuuf8G28mjs Anyhow -- the problem I've run into is that there doesn't seem to be any way to get access to the fd of the socket that holds the raw connection to the client when TLS encryption is active. It looks like the only connection I get is to the proxy process. Is there any guidance that someone can give with how to implement this? Thank you. -Kurt

2 1

[Sieve] How to retrieve part of message as a variable
by Adam Raszkiewicz 19 Sep '18

19 Sep '18

I need to retrieve some parts of a message which looks like Subject: Test message Message-Id: <2788db2f-b5c3-4b8c-881e-362c3df4f915(a)server.local> Mime-Version: 1.0 Content-Type: multipart/report; boundary="----=_Part_fba0c199dfcd4d60ae506b37a6320a84"; report-type=disposition-notification To: adam(a)test.local Date: Mon, 17 Sep 2018 16:21:45 -0400 (EDT) From: postmaster(a)server.local ------=_Part_fba0c199dfcd4d60ae506b37a6320a84 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit BTW, this is the correct notification ... it should be "failed" not "failure". ------=_Part_fba0c199dfcd4d60ae506b37a6320a84 Content-Type: message/notification Content-Transfer-Encoding: 7bit Final-Recipient: rfc822; eve(a)test.local Original-Message-ID: <06dbfc12-57a5-49bd-84bd-a24bc3ba1e42(a)fake.cernerasp> ------=_Part_fba0c199dfcd4d60ae506b37a6320a84— for an example I need to grab what is after “Final-Recipient: rfc822;” and “Original-Message-ID:” and store them as variables Thanks

1 0

Re: Auth process sometimes stop responding after upgrade
by Simone Lazzaris 19 Sep '18

19 Sep '18

In data martedì 18 settembre 2018 14:07:26 CEST, Aki Tuomi ha scritto: > If you are using systemd, create > /etc/systemd/system/dovecot.service.d/limits.conf and put > [Service]LimitCORE=infinity > > and run > systemctl daemon-reloadsystemctl restart dovecot Nope, I'm on a debian 7, without systemd. Anyway, I've resolved the issue: I had to set fs.suid_dumpable BEFORE starting dovecot. I've got a core dump, and here is the backtrace. Let me know if you want the core file. gdb) bt #0 0xb76e4428 in __kernel_vsyscall () #1 0xb74636c1 in raise () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 #2 0xb7466af2 in abort () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 #3 0xb76485ae in default_fatal_finish (type=<optimized out>, status=status@entry=0) at failures.c:201 #4 0xb7648641 in i_internal_fatal_handler (ctx=0xbf839cc0, format=0x805c274 "file %s: line %d (%s): assertion failed: (%s)", args=0xbf839ce4 "4\370\005\bI\004") at failures.c:670 #5 0xb75ce35e in i_panic (format=format@entry=0x805c274 "file %s: line %d (%s): assertion failed: (%s)") at failures.c:275 #6 0x080574f7 in doveadm_connection_deinit (_conn=_conn@entry=0xbf839d60) at doveadm-connection.c:1097 #7 0x08057f03 in doveadm_connection_input (conn=0x0) at doveadm-connection.c:1051 #8 0xb76613db in io_loop_call_io (io=0x82fb780) at ioloop.c:600 #9 0xb7662e1e in io_loop_handler_run_internal (ioloop=ioloop@entry=0x82bd648) at ioloop-epoll.c:223 #10 0xb7661496 in io_loop_handler_run (ioloop=ioloop@entry=0x82bd648) at ioloop.c:649 #11 0xb7661658 in io_loop_run (ioloop=0x82bd648) at ioloop.c:624 #12 0xb75da45e in master_service_run (service=0x82bd578, callback=callback@entry=0x804d360 <client_connected>) at master-service.c:719 #13 0x0804cf5e in main (argc=1, argv=0x82bd300) at main.c:366 *Simone Lazzaris* *Qcom S.p.A.* simone.lazzaris(a)qcom.it[1] | www.qcom.it[2] * LinkedIn[3]* | *Facebook*[4] [5] -------- [1] mailto:simone.lazzaris@qcom.it [2] https://www.qcom.it [3] https://www.linkedin.com/company/qcom-spa [4] http://www.facebook.com/qcomspa [5] https://www.qcom.it/includes/email-banner.gif

3 8

HTTP DoveAdm API - Possible bug?
by Chris Malton 19 Sep '18

19 Sep '18

Hi all, It seems that setting userMask in the doveadm http api's "user" command to anything involving wildcards ?s or *s causes the API to fail. When using the API with a userMask set to the fully qualified email address, I get the expected fields back. From a python shell, with an asterisk causes the following behaviour (note that doveadm is a python module that wrappers the dovecot API into Python's requests module). >>> doveadm.do_query("http://[xxxx:xxxx:xxxx:xxxx::ffff]:24280", "API_KEY_HERE", [ ["user", { "userMask": ["*"] }, "c01"] ]) '[["doveadmResponse",,"c01"]]' If I run the same command with a valid mailbox in the userMask Field, I get the response I expect: >>> doveadm.do_query("http://[[xxxx:xxxx:xxxx:xxxx::ffff]:24280", "API_KEY_HERE", [ ["user", { "userMask": ["test(a)test.mydomain.com"] }, "c01"] ]) '[["doveadmResponse",{"test(a)test.mydomain.com":{"uid":"x","gid":"y","home":"/mail/test.mydomain.com/users/test","mail":"maildir:/mail/test.mydomain.com/users/test/Maildir/:INDEX=/indexes/test.myd…","nice":"10"}},"c01"]]' Chatting with people in #dovecot revealed similar behaviour elsewhere so it doesn't appear to be just me. I've also selectively edited the above to avoid leaking data. We're running on the following # 2.2.34 (874deae): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.22 (22940fb7) # OS: Linux 4.9.0-7-amd64 x86_64 Debian 9.5 ext4 If you need a full copy of the config, please let me know and I'll talk to my client, however, I don't think this is config related. Thanks in advance. Regards, Chris Malton -- Delta V Technologies Limited 03333 402 402 www.deltav-tech.co.uk Office: 17 Elm Close, Southampton, SO16 7DT Company No. 11006104 Registered in England and Wales

2 2