August 2018 - dovecot

Dovecot Submission Proxy Auth
by Paul Hecker 12 Jan '19

12 Jan '19

Hi, Dovecot 2.3.1 (8e2f634). Could not get Dovecot to forward the (plain) authentication to the SMTP server using submission. Reason why I need it is sender spoofing (do not want my employees to send messages in behalf of me). In exim I can disable sender spoofing with the authenticated user. When sending through dovecot, exim either does not accept the email (need auth) or relay every sender address (because relaying from localhost). Am I missing a setting or do I need any additional field in the (MySQL) user_query/password_query to forward the password? You can find my config here: https://gist.github.com/lluuaapp/7daddf761131da47237b0f45e6bab5a8 Thanks, Paul

5 12

RE: Dovecot (doveadm, ssl, sync) - SSL error
by Arkadiusz Majewski 08 Jan '19

08 Jan '19

Hello! dsync SSL still doesn't work for replication, so I've disabled it and tried to sync without. But I have a problem with temp directory. Is it possible to change path to temp folder? I don't want to set permissions but change temporary folder for replication. Thanks in advance. dovecot: doveadm: Error: safe_mkstemp(/tmp/dovecot.doveadm.) failed: Permission denied Arkadiusz Majewski From: Arkadiusz Majewski <majewsa(a)gmail.com> Sent: Sunday, December 3, 2017 7:38 PM To: 'dovecot(a)dovecot.org' <dovecot(a)dovecot.org> Subject: Dovecot (doveadm, ssl, sync) - SSL error Hello! I've got a problem to run syncing between both dovecot services on the separate servers. The error indicates to the problem with SSL. Directly using openssl command to connect from one server to other and vice versa is passed without any errors. OS: FreeBSD 11.1-RELEASE-p4 Dovecot: 2.2.33.2_2 and the older one dovecot-2.2.32.1_1 (or similar) - build by ports. OpenSSL: 1.0.2k-freebsd 26 Jan 2017 dovecot: doveadm(10.18.1.15): Error: doveadm client disconnected before handshake: SSL_accept() failed: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol dovecot.conf (on both servers): mail_plugins = $mail_plugins notify replication service replicator { process_min_avail = 1 } service aggregator { fifo_listener replication-notify-fifo { user = dovecot } unix_listener replication-notify { user = dovecot } } service replicator { unix_listener replicator-doveadm { mode = 0600 } } replication_max_conns = 10 service doveadm { inet_listener { port = 12130 ssl = yes } } ssl = required ssl_protocols = SSLv3 TLSv1 TLSv1.1 TLSv1.2 ssl_cert = </usr/local/etc/dovecot/ssl/cert.crt ssl_key = </usr/local/etc/dovecot/ssl/cert.key ssl_client_ca_file = </usr/local/etc/dovecot/ssl/ca.pem ssl_client_ca_dir = /usr/local/etc/dovecot/ssl Thank you in advance for any help. Kind Regards, Arkadiusz Majewski

2 1

Limit IMAP/IMAPS listening IPs
by Dudi Goldenberg 26 Nov '18

26 Nov '18

Hello list. I am running dovecot v2.2.13 on debian 8.8; I tried to add a "address =" statement for only localhost and one private IP, but then I got a complaint from system: dovecot[20515]: Error: systemd listens on port 143, but it's not configured in Dovecot. Closing. What is the proper way to resolve this? TIA & Regards, Dudi

9 23

doveconf error upon boot up
by Jerry 31 Oct '18

31 Oct '18

FreeBSD 11.1-RELEASE-p9 amd64 doveconf # 2.3.1 (8e2f634): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.5.1 (d9bc6dfe) # OS: FreeBSD 11.1-RELEASE-p9 amd64 zfs # Hostname: localhost I can send the entire output if necessary. Error message upon boot up: doveconf: error: t_readlink(/var/run/dovecot/dovecot.conf) failed: readlink() failed: No such file or directory Starting dovecot. Dovecot appears to start correctly There is a link in the "/var/run/dovecot directory for dovecot.conf" lrwx------ 1 root wheel 35B 2018-04-10 10:21:52 EDT dovecot.conf@-> /usr/local/etc/dovecot/dovecot.conf This error message has only started since the update to the latest version of dovecot. I am assuming it is just a harmless error message; however, I would like to confirm that hypothesis. -- Jerry

3 4

OCSP Stapling and Certificate Transparency
by Felipe Gasper 31 Oct '18

31 Oct '18

Hi, For CAs that do not include a signed certificate timestamp in their newly-issued certificates, does Dovecot support either OCSP stapling or the Certificate Transparency TLS extension? If the TLS extension is supported, how does the admin configure the timestamp for each certificate? I’m wondering if any MUAs will follow Google’s lead and insist on CT. Thank you! -Felipe Gasper Mississauga, Ontario

4 3

Re: [ceph-users] dovecot + cephfs - sdbox vs mdbox
by Webert de Souza Lima 04 Oct '18

04 Oct '18

Thanks Jack. That's good to know. It is definitely something to consider. In a distributed storage scenario we might build a dedicated pool for that and tune the pool as more capacity or performance is needed. Regards, Webert Lima DevOps Engineer at MAV Tecnologia *Belo Horizonte - Brasil* *IRC NICK - WebertRLZ* On Wed, May 16, 2018 at 4:45 PM Jack <ceph(a)jack.fr.eu.org> wrote: > On 05/16/2018 09:35 PM, Webert de Souza Lima wrote: > > We'll soon do benchmarks of sdbox vs mdbox over cephfs with bluestore > > backend. > > We'll have to do some some work on how to simulate user traffic, for > writes > > and readings. That seems troublesome. > I would appreciate seeing these results ! > > > Thanks for the plugins recommendations. I'll take the change and ask you > > how is the SIS status? We have used it in the past and we've had some > > problems with it. > > I am using it since Dec 2016 with mdbox, with no issue at all (I am > currently using Dovecot 2.2.27-3 from Debian Stretch) > The only config I use is mail_attachment_dir, the rest lies as default > (mail_attachment_min_size = 128k, mail_attachment_fs = sis posix, > ail_attachment_hash = %{sha1}) > The backend storage is a local filesystem, and there is only one Dovecot > instance > > > > > Regards, > > > > Webert Lima > > DevOps Engineer at MAV Tecnologia > > *Belo Horizonte - Brasil* > > *IRC NICK - WebertRLZ* > > > > > > On Wed, May 16, 2018 at 4:19 PM Jack <ceph(a)jack.fr.eu.org> wrote: > > > >> Hi, > >> > >> Many (most ?) filesystems does not store multiple files on the same > block > >> > >> Thus, with sdbox, every single mail (you know, that kind of mail with 10 > >> lines in it) will eat an inode, and a block (4k here) > >> mdbox is more compact on this way > >> > >> Another difference: sdbox removes the message, mdbox does not : a single > >> metadata update is performed, which may be packed with others if many > >> files are deleted at once > >> > >> That said, I do not have experience with dovecot + cephfs, nor have made > >> tests for sdbox vs mdbox > >> > >> However, and this is a bit out of topic, I recommend you look at the > >> following dovecot's features (if not already done), as they are awesome > >> and will help you a lot: > >> - Compression (classic, https://wiki.dovecot.org/Plugins/Zlib) > >> - Single-Instance-Storage (aka sis, aka "attachment deduplication" : > >> https://www.dovecot.org/list/dovecot/2013-December/094276.html) > >> > >> Regards, > >> On 05/16/2018 08:37 PM, Webert de Souza Lima wrote: > >>> I'm sending this message to both dovecot and ceph-users ML so please > >> don't > >>> mind if something seems too obvious for you. > >>> > >>> Hi, > >>> > >>> I have a question for both dovecot and ceph lists and below I'll > explain > >>> what's going on. > >>> > >>> Regarding dbox format (https://wiki2.dovecot.org/MailboxFormat/dbox) > >> when > >>> using sdbox, a new file is stored for each email message. > >>> When using mdbox, multiple messages are appended to a single file until > >> it > >>> reaches/passes the rotate limit. > >>> > >>> I would like to understand better how the mdbox format impacts on IO > >>> performance. > >>> I think it's generally expected that fewer larger file translate to > less > >> IO > >>> and more troughput when compared to more small files, but how does > >> dovecot > >>> handle that with mdbox? > >>> If dovecot does flush data to storage upon each and every new email is > >>> arrived and appended to the corresponding file, would that mean that it > >>> generate the same ammount of IO as it would do with one file per > message? > >>> Also, if using mdbox many messages will be appended to a said file > >> before a > >>> new file is created. That should mean that a file descriptor is kept > open > >>> for sometime by dovecot process. > >>> Using cephfs as backend, how would this impact cluster performance > >>> regarding MDS caps and inodes cached when files from thousands of users > >> are > >>> opened and appended all over? > >>> > >>> I would like to understand this better. > >>> > >>> Why? > >>> We are a small Business Email Hosting provider with bare metal, self > >> hosted > >>> systems, using dovecot for servicing mailboxes and cephfs for email > >> storage. > >>> > >>> We are currently working on dovecot and storage redesign to be in > >>> production ASAP. The main objective is to serve more users with better > >>> performance, high availability and scalability. > >>> * high availability and load balancing is extremely important to us * > >>> > >>> On our current model, we're using mdbox format with dovecot, having > >>> dovecot's INDEXes stored in a replicated pool of SSDs, and messages > >> stored > >>> in a replicated pool of HDDs (under a Cache Tier with a pool of SSDs). > >>> All using cephfs / filestore backend. > >>> > >>> Currently there are 3 clusters running dovecot 2.2.34 and ceph Jewel > >>> (10.2.9-4). > >>> - ~25K users from a few thousands of domains per cluster > >>> - ~25TB of email data per cluster > >>> - ~70GB of dovecot INDEX [meta]data per cluster > >>> - ~100MB of cephfs metadata per cluster > >>> > >>> Our goal is to build a single ceph cluster for storage that could > expand > >> in > >>> capacity, be highly available and perform well enough. I know, that's > >> what > >>> everyone wants. > >>> > >>> Cephfs is an important choise because: > >>> - there can be multiple mountpoints, thus multiple dovecot instances > on > >>> different hosts > >>> - the same storage backend is used for all dovecot instances > >>> - no need of sharding domains > >>> - dovecot is easily load balanced (with director sticking users to the > >>> same dovecot backend) > >>> > >>> On the upcoming upgrade we intent to: > >>> - upgrade ceph to 12.X (Luminous) > >>> - drop the SSD Cache Tier (because it's deprecated) > >>> - use bluestore engine > >>> > >>> I was said on freenode/#dovecot that there are many cases where SDBOX > >> would > >>> perform better with NFS sharing. > >>> In case of cephfs, at first, I wouldn't think that would be true > because > >>> more files == more generated IO, but thinking about what I said in the > >>> beginning regarding sdbox vs mdbox that could be wrong. > >>> > >>> Any thoughts will be highlt appreciated. > >>> > >>> Regards, > >>> > >>> Webert Lima > >>> DevOps Engineer at MAV Tecnologia > >>> *Belo Horizonte - Brasil* > >>> *IRC NICK - WebertRLZ* > >>> > >>> > >>> > >>> _______________________________________________ > >>> ceph-users mailing list > >>> ceph-users(a)lists.ceph.com > >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>> > >> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users(a)lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > > >

1 1

Re: Dovecot User Listing Error - getpwent() failed: Invalid Argument
by Reuben Farrelly 04 Oct '18

04 Oct '18

Sure: https://www.reub.net/files/dovecot/lightning-dovecot.conf https://www.reub.net/files/dovecot/thunderstorm-dovecot.conf Updated nightly. Reuben On 1/09/2018 12:26 am, Aki Tuomi wrote: > Can you provide doveconf -n? > > > > --- > Aki Tuomi > Dovecot oy > > -------- Original message -------- > From: Reuben Farrelly <reuben-dovecot(a)reub.net> > Date: 31/08/2018 17:12 (GMT+02:00) > To: Aki Tuomi <aki.tuomi(a)dovecot.fi>, Dovecot Mailing List > <dovecot(a)dovecot.org> > Subject: Re: Dovecot User Listing Error - getpwent() failed: Invalid > Argument > > No. Neither of those are installed on either system. > > Reuben > > On 1/09/2018 12:09 am, Aki Tuomi wrote: > > Could apparmor or selinux be causing this! > > > > > > > > --- > > Aki Tuomi > > Dovecot oy > > > > -------- Original message -------- > > From: Reuben Farrelly <reuben-dovecot(a)reub.net> > > Date: 31/08/2018 16:50 (GMT+02:00) > > To: Dovecot Mailing List <dovecot(a)dovecot.org> > > Subject: Dovecot User Listing Error - getpwent() failed: Invalid Argument > > > > Hi, > > > > I'm running dovecot-2.3 git and seeing on 2 of my Dovecot installations, > > the following message logged quite frequently: > > > > Aug 31 16:55:53 lightning.reub.net dovecot[7698]: auth-worker(7707): > > Error: getpwent() failed: Invalid argument > > Aug 31 16:55:53 lightning.reub.net dovecot[7698]: replicator: Error: > > User listing returned failure > > Aug 31 16:55:53 lightning.reub.net dovecot[7698]: replicator: Error: > > listing users failed, can't replicate existing data > > > > It appears that this could be having an effect on replication, but it's > > unclear because the secondary/replica doesn't see much traffic. > > > > Authentication is via PAM and the system is Gentoo Linux x86_64. passb > > driver = pam, userdb driver = static . > > > > doveadm user -u '*' shows the system users listed twice like this (is > > this a problem?) and then prints: > > > > ---- > > reuben > > liam > > reuben > > liam > > <some other users> > > Error: User listing returned failure > > Fatal: user listing failed > > ---- > > > > To me that doesn't look right. > > > > Can anyone suggest what could be causing this? I don't recall seeing > > this message some time ago so I suspect it's a recent change in either > > dovecot, or pam or glibc etc.. > > > > Thanks, > > Reuben

2 10

Nightly builds of Debian packages
by Christoph Pleger 24 Sep '18

24 Sep '18

Hello, how is the future of the automatically built nightly debian packages of dovecot? Has that project been ceased? The current auto-built version is still 2.2.28 from April 28, though 2.2.29 (with a security patch) of dovecot is out now for a while. Regards Christoph

4 3

Re: Duplicate mails on pop3 expunge with dsync replication on 2.2.35 (2.2.33.2 works)
by Gerald Galster 21 Sep '18

21 Sep '18

Hi Tim, > Do you have any new insights on the problem with reappearing mail using dovecot replication + pop3? > > It's driving me mad. I'm running dovecot 2.2.34 (874deae) on OpenBSD and it looks like I have the same problem as you have. unfortunately there has been no response, I'm stuck with 2.2.33.2 for the time being. I can only suspect it has something to do with mailbox locking which was introduced in 2.2.34 (int dsync_mailbox_lock) or maybe with some cached values (drop_older_timestamp) as it just happens when the mail is deleted on the node that it has been synced to but not if it's deleted on the node where it has originally been received. Best regards, Gerald diff -Nru dovecot-2.2.33.2/src/doveadm/dsync/dsync-brain-mailbox.c dovecot-2.2.34/src/doveadm/dsync/dsync-brain-mailbox.c --- dovecot-2.2.33.2/src/doveadm/dsync/dsync-brain-mailbox.c 2017-10-05 19:10:44.000000000 +0200 +++ dovecot-2.2.34/src/doveadm/dsync/dsync-brain-mailbox.c 2018-02-28 15:45:34.000000000 +0100 @@ -522,25 +529,33 @@ } /* mailbox appears to have changed. do a full sync here and get the - state again */ + state again. Lock before syncing. */ + if (dsync_mailbox_lock(brain, box, &lock) < 0) { + brain->failed = TRUE; + mailbox_free(&box); + return -1; + } if (mailbox_sync(box, MAILBOX_SYNC_FLAG_FULL_READ) < 0) { i_error("Can't sync mailbox %s: %s", mailbox_get_vname(box), mailbox_get_last_internal_error(box, &brain->mail_error)); ... @@ -599,6 +615,7 @@ static void dsync_cache_fields_update(const struct dsync_mailbox *local_box, const struct dsync_mailbox *remote_box, + struct mailbox *box, struct mailbox_update *update) { ARRAY_TYPE(mailbox_cache_field) local_sorted, remote_sorted, changes; @@ -630,7 +647,8 @@ local_fields = array_get(&local_sorted, &local_count); remote_fields = array_get(&remote_sorted, &remote_count); t_array_init(&changes, local_count + remote_count); - drop_older_timestamp = ioloop_time - MAIL_CACHE_FIELD_DROP_SECS; + drop_older_timestamp = ioloop_time - + box->index->optimization_set.cache.unaccessed_field_drop_secs; diff -Nru dovecot-2.2.33.2/src/doveadm/dsync/dsync-mailbox.c dovecot-2.2.34/src/doveadm/dsync/dsync-mailbox.c --- dovecot-2.2.33.2/src/doveadm/dsync/dsync-mailbox.c 2017-06-23 13:18:28.000000000 +0200 +++ dovecot-2.2.34/src/doveadm/dsync/dsync-mailbox.c 2018-02-28 15:45:34.000000000 +0100 @@ -1,7 +1,9 @@ -/* Copyright (c) 2013-2017 Dovecot authors, see the included COPYING file */ +/* Copyright (c) 2013-2018 Dovecot authors, see the included COPYING file */ #include "lib.h" #include "istream.h" +#include "mail-storage-private.h" +#include "dsync-brain-private.h" #include "dsync-mailbox.h" void dsync_mailbox_attribute_dup(pool_t pool, @@ -20,3 +22,40 @@ dest_r->last_change = src->last_change; dest_r->modseq = src->modseq; } + +int dsync_mailbox_lock(struct dsync_brain *brain, struct mailbox *box, + struct file_lock **lock_r) +{ + const char *path, *error; + int ret; + + /* Make sure the mailbox is open - locking requires it */ + if (mailbox_open(box) < 0) { + i_error("Can't open mailbox %s: %s", mailbox_get_vname(box), + mailbox_get_last_internal_error(box, &brain->mail_error)); + return -1; + } + + ret = mailbox_get_path_to(box, MAILBOX_LIST_PATH_TYPE_INDEX, &path); + if (ret < 0) { + i_error("Can't get mailbox %s path: %s", mailbox_get_vname(box), + mailbox_get_last_internal_error(box, &brain->mail_error)); + return -1; + } + if (ret == 0) { + /* No index files - don't do any locking. In theory we still + could, but this lock is mainly meant to prevent replication + problems, and replication wouldn't work without indexes. */ + *lock_r = NULL; + return 0; + } + + if (mailbox_lock_file_create(box, DSYNC_MAILBOX_LOCK_FILENAME, + brain->mailbox_lock_timeout_secs, + lock_r, &error) <= 0) { + i_error("Failed to lock mailbox %s for dsyncing: %s", + box->vname, error); + return -1; + } + return 0; +}

2 2

auth_policy in a non-authenticating proxy chain
by Peter Mogensen 15 Sep '18

15 Sep '18

Hi, I was looking into the new Authentication Policy feature: https://wiki2.dovecot.org/Authentication/Policy I had kinda hoped that I would be able to enfore this in a proxy running in front of several backends. This proxy does not authenticate. It use "nopassword". But I realize that the "succes" reported in the final authpolicy req. (command=report) is not what is actaully happening on the IMAP protocol level, but rather the result of the passdb chain in the proxy. (I should probably have predicted this, it's kinda reasonable). However... since the proxy use "nopassword", ALL passdb lookups result in "success", so the proxy will never report an authentication failure to the authpolicy server. This, of course, forces me to do the authpolicy check on the backend with a shared state, but It would still have been nice to have the proxy being able to do the first "command=allow" req. and reject attemps already there even though the backend does "command=report". /Peter

3 4