June 2022 - dovecot - dovecot.org

dovecot and openssl3
by A. Schulze 21 Jun '22

21 Jun '22

Hello, I tried to build dovecot-2.3.19.1 with openssl-3.0.2 "make test" failed. Good, the test-suite found a problem! I found similar reports and patches: - https://sources.debian.org/src/dovecot/1%3A2.3.19%2Bdfsg1-1/debian/patches/… - https://bugzilla.redhat.com/show_bug.cgi?id=1962035 - https://bugs.launchpad.net/ubuntu/+source/dovecot/+bug/1945763 - https://launchpadlibrarian.net/569387479/dovecot.debdiff The patch looks easy and at least 'make test' pass again. Could the dovecot developer have a look at the patch and say something about the roadmap to support openssl3? Thanks Andreas

1 0

Malformed XLIST replies
by Bogdan Szczurek 21 Jun '22

21 Jun '22

Hi! Some (considerable) time ago, I, as one of the Mailspring (https://getmailspring.com/) users, had a problem with setting up IMAP email account hosted on home.pl. I was able to narrow down the problem to how the Mailspring handles server response regarding folder list. The problem is twofold: a) how the response is handled by Mailspring's IMAP libraries, b) how the server itself responds. As to the first part of the problem I submitted relevant bug reports (https://github.com/MailCore/mailcore2/issues/1908 and https://github.com/dinhvh/libetpan/issues/403) and but the second part appears to belong here, since home.pl uses Dovecot as their IMAP server. Now about the problem. When XLIST is used to get the list of the folders, I get, for example: 3 XLIST "" "*" * XLIST (\HasNoChildren \) "." "Archive" * XLIST (\HasNoChildren \) "." "Courses" * XLIST (\HasNoChildren \Draft) "." "DRAFTS" * XLIST (\HasNoChildren \Inbox) "." "INBOX" * XLIST (\HasChildren \) "." "PhD" * XLIST (\HasNoChildren \) "." "PhD.Advisor" * XLIST (\HasNoChildren \) "." "PhD.Archive" * XLIST (\HasNoChildren \) "." "PhD.Articles" * XLIST (\HasNoChildren \) "." "PhD.Conferences" * XLIST (\HasNoChildren \) "." "PhD.Journals" * XLIST (\HasNoChildren \) "." "PhD.Thesis" 3 OK Completed Compare this with the LIST output: 4 LIST "" "*" * LIST (\HasNoChildren) "." "Archive" * LIST (\HasNoChildren) "." "Courses" * LIST (\HasNoChildren \Drafts) "." "DRAFTS" * LIST (\HasNoChildren) "." "INBOX" * LIST (\HasChildren) "." "PhD" * LIST (\HasNoChildren) "." "PhD.Advisor" * LIST (\HasNoChildren) "." "PhD.Archive" * LIST (\HasNoChildren) "." "PhD.Articles" * LIST (\HasNoChildren) "." "PhD.Conferences" * LIST (\HasNoChildren) "." "PhD.Journals" * LIST (\HasNoChildren) "." "PhD.Thesis" 4 OK Completed Notice, how in the XLIST example, every item has an additional “empty” flag (“\”). This part causes libetpan to fail. While ignoring the problem as a library's problem would be perfectly understandable, I think it should be also considered if something shouldn't be done on Dovecot's end. Granted, XLIST has been long depreceated, however it still is implemented and it should be implemented properly. Perhaps I'm mistaken, but I failed to find any mention of a thing like “empty flag”. It might not be prohibited, but it strikes me as something bizarre. I am unable to get the information about the Dovecot version used by home.pl, but I suspect this problem may still be present up to the newest version. Also, if someone is willing to take a closer look at the problem, I can provide you with an email account for testing purposes. Best regards! Bogdan

2 1

v2.3.19.1 released
by Aki Tuomi 20 Jun '22

20 Jun '22

Hi everyone! Due to a severe bug in doveadm deduplicate, we are releasing patch release 2.3.19.1. Please find it at locations below: https://dovecot.org/releases/2.3/dovecot-2.3.19.1.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.19.1.tar.gz.sig Binary packages in https://repo.dovecot.org/ Docker images in https://hub.docker.com/r/dovecot/dovecot Aki Tuomi Open-Xchange oy --- - doveadm deduplicate: Non-duplicate mails were deleted. v2.3.19 regression. - auth: Crash would occur when iterating multiple backends. Fixes: Panic: file userdb-blocking.c: line 125 (userdb_blocking_iter_next): assertion failed: (ctx->conn != NULL)

3 7

Move sent emails to sent folder?
by Austin Witmer 20 Jun '22

20 Jun '22

Hello all! I have a server running dovecot & postfix. I have a user on my server who is sending email via smtp on an HP printer and because of that, a copy of the email is not placed in the sent folder like usually happens with clients like outlook and thunderbird. Is there any way to have the sent emails copied to the sent folder for just this user on my server? Thx! Austin Witmer

6 19

Home directories with colons in it
by Jonas Pasche 16 Jun '22

16 Jun '22

Howdy folks, we're using Dovecot 2.3.19 in combination with vmailmgr as the virtual user management backend (please don't judge, there are plans to move away from it for various reasons, but as of today it's there), using the wrapper from https://wiki.dovecot.org/HowTo/VMailMgr - and we're facing an interesting problem with colons in home directories. I think this problem is not specific to vmailmgr but happens in any scenario that includes ":" in directory names. tl;dr: - The checkvpw-dovecot-wrapper from the Dovecot wiki converts ":" in directory names to "::" which works fine with general IMAP use, but at least the autoexpunge plugin and ManageSieve don't handle this escaping and thus break on those users. - When patching the ":"->"::" conversion out of the checkvpw-dovecot-wrapper, Dovecot itself bails on it, interpreting it as a sub-setting. - The latter is quite surprising because a HOME/mail_home directory is supposed to be exactly one directory (AFAIK), with no sub-settings like e.g. mail_location, so I did not expect it to be "interpreted" in any way. Short story long: When you set up a user with a name containing dots, vmailmgr converts them to colons, pretty much like qmail does (qmail maps the username "xy.z" to a file named ".qmail-xy:z"; vmailmgr creates the directory "./users/xy:z" for that username). The wrapper from the dovecot wiki escapes ":" into "::" before exporting the user's directory as the HOME environment variable: /* escape possible ':' characters before finally exporting */ escape_colon(buffer, MAXLEN); setenv("HOME", buffer, 1); The result looks like that: [root@stardust ~]# doveadm auth login xy.z(a)coltest.uber.space supersecret passdb: xy.z(a)coltest.uber.space auth succeeded extra fields: user=xy.z(a)coltest.uber.space userdb extra fields: xy.z(a)coltest.uber.space home=/home/coltest/./users/xy::z <---------- notice the double colon here uid=1009 gid=1010 auth_mech=PLAIN So far, so good: Working with IMAP generally works fine, but there's a catch: Not every module seems prepared for this "::" syntax. The first notable exception is the autoexpunge plugin which simply bails on it: Error: autoexpunge: Couldn't create dovecot.autoexpunge.lock lock: file_create_locked(/home/coltest/./users/xy::z/dovecot.autoexpunge.lock) failed: safe_mkstemp(/home/coltest/./users/xy::z/dovecot.autoexpunge.lock) failed: No such file or directory The second notable exception is ManageSieve which takes the escaped directory name "as is", leading to a situation where the Maildir is in "xy:z" while the Sieve stuff is being put in "xy::z" where it effectively is never picked up during delivery (as the stuff is in the wrong location and the .dovecot.sieve symlink is - silently - not created): [coltest@stardust ~]$ ls -la users/* users/xy:z: total 28 drwx------. 9 coltest coltest 4096 Jun 16 11:39 . drwxr-xr-x. 4 coltest coltest 31 Jun 16 11:40 .. drwxr-xr-x. 2 coltest coltest 6 Jun 16 11:39 cur -rw-------. 1 coltest coltest 376 Jun 16 11:40 dovecot.index.log -rw-------. 1 coltest coltest 2676 Jun 16 11:39 dovecot.list.index.log -rw-------. 1 coltest coltest 96 Jun 16 11:39 dovecot.mailbox.log -rw-------. 1 coltest coltest 51 Jun 16 11:39 dovecot-uidlist -rw-------. 1 coltest coltest 8 Jun 16 11:39 dovecot-uidvalidity -r--r--r--. 1 coltest coltest 0 Jun 16 11:39 dovecot-uidvalidity.62aafa70 drwx------. 5 coltest coltest 108 Jun 16 11:39 .Drafts drwxr-xr-x. 2 coltest coltest 6 Jun 16 11:39 new drwx------. 5 coltest coltest 108 Jun 16 11:39 .Sent drwx------. 5 coltest coltest 108 Jun 16 11:39 .Spam -rw-------. 1 coltest coltest 28 Jun 16 11:39 subscriptions drwxr-xr-x. 2 coltest coltest 6 Jun 16 11:39 tmp drwx------. 5 coltest coltest 108 Jun 16 11:39 .Trash users/xy::z: total 0 drwx------. 3 coltest coltest 19 Jun 16 11:40 . drwxr-xr-x. 4 coltest coltest 31 Jun 16 11:40 .. drwx------. 3 coltest coltest 33 Jun 16 11:40 sieve So my initial thought was: Why is the ":" even escaped to a "::" in the first place? It shouldn't be necessary according to this clean workaround that I found in a posting from 2009 with Timo advising for this type of handling: > Hmm. Although now that I looked at your code, you seem to be using only > HOME. And using ~/ in mail_location should work around any ":" issues. > So what exactly is the problem you're seeing? https://www.dovecot.org/list/dovecot/2009-June/040590.html (In the same thread, Timo _also_ introduced the "::" handling in Dovecot.) This is exactly what we're doing at delivery time: Setting mail_home to the real pathname including a _single_ colon (not an escaped double colon) and then pointing mail_location to "~/": /usr/libexec/dovecot/dovecot-lda -a '$RECIPIENT' -f '$SENDER' -o mail_home='$MAILDIR' -o mail_location='maildir:~/' This works well. So my assumption was: If the checkvpw-dovecot-wrapper returns the HOME directory of a virtual user; where's even the need to escape a single colon to a double colon? So I gave it a try and removed that one line of escaping and recompiled the wrapper. Result: [root@stardust ~]# doveadm auth login xy.z(a)coltest.uber.space supersecret passdb: xy.z(a)coltest.uber.space auth succeeded extra fields: user=xy.z(a)coltest.uber.space userdb extra fields: xy.z(a)coltest.uber.space home=/home/coltest/./users/xy:z <---------- single colon here now uid=1009 gid=1010 auth_mech=PLAIN Using a real IMAP login, though, this lead to the "interesting" situation that the login itself worked correctly, but _then_ immediately an "Internal error occurred": [root@stardust ~]# nc localhost 143 * OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SPECIAL-USE LITERAL+ STARTTLS AUTH=PLAIN] Dovecot ready. 0 LOGIN xy.z(a)coltest.uber.space supersecret 0 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE SPECIAL-USE LITERAL+ NOTIFY SPECIAL-USE] Logged in * BYE Internal error occurred. Refer to server log for more information. This is the according log: Jun 16 10:04:37 stardust.uberspace.de dovecot[15268]: Jun 16 10:04:37 imap-login: Info: Login: user=<xy.z(a)coltest.uber.space>, method=PLAIN, rip=::1, lip=::1, mpid=22889, secured Jun 16 10:04:37 stardust.uberspace.de dovecot[15268]: Jun 16 10:04:37 imap(xy.z(a)coltest.uber.space)<22889><utbow43hgJoAAAAAAAAAAAAAAAAAAAAB>: Error: Namespace '': Unknown setting: z Jun 16 10:04:37 stardust.uberspace.de dovecot[15268]: Jun 16 10:04:37 imap(xy.z(a)coltest.uber.space)<22889><utbow43hgJoAAAAAAAAAAAAAAAAAAAAB>: Info: Disconnected: Namespace '': Unknown setting: z in=0 out=428 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0 So it looks like Dovecot tries to do some interpretation on the HOME value, splitting it at the ":" and interpret the "z" as some kind of sub-setting - which a HOME variable (in contrast to, e.g. mail_location) is not supposed to have. Here I'm a bit stuck. My idea is to work around this by eliminating the ":" from directory names altogether by passing an explicit directory name to vadduser: [coltest@stardust ~]$ echo supersecret | vadduser --directory=./users/ab.c ab.c vadduser: user 'ab.c' successfully added [coltest@stardust ~]$ ls -l users total 0 drwx------. 5 coltest coltest 39 Jun 16 10:26 ab.c drwx------. 5 coltest coltest 39 Jun 16 09:31 xy:z But still, it would be nicer if Dovecot cleanly supported the ":" in directory names, as it seems to _want_ to support it (by introducing the "::" escaping) but then only does it halfway. Could this be fixed in whatever direction, or is there any other advice how we should handle this? All the best, Jonas

1 0

Re: [EXT] Re: The meaning of "tarpit" as used in the auth policy docs?
by Hippo Man 15 Jun '22

15 Jun '22

I'm sorry. Gmail didn't include the list in the "To" header when I responded, and I didn't notice. I fixed it now. -- hippoman(a)gmail.com Take a hippopotamus to lunch today. On Tue, Jun 14, 2022 at 2:40 AM Aki Tuomi <aki.tuomi(a)open-xchange.com> wrote: > Hi! > > Can you please keep responses on the list? =) > > Aki > > > On 14/06/2022 03:57 Hippo Man <hippoman(a)gmail.com> wrote: > > > > > > Thank you very much for explaining what "tarpit" means. > > > > So, if I return a huge numeric status value from the "before" check, > could I > > make the client wait for hours or even days or weeks before continuing > > with the authentication? > > > > -- > > hippoman(a)gmail.com > > Take a hippopotamus to lunch today. > > > > > > > > On Wed, Jun 8, 2022 at 12:50 AM Aki Tuomi <aki.tuomi(a)open-xchange.com> > wrote: > > > > > > > On 08/06/2022 05:34 Hippo Man <hippoman(a)gmail.com> wrote: > > > > > > > > > > > > I'm trying to understand what is meant by "tarpit" in the > ffollowing quote of the Dovecot auth policy docs for the "before" check ... > > > > First query is done before password and user databases are > consulted. This means that any userdb/passdb attributes are left empty. > > > > The command used here is ‘allow’ and will appear on the URL as > command=allow. > > > > status result values: > > > > * -1: Reject > > > > * 0: Accept > > > > * (Any other positive value): Tarpit for this number of seconds. > > > > What actually happens if a positive status value is returned here? > > > > > > > > Does it mean that the client will not receive any response for the > given number of seconds, but that the given login attempt will still > continue after that delay? Or does it mean that the client will not receive > a response for the given number of seconds, and then the login will be > rejected after that delay? > > > > > > > > What if the client has connected via pop3 and is sending multple > "user" and "pass" combinations? If the "before" check returns, for example, > the integer "5" as the status, does this mean there will be a 5-second > delay between each login attempt that is being sent to this connection? > > > > > > > > Ideally, I'd like the following to occur, but I don't know if it is > possible: > > > > > > > > * Client connects via pop3 and intends to send a large batch of > "user"/"pass" commands via this connection. > > > > > > > > * The first "user"/"pass" command is made to wait for many seconds. > > > > > > > > * Then, the connection is dropped, so that the subsequent > "user"/"pass" commands do not even hit the dovecot server. > > > > > > > > I'm thinking that this is *not* what is meant by "tarpit" in the > auth policy docs, correct? > > > > > > > > Thank you for any clarification. > > > > > > > > -- > > > > hippoman(a)gmail.com > > > > Take a hippopotamus to lunch today. > > > > > > Hi! > > > > > > When client connects, and sends authentication request dovecot > performs a policy lookup before any actual authentication. If this policy > result has positive value, tarpit occurs. > > > > > > This means we simply add a timeout to auth process and leave it > hanging, while handling other requests, and then perform the actual > authentication. The authentication is not failed, however. > > > > > > Aki > > > >

1 0

Re: doveadm-deduplicate deletes non-duplicates
by gravitini 14 Jun '22

14 Jun '22

Replying to: https://dovecot.org/pipermail/dovecot/2022-May/124816.html Hi, Looking at the code (and tested via local build from source) it looks like doveadm deduplicate in 2.3.19 can cause significant data loss. A 2022-02-11 commit removed key duplication resulting in undefined behaviour which is often truncation of a mailbox to 67 entries. (HASH_TABLE_MIN_SIZE) https://github.com/dovecot/core/commit/320844f50cd669b602d30210e2e5216f65d2… diff --git a/src/doveadm/doveadm-mail-deduplicate.c b/src/doveadm/doveadm-mail-deduplicate.c index caec758112..2152482876 100644 --- a/src/doveadm/doveadm-mail-deduplicate.c +++ b/src/doveadm/doveadm-mail-deduplicate.c @@ -63,8 +63,10 @@ cmd_deduplicate_box(struct doveadm_mail_cmd_context *_ctx, if (key != NULL && *key != '\0') { if (hash_table_lookup(hash, key) != NULL) mail_expunge(mail); - else + else { + key = p_strdup(pool, key); hash_table_insert(hash, key, POINTER_CAST(1)); + } } }

4 8

Re: dovecot Digest, Vol 230, Issue 21
by Joseph Tam 10 Jun '22

10 Jun '22

On Thu, 9 Jun 2022, Richard wrote: > Rather than simply upping the limit I think a reasonable question to > ask is why/how they are managing to do that. That's a lot of open > folders. If this mail client behaves anything like Apple mail clients, these connection storms can come about when doing global searches. The mail clients will march through each mailbox (opening a connection for each mailbox) looking for a pattern. If you can narrow this scenario for one specific user (e.g. $user), you can deep dive what's going on by enabling IMAP session logs for this user protocol imap { ... rawlog_dir = /log/dir/%u } then (Make sure this user has write permissions into this directory) mkdir /log/dir/$user After you're done, you can disable logging, rm -rf /log/dir/$user Joseph Tam <jtam.home(a)gmail.com>

1 0

Re: Issue with one user only, exceeding connections
by Richard 09 Jun '22

09 Jun '22

> Date: Thursday, June 09, 2022 11:07:38 -0500 > From: Jeremy Schaeffer <kb9mfd(a)phonesplus.biz> > > On 6/9/2022 10:59, Richard wrote: >> >>> Date: Thursday, June 09, 2022 10:46:25 -0500 >>> From: Jeremy Schaeffer <kb9mfd(a)phonesplus.biz> >>> >>> That was the first thing I tried, I lowered the cache connections >>> in Thunderbird. Actually the max connections was 50, not 500, but >>> I could see why as I do have a lot of folders, but what is odd is >>> I have other mailboxes that have even more folders, but it's only >>> one mailbox that is trowing the error. >>> >>> "# ps -axww | grep imap" does not give me the same results - >>> >>> ..... >>> >>> 19897 ? S 0:00 dovecot/imap >>> 19900 ? S 0:02 dovecot/imap >>> 19901 ? S 0:00 dovecot/imap >>> 19902 ? S 0:00 dovecot/imap >>> ..... >>> >>> I wish it did give me the mailbox, is there a option to get it to >>> give me that information? >> >> Try "auxw" on your "ps". I.e., add in the "u" which will get you >> the user detail in the first column, otherwise you just get the >> process id. >> >> > Thank you! That worked, I piped the output to a file, grep the > username and sure enough there are 60 lines. So I guess going over > 50 was a possibility. > > Learn something new every day. I set the maximum to 100 so I should > not have any errors on that anymore. > Rather than simply upping the limit I think a reasonable question to ask is why/how they are managing to do that. That's a lot of open folders. By the way, the single command: ps auxw | grep imap | cut -d" " -f1 | sort | uniq -c will get you a nice list with the users and their connection counts.

4 4

Issue with one user only, exceeding connections
by Jeremy Schaeffer 09 Jun '22

09 Jun '22

I keep having this issue with one user, and I have to restart dovecot several times a day to clear it. What I have is a postfix / dovecot mail server (Centos 7) and about a dozen users. All mailboxes are imap ssl. I monitor about 4 mailboxes on my computer and tablet. I use Thunderbird on the computer (cache connections at 2) and K9 on the tablet, but one user of the four I keep getting "Maximum number of connections from user+IP exceeded" and I have the maximum at 50 "(mail_max_userip_connections=50)" so its hard for me to believe I am actually exceeding it unless dovecot/client is not dropping connections and keeps starting new ones until it reaches the maximum, but again, only for one user, even though I am monitoring 4 on the same devices. Any idea how to troubleshoot this? I don't know if I should be looking at dovecot or the clients, or what I need to look for. It's been going on since I put this server in use over a year ago. I also have issues with Outlook clients disconnecting, just outlook, is there any recommended settings to make Outlook work smoother? Thanks! - Jeremy Config - # 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-1160.11.1.el7.x86_64 x86_64 CentOS Linux release 7.9.2009 (Core) # Hostname: *** auth_mechanisms = plain login debug_log_path = /var/log/dovecot_debug.log first_valid_gid = 500 last_valid_gid = 600 last_valid_uid = 600 listen = * mail_location = maildir:~/Maildir mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = INBOX. separator = . } passdb { driver = pam } pop3_uidl_format = %f protocols = imap lmtp service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } process_min_avail = 1 service_count = 0 } service imap { process_limit = 1024 } service lmtp { unix_listener lmtp { mode = 0666 } } ssl = required ssl_cert = <*** ssl_cipher_list = ECDHE-RSA-CHACHA20-POLY1305:ALL:!LOW:!SSLv2:!EXP:!aNULL ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { driver = passwd } protocol imap { mail_max_userip_connections = 50 }

4 6