March 2024 - dovecot - dovecot.org

Separate index get dovecot lost
by Joan Moreau 30 Mar '24

30 Mar '24

Hi I have large number of email (~TB) and want to put the index in a separate, rapid drive Initially, I have mail_location = mdbox:/files/mail/%d/%n If I put mail_location = mdbox:/files/mail/%d/%n:INDEX=/data/mailindexes/%d/%n then dovecot gets totally lost and tries to reach mailboxes content and tree from the INDEX location instead of the original location What is wrong ? Thank you Hi I have large number of email (~TB) and want to put the index in a separate, rapid drive Initially, I have mail_location = mdbox:/files/mail/%d/%n If I put mail_location = mdbox:/files/mail/%d/%n:INDEX=/data/mailindexes/%d/%n then dovecot gets totally lost and tries to reach mailboxes content and tree from the INDEX location instead of the original location What is wrong ? Thank you

2 4

replication timeout
by Andy Balholm 28 Mar '24

28 Mar '24

I have two Dovecot mail servers that replicate to each other. Sometimes there are delays in the synchronization, and I notice that the mail log has entries like this: Error: dsync(spokane): I/O has stalled, no activity for 600 seconds (last sent=mailbox, last recv=mailbox_state) Five minutes seems like a long time to sit there waiting with nothing happening. Is there a way to reduce this timeout so that I don't have so many replicaton connections just sitting around doing nothing? (Of course, a way to prevent the I/O stalls would be great too, but with my limited upload bandwidth, they may be unavoidable.) Andy

2 9

panics
by mgrant＠grant.org 28 Mar '24

28 Mar '24

I am still getting panics. Today I'm seeing this panic only when 'eM Client' connects. Not other imap clients. It's almost as if there's a malformed message in my mailbox which might be causing this. I spent some time trying to do things like delete the last 20 messages or so but no luck. I tried to enable debugging in 10-logging but i get nothing at all in the log file /var/log/dovecot. Yes I've restarted dovecot. I'm running the version from here: https://repo.dovecot.org/ce-2.3-latest/debian/bullseye Any ideas?? Mar 26 19:23:14 strange dovecot: imap(mgrant)<2743779><EQ/3kpgUgvxRaQEP>: Panic: file istream-header-filter.c: line 663 (i_stream_header_filter_snapshot_free): assertion failed: (snapshot->mstream->snapshot_pending) Mar 26 19:23:14 strange dovecot: imap(mgrant)<2743779><EQ/3kpgUgvxRaQEP>: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42) [0x7f9d301ed482] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f9d301ed59e] -> /usr/lib/dovecot/libdovecot.so.0(+0x1041fb) [0x7f9d301fa1fb] -> /usr/lib/dovecot/libdovecot.so.0(+0x104291) [0x7f9d301fa291] -> /usr/lib/dovecot/libdovecot.so.0(+0x56670) [0x7f9d3014c670] -> /usr/lib/dovecot/libdovecot.so.0(+0x51ec5) [0x7f9d30147ec5] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_snapshot_free+0x18) [0x7f9d30205c68] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_unref+0x4c) [0x7f9d30205d0c] -> /usr/lib/dovecot/libdovecot.so.0(message_parse_header_deinit+0x15) [0x7f9d301d60e5] -> /usr/lib/dovecot/libdovecot.so.0(+0xe29b1) [0x7f9d301d89b1] -> /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_next_block+0x45) [0x7f9d301d84d5] -> /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_header+0x51) [0x7f9d301d8611] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_parse_headers_internal+0x11a) [0x7f9d3039c8ca] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_parse_headers+0x4c) [0x7f9d3039c9ec] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xd885a) [0x7f9d303a185a] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xd8a89) [0x7f9d303a1a89] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_get_special+0x122) [0x7f9d303a1d92] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_special+0xa) [0x7f9d3031cfba] -> dovecot/imap(+0x26808) [0x55f5622cd808] -> dovecot/imap(+0x240a4) [0x55f5622cb0a4] -> dovecot/imap(imap_fetch_more+0x39) [0x55f5622cc4a9] -> dovecot/imap(cmd_fetch+0x39f) [0x55f5622bc8af] -> dovecot/imap(command_exec+0xa4) [0x55f5622c95b4] -> dovecot/imap(+0x2052f) [0x55f5622c752f] -> dovecot/imap(+0x205da) [0x55f5622c75da] -> dovecot/imap(+0x208ed) [0x55f5622c78ed] -> dovecot/imap(client_handle_input+0x1b5) [0x55f5622c7ab5] -> dovecot/imap(client_input+0x70) [0x55f5622c8020] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x7f9d30210429] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x132) [0x7f9d30211b12] Mar 26 19:23:14 strange dovecot: imap(mgrant)<2743779><EQ/3kpgUgvxRaQEP>: Fatal: master: service(imap): child 2743779 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps)

4 5

SAVEDATE is using ctime instead of btime for Maildir
by Alex 28 Mar '24

28 Mar '24

Dovecot supports SAVEDATE IMAP extension [1]. This is however not implemented correctly for Maildir storage. Compared to "internal date", "save date" is supposed to be the same but well-defined for APPEND and preserved when COPY command is used. However, for Maildir storage INTERNALDATE is implemented using mtime (modification timestamp) and SAVEDATE is implemented using ctime (status change timestamp) [2]. ctime is changed even more often than mtime. For example, chown or chmod operations on the message change its ctime, but preserve its mtime. Dovecot should instead use btime (creation timestamp) when it is available and not implement SAVEDATE at all if there is no btime. Just one example of the message in Maildir: stat 1708031310.M769290P3712350.nine,S=7898,W=8002:2,Sa File: 1708031310.M769290P3712350.nine,S=7898,W=8002:2,Sa Size: 7898 Blocks: 16 IO Block: 4096 regular file Device: 9,2 Inode: 7864351 Links: 1 Access: (0600/-rw-------) Uid: ( 999/ vmail) Gid: ( 996/ vmail) Access: 2024-02-16 08:34:04.374097798 +0100 Modify: 2024-02-15 22:08:30.769813368 +0100 Change: 2024-02-21 13:39:14.878163244 +0100 Birth: 2024-02-15 22:08:30.769813368 +0100 Its Birth date (btime) and Modify date (mtime) are the same, but Change date (ctime) was changed, likely by someone doing chown/chmod manually. You can read more in the issue [3] where we have problems with incorrect INTERNALDATE and investigated the possibility of using SAVEDATE, but found that it is not implemented correctly and is changed even more often than INTERNALDATE. According to the standard it should be the other way round, SAVEDATE should be more stable and survive COPY. [1] <https://www.rfc-editor.org/rfc/rfc8514.html> [2] <https://github.com/dovecot/core/blob/93eb0663336b17f66a67324de43e64bf0bc796…> [3] <https://github.com/deltachat/deltachat-core-rust/issues/5339>

2 1

Post-Auth external check?
by Stephan von Krawczynski 28 Mar '24

28 Mar '24

Hello all, is it possible to include some post-auth check in the password authentication? So after dovecot has found a user allowed to login to execute some external script for checking additional conditions which gives back simple true or false wether the login should be allowed? -- Regards, Stephan

3 2

Re: [Dovecot] Re: Using mbox INBOX with Maildirs?
by Will 26 Mar '24

26 Mar '24

Hi, This is my first time using this group and I have the same question as the subject. I can't seem to get dovecot to read a mbox inbox that is in /var/mail/%u with /home/%u/Maildir as maildir. This just worked on my old server that I am now upgrading to AlmaLinux release 9.3 with dovecot version 2.3.16 (7e2e900c1a). Please help, this upgrade is very painful. Guess I shouldn't have waited 20 years...

3 3

How can I override settings in a dovecot protocol section?
by Christian Rößner 25 Mar '24

25 Mar '24

Hi, I have defined a service and a protocol that shall honor quota, but in an non-enforcing mode (required). Therefr I have done this on a test-server: ``` protocols = lmtp lmtp_noenforcing imap pop3 service lmtp_noenforcing { executable = lmtp inet_listener lmtp_noenforcing { port = 24000 } protocol = lmtp_noenforcing } protocol lmtp_noenforcing { postmaster_address = postmaster@**** mail_plugins = quota sieve fts fts_solr # Just for testing, if this works. userdb { driver = ldap args = /etc/dovecot/dovecot-ldap.conf.ext override_fields = quota=count:User quota:noenforcing skip = never result_success = continue result_failure = return-fail } # Just for testing, if this works. userdb { driver = lua args = file=/etc/dovecot/userdb.lua blocking=yes override_fields = quota=count:User quota:noenforcing result_success = return-ok result_failure = continue result_internalfail = continue } # Preferred plugin { quota = count:User quota:noenforcing } } ``` I tried with and without userdbs. With and without plugin{}. It seems that this whole section is never used. I removed "quota" from the mail_plugins parameter, but that seems to show me that the section is not used, because it always fails in a local swaks-test. The test shows that quota is exceeded. I would expect that the quota is ignored. One or the other hand. So why is the protocol not used for this service? We currently use Dovecot 2.3.20 from your repos. Kind regards and thanks in advance Christian Rößner -- Rößner-Network-Solutions Zertifizierter ITSiBe / CISO Karl-Bröger-Str. 10, 36304 Alsfeld Fax: +49 6631 78823409, Mobil: +49 171 9905345 USt-IdNr.: DE225643613, https://roessner.website PGP fingerprint: 658D 1342 B762 F484 2DDF 1E88 38A5 4346 D727 94E5

1 1

virtual user list in a passwd file (without linux user)
by paul＠virtuadv.com 23 Mar '24

23 Mar '24

Hi, I am trying to have a virtual user list in a passwd file (without linux user) but it does not seem to work. I have been trying to make this work for days reading the dovecot documentation and whatever I could find online... Note that I did not enable imap since I only use pop3s to retrieve my emails. I also tried to follow this guide: https://doc.dovecot.org/configuration_manual/howto/simple_virtual_install/ Note that once this works for dovecot, I plan to have postfix use the dovecot auth service to authenticate the virtual users and validate mail boxes. # cat /proc/version Linux version 6.1.0-17-cloud-amd64 (debian-kernel(a)lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) # dovecot version Feb 15 15:57:26 master: Info: Dovecot v2.3.19.1 (9b53102964) starting up for pop3, submission, lmtp # doveconf -n # 2.3.19.1 (9b53102964): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.19 (4eae2f79) # OS: Linux 6.1.0-17-cloud-amd64 x86_64 Debian 12.5 # Hostname: 85e10e78339f auth_verbose = yes debug_log_path = /dev/stdout first_valid_uid = 1000 info_log_path = /dev/stdout last_valid_uid = 1000 listen = * log_path = /dev/stdout login_greeting = greeting. mail_debug = yes mail_gid = 1000 mail_home = /srv/vmail/%d/%n mail_location = maildir:~/mail mail_uid = 1000 namespace { inbox = yes location = prefix = separator = / } passdb { args = username_format=%u /path/to/passwd driver = passwd-file } pop3_uidl_format = %g protocols = pop3 submission lmtp service auth { inet_listener { address = <ip> port = 12345 } user = vmail } service lmtp { inet_listener lmtp { address = <ip> port = 12344 } user = vmail } service pop3-login { client_limit = 1000 process_min_avail = 1 service_count = 0 } service submission-login { client_limit = 1000 process_min_avail = 1 service_count = 0 } ssl_cert = <cert.pem ssl_key = # hidden, use -P to show it userdb { args = username_format=%u /path/to/passwd default_fields = uid=vmail gid=vmail home=/srv/vmail/%d/%n driver = passwd-file } verbose_proctitle = yes local_name site1.com { ssl_cert = </etc/apache2/md-ssl/domains/site1.com/pubcert.pem ssl_key = # hidden, use -P to show it } local_name www.site1.com { ssl_cert = </etc/apache2/md-ssl/domains/site1.com/pubcert.pem ssl_key = # hidden, use -P to show it } The passwd file: # user:password:uid:gid:(gecos):home:(shell):extra_fields user1(a)site1.com{PLAIN}password1:::::: user2(a)site2.com{PLAIN}password2:::::: This is the log I am getting when trying to create an account with Thunderbird: Feb 15 15:57:26 master: Info: Dovecot v2.3.19.1 (9b53102964) starting up for pop3, submission, lmtp Feb 15 17:34:45 pop3-login: Info: Disconnected: Aborted login by logging out (no auth attempts in 0 secs): user=<>, rip=<ip>, lip=<ip>, TLS, session=<cMIeC28RVOImVwz3> Feb 15 17:35:02 auth: Info: passwd-file(user1(a)site1.com,<ip>,<AIAjDG8RVuImVwz3>): unknown user Feb 15 17:35:08 auth: Info: passwd-file(user1(a)site1.com,<ip>,<AIAjDG8RVuImVwz3>): unknown user Feb 15 17:35:10 pop3-login: Info: Disconnected: Aborted login by logging out (auth failed, 2 attempts in 8 secs): user=<user1(a)site1.com>, method=PLAIN, rip=<ip>, lip=<ip>, TLS, session=<AIAjDG8RVuImVwz3> I tried using doveadm: # doveadm user user1 userdb lookup: user user1 doesn't exist field value # doveadm user user1(a)site1.com userdb lookup: user user1(a)site1.com doesn't exist field value I also tried with the verbose on (removed all the 'usually intentional'): # doveadm -Dv user user1(a)site1.com Feb 15 17:38:17 doveadm(user1(a)site1.com)<137><>: Debug: auth-master: userdb lookup(user1(a)site1.com): Started userdb lookup Feb 15 17:38:17 doveadm(user1(a)site1.com)<137><>: Debug: auth-master: conn unix:/run/dovecot/auth-userdb: Connecting Feb 15 17:38:17 doveadm(user1(a)site1.com)<137><>: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=7,uid=0): Client connected (fd=9) Feb 15 17:38:17 doveadm(user1(a)site1.com)<137><>: Debug: auth-master: userdb lookup(user1(a)site1.com): auth USER input: Feb 15 17:38:17 doveadm(user1(a)site1.com)<137><>: Debug: auth-master: userdb lookup(user1(a)site1.com): Userdb lookup failed userdb lookup: user user1(a)site1.com doesn't exist Feb 15 17:38:17 Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=7,uid=0): Disconnected: Connection closed (fd=9) field value I could not find a way to list either the mailboxes or virtual user list that dovecot is using/serving.

4 17

Debug on FreeBSD: tries to open linux style /proc
by Larry Rosenman 22 Mar '24

22 Mar '24

Aki, Can you comment on this FreeBSD bug I got? https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=274540 Either here on in the bugzilla bug. Thanks! -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106 Aki, Can you comment on this FreeBSD bug I got? https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=274540 Either here on in the bugzilla bug. Thanks! -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

1 0

Thousands of dovecot/dict processes
by Alex 21 Mar '24

21 Mar '24

Hi, I have a fedora38 system with dovecot-2.3.21 for a few hundred users and just noticed there are 4000 dovecot/dict processes running: 2203938 ? S 0:15 dovecot/dict or sleeping, as the case may be. Some time ago I installed the last_login_dict plugin to record login details per user. Here is my config: plugin { last_login_dict = proxy::sql last_login_key = last-login/%{service}/%{user}/%{remote_ip} last_login_precision = ms } dict { sql = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } There are about 1100 total entries in the last_login table, most of which are localhost for webmail. Is this all to be expected? I don't know how much in terms of resources these 4000 processes are utilizing, and although the system is virtually idle, it does appear to be going into swap. # free -h total used free shared buff/cache available Mem: 62Gi 9.3Gi 2.5Gi 13Mi 51Gi 52Gi Swap: 15Gi 4.4Gi 11Gi Any ideas greatly appreciated. Thanks, Alex Hi, I have a fedora38 system with dovecot-2.3.21 for a few hundred users and just noticed there are 4000 dovecot/dict processes running: 2203938 ? S 0:15 dovecot/dict or sleeping, as the case may be. Some time ago I installed the last_login_dict plugin to record login details per user. Here is my config: plugin { last_login_dict = proxy::sql last_login_key = last-login/%{service}/%{user}/%{remote_ip} last_login_precision = ms } dict { sql = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } There are about 1100 total entries in the last_login table, most of which are localhost for webmail. Is this all to be expected? I don't know how much in terms of resources these 4000 processes are utilizing, and although the system is virtually idle, it does appear to be going into swap. # free -h total used free shared buff/cache available Mem: 62Gi 9.3Gi 2.5Gi 13Mi 51Gi 52Gi Swap: 15Gi 4.4Gi 11Gi Any ideas greatly appreciated. Thanks, Alex

1 0