October 2022 - dovecot

dovecot-fts-solr Solr9 support
by PGNet Dev 24 Apr '23

24 Apr '23

I run dovecot-2.3.18 , & am deploying Solr search with 'dovecot-fts-solr' I run Solr standalone. When creating a dedicated 'dovecot' core, cp'ing from Dovecot's included solr schemas ls -al /usr/share/doc/dovecot/solr-* -rw-r--r-- 1 root root 12K Dec 3 06:48 /usr/share/doc/dovecot/solr-config-7.7.0.xml -rw-r--r-- 1 root root 2.7K Dec 3 06:48 /usr/share/doc/dovecot/solr-schema-7.7.0.xml -rw-r--r-- 1 root root 2.9K Dec 3 06:48 /usr/share/doc/dovecot/solr-schema.xml is tested only for Solr 7.7.0 https://doc.dovecot.org/configuration_manual/fts/solr/ "The steps described in this page are tested for Solr 7.7.0. For other versions, this these steps may need to be adjusted." which, per solr docs https://solr.apache.org/downloads.html About versions and support Version Description 9.x Current major version for feature releases (STABLE) 8.11.x Previous major version may sometimes receive critical bugfix releases 10 Next major version, yet to be released (UNSTABLE) <8.11 All older versions are End Of Life (EOL) is EOL'd and unsupported. In the dovecot 7.7.0 schema, FastLRUCache/LRUCache are used as the caching class(es), grep class=.*LRU /usr/share/doc/dovecot/solr-config-7.7.0.xml <filterCache class="solr.FastLRUCache" <queryResultCache class="solr.LRUCache" <documentCache class="solr.LRUCache" class="solr.search.LRUCache" For Solr 9, cat CHANGES.txt Upgrade Notes --------------------- ... * SOLR-13817: Deprecate legacy SolrCache implementations. Users are encouraged to transition their configurations to use org.apache.solr.search.CaffeineCache instead. (ab) running solr9 + dovecot-fts-solr search, based on the 7.7.0 schemas, results in errors, failed scan & search. Modifying the config to use CaffeineCache cp solr-config-7.7.0.xml solr-config-9.0.0.xml perl -pi -e '\ s|solr.FastLRUCache|solr.CaffeineCache|g; \ s|solr.LRUCache|solr.CaffeineCache|g; \ s|solr.search.LRUCache|solr.search.CaffeineCache|g;' \ solr-config-9.0.0.xml appears to do the trick. Atm, indexing & search -- both manually triggered, and on delivery receipt to Dovecot store -- is working with no errors (yet). I've not tested exhaustively yet, & don't know if this Cache class change in the schema is both necessary & sufficient for dovecot-fts-solr search use with solr 9.0.0 Given that for Solr "<8.11 All older versions are End Of Life (EOL)" it'd be useful for Dovecot to 'retest' for current, not-EOL'd Solr, and publish/package compatible schemas accordingly.

5 8

mbox to maildir migration, same server?
by kjohnson＠eclypse.org 20 Apr '23

20 Apr '23

What is the recommended procedure to migrate from mbox to maildir on a server already running dovecot? Ideally, this procedure would convert messages in an existing mbox INBOX and in ~/mail mboxes (as defined in my dovecot configuration) to storage in a ~/Maildir. Some of the INBOXs are rather large -- 100's of MB (one of the reasons for migrating). What I have tried has not worked for me. Below, I describe what I tried, how I tested before I tried, what went wrong, and the current setup (pre-migration). What I tried: ============= stop exim4 service dovecot stop Make dovecot listen only locally, not to external connections. /etc/dovecot/local.conf protocol imap { listen = 127.0.0.1:143 ssl_listen = 127.0.0.1:993 } protocol pop3 { listen = 127.0.0.1:110 ssl_listen = 127.0.0.1:995 } Modify dovecot configuration for maildir: Watch capitalization! #in 10-mail.conf: mail_location = maildir:/home/%u/Maildir namespace inbox { separator = / } dovecot -n, to confirm results and perform limited syntax checking. service dovecot start for each user, in a bash script: dsync -v -u "$U" mirror "mbox:/home/$U/mail:INBOX=/var/mail/$U" (if the conversion worked for all users, then reconfigure exim to use maildir and restart it; reconfigure dovecot to listen to external interfaces and restart.) How I tested: ============= Before trying this on the real server, I set up a matching (same versions of Debian, Exim, and Dovecot) VM using mbox, with a couple of accounts. One using POP3, one using IMAP. The IMAP account was accessed from Claws and from Outlook. The test accounts had perhaps a dozen messages on the server. I tried the conversion procedure, and everything I tried worked flawlessly. (Yes, next time I will move some real inbox files from the live server). What went wrong: ================ I encountered two types of problems here. The first was several dsync warnings (username changed): dsync(example): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID 21bfa11a81f11763ac150000d09efc50 (maybe it was just deleted?) As I understand it, this can result in some or all messages being downloaded again by clients. My understanding may be wrong. I also don't understand how the mailbox could have changed -- exim4 was stopped, dovecot is only listening on 127.0.0.1, and I am the only user who uses a local mail client on that system, and this was not my account. The second problem I encountered seems to match Debian Bug report logs - #1011238. In part of the discussion of that report, the reporter mentions adding -1 to the command, as shown: dsync -v -u "$U" mirror -1 "mbox:/home/$U/mail:INBOX=/var/mail/$U" While this seemed to eliminate the dsync panic of #1011238, as far as I can tell, no old messages were actually moved from the old mbox files to the new maildirs. At that point I restored the backup mbox files and undid the changes to dovecot and exim4. Current setup: ============== Debian 11.4 Exim 4.94.2-7 dovecot 1:2.3.13+dfsg1-2 The current setup with Dovecot was a successful migration from IMAP-UW in December 2020. Output from dovecot -n (the list of trusted networks has been removed) # 2.3.13 (89f716dc2): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.13 (cdd19fe3) # OS: Linux 5.10.0-16-686-pae i686 Debian 11.4 # Hostname: localhost auth_verbose = yes mail_location = mbox:~/mail:INBOX=/var/mail/%u mail_privileged_group = mail namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } pop3_uidl_format = %08Xv%08Xu protocols = " imap pop3" ssl_cert = </etc/dovecot/dovecot.pem ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = passwd } Thank you to anyone who read this far. Ken

4 4

decrypt.rb
by Dave 11 Apr '23

11 Apr '23

Per the Dovecot site here: https://wiki.dovecot.org/Plugins/MailCrypt ... the "decrypt.rb" ruby script can be used to decrypt a Dovecot-encrypted message file from the command line. The script sort of runs successfully for me, in the sense that it doesn't error out, but it doesn't show the decrypted message. I've called it like so: decrypt.rb -k <PATH_TO_DOVECOT_KEYFILE> -f ./encrypted_message But when I do it just displays the info below and exits (I've subbed <DATA> tags for the actual data): ------------------[OUTPUT]----------------------- Key(s) (total: 1) - Key type : EC - Key digest: <DATA1> - Peer key : <DATA2> - Encrypted : <DATA3> - Kd hash : <DATA4> ------------------------------------------------------- I've tried specifying an output file as well, per the script's command line options, but the output file is 0 bytes. Does anyone have any suggestions? I *think* I'm using it the way it's intended to be used, but maybe I'm not?! -Dave

5 14

ot: how to t/s TBird problems ?
by Voytek Eymont 22 Mar '23

22 Mar '23

I have a Dovecot/Postfix/MariaDB on a Centos, just have a user ask me: ------------------ I recently upgraded my Thunderbird email client and have experienced problems since. It appears that when Tbird polls for new messages it gets held up waiting for a response from the server I'm using POP port 995. Any ideas as to why I'm having a problem ? ------------------- how to investigate such issue ? CentOS release 6.10 (Final) # dovecot --version 2.3.11.3 (502c39af9) load average on server 0.05/0.01 user log shows like: Oct 12 10:08:50 pop3-login: Info: Login: user=<user(a)tld.com.au>, method=PLAIN, rip=49.111.222.333, lip=103.xxx.yyy.zzz, mpid=17371, TLS, session=<E83xW8rqDPkxw3EW> Oct 12 10:09:34 pop3(user(a)tld.com.au)<17371><E83xW8rqDPkxw3EW>: Info: Disconnected: Logged out top=0/0, retr=0/0, del=0/114194, size=10616009472 Oct 12 10:14:36 imap-login: Info: Login: user=<user(a)tld.com.au>, method=PLAIN, rip=49.111.222.333, lip=103.xxx.yyy.zzz, mpid=17460, TLS, session=<ftiLcMrqR/gxw3EW> Oct 12 10:14:37 imap(user(a)tld.com.au)<17460><ftiLcMrqR/gxw3EW>: Info: Logged out in=152 out=69724 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=1 body_bytes=68628 Oct 12 10:28:27 pop3-login: Info: Login: user=<user(a)tld.com.au>, method=PLAIN, rip=49.111.222.333, lip=103.xxx.yyy.zzz, mpid=17714, TLS, session=<+WsXosrqK/gxw3EW> Oct 12 10:29:16 pop3(user(a)tld.com.au)<17714><+WsXosrqK/gxw3EW>: Info: Disconnected: Logged out top=0/0, retr=1/76174, del=0/114195, size=10616085600 Oct 12 10:32:19 imap-login: Info: Login: user=<user(a)tld.com.au>, method=PLAIN, rip=49.111.222.333, lip=103.xxx.yyy.zzz, mpid=17817, TLS, session=<bZftr8rqq/gxw3EW> Oct 12 10:32:20 imap(user(a)tld.com.au)<17817><bZftr8rqq/gxw3EW>: Info: Logged out in=153 out=135592 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=1 body_bytes=134503 Oct 12 10:45:58 imap-login: Info: Login: user=<user(a)tld.com.au>, method=PLAIN, rip=49.111.222.333, lip=103.xxx.yyy.zzz, mpid=18081, TLS, session=<U3674Mrqzfgxw3EW> Oct 12 10:45:59 imap(user(a)tld.com.au)<18081><U3674Mrqzfgxw3EW>: Info: Logged out in=152 out=24602 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=1 body_bytes=23514 Oct 12 10:48:49 pop3-login: Info: Login: user=<user(a)tld.com.au>, method=PLAIN, rip=49.111.222.333, lip=103.xxx.yyy.zzz, mpid=18135, TLS, session=<y1/r6srqtPsxw3EW> Oct 12 10:49:53 pop3(user(a)tld.com.au)<18135><y1/r6srqtPsxw3EW>: Info: Disconnected: Logged out top=0/0, retr=2/166190, del=0/114197, size=10616251743 Oct 12 10:50:42 imap(user(a)tld.com.au)<3274><cr10yqTqifgxw3EW>: Info: Logged out in=192516 out=5957294 deleted=0 expunged=0 trashed=0 hdr_count=263 hdr_bytes=358656 body_count=126 body_bytes=5110403 Oct 12 10:50:42 imap(user(a)tld.com.au)<3271><hLl0yqTqh/gxw3EW>: Info: Logged out in=78324 out=58677 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0 Voytek

4 3

2.3.14: stale imap-login processes due to bogus user setup tying up process limit
by Volker Stolz 16 Dec '22

16 Dec '22

I just dealt with an interesting Dovecot issue on our small server, and thought I might share it. I also feel that although in the end it was a configuration error, Dovecot should somehow avoid leaving this processes around and hence filling up the process limit quickly. - set up Dovecot IMAP on Debian buster (either with the older 2.3.x Debian package, or 2.3.14 from the dovecot-repo) - config: ... namespace inbox { inbox = yes location = ... default settings ... ... passdb { args = scheme=HMAC-MD5 username_format=%u /etc/dovecot/users driver = passwd-file } protocols = " imap" ... userdb { args = blocking=no driver = passwd } ... - secret sauce: have a user in the passdb that doesn't have a physical account on the system yet (we were migrating to a new machine). The log-file shows successful authentication for that user (after we flipped DNS to point to this new machine), but over time (2-3 times/day) we were left with an increasing number of `imap-login` processes in state `pre-login`, eventually requiring a dovecot restart. Dialling up process_limits didn't seem to help, we'd just get more of these stale processes. After creating the user on the system, the behaviour reverted to normal, no more stale processes. I do not know what error message that user received on their end, but since we announced we were migrating, they probably just ignored it and thought it as part of the transition. Best, Volker (a happy & long Dovecot user)

3 4

Doveadm Move Query
by Simon B 01 Dec '22

01 Dec '22

I have a production Dovecot problem and although I searched the mailing lists, I could not find an answer and I hope you can give me a quick answer/pointer in the right direction. I have mails for a user (user(a)domain.net) under /var/spool/mail/virtual/ domain.net/user and I want to move ALL the mails to /var/spool/mail/virtual/domain.com/user If I use #doveadm -Dv move -u user(a)domain.net Maildir:/var/spool/mail/virtual/ domain.net/user Maildir:/var/spool/mail/virtual/domain.com/user ALL I get doveadm(root): Fatal: Unknown argument MAILDIR:/var/spool/mail/virtual/ domain.com/user if I use #doveadm -Dv move -u user(a)domain.net Maildir:/var/spool/mail/virtual/ domain.net/user /var/spool/mail/virtual/domain.com/user ALL doveadm(root): Fatal: Unknown argument /var/spool/mail/virtual/ domain.com/user What the hell am I doing wrong!? :) Thanks. Simon

3 3

The end of Dovecot Director?
by Steff Majeur 02 Nov '22

02 Nov '22

I recently stumbled upon the following commit on the Dovecot core Github repository: https://github.com/dovecot/core/commit/4a187116dc2311804be22724007d35732300… Apparently, Dovecot Director is going to be removed in the next major version of Dovecot and the commercial Dovecot cluster architecture will be its successor: https://github.com/dovecot/documentation/blob/a85b742ec4fc2744db30a6943b3c2… This would be a huge blow for many organizations around the world that are currently using Dovecot with Director in a shared storage environment. Can anyone of the Dovecot developers maybe enlighten us about the future of Dovecot? - Will there still be the Director feature in the next community release of Dovecot? - If not, will there be a community feature that is on par with the current Director feature? - For how long will Dovecot version 2.3 still be supported (security fixes, bug fixes)? Is there any EOL plan? Thanks for any clarification! Steff

31 56

Dovecot does not start on MacOS Monterey
by Doro Rose 30 Oct '22

30 Oct '22

Hi, this is a duplicate of the "Dovecot does not start on MacOS Monterey 12.01" thread. I’m still seeing this on dovecot 2.3.19.1. Is there a recommended workaround? The last post in that thread indicated that some modifications in `./src/lib/restrict-process-size.c` might be necessary. Best regards Doro

3 6

how to configure imapsieve to be used per user
by Marc 29 Oct '22

29 Oct '22

I only see configurations that are active for all users, how to configure this in the user sieve rules. I only need this for specific users.

7 11

oauth2 + postgres = Panic: file http-client.c: line 646 (http_client_context_close)
by Sebastiano Degan 27 Oct '22

27 Oct '22

I could to reply to the original thread: https://www.mail-archive.com/dovecot@dovecot.org/msg85204.html But I managed to get a stack trace of the error: root@mail:~ # lldb /usr/local/libexec/dovecot/auth (lldb) target create "/usr/local/libexec/dovecot/auth" Current executable set to '/usr/local/libexec/dovecot/auth' (x86_64). (lldb) run Process 54354 launched: '/usr/local/libexec/dovecot/auth' (x86_64) Error: stats: open(old-stats-user) failed: No such file or directory Panic: file http-client.c: line 646 (http_client_context_close): assertion failed: (cctx->clients_list == NULL) Process 54354 stopped * thread #1, name = 'auth', stop reason = signal SIGABRT frame #0: 0x000000080159f33a libc.so.7`__sys_thr_kill + 10 libc.so.7`__sys_thr_kill: -> 0x80159f33a <+10>: jb 0x80159d498 0x80159f340 <+16>: retq 0x80159f341: int3 0x80159f342: int3 (lldb) bt all * thread #1, name = 'auth', stop reason = signal SIGABRT * frame #0: 0x000000080159f33a libc.so.7`__sys_thr_kill + 10 frame #1: 0x0000000801517c74 libc.so.7`__raise + 52 frame #2: 0x00000008015c9109 libc.so.7`abort + 73 frame #3: 0x00000008012e606b libdovecot.so.0`default_fatal_finish(type=LOG_TYPE_PANIC, status=0) at failures.c:465:3 frame #4: 0x00000008012e3f7c libdovecot.so.0`fatal_handler_real(ctx=0x00007fffffffe610, format="file %s: line %d (%s): assertion failed: (%s)", args=0x00007fffffffe5f0) at failures.c:477:2 frame #5: 0x00000008012e3f14 libdovecot.so.0`default_fatal_handler(ctx=0x00007fffffffe610, format="file %s: line %d (%s): assertion failed: (%s)", args=0x00007fffffffe5f0) at failures.c:485:2 frame #6: 0x00000008012e4253 libdovecot.so.0`i_panic(format="file %s: line %d (%s): assertion failed: (%s)") at failures.c:530:2 frame #7: 0x000000080124f2ed libdovecot.so.0`http_client_context_close(cctx=0x000000080205c048) at http-client.c:646:2 frame #8: 0x000000080124ef22 libdovecot.so.0`http_client_global_context_ioloop_switched(prev_ioloop=0x0000000802030640) at http-client.c:710:3 frame #9: 0x00000008013102e4 libdovecot.so.0`io_loop_set_current(ioloop=0x0000000000000000) at ioloop.c:982:4 frame #10: 0x00000000010aa0be auth`driver_pgsql_set_state(db=0x0000000802092200, state=SQL_DB_STATE_IDLE) at driver-pgsql.c:103:3 frame #11: 0x00000000010a9ff0 auth`connect_callback(db=0x0000000802092200) at driver-pgsql.c:206:3 frame #12: 0x000000080130f3b2 libdovecot.so.0`io_loop_call_io(io=0x00000008020105c0) at ioloop.c:737:2 frame #13: 0x0000000801313e0b libdovecot.so.0`io_loop_handler_run_internal(ioloop=0x0000000802030640) at ioloop-kqueue.c:164:4 frame #14: 0x000000080130fa15 libdovecot.so.0`io_loop_handler_run(ioloop=0x0000000802030640) at ioloop.c:789:2 frame #15: 0x000000080130f8fa libdovecot.so.0`io_loop_run(ioloop=0x0000000802030640) at ioloop.c:762:3 frame #16: 0x00000000010a7eef auth`driver_pgsql_wait(_db=0x0000000802092200) at driver-pgsql.c:1279:3 frame #17: 0x00000000010a4c59 auth`sql_wait(db=0x0000000802092200) at sql-api.c:820:3 frame #18: 0x00000000010a7fc0 auth`driver_pgsql_get_flags(db=0x0000000802092200) at driver-pgsql.c:306:3 frame #19: 0x00000000010a2fa9 auth`sql_get_flags(db=0x0000000802092200) at sql-api.c:171:10 frame #20: 0x00000000010a585e auth`driver_sqlpool_get_flags(_db=0x0000000802092000) at driver-sqlpool.c:450:9 frame #21: 0x00000000010a2fa9 auth`sql_get_flags(db=0x0000000802092000) at sql-api.c:171:10 frame #22: 0x000000000108c675 auth`passdb_sql_init(_module=0x0000000802041ca8) at passdb-sql.c:281:10 frame #23: 0x000000000107d577 auth`passdb_init(passdb=0x0000000802041ca8) at passdb.c:247:3 frame #24: 0x0000000001052a89 auth`auth_passdb_init(passdb=0x0000000802041ba8) at auth.c:308:2 frame #25: 0x0000000001051dbc auth`auth_init(auth=0x0000000802041b70) at auth.c:322:3 frame #26: 0x0000000001051d21 auth`auths_init at auth.c:427:3 frame #27: 0x0000000001050610 auth`main_init at main.c:242:2 frame #28: 0x00000000010503ad auth`main(argc=1, argv=0x00007fffffffeaf8) at main.c:391:2 frame #29: 0x000000000104ff7d auth`_start(ap=<unavailable>, cleanup=<unavailable>) at crt1_c.c:75:7

1 0