June 2021 - dovecot - dovecot.org

Doveadm encrypt/decrypt files manually with per-user folder keys
by Ben Burk 21 Jul '21

21 Jul '21

There is an example of manually encrypting/decrypting mail processed by the mail_crypt plugin here: https://doc.dovecot.org/configuration_manual/mail_crypt_plugin/ It outlines how one would manually encrypt/decrypt messages using global keys: doveadm fs get/put crypt private_key_path=foo:public_key_path=foo2:posix:prefix=/path/to/files/root path/to/file Is there a way to accomplish this using per-user folder keys? Is there some way of making sense of the information in the dovecot-attributes file to accomplish this? -- Ben Burk BURK.TECH System Administrator

2 6

Verifying replication
by Arnold Hendriks 15 Jul '21

15 Jul '21

Dear Dovecot people, I've set up Dovecot replication a couple of years ago. I'm watching the general server health and having Nagios check doveadm replicator status regularly. I'm seeing accounts replicating, disk space usage obviously shows that things work in principle, however, I still wonder whether the replication actually works properly. Accounts I check work, but 1-5% of the messages could easily be missing without me noticing it. These servers have I think about 50K accounts (a lot of them dormant) and between 1 to 2TB of mail. I'm trying to get some more confidence as to whether replication is actually working properly and whether I'm not missing anything that will burn me if I ever have to 'fallback'. Has anyone ever done some verification outside of simply watching doveadm replication stats, to see if they are missing anything ? Eg, I could imagine a process that generates a list of accounts, and then generates hashes on both sides of the replication for each mailbox folder. If the hashes match, the folder gets removed from the list, and once all folders are removed, the account gets removed. Iterating the account list should finally reduce it to 0, or a few extremely high traffic accounts which can be checked manually or be ignored. But I'm trying to avoid reinventing the wheel. Has anyone done anything like this, or can suggest a different approach? With regards, Arnold Hendriks

1 1

[ENHANCEMENT] Doveadm HTTP API and the mail-crypt plugin
by Kevin N. 13 Jul '21

13 Jul '21

Hello everybody, I was wondering, are there any plans for the near future to include the mail-crypt plugin related functions, like changing the user key password, into the doveadm http api? It would probably be possible to implement it in a more secure way than any other external ad-hoc solution. Wouldn't it? Cheers, K.

2 2

How to use xapian with non-text attachments
by André Rodier 03 Jul '21

03 Jul '21

Hello, I am developing a hosting platform on Debian Bullseye, with Dovecot amongst other tools. I am trying to use the xapian full test search plugin, but I can see the attachments are skipped: This is what I have in the logs when running the indexing in verbose mode: ----------------------------------------------------------------------- doveadm(camille): Info: FTS Xapian: fts_backend_xapian_check_access doveadm(camille): Info: FTS Xapian: Memory stats : Used = 56 MB, Free = 66 MB doveadm(camille): Info: FTS Xapian: fts_backend_xapian_index_hdr doveadm(camille): Info: FTS Xapian: fts_backend_xapian_query doveadm(camille): Info: FTS Xapian: Query= uid:"44" doveadm(camille): Info: FTS Xapian: Ngram(S) -> 63 items (total 0 KB) doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_unset_build_key doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_set_build_key doveadm(camille): Info: FTS Xapian: New part (Header=Message- Id,Type=(null),Disposition=(null)) doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_build_more doveadm(camille): Info: FTS Xapian: fts_backend_xapian_check_access doveadm(camille): Info: FTS Xapian: Memory stats : Used = 56 MB, Free = 66 MB doveadm(camille): Info: FTS Xapian: fts_backend_xapian_index_hdr doveadm(camille): Info: FTS Xapian: fts_backend_xapian_query doveadm(camille): Info: FTS Xapian: Query= uid:"44" doveadm(camille): Info: FTS Xapian: Ngram(XMID) -> 4 items (total 0 KB) doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_unset_build_key doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_set_build_key doveadm(camille): Info: FTS Xapian: New part (Header=X- Mailer,Type=(null),Disposition=(null)) doveadm(camille): Info: FTS Xapian: Unknown header (indexing) 'xmailer' doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_set_build_key doveadm(camille): Info: FTS Xapian: New part (Header=MIME- Version,Type=(null),Disposition=(null)) doveadm(camille): Info: FTS Xapian: Unknown header (indexing) 'mimeversion' doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_set_build_key doveadm(camille): Info: FTS Xapian: New part (Header=Content- Type,Type=(null),Disposition=(null)) doveadm(camille): Info: FTS Xapian: Unknown header (indexing) 'contenttype' doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_set_build_key doveadm(camille): Info: FTS Xapian: New part (Header=Authentication- Results,Type=(null),Disposition=(null)) doveadm(camille): Info: FTS Xapian: Unknown header (indexing) 'authenticationresults' doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_set_build_key doveadm(camille): Info: FTS Xapian: New part (Header=X-AV- Checked,Type=(null),Disposition=(null)) doveadm(camille): Info: FTS Xapian: Unknown header (indexing) 'xavchecked' doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_set_build_key doveadm(camille): Info: FTS Xapian: New part (Header=Content- Type,Type=(null),Disposition=(null)) doveadm(camille): Info: FTS Xapian: Unknown header (indexing) 'contenttype' doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_set_build_key doveadm(camille): Info: FTS Xapian: New part (Header=(null),Type=text/plain,Disposition=(null)) doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_build_more doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_unset_build_key doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_set_build_key doveadm(camille): Info: FTS Xapian: New part (Header=Content- Type,Type=(null),Disposition=(null)) doveadm(camille): Info: FTS Xapian: Unknown header (indexing) 'contenttype' doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_set_build_key doveadm(camille): Info: FTS Xapian: New part (Header=Content- Description,Type=(null),Disposition=(null)) doveadm(camille): Info: FTS Xapian: Unknown header (indexing) 'contentdescription' doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_set_build_key doveadm(camille): Info: FTS Xapian: New part (Header=Content- Disposition,Type=(null),Disposition=(null)) doveadm(camille): Info: FTS Xapian: Unknown header (indexing) 'contentdisposition' doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_set_build_key doveadm(camille): Info: FTS Xapian: New part (Header=Content-Transfer- Encoding,Type=(null),Disposition=(null)) doveadm(camille): Info: FTS Xapian: Unknown header (indexing) 'contenttransferencoding' doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_set_build_key doveadm(camille): Info: FTS Xapian: New part (Header=(null),Type=text/csv,Disposition=attachment; filename="file.csv") doveadm(camille): Info: FTS Xapian: Skipping part of type 'text/csv' and disposition 'attachment; filename="file.csv"' doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_set_mailbox doveadm(camille): Info: FTS Xapian: Unset box 'INBOX' (c0d4e304584e5460dae30000075d7e67) doveadm(camille): Info: FTS Xapian: fts_backend_xapian_oldbox doveadm(camille): Info: FTS Xapian: Done indexing 'INBOX' (c0d4e304584e5460dae30000075d7e67) (13 msgs in 261 ms, rate: 49.8) doveadm(camille): Info: FTS Xapian: fts_backend_xapian_release (unset_box) doveadm(camille): Info: FTS Xapian: Committed 'unset_box' in 17 ms doveadm(camille): Info: FTS Xapian: Box is empty doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_deinit (/home/users/camille/mails/indexes/xapian-indexes) doveadm(camille): Info: FTS Xapian: fts_backend_xapian_release (update_deinit) doveadm(camille): Info: FTS Xapian: Committed 'update_deinit' in 0 ms doveadm(camille): Info: FTS Xapian: Deinit /home/users/camille/mails/indexes/xapian-indexes) ----------------------------------------------------------------------- Especially thos line: doveadm(camille): Info: FTS Xapian: Skipping part of type 'text/csv' and disposition 'attachment; filename="file.csv"' I have seen this as a bug on fts-xapian: https://github.com/grosjo/fts-xapian/issues/68 Which then references a PR on Dovecot core itself: https://github.com/dovecot/core/pull/155 Any idea on how to use xapian with non-text attachments ? Thanks a lot for your help! André

2 2

Dovecot sync broke after upgrade to OpenBSD 6.9
by William Orr 01 Jul '21

01 Jul '21

Hey, I have two mailservers running OpenBSD 6.9, and I use bidirectional syncing of my maildirs through doveadm. After the upgrade, I noticed that the sync process was failing. OpenBSD 6.8 shipped with Dovecot 2.3.13, 6.9 ships with 2.3.14. Here's the output of the dovecot --build-options: kefka|~|03:01:50|89$ dovecot --build-options Build options: ioloop=kqueue notify=kqueue openssl io_block_size=8192 SQL driver plugins: mysql postgresql sqlite Passdb: bsdauth checkpassword ldap passwd passwd-file sql Userdb: checkpassword ldap(plugin) passwd prefetch passwd-file sql I did not compile this manually, this is from packages. More info about configure options passed are available here, in the Makefile from ports https://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/mail/dovecot/Makefile?rev=1… Here's a sample run of the sync: kefka|~|02:57:05|0$ doas doveadm sync -u worr(a)worrbase.com remote:sabin.worrbase.com doveadm(VERSION dsync 3 5): Error: User doesn't exist dsync-local(worr)<FNQIMvI9omB4VwEADTvxNg>: Error: read(sabin.worrbase.com) failed: EOF (version not received) dsync-local(worr)<FNQIMvI9omB4VwEADTvxNg>: Error: Remote command returned error 67: /usr/bin/ssh -i /root/.ssh/id_ed25519.dsync sabin.worrbase.com /usr/local/bin/dsync-in-wrapper.sh kefka|~|02:57:08|75$ cat /usr/local/bin/dsync-in-wrapper.sh #!/bin/ksh read username /usr/local/bin/doveadm dsync-server -u "$username" I ktraced the process, and noticed that in the communication with the remote mail server, that a bunch of doveadm plugins fail to load. It's worth noting that these are plugins that are dlopen(3)ed in response to certain commands sent over the wire, so they don't show up in ldd(1) output. kefka|~|03:00:08|0$ doas kdump | grep symbol "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_user_module' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_mailbox_get_aclobj' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_mailbox_list_get_backend' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_object_list_init' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_object_list_next' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_backend_rights_match_me' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_rights_get_id' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_object_list_deinit' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_object_get_my_rights' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_rights_update_import' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_mailbox_update_acl' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_lookup_dict_rebuild' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_backend_nonowner_lookups_iter_init' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_backend_nonowner_lookups_iter_next' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_backend_nonowner_lookups_iter_deinit' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_backend_nonowner_lookups_rebuild' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_lookup_dict_is_enabled' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_lookup_dict_iterate_visible_init' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_lookup_dict_iterate_visible_next' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol 'acl_lookup_dict_iterate_visible_deinit' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol 'quota_user_module' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol 'quota_root_get_resources' "doveadm:/usr/local/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol 'quota_get_resource' "doveadm:/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol 'fts_list_backend' "doveadm:/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol 'fts_backend_lookup' "doveadm:/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol 'fts_backend_lookup_done' "doveadm:/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol 'fts_search_args_expand' "doveadm:/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol 'fts_language_find' "doveadm:/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol 'fts_user_get_language_list' "doveadm:/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol 'fts_language_detect' "doveadm:/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol 'fts_language_list_get_first' "doveadm:/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol 'fts_user_language_find' "doveadm:/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol 'fts_tokenizer_reset' "doveadm:/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol 'fts_tokenizer_final' "doveadm:/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol 'fts_tokenizer_next' "doveadm:/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol 'fts_filter_filter' "doveadm:/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol 'fts_backend_optimize' "doveadm:/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol 'fts_backend_rescan' "doveadm:/usr/local/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol 'mail_crypt_user_get_public_key' "doveadm:/usr/local/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol 'mail_crypt_user_generate_keypair' "doveadm:/usr/local/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol 'mail_crypt_box_get_public_key' "doveadm:/usr/local/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol 'mail_crypt_box_generate_keypair' "doveadm:/usr/local/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol 'mail_crypt_box_get_pvt_digests' "doveadm:/usr/local/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol 'mail_crypt_acl_secure_sharing_enabled'" "doveadm:/usr/local/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol 'mail_crypt_box_share_private_keys' "doveadm:/usr/local/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol 'mail_crypt_box_set_shared_key' "doveadm:/usr/local/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol 'mail_crypt_get_private_key' I checked on both machines, and these symbols are all undefined in the associated libraries. I do notice though, that these symbols seem to be present in similar dovecot, non-doveadm plugins: kefka|~|03:04:56|130$ nm -A /usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so | grep acl_user_module /usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: U acl_user_module kefka|~|03:05:14|0$ nm -A /usr/local/lib/dovecot/lib01_acl_plugin.so | grep acl_user_module /usr/local/lib/dovecot/lib01_acl_plugin.so:0001b008 D acl_user_module However, the doveadm plugins don't link against them? kefka|~|03:04:24|1$ ldd /usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so /usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: Start End Type Open Ref GrpRef Name 000001d89e043000 000001d89e04d000 dlib 1 0 0 /usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so Is anyone else seeing issues like this? Is there perhaps a misconfiguration on my end? Here's the full output of dovecot -n # 2.3.14 (cee3cbc0d): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.14 (1b5c82b2) # OS: OpenBSD 6.9 amd64 # Hostname: kefka.worrbase.com auth_username_format = %n default_vsz_limit = 512 M doveadm_password = # hidden, use -P to show it dsync_remote_cmd = /usr/bin/ssh -i /root/.ssh/id_ed25519.dsync %{host} /usr/local/bin/dsync-in-wrapper.sh first_valid_uid = 1000 imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags mail_location = maildir:~/Maildir mail_plugins = " notify replication" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mbox_write_locks = fcntl mmap_disable = yes namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Sent { special_use = \Sent } mailbox Spam { special_use = \Junk } mailbox Trash { special_use = \Trash } prefix = separator = / } passdb { driver = bsdauth } plugin { mail_replica = remoteprefix:root@sabin.worrbase.com sieve = ~/.dovecot.sieve sieve_dir = ~/.sieve sieve_user_log = ~/.dovecot.sieve.log } pop3_client_workarounds = outlook-no-nuls oe-ns-eoh postmaster_address = postmaster(a)worrbase.com protocols = imap lmtp service aggregator { fifo_listener replication-notify-fifo { mode = 0666 } unix_listener replication-notify { mode = 0666 } } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } } service lmtp { inet_listener lmtp { address = localhost port = 2525 } } service managesieve-login { inet_listener sieve { port = 4190 } inet_listener sieve_deprecated { port = 2000 } } service pop3-login { inet_listener pop3 { port = 0 } inet_listener pop3s { port = 0 } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } ssl = required ssl_cert = </etc/mail/certs/mail.worrbase.com.crt ssl_cipher_list = ALL:HIGH:!TLSv1:!SSLv3:!SSLv2:!MEDIUM:!LOW:!EXP:!RC4:!MD5:!aNULL:@STRENGTH ssl_client_ca_file = /etc/ssl/cert.pem ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it ssl_min_protocol = TLSv1.1 userdb { driver = passwd } protocol lmtp { mail_plugins = " notify replication sieve" } protocol lda { mail_plugins = " notify replication sieve" } protocol imap { mail_max_userip_connections = 20 mail_plugins = " notify replication" } Let me know if I need to provide more info. Thanks so much for the help!

3 3

Pigeonhole - Auto create default per user script
by dovecot＠ptld.com 01 Jul '21

01 Jul '21

Is there a way to have a per-user default sieve script get auto created if it doesn't exist? Or created on first time use like when the directory set by plugin{sieve=file:~/} gets created? Not global scripts.

2 1

Pigeonhole sieve_user_log
by dovecot＠ptld.com 01 Jul '21

01 Jul '21

https://doc.dovecot.org/configuration_manual/sieve/configuration/ Pigeonhole creates logs separate from main dovecot logs. I see sieve_user_log allows you to change log location and it says if you don't set it logs are still made by default. I can not find anything about log rotation of these pigeonhole logs or if there is a way to disable this logging. Do i need to worry about this log growing out of control? Is there a way to disable it? Does it prune/purge itself? Any practical reason to keep these logs?

2 1

Feature/patch request
by Bucher, Dr. Damian 30 Jun '21

30 Jun '21

Hi, we ran into a problem, while using filesystem snapshots on a GPFS-filesystem. With this we want to give the users the opportunity to restore Mailboxes from this snapshots in case they deleted mail by accident. For each snapshot we create a namespace via a postlogin skript (with INDEX=MEMORY to avoid writing index files to a ro filesystem). The snapshot are of course read only. When accessing the mailboxes in the namespaces, dovecot generates an error (ro filesystem): Error: Mailbox #backup20/Mail/deleted-messages: open() failed with mbox: Read-only file system We tracked that down to a problem in src/lib-storage/index/mbox/mbox-file.c (yes, we are still using mbox ;) ) fd = open(mailbox_get_path(&mbox->box), mbox_is_backend_readonly(mbox) ? O_RDONLY : O_RDWR); if (fd == -1 && errno == EACCES && !mbox->backend_readonly) { mbox->backend_readonly = TRUE; fd = open(mailbox_get_path(&mbox->box), O_RDONLY); } Here the mailbox is opened and it is checked wether the file is ro or rw. Since the snapshot contains the original file permissions, the file is of course rw, but the filesystem is ro. Unfortunately the code only checks for file permissions (errno == EACCES), which doesn’t catch our error condition (FS ro). We think that exchanging this to (errno == EACCESS || errno == EROFS) would catch the file system ro error correctly and set the readonly flag correctly and open the mailbox ro. (this probably hast to be done for other mailbox formats in the corresponding files too) What do you think? Is there a chance to implement that in an upcoming release? Best regards Damian — Postmaster@WWU WWU IT Westfälische Wilhelms-Universität (WWU) Münster Röntgenstr. 7-13, 48149 Münster mailto:postmaster@uni-muenster.de

2 2

oauth2
by Andrea Gabellini 30 Jun '21

30 Jun '21

Hello, Dovecot 2.3.15 installed from repository on CentOS 7.9 I would like to try the oauth2 mechanism to autenticate my users. I have installed a working demo of KeyCloak and I have a working installation of Dovecot 2.3.15 with LDAP backend and plain authentication. My Thunderbird client always says "that the server doesn't manage the authentication method", but with a "openssl s_client -connect" command I get: * OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN AUTH=OAUTHBEARER AUTH=XOAUTH2] IMAP ready. Putting Dovecot in debug I can't see any error or warning on oauth2 authentication. I'm following the offical guide: auth_mechanisms = plain oauthbearer xoauth2 passdb { driver = oauth2 mechanisms = xoauth2 oauthbearer args = /etc/dovecot/dovecot-oauth2.conf.ext } /etc/dovecot/dovecot-oauth2.conf.ext: tokeninfo_url = https://www.googleapis.com/oauth2/v3/tokeninfo?access_token= introspection_url = https://www.googleapis.com/oauth2/v2/userinfo #force_introspection = yes username_attribute = email tls_ca_cert_file = /etc/ssl/certs/ca-certificates.crt I'm using an explicit wrong configuration to thrown some errors and verify that Dovecot is really using the OAUTH2 mechanism, but I don't get any errors... What I'm doing wrong? Thanks, Andrea -- __________________________ Okay, okay, I'll take it back ... UNfuck you ! __________________________ TIM San Marino S.p.A. Andrea Gabellini Engineering R&D TIM San Marino S.p.A. - https://www.telecomitalia.sm Via Ventotto Luglio, 212 - Piano -2 47893 - Borgo Maggiore - Republic of San Marino Tel: (+378) 0549 886237 Fax: (+378) 0549 886188 -- Informativa Privacy Questa email ha per destinatari dei contatti presenti negli archivi di TIM San Marino S.p.A.. Tutte le informazioni vengono trattate e tutelate nel rispetto della normativa vigente sulla protezione dei dati personali (Reg. EU 2016/679). Per richiedere informazioni e/o variazioni e/o la cancellazione dei vostri dati presenti nei nostri archivi potete inviare una email a privacy(a)telecomitalia.sm. Avviso di Riservatezza Il contenuto di questa e-mail e degli eventuali allegati e' strettamente confidenziale e destinato alla/e persona/e a cui e' indirizzato. Se avete ricevuto per errore questa e-mail, vi preghiamo di segnalarcelo immediatamente e di cancellarla dal vostro computer. E' fatto divieto di copiare e divulgare il contenuto di questa e-mail. Ogni utilizzo abusivo delle informazioni qui contenute da parte di persone terze o comunque non indicate nella presente e-mail potra' essere perseguito ai sensi di legge.

2 7

Panic indexer-worker on Dovecot 2.3.15
by Alessio Cecchi 30 Jun '21

30 Jun '21

Hi, after the upgrade to dovecot 2.3.15 the Panic: indexer-worker error is still present. I don't know if is related to searching in a Virtual folder (look for my previous email on list with subject "Panic on indexer-worker and Dovecot stop to search on Solr" for more details). Here the relevant configuration of FTS plugin: plugin { fts = solr fts_autoindex = no fts_autoindex_exclude = \Junk fts_autoindex_exclude2 = \Drafts fts_decoder = decode2text fts_enforced = body fts_index_timeout = 5s fts_solr = url=http://10.0.0.32:8983/solr/dovecot/ [...] I'm running Dovecot with FTS Solr 7.7, here the log error and the bt full: Jun 29 13:06:26 Panic: indexer-worker(user1(a)company.net) session=<SLTLmOXFFIy5Ydlu:6BC4HLH+2mBHQwAAxbYlJw>: file http-clien t-request.c: line 1240 (http_client_request_send_more): assertion failed: (req->payload_input != NULL) Jun 29 13:06:26 Error: indexer-worker(user1(a)company.net) session=<SLTLmOXFFIy5Ydlu:6BC4HLH+2mBHQwAAxbYlJw>: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x42) [0x7f7212b36862] -> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f7212b3696e] -> /usr/lib64/dovecot/libdovecot.so.0(+0xf50fe) [0x7f7212b440fe] -> /usr/lib64/dovecot/libdovecot.so.0(+0xf51a1) [ 0x7f7212b441a1] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f7212a9460c] -> /usr/lib64/dovecot/libdovecot.so.0(http_client_ request_send_more+0x3dd) [0x7f7212add54d] -> /usr/lib64/dovecot/libdovecot.so.0(http_client_connection_output+0xf1) [0x7f7212ae1ca1] -> /usr/lib64/dovecot/libdovecot.so.0(+0x11e870) [0x7f7212b6d870] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x65) [0x7f 7212b5c425] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12b) [0x7f7212b5ddab] -> /usr/lib64/dovecot/libdove cot.so.0(io_loop_handler_run+0x59) [0x7f7212b5c529] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f7212b5c768] -> /usr /lib64/dovecot/libdovecot.so.0(http_client_wait+0xcd) [0x7f7212aeadbd] -> /usr/lib64/dovecot/lib21_fts_solr_plugin.so(solr_connectio n_select+0xe5) [0x7f7210a79265] -> /usr/lib64/dovecot/lib21_fts_solr_plugin.so(+0x4684) [0x7f7210a75684] -> /usr/lib64/dovecot/lib20 _fts_plugin.so(fts_backend_get_last_uid+0x6e) [0x7f7211e1944e] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0xf9d2) [0x7f7211e1f9d2] - > /usr/lib64/dovecot/lib20_fts_plugin.so(+0x10f56) [0x7f7211e20f56] -> /usr/lib64/dovecot/lib20_virtual_plugin.so(+0x94da) [0x7f7211 a014da] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0x10ba6) [0x7f7211e20ba6] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_prec ache+0x2e) [0x7f7212e48fae] -> dovecot/indexer-worker [user1(a)company.net Virtual/All](+0x2bb1) [0x55559e522bb1] -> /usr/lib 64/dovecot/libdovecot.so.0(io_loop_call_io+0x65) [0x7f7212b5c425] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal +0x12b) [0x7f7212b5ddab] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x59) [0x7f7212b5c529] -> /usr/lib64/dovecot/libd ovecot.so.0(io_loop_run+0x38) [0x7f7212b5c768] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f7212ac63c3] -> do vecot/indexer-worker [user1(a)company.net Virtual/All](main+0xd7) [0x55559e522437] -> /lib64/libc.so.6(__libc_start_main+0xf5 ) [0x7f72126a3555] -> dovecot/indexer-worker [user1(a)company.net Virtual/All](+0x24fa) [0x55559e5224fa] Jun 29 13:06:26 Fatal: indexer-worker(user1(a)company.net) session=<SLTLmOXFFIy5Ydlu:6BC4HLH+2mBHQwAAxbYlJw>: master: service(indexer-worker): child 17223 killed with signal 6 (core dumped) # gdb /usr/libexec/dovecot/indexer-worker /var/core/core.indexer-worker.17223 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/libexec/dovecot/indexer-worker...Reading symbols from /usr/lib/debug/usr/libexec/dovecot/indexer-worker.debug...done. done. [New LWP 17223] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `dovecot/indexer-worker'. Program terminated with signal 6, Aborted. #0 0x00007f72126b7387 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 expat-2.1.0-12.el7.x86_64 glibc-2.17-324.el7_9.x86_64 libgcc-4.8.5-44.el7.x86_64 libzstd-1.5.0-1.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-19.el7_9.x86_64 (gdb) bt full #0 0x00007f72126b7387 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f72126b8a78 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f7212b440b7 in default_fatal_finish (status=0, type=<optimized out>) at failures.c:459 backtrace = 0x55559eb75970 "/usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x42) [0x7f7212b36862] -> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f7212b3696e] -> /usr/lib64/dovecot/libdovecot.so.0(+0xf50fe) [0"... recursed = 0 #3 fatal_handler_real (ctx=<optimized out>, format=<optimized out>, args=<optimized out>) at failures.c:471 status = 0 #4 0x00007f7212b441a1 in i_internal_fatal_handler (ctx=<optimized out>, format=<optimized out>, args=<optimized out>) at failures.c:866 No locals. #5 0x00007f7212a9460c in i_panic (format=format@entry=0x7f7212b85cf8 "file %s: line %d (%s): assertion failed: (%s)") at failures.c:523 ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0, timestamp_usecs = 0, log_prefix = 0x0, log_prefix_type_pos = 0} args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 0x7ffe7b6e7150, reg_save_area = 0x7ffe7b6e7090}} #6 0x00007f7212add54d in http_client_request_send_more (req=req@entry=0x55559eca9db8, pipelined=pipelined@entry=false) at http-client-request.c:1240 conn = 0x55559ec8b750 cctx = 0x55559ebb63a8 output = 0x55559ecafa70 res = <optimized out> error = <optimized out> offset = <optimized out> __func__ = "http_client_request_send_more" #7 0x00007f7212ae1ca1 in http_client_connection_continue_request (conn=0x55559ec8b750) at http-client-connection.c:1342 tmp_conn = 0x55559ec8b750 count = 1 ret = <optimized out> reqs = <optimized out> req = 0x55559eca9db8 pipelined = false #8 http_client_connection_output (conn=0x55559ec8b750) at http-client-connection.c:1374 output = <optimized out> ret = <optimized out> #9 0x00007f7212b6d870 in stream_send_io (fstream=0x55559ec8cb20) at ostream-file.c:526 ostream = 0x55559ec8cbc0 am = 0x55559ec8cb20 use_cork = true ret = <optimized out> ---Type <return> to continue, or q <return> to quit--- #10 0x00007f7212b5c425 in io_loop_call_io (io=0x55559ec925f0) at ioloop.c:715 ioloop = 0x55559ecaf4d0 t_id = 6 __func__ = "io_loop_call_io" #11 0x00007f7212b5ddab in io_loop_handler_run_internal (ioloop=ioloop@entry=0x55559ecaf4d0) at ioloop-epoll.c:222 ctx = 0x55559ecab870 events = <optimized out> list = 0x55559ec8c8e0 io = <optimized out> tv = {tv_sec = 0, tv_usec = 0} events_count = <optimized out> msecs = <optimized out> ret = 1 i = 0 call = <optimized out> __func__ = "io_loop_handler_run_internal" #12 0x00007f7212b5c529 in io_loop_handler_run (ioloop=ioloop@entry=0x55559ecaf4d0) at ioloop.c:767 __func__ = "io_loop_handler_run" #13 0x00007f7212b5c768 in io_loop_run (ioloop=0x55559ecaf4d0) at ioloop.c:740 __func__ = "io_loop_run" #14 0x00007f7212aeadbd in http_client_wait () at http-client.c:339 prev_ioloop = 0x55559eb7b050 client_ioloop = 0x55559ecaf4d0 prev_client_ioloop = 0x55559eb7b050 __func__ = "http_client_wait" #15 0x00007f7210a79265 in solr_connection_select (conn=0x55559eb8b150, query=<optimized out>, pool=<optimized out>, box_results_r=0x7ffe7b6e7400) at solr-connection.c:211 lctx = {result_pool = 0x55559ec03910, payload = 0x0, io = 0x0, request_status = 0, parser = 0x0, results = 0x0} http_req = 0x55559ecb3b48 url = <optimized out> #16 0x00007f7210a75684 in fts_backend_solr_get_last_uid (_backend=0x55559eb8b090, box=0x55559ebe59a8, last_uid_r=0x7ffe7b6e74a4) at fts-backend-solr.c:238 hdr = {last_indexed_uid = 0, settings_checksum = 0, unused = 0} #17 0x00007f7211e1944e in fts_backend_get_last_uid (backend=<optimized out>, box=box@entry=0x55559ebe59a8, last_uid_r=last_uid_r@entry=0x7ffe7b6e74a4) at fts-api.c:106 hdr = {last_indexed_uid = 2, settings_checksum = 0, unused = 317606002} #18 0x00007f7211e1f9d2 in fts_mailbox_get_last_cached_seq (box=box@entry=0x55559ebe59a8, seq_r=seq_r@entry=0x7ffe7b6e74f4) at fts-storage.c:85 flist = <optimized out> seq1 = 21845 seq2 = 2663132384 last_uid = 21845 ---Type <return> to continue, or q <return> to quit--- #19 0x00007f7211e20f56 in fts_mail_precache_init (_mail=<optimized out>, _mail=<optimized out>) at fts-storage.c:478 ft = 0x55559ec8c820 flist = 0x55559ebca0a0 last_seq = 21845 #20 fts_mail_index (_mail=0x55559ecaed08) at fts-storage.c:501 ft = 0x55559ec8c820 flist = 0x55559ebca0a0 pmail = 0x55559ecaed08 #21 fts_mail_precache (_mail=0x55559ecaed08) at fts-storage.c:562 _data_stack_cur_id = 5 mail = 0x55559ecaed08 fmail = <optimized out> ft = 0x55559ec8c820 ret = 0 __func__ = "fts_mail_precache" #22 0x00007f7211a014da in virtual_mail_precache (mail=<optimized out>) at virtual-mail.c:243 vmail = <optimized out> backend_mail = 0x55559ecaed08 p = <optimized out> #23 0x00007f7211e20ba6 in fts_mail_precache (_mail=0x55559ec027f8) at fts-storage.c:555 mail = 0x55559ec027f8 fmail = 0x55559ec02f70 ft = 0x55559ebff760 ret = 0 __func__ = "fts_mail_precache" #24 0x00007f7212e48fae in mail_precache (mail=0x55559ec027f8) at mail.c:455 _data_stack_cur_id = 4 p = 0x55559ec027f8 ret = <optimized out> #25 0x000055559e522bb1 in index_mailbox_precache (conn=<optimized out>, box=0x55559eb9c498) at master-connection.c:119 counter = 1 max = 93 percentage_sent = 0 storage = <optimized out> status = {messages = 93, recent = 0, unseen = 0, uidvalidity = 1624964785, uidnext = 94, first_unseen_seq = 0, first_recent_uid = 94, last_cached_seq = 0, highest_modseq = 0, highest_pvt_modseq = 0, keywords = 0x0, permanent_flags = 0, flags = 0, permanent_keywords = false, allow_new_keywords = false, nonpermanent_modseqs = false, no_modseq_tracking = false, have_guids = true, have_save_guids = true, have_only_guid128 = false} uids = <optimized out> username = 0x55559eba0e18 "user1(a)company.net" first_uid = 1 percentage_str = "\261\376\332`" ---Type <return> to continue, or q <return> to quit--- percentage = <optimized out> error = MAIL_ERROR_NONE trans = 0x55559ebf1b00 ctx = 0x55559ebf3640 last_uid = 2 ret = 0 index_event = 0x55559ebf3250 box_vname = 0x55559eb9c830 "Virtual/All" errstr = <optimized out> search_args = 0x0 mail = 0x55559ec027f8 metadata = {guid = '\000' <repeats 15 times>, virtual_size = 0, physical_size = 0, first_save_date = 0, cache_fields = 0x0, precache_fields = (MAIL_FETCH_STREAM_HEADER | MAIL_FETCH_STREAM_BODY | MAIL_FETCH_RECEIVED_DATE | MAIL_FETCH_SAVE_DATE | MAIL_FETCH_PHYSICAL_SIZE | MAIL_FETCH_VIRTUAL_SIZE | MAIL_FETCH_UIDL_BACKEND | MAIL_FETCH_GUID | MAIL_FETCH_POP3_ORDER), backend_ns_prefix = 0x0, backend_ns_type = (unknown: 0)} seq = <optimized out> #26 index_mailbox (user=<optimized out>, user=<optimized out>, what=<optimized out>, max_recent_msgs=<optimized out>, mailbox=<optimized out>, conn=0x55559ebd8a80) at master-connection.c:238 box = 0x55559eb9c498 status = {messages = 2662839088, recent = 21845, unseen = 1, uidvalidity = 0, uidnext = 2662837936, first_unseen_seq = 21845, first_recent_uid = 313289502, last_cached_seq = 32626, highest_modseq = 140730969257744, highest_pvt_modseq = 93826223419056, keywords = 0x55559eb7aeb0, permanent_flags = 313289927, flags = 32626, permanent_keywords = false, allow_new_keywords = false, nonpermanent_modseqs = false, no_modseq_tracking = false, have_guids = true, have_save_guids = false, have_only_guid128 = false} errstr = <optimized out> ret = 0 ns = <optimized out> path = 0x55559eb74a50 "/home/vmail/domains/2/company.net/0/user1/Maildir/virtual/All" error = 93 sync_flags = <optimized out> #27 master_connection_input_line (line=<optimized out>, conn=0x55559ebd8a80) at master-connection.c:280 str = <optimized out> max_recent_msgs = 0 ret = <optimized out> args = <optimized out> input = {event_parent = 0x0, module = 0x55559e52343f "mail", service = 0x55559e523430 "indexer-worker", username = 0x55559eb72038 "user1(a)company.net", session_id = 0x0, session_id_prefix = 0x55559eb7205f "SLTLmOXFFIy5Ydlu", session_create_time = 0, local_ip = {family = 0, u = {ip6 = { __in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ip4 = {s_addr = 0}}}, remote_ip = {family = 0, u = {ip6 = {__in6_u = { __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ---Type <return> to continue, or q <return> to quit--- ip4 = {s_addr = 0}}}, local_port = 0, remote_port = 0, userdb_fields = 0x0, forward_fields = 0x0, flags_override_add = (unknown: 0), flags_override_remove = (unknown: 0), no_userdb_lookup = false, debug = false, conn_secured = false, conn_ssl_secured = false} service_user = 0x55559ebcb318 user = 0x55559eba0cf8 error = 0x0 #28 master_connection_input (conn=0x55559ebd8a80) at master-connection.c:320 _data_stack_cur_id = 3 line = <optimized out> #29 0x00007f7212b5c425 in io_loop_call_io (io=0x55559ec54760) at ioloop.c:715 ioloop = 0x55559eb7b050 t_id = 2 __func__ = "io_loop_call_io" #30 0x00007f7212b5ddab in io_loop_handler_run_internal (ioloop=ioloop@entry=0x55559eb7b050) at ioloop-epoll.c:222 ctx = 0x55559eb7bc40 events = <optimized out> list = 0x55559eb86770 io = <optimized out> tv = {tv_sec = 2147483, tv_usec = 0} events_count = <optimized out> msecs = <optimized out> ret = 1 i = 0 call = <optimized out> __func__ = "io_loop_handler_run_internal" #31 0x00007f7212b5c529 in io_loop_handler_run (ioloop=ioloop@entry=0x55559eb7b050) at ioloop.c:767 __func__ = "io_loop_handler_run" #32 0x00007f7212b5c768 in io_loop_run (ioloop=0x55559eb7b050) at ioloop.c:740 __func__ = "io_loop_run" #33 0x00007f7212ac63c3 in master_service_run (service=0x55559eb7aeb0, callback=callback@entry=0x55559e5225f0 <client_connected>) at master-service.c:862 No locals. #34 0x000055559e522437 in main (argc=1, argv=0x55559eb7ab90) at indexer-worker.c:76 storage_service_flags = (MAIL_STORAGE_SERVICE_FLAG_USERDB_LOOKUP | MAIL_STORAGE_SERVICE_FLAG_TEMP_PRIV_DROP | MAIL_STORAGE_SERVICE_FLAG_NO_IDLE_TIMEOUT) c = <optimized out> (gdb) quit Hope can be fixed. Thanks -- Alessio Cecchi Postmaster @ http://www.qboxmail.it https://www.linkedin.com/in/alessice

1 0