November 2021 - dovecot

ZFS storage and backup
by infoomatic 22 Nov '21

22 Nov '21

Hi listmembers, I am about to migrate our mailservices to FreeBSD + ZFS. Thus, before entering the sheer endless stage of performance testing, I thought I would ask here kindly for all kinds of information. My setups are nothing special with few users, however, I would like to have a nice setup, maybe some of you could contribute to this thread. We are using slow spinning disks, but we may consider using ssds in a not-so-distant future. *) storages: any infos on ZFS options or whether to use mdbox or sdbox, and what configs/options regarding compression etc. *) backup: what is a best practice regarding backups? - using only the dovecot tools or leveraging the great features of ZFS (or both) with snapshots etc.? Thanks for all sorts of infos, probably saving me quite some time evaluating different options! Robert

5 6

Mailinglistserver IPv6 address blacklisted
by Arjen de Korte 21 Nov '21

21 Nov '21

It looks like the mailinglist server IPv6 address has been blacklisted: https://check.spamhaus.org/listed/?searchterm=2a04:3545:1000:720:acc1:5bff:… The 'Show details' shows the entire 2a04:3545:1000:720::/64 network is blacklisted. Whois tells this is a network used by UpCloud Cloud Servers. So either the mailinglistserver or a server it shares its network with is compromised. If UpCloud gives its users their own /64, it would probably be the former. Otherwise it might make sense to ask them for one, as sharing IPv6 networks between customers is a no-no in the IPv6 world.

2 1

Solr FTS - message deletes not working as expected
by Shawn Heisey 19 Nov '21

19 Nov '21

I have Solr FTS enabled in dovecot. It is behaving differently than I would expect. When I delete messages in Thunderbird normally, using IMAP to talk to dovecot, the deletion hits Solr right away. If I then ask Thunderbird to empty the trash, that delete also hits Solr immediately. But if I use shift-delete in Thunderbird, which deletes the message immediately without going through Trash, things are different. One delete that I did (6 messages) took at least a minute before Solr saw the request. And on other tests that I ran, deleting one message, I never did see the delete request hit Solr. I waited for several minutes. I use the shift-delete mechanism a lot more than the method that sends to trash. That way I do not need to remember to empty the trash. Dovecot version is 2:2.3.16-2+ubuntu20.04 installed from the dovecot repo, not from distro packages. I don't think the Solr version matters, but just in case I am wrong, that is 8.10.1. I don't recall exactly what I changed in the solrconfig.xml and managed-schema files, but it should not be anything that affects how Solr FTS works. The problem seems to be that Solr FTS isn't sending deletes, not that Solr is behaving incorrectly. I just discovered that trying to get the dovecot config failed: root@bilbo:~# doveconf -n # 2.3.16 (7e2e900c1a): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.16 (09c29328) doveconf: Fatal: Error in configuration file /etc/dovecot/conf.d/10-master.conf line 64: Invalid size: $default_vsz_limit After fixing that, this is what I get for the config: root@bilbo:/etc/dovecot# doveconf -n # 2.3.16 (7e2e900c1a): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.16 (09c29328) # OS: Linux 5.11.0-1020-aws x86_64 Ubuntu 20.04.3 LTS ext4 # Hostname: REDACTED default_vsz_limit = 1 G dict { quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } first_valid_uid = 150 last_valid_uid = 150 listen = * mail_gid = mail mail_location = maildir:/var/vmail/%d/%u mail_plugins = " quota fts fts_solr" mail_privileged_group = mail mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { fts = solr fts_autoindex = yes fts_solr = url=http://localhost:8983/solr/dovecot/ quota = dict:User quota::proxy::quota quota_grace = 10%% quota_max_mail_size = 100M quota_status_nouser = DUNNO quota_status_overquota = 552 5.2.2 Mailbox is full quota_status_success = DUNNO sieve = file:/var/vmail/sieve/%d/%u/sieve;active=/var/vmail/sieve/%d/%u/.dovecot.sieve sieve_default = file:/var/vmail/global.sieve } postmaster_address = postmaster@REDACTED protocols = " imap lmtp sieve pop3" service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { group = mail mode = 0666 user = vmail } } service dict { unix_listener dict { group = postfix mode = 0660 user = vmail } } service imap { vsz_limit = 1 G } service stats { unix_listener stats-reader { group = vmail mode = 0660 user = vmail } unix_listener stats-writer { group = vmail mode = 0660 user = vmail } } ssl_cert = </etc/ssl/certs/local/mainwildcards.pem ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lmtp { mail_plugins = " quota fts fts_solr sieve" } protocol lda { mail_plugins = " quota fts fts_solr sieve" } protocol imap { mail_max_userip_connections = 20 mail_plugins = " quota fts fts_solr imap_quota" } I repeated the delete test after fixing the config and there was no change -- the delete request never made it to Solr. Reindexing completely for my dovecot install takes about 8 minutes. I have less than 150K messages. Reindexing for some could take hours or days. This is what I use to initiate a full reindex: root@bilbo:~# cat /usr/local/sbin/reindexsolr #!/bin/sh DOVECOT_SOLR_URL_BASE="http://localhost:8983/solr/dovecot" DEL_ALL_QUERY_XML="<delete><query>*:*</query></delete>" MAIL_ROOT=/var/vmail service dovecot stop curl \ "${DOVECOT_SOLR_URL_BASE}/update?commit=true&optimize=true" \ -H "Content-Type: text/xml" \ --data-binary "${DEL_ALL_QUERY_XML}" find ${MAIL_ROOT} -name "dovecot.index*" -print0 | xargs -0 rm -f service dovecot start doveadm index -A -q '*'

3 11

FTS config - cannot search substrings
by infoomatic 18 Nov '21

18 Nov '21

Hi, I have an issue with FTS, maybe someone here can help me out. Basically my setup is working with solr + tika enabled, however, it seems my search is not configured the way I expect it. Eg when I receive a pdf document containing the word "myfoobar" I cannot search for "myfoo" or "foobar", but searching for "myfoobar" shows the right result. Any ideas? Thanks!

2 2

Can sieve-script run when saving mail in Sent folder?
by Kees van Vloten 18 Nov '21

18 Nov '21

Hi everyone, Would it be possible to run a sieve-script when the mail-client sends a mail (no because smtp) on the action where the mail-client stores a copy in the Sent folder over imap? I was expecting through imapsieve_mailboxXXX_* but can find a combination of parameters are correct. Does anybody have this working? and how? - Kees

3 4

Subfolder in sieve not working as expected
by Kees van Vloten 17 Nov '21

17 Nov '21

Hi everybody, I am trying to move incoming mails into subfolders with this sieve script: require ["fileinto", "variables", "mailbox"]; if header :matches "Delivered-To" "*@*" { fileinto :create "INBOX/${2}"; } On a message to user(a)example.com I would expect it to be stored in the folder 'example.com' under 'INBOX', but instead a folder named 'INBOX\2fexample.com' is created. I tried several options: - replace / with \/ or \\/ - replace / with . - disable listescape in dovecot.conf Unfortunately nothing did the trick. Any ideas what could fix this? dovecot -n # 2.3.13 (89f716dc2): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.13 (cdd19fe3) # OS: Linux 5.10.0-9-amd64 x86_64 Debian 11.1 # Hostname: servers.example.com auth_default_realm = EXAMPLE.COM auth_gssapi_hostname = servers.example.com auth_krb5_keytab = /etc/keytab/dovecot.keytab auth_master_user_separator = * auth_mechanisms = gssapi gss-spnego plain auth_realms = EXAMPLE.COM first_valid_gid = 986 first_valid_uid = 990 imap_client_workarounds = tb-extra-mailbox-sep login_greeting = Dovecot ready. mail_debug = yes mail_gid = 986 mail_location = maildir:%h/%d/%n/store:LAYOUT=fs:FULLDIRNAME=0_FolderContent:UTF-8:INDEX=%h/%d/%n/index:CONTROL=%h/%d/%n/control:VOLATILEDIR=%h/%d/%n/volatile mail_plugins = zlib notify push_notification listescape acl mail_privileged_group = mail mail_shared_explicit_inbox = yes mail_uid = 990 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext vacation-seconds spamtest imapflags notify imapsieve vnd.dovecot.imapsieve namespace inbox { inbox = yes list = yes mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Inbox { auto = subscribe } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = ~ subscriptions = yes type = private } passdb { args = username_format=%u /etc/dovecot/master-users driver = passwd-file master = yes } passdb { args = /etc/dovecot/ldap_user_to_principal.conf.ext driver = ldap pass = yes } passdb { driver = pam } plugin { acl = vfile:/var/lib/dovecot/global-acls:cache_secs=1 acl_shared_dict = file:/var/lib/dovecot/shared-mailboxes autocreate = Trash autocreate2 = Drafts autocreate3 = Sent autosubscribe = Trash autosubscribe2 = Drafts autosubscribe3 = Sent imapsieve_mailbox1_before = file:/var/lib/dovecot/sieve/imap/learn-spam.sieve imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_name = Junk imapsieve_mailbox2_before = file:/var/lib/dovecot/sieve/imap/learn-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = Junk imapsieve_mailbox2_name = * listescape_char = \ sieve = file:%h/%d/%n/sieve/sieve;active=%h/%d/%n/sieve/active.sieve sieve_after = /var/lib/dovecot/sieve/after sieve_before = /var/lib/dovecot/sieve/before sieve_default = /var/lib/dovecot/sieve/default.sieve sieve_extensions = +vacation-seconds +reject +notify +imapflags +spamtest sieve_global_extensions = +vnd.dovecot.pipe sieve_pipe_bin_dir = /var/lib/dovecot/sieve/bin sieve_plugins = sieve_imapsieve sieve_extprograms sieve_vacation_default_period = 1d sieve_vacation_max_period = 1d sieve_vacation_min_period = 10s sieve_vacation_use_original_recipient = yes } postmaster_address = sysadmin(a)cvanvloten.nl protocols = " imap lmtp sieve submission" service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } service imap-login { inet_listener imap { port = 0 } inet_listener imaps { port = 993 ssl = yes } process_min_avail = 1 } service imap { executable = imap } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service submission-login { inet_listener submission { port = 465 ssl = yes } } ssl = required ssl_cert = </etc/dovecot/private/dovecot.pem ssl_cipher_list = EECDH+AESGCM:EDH+AESGCM ssl_client_ca_dir = /etc/ssl/certs ssl_curve_list = X25519:secp521r1:secp384r1:prime256v1 ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it ssl_min_protocol = TLSv1.2 submission_client_workarounds = whitespace-before-path submission_relay_host = server.example.com submission_relay_password = # hidden, use -P to show it submission_relay_ssl = smtps submission_relay_user = dovecot@server userdb { args = /etc/dovecot/ldap_list_users.conf.ext driver = ldap result_success = continue } userdb { args = /etc/dovecot/ldap_user_domain.conf.ext driver = ldap result_success = continue } userdb { args = uid=vmail gid=vmail home=/srv/mail/vmail allow_all_users=yes driver = static } protocol lmtp { mail_plugins = zlib notify push_notification listescape acl sieve } protocol imap { mail_max_userip_connections = 50 mail_plugins = zlib notify push_notification listescape acl imap_acl imap_sieve } - Kees

3 4

Bug report: Global Mailbox UID mismatch on replication when upgrading from 2.3.13 to 2.3.17
by Dennis Højgaard | Powerhosting Support 17 Nov '21

17 Nov '21

Hi. We use dovecot for replication on two dovecot nodes (using Maildir). Worked perfectly until upgrading from 2.3.13 to 2.3.17. I found out about it a couple of days after where Queued 'full resync' requests had gone up. And then i saw some failed requests. But nothing in the log. I found out which users having trouble and saw that it was new user so i ran doveadm sync manually and got the following error: doveadm(2849817): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID 38e69e0ba98d936142ac29004e3fea09 (maybe it was just deleted?) So i found out that dovecot-uidlist had a mismatch on the Global Mailbox UID, so global mailbox uid is not the same on both servers for new users. I don't know if there is something that has changed with regards to how this GUID is generated, because when i look at old mailboxes they are always identical on both servers. I fix this issue manually by editing dovecot-uidlist on one of the servers making the global mailbox uid on both servers identical, and then run force-resync. This works, but i guess something has changed. -- Dennis Højgaard Powerhosting Support t: +45 7222 4457 | e: dh(a)powerhosting.dk | w: https://powerhosting.dk

1 0

Postfix -> bogogilter -> Dovecot -> Sieve
by Mike 17 Nov '21

17 Nov '21

I've been trying to work out how to get postfix to accept mail, send it to bogofilter, then deliver using dovecot while allowing a global sieve filter and users able to filter mail based on the bogofilter header. I've been successful at getting it to add the bogofilter header as needed, but don't understand what I need to do in order to have it get routed to sieve. Anyone do anything like this already and have a working config? I THINK I need to make dovecot deliver mail locally using lmtp or lda, but I'm not exactly sure. Thanks for any thoughts or ideas that you might have to accomplish this. Mike.

3 2

Help needed for OAuth and WSO2 Identity Server
by Tyler Montney 16 Nov '21

16 Nov '21

As outlined here: https://doc.dovecot.org/configuration_manual/authentication/oauth2/ Can Postman https://identity.foo.mylocal:9443/oauth2/token OK. Using this command to generate the base64 token: echo -en 'n,a=test(a)foo.com,\001host=localhost\001port=143\001auth=Bearer S3cure!Password\001\001' | base64 -w0; echo I telnet to localhost 143, and run 01 AUTHENTICATE OAUTHBEARER {TOKEN}. Get 'User id is not available for user: FOO.MYLOCAL/test(a)carbon.super' (HTTP 500). It could be because Dovecot is just sending a username instead of the full email address? I can generate the same 500 error by just sending the username in Postman. ***dovecot-oauth.conf.ext*** introspection_mode = post introspection_url = https://adminusername:adminpassword@identity.foo.mylocal :9443/oauth2/introspect username_attribute = username tls_allow_invalid_cert = yes active_attribute = active active_value = true use_grant_password = yes # Have tried this, no change. #username_format = %n client_id = {CLIENTID} client_secret = {CLIENTSECRET} grant_url = https://identity.foo.mylocal:9443/oauth2/token tokeninfo_url = https://identity.foo.mylocal:9443/oauth2/tokeninfo?oauth= pass_attrs = pass=%{oauth2:access_token} ***dovecot.conf*** auth_mechanisms = $auth_mechanisms oauthbearer xoauth2 passdb { driver = oauth2 mechanisms = xoauth2 oauthbearer args = /etc/dovecot/dovecot-oauth2.conf.ext }

1 0

Dovecot and certificates/keys on PKCS11 smartcards/HSMs
by Graham Leggett 16 Nov '21

16 Nov '21

Hi all, To configure SSL on dovecot using certificates and keys in files on disk, you specify the following: # Preferred permissions: root:root 0400 ssl_cert = </etc/ssl/dovecot.pem ssl_key = </etc/ssl/dovecot.pem Is there a syntax that will allow a certificate and key to be specified on a smartcard/HSM, as is possible (for example) for httpd at https://httpd.apache.org/docs/trunk/mod/mod_ssl.html#sslcertificatefile: # Example use of a certificate and private key from a PKCS#11 token: SSLCertificateFile "pkcs11:token=My%20Token%20Name;id=45” Regards, Graham —

1 0