September 2022 - dovecot

Warning sqlpool(mysql): Query failed
by Alessio Cecchi 27 Jun '23

27 Jun '23

Hi, with the last dovecot version (2.3.13) I see many of this warnings via "doveadm log errors": Feb 17 03:26:28 Warning: dict(31513): conn unix:dict (pid=31330,uid=95): dict(mysql)<email1(a)company1.com>: sqlpool(mysql): Query failed, retrying: MySQL server has gone away (idled for 679 secs) Feb 17 03:48:53 Warning: dict(31013): conn unix:dict (pid=31805,uid=95): dict(mysql)<email2(a)company2.com>: sqlpool(mysql): Query failed, retrying: MySQL server has gone away (idled for 685 secs) Feb 17 03:54:12 Warning: dict(32529): conn unix:dict (pid=3147,uid=95): dict(mysql)<email3(a)company3.com>: sqlpool(mysql): Query failed, retrying: MySQL server has gone away (idled for 670 secs) What do they mean? I thinks are related to inactivity timeout on mysql server (where I have a wait_timeout=600), but these warnings could be problems or can be fixed with some setting in dovecot? My dovecot configuration about dict: dict { acl = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext sieve = mysql:/etc/dovecot/dovecot-dict-sieve-sql.conf.ext sqlquota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } plugin { [...] acl_shared_dict = proxy::acl expire_dict = proxy::expire quota2 = dict:Quota Usage::noenforcing:proxy::sqlquota sieve = file:~/sieve;active=~/.dovecot.sieve sieve_before = dict:proxy::sieve;name=activesql [...] } service dict { process_limit = 500 unix_listener dict { group = vmail mode = 0660 user = vmail } } Thanks -- Alessio Cecchi Postmaster @ http://www.qboxmail.it https://www.linkedin.com/in/alessice

2 1

dovecot-fts-solr Solr9 support
by PGNet Dev 24 Apr '23

24 Apr '23

I run dovecot-2.3.18 , & am deploying Solr search with 'dovecot-fts-solr' I run Solr standalone. When creating a dedicated 'dovecot' core, cp'ing from Dovecot's included solr schemas ls -al /usr/share/doc/dovecot/solr-* -rw-r--r-- 1 root root 12K Dec 3 06:48 /usr/share/doc/dovecot/solr-config-7.7.0.xml -rw-r--r-- 1 root root 2.7K Dec 3 06:48 /usr/share/doc/dovecot/solr-schema-7.7.0.xml -rw-r--r-- 1 root root 2.9K Dec 3 06:48 /usr/share/doc/dovecot/solr-schema.xml is tested only for Solr 7.7.0 https://doc.dovecot.org/configuration_manual/fts/solr/ "The steps described in this page are tested for Solr 7.7.0. For other versions, this these steps may need to be adjusted." which, per solr docs https://solr.apache.org/downloads.html About versions and support Version Description 9.x Current major version for feature releases (STABLE) 8.11.x Previous major version may sometimes receive critical bugfix releases 10 Next major version, yet to be released (UNSTABLE) <8.11 All older versions are End Of Life (EOL) is EOL'd and unsupported. In the dovecot 7.7.0 schema, FastLRUCache/LRUCache are used as the caching class(es), grep class=.*LRU /usr/share/doc/dovecot/solr-config-7.7.0.xml <filterCache class="solr.FastLRUCache" <queryResultCache class="solr.LRUCache" <documentCache class="solr.LRUCache" class="solr.search.LRUCache" For Solr 9, cat CHANGES.txt Upgrade Notes --------------------- ... * SOLR-13817: Deprecate legacy SolrCache implementations. Users are encouraged to transition their configurations to use org.apache.solr.search.CaffeineCache instead. (ab) running solr9 + dovecot-fts-solr search, based on the 7.7.0 schemas, results in errors, failed scan & search. Modifying the config to use CaffeineCache cp solr-config-7.7.0.xml solr-config-9.0.0.xml perl -pi -e '\ s|solr.FastLRUCache|solr.CaffeineCache|g; \ s|solr.LRUCache|solr.CaffeineCache|g; \ s|solr.search.LRUCache|solr.search.CaffeineCache|g;' \ solr-config-9.0.0.xml appears to do the trick. Atm, indexing & search -- both manually triggered, and on delivery receipt to Dovecot store -- is working with no errors (yet). I've not tested exhaustively yet, & don't know if this Cache class change in the schema is both necessary & sufficient for dovecot-fts-solr search use with solr 9.0.0 Given that for Solr "<8.11 All older versions are End Of Life (EOL)" it'd be useful for Dovecot to 'retest' for current, not-EOL'd Solr, and publish/package compatible schemas accordingly.

5 8

mbox to maildir migration, same server?
by kjohnson＠eclypse.org 20 Apr '23

20 Apr '23

What is the recommended procedure to migrate from mbox to maildir on a server already running dovecot? Ideally, this procedure would convert messages in an existing mbox INBOX and in ~/mail mboxes (as defined in my dovecot configuration) to storage in a ~/Maildir. Some of the INBOXs are rather large -- 100's of MB (one of the reasons for migrating). What I have tried has not worked for me. Below, I describe what I tried, how I tested before I tried, what went wrong, and the current setup (pre-migration). What I tried: ============= stop exim4 service dovecot stop Make dovecot listen only locally, not to external connections. /etc/dovecot/local.conf protocol imap { listen = 127.0.0.1:143 ssl_listen = 127.0.0.1:993 } protocol pop3 { listen = 127.0.0.1:110 ssl_listen = 127.0.0.1:995 } Modify dovecot configuration for maildir: Watch capitalization! #in 10-mail.conf: mail_location = maildir:/home/%u/Maildir namespace inbox { separator = / } dovecot -n, to confirm results and perform limited syntax checking. service dovecot start for each user, in a bash script: dsync -v -u "$U" mirror "mbox:/home/$U/mail:INBOX=/var/mail/$U" (if the conversion worked for all users, then reconfigure exim to use maildir and restart it; reconfigure dovecot to listen to external interfaces and restart.) How I tested: ============= Before trying this on the real server, I set up a matching (same versions of Debian, Exim, and Dovecot) VM using mbox, with a couple of accounts. One using POP3, one using IMAP. The IMAP account was accessed from Claws and from Outlook. The test accounts had perhaps a dozen messages on the server. I tried the conversion procedure, and everything I tried worked flawlessly. (Yes, next time I will move some real inbox files from the live server). What went wrong: ================ I encountered two types of problems here. The first was several dsync warnings (username changed): dsync(example): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID 21bfa11a81f11763ac150000d09efc50 (maybe it was just deleted?) As I understand it, this can result in some or all messages being downloaded again by clients. My understanding may be wrong. I also don't understand how the mailbox could have changed -- exim4 was stopped, dovecot is only listening on 127.0.0.1, and I am the only user who uses a local mail client on that system, and this was not my account. The second problem I encountered seems to match Debian Bug report logs - #1011238. In part of the discussion of that report, the reporter mentions adding -1 to the command, as shown: dsync -v -u "$U" mirror -1 "mbox:/home/$U/mail:INBOX=/var/mail/$U" While this seemed to eliminate the dsync panic of #1011238, as far as I can tell, no old messages were actually moved from the old mbox files to the new maildirs. At that point I restored the backup mbox files and undid the changes to dovecot and exim4. Current setup: ============== Debian 11.4 Exim 4.94.2-7 dovecot 1:2.3.13+dfsg1-2 The current setup with Dovecot was a successful migration from IMAP-UW in December 2020. Output from dovecot -n (the list of trusted networks has been removed) # 2.3.13 (89f716dc2): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.13 (cdd19fe3) # OS: Linux 5.10.0-16-686-pae i686 Debian 11.4 # Hostname: localhost auth_verbose = yes mail_location = mbox:~/mail:INBOX=/var/mail/%u mail_privileged_group = mail namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } pop3_uidl_format = %08Xv%08Xu protocols = " imap pop3" ssl_cert = </etc/dovecot/dovecot.pem ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = passwd } Thank you to anyone who read this far. Ken

4 4

decrypt.rb
by Dave 11 Apr '23

11 Apr '23

Per the Dovecot site here: https://wiki.dovecot.org/Plugins/MailCrypt ... the "decrypt.rb" ruby script can be used to decrypt a Dovecot-encrypted message file from the command line. The script sort of runs successfully for me, in the sense that it doesn't error out, but it doesn't show the decrypted message. I've called it like so: decrypt.rb -k <PATH_TO_DOVECOT_KEYFILE> -f ./encrypted_message But when I do it just displays the info below and exits (I've subbed <DATA> tags for the actual data): ------------------[OUTPUT]----------------------- Key(s) (total: 1) - Key type : EC - Key digest: <DATA1> - Peer key : <DATA2> - Encrypted : <DATA3> - Kd hash : <DATA4> ------------------------------------------------------- I've tried specifying an output file as well, per the script's command line options, but the output file is 0 bytes. Does anyone have any suggestions? I *think* I'm using it the way it's intended to be used, but maybe I'm not?! -Dave

5 14

2.3.14: stale imap-login processes due to bogus user setup tying up process limit
by Volker Stolz 16 Dec '22

16 Dec '22

I just dealt with an interesting Dovecot issue on our small server, and thought I might share it. I also feel that although in the end it was a configuration error, Dovecot should somehow avoid leaving this processes around and hence filling up the process limit quickly. - set up Dovecot IMAP on Debian buster (either with the older 2.3.x Debian package, or 2.3.14 from the dovecot-repo) - config: ... namespace inbox { inbox = yes location = ... default settings ... ... passdb { args = scheme=HMAC-MD5 username_format=%u /etc/dovecot/users driver = passwd-file } protocols = " imap" ... userdb { args = blocking=no driver = passwd } ... - secret sauce: have a user in the passdb that doesn't have a physical account on the system yet (we were migrating to a new machine). The log-file shows successful authentication for that user (after we flipped DNS to point to this new machine), but over time (2-3 times/day) we were left with an increasing number of `imap-login` processes in state `pre-login`, eventually requiring a dovecot restart. Dialling up process_limits didn't seem to help, we'd just get more of these stale processes. After creating the user on the system, the behaviour reverted to normal, no more stale processes. I do not know what error message that user received on their end, but since we announced we were migrating, they probably just ignored it and thought it as part of the transition. Best, Volker (a happy & long Dovecot user)

3 4

Doveadm Move Query
by Simon B 01 Dec '22

01 Dec '22

I have a production Dovecot problem and although I searched the mailing lists, I could not find an answer and I hope you can give me a quick answer/pointer in the right direction. I have mails for a user (user(a)domain.net) under /var/spool/mail/virtual/ domain.net/user and I want to move ALL the mails to /var/spool/mail/virtual/domain.com/user If I use #doveadm -Dv move -u user(a)domain.net Maildir:/var/spool/mail/virtual/ domain.net/user Maildir:/var/spool/mail/virtual/domain.com/user ALL I get doveadm(root): Fatal: Unknown argument MAILDIR:/var/spool/mail/virtual/ domain.com/user if I use #doveadm -Dv move -u user(a)domain.net Maildir:/var/spool/mail/virtual/ domain.net/user /var/spool/mail/virtual/domain.com/user ALL doveadm(root): Fatal: Unknown argument /var/spool/mail/virtual/ domain.com/user What the hell am I doing wrong!? :) Thanks. Simon

3 3

Catch all for Metadata storage in SQL database
by Steffen Kaiser 27 Oct '22

27 Oct '22

Hi, the https://doc.dovecot.org/configuration_manual/imap_metadata/ sample uses mail_attribute_dict = file:%h/Maildir/dovecot-attributes which stores all keys=value pairs in the file. http://dovecot.2317879.n4.nabble.com/Dovecot-v2-3-9-3-HTTP-API-Endpoint-for… uses a SQL dict, however very specific ones. How can I use a SQL dict to store _all_ keys, as with a file based storage? I cannot find a documentation for "pattern" specification, that works as "catch all", in order to store anything not catched by patterns into the database. https://wiki.dovecot.org/Dictionary does not give any hint (in my eyes). -- Steffen Kaiser

3 4

dovecot mailing list (this mailing list), DKIM, SPF and DMARC
by Lev Serebryakov 22 Oct '22

22 Oct '22

My domain (serebrtyajov.spb.ru) has all these "new" e-mail technologies configured. It works fine till I write to this mailing list. After that I've got several DMARC reports about "spam" from my domain. All these reports are about my mailing list post. I don't have such problems with other mailing lists (FreeBSD ones, OpenJDK ones, and others). Looks like mailing list software for this mailing list is misconfigured. I'm sure, I'll get new after this message. -- // Black Lion AKA Lev Serebryakov

15 38

Unseen field reported by imap status command returns wrong count for shared mailboxes on dovecot cluster
by Nikolaos Pyrgiotis 19 Oct '22

19 Oct '22

Hello, We recently migrated our mail server to a dovecot cluster of 5nodes, a dovecot proxy, 2directors and 2dovecot backends. All dovecot nodes run version 2.19.1. We use a glusterfs mounted volume on the backends forthe mail storage. We noticed that when issuing the IMAP command to check forUNSEEN messages formails on the shared namespace instead of seeing the value of the unseen messages of the user that the mailbox has been shared to, the value of the unseen messages of the owner of the mailbox is returned. This behavior causes thunderbird when getting new messages forshared mailboxes, to show briefly all messages of the mailbox as unseen forthe user before showing the correct unseen count forthe shared mailbox. The mail location of the shared mailboxes is defined with imapc storage type as described inthe documentation. Is this a dovecot bug? Can we configure a different dovecot configuration setting so that imap status command reports the correct unseen field count when mail location is an imapc storage location? An examples is given below when running doveadm command from one the 2 dovecot directors: When issuing the status command we see that the unseen count is 2: root@doved0-rmt0-cn1:/etc/cron.d# doveadm mailbox status -u npyrgiotis all shared.sysadmins doveadm(npyrgiotis): Info: remote(10.101.0.71:8080): doveadm(npyrgiotis(a)domie02.com)<19078><9l8IJTb+GWOGSgAAEU9A+w>: imapc(10.101.0.75:143): Connected to 10.101.0.75:143 (local 10.101.0.71:33476) shared.sysadmins messages=2 recent=0 uidnext=3 uidvalidity=1662640492 unseen=2 highestmodseq=3 vsize=3950 guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=1662643853 shared.sysadmins messages=2recent=0uidnext=3uidvalidity=1662640492unseen=2highestmodseq=3vsize=3950guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=1662643853 But when fetching the emails of the mailboxes we can see that the \Seen flag is set for both emails root@doved0-rmt0-cn1:/etc/cron.d# doveadm fetch -u npyrgiotis flags mailbox shared.sysadmins ALL doveadm(npyrgiotis): Info: remote(10.101.0.71:8080): doveadm(npyrgiotis(a)domie02.com)<19074><e7rWDgj+GWOCSgAAEU9A+w>: imapc(10.101.0.75:143): Connected to 10.101.0.75:143 (local 10.101.0.71:38750) flags: \Seen flags: \Seen Below i post the dovecot configuration of one of the two dovecot backends: # 2.3.19.1 (9b53102964): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.19 (4eae2f79) # OS: Linux 5.10.0-17-amd64 x86_64 Debian 11.4 # Hostname: doveb0-rmt0-cn1 auth_cache_negative_ttl = 5mins auth_cache_size = 50M auth_debug = yes auth_default_realm = example.com auth_master_user_separator = * auth_mechanisms = plain login auth_verbose = yes auth_worker_max_count = 16 disable_plaintext_auth = no first_valid_uid = 499 hostname = smtp.example.com imapc_features = fetch-bodystructure fetch-headers rfc822.size search modseq acl delay-login imapc_host = 10.101.0.75 imapc_password = # hidden, use -P to show it imapc_sasl_mechanisms = plain login imapc_ssl = starttls imapc_ssl_verify = no last_valid_uid = 499 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes login_greeting = You have successfully logged into example.com IMAP server login_trusted_networks = 10.101.0.73 10.101.0.74 mail_always_cache_fields = flags hdr.* date.received date.sent mail_cache_fields = flags date.received guid size.physical size.virtual imap.bodystructure body.snippet mail_debug = yes mail_fsync = always mail_plugins = " notify mail_log zlib acl" mail_privileged_group = mail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 200M mmap_disable = yes namespace example { list = children location = imapc:~/shared/%%n:INDEXPVT=~/shared-pvt/%%n prefix = shared.%%n. separator = . subscriptions = no type = shared } namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam.Virus { auto = subscribe } mailbox Spam { auto = subscribe } mailbox Trash { special_use = \Trash } prefix = separator = . subscriptions = yes type = private } passdb { args = password=#hidden_use-P_to_show# driver = static master = yes override_fields = allow_real_nets=127.0.0.0/8,local,10.101.0.73,10.101.0.74 userdb_acl_user=%{auth_user} userdb_namespace/example/disabled=yes result_success = continue } passdb { args = password=#hidden_use-P_to_show# driver = static override_fields = allow_real_nets=127.0.0.0/8,local,10.101.0.73,10.101.0.74 result_success = continue } passdb { args = /etc/dovecot/dovecot-ldap-users.conf.ext driver = ldap } passdb { args = /etc/dovecot/dovecot-ldap-groups.conf.ext driver = ldap } plugin { acl = vfile:/etc/dovecot/global-acls/%d/master-dovecot-acl:cache_secs=300 acl_defaults_from_inbox = yes acl_globals_only = no acl_shared_dict = file:/var/dovecot-acl/%d/shared-mailboxes.db imap_compress_deflate_level = 6 mail_log_events = flag_change mail_log_fields = uid box flags sieve = file:~/sieve;active=~/sieve/%n.sieve sieve_before = /var/vmail/global-sieve/global.sieve sieve_env_email_address = %u sieve_env_mailbox_prefix = %{userdb:mailbox_prefix} sieve_global = /var/vmail/sieve sieve_global_extensions = +editheader +vnd.dovecot.environment zlib_save = zstd zlib_save_level = 5 } protocols = " imap lmtp sieve submission" service auth-worker { user = dovecot } service auth { unix_listener auth-userdb { group = mail mode = 0666 user = vmail } } service doveadm { inet_listener dovadm-dom-mgmt { address = 10.100.1.71 port = 8080 ssl = no } inet_listener dovadm-dom { address = 10.101.0.71 port = 8080 ssl = no } vsz_limit = 256M } service imap-login { inet_listener imap-dom { address = 10.101.0.71 port = 143 ssl = no } inet_listener imap { port = 0 } inet_listener imaps-dom { address = 10.101.0.71 port = 993 ssl = yes } inet_listener imaps { port = 0 } process_limit = 400 process_min_avail = 2 service_count = 20 vsz_limit = 512M } service imap { process_limit = 2048 process_min_avail = 0 service_count = 1 vsz_limit = 4G } service lmtp { inet_listener lmtp-dom { address = 10.101.0.71 port = 24 ssl = no } process_limit = 0 process_min_avail = 0 service_count = 0 unix_listener lmtp { mode = 0666 } vsz_limit = 256M } service managesieve-login { inet_listener sieve-dom { address = 10.101.0.71 port = 4190 ssl = no } inet_listener sieve { port = 0 } process_limit = 0 process_min_avail = 0 service_count = 0 vsz_limit = 64M } service managesieve { process_limit = 1024 process_min_avail = 0 service_count = 1 vsz_limit = 256M } service submission-login { inet_listener submission-dom { address = 10.101.0.71 port = 587 ssl = no } inet_listener submission { port = 0 } inet_listener submissions-dom { address = 10.101.0.71 port = 465 ssl = yes } process_limit = 250 process_min_avail = 2 service_count = 1 vsz_limit = 256M } service submission { process_limit = 1024 process_min_avail = 0 service_count = 1 vsz_limit = 256M } ssl = required ssl_cert = </etc/ssl/certs/doveb0-rmt0-cn1.mail.rmt.example.com.pem ssl_client_ca_dir = /etc/ssl/certs ssl_key = # hidden, use -P to show it submission_max_mail_size = 30000k submission_relay_host = 10.101.0.7 submission_relay_port = 587 submission_relay_ssl = starttls submission_relay_ssl_verify = no submission_relay_trusted = yes userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-ldap-users.conf.ext driver = ldap } userdb { args = /etc/dovecot/dovecot-ldap-groups.conf.ext driver = ldap } protocol lmtp { mail_plugins = " notify mail_log zlib acl sieve acl" } protocol lda { mail_plugins = " notify mail_log zlib acl sieve acl" } protocol imap { mail_max_userip_connections = 100 mail_plugins = " notify mail_log zlib acl" } protocol sieve { mail_max_userip_connections = 40 } protocol submission { mail_max_userip_connections = 70 } local10.101.0.71 { doveadm_allowed_commands = doveadm_password = # hidden, use -P to show it } local10.100.1.71 { doveadm_allowed_commands = doveadm_password = # hidden, use -P to show it } Thank you, Nikolaos Pyrgiotis

2 3

Dovecot mail-crypt webmail can't read encrypted messages
by Serveria Support 18 Oct '22

18 Oct '22

I'm trying to setup Dovecot with mail-crypt plugin with per-user encryption. I have configured mail-crypt plugin as per official guide here: https://doc.dovecot.org/configuration_manual/mail_crypt_plugin/ After that I created a user and an encrypted key by running this command: doveadm -o \plugin/mail_crypt_private_password=12345 mailbox cryptokey generate -u mail(a)example.org -URf (replacing dummy data ofc) I can log in to webmail (and Dovecot) just fine, emails are getting sent and delivered. I have also checked the storage and the messages seem to be stored encrypted. However, I can't read the emails in webmail (just headers can be seen) and in Dovecot logs I can see the following error: failed: Private key not available: Cannot decrypt key ### Cannot decrypt key ### <8632: Password not available (FETCH RFC822.HEADER) There seems to be an issue with mySQL query. The query I'm using (Select username as "user", password,"%w" as userdb_mail_crypt_private_password from mailbox;) seems to work just fine, when run from mysql prompt it outputs the usernames and passwords, but the error is still there (Cannot decrypt key ### Password not available). Any ideas? What am I missing?

16 70