February 2018 - dovecot

Really slow IMAP performance
by Neil Jerram 26 Feb '18

26 Feb '18

Please could you help me to understand and fix why my dovecot IMAP performance is so bad? I've read through a lot of the performance-related material on the website, but I don't think that any of it could account for slowness at the level that I am seeing. The simplest scenario is moving a message from my Inbox to another IMAP folder. Using Gnus as the client, the whole UI freezes for about 2 minutes (which I assume is until the move is complete), and journalctl on the dovecot server says: Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: Transaction log file /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked for 98 seconds (Mailbox was synchronized) or the same message with (rotating while syncing). There must be something badly wrong in my setup, or perhaps in the spec of the server that dovecot is running on. What should I look at to start understanding this better? Many thanks - Neil

6 11

move more than one subvolder doesn't works
by Arno Schneider 26 Feb '18

26 Feb '18

Our setup: 2 replicated dovecot 2.2.33.2 servers thunderbird client When I select more than one mail subfolder in thunderbird and drag & drop them to another subfolder only one subfolger is moved. In the following I can't move another mail folder to the first target. Any ideas why this happens? best regards, arno

1 0

understanding dovecot director passdb configuration
by Kalyana sundaram 26 Feb '18

26 Feb '18

Hey All I am very new to dovecot ecosystem. Found the software really robust and secure. Kudos to the team!!! We are setting up dovecot imap servers sharing a single nfs mount point. So to avoid nfs cache issues, we are setting up dovecot director. We are using dovecot version 2.2.10. While going through the documentation of dovecot director I stumbled across the following lines in passdb configuration https://wiki2.dovecot.org/Director "Note that while this is the simplest director configuration, users will be assigned to a backend before they have been authenticated. A director configured this way can be attacked by sending it a large number of unknown users. To prevent this, the director should be configured to authenticate the user and might make use of a master password to log into the backend servers." I understand on static passdb config dovecot assigns a user to a machine in the list of backends by using md5(username)%number_of_mail_servers. But other than this calculation it does not incur any other resources. It does have tcp connection with the system which is trying to do bruteforce. If we move to authenticating users directly at the director server, the director servers imap-login director service should be anyways loaded on an attack. Is it anything to do that the imap-login will contact auth process asynchronously and keep itself free? I am pretty sure I am overlooking some point on the above statement. Can somebody throw some light on that? -- Kalyanasundaram http://blogs.eskratch.com/ https://github.com/kalyanceg/

2 1

imap-login director is not restarted after each auth request
by Kalyana sundaram 26 Feb '18

26 Feb '18

Hi imap-login by default in mail server is restarted after every login request keeping security in mind to avoid leak of critical data. But in a director config, the imap-login director process is not restarted on each auth request. Can somebody explain why this design decision is taken? -- Kalyanasundaram http://blogs.eskratch.com/ https://github.com/kalyanceg/

1 0

dovecot solr and users
by David Mehler 26 Feb '18

26 Feb '18

Hello, I'm trying to get fts searching done with dovecot. I keep getting the below error. Any help appreciated? Thanks. Dave. #doveadm index -A inbox doveadm(user(a)example.com): Info: User no longer exists, skipping #doveconf -n # 2.2.33.2 (d6601f4ec): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.21 (92477967) # OS: FreeBSD 11.1-RELEASE-p4 amd64 auth_cache_size = 10 M auth_default_realm = example.com auth_realms = example.com example2.com dict { acl = mysql:/usr/local/etc/dovecot/shared-folders.conf sqlquota = mysql:/usr/local/etc/dovecot/quota.conf } first_valid_gid = 999 first_valid_uid = 999 hostname = mail.example.com imap_idle_notify_interval = 10 mins last_valid_gid = 999 last_valid_uid = 999 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes listen = 127.0.0.1 lmtp_rcpt_check_quota = yes mail_access_groups = vmail mail_fsync = never mail_gid = vmail mail_home = /home/vmail/mailboxes/%d/%n mail_location = maildir:~/mail:LAYOUT=fs mail_plugins = acl mail_log notify quota quota_clone trash virtual welcome zlib fts fts_solr mail_privileged_group = vmail mail_server_admin = mailto:postmaster@example.com mail_uid = vmail mailbox_list_index = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext imapflags notify imapsieve vnd.dovecot.imapsieve namespace { hidden = no list = yes location = maildir:/home/vmail/public/:LAYOUT=fs:CONTROL=~/mail/public:INDEXPVT=~/mail/public:INDEX=~/mail/public mailbox TestFolder { auto = subscribe comment = Public Folder for message sharing } prefix = public/ separator = / subscriptions = yes type = public } namespace { list = yes location = maildir:~/mail/:INDEX=~/mail/shared/%%Ld/%%Ln prefix = shared/%%u/ separator = / subscriptions = yes type = shared } namespace { location = virtual:/usr/local/etc/dovecot/virtual mailbox All { auto = subscribe comment = All my messages special_use = \All } prefix = virtual/ separator = / } namespace inbox { inbox = yes location = mailbox Archive { auto = no special_use = \Archive } mailbox Archives { auto = subscribe special_use = \Archive } mailbox "Deleted Messages" { auto = no autoexpunge = 30 days special_use = \Trash } mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = no autoexpunge = 30 days special_use = \Junk } mailbox "Junk E-mail" { auto = no autoexpunge = 30 days special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Items" { auto = no special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Spam { auto = subscribe autoexpunge = 30 days special_use = \Junk } mailbox Trash { auto = subscribe autoexpunge = 30 days special_use = \Trash } mailbox virtual/All { comment = All my messages special_use = \All } prefix = separator = / type = private } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { acl = vfile:/usr/local/etc/dovecot/global-acls:cache_secs=300 acl_anyone = allow acl_globals_only = yes acl_shared_dict = proxy::acl fts = solr fts_autoindex = yes fts_solr = url=http://127.0.0.1:8983/solr/dovecot/ imapsieve_mailbox1_before = file:/home/vmail/sieve/global/learn-spam.sieve imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_name = Spam imapsieve_mailbox2_before = file:/home/vmail/sieve/global/learn-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = Spam imapsieve_mailbox2_name = * last_login_dict = proxy::lastlogin last_login_key = last-login/%u mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size quota = count:User quota quota_clone_dict = proxy::sqlquota quota_exceeded_message = Storage quota for this account has been exceeded, please try again later. quota_grace = 10%% quota_status_nouser = DUNNO quota_status_overquota = 552 5.2.2 Mailbox is full quota_status_success = DUNNO quota_vsizes = true quota_warning = storage=100%% quota-exceeded 100 %u quota_warning2 = storage=95%% quota-warning 95 %u quota_warning3 = storage=90%% quota-warning 90 %u quota_warning4 = storage=85%% quota-warning 85 %u quota_warning5 = storage=75%% quota-warning 75 %u sieve = ~/.dovecot.sieve sieve_before = /home/vmail/sieve/before.d sieve_default = /home/vmail/sieve/default.sieve sieve_dir = ~/sieve sieve_extensions = +notify +imapflags sieve_global_dir = /home/vmail/sieve sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.execute sieve_max_redirects = 30 sieve_max_script_size = 1M sieve_pipe_bin_dir = /usr/local/bin sieve_plugins = sieve_imapsieve sieve_extprograms sieve_user_log = /home/vmail/sieve/sieve_error.log trash = /usr/local/etc/dovecot/trash.conf welcome_script = welcome %u welcome_wait = yes } postmaster_address = postmaster(a)example.com protocols = imap lmtp sieve sendmail_path = /usr/local/sbin/sendmail service auth-worker { user = vmail } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = vmail mode = 0660 user = vmail } } service dict { unix_listener dict { group = vmail mode = 0660 user = vmail } } service imap-login { inet_listener imap { address = 127.0.0.1 port = 143 } inet_listener imaps { address = xxx.xxx.xxx.xxx port = 993 ssl = yes } } service imap { executable = imap } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } } service managesieve-login { inet_listener sieve { address = 127.0.0.1 port = 4190 } } service quota-status { client_limit = 1 executable = quota-status -p postfix unix_listener /var/spool/postfix/private/dovecot-quota { group = postfix mode = 0660 user = postfix } } service quota-warning { executable = script /usr/local/etc/dovecot/quota-warning.sh unix_listener quota-warning { group = vmail mode = 0660 user = vmail } user = vmail } service welcome { executable = script /usr/local/etc/dovecot/welcome.sh unix_listener welcome { user = vmail } user = vmail } ssl = required ssl_cert = <cert location ssl_cipher_list = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256@STRENGTH ssl_dh_parameters_length = 4096 ssl_key = # hidden, use -P to show it ssl_options = no_compression ssl_prefer_server_ciphers = yes ssl_protocols = !SSLv2 !SSLv3 !TLSv1 !TLSv1.1 TLSv1.2 userdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lmtp { mail_fsync = optimized mail_plugins = acl mail_log notify quota quota_clone trash virtual welcome zlib fts fts_solr quota sieve } protocol lda { mail_fsync = optimized mail_plugins = acl mail_log notify quota quota_clone trash virtual welcome zlib fts fts_solr sieve } protocol imap { mail_max_userip_connections = 20 mail_plugins = acl mail_log notify quota quota_clone trash virtual welcome zlib fts fts_solr imap_acl imap_quota imap_sieve imap_zlib last_login }

2 1

2.2.33.2 - Virtual Mailbox Issue bug maybe ?
by Matt Bryant 25 Feb '18

25 Feb '18

Been experimenting with virtual mailbox and have a question :- I have defined 2 virtual mailboxes 'all' and 'combined' as follows ... virtual/all/dovecot-virtual virtual/combined inthread refs x-mailbox INBOX virtual/combined/dovecot-virtual !INBOX INBOX/Sent INBOX/Drafts all When new mail comes I would assume that both virtual mailboxes should actually see the message but whilst an imap search command gets a result in the virtual/combined for some reason not every message is seen in virtual all - so was wondering what exactly does 'inthread refs x-mailbox INBOX' do ? I would have thought new mail to inbox would just have a thread of 1 and be seen. If i remove the indexes from virtual then the messages appear so am thinking something not quite triggering an action. Any ideas ? What can I do to try and track down where problem lies or am i misunderstanding what 'inthread refs x-mailbox INBOX' should do ???? == 8< ========== # doveconf -n # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.21 (92477967) doveconf: Warning: service auth { client_limit=5000 } is lower than required under max. load (26096) doveconf: Warning: service anvil { client_limit=5000 } is lower than required under max. load (9715) # OS: Linux 3.10.0-693.11.6.el7.x86_64 x86_64 CentOS Linux release 7.3.1611 (Core) auth_cache_negative_ttl = 2 mins auth_cache_ttl = 3 mins auth_master_user_separator = * auth_mechanisms = plain login auth_verbose = yes auth_worker_max_count = 100 default_client_limit = 8192 default_process_limit = 8192 default_vsz_limit = 512 M disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags imap_idle_notify_interval = 1 mins login_greeting = IMAP/POP3 ready - us11-001dm mail_attachment_dir = /var/mail/sis/%Ld mail_cache_min_mail_count = 5 mail_debug = yes mail_plugins = " quota virtual" mailbox_list_index = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext vacation-seconds spamtest spamtestplus editheader mbox_write_locks = fcntl mmap_disable = yes namespace inbox { inbox = yes list = yes location = mailbox Archive { auto = create special_use = \Archive } mailbox Drafts { auto = create special_use = \Drafts } mailbox Sent { auto = create special_use = \Sent } mailbox Spam { auto = create special_use = \Junk } mailbox Trash { auto = create special_use = \Trash } prefix = INBOX/ separator = / type = private } namespace virtual { list = no location = virtual:/var/lib/mail/dovecot/virtual:INDEX=~/virtual prefix = virtual/ separator = / type = private } passdb { args = /etc/dovecot/sql_users.conf driver = sql } passdb { args = /etc/dovecot/sql_admin.conf driver = sql } plugin { mail_log_events = delete expunge mail_log_fields = uid box msgid size quota = dict:UserQuota::file:%h/dovecot-quota quota_rule2 = INBOX/Trash:storage=+10%% sieve = file:~/sieve/user;active=~/.dovecot.sieve sieve_default = file:/var/lib/mail/dovecot/sieve/default.sieve sieve_default_name = default sieve_editheader_max_header_size = 1k sieve_extensions = +spamtest +spamtestplus +editheader +vacation-seconds sieve_global = file:/var/lib/mail/dovecot/sieve sieve_quota_max_scripts = 5 sieve_spamtest_max_value = 200 sieve_spamtest_status_header = X-spam-score-int: -?([[:digit:]]+) sieve_spamtest_status_type = score sieve_vacation_default_period = 10d sieve_vacation_max_period = 30d sieve_vacation_min_period = 1s } pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_fast_size_lookups = yes protocols = imap pop3 lmtp sieve service anvil { client_limit = 5000 } service auth-worker { user = $default_internal_user } service auth { client_limit = 5000 unix_listener auth_client { mode = 0660 user = exim } } service doveadm { client_limit = 10 inet_listener { address = * port = 4000 } process_limit = 100 process_min_avail = 10 service_count = 1 } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } process_limit = 500 process_min_avail = 10 service_count = 1 } service imap { executable = imap postlogin process_limit = 500 } service managesieve-login { inet_listener sieve { port = 4190 } process_limit = 20 process_min_avail = 10 service_count = 1 } service managesieve { process_limit = 1024 } service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 ssl = yes } process_limit = 1000 process_min_avail = 5 service_count = 1 } service pop3 { process_limit = 500 } service postlogin { executable = script-login -d rawlog } shutdown_clients = no ssl = required ssl_cert = </etc/pki/dovecot/certs/dovecotl.com.pem ssl_key = # hidden, use -P to show it userdb { driver = prefetch } userdb { args = /etc/dovecot/sql_users.conf driver = sql } protocol imap { imap_capability = +SPECIAL-USE mail_max_userip_connections = 50 mail_plugins = " quota virtual imap_quota" } protocol pop3 { mail_max_userip_connections = 20 mail_plugins = " quota virtual" } protocol lmtp { auth_username_format = %Lu mail_plugins = " quota virtual sieve quota" postmaster_address = mailer-daemon } ======= 8< ========== The below were searches in performed in the two mailboxes .. showing the found/not found status. search in vritual/all ? search header Message-ID < 5A8663D6.5000704(a)the-bryants.net <mailto:5A8663D6.5000704@the-bryants.net> > * SEARCH ? OK Search completed (0.246 + 0.000 + 0.245 secs). search in virtual combined ? search header Message-ID < 5A8663D6.5000704(a)the-bryants.net <mailto:5A8663D6.5000704@the-bryants.net> > * SEARCH 29268 ? OK Search completed (0.179 + 0.000 + 0.178 secs). rgds Matt -- <https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campai…>

2 6

Dovecot With DSpam AntiSpam Plugin Replacement With Sieve
by colinh＠nxtgn.com 25 Feb '18

25 Feb '18

Hello I have recently had to rebuild my email server, it is a small server for my various email accounts and I also host the email for a clients business. Switched to Dovecot from Courier many years ago, and it has been a very reliable solution With the new rebuild I have updated it to Dovecot 2.2.33.2 with PigeonHole 4.21, it runs DSpam 3.10.2 to scan and tag all incoming mails as Spam or not then a Sieve rule to put the Spam emails into the Spam folder, I was also using the DSpam Antispam plugin to enable moving messages from one folder to another to teach failures to DSpam, this works well for my and my client. Doing the rebuild I saw that the AntiSpam plugin had been deprecated and the preferred solution now was to use Sieve Filters, so I implemented that and had struggled with it for a few hours until I finally got it working. It may be something about my setup but I am not sure it is, but the suggested way of doing it here https://wiki.dovecot.org/HowTo/AntispamWithSieve did not work for me and I don't think it was ever going to. Here are the shell scripts and sieve filters that I came up with that work for me, your mileage may vary. I use DSpam as a standalone binary not as a daemon so you may need to add the --client option on the dspam command line in the shell scripts if you run DSpam in daemon mode. === report-spam.sieve === require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"]; if environment :matches "imap.user" "*" { set "userid" "${1}"; if header :matches "X-DSPAM-Signature" "*" { set "signature" "${1}"; } } pipe :copy "learn-spam.sh" [ "${userid}", "${signature}" ]; === report-ham.sieve === require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"]; if environment :matches "imap.mailbox" "*" { set "mailbox" "${1}"; } if string "${mailbox}" "Trash" { stop; } if environment :matches "imap.user" "*" { set "userid" "${1}"; if header :matches "X-DSPAM-Signature" "*" { set "signature" "${1}"; } } pipe :copy "learn-ham.sh" [ "${userid}", "${signature}" ]; === learn-spam.sh === #!/bin/sh exec /usr/local/bin/dspam --user ${1} --signature=${2} --class=spam --source=error === learn-ham.sh === #!/bin/sh exec /usr/local/bin/dspam --user ${1} --signature=${2} --class=innocent --source=error I hope this saves someone some time or gives someone some tips -- Regards Colin

2 1

Remove "Duplicate" emails
by ＠lbutlr 24 Feb '18

24 Feb '18

In a quest to remove “duplicate” messages sent to both me and lists I subscribe to I came up with this, which I think should clean out my Archive folder, but I’ve been unable to get it to work for scanning all on my list-user email. $ doveadm -f table fetch -u kremels 'hdr.message-id guid uid hdr.x-listname' mailbox "Archive" | sort| awk 'cnt[$1]++{if (cnt[$1]==2) print prev[$1]; print} {prev[$1]=$0}' |grep -E "[0-9] +$" |awk '{print "doveadm expunge -u kremels MAILBOX-GUID "$2" UID "$3}’ X-listname is a header that my list user applies to each message that comes from a list, and the output is just text to the screen that I can then run manually (I am not confident enough to automatically delete the messages). I include the X-listname header at the end so that I can exclude lines that don’t end in a number, which means the copies sent directly to me are the ones expunged and the ones from the list are preserved. So far so good. But ther are issues. First, even after expunging a message and running doveadm index -u kremels “Archive”, subsequent runs still show the same duplicate messages. Second, what I really want to do is run this over ALL the mailboxes, except for Junk and Sent but if that is possible I can’t find the right syntax. -- "He has all the virtues I dislike and none of the vices I admire." Winston Churchill

2 4

Sieve
by IT 24 Feb '18

24 Feb '18

Hello, Im newbie at sieve. I have a nice script running on my boxes, and I have the following question: Using Sieve, is there a RULE to find out if String "MYTESTSTRING" is ANYWHERE inside the Email, or ANYWHERE inside the HEADER at least -- in order to trigger actions like "discard;" consequently ? Thank you so much in advance best Regards Florian -- Mit freundlichen Grüßen / best regards Florian Bojack it(a)u-eitner.de <mailto:it@u-eitner.de> logoEitner U.Eitner Formenbau, Erodier-& Kunststofftechnik GmbH Benzstraße 7 D-64807 Dieburg Geschäftsführer Ulrich und Lars Eitner Tel: +49 (0) 6071 23262 Fax: +49 (0) 6071 5562 Web: www.info@u-eitner.de Der Inhalt dieser Email und etwaiger Anhänge ist vertraulich und ausschließlich für den bezeichneten Adressaten bestimmt. Jede unbefugte Kenntnisnahme, Veröffentlichung,Vervielfältigung oder Weitergabe ist unzulässig. Sollten Sie nicht der vorgesehene Empfänger sein, bitten wir Sie, sich mit uns in Verbindung zu setzen und die Email zu löschen sowie alle Kopien zu vernichten. This e-mail is intended solely for the addressee and may contain confidential and/or privileged information. If you are not the intended recipient, please notify the sender immediately and destroy this e-mail. In this case any form of reproduction, disclosure, distribution or any action taken or refrained from in reliance on it, is strictly prohibited.

3 2

Stale mailbox lock file
by Raffaele Gambelli 24 Feb '18

24 Feb '18

Hi all, I would like to understand the meaning of "Stale mailbox lock file detected" In index-storage.c (https://mis.dopa.go.th/atmail/server_source/dovecot/src/lib-storage/index/i…) there is this notice "Stale mailbox lock file detected" ... I have a java application which connects via javamail to a dovecot server and I have some anomalies when that message comes. Thanks, bye Raffaele Gambelli

2 7