Thank you ! I was little concerned that the following settings are not in line with the new version:

mail_nfs_index = yes mail_nfs_storage = yes mail_fsync = always mailbox_list_index = yes maildir_stat_dirs = yes mmap_disable = yes

On Wed, Nov 14, 2018 at 10:19 AM Aki Tuomi aki.tuomi@open-xchange.com wrote:

It should eventually wind down once all the problems are fixed. Of course if it does not happen, you can always run force-resync for the problem users.

Aki On 14.11.2018 10.08, Adrian M wrote:

Hi, we upgraded our servers from version 2.2.13 to 2.2.27. After the upgrade we notice a 10x increase in traffic with the nfs storage an errors like this in the logfile:

Nov 12 09:48:16 mail dovecot: imap(...): Error: Corrupted index cache file /.../dovecot.index.cache: invalid record size Nov 12 09:48:16 mail dovecot: imap(...): Error: unlink(/.../dovecot.index.cache) failed: No such file or directory (in mail-cache.c:29) Nov 12 09:48:16 mail dovecot: imap(...): Error: Corrupted index cache file /.../dovecot.index.cache: invalid record size Nov 12 09:48:16 mail dovecot: imap(...): Error: Broken file /.../dovecot-uidlist line 8: Invalid data:

Is this normal ? Will the activity wind down ? Can we do something, like deleting the old dovecot.index* or dovecot-uidlist files from maildirs, or doing an doveadm force-refresh for all inboxes ?

Thank you !

Here's my configuration:

# 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.9.0-8-amd64 x86_64 Debian 9.6 nfs auth_failure_delay = 15 secs auth_mechanisms = plain login auth_verbose = yes auth_worker_max_count = 256 default_client_limit = 40000 default_process_limit = 512 dict { lastlogin = mysql:/etc/dovecot/mysql/dovecot-dict-lastlogin.conf quotadict = mysql:/etc/dovecot/mysql/dovecot-dict-quota.conf } disable_plaintext_auth = no first_valid_uid = 100 log_timestamp = "%Y-%m-%d %H:%M:%S " login_log_format_elements = user=<%u> method=%m rip=%r lip=%l pid=%p %c mail_fsync = always mail_location = maildir:*/home/virtual/*%d/%u mail_max_userip_connections = 16 mail_nfs_index = yes mail_nfs_storage = yes mail_plugins = zlib quota mail_log notify mail_privileged_group = mail mailbox_list_index = yes maildir_stat_dirs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave mmap_disable = yes namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { auto = subscribe special_use = \Junk } mailbox Trash { auto = subscribe special_use = \Trash } mailbox virtual/All { special_use = \All } prefix = } passdb { args = /etc/dovecot/mysql/dovecot-sql.conf driver = sql } plugin { last_login_dict = proxy::lastlogin last_login_key = last-login/%u mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size quota = dict:user::proxy::quotadict quota_rule2 = Trash:ignore quota_rule3 = Spam:ignore quota_status_nouser = DUNNO quota_status_overquota = 552 5.2.2 The email account that you tried to reach is over quota quota_status_success = DUNNO quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = ~/.dovecot.sieve sieve_before = /etc/dovecot/sieve/default.sieve sieve_dir = ~/sieve stats_refresh = 30 secs stats_track_cmds = yes trash = /etc/dovecot/dovecot-trash.conf zlib_save = gz zlib_save_level = 6 } protocols = imap sieve pop3 lmtp service anvil { client_limit = 20000 unix_listener anvil-auth-penalty { mode = 00 } } service auth { client_limit = 20000 unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { group = mail mode = 0660 user = vmail } user = root vsz_limit = 1 G } service dict { unix_listener dict { mode = 0660 user = vmail } } service imap-login { client_limit = 4096 executable = /usr/lib/dovecot/imap-login inet_listener imap { address = * port = 143 } inet_listener imaps { address = * port = 993 } process_limit = 4096 process_min_avail = 24 service_count = 100 } service imap { executable = /usr/lib/dovecot/imap process_limit = 4096 process_min_avail = 24 vsz_limit = 1 G } service lmtp { process_min_avail = 24 unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } vsz_limit = 1 G } service managesieve-login { client_limit = 4096 inet_listener sieve { port = 4190 } process_limit = 4096 process_min_avail = 24 service_count = 1 } service managesieve { process_limit = 4096 vsz_limit = 1 G } service pop3-login { client_limit = 4096 executable = /usr/lib/dovecot/pop3-login inet_listener pop3 { address = * port = 0 } inet_listener pop3s { address = * port = 995 } process_limit = 4096 process_min_avail = 24 service_count = 100 } service pop3 { executable = /usr/lib/dovecot/pop3 process_limit = 4096 process_min_avail = 24 vsz_limit = 1 G } service quota-status { client_limit = 4096 executable = /usr/lib/dovecot/quota-status -p postfix process_min_avail = 24 unix_listener /var/spool/postfix/private/quota-status { group = postfix mode = 0660 user = postfix } } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { mode = 0660 user = vmail } user = vmail }

userdb { args = /etc/dovecot/mysql/dovecot-sql.conf driver = sql } protocol imap { imap_idle_notify_interval = 4 mins imap_max_line_length = 64 k mail_max_userip_connections = 16 mail_plugins = zlib quota mail_log notify trash imap_quota } protocol pop3 { mail_max_userip_connections = 16 mail_plugins = zlib quota mail_log notify pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_fast_size_lookups = yes pop3_no_flag_updates = yes pop3_uidl_format = %08Xu%08Xv } protocol lda { auth_socket_path = /var/run/dovecot/auth-master mail_plugins = zlib quota mail_log notify sieve sendmail_path = /usr/lib/sendmail syslog_facility = mail } protocol sieve { mail_max_userip_connections = 16 managesieve_max_compile_errors = 5 } protocol lmtp { mail_plugins = zlib quota mail_log notify sieve syslog_facility = mail }

Thanks, they are as in example, except for "mailbox_list_index = yes" witch is from https://wiki.dovecot.org/PerformanceTuning

On Wed, Nov 14, 2018 at 12:18 PM Aki Tuomi aki.tuomi@open-xchange.com wrote:

You should review https://wiki2.dovecot.org/NFS to see that the settings make sense.

Aki On 14.11.2018 12.00, Adrian M wrote:

Thank you ! I was little concerned that the following settings are not in line with the new version:

mail_nfs_index = yes mail_nfs_storage = yes mail_fsync = always mailbox_list_index = yes maildir_stat_dirs = yes mmap_disable = yes

On Wed, Nov 14, 2018 at 10:19 AM Aki Tuomi aki.tuomi@open-xchange.com wrote:

...

It should eventually wind down once all the problems are fixed. Of course if it does not happen, you can always run force-resync for the problem users.

Aki On 14.11.2018 10.08, Adrian M wrote:

Hi, we upgraded our servers from version 2.2.13 to 2.2.27. After the upgrade we notice a 10x increase in traffic with the nfs storage an errors like this in the logfile:

Nov 12 09:48:16 mail dovecot: imap(...): Error: Corrupted index cache file /.../dovecot.index.cache: invalid record size Nov 12 09:48:16 mail dovecot: imap(...): Error: unlink(/.../dovecot.index.cache) failed: No such file or directory (in mail-cache.c:29) Nov 12 09:48:16 mail dovecot: imap(...): Error: Corrupted index cache file /.../dovecot.index.cache: invalid record size Nov 12 09:48:16 mail dovecot: imap(...): Error: Broken file /.../dovecot-uidlist line 8: Invalid data:

Is this normal ? Will the activity wind down ? Can we do something, like deleting the old dovecot.index* or dovecot-uidlist files from maildirs, or doing an doveadm force-refresh for all inboxes ?

Thank you !

Here's my configuration:

# 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.9.0-8-amd64 x86_64 Debian 9.6 nfs auth_failure_delay = 15 secs auth_mechanisms = plain login auth_verbose = yes auth_worker_max_count = 256 default_client_limit = 40000 default_process_limit = 512 dict { lastlogin = mysql:/etc/dovecot/mysql/dovecot-dict-lastlogin.conf quotadict = mysql:/etc/dovecot/mysql/dovecot-dict-quota.conf } disable_plaintext_auth = no first_valid_uid = 100 log_timestamp = "%Y-%m-%d %H:%M:%S " login_log_format_elements = user=<%u> method=%m rip=%r lip=%l pid=%p %c mail_fsync = always mail_location = maildir:*/home/virtual/*%d/%u mail_max_userip_connections = 16 mail_nfs_index = yes mail_nfs_storage = yes mail_plugins = zlib quota mail_log notify mail_privileged_group = mail mailbox_list_index = yes maildir_stat_dirs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave mmap_disable = yes namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { auto = subscribe special_use = \Junk } mailbox Trash { auto = subscribe special_use = \Trash } mailbox virtual/All { special_use = \All } prefix = } passdb { args = /etc/dovecot/mysql/dovecot-sql.conf driver = sql } plugin { last_login_dict = proxy::lastlogin last_login_key = last-login/%u mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size quota = dict:user::proxy::quotadict quota_rule2 = Trash:ignore quota_rule3 = Spam:ignore quota_status_nouser = DUNNO quota_status_overquota = 552 5.2.2 The email account that you tried to reach is over quota quota_status_success = DUNNO quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = ~/.dovecot.sieve sieve_before = /etc/dovecot/sieve/default.sieve sieve_dir = ~/sieve stats_refresh = 30 secs stats_track_cmds = yes trash = /etc/dovecot/dovecot-trash.conf zlib_save = gz zlib_save_level = 6 } protocols = imap sieve pop3 lmtp service anvil { client_limit = 20000 unix_listener anvil-auth-penalty { mode = 00 } } service auth { client_limit = 20000 unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { group = mail mode = 0660 user = vmail } user = root vsz_limit = 1 G } service dict { unix_listener dict { mode = 0660 user = vmail } } service imap-login { client_limit = 4096 executable = /usr/lib/dovecot/imap-login inet_listener imap { address = * port = 143 } inet_listener imaps { address = * port = 993 } process_limit = 4096 process_min_avail = 24 service_count = 100 } service imap { executable = /usr/lib/dovecot/imap process_limit = 4096 process_min_avail = 24 vsz_limit = 1 G } service lmtp { process_min_avail = 24 unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } vsz_limit = 1 G } service managesieve-login { client_limit = 4096 inet_listener sieve { port = 4190 } process_limit = 4096 process_min_avail = 24 service_count = 1 } service managesieve { process_limit = 4096 vsz_limit = 1 G } service pop3-login { client_limit = 4096 executable = /usr/lib/dovecot/pop3-login inet_listener pop3 { address = * port = 0 } inet_listener pop3s { address = * port = 995 } process_limit = 4096 process_min_avail = 24 service_count = 100 } service pop3 { executable = /usr/lib/dovecot/pop3 process_limit = 4096 process_min_avail = 24 vsz_limit = 1 G } service quota-status { client_limit = 4096 executable = /usr/lib/dovecot/quota-status -p postfix process_min_avail = 24 unix_listener /var/spool/postfix/private/quota-status { group = postfix mode = 0660 user = postfix } } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { mode = 0660 user = vmail } user = vmail }

userdb { args = /etc/dovecot/mysql/dovecot-sql.conf driver = sql } protocol imap { imap_idle_notify_interval = 4 mins imap_max_line_length = 64 k mail_max_userip_connections = 16 mail_plugins = zlib quota mail_log notify trash imap_quota } protocol pop3 { mail_max_userip_connections = 16 mail_plugins = zlib quota mail_log notify pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_fast_size_lookups = yes pop3_no_flag_updates = yes pop3_uidl_format = %08Xu%08Xv } protocol lda { auth_socket_path = /var/run/dovecot/auth-master mail_plugins = zlib quota mail_log notify sieve sendmail_path = /usr/lib/sendmail syslog_facility = mail } protocol sieve { mail_max_userip_connections = 16 managesieve_max_compile_errors = 5 } protocol lmtp { mail_plugins = zlib quota mail_log notify sieve syslog_facility = mail }

Yes, multiple imap servers using one shared nfs storage. With the same config on 2.2.13 the public interface traffic was similar to the storage interface, around 100 mbps.

After we switch to 2.2.27 the storage interface traffic jumped 10 times while the public interface stayed the same. This make us thinking that something is wrong and each time a user logs in the whole Inbox content is read by dovecot.

What you are suggesting goes against the documentations and it may not be save, but I will give a  thought.

On 11/15/18 6:23 AM, Adi Pircalabu wrote:

Are you connecting to the same mailbox over NFS from multiple IMAP servers? If not and, at any given time, any mailbox will be accessed from a single NFS client, try to "dupe" Dovecot into thinking it's not using NFS. We're running quite successfully such setup with NFSv3 over TCP, which turned out to be the fastest and most reliable throughout the years. Here are the mount options: rw,noatime,nodiratime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,mountvers=3,mountport=1892,mountproto=tcp,local_lock=none

On the Dovecot side we're running with: lock_method = dotlock mail_fsync = never mail_nfs_index = no mail_nfs_storage = no maildir_very_dirty_syncs = yes mmap_disable = yes protocol lda { mail_fsync = optimized } protocol lmtp { mail_fsync = optimized }

Note: we're using Maildir and the usual "works for me(c), may not work for everyone" applies.

-- Best regards, Adrian Minta

Yes, more than one server.

On Thu, Nov 15, 2018 at 11:42 PM Adi Pircalabu adi@ddns.com.au wrote:

On 2018-11-16 07:24, Adrian Minta wrote:

...

Yes, multiple imap servers using one shared nfs storage. With the same config on 2.2.13 the public interface traffic was similar to the storage interface, around 100 mbps.

After we switch to 2.2.27 the storage interface traffic jumped 10 times while the public interface stayed the same. This make us thinking that something is wrong and each time a user logs in the whole Inbox content is read by dovecot.

What you are suggesting goes against the documentations and it may not be save, but I will give a thought.

I was expecting you have multiple IMAP servers using the same shared NFS storage, however my question was: are the *individual mailboxes* on that share accessed *at the same time* from more than one IMAP server?

-- Adi Pircalabu

...

On 11/15/18 6:23 AM, Adi Pircalabu wrote:

...

Are you connecting to the same mailbox over NFS from multiple IMAP servers? If not and, at any given time, any mailbox will be accessed from a single NFS client, try to "dupe" Dovecot into thinking it's not using NFS. We're running quite successfully such setup with NFSv3 over TCP, which turned out to be the fastest and most reliable throughout the years. Here are the mount options:

rw,noatime,nodiratime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,mountvers=3,mountport=1892,mountproto=tcp,local_lock=none

...

...

On the Dovecot side we're running with: lock_method = dotlock mail_fsync = never mail_nfs_index = no mail_nfs_storage = no maildir_very_dirty_syncs = yes mmap_disable = yes protocol lda { mail_fsync = optimized } protocol lmtp { mail_fsync = optimized }

Note: we're using Maildir and the usual "works for me(c), may not work for everyone" applies.