Single mailbox optimizations

[Alex Shishkin]

Hello,
I have unconventional load profile for my email server and I would like to
ask if someone can suggest optimizations in my case.
I have only one mailbox on the server for collecting emails from variety of
external mail servers. I use Postfix for collecting emails and Dovecot to
provide IMAP for downloading emails from this mailbox. Currently Postfix
configured to use LMTP for transferring emails to Dovecot. Dovecot mail
storage format is mdbox.
Everything was in order (no incoming and active queues in Postfix) when
daily incoming email count was up to 1 million emails.
But now I have to process up to 2 million incoming emails and I faced
bottleneck of 10 emails per second for saving emails from Postfix active
queue to mdbox using LMTP. Currently Postfix receives about 20 emails per
second, incoming and active queues in Postfix are growing and I have to
find the way to speed up transferring process from Postfix to mdbox.

This setup is running on the virtual machine with 8 CPU and 8 GB RAM.
CPU loaded only by dovecot/lmtp process but not more than 20% per core.
I/O wait is 2-3%. Disk load is about 200 wMB/sec and 300 rKB/sec (and this
sounds quite a bit for our storage capacity) .
Postfix has destination concurrency limit 20 for default and virtual
transport (I’m using virtual transport due to virtual domains
configuration).
Strace showed up that dovecot/lmtp processes have some delays only on
waiting for write lock on truncation log file dovecot.index.log. I have
temporary moved all index files including truncation log to tmpfs, but this
did not help.
The only thing that helped me to increase speed up to 20 emails per second
is removing almost all emails from the mailbox (with empty Inbox I got even
30 emails per second). But I have to keep all incoming emails in the
mailbox at least for one week so this way is unacceptable.

So I would like to ask for suggestions, because I can’t find the root cause
of this bottleneck.

Dovecot version 2.2.31
Postfix version  2.10.1

------------------------------------------------Dovecot conf
------------------------------------------------

mail_location = mdbox:~/mdbox
mdbox_rotate_interval = 1 days
mdbox_rotate_size = 200MB
mail_plugins = " zlib”
plugin {
  zlib_save = gz
  zlib_save_level = 6
}

userdb {
  args = /etc/dovecot/passwd
  default_fields = uid=vmail gid=vmail home=/opt/vmail/%u
  driver = passwd-file
}
passdb {
  args = /etc/dovecot/passwd
  driver = passwd-file
}

service lmtp {
  process_min_avail = 8
  service_count = 0
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0600
    user = postfix
  }
  vsz_limit = 0
}
------------------------------------------------------------------------------------------------



--------------------------------------Postfix
conf-------------------------------------------
virtual_alias_maps = hash:/etc/postfix/virtual
virtual_mailbox_domains = test.com, example.com
virtual_mailbox_maps = hash:/etc/postfix/vmailbox
virtual_transport = lmtp:unix:private/dovecot-lmtp
default_destination_concurrency_limit = 20
virtual_destination_concurrency_limit = 20
------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20180125/96e540d7/attachment.html>