Very slow mail download/notification with dovecot 2.3.7 and Thunderbird​

[ml_dovecot at thorsten-reichelt.de]

Hi!

I know that there are many results if I search for "dovecot thunderbird
very slow" on Google but none of them helped me with my problem. :(  (I
already posted this on February 2020, but forgot to answer. O.o)

I am running Ubuntu 20.04 (4GB RAM, 40GB HDD with EXT4) server with the
following software:

- Dovecot 2.3.7.2
---- All accounts are using Maildir
---- Sieve filters

- Postfix 3.4.13
---- with opendkim, spamassassin, razor, pyzor, ....

And as IMAP client I am using Thunderbird 78.6 on Windows 10.
Sieve is used to sort some incoming mails into my 228 folders (56 INBOX.
folders and its subfolders (output of "find .INBOX.* -maxdepth 4 -type d
-print |wc -l")).  The total size of my Maildir directory is 5.5GB.

My problem is that it always takes very very long to synchronize the
INBOX and its subfolders. Even if there are only a few new messages to
be downloaded. Sometimes Thunderbird outputs something like "downloading
message 1 of 19" and it takes at least 60 seconds to download all of
them. None of these mails is larger than 5kB. And sometimes it seems
that the transfer is suspended for a long time and it takes >5 Minutes
to synchronize only a few mails. :(

And it seems that some folders are never updated in TB. In example I
sort all messages from this list into a "INBOX.Mailinglists.ML-Dovecot"
subfolder by using a simple sieve rule. But even after 10 minutes TB
thinks that there are no new messages. As soon as I click on the
ML-Dovecot folder I see in the /var/log/dovecot-info.log file that TB
logs on to the server and then displays hundreds, of new messages. I
checked twice but I have subscribed to all 228 folders.

How can I find the bottleneck?
Is there a test tool available that simulates a simple mail client and
can analyse an IMAP server?

This is the output of "dovecot -n":

===============================
# 2.3.7.2 (3c910f64b): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.7.2 ()
# OS: Linux 5.4.0-54-generic x86_64 Ubuntu 20.04.1 LTS
# Hostname: mail.XXXXXXX.de
auth_mechanisms = plain login
debug_log_path = /var/log/dovecot-debug.log
default_client_limit = 15653
default_process_limit = 5200
first_valid_uid = 150
info_log_path = /var/log/dovecot-info.log
last_valid_uid = 150
lda_original_recipient_header = X-Original-To
log_path = /var/log/dovecot-error.log
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c %k
mail_gid = mail
mail_home = /var/vmail/%d/%n
mail_location = maildir:~/Maildir
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart extracttext
namespace inbox {
  inbox = yes
  location =
  mailbox Archive {
    special_use = \Archive
  }
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix =
}
passdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  sieve = ~/.dovecot.sieve
  sieve_default = /var/vmail/global.sieve
  sieve_dir = ~/sieve
}
postmaster_address = thorsten at MY_DOMAIN.TLD
protocols = " imap sieve pop3"
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0666
    user = postfix
  }
  unix_listener auth-userdb {
    group = mail
    mode = 0666
    user = vmail
  }
}
service imap-login {
  inet_listener imap {
    port = 143
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
  process_limit = 50
  process_min_avail = 2
  service_count = 20
}
service imap {
  process_limit = 1024
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
  service_count = 0
}
ssl_cert = </etc/ssl/mail/mail.XXXXXXX.de.fullchain.pem
ssl_cipher_list =
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
ssl_min_protocol = TLSv1.2
syslog_facility = local5
userdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
protocol lda {
  deliver_log_format = msgid=%m: %$
  mail_plugins = " sieve"
  rejection_reason = Your message to <%t> was automatically rejected:%n%r
}
protocol imap {
  imap_client_workarounds = tb-extra-mailbox-sep
  imap_idle_notify_interval = 10 mins
  mail_max_userip_connections = 100
  mail_plugins =
}
protocol sieve {
  info_log_path = /var/log/dovecot-sieve.log
  mail_max_userip_connections = 100
  managesieve_implementation_string = Dovecot Pigeonhole
  managesieve_max_line_length = 65536
===============================

Output of "free -h":
===============================
free -h
              total        used        free      shared  buff/cache  
available
Mem:          3.8Gi       1.7Gi       1.1Gi        10Mi      
958Mi       1.7Gi
Swap:         1.0Gi       375Mi       648Mi
===============================

Thank you very much!
Thorsten