dovecot 2.3.7.2-1~bionic: Performance issues caused by excessive IO to ~/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.tmp

[Timo Sirainen]

On 1 Oct 2019, at 16.31, Ralf Hildebrandt via dovecot <dovecot at dovecot.org> wrote:
> 
> I set up system copying all mails to a backup system.
> 
> This used to work without a hitch - now in the last few days mails
> would pile up in the Postfix Queue, waiting to be delivered using the
> lmtp transport into dovecot.
> 
> So dovecot was being slow, but why? After all, nothing changed.
> 
> After reading some articles on stackoverflow I found a way of finding
> out which file gets the most IO:
> 
> % sysdig -c topfiles_bytes;
> 
> This command quickly pointed to ~/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.tmp
> That file was written excessively. 

Was it one user's dovecot.index.tmp or for a lot of users? This means that dovecot.index is being rewritten, which should happen only once in a while, but now it sounds like it's happening maybe for every mail delivery. If it's still happening, could you send me one folder's dovecot.index and dovecot.index.log files? (They don't contain anything sensitive other than maybe message flags.)

> I then put ~/mdbox/mailboxes/INBOX/dbox-Mails/ into tmpfs and alas, the queue would drain quickly.
> 
> But why is that? Why would the index file be updated so often?
> 
> This is dovecot 2.3.7.2-1~bionic

So you had been running this version already for a while, and then it just suddenly started getting slow?

I tried to reproduce this with imaptest and Dovecot that is patched to log when dovecot.index is being rewritten, but there doesn't seem to be any difference with v2.2.36, v2.3.7 or git master.