sudden performance drop - i/o related

[Aki Tuomi]

> On 11/05/2021 01:07 Marcin Gryszkalis <mg at fork.pl> wrote:
> 
>  
> Hi
> I have exim/dovecot server that worked great for last few years and two 
> weeks ago it got ill ;)
> First were users reporting errors on saving mails to Sent (timeouts).
> Now the logs are infested with warnings about long waits:
> 
> May 10 10:18: Maildir /mail/xxx Synchronization took 193 seconds (1 new 
> msgs, 0 flag change attempts, 0 expunge attempts)
> May 10 10:18: Maildir /mail/xxx Synchronization took 125 seconds (1 new 
> msgs, 0 flag change attempts, 0 expunge attempts)
> May 10 10:18: Maildir /mail/xxx Synchronization took 211 seconds (1 new 
> msgs, 0 flag change attempts, 0 expunge attempts)
> May 10 10:18: Maildir /mail/xxx Synchronization took 107 seconds (8 new 
> msgs, 0 flag change attempts, 0 expunge attempts)
> May 10 10:18: Transaction log file /mail/xxx was locked for 36 seconds 
> (Mailbox was synchronized)
> May 10 10:18: Transaction log file /mail/xxx was locked for 160 seconds 
> (Mailbox was synchronized)
> May 10 10:18: Transaction log file /mail/xxx was locked for 72 seconds 
> (Mailbox was synchronized)
> May 10 10:18: Locking transaction log file /mail/xxx took 60 seconds 
> (syncing)
> May 10 10:18: Locking transaction log file /mail/xxx took 38 seconds 
> (syncing)
> May 10 10:18: Locking transaction log file /mail/xxx took 35 seconds 
> (syncing)
> 
> It looks like i/o risen from 150writes/s to 500writes/s (in top hours) - 
> but there's no real change in number of emails or the volume. Number of 
> users is steady (~100 active users, ~250 imap sessions), number of 
> emails (by count or by volume) is rising and falling within 15% margin.
> 
> The box is FreeBSD 11.4, dovecot is 2.3.13.
> Filesystem is ZFS, disks are fine, free space is around 20% (~200GB)
> Layout is Maildir. CPU is not overloaded (2x6core), same with memory (48GB).
> 
> I didn't change anything in configuration.
> 
> Tonight I did some finetuning like maildir_copy_with_hardlinks=yes or 
> mail_fsync=never/optimized (I'm not happy with that but I'm afraid it 
> won't really help and I'll be able to revert that). I'm also thinking 
> about switching from Maildir to sdbox (I know it won't hurt).
> 
> I don't know where to look to find where the i/o goes. I don't have any 
> metrics/stats enabled (I looked at the docs but it looks it's not really 
> simple and needs some digging to get valuable config). Maybe somebody 
> has suggestions what to look for?
> 
> For detailed per-process stats I need to rebuild kernel with dtrace 
> (other night I guess)... Simple top (in i/o mode - similar to linux's 
> iotop) doesn't catch short living processes (like LDA deliveries).
> 
> best regards
> -- 
> Marcin Gryszkalis, PGP 0xA5DBEEC7 http://fork.pl/gpg.txt

One thing that does come to mind is that you are delivering outside dovecot. Without knowing your system better, I would suggest that one thing to try would be to use dovecot-lda to deliver mail.

Are your users directly accessing the maildir?

Aki