[Dovecot] Maildir Synchronization warnings

[Stan Hoeppner]

On 8/2/2013 6:57 AM, Timo Sirainen wrote:
> On Fri, 2013-08-02 at 15:34 +0800, Kavish Karkera wrote:
>> Hi,
>>
>> We are repeatedly getting these below warnings for some of our users, al though we have no complaints from them yet,
>>  we need to know why these warning occurs.
>>
>> So it would be help full if some one explain these warning msg in detail.
> ..
>> Aug  2 12:52:55 blade8 dovecot: imap(kavish.karkera at example.com): Warning: Maildir: Scanning /mail/v3store/example.com/kavish.karkera at example.com/Maildir/cur took 94 seconds (23191 readdir()s, 0 rename()s to cur/, why=0x1)
> 
> It means that the maildir INBOX is huge, and it takes a long time to
> access them with your available disk IO. Possibilities:

Yes, 23K+ and 43K+ emails in a maildir INBOX will generate quite a bit
of read IO.  Are these shared mailboxes used for some business process,
or normal user mailboxes?  If the former are these tends of thousands of
emails arriving in a single day?

> a) Move move of your mails away from INBOX.

If the former you'll want to create a workflow that moves each email to
archival storage, or some temporary location, after each email has been
processed, either by your automated system or human beings.  As you've
discovered, allowing so many emails to pile up in a maildir INBOX causes
problems.

If the latter, the users need to be properly educated on mail folder
hierarchy, sorting, and storage best practices.

> b) Switch to different mailbox format that can handle large mailboxes,
> such as mdbox or sdbox.

This will help but not nearly as much as Timo's first suggestion.


For many reasons it's not always possible or feasible for an
organization to switch mailbox storage formats.  So the last option,
which is always the least bang for the buck, is optimizing your storage
stack on the NFS server for maximum IOPS.

For instance if you're currently using parity RAID, migrating to RAID10
will give you a 5-15x+ boost in mixed read/write random IOPS throughput.
 If your RAID has large strips (chunks) of say 128KB+ switching to small
strips of 16-32KB will increase IOPS, especially if you're using parity
RAID--smaller chunks significantly decrease RMW latency.  Switching from
EXT3/4 to XFS w/inode64 will provide a boost due to parallelism across
allocation groups, and files being clustered around their metadata
inodes which decreases head seek latency--this is especially beneficial
to maildir which manipulates metadata more than file data.

And then there's always the option of adding more hardware, typically
more/faster spindles, maybe more HW RAID cache and more system RAM for
greater filesystem buffer cache, or all three.

-- 
Stan