[Dovecot] Performance with 200k messages in Maildir

[Ricardo Branco]

I agree with yr points on TBird, moving large amounts of messages can cause it to hang with CPU pegged at max for ages.
TBird v2 was nice and nippy, v3 acceptable, v4/v5 are just awfully slow overall.
TBird uses mbox storage format which probably stuffs it up on large deletes/moves etc.

Just did a count on our server, 350G of email (largest single mailbox is 40G, that is 350k messages), total messages is 
3.6mil+, biggest problem is on backup, ive read that the latest rsync has fast start now rather than wait to finish 
scanning.
Im intrested in the latest mdbox format to reduce how many files we have.
Try backing up small files fast enough to LTO5, tar it all up first before backup I think.
Ile move all our maildirs to 10k SAS soon hopefully to lower the load on the SATA disks.

Thomas Harold wrote, On 19/07/2011 16:03:
> On 7/19/2011 5:54 AM, Ricardo Branco wrote:
>> If you have 200k all within one folder progs like TB will have issues
>> loading it all up and may hang when you try to do moves/deletes etc, not
>> sure if mutt stores a local cache of headers, thats the biggest worry.
>> Biggest single folder ive seen at our office had 60k messages, it loads
>> slowly on a cold cache in TB.
>> Biggest mailbox has over 350k, my mailbox is over 250k, ofcourse thats
>> across several folders.
>> Mailserver is on VMware server (local drives), datastore (with maildirs)
>> is separate NFS server on 11x2TB SATA R6 array (has other SAS disks for
>> other things).
>> If you have it spread out in different folders then it wont be so bad.
>>
>
> 60k in a single folder is about the upper limit for TBird (TBird v2 was actually better suited for this).  But 
> drag-n-drop breaks if you try to do more then 3-5k messages at a time.  When a mailbox gets over 30-50k messages, I 
> archive some of them off to a sub-folder in Thunderbird. One of my TBird mailboxes is about 880,000 messages, almost 
> 6GB of email, spread across dozens of directories.
>
> Assuming MailDir storage, the bigger issue will be (a) how well the filesystem handles tens of thousands of files in a 
> single folder (b) the physical disks / speed / number of spindles (c) how busy the CPU is on the server and maybe (d) 
> the amount of server RAM that can be used as cache/buffer.  Ext3 is probably fine as long as directory indexing is 
> turned on, but ext4 might be better (or something else that deals well with lots of small files).
>
> The other side is how fast the disks are on the local client.  An SSD drive or 10k RPM drive on the local desktop 
> helps a lot when you get up into the larger mailboxes.