[Dovecot] Spliting Folders for Efficiency

[Daniel Watts]

Curtis Maloney wrote:
> Daniel Watts wrote:
>> Dear Timo,
>>
>> Would there be any sense in giving Dovecot the option to split 
>> folders into multiple subfolders when they reached a specified size 
>> (probably message count) limit?
>
> My understanding is this is partially covered in Timo's "dbox" format, 
> which tries to take the best features of mbox and Maildir.
Is dbox production ready? It looks interesting.
http://wiki.dovecot.org/MailboxFormat/dbox this page says it is not 
finished.

What actually ARE the advantages of a 'one file per folder' format?? We 
switched to Maildir because mbox was killing our server. I wouldn't ever 
switch back.
The only thing perhaps is faster Search since you don't have to open 
lots of files. But for this I reckon it would be best to keep a separate 
index of content. Dreams of offering a 'google like' imap-search 
function anyone? =) Are there any (preferably open source) products out 
there for this?
>
>> .Folder.new
>> .Folder.cur
>> .Folder.tmp
>>
>> could become:
>>
>> .Folder__1.new
>> .Folder__1.cur
>> .Folder__1.tmp
>> and
>> .Folder__2.new
>> .Folder__2.cur
>> .Folder__2.tmp
>
> You would only need to split "cur", unless you expect someone to get 
> over 10,000 new message waiting.  "tmp" is only used _whilst_ message 
> are being delivered, so mail clients don't see a partially written 
> message.
Ah yes this is true.
>
>> This could be further extended so that Dovecot could be configured to 
>> store 'old' message folders in a separate location. We could then 
>> have slower+cheaper+larger storage mounted so that 'old mail' does 
>> not take up the expensive local SCSI disks on the machine. Mail from 
>> 2 years ago is much less likely to be accessed than mail from the 
>> last week.
>
> Also, instead of __N, you could try a different path, so 
> /foo/bar/User/ is for new mail, and /old/slow/disk/User is for older 
> stuff.
ah yes - and if it is on the same disk it could just be 
$HOME/Maildir/cur and $HOME/Maildir/old/cur
>
>> This would provide very neat behind-the-scenes archiving functionality.
>
> There's really two ideas here... one is the mechanism of 
> multi-directory folders, the other is the policy of separating by age.
Ideally there would be a few limits set by the system admin:
Min Age of mail
Max Age of mail
Min number of messages.
Max number of messages.

You can then split by either volume or age and control how many emails 
to keep in 'fast' storage as a minimum - eg always have the most recent 
50 emails in local storage, regardless of age.

Dan