[Dovecot] Creating an IMAP repo for ~100 users need some advice

[Kaya Saman]

On 03/17/2012 10:28 PM, Sven Hartge wrote:
> Kaya Saman<kayasaman at gmail.com>  wrote:
>> <snip>
>>> Question: do you need public or shared folders?
>> I don't need anything apart from an IMAP storage solution. I don't
>> intend to tie in Dovecot with an MTA either as I will simply be using
>> this for storage.
>> Long story but we don't have any control over our mail server which is
>> handled by the parent company abroad and is on MS Exchange.
>> To use an IMAP storage solution is the only way to get rid of pesky MS
>> .pst files which have been causing everyone grief and havoc.
> So, how do you plan to get the mails into this storage? offlineimap?
> imapsync? mbsync? fetchmail?

Since everything is blocked at the Exchange end, users will have to 
manually transfer for now through MS Outlook.

Currently that's what they're doing to their PST's....

>
>>>> Hmm..... so FreeBSD coupled together with a ZFS repo for mail should
>>>> take care of 'Snapshot' issues.
>>> Yes. Or using LVM on Linux.
>> Yeah.... true but I specified ZFS as I'm a fan and also am quite
>> comfortable with Solaris/*BSD too......
> If you know ZFS and are familiar with it, then, by all means, go for it.

:-)

>
>>> Depending on the amount of mail a user collects inside a folder,
>>> Maildir is not the best storage format. You may want to check into
>>> mdbox, if your users are kind of "mail hoarders" (like some of my
>>> users are).
>>>
>>> In my opinion, Maildir has outlived its usefullnes. It was fine when
>>> users had 1,000 mails in some 10 folders, but today, users collect
>>> over 100,000 mails a year and Maildir is causing serious I/O trouble
>>> and the need to heavily fine tune your storage and filesystems to
>>> cope with those demands.
>>>
>>> I cannot thank Timo enough for inventing mdbox, as this format breaks
>>> this viciuos cycle and, as someone else said "it ends the battle at
>>> the I/O front forever".
>> So mdbox is a 'new' mailbox standard? ie. one can replace Maildir
>> format with this and use mdbox instead. {Note to self: time to
>> browse!}
> mdbox is a format invented by Timo for dovecot. But dovecot can use
> nearly all common mailbox formats (except MH, but no one uses that one
> today).

Ok so if you claim that mdbox is the 'best' mailbox storage solution 
then I'll look at implementing this.

>
>> Since where I'm implementing this is mainly an MS based environment
>> they are concerned about /flat/ files.... which MS seems to typically
>> do (although never used MS before so I wouldn't know). So there is
>> some concern over performance, efficiency and manageability.
> Ye olde MBOX flat file format, as used in UW-imapd for ages, is a nightmare, no
> doubt about this.
>
> But even with this crappy format, dovecot is able to deliver astounding
> performance by use of separete index files which allow it to access the
> storage in an efficient manner.
>
> mbox has big problems with concurrent writes, the bigger the mbox is,
> the more problems you get. This is mainly caused by the meta-data of a
> message (meaning flags, status, etc.) which is stored inside the mbox
> file itself. Flagging a message as read or replied causes the whole mbox
> file to be rewritten.
>
> mdbox solves this problem by a) storing all meta-data in the index and
> b) by only ever appending to a mdbox storage file, c) never
> truncating an existing mdbox storage file and d) using more than one
> mdbox storage file. Max size and TTL are configurable.
>
> But this also means deleted mails are still inside a mdbox storage file
> and need to be finally removed by copying all remaining files into a new
> file. This process has to be manually run during low traffic hours, for
> example using a cronjob.
>
> You can say, mdbox is like mbox on steroids. ;)
>
> Flat files are not evil or bad or slow per se, but you have to use them
> the right way.

Thanks a lot for that info. I will research more into this but I maybe 
overridden at some point :-(

Need to make a strong case!

>
>> However, if like you say mdbox is the way to go then I will put a
>> strong case together!
> You may want to start with something familiar and convert later, which
> is no problem with dovecot.

Maildir is what I'm familiar with currently and mbox format - though 
only use mbox as an unfortunate side product of /system mail/ accounts.

Works well with Alpine client though!

>
> Grüße,
> Sven
>

Regards,

Kaya