[Dovecot] Creating an IMAP repo for ~100 users need some advice

[Sven Hartge]

Kaya Saman <kayasaman at gmail.com> wrote:
> <snip>
>> Question: do you need public or shared folders?

> I don't need anything apart from an IMAP storage solution. I don't
> intend to tie in Dovecot with an MTA either as I will simply be using
> this for storage.

> Long story but we don't have any control over our mail server which is
> handled by the parent company abroad and is on MS Exchange.

> To use an IMAP storage solution is the only way to get rid of pesky MS
> .pst files which have been causing everyone grief and havoc.

So, how do you plan to get the mails into this storage? offlineimap?
imapsync? mbsync? fetchmail?

>>> Hmm..... so FreeBSD coupled together with a ZFS repo for mail should
>>> take care of 'Snapshot' issues.
>> Yes. Or using LVM on Linux.

> Yeah.... true but I specified ZFS as I'm a fan and also am quite
> comfortable with Solaris/*BSD too......

If you know ZFS and are familiar with it, then, by all means, go for it.

>> Depending on the amount of mail a user collects inside a folder,
>> Maildir is not the best storage format. You may want to check into
>> mdbox, if your users are kind of "mail hoarders" (like some of my
>> users are).
>>
>> In my opinion, Maildir has outlived its usefullnes. It was fine when
>> users had 1,000 mails in some 10 folders, but today, users collect
>> over 100,000 mails a year and Maildir is causing serious I/O trouble
>> and the need to heavily fine tune your storage and filesystems to
>> cope with those demands.
>>
>> I cannot thank Timo enough for inventing mdbox, as this format breaks
>> this viciuos cycle and, as someone else said "it ends the battle at
>> the I/O front forever".

> So mdbox is a 'new' mailbox standard? ie. one can replace Maildir
> format with this and use mdbox instead. {Note to self: time to
> browse!}

mdbox is a format invented by Timo for dovecot. But dovecot can use
nearly all common mailbox formats (except MH, but no one uses that one
today).

> Since where I'm implementing this is mainly an MS based environment
> they are concerned about /flat/ files.... which MS seems to typically
> do (although never used MS before so I wouldn't know). So there is
> some concern over performance, efficiency and manageability.

Ye olde MBOX flat file format, as used in UW-imapd for ages, is a nightmare, no
doubt about this.

But even with this crappy format, dovecot is able to deliver astounding
performance by use of separete index files which allow it to access the
storage in an efficient manner.

mbox has big problems with concurrent writes, the bigger the mbox is,
the more problems you get. This is mainly caused by the meta-data of a
message (meaning flags, status, etc.) which is stored inside the mbox
file itself. Flagging a message as read or replied causes the whole mbox
file to be rewritten.

mdbox solves this problem by a) storing all meta-data in the index and
b) by only ever appending to a mdbox storage file, c) never
truncating an existing mdbox storage file and d) using more than one
mdbox storage file. Max size and TTL are configurable.

But this also means deleted mails are still inside a mdbox storage file
and need to be finally removed by copying all remaining files into a new
file. This process has to be manually run during low traffic hours, for
example using a cronjob.

You can say, mdbox is like mbox on steroids. ;)

Flat files are not evil or bad or slow per se, but you have to use them
the right way.

> However, if like you say mdbox is the way to go then I will put a
> strong case together!

You may want to start with something familiar and convert later, which
is no problem with dovecot.

Grüße,
Sven

-- 
Sigmentation fault. Core dumped.