[Dovecot] nfs director

[Cor Bosman]

Hi,

> We use Dovecot only recent, but many I speak  use for very many year, if
> director was really need, why it only come about now and not 5 or more year
> ago, all many mail network run broken for so many year? I no think so.
> It might compliment some situation, but not suitable or advantage for all.
> Imap not likely happen here for user, hardware  no so cheap here,such is
> storage space not big deal, not until we buy hardware for near cheap as
> U.S., which 50-70% less than we pay.

The director is not needed in most cases. If you feel you dont need the
director, then just forget about its existence and ignore all emails about it. 
Just pretend it doesnt exist :)

I'll try to explain why the director is needed in some situations.

dovecot has an index file. If you look in your configuration file, you'll
see something like:

mail_location = maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u

The index file holds an index so dovecot can search in your mailbox really
fast. (sorry if you know all this). So, if for instance you want to sort
your mailbox on subject, dovecot doesnt have to read through all your emails
to find all the subjects. Instead, it can read the index file really fast,
and return a sorted mailbox basically instantly. A lot of IMAP servers dont
even have this feature!  Imagine the alternative, where dovecout would have
to read through 25.000 emails, just so it can sort your mailbox on subject. 
A nightmare for performance, but like I said, some imap servers still do that!
Dovecot luckily uses an index, and is therefore faster than a lot of other
imap servers.

So, we have concluded that dovecot has a really cool performance feature
in the form of an index file. 

Now imagine what happens if you have more than 1 imap server. We have 35
imap servers right now. Each server needs access to the index file.
One popular way to do this is by using NFS. With NFS you have a central
file server, and each imap server uses the remote fileserver for storage.
So now, all 35 servers have access to a cool central index file. 

But, there is a slight snag. NFS has a problem where caching of the files
can cause each imap server to see outdated information. Server 1 could
have just written some updates into the index file, but Server 2 doesnt
see this yet, and writes conflicting data into the index file. The index
file can then end up being corrupted. This causes dovecot to give faulty
data back to the client, which in practice ends up looking like the mailbox
of the user is empty. Not good. This causes calls to support desks about
why a user has just lost all his mail. It's not lost, it's just that the index
file is corrupt, and now dovecot thinks the mail is gone. 

You may wonder rightly so, why this is a problem only now. Well, it hasnt
been a problem only now. We've been dealing with this issue for years. We've
had discussions with Timo over the years on how to solve this problem. But
Timo has only so much time, and this problem was not important enough to fix.
He had more important bugs to fix. So, we lived with the problem, or worked
around it in ugly ways. 

Recently Timo has found the time (I believe an ISP paid Timo to solve this
problem) to think about ways to solve it. It's really not an easy problem
to fix. We've known for years that the best way to fix this would be to have a
user always end up on the same server, because thats when that specific NFS
problem doesnt occur. If 2 sessions of the same user both end up on Server 1,
then both sessions will see the same data, and no problems can occur.

But how to get all sessions of a user on the same server? Most loadbalancers
dont support this feature. Our Foundries dont, and our previous Alteons didnt
either. So Timo thought of a way how dovecot itself could make sure all 
sessions of a user end up on the same server. Basically by turning dovecot
into a proxy, that proxies connections to another server. It maintains an
internal state on what user belongs on what server, and redirects connections
appropriately. And thats all the director is. A proxy with an internal
state of where to proxy a connection to. 

So why dont you see this problem with your setup? Like I said, this is only
a problem with specific setups. 

- you need to be using NFS for central index storage. If you dont use NFS for
  your indexes, you wont see this problem, and you wont need the director.
  carry on as you always have. 

- even if you use NFS, you may not run into this problem. Because you also
  need to have situations where multiple servers could be updating the indexes
  at the same time. There are only 2 situations where this could happen:

  1) you're using dovecot-lda, and have index updating turned on. Index
     updating in the lda is cool, because it means dovecot will always have
     the latest information about your email in the index files. 
  
  and/or

  2) you allow users to have sessions to multiple servers. For instance,
     they connect with Outlook and their iPhone at the same time. And maybe
     even a webmail connection as well. 


  If you dont have one of these 2 situations, you wont ever see a problem
  and you wont need the director. carry on as you always have. 


But, for those (mostly larger) setups where you are using NFS for indexes,
AND you can have multiple sessions of the same user on different servers
(and they can be either LDA or IMAP sessions), you're probably better off
using the director. 

Again, if this doesnt apply to you, simply..... do not use the director.

Regards,

Cor