[Dovecot] Dovecot load balancing

[Thomas Hummel]

Hello,

I'd like to figure out how to set up a site running dovecot on multiple
computers for load balancing reasons.

I'm currently running dovecot-1.0.14 on a 8GB RAM, 2 quad-core 2.66Ghz Xeon
single server which runs FreeBSD 6.2-STABLE SMP.

  . the mailbox format was mbox and I successfully migrated to Maildir only
  . mailboxes are on an NFS NetApp server, attribute caching is off
  . indexes are locally stored
  . the number of users is around 2500, some of them having huge mailboxes (~ 1 or 2 GB) 
  . a large amount of users is using IMAP but there are also many POP3 users
  . the passdb/userdb is openLDAP
  . for now, I neither use namespaces, nor quotas nor deliver, nor any plugin,
    but I plan to use namespaces and ACL to implement shared mailboxes.

The same machine is running postfix-2.4.6 with amavisd-new-2.5.4 (no
spamassassin) and clamav-0.93.1. I'm using procmail-3.22 as the LDA.

I installed dovecot-1.1.2 on non-standard ports for test and plan to switch to
it as I read on this list it performs better, load wise.

This setup (1.0.14) works great except that, compared to the time mailboxes
were in mbox format, the load average (as shown by 'top') goes higher from a
significant amout on a regular basis and sometimes goes over the top.

On "normal" circumstances, the number of processes is around 1400 and the load
oscillates between 1 to 10 and 30 to 60. Most of the time, I'd say the load
average is around 20.

Every friday a message (no attachement, decent size) is sent to all users : in
such circumstances (but only sometimes, not every time : let's say half the
time such a message is sent), the load goes so high that I have to stop dovecot
to let procmail deliver the message or even reboot the machine.

Sometimes, the load climbs up to something like 150, then goes back to the
'normal' case described above.

I know multi-master replication is on the roadmap. I know some dovecot sites
use in the meantime several dovecot servers. I'd like to know how those sites
do the load balancing, the main problem beeing to be able to direct each user
to the same dovecot server each time as stated in the Wiki
(http://wiki.dovecot.org/NFS).

I don't think that DNS round robin would do the trick because some UA (for
instance Thunderbird) often open up to 5 connexions for the same user, unless
maybe such UA makes a single 'gethostbyname()', thus connecting 5 times to the
same physical server ?

What are the options to achieve such a setup ? Any successful experiences ?

Thank you

--
Thomas Hummel 	    | Institut Pasteur
<hummel at pasteur.fr> | Pôle informatique - systèmes et réseau