[Dovecot] Please help to make decision

[Urban Loesch]

Hi,

we have similar setup like Thierry, but not so big. Only 40k users and 
1,2T of used space. Only 300 concurrent POP3 and 1600 IMAP sessions. 
Imap is increasing continously.

Due to the fact that we have a low budget we impelented the following 
small solution.

- 2 static IMAP/POP3 Proxies (no director) load balanced with the well 
known CLUSTERIP module from iptables (poors man loadbalancing, but works 
only in layer 2 envirmonments. Works great for our needs and would be 
scalable too)
- 2 static SMTP relayservers load balanced the same way as above.
- 4 storage machines in active/passive setup with DRBD on top of LVM2. 
On each active node are running 4-5 virtual containers (based on 
http://linux-vserver.org). All 40k accounts a spread on this 8 
containers. This has the advantage to quickly move the hole container 
from one storage machine to another wihout dsync if there is not enough 
space on some node.
- 2 Mysql master/master containers to store userinformation which then 
are be cached by dovecot itself. This extremly reduces database load.

All servers (proxies, relayserver dovecot, mysql are containers). So we 
can move them around on different hardware without changing any 
configuration. But this happens rarely.

Dovecot uses mdbox storage format with compression enabled. No problems 
yet. Index and mdbox files are stored on different mount points. This
gives us the chance to move them easily to different spindles if we 
need. In the future we plan to store indexes on SSD's and mdbox files on 
SATA drives, as in fact the main IO happens on index files and the
use of disk space is increasing.

As mentioned above, this is not a big setup, but for our needs it works
very good and stable. Helps us to save money and problems with NFS and 
SAN's, etc. And it can be scaled out very easy.

Regards
Urban

Am 25.03.2013 19:47, schrieb Thierry de Montaudry:
> Hi Tigran,
>
> Managing a mail system for 1M odd users, we did run for a few years on some high range SAN system (NetApp, then EMC), but were not happy with the performance, whatever double head, fibre, and so on, it just couldn't handle the IOs. I must just say that at this time, we were not using dovecot.
>
> Then we moved to a completely different structure: 24 storage machines (plain CentOS as NFS servers), 7 frontend (webmail through IMAP + POP3 server) and 5 MXs, and all front end machines running dovecot. That was a major change in the system performances, but not happy yet with the 50T total storage we had. Having huge traffic between front end machine and storage, and at this time, I was not sure the switches were handling the load properly. Not talking about the load on the front end machine which some times needed a hard reboot to recover from NFS timeouts. Even after trying some heavy optimizations all around, and particularly on NFS.
>
> Then we did look at the Dovecot director, but not sure how it would handle 1M users, we moved to the proxy solution: we are now running dovecot on the 24 storage machines, our webmail system connecting with IMAP to the final storage machine, as well as the MXs with LMTP, we only use dovecot proxy for the POP3 access on the 7 front end machines. And I must say, what a change. Since then the system is running smoothly, no more worries about NFS timeouts and the loadavg on all machine is down to almost nothing, as well as the internal traffic on the switches and our stress. And most important, the feed back from our users told us that we did the right thing.
>
> Only trouble: now and then we have to move users around, as if a machine gets full, the only solution is to move data to one that has more space. But this is achieved easily with  the dsync tool.
>
> This is just my experience, it might not be the best, but with the (limited) budget we had, we finally came up with a solutions that can handle the load and got us away from SAN systems which could never handle the IOs for mail access. Just for the sake of it, our storage machines only have each 4 x 1T SATA drives in RAID 10, and 16G of mem, which I've been told would never do the job, but it just works. Thanks Timo.
>
> Hoping this will help in your decision,
>
> Regards,
>
> 	Thierry
>
>
> On 24 Mar 2013, at 18:12, Tigran Petrosyan<tpetrosy at gmail.com>  wrote:
>
>> Hi
>> We are going to implement the "Dovecot" for 1 million users. We are going
>> to use more than 100T storage space. Now we examine 2 solutions NFS or GFS2
>> via (Fibre Channel storage).
>> Can someone help to make decision? What kind of storage solution we can use
>> to achieve good performance and scalability.
>