On Thu, 13 Jan 2011 10:33:34 -0200, Henrique Fernandes <sf.rique@gmail.com> wrote: I use ocfs2 with 3 dovecots. one only for mailman.
We have problens with IO. Have about 4k active users.
We are now testing more ocfs2 clusters, becasue one of yours theorys is that iff all mail resides in only one ocfs2 cluster, it takes too long to find the file. ocfs2 i guess does not support index. using ocfs2 1.4
My last production environment using OCFS2 was with quite recent ocfs2/dovecot - linux 2.6.35 and dovecot 1.2.15 with dbox mail storage. We got a lot of problems - high IO, fragmentation and exponential grow of access time etc. We tested also with directory indexes but this hasn't helped a lot.
Finaly we scrapped the ocfs2 setup and moved to less advanced setup: We created distinct volumes for every worker on the SAN, formated it with with XFS. The volumes got mounted on different mountpoints on workers. We setup a Pacemaker as cluster manager on the workers, so if worker dies its volume gets mounted on another worker and its service IP is brought up there.
As a result we are using a fraction of the IO compared with OCFS, the wait time on the workers dropped significantly, the service got better.
You have different options to distribute mailboxes through the workers. In owr setup the load is distributed by domain, because we are servicing hundreds of domains. So every domain MX/pop3/imap was changed to the service IP of the worker. If there are a lot of mailboxes in one domain you should put a balancer that knows on which server the mailbox is located and forward the requests there.
So now, we are gettins smallers luns from your storages and mounting 3 ocfs2 clusters that way we think the DLM will work better.
Sorry if i did not answer your question.
Anyway, we had some tests with NFS and it wasn't good also. We prefere sticky with ocfs2.
My test with NFS3/NFS4 were not good also, so it was not considered an option.
We are balacing with IPVS, not using dovecot director.
With IPVS you could not stick the same mailbox to the same server - this is important with ocfs setup because of filesystem caches and the locks. We were using nginx as proxy/balancer that could stick the same mailbox to the same backend - we did this before there was director service in dovecot but now you could use the director.
Best regards
-- Luben Karavelov