[Dovecot] Dsync replication
I have been loosely following discussions dsync replication, but I am wondering if this tool still in a testing phase or has it been committed as a production part of Dovecot? I would like to do some testing with the protocol to see if it's capable of handling the change rate our mail service generates, which has about 23,000 active users. We are trying to find solutions for a two site setup where mail processing is either being done at the primary data center A and fails over to a hot standby cluster at the secondary data center B, or where both data centers are doing active processing. I have been unable to find documentation on dsync replication on the wiki, so if there is any documentation available on how to setup dsync replication I would appreciate a nudge in the right direction.
use the search, Luke)
http://www.dovecot.org/list/dovecot/2012-March/064512.html this thread was all that I needed to setup replication for testing.
-----Original Message----- From: dovecot-bounces@dovecot.org [mailto:dovecot-bounces@dovecot.org] On Behalf Of list@airstreamcomm.net Sent: Friday, July 06, 2012 3:28 AM To: dovecot@dovecot.org Subject: [Dovecot] Dsync replication
I have been loosely following discussions dsync replication, but I am wondering if this tool still in a testing phase or has it been committed as a production part of Dovecot? I would like to do some testing with the protocol to see if it's capable of handling the change rate our mail service generates, which has about 23,000 active users. We are trying to find solutions for a two site setup where mail processing is either being done at the primary data center A and fails over to a hot standby cluster at the secondary data center B, or where both data centers are doing active processing. I have been unable to find documentation on dsync replication on the wiki, so if there is any documentation available on how to setup dsync replication I would appreciate a nudge in the right direction.
On 7/5/12 10:08 PM, Костырев Александр Алексеевич wrote:
use the search, Luke)
http://www.dovecot.org/list/dovecot/2012-March/064512.html this thread was all that I needed to setup replication for testing.
-----Original Message----- From: dovecot-bounces@dovecot.org [mailto:dovecot-bounces@dovecot.org] On Behalf Of list@airstreamcomm.net Sent: Friday, July 06, 2012 3:28 AM To: dovecot@dovecot.org Subject: [Dovecot] Dsync replication
I have been loosely following discussions dsync replication, but I am wondering if this tool still in a testing phase or has it been committed as a production part of Dovecot? I would like to do some testing with the protocol to see if it's capable of handling the change rate our mail service generates, which has about 23,000 active users. We are trying to find solutions for a two site setup where mail processing is either being done at the primary data center A and fails over to a hot standby cluster at the secondary data center B, or where both data centers are doing active processing. I have been unable to find documentation on dsync replication on the wiki, so if there is any documentation available on how to setup dsync replication I would appreciate a nudge in the right direction.
Thanks, that certainly helps identify the configuration options. However
I am more concerned about the experiences of others who have actually
used the replication. What is the rate of change on your mail cluster,
how many concurrent users do you support with replication enabled, do
you use synchronous or asynchronous replication, are you using it in an
active/active or active/passive state, is it possible to have a cluster
with multiple servers at each site hosting the same mail data, does
dysnc replication scale well (10,000 -> 100,000 -> 1,000,000 users)?
Just trying to get a good feel for whether dsync replication is capable
of handling the use case I am proposing before investing too much time
in testing it.
On 6.7.2012, at 23.28, list@airstreamcomm.net wrote:
Thanks, that certainly helps identify the configuration options. However I am more concerned about the experiences of others who have actually used the replication. What is the rate of change on your mail cluster, how many concurrent users do you support with replication enabled, do you use synchronous or asynchronous replication, are you using it in an active/active or active/passive state, is it possible to have a cluster with multiple servers at each site hosting the same mail data, does dysnc replication scale well (10,000 -> 100,000 -> 1,000,000 users)? Just trying to get a good feel for whether dsync replication is capable of handling the use case I am proposing before investing too much time in testing it.
I wouldn't use it for large systems yet. It is still pretty inefficient. v2.2 will have a redesigned dsync that can do incremental syncs much faster and with less bandwidth.
Anyway, in my small installation I'm using it in active-active mode and it works well enough. I've even configured my clients intentionally so that they use different servers.
On 7/6/12 9:56 PM, Timo Sirainen wrote:
On 6.7.2012, at 23.28, list@airstreamcomm.net wrote:
Thanks, that certainly helps identify the configuration options. However I am more concerned about the experiences of others who have actually used the replication. What is the rate of change on your mail cluster, how many concurrent users do you support with replication enabled, do you use synchronous or asynchronous replication, are you using it in an active/active or active/passive state, is it possible to have a cluster with multiple servers at each site hosting the same mail data, does dysnc replication scale well (10,000 -> 100,000 -> 1,000,000 users)? Just trying to get a good feel for whether dsync replication is capable of handling the use case I am proposing before investing too much time in testing it. I wouldn't use it for large systems yet. It is still pretty inefficient. v2.2 will have a redesigned dsync that can do incremental syncs much faster and with less bandwidth.
Anyway, in my small installation I'm using it in active-active mode and it works well enough. I've even configured my clients intentionally so that they use different servers.
Timo,
Does dsync replication only work between two hosts? In my scenario I would have two sites with X number of nodes at each with an NFS backend for each site. For this example lets say I have site A with two nodes that mount one NFS share, and site B with two nodes that mount one NFS share. Is it possible to implement dsync replication between these two clusters of nodes?
list@airstreamcomm.net wrote:
On 7/6/12 9:56 PM, Timo Sirainen wrote:
On 6.7.2012, at 23.28, list@airstreamcomm.net wrote:
Thanks, that certainly helps identify the configuration options. However I am more concerned about the experiences of others who have actually used the replication. What is the rate of change on your mail cluster, how many concurrent users do you support with replication enabled, do you use synchronous or asynchronous replication, are you using it in an active/active or active/passive state, is it possible to have a cluster with multiple servers at each site hosting the same mail data, does dysnc replication scale well (10,000 -> 100,000 -> 1,000,000 users)? Just trying to get a good feel for whether dsync replication is capable of handling the use case I am proposing before investing too much time in testing it.
I wouldn't use it for large systems yet. It is still pretty inefficient. v2.2 will have a redesigned dsync that can do incremental syncs much faster and with less bandwidth. Anyway, in my small installation I'm using it in active-active mode and it works well enough. I've even configured my clients intentionally so that they use different servers.
Does dsync replication only work between two hosts? In my scenario I would have two sites with X number of nodes at each with an NFS backend for each site. For this example lets say I have site A with two nodes that mount one NFS share, and site B with two nodes that mount one NFS share. Is it possible to implement dsync replication between these two clusters of nodes?
If you respect http://wiki2.dovecot.org/NFS and setup a director http://wiki2.dovecot.org/Director including a doveadm proxy it could work.
dsync is part of "doveadm backup", so you just have to get your director setup right so that imap, pop3, lmtp and doveadm services are always proxied to the correct nfs-client-node at the local site.
Regards Daniel
On 18.7.2012, at 19.24, list@airstreamcomm.net wrote:
Anyway, in my small installation I'm using it in active-active mode and it works well enough. I've even configured my clients intentionally so that they use different servers.
Does dsync replication only work between two hosts? In my scenario I would have two sites with X number of nodes at each with an NFS backend for each site. For this example lets say I have site A with two nodes that mount one NFS share, and site B with two nodes that mount one NFS share. Is it possible to implement dsync replication between these two clusters of nodes?
You can have as many hosts as you want, but you most likely don't want to use active-active setup via NFS or you'll run into NFS caching (= corruption) problems.
participants (4)
-
Daniel Parthey
-
list＠airstreamcomm.net
-
Timo Sirainen
-
Костырев Александр Алексеевич