Hello,
Does the community version of dovecot have the replication feature? When one dovecot was down, another one could take over the tasks.
Thanks.
Your E-Mail. Your Cloud. Your Office. eclipso Mail Europe. https://www.eclipso.de
On 12/07/2024 13:05, Jeff Pang via dovecot wrote:
Hello,
Does the community version of dovecot have the replication feature? When one dovecot was down, another one could take over the tasks.
Thanks.
Jeff
Replication is in the current dovecot version but will go away in 2.4.
The doveadm sync feature is staying. So with some work you can set it up what you are requesting.
I used to use replication and now I'm thinking about using sync but have not implemented it. The following are thoughs on it.
There are some points to be addressed that are outside dovecot. I think you'd have to make sure that your sync happened frequently enough that you could live with losing the emails that arrives bewteen syncs for example. That would tend to lead to a requirement to sync more frequently and reduce risk of email loss. But then you'd need to avoid more than one sync being active simultaneously (that is my assumption that this would not work, but I don't know if it is a real problem).
The failover would require you to stop people accessing the old server, stop syncing and start the backup instance. When the main instance is available to come back on line, you'd need to stop the backup instance, sync in the opposite direction and then start dovecot and re-enable the sync mechansim towards the backup.
You need to ensure people don't connect simultaneously to both instances. So some thought would be needed about those cases where the main node goes and then comes backup to ensure that your sync is not still active at that point and replicates the old state onto the backup server and to ensure then people don't start connecting to it again without being able to control it.
It's a disaster recover rather than high availability solution, but I think it can work. Others may already have got working implementations and thought through some of the implications.
John
Replication is in the current dovecot version but will go away in 2.4.
The doveadm sync feature is staying. So with some work you can set it up what you are requesting.
I used to use replication and now I'm thinking about using sync but have not implemented it. The following are thoughs on it.
There are some points to be addressed that are outside dovecot. I think you'd have to make sure that your sync happened frequently enough that you could live with losing the emails that arrives bewteen syncs for example.
I have been thinking of writing a hacky delivery script that passes the email on to dovecot-lda, then runs doveadm sync, and only returns success after the sync is done (or after a timeout). No idea what problems I will run into, but the idea is to never accept an email until it's guaranteed replicated.
That would tend to lead to a requirement to sync more frequently and reduce risk of email loss. But then you'd need to avoid more than one sync being active simultaneously (that is my assumption that this would not work, but I don't know if it is a real problem).
Doesn't the -l option protect against simultaneous syncs? Just based on reading the doveadm-sync man page. (I guess it could cause a problem if you start sync processes more quickly than they can finish.)
NB I'm just running a one-person email server so don't take my ideas too seriously :)
-- James
On 12/07/2024 17:38, James Cook via dovecot wrote:
Replication is in the current dovecot version but will go away in 2.4.
The doveadm sync feature is staying. So with some work you can set it up what you are requesting.
I used to use replication and now I'm thinking about using sync but have not implemented it. The following are thoughs on it.
There are some points to be addressed that are outside dovecot. I think you'd have to make sure that your sync happened frequently enough that you could live with losing the emails that arrives bewteen syncs for example.
I have been thinking of writing a hacky delivery script that passes the email on to dovecot-lda, then runs doveadm sync, and only returns success after the sync is done (or after a timeout). No idea what problems I will run into, but the idea is to never accept an email until it's guaranteed replicated.
That would tend to lead to a requirement to sync more frequently and reduce risk of email loss. But then you'd need to avoid more than one sync being active simultaneously (that is my assumption that this would not work, but I don't know if it is a real problem).
Doesn't the -l option protect against simultaneous syncs? Just based on reading the doveadm-sync man page. (I guess it could cause a problem if you start sync processes more quickly than they can finish.)
NB I'm just running a one-person email server so don't take my ideas too seriously :)
Hi James
I want to avoid the -1 parameter because it doesn't do deletes in the target. It might not be an issue, but I'd like to keep the target the same as the source. I don't know if it would protect against simultaneous jobs, but I dont' know even if that is an issue. I was making the assumption it could be so I plan to avoid it somehow.
As for the lda to doveadm sync script, I have been wondering too about how to close the gap for emails arriving between syncs, even though the risk might not be so significant.
With delivering to two dovecot servers before accepting the email, either one going down will stop email delivery. That's an inconvenience but without necessarily introducing risks for losing the emails. Ultimately it depends on where those undelivered emails are being kept in the meantime (presumably MTA queue) and whether they are safer there than being delivered to a single instance. Though that is related only to the downtime. When both are up the risk would be mitigated by the solution.
Other idea I was thinking of is a replicated file system. That could make emails available in real time to both dovecot instances assuming a functionality to sync to disk on both/multiple nodes before replying back to dovecot to accept the email. I believe there would still need to be only one dovecot instance active at a time. However, I have heard of but don't know personally of horror stories about email outages on clustered file systems, which leads me to believe that there would still be sufficient residual risk to require a backup copy of the mailboxes with doveadm sync or other tools.
John
John
On Fri, Jul 12, 2024 at 06:28:13PM GMT, John Fawcett via dovecot wrote:
Hi James
I want to avoid the -1 parameter because it doesn't do deletes in the target.
-l, not -1.
As for the lda to doveadm sync script, I have been wondering too about how to close the gap for emails arriving between syncs, even though the risk might not be so significant.
With delivering to two dovecot servers before accepting the email, either one going down will stop email delivery.
I was thinking my script will accept the email anyway if the sync fails. It would do this:
Pass to dovecot-lda. If dovecot-lda fails, something is seriously wrong, so stop and fail.
Fork a background process that attempts to doveadm sync.
Wait for the background process to finish, or a maximum of 2 seconds. Then return success regardless of whether sync worked.
This guards against a hard disk crash or filesystem failure on one machine, but falls back to single-homing the email if a server is down.
This is inspired by the documentation at https://doc.dovecot.org/configuration_manual/replication/
-- James
On 07/12/2024 1:14 PM MDT James Cook via dovecot <dovecot@dovecot.org> wrote:
On Fri, Jul 12, 2024 at 06:28:13PM GMT, John Fawcett via dovecot wrote:
Hi James
I want to avoid the -1 parameter because it doesn't do deletes in the target.
-l, not -1.
No, it's -1 - as in one(1)-way sync.
-l (lowercase L) is for locking.
michael
That's a nice suggestion. Thanks John.
On Fri, Jul 12, 2024 at 11:25 PM John Fawcett via dovecot <dovecot@dovecot.org> wrote:
On 12/07/2024 13:05, Jeff Pang via dovecot wrote:
Hello,
Does the community version of dovecot have the replication feature? When one dovecot was down, another one could take over the tasks.
Thanks.
Jeff
Replication is in the current dovecot version but will go away in 2.4.
The doveadm sync feature is staying. So with some work you can set it up what you are requesting.
I used to use replication and now I'm thinking about using sync but have not implemented it. The following are thoughs on it.
There are some points to be addressed that are outside dovecot. I think you'd have to make sure that your sync happened frequently enough that you could live with losing the emails that arrives bewteen syncs for example. That would tend to lead to a requirement to sync more frequently and reduce risk of email loss. But then you'd need to avoid more than one sync being active simultaneously (that is my assumption that this would not work, but I don't know if it is a real problem).
The failover would require you to stop people accessing the old server, stop syncing and start the backup instance. When the main instance is available to come back on line, you'd need to stop the backup instance, sync in the opposite direction and then start dovecot and re-enable the sync mechansim towards the backup.
You need to ensure people don't connect simultaneously to both instances. So some thought would be needed about those cases where the main node goes and then comes backup to ensure that your sync is not still active at that point and replicates the old state onto the backup server and to ensure then people don't start connecting to it again without being able to control it.
It's a disaster recover rather than high availability solution, but I think it can work. Others may already have got working implementations and thought through some of the implications.
John
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
Your E-Mail. Your Cloud. Your Office. eclipso Mail Europe. https://www.eclipso.de
participants (4)
-
James Cook
-
Jeff Pang
-
John Fawcett
-
Michael Slusarz