Replication going away?
Hello everyone.
Just saw this commit in the official Github repo:
https://github.com/dovecot/core/commit/4c04e4c30fd4817a8b0e11d04d9681173f696...
Looking at the commit details, it appears that it completely removes the replication feature. I'm a bit perplexed by this change and am not sure what might be the justification for it. Personally, I find replication to be very useful, as it allows me to maintain a synchronized mirror of all of my mailboxes on my home server, for use as backup in case the primary server goes down for some reason.
Perhaps there's some sort of replacement being planned for this feature? Or maybe the relevant code is simply going to be refactored to a plugin or external program, and there's nothing to worry about at all?
In any case, I'd greatly appreciate if one of the developers could comment on this change.
Hello,
Just like Vladimir, I'm a bit concerned about this change, and I'd really appreciate if someone could let us know if the replication feature (that works so well!) will be replaced or removed; and, in case of removal, what would be recommended replacement? Thanks in advance and best regards, Daniele
On 09-Jul-23 9:36 PM, Vladimir Mishonov via dovecot wrote:
Hello everyone.
Just saw this commit in the official Github repo:
https://github.com/dovecot/core/commit/4c04e4c30fd4817a8b0e11d04d9681173f696...
Looking at the commit details, it appears that it completely removes the replication feature. I'm a bit perplexed by this change and am not sure what might be the justification for it. Personally, I find replication to be very useful, as it allows me to maintain a synchronized mirror of all of my mailboxes on my home server, for use as backup in case the primary server goes down for some reason.
Perhaps there's some sort of replacement being planned for this feature? Or maybe the relevant code is simply going to be refactored to a plugin or external program, and there's nothing to worry about at all?
In any case, I'd greatly appreciate if one of the developers could comment on this change.
Top posting because nothing specific to reply to, sorry. Not exactly sure, but there’s another thread about the removal of Director in favour of Dovecot Pro on 3.x. Perhaps this change is related.
William Edwards
Op 16 jul. 2023 om 16:33 heeft Daniele danix@kernel-panic.it het volgende geschreven:
Hello,
Just like Vladimir, I'm a bit concerned about this change, and I'd really appreciate if someone could let us know if the replication feature (that works so well!) will be replaced or removed; and, in case of removal, what would be recommended replacement? Thanks in advance and best regards, Daniele
On 09-Jul-23 9:36 PM, Vladimir Mishonov via dovecot wrote: Hello everyone.
Just saw this commit in the official Github repo:
https://github.com/dovecot/core/commit/4c04e4c30fd4817a8b0e11d04d9681173f696...
Looking at the commit details, it appears that it completely removes the replication feature. I'm a bit perplexed by this change and am not sure what might be the justification for it. Personally, I find replication to be very useful, as it allows me to maintain a synchronized mirror of all of my mailboxes on my home server, for use as backup in case the primary server goes down for some reason.
Perhaps there's some sort of replacement being planned for this feature? Or maybe the relevant code is simply going to be refactored to a plugin or external program, and there's nothing to worry about at all?
In any case, I'd greatly appreciate if one of the developers could comment on this change.
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
Hi!
Yes, director and replicator are removed, and won't be available for pro users either.
For NFS setups (or similar shared setups), we have documented a way to use Lua to run a director-like setup, see
https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/
Regards to replication, doveadm sync is not being removed. So you can still run doveadm sync on your system to have a primary / backup setup.
Aki
On 16/07/2023 18:34 EEST William Edwards via dovecot dovecot@dovecot.org wrote:
Top posting because nothing specific to reply to, sorry. Not exactly sure, but there’s another thread about the removal of Director in favour of Dovecot Pro on 3.x. Perhaps this change is related.
William Edwards
Op 16 jul. 2023 om 16:33 heeft Daniele danix@kernel-panic.it het volgende geschreven:
Hello,
Just like Vladimir, I'm a bit concerned about this change, and I'd really appreciate if someone could let us know if the replication feature (that works so well!) will be replaced or removed; and, in case of removal, what would be recommended replacement? Thanks in advance and best regards, Daniele
On 09-Jul-23 9:36 PM, Vladimir Mishonov via dovecot wrote: Hello everyone.
Just saw this commit in the official Github repo:
https://github.com/dovecot/core/commit/4c04e4c30fd4817a8b0e11d04d9681173f696...
Looking at the commit details, it appears that it completely removes the replication feature. I'm a bit perplexed by this change and am not sure what might be the justification for it. Personally, I find replication to be very useful, as it allows me to maintain a synchronized mirror of all of my mailboxes on my home server, for use as backup in case the primary server goes down for some reason.
Perhaps there's some sort of replacement being planned for this feature? Or maybe the relevant code is simply going to be refactored to a plugin or external program, and there's nothing to worry about at all?
In any case, I'd greatly appreciate if one of the developers could comment on this change.
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
Le dim. 16 juil. 2023, 18:55, Aki Tuomi via dovecot dovecot@dovecot.org a écrit :
Hi!
Yes, director and replicator are removed, and won't be available for pro users either.
For NFS setups (or similar shared setups), we have documented a way to use Lua to run a director-like setup, see
https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/
Regards to replication, doveadm sync is not being removed. So you can still run doveadm sync on your system to have a primary / backup setup
That's completely crazy ! What is the rationale ?
Emmanuel.
Emmanuel Fusté manu.fuste@gmail.com wrote:
Le dim. 16 juil. 2023, 18:55, Aki Tuomi via dovecot dovecot@dovecot.org a écrit :
Yes, director and replicator are removed, and won't be available for pro users either.
Why in hell would one remove replicator? It's working for years now. Yes, I recall issues in the beginning, and others and me helped Timo in debugging/testing. After that it runs without any flaws.
So why removing it?
Regards to replication, doveadm sync is not being removed. So you can still run doveadm sync on your system to have a primary / backup setup
AND: What do you believe an alternative should be, for a failover scenario of two IMAP servers? doveadm sync is not! That's why replicator has been implemented!
That's completely crazy !
+1
Regards, Michael
On 16/07/2023 17:54, Aki Tuomi via dovecot wrote:
Hi!
Yes, director and replicator are removed, and won't be available for pro users either.
For NFS setups (or similar shared setups), we have documented a way to use Lua to run a director-like setup, see
https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/
With respect, I'm not sure why these scripts are considered a suitable replacement, because they're not, and it's obvious no real attempt was made to make them so.
Putting aside things relatively trivial to add, like weighted instead of random mappings, the meat of the issue is:
When a backend is removed (fails, or gracefully taken out), the script remaps connecting users that were mapped to it to a different backend. This sounds obvious enough.
However, when that backend comes back up, users aren't mapped back onto it - the users now have mappings elsewhere. Maybe you got lucky and some users that were mapped onto it just didn't connect the whole time it was down, maybe you got a few new users, but for the majority of your active user base, you now have a large imbalance in your user mappings. The backend has stopped existing for them.
So you have to rebalance your user mappings manually.
Rebalancing requires going through all your user mappings. This quickly becomes prohibitively expensive as number of users increases. However, it also leads us on to the next issue.
Adding a new backend, or returning a failed backend, requires remapping your users, but you can't remap one that's currently connected to a backend without risking a new connection to the wrong backend.
Either you don't balance/add backends in a way that covers any connected user going forward (which isn't really useful) or you have to start kicking your users (on all of your balancers individually, you don't have the control director gave you any more!) and remapping to rebalance the overall configuration.
We know that a user will immediately attempt to reconnect when it's kicked, this is the nature of imap clients. You now have a race condition, the only solution to which is to lock that user out while you update the database. I'm not even sure how you'd cover this.
So it's all gone from automatic to painfully manual with large overheads and race conditions.
At the very least, to retain the very basic level of suitability would require:
- mappings in the database being given a TTL after which they're no longer
considered valid. This is trivial, however: 2. importantly, mappings exceeding TTL must *still* be considered valid if the user has existing connections on any balancer. Good luck with that!
And all this doesn't even cover the other deficiencies. Some easy enough to add, some (involving addressing all balancers at once) far less so.
Unfortunately I can't speak for our future plans; but I'd personally remain on 2.x director for the proxies for as long as possible, then potentially look at the feasibility of porting the director code to 3.x.
Again, I really don't understand the thinking behind saying the lua scripts are anything like a suitable replacement. Our team were considering using Pro when the decision to drop director was announced, we backed off quickly as a result.
On 17/07/2023 13:24 EEST David Zambonini dzambonini@names.co.uk wrote:
On 16/07/2023 17:54, Aki Tuomi via dovecot wrote:
Hi!
Yes, director and replicator are removed, and won't be available for pro users either.
For NFS setups (or similar shared setups), we have documented a way to use Lua to run a director-like setup, see
https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/
With respect, I'm not sure why these scripts are considered a suitable replacement, because they're not, and it's obvious no real attempt was made to make them so.
Putting aside things relatively trivial to add, like weighted instead of random mappings, the meat of the issue is:
When a backend is removed (fails, or gracefully taken out), the script remaps connecting users that were mapped to it to a different backend. This sounds obvious enough.
However, when that backend comes back up, users aren't mapped back onto it - the users now have mappings elsewhere. Maybe you got lucky and some users that were mapped onto it just didn't connect the whole time it was down, maybe you got a few new users, but for the majority of your active user base, you now have a large imbalance in your user mappings. The backend has stopped existing for them.
So you have to rebalance your user mappings manually.
Rebalancing requires going through all your user mappings. This quickly becomes prohibitively expensive as number of users increases. However, it also leads us on to the next issue.
Adding a new backend, or returning a failed backend, requires remapping your users, but you can't remap one that's currently connected to a backend without risking a new connection to the wrong backend.
Either you don't balance/add backends in a way that covers any connected user going forward (which isn't really useful) or you have to start kicking your users (on all of your balancers individually, you don't have the control director gave you any more!) and remapping to rebalance the overall configuration.
We know that a user will immediately attempt to reconnect when it's kicked, this is the nature of imap clients. You now have a race condition, the only solution to which is to lock that user out while you update the database. I'm not even sure how you'd cover this.
So it's all gone from automatic to painfully manual with large overheads and race conditions.
At the very least, to retain the very basic level of suitability would require:
- mappings in the database being given a TTL after which they're no longer
considered valid. This is trivial, however: 2. importantly, mappings exceeding TTL must *still* be considered valid if the user has existing connections on any balancer. Good luck with that!
And all this doesn't even cover the other deficiencies. Some easy enough to add, some (involving addressing all balancers at once) far less so.
Unfortunately I can't speak for our future plans; but I'd personally remain on 2.x director for the proxies for as long as possible, then potentially look at the feasibility of porting the director code to 3.x.
Again, I really don't understand the thinking behind saying the lua scripts are anything like a suitable replacement. Our team were considering using Pro when the decision to drop director was announced, we backed off quickly as a result.
-- David Zambonini
You do note that most of the issues you just described were actually issues with director itself. The director did not do any automatic healing, load balancing etc, which is why we have a Pro offering that provides actual clustering component (called Palomar architecture).
Those Lua scripts are OK replacement for a do it yourself environment.
Aki
On 17/07/2023 11:41, Aki Tuomi wrote:
You do note that most of the issues you just described were actually issues with director itself. The director did not do any automatic healing, load balancing etc, which is why we have a Pro offering that provides actual clustering component (called Palomar architecture).
I'd not raised those points. Load balancers handle first level load balancing in both scenarios, in any case. poolmon has always adequately managed healing.
To reiterate, we know director mappings are a live mapping set, not a permanent record; there's literally a timeout. Unless I'm in a hurry to re-equalise, I've seldom had to manually intervene. It sorts itself out in a few days.
What I've never had to do is constantly mess around in a database and kick users manually all the time as a _requirement_ just to keep things running. We both know it's not an adequate replacement for director.
Those Lua scripts are OK replacement for a do it yourself environment.
Putting aside the subtext of the retiring of enterprise features outside of Pro, with Pro the issue of the cost involved in being forced to shift to an entirely different architecture now exists.
On 17/7/2023 1:24 μ.μ., David Zambonini wrote:
With respect, I'm not sure why these scripts are considered a suitable replacement, because they're not, and it's obvious no real attempt was made to make them so. ...
+1000
We recognize and respect all Dovecot Team's efforts and goodwill for many years now. You guys rock.
But please do not mess thousands of mail setups around the globe where Replication is already in use.
Please continue the support of these really valuable - for a really huge number of mail admins - features.
I strongly believe that the vast majority of the community will be very disappointed by this feature removal and it will be a pity for such a rich and collaborative community.
My 2c.
All the best, Nick
Just to understand that correctly: I could setup a (cron) based process for doveadm sync, but no longer a setup like plugin { mail_replica = tcp:$IMAP_REPLICA_SERVER:$IMAP_REPLICA_PORT } where the cron would lead to some delay and would have to check for concurrent jobs?
On 18/07/2023 15:19 EEST info@joergschulz.de wrote:
Just to understand that correctly: I could setup a (cron) based process for doveadm sync, but no longer a setup like plugin { mail_replica = tcp:$IMAP_REPLICA_SERVER:$IMAP_REPLICA_PORT } where the cron would lead to some delay and would have to check for concurrent jobs?
You can also have that too.
doveadm sync -d
makes it use mail_replica setting.
Aki
Just to understand that correctly: I could setup a (cron) based process for doveadm sync, but no longer a setup like plugin { mail_replica = tcp:$IMAP_REPLICA_SERVER:$IMAP_REPLICA_PORT } where the cron would lead to some delay and would have to check for concurrent jobs?
You can also have that too.
doveadm sync -d
makes it use mail_replica setting.
@Aki:
Is it possible to monitor actions that would have triggered replication in dovecot < 2.4, e.g. parsing logs or a lua script to imitate the previous behaviour?
@Michael Slusarz:
While I understand it takes effort to maintain the replication plugin, this is especially problematic for small active/active high-availability deployments. I guess there are lots of servers that use replication for just 50 or 100 mailboxes. Cloudstorage (like S3) would be overkill for these.
Do you provide dovecot pro subscriptions for such small deployments?
The basic replication functionality with dsync seems to be available in future versions. Would you consider releasing a "dovecot smb" version for small/medium businesses that maintains the replication plugin (without director) for a yearly subscription fee? Otherwise those affected would have to look for alternatives. The recent release of rust-written Stalwart mailserver comes to mind, that bundles a lot of functionality (smtp, pop, imap, jmap, s3, sieve, dkim/arc/spf/dmarc, dane, mta-sas, ...): https://stalw.art/blog/
Best regards, Gerald
While I understand it takes effort to maintain the replication plugin, this is especially problematic for small active/active high-availability deployments. I guess there are lots of servers that use replication for just 50 or 100 mailboxes. Cloudstorage (like S3) would be overkill for these.
Do you provide dovecot pro subscriptions for such small deployments?
I’m running such a setup for 5-10 mailboxes.. :D
Hi Guys,
Just asking, i don't see a new dovecot version since last December, and the girhub repo has no new requests/updates. Is there a new place for dovecot versions, or it's happening something?
Thanks in advanced, Jorge,
On 19/07/2023 01:22 EEST Jorge Bastos mysql.jorge@decimal.pt wrote:
Hi Guys, Just asking, i don't see a new dovecot version since last December, and the girhub repo has no new requests/updates. Is there a new place for dovecot versions, or it's happening something? Thanks in advanced, Jorge,
Our github repo has been last updated 2 days ago, so not sure what you mean... We are working on new major release, so this usually causes a longer pause between releases.
Aki
Ops, my bad, you're right.
Ignore my email.
On 2023-07-19 5:58, Aki Tuomi wrote:
On 19/07/2023 01:22 EEST Jorge Bastos mysql.jorge@decimal.pt wrote:
Hi Guys, Just asking, i don't see a new dovecot version since last December, and the girhub repo has no new requests/updates. Is there a new place for dovecot versions, or it's happening something? Thanks in advanced, Jorge,
Our github repo has been last updated 2 days ago, so not sure what you mean... We are working on new major release, so this usually causes a longer pause between releases.
Aki
On 2023-07-18, Gerald Galster list+dovecot@gcore.biz wrote:
While I understand it takes effort to maintain the replication plugin, this is especially problematic for small active/active high-availability deployments. I guess there are lots of servers that use replication for just 50 or 100 mailboxes. Cloudstorage (like S3) would be overkill for these.
Even without active/active, it's super useful for the simple active/backup configuration which I use on my personal mail server setup (one colo box, one home server) and a small company mail server; as such I'm pretty sad to see it go. Still, it is up to OX where they want to put their resources.
I guess losing repl probably doesn't affect larger ISP type setups so much; it seems a bit more common to use shared storage (e.g. maildirs on an nfs appliance or similar) in those cases if they're actually running their own storage.
Do you provide dovecot pro subscriptions for such small deployments?
Unless I misunderstood the message (and I don't think I did), repl was removed in pro too. (I don't expect that pro is available on my usual choice of OS anyway..).
While I understand it takes effort to maintain the replication plugin, this is especially problematic for small active/active high-availability deployments. I guess there are lots of servers that use replication for just 50 or 100 mailboxes. Cloudstorage (like S3) would be overkill for these.
Even without active/active, it's super useful for the simple active/backup configuration which I use on my personal mail server
This depends heavily on individual usage. Coming from an active/active deployment it's a major step backwards though: usually two servers are running independently in geographically dispersed datacenters. High-availabilty is achieved by a simple DNS entry that returns two ip addresses, one from each datacenter. Under normal circumstances that gives you 50/50 loadbalancing without loadbalancers, without additional components that can fail. In case one datacenter goes down, and that happens to every datacenter at some time, the other datacenter takes over - automatically, without any configuration changes. Additionally mail user agents (Outlook, Thunderbird, ...) don't need special configuration. If one ip address is unrechable they connect to the other one obtained via DNS and users can quite seemlessly send and receive email again. After the outage ceased and the other datacenter is back online again, there is nothing to do. No configuration changes, no error prone manual synchronization or promoting passive to active - it just works and heals itself. Being used to a carefree setup like that you don't want to go back.
Of course there are other possibilities like nfs, glusterfs, gfs2, zfs snapshots, ceph, minio or dsync backup but they all have their own drawbacks. For small mailservers that want high availability dsync replication is quite the perfect solution.
setup (one colo box, one home server) and a small company mail server; as such I'm pretty sad to see it go. Still, it is up to OX where they want to put their resources.
Well, it seems the dsync replication function is still there, just the replication plugin that notifies what to replicate is deprectated. Of course it's OX's decision, I'm just hoping they were not aware how useful replication is in the before mentioned scenario.
Moreover I'm quite sure this kind of small-scale replication does not have any impact on customers upgrading to the new cloud architecture. Big customers will go for cloud because it scales way better and does not have replication induced performance penalties and small customers probably can't afford to upgrade because it's too pricey.
I guess losing repl probably doesn't affect larger ISP type setups so much; it seems a bit more common to use shared storage (e.g. maildirs on an nfs appliance or similar) in those cases if they're actually running their own storage.
Do you provide dovecot pro subscriptions for such small deployments?
Unless I misunderstood the message (and I don't think I did), repl was removed in pro too. (I don't expect that pro is available on my usual choice of OS anyway..).
As I understood it dsync is still working. Replication configured via ssh is calling dsync under the hood, so if local storage and index/log formats don't change for single deployments, it seems to be more of a political decision. I know maintenance is not for free, that's why I suggested to think about a dovecot small/medium business edition with a more affordable price tag.
Best regards, Gerald
Real world is a bit different.. DNS Caching.. While DNS Round Robin is good enough to distribute loads, it isnt' a very good method for failover, even with a very short TTL. Many home routers, still insist on caching results for a long time, no matter what the TTL says, and of course Windows internal caching etc..
Should not confuse the issue.. call it a 'poor man's load balancer' if you will, but it more of a last line failover, and during the time it takes for DNS to retry, and find another active node, an AWFUL lot of disgruntled customers will be calling ;)
Also so interesting to see some resolvers that don't think of using the second record, if the first one is down..
On 2023-07-18 17:09, Gerald Galster wrote:
While I understand it takes effort to maintain the replication plugin, this is especially problematic for small active/active high-availability deployments. I guess there are lots of servers that use replication for just 50 or 100 mailboxes. Cloudstorage (like S3) would be overkill for these.
Even without active/active, it's super useful for the simple active/backup configuration which I use on my personal mail server
This depends heavily on individual usage. Coming from an active/active deployment it's a major step backwards though: usually two servers are running independently in geographically dispersed datacenters. High-availabilty is achieved by a simple DNS entry that returns two ip addresses, one from each datacenter. Under normal circumstances that gives you 50/50 loadbalancing without loadbalancers, without additional components that can fail. In case one datacenter goes down, and that happens to every datacenter at some time, the other datacenter takes over - automatically, without any configuration changes. Additionally mail user agents (Outlook, Thunderbird, ...) don't need special configuration. If one ip address is unrechable they connect to the other one obtained via DNS and users can quite seemlessly send and receive email again. After the outage ceased and the other datacenter is back online again, there is nothing to do. No configuration changes, no error prone manual synchronization or promoting passive to active - it just works and heals itself. Being used to a carefree setup like that you don't want to go back.
Of course there are other possibilities like nfs, glusterfs, gfs2, zfs snapshots, ceph, minio or dsync backup but they all have their own drawbacks. For small mailservers that want high availability dsync replication is quite the perfect solution.
setup (one colo box, one home server) and a small company mail server; as such I'm pretty sad to see it go. Still, it is up to OX where they want to put their resources.
Well, it seems the dsync replication function is still there, just the replication plugin that notifies what to replicate is deprectated. Of course it's OX's decision, I'm just hoping they were not aware how useful replication is in the before mentioned scenario.
Moreover I'm quite sure this kind of small-scale replication does not have any impact on customers upgrading to the new cloud architecture. Big customers will go for cloud because it scales way better and does not have replication induced performance penalties and small customers probably can't afford to upgrade because it's too pricey.
I guess losing repl probably doesn't affect larger ISP type setups so much; it seems a bit more common to use shared storage (e.g. maildirs on an nfs appliance or similar) in those cases if they're actually running their own storage.
Do you provide dovecot pro subscriptions for such small deployments?
Unless I misunderstood the message (and I don't think I did), repl was removed in pro too. (I don't expect that pro is available on my usual choice of OS anyway..).
As I understood it dsync is still working. Replication configured via ssh is calling dsync under the hood, so if local storage and index/log formats don't change for single deployments, it seems to be more of a political decision. I know maintenance is not for free, that's why I suggested to think about a dovecot small/medium business edition with a more affordable price tag.
Best regards, Gerald _______________________________________________ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
Le 19/07/2023 à 19:53, Michael Peddemors a écrit :
Real world is a bit different.. DNS Caching.. While DNS Round Robin is good enough to distribute loads, it isnt' a very good method for failover, even with a very short TTL. Many home routers, still insist on caching results for a long time, no matter what the TTL says, and of course Windows internal caching etc..
Should not confuse the issue.. call it a 'poor man's load balancer' if you will, but it more of a last line failover, and during the time it takes for DNS to retry, and find another active node, an AWFUL lot of disgruntled customers will be calling ;)
Also so interesting to see some resolvers that don't think of using the second record, if the first one is down..
You're mixing things : DNS and Mail client behavior. It is a non sense. A resolver will serve records, It does not use them and do not care of what is behind the record. A good client use the lists (of A or AAAA) records to connect to the server and will iterate on the list if the server behind the record is down. And DNS caching do it job nothing less, nothing more and is out of the picture.
Emmanuel.
Le 19/07/2023 à 19:53, Michael Peddemors a écrit :
Real world is a bit different.. DNS Caching.. While DNS Round Robin is good enough to distribute loads, it isnt' a very good method for failover, even with a very short TTL. Many home routers, still insist on caching results for a long time, no matter what the TTL says, and of course Windows internal caching etc..
Should not confuse the issue.. call it a 'poor man's load balancer' if you will, but it more of a last line failover, and during the time it takes for DNS to retry, and find another active node, an AWFUL lot of disgruntled customers will be calling ;)
Also so interesting to see some resolvers that don't think of using the second record, if the first one is down..
You're mixing things : DNS and Mail client behavior. It is a non sense. A resolver will serve records, It does not use them and do not care of what is behind the record. A good client use the lists (of A or AAAA) records to connect to the server and will iterate on the list if the server behind the record is down. And DNS caching do it job nothing less, nothing more and is out of the picture.
Emmanuel is right. Here's an example to clarify:
$ dig imap.web.de
;; ANSWER SECTION: imap.web.de. 226 IN A 212.227.17.178 imap.web.de. 226 IN A 212.227.17.162
A dns query for imap.web.de address records (IN A) returns two ip addresses. A local resolver receives those two ip addresses and usually passes them on to clients while it may rotate the order, so that some clients will see 212.227.17.178, 212.227.17.162 and others will see 212.227.17.162, 212.227.17.178. It is possible to get the same order for subsequent requests but on a *global* scale that roughly equals 50/50 loadbalancing.
Mail clients then connect to e.g. 212.227.17.178 and try 212.227.17.162 on connection failure without any further dns involvement. Dns caching (ttl) is irrelevant in that case.
Best regards, Gerald
On 2023-07-19 12:55, Gerald Galster wrote:
Le 19/07/2023 à 19:53, Michael Peddemors a écrit :
Real world is a bit different.. DNS Caching.. While DNS Round Robin is good enough to distribute loads, it isnt' a very good method for failover, even with a very short TTL. Many home routers, still insist on caching results for a long time, no matter what the TTL says, and of course Windows internal caching etc..
Should not confuse the issue.. call it a 'poor man's load balancer' if you will, but it more of a last line failover, and during the time it takes for DNS to retry, and find another active node, an AWFUL lot of disgruntled customers will be calling ;)
Also so interesting to see some resolvers that don't think of using the second record, if the first one is down..
You're mixing things : DNS and Mail client behavior. It is a non sense. A resolver will serve records, It does not use them and do not care of what is behind the record. A good client use the lists (of A or AAAA) records to connect to the server and will iterate on the list if the server behind the record is down. And DNS caching do it job nothing less, nothing more and is out of the picture.
Emmanuel is right. Here's an example to clarify:
$ dig imap.web.de
;; ANSWER SECTION: imap.web.de. 226 IN A 212.227.17.178 imap.web.de. 226 IN A 212.227.17.162
A dns query for imap.web.de address records (IN A) returns two ip addresses. A local resolver receives those two ip addresses and usually passes them on to clients while it may rotate the order, so that some clients will see 212.227.17.178, 212.227.17.162 and others will see 212.227.17.162, 212.227.17.178. It is possible to get the same order for subsequent requests but on a *global* scale that roughly equals 50/50 loadbalancing.
Mail clients then connect to e.g. 212.227.17.178 and try 212.227.17.162 on connection failure without any further dns involvement. Dns caching (ttl) is irrelevant in that case.
In theory, that is how it is SUPPOSED to work, in practice (and we have lots of history where customers ran into this problem when one went down), I believe that it was Outlook that didn't try an alternative IP address for a 20 min internal cache for instance, before a requery of the DNS was done, at which time it again would choose which IP to connect to. As well, SOME modems would get the two results, and return only one to the client. And lots of libraries we see, do the DNS query, get two IP results, but then only use the first one returned, etc..
Not arguing how it is supposed to work, just forewarning those to be ready when it doesn't work like the manual says.. (Everyone hates phone calls about email being down).
If you want to be certain, only a true load balancer will fit the bill.
Oh, and another PS.. IF you are going to do round robin, suggest you make two (2) MX records, and put two IPs in both, and then equal weight the two MX's.
Keeps a more even load, given those that only prefer the first MX returned, and those that prefer the last (spammers)
Le 19/07/2023 à 23:03, Michael Peddemors a écrit :
In theory, that is how it is SUPPOSED to work, in practice (and we have lots of history where customers ran into this problem when one went down), I believe that it was Outlook that didn't try an alternative IP address for a 20 min internal cache for instance, before a requery of the DNS was done, at which time it again would choose which IP to connect to. As well, SOME modems would get the two results, and return only one to the client. And lots of libraries we see, do the DNS query, get two IP results, but then only use the first one returned, etc..
The windows cache is supposed (and is confirmed on my side ) to work the same as other DNS cache: It will cache all the A records. Outlook being a good IMAP client is another story :-)
Not arguing how it is supposed to work, just forewarning those to be ready when it doesn't work like the manual says.. (Everyone hates phone calls about email being down).
If you want to be certain, only a true load balancer will fit the bill.
Oh, and another PS.. IF you are going to do round robin, suggest you make two (2) MX records, and put two IPs in both, and then equal weight the two MX's.
That is exactly what should not be done. Never put more than one IPv4 or IPv6 behind a FQDN pointed by a MX. It will kill the proper HA algorithm build in the SMTP/MX protocol. You will introduce some unnecessary delivery delays/retry backoff in case of one server failure. Put as many MX records has you have SMTP gateways. Or group some gateways behind some LB VIP if you have/need a high count of gateways.
Keeps a more even load, given those that only prefer the first MX returned, and those that prefer the last (spammers)
There is no ordering, round robin apply here too.
MX are for MTA to MTA communications. Talking about MUA/clients, they don't care/use MX.
Emmanuel.
On 20/07/2023 05:55, Gerald Galster wrote:
A dns query for imap.web.de address records (IN A) returns two ip addresses.
And I'm betting each IP is a hardware load balancer with crap load of servers behind each :)
If you are not the intended recipient, please notify the sender then delete all copies of this message including attachments immediately. Confidentiality, copyright, and legal privilege are not waived or lost by reason of the mistaken delivery of this message.
I know this might have already been answered
Can some one give a link to the paid site that does what dovecot project does now ++++
more then happy to keep the lights on !
pls advise link ?
Happy Wednesday !!! Thanks - paul
Paul Kudla
Scom.ca Internet Services http://www.scom.ca 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3
Toronto 416.642.7266 Main 1.866.411.7266 Fax 1.888.892.7266 Email paul@scom.ca
On 7/26/2023 5:12 AM, Noel Butler via dovecot wrote:
On 20/07/2023 05:55, Gerald Galster wrote:
A dns query for imap.web.de address records (IN A) returns two ip addresses.
And I'm betting each IP is a hardware load balancer with crap load of servers behind each :) --
Regards, Noel Butler
This Email, including attachments, may contain legally privileged information, therefore at all times remains confidential and subject to copyright protected under international law. You may not disseminate this message without the authors express written authority to do so.
If you are not the intended recipient, please notify the sender then delete all copies of this message including attachments immediately. Confidentiality, copyright, and legal privilege are not waived or lost by reason of the mistaken delivery of this message.-- This message has been scanned for viruses and dangerous content by *MailScanner* http://www.mailscanner.info/, and is believed to be clean.
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
On Jul 26, 2023, at 05:01, Paul Kudla paul@scom.ca wrote:
I know this might have already been answered
Can some one give a link to the paid site that does what dovecot project does now ++++
more then happy to keep the lights on !
pls advise link ?
I believe the URL is https://www.open-xchange.com https://www.open-xchange.com/
-- Doug
A dns query for imap.web.de address records (IN A) returns two ip addresses.
And I'm betting each IP is a hardware load balancer with crap load of servers behind each :)
I am converting a bit to containers and there are so many applications that are not able to properly resolve and handle errors. Once they have an ip they stop doing anything. That it is nicely setup on the server side means nothing. If you do this for outgoing email, lots of email clients fail switching to the 2nd ip.
On 26/07/2023 22:43, Marc wrote:
A dns query for imap.web.de address records (IN A) returns two ip addresses.
And I'm betting each IP is a hardware load balancer with crap load of servers behind each :)
I am converting a bit to containers and there are so many applications that are not able to properly resolve and handle errors. Once they have an ip they stop doing anything. That it is nicely setup on the server side means nothing. If you do this for outgoing email, lots of email clients fail switching to the 2nd ip.
Interesting, if server end is using L4 DSR they shouldnt tell the difference, but I can't comment on containers or VM's, as we do not play in the virtual world, these things need as much raw power as possible.
If you are not the intended recipient, please notify the sender then delete all copies of this message including attachments immediately. Confidentiality, copyright, and legal privilege are not waived or lost by reason of the mistaken delivery of this message.
(I'm very late to this party so my comments may have been said at some point)
On 20/07/2023 03:53, Michael Peddemors wrote:
Real world is a bit different.. DNS Caching.. While DNS Round Robin is good enough to distribute loads, it isnt' a very good method for failover, even with a very short TTL. Many home
No, history showed DNS round robin proved abysmal, it led to real load balancing hardware and software being born.
These changes don't affect us, we've never used director, hardware load balancers FTW, and no replicator, nightly snapshots and multiple levels of raid on a NAS backend, but I do see smaller installs where it may be preferable to buying a $200k NAS :)
However, for those with shoe string budgets, for load balancing, this can be overcome by a software version, there are some for no cost if you have a spare machine, you might even pick up a real cheap old hardware balancer on likes of ebay.
but it more of a last line failover, and during the time it takes for DNS to retry, and find another active node, an AWFUL lot of disgruntled customers will be calling ;)
Ahhh reminds me on the very early 90's :)
If you are not the intended recipient, please notify the sender then delete all copies of this message including attachments immediately. Confidentiality, copyright, and legal privilege are not waived or lost by reason of the mistaken delivery of this message.
On 07/18/2023 9:00 AM MDT Gerald Galster list+dovecot@gcore.biz wrote:
While I understand it takes effort to maintain the replication plugin, this is especially problematic for small active/active high-availability deployments.
To clarify: replication absolutely does not provide "active/active". Replication was meant to copy data to a standby server, but you can't have concurrent mailbox access. This is why directors existed.
I guess there are lots of servers that use replication for just 50 or 100 mailboxes. Cloudstorage (like S3) would be overkill for these.
Do you provide dovecot pro subscriptions for such small deployments?
A 50-100 mailbox user server will run Dovecot CE just fine. Pro would be overkill.
All current Dovecot development assumes that storage is decoupled from the system. Shared (as in network available) storage is what you need if you want high availability, whether in Pro or CE.
michael
On 07/19/2023 12:51 PM MDT Marc marc@f1-outsourcing.eu wrote:
A 50-100 mailbox user server will run Dovecot CE just fine. Pro would be overkill.
What is overkill? I always thought it had a bit more features and support.
For Pro 2.3, you need (at minimum) 7 Dovecot nodes + HA authentication + HA storage + (minimum) 3 Cassandra nodes if using object storage. This is per site; most of our customers require data center redundancy as well, so multiply as needed. And this is only email retrieval; this doesn't even begin to touch upon email transfer.
Email high availability isn't cheap. (I would argue that if you truly need this sort of carrier-grade HA for 50 users, it makes much more sense to use email as-a-service than trying to do it yourself these days. Unless you have very specific reasons and a ton of cash.)
michael
A 50-100 mailbox user server will run Dovecot CE just fine. Pro would be overkill.
What is overkill? I always thought it had a bit more features and support.
For Pro 2.3, you need (at minimum) 7 Dovecot nodes + HA authentication + HA storage + (minimum) 3 Cassandra nodes if using object storage. This is per site; most of our customers require data center redundancy as well, so multiply as needed. And this is only email retrieval; this doesn't even begin to touch upon email transfer.
Email high availability isn't cheap. (I would argue that if you truly need this sort of carrier-grade HA for 50 users, it makes much more sense to use email as-a-service than trying to do it yourself these days. Unless you have very specific reasons and a ton of cash.)
High availability currently is cheap with a small two server setup: You need 3 servers or virtual machines: dovecot (and maybe postfix) running on two of them and mysql galera on all three. This provides very affordable active/active geo-redundancy.
No offence, it's just a pity to see that feature disappering.
Best regards, Gerald
Le 19/07/2023 à 22:08, Gerald Galster a écrit :
A 50-100 mailbox user server will run Dovecot CE just fine. Pro would be overkill.
What is overkill? I always thought it had a bit more features and support.
For Pro 2.3, you need (at minimum) 7 Dovecot nodes + HA authentication + HA storage + (minimum) 3 Cassandra nodes if using object storage. This is per site; most of our customers require data center redundancy as well, so multiply as needed. And this is only email retrieval; this doesn't even begin to touch upon email transfer.
Email high availability isn't cheap. (I would argue that if you truly need this sort of carrier-grade HA for 50 users, it makes much more sense to use email as-a-service than trying to do it yourself these days. Unless you have very specific reasons and a ton of cash.)
High availability currently is cheap with a small two server setup: You need 3 servers or virtual machines: dovecot (and maybe postfix) running on two of them and mysql galera on all three. This provides very affordable active/active geo-redundancy.
No offence, it's just a pity to see that feature disappering.
Could not say more : all my past setups were like this. With postgres or openldap instead of mysql. It was a feature I waited for years and heavy used for last ten years. Big complicated shared storage was NEVER needed or suitable alone in all my encountered scenario. And in case of active/active Geo, even with shared storage, the geo replication was entrusted to dovecot and not to the underlying storage for efficiency and consistency purpose.
Emmanuel.
On 2023-07-19 4:08 pm, Gerald Galster wrote:
A 50-100 mailbox user server will run Dovecot CE just fine. Pro would be overkill. What is overkill? I always thought it had a bit more features and support.
For Pro 2.3, you need (at minimum) 7 Dovecot nodes + HA authentication
- HA storage + (minimum) 3 Cassandra nodes if using object storage. This
is per site; most of our customers require data center redundancy as well, so multiply as needed. And this is only email retrieval; this doesn't even begin to touch upon email transfer. Email high availability isn't cheap. (I would argue that if you truly need this sort of carrier-grade HA for 50 users, it makes much more sense to use email as-a-service than trying to do it yourself these days. Unless you have very specific reasons and a ton of cash.)
High availability currently is cheap with a small two server setup: You need 3 servers or virtual machines: dovecot (and maybe postfix) running on two of them and mysql galera on all three. This provides very affordable active/active geo-redundancy.
No offence, it's just a pity to see that feature disappering.
That's exactly how my own 3-node personal setup works. I shove all I can into mariadb with galera (dovecot auth, spamassassin, etc) across the 3 nodes. Dovecot replication keeps the 2 dovecot instances in sync, the 3rd node is the quorum node for galera.
This is is on 3 cheap VPS' in 3 locations around the US. Mesh VPN between them for the encrypted connectivity. It works, and it works well.
And now replication is going away ? A perfectly-well working feature is being removed ?? It's not as if it's a problematic one, nor would it interfere with anything if it remained ...
I only see a couple of routes forward, at least for me.
* Stay on the last dovecot release that supports replication.
* Switch away from dovecot and cobble something else together.
* Move to gmail <shudder>
The removal of replication feels very arbitrary.
On 2023-07-20 5:31 pm, deano-dovecot@areyes.com wrote:
On 2023-07-19 4:08 pm, Gerald Galster wrote: A 50-100 mailbox user server will run Dovecot CE just fine. Pro would be overkill. What is overkill? I always thought it had a bit more features and support.
For Pro 2.3, you need (at minimum) 7 Dovecot nodes + HA authentication
- HA storage + (minimum) 3 Cassandra nodes if using object storage. This
is per site; most of our customers require data center redundancy as well, so multiply as needed. And this is only email retrieval; this doesn't even begin to touch upon email transfer. Email high availability isn't cheap. (I would argue that if you truly need this sort of carrier-grade HA for 50 users, it makes much more sense to use email as-a-service than trying to do it yourself these days. Unless you have very specific reasons and a ton of cash.)
High availability currently is cheap with a small two server setup: You need 3 servers or virtual machines: dovecot (and maybe postfix) running on two of them and mysql galera on all three. This provides very affordable active/active geo-redundancy.
No offence, it's just a pity to see that feature disappering.
That's exactly how my own 3-node personal setup works. I shove all I can into mariadb with galera (dovecot auth, spamassassin, etc) across the 3 nodes. Dovecot replication keeps the 2 dovecot instances in sync, the 3rd node is the quorum node for galera.
This is is on 3 cheap VPS' in 3 locations around the US. Mesh VPN between them for the encrypted connectivity. It works, and it works well.
And now replication is going away ? A perfectly-well working feature is being removed ?? It's not as if it's a problematic one, nor would it interfere with anything if it remained ...
I only see a couple of routes forward, at least for me.
* Stay on the last dovecot release that supports replication.
* Switch away from dovecot and cobble something else together.
* Move to gmail <shudder>
The removal of replication feels very arbitrary.
At what version is replication being removed ? 2.4 I think ? Or, perhaps the question is which is the last version that WILL still have replication ?
For now I'll be going with option 1 above, staying with the last version to support it. Sad.
On 2023-07-20 5:31 pm, deano-dovecot@areyes.com wrote: On 2023-07-19 4:08 pm, Gerald Galster wrote: A 50-100 mailbox user server will run Dovecot CE just fine. Pro would be overkill. What is overkill? I always thought it had a bit more features and support. For Pro 2.3, you need (at minimum) 7 Dovecot nodes + HA authentication + HA storage + (minimum) 3 Cassandra nodes if using object storage. This is per site; most of our customers require data center redundancy as well, so multiply as needed. And this is only email retrieval; this doesn't even begin to touch upon email transfer. Email high availability isn't cheap. (I would argue that if you truly need this sort of carrier-grade HA for 50 users, it makes much more sense to use email as-a-service than trying to do it yourself these days. Unless you have very specific reasons and a ton of cash.) High availability currently is cheap with a small two server setup: You need 3 servers or virtual machines: dovecot (and maybe postfix) running on two of them and mysql galera on all three. This provides very affordable active/active geo-redundancy.
No offence, it's just a pity to see that feature
disappering.
That's exactly how my own 3-node personal setup works. I shove all I
can into mariadb with galera (dovecot auth, spamassassin, etc) across
the 3 nodes. Dovecot replication keeps the 2 dovecot instances in
sync, the 3rd node is the quorum node for galera.
This is is on 3 cheap VPS' in 3 locations around the US. Mesh VPN
between them for the encrypted connectivity. It works, and it works
well.
And now replication is going away ? A perfectly-well working feature
is being removed ?? It's not as if it's a problematic one, nor would
it interfere with anything if it remained ...
I only see a couple of routes forward, at least for me.
* Stay on the last dovecot release that supports replication.
* Switch away from dovecot and cobble something else together.
* Move to gmail <shudder>
The removal of replication feels very arbitrary.
At what version is replication being removed ? 2.4 I think ? Or, perhaps the question is which is the last version that WILL still have replication ? For now I'll be going with option 1 above, staying with the last version to support it. Sad.
Does anyone already have a dovecot (CE with Maildir) setup running using shared storage (i.ex. GlusterFS) underneath?
This will be my current „migration plan“ to dovecot nmot supporting replication anymore:
2 x Loadblancers (accross two sites) with keepalived and haproxy 3x GlusterFS nodes 3x Galera Cluster mariadb nodes 2x dovecot IMAP with Postfix SMTP (wihout director there’s no need anynmore to use dovecot SMTP implementation)
The Loadbalancer-, Galera- and Gluster modes are shared wit the existing web service infrastructure, so not dedicated to dovecot.
So basically, this way I will not have to deploy a single node more without director/repication.
If the HA storage and database clusters will be used for mail dedicated, this makes 11 nodes for a fully HA cluster accross two sites. With virtualization infrastructure that should get very expensive.
Or am I missing something?
Steven
On Wed, 19 Jul 2023 at 20:40, Michael Slusarz via dovecot dovecot@dovecot.org wrote:
On 07/18/2023 9:00 AM MDT Gerald Galster list+dovecot@gcore.biz wrote:
While I understand it takes effort to maintain the replication plugin, this is especially problematic for small active/active high-availability deployments.
To clarify: replication absolutely does not provide "active/active". Replication was meant to copy data to a standby server, but you can't have concurrent mailbox access. This is why directors existed.
I guess there are lots of servers that use replication for just 50 or 100 mailboxes. Cloudstorage (like S3) would be overkill for these.
Do you provide dovecot pro subscriptions for such small deployments?
A 50-100 mailbox user server will run Dovecot CE just fine. Pro would be overkill.
All current Dovecot development assumes that storage is decoupled from the system. Shared (as in network available) storage is what you need if you want high availability, whether in Pro or CE.
That seems like an extraordinary callous and short-sighted decision, but nevertheless does give some technical weight to your earlier statement.
Regards
Simon
While I understand it takes effort to maintain the replication plugin, this is especially problematic for small active/active high-availability deployments.
To clarify: replication absolutely does not provide "active/active". Replication was meant to copy data to a standby server, but you can't have concurrent mailbox access. This is why directors existed.
I have to disagree, active/active with two distinct servers is a special case and two-way replication has been working reliably in production for years. Here's why: those two imap servers are separate instances with local storage, local quota and local pruning. There is no shared medium access like nfs that could lead to data corruption and hence no directors needed. Moreover the dsync manpage states "doveadm-sync - Dovecot's two-way mailbox synchronization utility". It's more like IMAP-clients copying mails from one server to another where the most current state wins.
I guess there are lots of servers that use replication for just 50 or 100 mailboxes. Cloudstorage (like S3) would be overkill for these.
Do you provide dovecot pro subscriptions for such small deployments?
A 50-100 mailbox user server will run Dovecot CE just fine. Pro would be overkill.
All current Dovecot development assumes that storage is decoupled from the system. Shared (as in network available) storage is what you need if you want high availability, whether in Pro or CE.
Thanks for clarification. So even if mdbox is still available and replication (backup) would work with dsync on the command line, there is no signaling layer to auto-trigger replication because storage is decoupled and this functionality is not needed anymore.
Best regards, Gerald
Michael Slusarz via dovecot dovecot@dovecot.org wrote:
On 07/18/2023 9:00 AM MDT Gerald Galster list+dovecot@gcore.biz wrote:
While I understand it takes effort to maintain the replication plugin, this is especially problematic for small active/active high-availability deployments.
To clarify: replication absolutely does not provide "active/active". Replication was meant to copy data to a standby server, but you can't have concurrent mailbox access. This is why directors existed.
That simply isn't true, and I am baffled that you don't know that replication works with a two server active/active setup for years now! Two separate instances (active/active) on two different continents are a completely reliable failover scenario for years now.
Very irritating to read such a statement.
Regards, Michael
That simply isn't true, and I am baffled that you don't know that replication works with a two server active/active setup for years now! Two separate instances (active/active) on two different continents are a completely reliable failover scenario for years now.
Maybe it works like this in your environment? Maybe if the load increases you run into trouble? The director is making sure you never utilize an active/active situation from the perspective of user access. The user is only accessing one server. It is quite a different story when the same user starts writing to both servers at the same time.
Marc Marc@f1-outsourcing.eu wrote:
That simply isn't true, and I am baffled that you don't know that replication works with a two server active/active setup for years now! Two separate instances (active/active) on two different continents are a completely reliable failover scenario for years now.
Maybe it works like this in your environment? Maybe if the load increases you run into trouble? The director is making sure you never utilize an active/active situation from the perspective of user access. The user is only accessing one server. It is quite a different story when the same user starts writing to both servers at the same time.
If I do rapidly inject tens of thousands of mails locally on both servers SIMULTANEOUSLY for the very same user I never ever loose one of it. Tested numerous times before rolling it out. In the very beginning of Timo's publishing replication it had had flaws, but other users and myself tested it while Timo enhances his code (and IIRC once even rewrote it from scratch). For years now it runs as expected and documented.
As mentioned in this thread this ist true for small setups.
Regards, Michael
On 07/19/2023 2:54 PM MDT Michael Grimm via dovecot dovecot@dovecot.org wrote:
Michael Slusarz via dovecot dovecot@dovecot.org wrote:
On 07/18/2023 9:00 AM MDT Gerald Galster list+dovecot@gcore.biz wrote:
While I understand it takes effort to maintain the replication plugin, this is especially problematic for small active/active high-availability deployments.
To clarify: replication absolutely does not provide "active/active". Replication was meant to copy data to a standby server, but you can't have concurrent mailbox access. This is why directors existed.
That simply isn't true, and I am baffled that you don't know that replication works with a two server active/active setup for years now! Two separate instances (active/active) on two different continents are a completely reliable failover scenario for years now.
Very irritating to read such a statement.
You may be irritated, but my statement is accurate.
Eventually consistent replication is *NOT* active/active. active/active has a very specific meaning (and is not the same as master/master).
Quotas and shared mailboxes are two troublesome concepts with replicator. Inconsistent mailbox views are a call center driver. Neither of these would be an issue in a true active/active setup. Forcing a user to a single node at any given time will prevent some (but not all) issues.
Replicator's scaling issue can't really be worked around, and was a main driver why Dovecot Pro was developed (example: one Pro customer migrating from CE/replicators saw a 90% decrease in server count).
Your positive individual experience does not change the inherent characteristics, and limitations, of the design. If your setup works for you, in your particular circumstances, great! But it doesn't work for everyone. There is a reason Dovecot development moved on from replicator based architecture 10+ years ago.
michael
Le 20/07/2023 à 00:57, Michael Slusarz via dovecot a écrit :
On 07/19/2023 2:54 PM MDT Michael Grimm via dovecotdovecot@dovecot.org wrote:
Michael Slusarz via dovecotdovecot@dovecot.org wrote:
On 07/18/2023 9:00 AM MDT Gerald Galsterlist+dovecot@gcore.biz wrote: While I understand it takes effort to maintain the replication plugin, this is especially problematic for small active/active high-availability deployments.
To clarify: replication absolutely does not provide "active/active". Replication was meant to copy data to a standby server, but you can't have concurrent mailbox access. This is why directors existed.
That simply isn't true, and I am baffled that you don't know that replication works with a two server active/active setup for years now! Two separate instances (active/active) on two different continents are a completely reliable failover scenario for years now.
Very irritating to read such a statement.
You may be irritated, but my statement is accurate.
Eventually consistent replication is *NOT* active/active. active/active has a very specific meaning (and is not the same as master/master).
Quotas and shared mailboxes are two troublesome concepts with replicator. Inconsistent mailbox views are a call center driver. Neither of these would be an issue in a true active/active setup. Forcing a user to a single node at any given time will prevent some (but not all) issues.
Replicator's scaling issue can't really be worked around, and was a main driver why Dovecot Pro was developed (example: one Pro customer migrating from CE/replicators saw a 90% decrease in server count).
Your positive individual experience does not change the inherent characteristics, and limitations, of the design. If your setup works for you, in your particular circumstances, great! But it doesn't work for everyone. There is a reason Dovecot development moved on from replicator based architecture 10+ years ago.
Your focusing on a very specific scenario, and throw the baby out with the bathwater. Multiple shared client access on the same mailbox is a very specific scenario avoidable 99.99% of time as "true" active/active is never a real need. No offense too but it seems that you are biased by your business/customers needs that real world main usage. It is a little bit sad. We are slowly but surely FORCED to use external/cloud services as Open (and even closed) source tools are no longer use-able on own/private infrastructures or need nuclear plan like budget and implementation time/ressources (Yes it is a bit exaggerated ;-) )
Regards, Emmanuel.
We are slowly but surely FORCED to use external/cloud services as Open (and even closed) source tools are no longer use-able on own/private infrastructures or need nuclear plan like budget and implementation time/ressources (Yes it is a bit exaggerated ;-) )
There are still people out there that do not want to force everyone in the US cloud, they work on development strategies to support and facilitate micro organisations. Fact will always be that micro organisations have much better client support/relationship and people are just sick of the monopolistic cloud providers. Which at that point do not have even a financial advantage any more. Software designs like eg dovecot will be phased out in the near future and replaced by modern applications, this will happen sooner than you think. I sincerely hope I can have a significant contribution in such transition.
In order to prepare for the upcoming release:
Is there an estimated release date when replication will be removed?
What is the best way (reconfigure, turn off auto replication, etc.) to migrate to an environment to test and run using ‘doveadm sync -d’ in a scheduled job (e.g., cron)?
Sent from my iPad
On Jul 18, 2023, at 9:42 AM, Aki Tuomi via dovecot dovecot@dovecot.org wrote:
On 18/07/2023 15:19 EEST info@joergschulz.de wrote:
Just to understand that correctly: I could setup a (cron) based process for doveadm sync, but no longer a setup like plugin { mail_replica = tcp:$IMAP_REPLICA_SERVER:$IMAP_REPLICA_PORT } where the cron would lead to some delay and would have to check for concurrent jobs?
You can also have that too.
doveadm sync -d
makes it use mail_replica setting.
Aki _______________________________________________ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
Respectfully, I would like to ask: please do not remove replication, please rethink this.
Currently, replication is my life saver. I run two postfix/dovecot combos (on different operating systems), with dovecot synchronising via replication. Both are behind a HAProxy running on the router (OPNsense), one as active, one as backup.
If one of the two fails, the other takes over, and when it comes up again everything works fine and is up to date. I have had these kinds of system failures (very hard to find and turned to be hardware related) and it was the replication that made me survive the issues (even when I was far away from my systems). Mail for my small group of users (about 8) never went down, no mail message was ever lost, no manual interventions to sync were ever needed.
If I want to create the same level of availability without replication, I need those two dovecots to use shared (NFS cluster) storage. But then, I have another single point of failure (NFS storage) again. So, I need two separate NFS machines that synchronise, Apart from the nightmare of making NFS secure, it means that I need to double my hardware (from two systems to four) to be protected against hardware failure (which is my goal).
The replication service is the perfect small scale solution. Together with HAProxy, it enables HA in the most simple and effective way. Going the 'NFS cluster' route is not feasible for me, so if replication is removed and I am forced to upgrade, I will lose HA.
So please, take small scale users like me into account.
Gerben Wierda (LinkedIn https://www.linkedin.com/in/gerbenwierda, Mastodon https://newsie.social/@gctwnl) R&A IT Strategy https://ea.rna.nl/ (main site) Book: Chess and the Art of Enterprise Architecture https://ea.rna.nl/the-book/ Book: Mastering ArchiMate https://ea.rna.nl/the-book-edition-iii/ YouTube Channel http://www.youtube.com/@GerbenWierda
On 16 Jul 2023, at 18:54, Aki Tuomi via dovecot dovecot@dovecot.org wrote:
Hi!
Yes, director and replicator are removed, and won't be available for pro users either.
For NFS setups (or similar shared setups), we have documented a way to use Lua to run a director-like setup, see
https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/
Regards to replication, doveadm sync is not being removed. So you can still run doveadm sync on your system to have a primary / backup setup.
Aki
On 16/07/2023 18:34 EEST William Edwards via dovecot dovecot@dovecot.org wrote:
Top posting because nothing specific to reply to, sorry. Not exactly sure, but there’s another thread about the removal of Director in favour of Dovecot Pro on 3.x. Perhaps this change is related.
William Edwards
Op 16 jul. 2023 om 16:33 heeft Daniele danix@kernel-panic.it het volgende geschreven:
Hello,
Just like Vladimir, I'm a bit concerned about this change, and I'd really appreciate if someone could let us know if the replication feature (that works so well!) will be replaced or removed; and, in case of removal, what would be recommended replacement? Thanks in advance and best regards, Daniele
On 09-Jul-23 9:36 PM, Vladimir Mishonov via dovecot wrote: Hello everyone.
Just saw this commit in the official Github repo:
https://github.com/dovecot/core/commit/4c04e4c30fd4817a8b0e11d04d9681173f696...
Looking at the commit details, it appears that it completely removes the replication feature. I'm a bit perplexed by this change and am not sure what might be the justification for it. Personally, I find replication to be very useful, as it allows me to maintain a synchronized mirror of all of my mailboxes on my home server, for use as backup in case the primary server goes down for some reason.
Perhaps there's some sort of replacement being planned for this feature? Or maybe the relevant code is simply going to be refactored to a plugin or external program, and there's nothing to worry about at all?
In any case, I'd greatly appreciate if one of the developers could comment on this change.
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
Respectfully, I would like to ask: please do not remove replication, please rethink this.
Currently, replication is my life saver. I run two postfix/dovecot combos (on different operating systems), with dovecot synchronising via replication. Both are behind a HAProxy running on the router (OPNsense), one as active, one as backup.
If one of the two fails, the other takes over, and when it comes up again everything works fine and is up to date. I have had these kinds of system failures (very hard to find and turned to be hardware related) and it was the replication that made me survive the issues (even when I was far away from my systems). Mail for my small group of users (about 8) never went down, no mail message was ever lost, no manual interventions to sync were ever needed.
If I want to create the same level of availability without replication, I need those two dovecots to use shared (NFS cluster) storage. But then, I have another single point of failure (NFS storage) again. So, I need two separate NFS machines that synchronise, Apart from the nightmare of making NFS secure, it means that I need to double my hardware (from two systems to four) to be protected against hardware failure (which is my goal).
The replication service is the perfect small scale solution. Together with HAProxy, it enables HA in the most simple and effective way. Going the 'NFS cluster' route is not feasible for me, so if replication is removed and I am forced to upgrade, I will lose HA.
So please, take small scale users like me into account.
Gerben Wierda (LinkedIn, Mastodon) R&A_IT_Strategy (main site) Book: Chess_and_the_Art_of_Enterprise Architecture Book: Mastering_ArchiMate YouTube_Channel
On 16 Jul 2023, at 18:54, Aki Tuomi via dovecot <dovecot@dovecot.org>
wrote:
Hi!
Yes, director and replicator are removed, and won't be available for
pro users either.
For NFS setups (or similar shared setups), we have documented a way
to use Lua to run a director-like setup, see
https://doc.dovecot.org/3.0/configuration_manual/howto/
director_with_lua/
Regards to replication, doveadm sync is not being removed. So you can
still run doveadm sync on your system to have a primary / backup
setup.
Aki
On 16/07/2023 18:34 EEST William Edwards via dovecot
<dovecot@dovecot.org> wrote:
Top posting because nothing specific to reply to, sorry.
Not exactly sure, but there’s another thread about the
removal of Director in favour of Dovecot Pro on 3.x.
Perhaps this change is related.
William Edwards
Op 16 jul. 2023 om 16:33 heeft Daniele
<danix@kernel-panic.it> het volgende geschreven:
Hello,
Just like Vladimir, I'm a bit concerned about
this change, and I'd really appreciate if someone
could let us know if the replication feature
(that works so well!) will be replaced or
removed; and, in case of removal, what would be
recommended replacement?
Thanks in advance and best regards,
Daniele
On 09-Jul-23 9:36 PM, Vladimir Mishonov
via dovecot wrote:
Hello everyone.
Just saw this commit in the official
Github repo:
https://github.com/dovecot/core/commit/
4c04e4c30fd4817a8b0e11d04d9681173f696f41#diff-
5f643d8b0d1eea65d0f3c749d14d42b25a9d60f0f149bface862f5ff348412c8
Looking at the commit details, it
appears that it completely removes the
replication feature. I'm a bit
perplexed by this change and am not
sure what might be the justification
for it. Personally, I find replication
to be very useful, as it allows me to
maintain a synchronized mirror of all
of my mailboxes on my home server, for
use as backup in case the primary
server goes down for some reason.
Perhaps there's some sort of
replacement being planned for this
feature? Or maybe the relevant code is
simply going to be refactored to a
plugin or external program, and there's
nothing to worry about at all?
In any case, I'd greatly appreciate if
one of the developers could comment on
this change.
_______________________________________________
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-
leave@dovecot.org
_______________________________________________
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-leave@dovecot.org
_______________________________________________
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-leave@dovecot.org
Although I’m also a very happy dovecot replication user, I don’t think this decision will be reverted, sadly.
However, despite of messing with NFS, I will try setting up a three-node GlusterFS Cluster to give redundant storage to dovecote as mail store and hope it performs well enough… Has anyone else such a setup (or alternatively with Ceph) in production?
Steven
On 2024-01-24 16:35, Steven Varco wrote:
Although I’m also a very happy dovecot replication user, I don’t think this decision will be reverted, sadly.
However, despite of messing with NFS, I will try setting up a three-node GlusterFS Cluster to give redundant storage to dovecote as mail store and hope it performs well enough… Has anyone else such a setup (or alternatively with Ceph) in production?
Steven
Seen some Gluster backends blow up spectacularly..
Always say.. keep it simple. Every thought of NFS backend, and let the NetApp do the job? Scales well, and haven't seen one go down in production yet.. knock on wood.. and the costs have really dropped.
How many servers do you have? How many active clients do you have?
Respectfully, I would like to ask: please do not remove replication, please rethink this.
Currently, replication is my life saver. I run two postfix/dovecot combos (on different operating systems), with dovecot synchronising via replication. Both are behind a HAProxy running on the router (OPNsense), one as active, one as backup.
If one of the two fails, the other takes over, and when it comes up again everything works fine and is up to date. I have had these kinds of system failures (very hard to find and turned to be hardware related) and it was the replication that made me survive the issues (even when I was far away from my systems). Mail for my small group of users (about 8) never went down, no mail message was ever lost, no manual interventions to sync were ever needed.
If I want to create the same level of availability without replication, I need those two dovecots to use shared (NFS cluster) storage. But then, I have another single point of failure (NFS storage) again. So, I need two separate NFS machines that synchronise, Apart from the nightmare of making NFS secure, it means that I need to double my hardware (from two systems to four) to be protected against hardware failure (which is my goal).
The replication service is the perfect small scale solution. Together with HAProxy, it enables HA in the most simple and effective way. Going the 'NFS cluster' route is not feasible for me, so if replication is removed and I am forced to upgrade, I will lose HA.
So please, take small scale users like me into account.
Gerben Wierda (LinkedIn https://www.linkedin.com/in/gerbenwierda, Mastodon https://newsie.social/@gctwnl) R&A IT Strategy https://ea.rna.nl/ (main site) Book: Chess and the Art of Enterprise Architecture https://ea.rna.nl/the- book/ Book: Mastering ArchiMate https://ea.rna.nl/the-book-edition-iii/ YouTube Channel http://www.youtube.com/@GerbenWierda
On 16 Jul 2023, at 18:54, Aki Tuomi via dovecot dovecot@dovecot.org
wrote:
Hi!
Yes, director and replicator are removed, and won't be available for pro
users either.
For NFS setups (or similar shared setups), we have documented a way to use
Lua to run a director-like setup, see
https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/
Regards to replication, doveadm sync is not being removed. So you can
still run doveadm sync on your system to have a primary / backup setup.
Aki
On 16/07/2023 18:34 EEST William Edwards via dovecot
dovecot@dovecot.org wrote:
Top posting because nothing specific to reply to, sorry. Not exactly
sure, but there’s another thread about the removal of Director in favour of Dovecot Pro on 3.x. Perhaps this change is related.
William Edwards
Op 16 jul. 2023 om 16:33 heeft Daniele danix@kernel-panic.it het
volgende geschreven:
Hello,
Just like Vladimir, I'm a bit concerned about this change, and I'd
really appreciate if someone could let us know if the replication feature (that works so well!) will be replaced or removed; and, in case of removal, what would be recommended replacement?
Thanks in advance and best regards, Daniele
On 09-Jul-23 9:36 PM, Vladimir Mishonov via dovecot wrote: Hello everyone.
Just saw this commit in the official Github repo:
https://github.com/dovecot/core/commit/4c04e4c30fd4817a8b0e11d04d9681173f696 f41#diff-5f643d8b0d1eea65d0f3c749d14d42b25a9d60f0f149bface862f5ff348412c8
Looking at the commit details, it appears that it completely removes
the replication feature. I'm a bit perplexed by this change and am not sure what might be the justification for it. Personally, I find replication to be very useful, as it allows me to maintain a synchronized mirror of all of my mailboxes on my home server, for use as backup in case the primary server goes down for some reason.
Perhaps there's some sort of replacement being planned for this
feature? Or maybe the relevant code is simply going to be refactored to a plugin or external program, and there's nothing to worry about at all?
In any case, I'd greatly appreciate if one of the developers could
comment on this change.
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
I have two postfix/(replicating)dovecot/rspamd/etc servers, 6-8 human users, on a total of 15-20 devices (iOS and macOS)
Yours,
Gerben Wierda (LinkedIn https://www.linkedin.com/in/gerbenwierda, Mastodon https://newsie.social/@gctwnl) R&A IT Strategy https://ea.rna.nl/ (main site) Book: Chess and the Art of Enterprise Architecture https://ea.rna.nl/the-book/ Book: Mastering ArchiMate https://ea.rna.nl/the-book-edition-iii/ YouTube Channel http://www.youtube.com/@GerbenWierda
On 25 Jan 2024, at 10:17, Marc Marc@f1-outsourcing.eu wrote:
How many servers do you have? How many active clients do you have?
Respectfully, I would like to ask: please do not remove replication, please rethink this.
Currently, replication is my life saver. I run two postfix/dovecot combos (on different operating systems), with dovecot synchronising via replication. Both are behind a HAProxy running on the router (OPNsense), one as active, one as backup.
If one of the two fails, the other takes over, and when it comes up again everything works fine and is up to date. I have had these kinds of system failures (very hard to find and turned to be hardware related) and it was the replication that made me survive the issues (even when I was far away from my systems). Mail for my small group of users (about 8) never went down, no mail message was ever lost, no manual interventions to sync were ever needed.
If I want to create the same level of availability without replication, I need those two dovecots to use shared (NFS cluster) storage. But then, I have another single point of failure (NFS storage) again. So, I need two separate NFS machines that synchronise, Apart from the nightmare of making NFS secure, it means that I need to double my hardware (from two systems to four) to be protected against hardware failure (which is my goal).
The replication service is the perfect small scale solution. Together with HAProxy, it enables HA in the most simple and effective way. Going the 'NFS cluster' route is not feasible for me, so if replication is removed and I am forced to upgrade, I will lose HA.
So please, take small scale users like me into account.
Gerben Wierda (LinkedIn https://www.linkedin.com/in/gerbenwierda, Mastodon https://newsie.social/@gctwnl) R&A IT Strategy https://ea.rna.nl/ (main site) Book: Chess and the Art of Enterprise Architecture https://ea.rna.nl/the- book/ Book: Mastering ArchiMate https://ea.rna.nl/the-book-edition-iii/ YouTube Channel http://www.youtube.com/@GerbenWierda
On 16 Jul 2023, at 18:54, Aki Tuomi via dovecot dovecot@dovecot.org
wrote:
Hi!
Yes, director and replicator are removed, and won't be available for pro
users either.
For NFS setups (or similar shared setups), we have documented a way to use
Lua to run a director-like setup, see
https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/
Regards to replication, doveadm sync is not being removed. So you can
still run doveadm sync on your system to have a primary / backup setup.
Aki
On 16/07/2023 18:34 EEST William Edwards via dovecot
dovecot@dovecot.org wrote:
Top posting because nothing specific to reply to, sorry. Not exactly
sure, but there’s another thread about the removal of Director in favour of Dovecot Pro on 3.x. Perhaps this change is related.
William Edwards
Op 16 jul. 2023 om 16:33 heeft Daniele danix@kernel-panic.it het
volgende geschreven:
Hello,
Just like Vladimir, I'm a bit concerned about this change, and I'd
really appreciate if someone could let us know if the replication feature (that works so well!) will be replaced or removed; and, in case of removal, what would be recommended replacement?
Thanks in advance and best regards, Daniele
On 09-Jul-23 9:36 PM, Vladimir Mishonov via dovecot wrote: Hello everyone.
Just saw this commit in the official Github repo:
https://github.com/dovecot/core/commit/4c04e4c30fd4817a8b0e11d04d9681173f696 f41#diff-5f643d8b0d1eea65d0f3c749d14d42b25a9d60f0f149bface862f5ff348412c8
Looking at the commit details, it appears that it completely removes
the replication feature. I'm a bit perplexed by this change and am not sure what might be the justification for it. Personally, I find replication to be very useful, as it allows me to maintain a synchronized mirror of all of my mailboxes on my home server, for use as backup in case the primary server goes down for some reason.
Perhaps there's some sort of replacement being planned for this
feature? Or maybe the relevant code is simply going to be refactored to a plugin or external program, and there's nothing to worry about at all?
In any case, I'd greatly appreciate if one of the developers could
comment on this change.
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
I have two postfix/(replicating)dovecot/rspamd/etc servers, 6-8 human users, on a total of 15-20 devices (iOS and macOS)
Yours,
Gerben Wierda (LinkedIn, Mastodon) R&A_IT_Strategy (main site) Book: Chess_and_the_Art_of_Enterprise Architecture Book: Mastering_ArchiMate YouTube_Channel
On 25 Jan 2024, at 10:17, Marc <Marc@f1-outsourcing.eu> wrote:
How many servers do you have? How many active clients do you have?
Respectfully, I would like to ask: please do not remove
replication, please
rethink this.
Currently, replication is my life saver. I run two postfix/
dovecot combos
(on different operating systems), with dovecot
synchronising via
replication. Both are behind a HAProxy running on the
router (OPNsense), one
as active, one as backup.
If one of the two fails, the other takes over, and when it
comes up again
everything works fine and is up to date. I have had these
kinds of system
failures (very hard to find and turned to be hardware
related) and it was
the replication that made me survive the issues (even when
I was far away
from my systems). Mail for my small group of users (about
8) never went
down, no mail message was ever lost, no manual
interventions to sync were
ever needed.
If I want to create the same level of availability without
replication, I
need those two dovecots to use shared (NFS cluster)
storage. But then, I
have another single point of failure (NFS storage) again.
So, I need two
separate NFS machines that synchronise, Apart from the
nightmare of making
NFS secure, it means that I need to double my hardware
(from two systems to
four) to be protected against hardware failure (which is my
goal).
The replication service is the perfect small scale
solution. Together with
HAProxy, it enables HA in the most simple and effective
way. Going the 'NFS
cluster' route is not feasible for me, so if replication is
removed and I am
forced to upgrade, I will lose HA.
So please, take small scale users like me into account.
Gerben Wierda (LinkedIn <https://www.linkedin.com/in/
gerbenwierda>, Mastodon
<https://newsie.social/@gctwnl>)
R&A IT Strategy <https://ea.rna.nl/> (main site)
Book: Chess and the Art of Enterprise Architecture <https:/
/ea.rna.nl/the-
book/>
Book: Mastering ArchiMate <https://ea.rna.nl/the-book-
edition-iii/>
YouTube Channel <http://www.youtube.com/@GerbenWierda>
On 16 Jul 2023, at 18:54, Aki Tuomi via dovecot
<dovecot@dovecot.org>
wrote:
Hi!
Yes, director and replicator are removed, and
won't be available for pro
users either.
For NFS setups (or similar shared setups), we
have documented a way to use
Lua to run a director-like setup, see
https://doc.dovecot.org/3.0/configuration_manual/
howto/director_with_lua/
Regards to replication, doveadm sync is not being
removed. So you can
still run doveadm sync on your system to have a primary /
backup setup.
Aki
On 16/07/2023 18:34 EEST William
Edwards via dovecot
<dovecot@dovecot.org> wrote:
Top posting because nothing specific to
reply to, sorry. Not exactly
sure, but there’s another thread about the removal of
Director in favour of
Dovecot Pro on 3.x. Perhaps this change is related.
William Edwards
Op 16 jul. 2023 om 16:33
heeft Daniele <danix@kernel-
panic.it> het
volgende geschreven:
Hello,
Just like Vladimir, I'm a bit
concerned about this change,
and I'd
really appreciate if someone could let us know if the
replication feature
(that works so well!) will be replaced or removed; and, in
case of removal,
what would be recommended replacement?
Thanks in advance and best
regards,
Daniele
On 09-Jul-23 9:36
PM, Vladimir
Mishonov via
dovecot wrote:
Hello everyone.
Just saw this
commit in the
official Github
repo:
https://github.com/dovecot/core/commit/
4c04e4c30fd4817a8b0e11d04d9681173f696
f41#diff-
5f643d8b0d1eea65d0f3c749d14d42b25a9d60f0f149bface862f5ff348412c8
Looking at the
commit details, it
appears that it
completely removes
the replication feature. I'm a bit perplexed by this change
and am not sure
what might be the justification for it. Personally, I find
replication to be
very useful, as it allows me to maintain a synchronized
mirror of all of my
mailboxes on my home server, for use as backup in case the
primary server
goes down for some reason.
Perhaps there's
some sort of
replacement being
planned for this
feature? Or maybe the relevant code is simply going to be
refactored to a
plugin or external program, and there's nothing to worry
about at all?
In any case, I'd
greatly appreciate
if one of the
developers could
comment on this change.
_______________________________________________
dovecot mailing list -
- dovecot@dovecot.org
To unsubscribe send an email
to dovecot-leave@dovecot.org
_______________________________________________
dovecot mailing list -
- dovecot@dovecot.org
To unsubscribe send an email to
dovecot-leave@dovecot.org
_______________________________________________
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-
leave@dovecot.org
Ok, very very nice of you to make such redundant setup for these people. For such small amount I would just keep one server, and have everyone wait when it fails. ;)
Ok with such small user group you do not have any issues with performance with any hardware. I was creating recently a test cluster with ceph for a data centre move and changed on it the failure domain from host to osd. That worked pretty well with just 2 servers. So what you could do is run ceph on both servers with osd failure domain and use the shared rbd image as a 'replication' alternative. This way you have an instant backup available. You don't really need to configure 2 dovecot servers. You can move/start the vm from 1 host to the other. Actually if all disks fail on host, the vm's on that host keep running, connecting to the disks on the other host. Really nice.
I have two postfix/(replicating)dovecot/rspamd/etc servers, 6-8 human users, on a total of 15-20 devices (iOS and macOS)
Yours,
Gerben Wierda (LinkedIn https://www.linkedin.com/in/gerbenwierda , Mastodon https://newsie.social/@gctwnl ) R&A IT Strategy https://ea.rna.nl/ (main site) Book: Chess and the Art of Enterprise Architecture https://ea.rna.nl/the- book/ Book: Mastering ArchiMate https://ea.rna.nl/the-book-edition-iii/ YouTube Channel http://www.youtube.com/@GerbenWierda
On 25 Jan 2024, at 10:17, Marc Marc@f1-outsourcing.eu wrote:
How many servers do you have? How many active clients do you have?
Respectfully, I would like to ask: please do not removereplication, please rethink this.
Currently, replication is my life saver. I run twopostfix/dovecot combos (on different operating systems), with dovecot synchronising via replication. Both are behind a HAProxy running on the router (OPNsense), one as active, one as backup.
If one of the two fails, the other takes over, and when it comesup again everything works fine and is up to date. I have had these kinds of system failures (very hard to find and turned to be hardware related) and it was the replication that made me survive the issues (even when I was far away from my systems). Mail for my small group of users (about 8) never went down, no mail message was ever lost, no manual interventions to sync were ever needed.
If I want to create the same level of availability withoutreplication, I need those two dovecots to use shared (NFS cluster) storage. But then, I have another single point of failure (NFS storage) again. So, I need two separate NFS machines that synchronise, Apart from the nightmare of making NFS secure, it means that I need to double my hardware (from two systems to four) to be protected against hardware failure (which is my goal).
The replication service is the perfect small scale solution.Together with HAProxy, it enables HA in the most simple and effective way. Going the 'NFS cluster' route is not feasible for me, so if replication is removed and I am forced to upgrade, I will lose HA.
So please, take small scale users like me into account. Gerben Wierda (LinkedInhttps://www.linkedin.com/in/gerbenwierda, Mastodon https://newsie.social/@gctwnl) R&A IT Strategy https://ea.rna.nl/ (main site) Book: Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the- book/> Book: Mastering ArchiMate https://ea.rna.nl/the-book-edition- iii/ YouTube Channel http://www.youtube.com/@GerbenWierda
On 16 Jul 2023, at 18:54, Aki Tuomi via dovecotwrote: Hi! Yes, director and replicator are removed, and won't beavailable for pro
users either. For NFS setups (or similar shared setups), we havedocumented a way to use
Lua to run a director-like setup, seehttps://doc.dovecot.org/3.0/configuration_manual/howto/director_with_l ua/
Regards to replication, doveadm sync is not being removed.So you can
still run doveadm sync on your system to have a primary / backupsetup.
Aki On 16/07/2023 18:34 EEST William Edwards via dovecot <dovecot@dovecot.org> wrote: Top posting because nothing specific to reply to,sorry. Not exactly
sure, but there’s another thread about the removal of Director infavour of Dovecot Pro on 3.x. Perhaps this change is related.
William Edwards Op 16 jul. 2023 om 16:33 heeft Danielevolgende geschreven: Hello, Just like Vladimir, I'm a bit concerned aboutthis change, and I'd
really appreciate if someone could let us know if the replicationfeature (that works so well!) will be replaced or removed; and, in case of removal, what would be recommended replacement?
Thanks in advance and best regards, Daniele On 09-Jul-23 9:36 PM, Vladimir Mishonov viadovecot wrote: Hello everyone.
Just saw this commit in the official Githubrepo:
https://github.com/dovecot/core/commit/4c04e4c30fd4817a8b0e11d04d96811 73f696 f41#diff- 5f643d8b0d1eea65d0f3c749d14d42b25a9d60f0f149bface862f5ff348412c8
Looking at the commit details, it appears thatit completely removes
the replication feature. I'm a bit perplexed by this change andam not sure what might be the justification for it. Personally, I find replication to be very useful, as it allows me to maintain a synchronized mirror of all of my mailboxes on my home server, for use as backup in case the primary server goes down for some reason.
Perhaps there's some sort of replacement beingplanned for this
feature? Or maybe the relevant code is simply going to berefactored to a plugin or external program, and there's nothing to worry about at all?
In any case, I'd greatly appreciate if one ofthe developers could
comment on this change. _______________________________________________ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
_______________________________________________ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
_______________________________________________ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
I keep seeing this come up over and over. My understanding is it’s not getting removed, it’s just moving to the paid version of Dovecot. What is the cost for a small user license of dovecot that incudes replication anyway? Is the price that outrageous?
I keep seeing this come up over and over. My understanding is it’s not getting removed, it’s just moving to the paid version of Dovecot. What is the cost for a small user license of dovecot that incudes replication anyway? Is the price that outrageous?
On 25/01/2024 11:57 EET Michael Grant via dovecot dovecot@dovecot.org wrote:
I keep seeing this come up over and over. My understanding is it’s not getting removed, it’s just moving to the paid version of Dovecot. What is the cost for a small user license of dovecot that incudes replication anyway? Is the price that outrageous?
I keep seeing this come up over and over. My understanding is it’s not getting removed, it’s just moving to the paid version of Dovecot. What is the cost for a small user license of dovecot that incudes replication anyway? Is the price that outrageous?
Hi!
Your understanding is wrong though, we are not moving replicator or director to a paid version of Dovecot, they are fully removed.
Aki
+1
El 24/1/24 a les 23:33, Gerben Wierda ha escrit:
Respectfully, I would like to ask: please do not remove replication, please rethink this.
Currently, replication is my life saver. I run two postfix/dovecot combos (on different operating systems), with dovecot synchronising via replication. Both are behind a HAProxy running on the router (OPNsense), one as active, one as backup.
If one of the two fails, the other takes over, and when it comes up again everything works fine and is up to date. I have had these kinds of system failures (very hard to find and turned to be hardware related) and it was the replication that made me survive the issues (even when I was far away from my systems). Mail for my small group of users (about 8) never went down, no mail message was ever lost, no manual interventions to sync were ever needed.
If I want to create the same level of availability without replication, I need those two dovecots to use shared (NFS cluster) storage. But then, I have another single point of failure (NFS storage) again. So, I need two separate NFS machines that synchronise, Apart from the nightmare of making NFS secure, it means that I need to double my hardware (from two systems to four) to be protected against hardware failure (which is my goal).
The replication service is the perfect small scale solution. Together with HAProxy, it enables HA in the most simple and effective way. Going the 'NFS cluster' route is not feasible for me, so if replication is removed and I am forced to upgrade, I will lose HA.
So please, take small scale users like me into account.
Gerben Wierda (LinkedIn, Mastodon) R&A_IT_Strategy (main site) Book: Chess_and_the_Art_of_Enterprise Architecture Book: Mastering_ArchiMate YouTube_Channel
On 16 Jul 2023, at 18:54, Aki Tuomi via dovecot <dovecot@dovecot.org> wrote: Hi! Yes, director and replicator are removed, and won't be available for pro users either. For NFS setups (or similar shared setups), we have documented a way to use Lua to run a director-like setup, see https://doc.dovecot.org/3.0/configuration_manual/howto/ director_with_lua/ Regards to replication, doveadm sync is not being removed. So you can still run doveadm sync on your system to have a primary / backup setup. Aki On 16/07/2023 18:34 EEST William Edwards via dovecot <dovecot@dovecot.org> wrote: Top posting because nothing specific to reply to, sorry. Not exactly sure, but there’s another thread about the removal of Director in favour of Dovecot Pro on 3.x. Perhaps this change is related. William Edwards Op 16 jul. 2023 om 16:33 heeft Daniele <danix@kernel-panic.it> het volgende geschreven: Hello, Just like Vladimir, I'm a bit concerned about this change, and I'd really appreciate if someone could let us know if the replication feature (that works so well!) will be replaced or removed; and, in case of removal, what would be recommended replacement? Thanks in advance and best regards, Daniele On 09-Jul-23 9:36 PM, Vladimir Mishonov via dovecot wrote: Hello everyone. Just saw this commit in the official Github repo: https://github.com/dovecot/core/commit/ 4c04e4c30fd4817a8b0e11d04d9681173f696f41#diff- 5f643d8b0d1eea65d0f3c749d14d42b25a9d60f0f149bface862f5ff348412c8 Looking at the commit details, it appears that it completely removes the replication feature. I'm a bit perplexed by this change and am not sure what might be the justification for it. Personally, I find replication to be very useful, as it allows me to maintain a synchronized mirror of all of my mailboxes on my home server, for use as backup in case the primary server goes down for some reason. Perhaps there's some sort of replacement being planned for this feature? Or maybe the relevant code is simply going to be refactored to a plugin or external program, and there's nothing to worry about at all? In any case, I'd greatly appreciate if one of the developers could comment on this change. _______________________________________________ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot- leave@dovecot.org _______________________________________________ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org _______________________________________________ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
Currently, replication is my life saver.
depending on your use case, and how critical that is, there _are_ other options available.
not least of which is,
https://www.cyrusimap.org/imap/reference/admin/sop/replication.html
it's not "keeping replication in Dovecot". but it is well-supported FOSS replication.
ymmv.
and, as stated in this thread, the doveadm sync command will persist.
For a 8 user environment, it should be enough to do this via crontab, every some minutes.
Am 25.01.24 um 19:22 schrieb pgnd:
Currently, replication is my life saver.
depending on your use case, and how critical that is, there _are_ other options available.
not least of which is,
https://www.cyrusimap.org/imap/reference/admin/sop/replication.html
it's not "keeping replication in Dovecot". but it is well-supported FOSS replication.
ymmv. _______________________________________________ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
On Sun, Jul 09, 2023 at 10:36:35PM +0300, Vladimir Mishonov via dovecot wrote:
Looking at the commit details, it appears that it completely removes the replication feature.
I am not very upset with that news, since all I have been able to do with replicator was loosing mail.
However, I understand some had a better experience with it. I am curious if someone will fork dovecot and restore the beloved feature.
However, I understand some had a better experience with it. I am curious if someone will fork dovecot and restore the beloved feature.
With all the recent actions going on, clearly targeting in getting more paying „pro“ customers (and nothing else!) the dovecot project will walk on thin ice and there is a risk (or chance :) ) that it will get forked, all the good dovecot developers walking over to the fork and dovecot let down to a product no one uses anymore. It happend before, i.ex. nagios/icinga, CentOS/Alma-/Rocky Linux and other big, commonly used software projects.
I for my part will also try to stay as long as possible on dovecot 2.x with director and replicator, hoping for a fork which includes those features again in a distant future.
Steven
Op 17-7-2023 om 17:54 schreef Steven Varco:
However, I understand some had a better experience with it. I am curious if someone will fork dovecot and restore the beloved feature.
With all the recent actions going on, clearly targeting in getting more paying „pro“ customers (and nothing else!) the dovecot project will walk on thin ice and there is a risk (or chance :) ) that it will get forked, all the good dovecot developers walking over to the fork and dovecot let down to a product no one uses anymore. It happend before, i.ex. nagios/icinga, CentOS/Alma-/Rocky Linux and other big, commonly used software projects.
Sadly, Dovecot is a project with only a few prolific developers, all of which work for the company as far as I know. So, the scenario you describe is pretty unlikely. Maybe new community developers will stand up now, but that is something I think we can only welcome. And forks don't need to be adversarial like that.
Regards,
Stephan.
On 7/17/23 12:21, Stephan Bosch wrote:
Op 17-7-2023 om 17:54 schreef Steven Varco:
However, I understand some had a better experience with it. I am curious if someone will fork dovecot and restore the beloved feature.
With all the recent actions going on, clearly targeting in getting more paying „pro“ customers (and nothing else!) the dovecot project will walk on thin ice and there is a risk (or chance :) ) that it will get forked, all the good dovecot developers walking over to the fork and dovecot let down to a product no one uses anymore. It happend before, i.ex. nagios/icinga, CentOS/Alma-/Rocky Linux and other big, commonly used software projects.
Sadly, Dovecot is a project with only a few prolific developers, all of which work for the company as far as I know. So, the scenario you describe is pretty unlikely. Maybe new community developers will stand up now, but that is something I think we can only welcome. And forks don't need to be adversarial like that.
Regards,
This is a point where we should all remember TANSTAAFL. Its a law you cannot break no matter how many useless MBA's you hire. Developers like to eat regular and they can't do that on nothing but our good will. Dovecot has been good to us, so much so its become the default mail server all over this pale blue dot. Other than the monthly fees we pay our ISP's has any one of us donated a stock tank full of ice & beer for a summer picnic? Or $500 for a costly bug fixed in 2 hours? TANSTAAFL folks, There really Ain't No Such Thing As A Free Lunch.
Stephan.
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
Cheers, Gene Heskett.
"There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author, 1940) If we desire respect for the law, we must first make the law respectable.
- Louis D. Brandeis
Genes Web page http://geneslinuxbox.net:6309/
Hello all,
I want to provide a brief overview regarding various questions surrounding features that are being removed from Dovecot CE going forward.
We are currently working on providing updated/improved website info and documentation that will better explain exactly what is being maintained in CE. However, the desire to have unified messaging clashes with the Engineering Team's desire to continue to push code to the open source repository when it is ready...
So I want to educate on just a few points here, with the promise that further information will be provided in the future.
A reminder that Dovecot is commercial software, and has been since Timo made this decision 13 years ago. Dovecot is not maintained by a community of volunteers. We continue to be lucky that Timo remains Dovecot's Chief Architect today, but there are 20 dedicated Dovecot employees, plus additional Open-Xchange support staff, that are working on the software everyday. This is carrier-grade software, which requires significant resources to maintain.
Dovecot CE is the open source version of this commercial product (currently, Dovecot Pro). Dovecot CE is not a separate project - it is maintained as part of the day-to-day maintenance of Pro.
Every single person that works for Dovecot/OX is extremely proud and dedicated to releasing as much software as we can to open source. CE is able to take advantage of this situation to provide features that would not be allowed in a purely voluntary project (for example, there are 5 full time QA people working on what is eventually released as Dovecot CE).
However, there remains a delicate balance of what we can openly release and what we need to be able to commercially provide in order to keep the lights on (which allows us to continue to provide open releases...). This is a difficult juggling act, and is one that is always prone to recalibration in any open software product, not just Dovecot.
Dovecot CE has always been 100% open source, and will continue to be so. Nothing is changing in the future. Dovecot CE has been, and will always continue to be, fully compliant with open source principles (see https://opensource.org/osd/).
For a variety of software, maintenance, and (yes) business reasons, there comes a time when decisions need to be made to move beyond existing software. This is completely normal in software development, and there is no "open source" duty to continue to maintain software that is no longer useful (or, is broken or is unmaintained or is not longer best practices or is no longer commercially viable or is duplicative of other features that exist or ....) That decision is what is being done for a selection of longstanding Dovecot features. It is time to move on from them. There are valid reasons to do so.
If you disagree: the software is open source. You can continue to use the existing software, adapt it to your needs, move to a different solution, or whatever else.
To focus development efforts, and to provide extreme clarity for users going forward, Dovecot CE for the first time has adopted a defined Vision Statement: "To provide the world's premier open source, standards compliant, full-featured, single node email backend server." This vision formulation was made to ensure that CE users continue to receive world class, stable, tested, modern, secure email software going forward. Maintaining features that have existed since the mid-2000s (replication, Directors), at the expense of moving the software forward to adapt to new paradigms (cloud, containers, storage-layer replication, statelessness) is not the proper choice.
These Dovecot CE feature decisions are mine. If you are unhappy with them, I ask that you direct your vitriol directly (and privately) to me. The Dovecot Team does fantastic work and has provided software, under open source principles, that runs millions of email servers around the world. They continue to provide invaluable feedback internally in determining the proper balance between open and commercial considerations. They deserve to be thanked by the community, not vilified.
michael
On Tue, 18 Jul 2023 at 06:47, Michael Slusarz via dovecot dovecot@dovecot.org wrote:
Hello all,
I want to provide a brief overview regarding various questions surrounding features that are being removed from Dovecot CE going forward.
We are currently working on providing updated/improved website info and documentation that will better explain exactly what is being maintained in CE. However, the desire to have unified messaging clashes with the Engineering Team's desire to continue to push code to the open source repository when it is ready...
So I want to educate on just a few points here, with the promise that further information will be provided in the future.
A reminder that Dovecot is commercial software, and has been since Timo made this decision 13 years ago. Dovecot is not maintained by a community of volunteers. We continue to be lucky that Timo remains Dovecot's Chief Architect today, but there are 20 dedicated Dovecot employees, plus additional Open-Xchange support staff, that are working on the software everyday. This is carrier-grade software, which requires significant resources to maintain.
Dovecot CE is the open source version of this commercial product (currently, Dovecot Pro). Dovecot CE is not a separate project - it is maintained as part of the day-to-day maintenance of Pro.
Every single person that works for Dovecot/OX is extremely proud and dedicated to releasing as much software as we can to open source. CE is able to take advantage of this situation to provide features that would not be allowed in a purely voluntary project (for example, there are 5 full time QA people working on what is eventually released as Dovecot CE).
However, there remains a delicate balance of what we can openly release and what we need to be able to commercially provide in order to keep the lights on (which allows us to continue to provide open releases...). This is a difficult juggling act, and is one that is always prone to recalibration in any open software product, not just Dovecot.
Dovecot CE has always been 100% open source, and will continue to be so. Nothing is changing in the future. Dovecot CE has been, and will always continue to be, fully compliant with open source principles (see https://opensource.org/osd/).
For a variety of software, maintenance, and (yes) business reasons, there comes a time when decisions need to be made to move beyond existing software. This is completely normal in software development, and there is no "open source" duty to continue to maintain software that is no longer useful (or, is broken or is unmaintained or is not longer best practices or is no longer commercially viable or is duplicative of other features that exist or ....) That decision is what is being done for a selection of longstanding Dovecot features. It is time to move on from them. There are valid reasons to do so.
If you disagree: the software is open source. You can continue to use the existing software, adapt it to your needs, move to a different solution, or whatever else.
To focus development efforts, and to provide extreme clarity for users going forward, Dovecot CE for the first time has adopted a defined Vision Statement: "To provide the world's premier open source, standards compliant, full-featured, single node email backend server." This vision formulation was made to ensure that CE users continue to receive world class, stable, tested, modern, secure email software going forward. Maintaining features that have existed since the mid-2000s (replication, Directors), at the expense of moving the software forward to adapt to new paradigms (cloud, containers, storage-layer replication, statelessness) is not the proper choice.
I cannot imagine you will devote any time to justifying this statement, so I won't ask, but for a list of mainly technical participants, I imagine people cleverer than me have issues with the lack of substance.
These Dovecot CE feature decisions are mine. If you are unhappy with them, I ask that you direct your vitriol directly (and privately) to me. The Dovecot Team does fantastic work and has provided software, under open source principles, that runs millions of email servers around the world. They continue to provide invaluable feedback internally in determining the proper balance between open and commercial considerations. They deserve to be thanked by the community, not vilified.
Just to be clear, no one - involved with the project, or the commercial operations - should be vilified for any decision taken at a senior level.
But from my detached perspective - I do not use the feature - the real failure here is the lack of communication from the senior level. Aki has had to defend this decision for months without your support and communication.
Similar to the complete absence of communication around the Horde Project/Horde LLC.
Regards
Simon
On Mon, Jul 17, 2023 at 10:30:32PM -0600, Michael Slusarz via dovecot wrote:
For a variety of software, maintenance, and (yes) business reasons, there comes a time when decisions need to be made to move beyond existing software. This is completely normal in software development, and there is no "open source" duty to continue to maintain software that is no longer useful (or, is broken or is unmaintained or is not longer best practices or is no longer commercially viable or is duplicative of other features that exist or ....) That decision is what is being done for a selection of longstanding Dovecot features. It is time to move on from them. There are valid reasons to do so.
You are breaking the social contract with your (unpaying) customers. Expect push back and loss of goodwill. And hope that loss of goodwill does not amount to much in the long run.
These Dovecot CE feature decisions are mine. If you are unhappy with them, I ask that you direct your vitriol directly (and privately) to me. The Dovecot Team does fantastic work and has provided software, under open source principles, that runs millions of email servers around the world. They continue to provide invaluable feedback internally in determining the proper balance between open and commercial considerations. They deserve to be thanked by the community, not vilified.
The people I interacted with working on Dovecot are passinate about the work they do. And those kind of people do not usually say "I dont care. I just work here". Whether voiced or acted upon or not, expect loss of goodwill in your developers as well.
From what version on is replication gone? I am running 2.3.20 and it still there.
participants (34)
-
Aki Tuomi
-
Daniele
-
David Morsberger
-
David Zambonini
-
Dean Carpenter
-
deano-dovecot＠areyes.com
-
Doug Hardie
-
Emmanuel Dreyfus
-
Emmanuel Fusté
-
Eray Aslan
-
gene heskett
-
Gerald Galster
-
Gerben Wierda
-
gerben.wierda＠rna.nl
-
info＠joergschulz.de
-
Jim Popovitch
-
Jorge Bastos
-
Jörg Schulz
-
Marc
-
Michael Grant
-
Michael Grimm
-
Michael Peddemors
-
Michael Slusarz
-
Narcis Garcia
-
Nikolaos Milas
-
Noel Butler
-
Paul Kudla
-
pgnd
-
Simon B
-
Stephan Bosch
-
Steven Varco
-
Stuart Henderson
-
Vladimir Mishonov
-
William Edwards