[Dovecot] problems with SSH-based clustering dovecot 2.1.1
Hello,
I'm using dovecot 2.1.1 with vpopmail 5.4.30 with multiples domains and I have problems setting up synchronization in between multiple computers.
All act like master (my clients can connect to any of the them and read their emails either via POP3 either via IMAP, inbound email gets on any of the machines). Each machine is on a different continent, there is no shared drive in between and the synchronization is supposed to be asynchronous via cron scripts.
To simplify the case, let's consider 2 machines (mx1.a and mx2.a) with 3 virtual domains (a and b and c). On both machine domain a is the default domain (needs only username to connect to imap).
Initially I synchronize mx1.a with mx2.a using rsync. I check that I can login using dovecot.
I tried any of the following commands to synchronize the 2 machines:
mx1.a$ doveadm -Dv sync -u user1@a -f ssh mx2.a doveadm dsync-server -u user1@a
mx1.a$ doveadm -Dv sync -u user1@a ssh mx2.a doveadm -u user1@a
mx1.a$ doveadm -Dv sync -u user1@a user1@a
The only thing that happens is that the on each machine the folders get doubled with some random extension (eg. Inbox becomes Inbox_3e3ff3g3gb3bb3b22).
Also, another bug, if there is a domain setup as default (auth_default_realm) dsync simply ignores the specified -u <user> and attempts to sync the first email in the default domain.
Please advise.
# 2.1.1: /etc/dovecot/dovecot/dovecot.conf # OS: Linux 2.6.38-b i686 Slackware 13.0.0.0.0 auth_debug = yes auth_debug_passwords = yes auth_default_realm = a first_valid_gid = 89 first_valid_uid = 89 last_valid_gid = 89 last_valid_uid = 89 listen = * log_path = /dev/stderr login_greeting = A login_trusted_networks = 192.168.20.64/26 mail_debug = yes mail_gid = vchkpw mail_location = maildir:~/Maildir mail_privileged_group = vchkpw mail_uid = vpopmail passdb { driver = vpopmail } protocols = imap pop3 service auth-worker { unix_listener auth-worker { user = vpopmail } user = vpopmail } service auth { user = vpopmail } service imap-login { user = vpopmail } service pop3-login { user = vpopmail } ssl = no userdb { driver = vpopmail }
On 13.3.2012, at 7.41, Michescu Andrei wrote:
Initially I synchronize mx1.a with mx2.a using rsync. I check that I can login using dovecot. .. The only thing that happens is that the on each machine the folders get doubled with some random extension (eg. Inbox becomes Inbox_3e3ff3g3gb3bb3b22).
This is kind of a feature. Currently if two mailboxes have a same name, but different GUID, dsync doesn't even try to merge them but instead renames one of them.
So don't do initial sync with rsync, but with dsync. Alternatively you need to first get each mailbox assigned a GUID, for example: doveadm -A mailbox status guid '*'
Also, another bug, if there is a domain setup as default (auth_default_realm) dsync simply ignores the specified -u <user> and attempts to sync the first email in the default domain.
That can't be possible, something else is happening. What does dsync and auth log with debugs enabled when this happens?
Hello,
Thank you for your reply.
I'm attaching you the output for the 2nd bug. All the folders that you see in there does not exists in user1@b but they belong to first_user@a (which is NOT involved in this sync), BUT a is the default domain.
Also for the first suggestion:
- how do you sync initially the 2 machines? Because if you create the account on both machines, already the Inbox has 2 different guids
- if you know the guid, how do you change them? Because then I can do the rsync and after I can correct the guid on the other machine
Thank you, Andrei
On 13.3.2012, at 7.41, Michescu Andrei wrote:
Initially I synchronize mx1.a with mx2.a using rsync. I check that I can login using dovecot. .. The only thing that happens is that the on each machine the folders get doubled with some random extension (eg. Inbox becomes Inbox_3e3ff3g3gb3bb3b22).
This is kind of a feature. Currently if two mailboxes have a same name, but different GUID, dsync doesn't even try to merge them but instead renames one of them.
So don't do initial sync with rsync, but with dsync. Alternatively you need to first get each mailbox assigned a GUID, for example: doveadm -A mailbox status guid '*'
Also, another bug, if there is a domain setup as default (auth_default_realm) dsync simply ignores the specified -u <user> and attempts to sync the first email in the default domain.
That can't be possible, something else is happening. What does dsync and auth log with debugs enabled when this happens?
!DSPAM:4f5efb4c315461389012818!
On 13.3.2012, at 20.22, Michescu Andrei wrote:
I'm attaching you the output for the 2nd bug. All the folders that you see in there does not exists in user1@b but they belong to first_user@a (which is NOT involved in this sync), BUT a is the default domain.
The output showed debug output from the local dsync, but not from the remote. I think you'll see that if you do:
sudo -u vpopmail doveadm sync -u user1@b -f ssh mx2.a doveadm -Dv dsync-server -u user1@b
Also it's possible that in v2.1.1 there was some bug related to this.. You could try the latest nightly snapshot that has several fixes related to dsync: http://www.dovecot.org/nightly/
Also for the first suggestion:
- how do you sync initially the 2 machines? Because if you create the account on both machines, already the Inbox has 2 different guids
doveadm sync should be run before the destination Maildir exists at all. If vpopmail creates that, I guess it would just have to be deleted manually..
- if you know the guid, how do you change them? Because then I can do the rsync and after I can correct the guid on the other machine
The GUID is generated the first time it's used, which normally means when you run dsync for the first time. Alternatively you can also use doveadm to ask for the mailbox's GUID and it gets generated:
doveadm mailbox status -u user@domain guid '*'
Running rsync after this is done also copies the GUID (it's stored in dovecot-uidlist).
Hello,
Thank you very much... Using the nightly build and a combination of mailbox status + rsync + dsync made it happen.
So the *full* procedure was:
for every domain dom for every user u in dom doveadm mailbox status -u user@domain guid '*' rsync /home/vpopmail/domains/$dom/$u mx2.a:/home/vpopmail/domains/$dom/$u doveadm -Dv sync -u $u@$dom -f ssh mx2.a doveadm dsync-server -u $u@$dom loop $u loop $dom
Now, as long as I touch the mailbox of user1 only on mx1.a doveadm sync keeps them in sync (cron job every 5 minutes).
The problem comes when I start using the master-master model: emails starts getting duplicate with different ids.
Here is the example:
mx1.a receives an email for user1 (next line is the ls on the Maildir/new on mx1.a): -rw------- 1 vpopmail vchkpw 278 Mar 14 01:04 1331701451.24233.mx1,S\=278
mx2.a receives another email for user1: -rw------- 1 vpopmail vchkpw 273 Mar 14 07:05 1331701504.32564.mx2,S\=273
cron job starts on mx1.a: sudo -u vpopmail doveadm -Dv sync -u user1@b -f ssh mx2.a doveadm dsync-server -u user1@b doveadm(vpopmail): Debug: Loading modules from directory: /usr/lib/dovecot/doveadm doveadm(vpopmail): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(user1@b): Debug: auth input: user1@b uid=89 gid=89 home=/home/vpopmail/domains/b/user1 doveadm(user1@b): Debug: Effective uid=89, gid=89, home=/home/vpopmail/domains/b/user1 doveadm(user1@b): Debug: maildir++: root=/home/vpopmail/domains/b/user1/Maildir, index=, control=, inbox=/home/vpopmail/domains/b/user1/Maildir, alt= dsync-local(user1@b): Debug: Namespace : Using permissions from /home/vpopmail/domains/b/user1/Maildir: mode=0700 gid=-1 dsync-local(user1@b): Info: INBOX: Ignored 1 modseq changes dsync-local(user1@b): Info: INBOX: Couldn't keep all uids dsync-local(user1@b): Warning: Mailbox changes caused a desync. You may want to run dsync again.
--due to desync it runs again (via the cron script): sudo -u vpopmail doveadm -Dv sync -u user1@b -f ssh mx2.a doveadm dsync-server -u user1@b doveadm(vpopmail): Debug: Loading modules from directory: /usr/lib/dovecot/doveadm doveadm(vpopmail): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(user1@b): Debug: auth input: user1@b uid=89 gid=89 home=/home/vpopmail/domains/b/user1 doveadm(user1@b): Debug: Effective uid=89, gid=89, home=/home/vpopmail/domains/b/user1 doveadm(user1@b): Debug: maildir++: root=/home/vpopmail/domains/b/user1/Maildir, index=, control=, inbox=/home/vpopmail/domains/b/user1/Maildir, alt= dsync-local(user1@b): Debug: Namespace : Using permissions from /home/vpopmail/domains/b/user1/Maildir: mode=0700 gid=-1
The expected result would be that in user1's Mailbox on both mx1.a and mx2.a there would be 2 files... which is not happening
ls on mx1.a -rw------- 2 vpopmail vchkpw 278 Mar 14 01:04 1331701451.24233.mx1,S\=278 -rw------- 1 vpopmail vchkpw 273 Mar 14 01:05 1331701504.32564.mx2,S\=273 -rw------- 2 vpopmail vchkpw 278 Mar 14 01:04 1331702193.M868989P24524.mx1,S\=278
ls on mx2.a -rw------- 1 vpopmail vchkpw 278 Mar 14 07:04 1331701451.24233.mx1,S\=278 -rw------- 2 vpopmail vchkpw 273 Mar 14 07:05 1331701504.32564.mx1,S\=273 -rw------- 2 vpopmail vchkpw 273 Mar 14 07:05 1331702193.M798223P32571.mx2,S\=273
As you can see on every machine the original email gets duplicated.
Please advise on how I can fix this issue.
Thank you and Best regards, Andrei
On 13.3.2012, at 20.22, Michescu Andrei wrote:
I'm attaching you the output for the 2nd bug. All the folders that you see in there does not exists in user1@b but they belong to first_user@a (which is NOT involved in this sync), BUT a is the default domain.
The output showed debug output from the local dsync, but not from the remote. I think you'll see that if you do:
sudo -u vpopmail doveadm sync -u user1@b -f ssh mx2.a doveadm -Dv dsync-server -u user1@b
Also it's possible that in v2.1.1 there was some bug related to this.. You could try the latest nightly snapshot that has several fixes related to dsync: http://www.dovecot.org/nightly/
Also for the first suggestion:
- how do you sync initially the 2 machines? Because if you create the account on both machines, already the Inbox has 2 different guids
doveadm sync should be run before the destination Maildir exists at all. If vpopmail creates that, I guess it would just have to be deleted manually..
- if you know the guid, how do you change them? Because then I can do the rsync and after I can correct the guid on the other machine
The GUID is generated the first time it's used, which normally means when you run dsync for the first time. Alternatively you can also use doveadm to ask for the mailbox's GUID and it gets generated:
doveadm mailbox status -u user@domain guid '*'
Running rsync after this is done also copies the GUID (it's stored in dovecot-uidlist). !DSPAM:4f5f972f80146209382307!
On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote:
Now, as long as I touch the mailbox of user1 only on mx1.a doveadm sync keeps them in sync (cron job every 5 minutes).
The problem comes when I start using the master-master model: emails starts getting duplicate with different ids.
I was testing this a bit, and I guess in your tests dsync was running during a mail delivery, which seems to make it duplicate mails sometimes. I'll probably fix this at some point (I've actually been thinking about a larger dsync redesign), but anyway:
Even if dsync worked perfectly and didn't duplicate mails, it's not a great idea to do deliver mails randomly to both servers. Each time dsync notices that both sides have had new mails, it needs to reassign new IMAP UIDs for the messages, which means that IMAP clients may need to redownload the mails. Better to give one MX a higher priority so mails typically are delivered through it.
Hello,
Nope dsync was not running during the email delivery on that account. I've simulated in a controlled environment.
Servers are having different priorities, but this was a basic scenario to test the master-master synchronization.
Think that for incoming SMTP I can even restrict which server is the master (forcing all other to redeliver to this one). BUT, for a distributed IMAP cluster there is no way to restrict users to perform changes on only one server. This would defeat the model and the purpose of a distributed cluster...
One idea might be to have the IDs dependent on server where they appear first time so that they keep the ID once they get replicated. Here there are many options:
- the DB model = each server has a set of ids that can give (either ranges, either increment with step different then 1)
- the vpopmail/qmail model = append the server name (as you saw in the previous email in the listings the email files contained the hostname mx1.a and mx2.a)
Thank you, Andrei
On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote:
Now, as long as I touch the mailbox of user1 only on mx1.a doveadm sync keeps them in sync (cron job every 5 minutes).
The problem comes when I start using the master-master model: emails starts getting duplicate with different ids.
I was testing this a bit, and I guess in your tests dsync was running during a mail delivery, which seems to make it duplicate mails sometimes. I'll probably fix this at some point (I've actually been thinking about a larger dsync redesign), but anyway:
Even if dsync worked perfectly and didn't duplicate mails, it's not a great idea to do deliver mails randomly to both servers. Each time dsync notices that both sides have had new mails, it needs to reassign new IMAP UIDs for the messages, which means that IMAP clients may need to redownload the mails. Better to give one MX a higher priority so mails typically are delivered through it.
!DSPAM:4f60a6b137151972926802!
On 14.3.2012, at 18.45, Michescu Andrei wrote:
Nope dsync was not running during the email delivery on that account. I've simulated in a controlled environment.
How? You mean simply deliver mail to server A and to server B and run dsync and it duplicates it? I can't reproduce it that way, only if I run dsync during a flood of new mails.
Think that for incoming SMTP I can even restrict which server is the master (forcing all other to redeliver to this one). BUT, for a distributed IMAP cluster there is no way to restrict users to perform changes on only one server. This would defeat the model and the purpose of a distributed cluster...
For IMAP it's not much of a problem, because user typically still uses only one client actively, so clients aren't uploading mails to multiple servers at the same time.
One idea might be to have the IDs dependent on server where they appear first time so that they keep the ID once they get replicated. Here there are many options:
The messages have GUIDs that stay the same always, but IMAP UIDs are required to be ascending from client's point of view, and several clients rely on that, so when UID conflict happens the only safe thing to do is to assign new UIDs for all of the conflicting mails.
On 14.3.2012, at 18.45, Michescu Andrei wrote:
Nope dsync was not running during the email delivery on that account. I've simulated in a controlled environment.
How? You mean simply deliver mail to server A and to server B and run dsync and it duplicates it? I can't reproduce it that way, only if I run dsync during a flood of new mails.
YES. simply deliver mail to server A and then to server B (to the same user_1). After run dsync and you get exactly what you saw in my previous email. That's why I included the ls for the both servers, so that you can see the email files too. because each server duplicates only its own email (so brings the email from the other server and duplicates its own email).
Think that for incoming SMTP I can even restrict which server is the master (forcing all other to redeliver to this one). BUT, for a distributed IMAP cluster there is no way to restrict users to perform changes on only one server. This would defeat the model and the purpose of a distributed cluster...
For IMAP it's not much of a problem, because user typically still uses only one client actively, so clients aren't uploading mails to multiple servers at the same time.
hehe... one would think so, but when you have road-warriors that roam you can not insure that the server where they connect for IMAP (closest based on geo-ip) is the same as the server that you have picked for inbound SMTP. So you already have 2 servers that mess-up with user's mailbox.
The second case where you can not control this is for mobile devices that flip-in/out of wi-fi (my iPhone is in Canada when it is on 3G and in Europe when it is WiFi due to vpn tunneling, and this can change every couple of minutes... :( )
One idea might be to have the IDs dependent on server where they appear first time so that they keep the ID once they get replicated. Here there are many options:
The messages have GUIDs that stay the same always, but IMAP UIDs are required to be ascending from client's point of view, and several clients rely on that, so when UID conflict happens the only safe thing to do is to assign new UIDs for all of the conflicting mails.
well I don't know much about IMAP standard (you guys are the experts :)! here). If the GUID stays the same then this can be used to prevent the duplication error.
Also, as you can detect if the email is new or not (a client has already seen it or not): in the case that no one has seen it then it is safe to assign any UIDs that fits. In case that on only one server it has been seen then you can give it that UIDs on all servers, and reassign all the unseen ones. So the only messed-up case is if on both servers the message has been seen with different UIDs :(
Thank you very much for your time and patience. I know that our setup is pretty atypical. And think that this model with only 2 servers I'm showing you is only for simplicity as the real deployment has multiple servers geographically sparse connected by slow intercontinental internet links... :)) Otherwise we'd use a distribute file system and have only a unified storage :P
Best regards. Andrei
Hi --
On 14.03.2012, at 15:09, Timo Sirainen wrote:
On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote:
The problem comes when I start using the master-master model: emails starts getting duplicate with different ids.
I was testing this a bit, and I guess in your tests dsync was running during a mail delivery, which seems to make it duplicate mails sometimes. I'll probably fix this at some point (I've actually been thinking about a larger dsync redesign),
Good to hear ;-)
but anyway:
Even if dsync worked perfectly and didn't duplicate mails, it's not a great idea to do deliver mails randomly to both servers.
Sometimes croncobs are running on both servers at the same time producing locally delivered mails simultaneously, though. Ok, one can modify run times accordingly ...
Better to give one MX a higher priority so mails typically are delivered through it.
That's what I did. Now dsync/replicator is performing great, if the mail volume is rather low. I'm very satisfied, because this is the best performance ever. (Before I was running unison and dsync 2.0.)
But, whenever the high priority server will show delays during stress situations like huge mail loads, the low priority server will receive loads of mails as well. A dsync/replicator setup will then most probably produce duplicates (and multiples). That is a rather unrealistic scenario for my little severs, but others might have more difficulties.
And spammers don't care about mx priorities at all :-(
Regards, Michael
Hi --
On 14.03.2012, at 15:09, Timo Sirainen wrote:
On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote:
The problem comes when I start using the master-master model: emails starts getting duplicate with different ids.
I was testing this a bit, and I guess in your tests dsync was running during a mail delivery, which seems to make it duplicate mails sometimes. I'll probably fix this at some point (I've actually been thinking about a larger dsync redesign),
Good to hear ;-)
but anyway:
Even if dsync worked perfectly and didn't duplicate mails, it's not a great idea to do deliver mails randomly to both servers.
Sometimes croncobs are running on both servers at the same time producing locally delivered mails simultaneously, though. Ok, one can modify run times accordingly ...
Why do you run the crontab on all the servers? You can run a start-based system where only one ("main"-master) syncs all the other masters. And like this you avoid the time synch'ing of crontabs (especially if you don't always know how longer it will take for a dsync to finish).
Better to give one MX a higher priority so mails typically are delivered through it.
That's what I did. Now dsync/replicator is performing great, if the mail volume is rather low. I'm very satisfied, because this is the best performance ever. (Before I was running unison and dsync 2.0.)
But, whenever the high priority server will show delays during stress situations like huge mail loads, the low priority server will receive loads of mails as well. A dsync/replicator setup will then most probably produce duplicates (and multiples). That is a rather unrealistic scenario for my little severs, but others might have more difficulties.
This is another nice case to "motivate" Timo to look for solutions ;) I tried to push 2 ideas in the same direction earlier :P Especially that he confirmed that every single email has a unique GID (which should help prevent duplication/multiplication)...
And spammers don't care about mx priorities at all :-(
Actually, statistically speaking, spammers select the low priority ones.
Regards, Michael
Nice to hear that we are not the only ones out there to try to run something like this over dovecot :P
Thnx, Andrei
Hi --
On 14.03.2012, at 21:58, Michescu Andrei wrote:
Sometimes croncobs are running on both servers at the same time producing locally delivered mails simultaneously, though. Ok, one can modify run times accordingly ...
Why do you run the crontab on all the servers? You can run a start-based system where only one ("main"-master) syncs all the other masters.
You misunderstood. I was referring to system cronjob's mail reports from cron.daily jobs like security reports et al. Those reports normally run at identical times.
And spammers don't care about mx priorities at all :-(
Actually, statistically speaking, spammers select the low priority ones.
Actually: you are right ;-)
Nice to hear that we are not the only ones out there to try to run something like this over dovecot :P
Yes. I never loved the idea of a clusterfs for my small mail servers, I always considered such clusterfs an overkill. Well, my servers do reside in the same housing building, thus it could be done without performance loss. But a scenario of worldwide distributed mail servers desires a dsync/replicator scheme, IMHO ;-)
Regards, Michael
On 2012-03-14 5:51 PM, Michael Grimm <trashcan@odo.in-berlin.de> wrote:
You misunderstood. I was referring to system cronjob's mail reports from cron.daily jobs like security reports et al. Those reports normally run at identical times.
But are these really 'duplicate' mails? It sounds to me like they are individual to each system.
I'm also confused - are you actually delivering the exact *same* mail to two (or multiple) *different* servers simultaneously? If only one copy of the mail gets delivered, regardless of which server it gets delivered to, when dsync runs, there would be no duplicates, right?
I'm asking for clarification because I was considering a similar setup.
--
Best regards,
Charles
Hi --
On 15.03.2012 12:21, Charles Marcus wrote:
On 2012-03-14 5:51 PM, Michael Grimm <trashcan@odo.in-berlin.de> wrote:
You misunderstood. I was referring to system cronjob's mail reports from cron.daily jobs like security reports et al. Those reports normally run at identical times.
But are these really 'duplicate' mails? It sounds to me like they are individual to each system.
I'm also confused - are you actually delivering the exact *same* mail to two (or multiple) *different* servers simultaneously? If only one copy of the mail gets delivered, regardless of which server it gets delivered to, when dsync runs, there would be no duplicates, right?
Well, let me explain it in more detail:
Given there are two servers called mx1 and mx2. They both have cron.daily jobs running, and let's say those cronjobs are meant to create at 3:00 a postfix-logwatch report on every server. Thus, the cronjob at mx1 sends his final report to the admin of mx1, and the one at mx2 to the admin of mx2. I happen to be the one who will finally receive those reports, and therefore I did tell sieve to drop them into some folder of mine, let's say REPORTS.
Thus, at 3:01 one report from mx1 will be delivered at mx1 into mailfolder REPORTS and at 3:01 one report from mx2 will be delivered at mx2 into the mailfolder REPORTS. Important: both mails are different but they arrive in the mailfolder REPORTS at the same time, one at mx1 the other at mx2. And, let's call the report from mx1 cronjob "mx1-report" and that from mx2 "mx2-report".
I had dsync running every minute. Thus at 3:00 the final sync has been initiated, and at 3:01 dsync will find two mails to sync in REPORTS. In 99.9% of all synchronizations the final result at both server's REPORTS mailbox is as expected and as follows:
mx1-report 3:01 mx2-report 3:01
But occasionally, and what I refer to as duplicates, I did find either ...
mx1-report 3:01 mx1-report 3:01 mx2-report 3:01
... or ...
mx1-report 3:01 mx2-report 3:01 mx2-report 3:01
Actually, that was when I started to investigate how dsync will behave when many mails arrive at two servers simultaneously with identical final mailboxes.
The day I switched to the new replicator/dsync technique, those duplicates are history, but I'm still able to produce duplicates (and multiples) if I simultaneously produce *many* mails at every server with identical mailbox destinations in a minute (see my other report a couple of days ago). Timo is suspecting the combination of arriving mails while running dsync to be a possible cause of such duplicates, if I didn't get him wrong.
Again, if your servers aren't receiving loads of mails for the very same mailboxes within very short time, the current dsync/replicator works great.
HTH and regards, Michael
On 2012-03-15 9:46 AM, Michael Grimm <trashcan@odo.in-berlin.de> wrote:
Thus, at 3:01 one report from mx1 will be delivered at mx1 into mailfolder REPORTS and at 3:01 one report from mx2 will be delivered at mx2 into the mailfolder REPORTS. Important: both mails are different but they arrive in the mailfolder REPORTS at the same time, one at mx1 the other at mx2. And, let's call the report from mx1 cronjob "mx1-report" and that from mx2 "mx2-report".
so these are LOCAL mails delivered to local user accounts? The easiest thing to do for this is simply alias the local address(es) so that they all go to one single server/account (I would use only virtual, but you can do it with system accounts too).
I see lots of potential problems doing it the way you are doing it.
--
Best regards,
Charles
Hi --
On 15.03.2012 15:04, Charles Marcus wrote:
On 2012-03-15 9:46 AM, Michael Grimm <trashcan@odo.in-berlin.de> wrote:
Thus, at 3:01 one report from mx1 will be delivered at mx1 into mailfolder REPORTS and at 3:01 one report from mx2 will be delivered at mx2 into the mailfolder REPORTS. Important: both mails are different but they arrive in the mailfolder REPORTS at the same time, one at mx1 the other at mx2. And, let's call the report from mx1 cronjob "mx1-report" and that from mx2 "mx2-report".
so these are LOCAL mails delivered to local user accounts?
All locally produced mails are aliased to the very same virtual user, namely myself.
The easiest thing to do for this is simply alias the local address(es) so that they all go to one single server/account (I would use only virtual, but you can do it with system accounts too).
That is exactly what I'm doing, I'm running virtual, only. No local user accounts here. Every locally produced system mail end in virtual mailboxes of myself.
In the given example "mx1-report" is delivered to REPORTS@mx1 and "mx2-report" to REPORTS@mx2. Now, I want to access them via IMAP for instance at my mx1 mail account. Without dsync I would only be able to access "mx1-report", thus I do need to sync REPORTS to see both at mx1.
I see lots of potential problems doing it the way you are doing it.
Hmm, now, I don't understand you.
Regards, Michael
The day I switched to the new replicator/dsync technique, those duplicates are history, but I'm still able to produce duplicates (and multiples) if
Hello,
Can you get a little bit more in details about this replicator/dsync techique? As my main problem is that EVERYTHING (that gets created on different servers in the same time) gets duplicated.
I only do replication using the doveadm sync command.
My servers are geographically distributed as you might remember from previous posts so I run doveadm every 5 minutes, and only 1 instance of doveadm runs at any given times (so let's say that due to a HUGE volume the doveamd take 30 minutes to complete, then all in-between 5minutes are skipped).
Thnx, Andrei
On 15.3.2012, at 19.49, Michescu Andrei wrote:
Can you get a little bit more in details about this replicator/dsync techique? As my main problem is that EVERYTHING (that gets created on different servers in the same time) gets duplicated.
I only do replication using the doveadm sync command.
Try at least v2.1.2 first, since it has some fixes. Also post your doveconf -n output.
Hello Timo,
I have update the repository with hg pull -u, recompiled and redeployed and somehow the dovecot -n still shows 2.1.1... :(
I ran exactly the same test: starting for 1 clean user1, I create 2 emails, one on mx1.a and one on mx2.a and I sync them with doveadm.
The output is exactly as previously sent :(
Here is my conf:
# 2.1.1: /etc/dovecot/dovecot/dovecot.conf # OS: Linux 2.6.38-b i686 Slackware 13.0.0.0.0 auth_debug = yes auth_debug_passwords = yes auth_default_realm = a first_valid_gid = 89 first_valid_uid = 89 last_valid_gid = 89 last_valid_uid = 89 listen = * log_path = /dev/stderr login_greeting = WebMail MX1.A login_trusted_networks = 192.168.20.64/26 mail_debug = yes mail_gid = vchkpw mail_location = maildir:~/Maildir mail_privileged_group = vchkpw mail_uid = vpopmail passdb { driver = vpopmail } protocols = imap pop3 service auth-worker { unix_listener auth-worker { user = vpopmail } user = vpopmail } service auth { user = vpopmail } service imap-login { user = vpopmail } service pop3-login { user = vpopmail } ssl = no userdb { driver = vpopmail }
Thank you, Andrei
On 15.3.2012, at 19.49, Michescu Andrei wrote:
Can you get a little bit more in details about this replicator/dsync techique? As my main problem is that EVERYTHING (that gets created on different servers in the same time) gets duplicated.
I only do replication using the doveadm sync command.
Try at least v2.1.2 first, since it has some fixes. Also post your doveconf -n output.
!DSPAM:4f622cb881591647615726!
Hi --
On 15.03.2012, at 18:49, Michescu Andrei wrote:
Can you get a little bit more in details about this replicator/dsync techique?
http://blog.dovecot.org/2012/02/dovecot-clustering-with-dsync-based.html and http://www.dovecot.org/img/dsync-director-replication-ssh.png helped me a lot understand the idea behind it.
As my main problem is that EVERYTHING (that gets created on different servers in the same time) gets duplicated.
As Timo recommended already, you better upgrade to 2.1.2 first. I can confirm that he fixed a lot compared to older dsync versions.
Regards, Michael
hello,
So I upgraded to 2.1.2 (not from repository because that one still says 2.1.1, but from the release).
I ran exactly the same test with exactly the same behaviour. (new account, synced successfully on 2 servers, deliver 1 email to each server, run doveadm sync)...
Please find below the dovecot-uidlists:
on mx1.a: 3 V1331851700 N1 Gc9e2a526b471624f70760000498f706b 1 :1331852540.19862.mx2,S=272 2 G1331852540.19862.mx2,S=272 :1331852573.M89342P19877.mx2,S=272 3 :1331852488.30409.mx1,S=268
on mx2.a: 3 V1331851700 N1 Gc9e2a526b471624f70760000498f706b 1 :1331852488.30409.mx1,S=268 2 :1331852540.19862.mx2,S=272 3 G1331852488.30409.mx1,S=268 :1331852572.M622052P30410.mx1,S=268
As you can see both servers duplicated the email that was delivered first to them (1 in both cases, because the user1 is a clean account). There is the same effect in the folders: initial there is only one file on each server and after sync there are 3 files instead of only 2...
Also, after the sync, there should be 2 new emails (N2 if I interpret correctly that N1 means only one new).
Thank you. Andrei
PS: also I need to run dsync twice, because first time I receive:
dsync-local(user1@a): Info: INBOX: Ignored 1 modseq changes dsync-local(user1@a): Info: INBOX: Couldn't keep all uids dsync-local(user1@a): Warning: Mailbox changes caused a desync. You may want to run dsync again.
The config is below: # 2.1.2: /etc/dovecot/dovecot/dovecot.conf # OS: Linux 2.6.38-b i686 Slackware 13.0.0.0.0 auth_debug = yes auth_debug_passwords = yes auth_default_realm = a first_valid_gid = 89 first_valid_uid = 89 last_valid_gid = 89 last_valid_uid = 89 listen = * log_path = /dev/stderr login_greeting = WebMail MX1.A login_trusted_networks = 192.168.20.64/26 mail_debug = yes mail_gid = vchkpw mail_location = maildir:~/Maildir mail_privileged_group = vchkpw mail_uid = vpopmail passdb { driver = vpopmail } protocols = imap pop3 service auth-worker { unix_listener auth-worker { user = vpopmail } user = vpopmail } service auth { user = vpopmail } service imap-login { user = vpopmail } service pop3-login { user = vpopmail } ssl = no userdb { driver = vpopmail }
Hi -
On 14.03.2012, at 22:14, Timo Sirainen wrote:
On 14.3.2012, at 22.36, Michael Grimm wrote:
And spammers don't care about mx priorities at all :-(
But spams go to spam mailbox where duplicates don't really matter. :)
True ;-) But spam mails might interfere with syncing of legitimate mail arriving at the same time.
Regards, Michael
On 14.3.2012, at 22.36, Michael Grimm wrote:
And spammers don't care about mx priorities at all :-(
But spams go to spam mailbox where duplicates don't really matter. :)
In an ideal world yes... or no.
In our deployment spam is simply header tagged and left in INBOX. Each user can decide after what they want to do with it (client side rules).
And, in the end, it is the same discussion, because the spam mailbox get replicated too and if the spam gets duplicated we are in the worst scenarios... knowing that spam represents 95% of all email traffic (in a real-world public-facing system).
;)
hehe... in the meanwhile I looked a little on the ietf and there are different RFCs out there on disconnected clients and UIDPLUS and other nice features ;) let me know if you are interested to get some help in implementing it :D
participants (5)
-
Charles Marcus
-
Michael Grimm
-
Michescu Andrei
-
Michescu Andrei
-
Timo Sirainen