Duplicate messages if message is moved when using dsync

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok - question

when you say dsync are you running that manually or as a cronjob ??

or using replication via dsync

there is a difference

replication is designed to handle mailbox changes by quing the changes 
on the fly and then update in the background.

replication tracks all changes in a mailbox and then sends them one at a 
time to the target server.

i run replication with tcp (not tcps or dsync)

I find (other then the 300+folder issues) that it runs pretty good in 
real time.

if you are running dsync manually then yes timing on its own will be a 
hit and miss and it will never sync properly if something appears during 
the dsync process.

if you have the sive running as discussed that is the first step

you should setup replication so things are handled in real time.

pls advise if you have already done replication or just manually dsyncing.

Below is basic replication setup but you need to make sure various 
pre-requirements are in place a head of time.

1. you need / preferably use a database for your user auth

    you need to setup an iterate sql statement that returns the user
    list of users to be replicated

Example (i use pgsql) :

# cat dovecot-pgsql.conf
driver = pgsql
connect = host=localhost port=5433 dbname=scom_billing user=pgsql password=
default_pass_scheme = PLAIN

password_query = SELECT username as user, password FROM email_users 
WHERE username = '%u' and password <> 'alias' and status = True and 
destination = '%u'

user_query = SELECT home, uid, gid FROM email_users WHERE username = 
'%u' and password <> 'alias' and status = True and destination = '%u'

iterate_query = SELECT "username" as user, domain FROM email_users WHERE 
status = True and alias_flag = False

#iterate_query  above is what returns the user lists

2. run the command doveadm user '*' - it needs to return a list of users 
that will trigger replication (after doing config above)

3. in general here is my relative parts for tcp replication

In dovecot.conf

---------------------------------------------------------------------------

mail_plugins = " virtual notify replication fts fts_lucene "

replication_dsync_parameters = -d -N -l 300 -U

dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u

plugin {

.....
   mail_replica = tcp:10.221.0.19:12345

#(note this is the ip address of the other server)
#it is the only config change ie server one above
#server2 would be the other server for communicating.



.....

}

protocol imap {
   mail_max_userip_connections = 50
   mail_plugins = $mail_plugins notify replication
}

protocol pop3 {
   mail_max_userip_connections = 50
   mail_plugins = $mail_plugins notify replication
}

protocol imaps {
   mail_max_userip_connections = 25
   mail_plugins = $mail_plugins notify replication
}

protocol pop3s {
   mail_max_userip_connections = 25
   mail_plugins = $mail_plugins notify replication
}

replication_max_conns = 100 (optional)

replication_full_sync_interval = 1d

service replicator {
   client_limit = 0
   drop_priv_before_exec = no
   idle_kill = 4294967295s
   process_limit = 1
   process_min_avail = 0
   service_count = 0
   vsz_limit = 8g
         unix_listener replicator-doveadm {
     mode = 0600
     user = vmail
   }
   vsz_limit = 8192M
}

service aggregator {
   process_limit = 1000
   #vsz_limit = 1g
   fifo_listener replication-notify-fifo {
     user = vmail
     group = vmail
     mode = 0666
   }

}



----------------------------------------------------------------------------

that should do it?












Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 5/18/2022 3:39 PM, Thom Pol wrote:
> Never mind. It did not work, issue still exists. I guess my network was 
> a little slower when I thought it worked, so the emails were moved by my 
> local client only after the sync had already happened between the servers.
> 
> So, back to the original message. Anyone any idea how this moving on 
> local client while the remote servers are still running the dsync can 
> work correctly without getting a duplicate?
> 
> Kind regards,
> 
> Thom Pol
> 
> 
> 
> On 18/05/2022 16:24, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:
>> Your Welcome
>>
>>
>>
>> Happy Wednesday !!!
>> Thanks - paul
>>
>> Paul Kudla
>>
>>
>> Scom.ca Internet Services<http://www.scom.ca>
>> 004-1009 Byron Street South
>> Whitby, Ontario - Canada
>> L1N 4S3
>>
>> Toronto 416.642.7266
>> Main?1.866.411.7266
>> Fax?1.888.892.7266
> 
> -- 
> This message has been scanned for viruses and
> dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
> believed to be clean.