Replication not working - GUIDs conflict - will be merged later
Paul Kudla (SCOM.CA Internet Services Inc.)
paul at scom.ca
Mon Aug 1 10:47:18 UTC 2022
OK thanks for the updates
Long story short i went through a bunch of replication issues when i was
first setting up dovecot.
Most of which were fixed in 2.3.19 and i have not seen any issues since.
in general i had to turn on the debugging mode (mail_debug = yes)
and filter syslog by "replication"
It was discovered through some work that any account over 300 physical
folders (give or take) would not replicate (physical size of mailbox had
nothing to do with the issue just folder count) and would fail without
error (why i asked the version this was an issue in 2.3.18 and before?)
when a replication sync failed it did say in the logs replication
requested for <email account> but it would fail and not log the error
why, replication timeouts were however recorded?
I my self ended up patching the c code in the replicator to get more
detail how far a replication sync would go (ie i added a bunch of
logging code to track the issues better)
maybe look at folder counts? - It was only effecting 5 of my customers
but was a pain to find the issue.
that being said i had to make some scripts to show user replication and
what was outstanding between the servers (i had to run them on both
servers seperately to acurately get the replication status going in both
directions.
I now run these scripts in the background every 5 minutes to make sure
replication on both sides are in sync.
also i found tcpip replication (without ssl) worked the best? tcpip over
SSL had timing errors, ssl in my case was not required as the two
servers were hardwired together (ie no security leaks)
if your two servers are at two different sites consider a small vpn setup
i like using gre for this as it is hardwired by static ip addresses on
both sides and you can set a 10.x.x.x (or whatever) to communicate
between the servers ??
also on using the doveadm ssh scripts introduced rights issues between
the two servers file systems (even though they were identical)
Basically i tried everything !
tcpip - non ssl just seem to work the best.
sync.status :
doveadm replicator status
echo ' '
doveadm replicator dsync-status | grep -v 'Not connected'
which outputs :
# sync.status
Queued 'sync' requests 0
Queued 'high' requests 0
Queued 'low' requests 0
Queued 'failed' requests 0
Queued 'full resync' requests 0
Waiting 'failed' requests 0
Total number of known users 269
oh and another major thing was the replication selecting from the
database properly.
i use postgresql
#iterate_query = SELECT user, password FROM email_users WHERE username =
'%u' and password <> 'alias' and status = True and destination = '%u'
iterate_query = SELECT "username" as user, domain FROM email_users WHERE
status = True and alias_flag = False
note my db setup uses status=True for an active user and
alias_flag=False for an alias email redirect inside postfix (fyi), you
can ignore these based on how your database is setup.
dovecot is very intelligent if an active box gets activity on one server
then it wont replicate to the other server if that mailbox is not
returned to sync users list (but it will setup / activate replication on
the server that received the email) - this took a bit to figure out as well.
# cat sync.users
doveadm replicator status '*' | grep ' y'
would only sync what was bad in the replication??
try running
# doveadm user '*' | wc
269 269 5244
on both servers the account count (269) in my case should be the same on
both servers.
Here is my dovecot.conf config maybe it will help
replication configs (other the the server ip address) must be the same
on both sides.
# cat dovecot.conf
# 2.3.14 (cee3cbc0d): /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 12.1-RELEASE amd64
# Hostname: mail18.scom.ca
auth_debug = no
auth_debug_passwords = no
default_process_limit = 16384
mail_debug = no
#lock_method = dotlock
#mail_max_lock_timeout = 300s
#mbox_read_locks = dotlock
#mbox_write_locks = dotlock
mmap_disable = yes
dotlock_use_excl = no
mail_fsync = always
mail_nfs_storage = no
mail_nfs_index = no
auth_mechanisms = plain login
auth_verbose = yes
base_dir = /data/dovecot/run/
debug_log_path = syslog
disable_plaintext_auth = no
dsync_features = empty-header-workaround
info_log_path = syslog
login_greeting = SCOM.CA Internet Services Inc. - Dovecot ready
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
mail_location = maildir:~/
mail_plugins = " virtual notify replication fts fts_lucene "
mail_prefetch_count = 20
protocols = imap pop3 lmtp sieve
protocol lmtp {
mail_plugins = $mail_plugins sieve
postmaster_address = monitor at scom.ca
}
service lmtp {
process_limit=1000
vsz_limit = 512m
client_limit=1
unix_listener /usr/home/postfix.local/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
}
protocol lda {
mail_plugins = $mail_plugins sieve
}
service lda {
process_limit=1000
vsz_limit = 512m
}
service imap {
process_limit=4096
vsz_limit = 2g
client_limit=1
}
service pop3 {
process_limit=1000
vsz_limit = 512m
client_limit=1
}
namespace inbox {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
separator = /
}
passdb {
args = /usr/local/etc/dovecot/dovecot-pgsql.conf
driver = sql
}
doveadm_port = 12345
doveadm_password = secretxxxx
service doveadm {
process_limit = 0
process_min_avail = 0
idle_kill = 0
client_limit = 1
user = vmail
inet_listener {
port = 12345
}
}
service config {
unix_listener config {
user = vmail
}
}
dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u
#dsync_remote_cmd = doveadm sync -d -u%u
replication_dsync_parameters = -d -N -l 300 -U
plugin {
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename
mail_log_fields = uid, box, msgid, from, subject, size, vsize, flags
push_notification_driver = dlog
sieve = file:~/sieve;active=~/sieve/.dovecot.sieve
#sieve = ~/.dovecot.sieve
sieve_duplicate_default_period = 1h
sieve_duplicate_max_period = 1h
sieve_extensions = +duplicate +notify +imapflags +vacation-seconds
sieve_global_dir = /usr/local/etc/dovecot/sieve
sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve
mail_replica = tcp:10.221.0.19:12345
#mail_replica = remote:vmail at 10.221.0.19
#replication_sync_timeout = 2
fts = lucene
fts_lucene = whitespace_chars=@.
fts_autoindex = yes
fts_languages = en
}
#sieve_extensions = vnd.dovecot.duplicate
#sieve_plugins = vnd.dovecot.duplicate
service anvil {
process_limit = 1
client_limit=5000
vsz_limit = 512m
unix_listener anvil {
group = vmail
mode = 0666
}
}
service indexer-worker {
vsz_limit = 2g
}
service auth {
process_limit = 1
client_limit=5000
vsz_limit = 1g
unix_listener auth-userdb {
mode = 0660
user = vmail
group = vmail
}
unix_listener /var/spool/postfix/private/auth {
mode = 0666
}
}
service stats {
process_limit = 1000
vsz_limit = 1g
unix_listener stats-reader {
group = vmail
mode = 0666
}
unix_listener stats-writer {
group = vmail
mode = 0666
}
}
userdb {
args = /usr/local/etc/dovecot/dovecot-pgsql.conf
driver = sql
}
protocol imap {
mail_max_userip_connections = 50
mail_plugins = $mail_plugins notify replication
}
protocol pop3 {
mail_max_userip_connections = 50
mail_plugins = $mail_plugins notify replication
}
protocol imaps {
mail_max_userip_connections = 25
mail_plugins = $mail_plugins notify replication
}
protocol pop3s {
mail_max_userip_connections = 25
mail_plugins = $mail_plugins notify replication
}
service managesieve-login {
process_limit = 1000
vsz_limit = 1g
inet_listener sieve {
port = 4190
}
}
verbose_proctitle = yes
replication_max_conns = 100
replication_full_sync_interval = 1d
service replicator {
client_limit = 0
drop_priv_before_exec = no
idle_kill = 4294967295s
process_limit = 1
process_min_avail = 0
service_count = 0
vsz_limit = 8g
unix_listener replicator-doveadm {
mode = 0600
user = vmail
}
vsz_limit = 8192M
}
service aggregator {
process_limit = 1000
#vsz_limit = 1g
fifo_listener replication-notify-fifo {
user = vmail
group = vmail
mode = 0666
}
}
service pop3-login {
process_limit = 1000
client_limit = 100
vsz_limit = 512m
}
service imap-urlauth-login {
process_limit = 1000
client_limit = 1000
vsz_limit = 1g
}
service imap-login {
process_limit=1000
client_limit = 1000
vsz_limit = 1g
}
protocol sieve {
managesieve_implementation_string = Dovecot Pigeonhole
managesieve_max_line_length = 65536
}
#Addition ssl config
!include sni.conf
# cat dovecot-pgsql.conf
driver = pgsql
connect = host=localhost port=5433 dbname= user= password=
default_pass_scheme = PLAIN
password_query = SELECT username as user, password FROM email_users
WHERE username = '%u' and password <> 'alias' and status = True and
destination = '%u'
user_query = SELECT home, uid, gid FROM email_users WHERE username =
'%u' and password <> 'alias' and status = True and destination = '%u'
#iterate_query = SELECT user, password FROM email_users WHERE username =
'%u' and password <> 'alias' and status = True and destination = '%u'
iterate_query = SELECT "username" as user, domain FROM email_users WHERE
status = True and alias_flag = False
Please note the above selects return a full email address (how i do it)
- ie paul at scom.ca
# cat sni.conf
#sni.conf
ssl = yes
verbose_ssl = yes
ssl_dh =</usr/local/etc/dovecot/dh-4096.pem
ssl_prefer_server_ciphers = yes
#ssl_min_protocol = TLSv1.2
#Default *.scom.ca
ssl_key =</usr/local/etc/dovecot/scom.pem
ssl_cert =</usr/local/etc/dovecot/scom.pem
ssl_ca =</usr/local/etc/dovecot/scom.pem
local_name .scom.ca {
ssl_key = /programs/common/getssl.cert -c *.scom.ca -q yes
ssl_cert = /programs/common/getssl.cert -c *.scom.ca -q yes
ssl_ca = /programs/common/getssl.cert -c *.scom.ca -q yes
}
local_name mail.clancyca.com {
ssl_key = /programs/common/getssl.cert -c mail.clancyca.com -q yes
ssl_cert = /programs/common/getssl.cert -c mail.clancyca.com -q yes
ssl_ca = /programs/common/getssl.cert -c mail.clancyca.com -q yes
}
local_name mail.paulkudla.net {
ssl_key = /programs/common/getssl.cert -c mail.paulkudla.net -q yes
ssl_cert = /programs/common/getssl.cert -c mail.paulkudla.net -q yes
ssl_ca = /programs/common/getssl.cert -c mail.paulkudla.net -q yes
}
local_name secure.clancyca.com {
ssl_key = /programs/common/getssl.cert -c secure.clancyca.com -q yes
ssl_cert = /programs/common/getssl.cert -c secure.clancyca.com -q yes
ssl_ca = /programs/common/getssl.cert -c secure.clancyca.com -q yes
}
local_name mail.ekst.ca {
ssl_key = /programs/common/getssl.cert -c mail.ekst.ca -q yes
ssl_cert = /programs/common/getssl.cert -c mail.ekst.ca -q yes
ssl_ca = /programs/common/getssl.cert -c mail.ekst.ca -q yes
}
local_name mail.hamletdevelopments.ca {
ssl_key = /programs/common/getssl.cert -c mail.hamletdevelopments.ca
-q yes
ssl_cert = /programs/common/getssl.cert -c mail.hamletdevelopments.ca
-q yes
ssl_ca = /programs/common/getssl.cert -c mail.hamletdevelopments.ca
-q yes
}
note the sni.conf above suck in the certs from a db.
Another thought is are you running duplicate supression, i am not sure
how that would work when using imapsync (ie i have to assume a lot of
emails when you run a sync would carry the same info)
duplicate suppression seems to pickup on job numbers, to, from etc to
decide if an email is a duplicate. Maybe this is also an issue.
# cat duplicates.sieve
require "duplicate"; # for dovecot >= 2.2.18
if duplicate {
discard;
stop;
}
Happy Monday !!!
Thanks - paul
Paul Kudla
Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3
Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email paul at scom.ca
On 8/1/2022 5:15 AM, Patrick Westenberg wrote:
>
> Very interesting new insights:
>
> When I use imapsync and let it synchronize mails from INBOX to
> INBOX/testfolder, the automatic replication works fine.
> All mails are synchronized between my two backends.
>
>
> When I move the mails to the INBOX (doveadm move -u mail at example.com
> INBOX mailbox INBOX/testfolder all), these mails are lost on the
> replica! They are neither in INBOX, nor in INBOX/testfolder
>
> Regards
> Patrick
>
More information about the dovecot
mailing list