Replication issues with 2.3.18 - Replication fails after xxx folders reached.

[Paul Kudla (SCOM.CA Internet)]

please appreciate that i am trying to help ....

after much testing i can now reporduce the replication issues at hand

I am running on freebsd 12 & 13 stable (both test and production servers)

sdram drives etc ...

Basically replication works fine until reaching a folder quantity of ~ 
256 or more

to reproduce using doveadm i created folders like

INBOX/folder-0
INBOX/folder-1
INBOX/folder-2
INBOX/folder-3
and so forth ......

I created 200 folders and they replicated ok on both servers

I created another 200 (400 total) and the replicator got stuck and would 
not update the mbox on the alternate server anymore and is still 
updating 4 days later ?

basically replicator goes so far and either hangs or more likely bails 
on an error that is not reported to the debug reporting ?

however dsync will sync the two servers but only when run manually (ie 
all the folders will sync)

I have two test servers avaliable if you need any kind of access - again 
here to help.

[07:28:42] mail18.scom.ca [root:0] ~
# sync.status
Queued 'sync' requests 0
Queued 'high' requests 6
Queued 'low' requests 0
Queued 'failed' requests 0
Queued 'full resync' requests 0
Waiting 'failed' requests 0
Total number of known users 255

username                       type status
paul at scom.ca                   normal      Waiting for dsync to finish
keith at elirpa.com               incremental Waiting for dsync to finish
ed.hanna at dssmgmt.com           incremental Waiting for dsync to finish
ed at scom.ca                     incremental Waiting for dsync to finish
nick at elirpa.com                incremental Waiting for dsync to finish
paul at paulkudla.net             incremental Waiting for dsync to finish


i have been going through the c code and it seems the replication gets 
requested ok

replicator.db does get updated ok with the replicated request for the 
mbox in question.

however i am still looking for the actual replicator function in the 
lib's that do the actual replication requests

the number of folders & subfolders is defanately the issue - not the 
mbox pyhsical size as thought origionally.

if someone can point me in the right direction, it seems either the 
replicator is not picking up on the number of folders to replicat 
properly or it has a hard set limit like 256 / 512 / 65535 etc and stops 
the replication request thereafter.

I am mainly a machine code programmer from the 80's and have 
concentrated on python as of late, 'c' i am starting to go through just 
to give you a background on my talents.

It took 2 months to finger this out.

this issue also seems to be indirectly causing the duplicate messages 
supression not to work as well.

python programming to reproduce issue (loops are for last run started @ 
200 - fyi) :

# cat mbox.gen
#!/usr/local/bin/python2

import os,sys

from lib import *


user = 'paul at paulkudla.net'

"""
for count in range (0,600) :
         box = 'INBOX/folder-%s' %count
         print count
         command = '/usr/local/bin/doveadm mailbox create -s -u %s %s' 
%(user,box)
         print command
         a = commands.getoutput(command)
         print a
"""

for count in range (0,600) :
         box = 'INBOX/folder-0/sub-%' %count
         print count
         command = '/usr/local/bin/doveadm mailbox create -s -u %s %s' 
%(user,box)
         print command
         a = commands.getoutput(command)
         print a



         #sys.exit()

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20220402/8a1413cf/attachment.htm>