flags not synced correctly with dovecot sync (dsync)

[Dan Christensen]

Over many months I've been getting frequent errors in synchronization
using dsync.  They are hard to reproduce, and aren't always of the same
nature, but there's a particular case that I can reproduce 100% of the
time.  It involves read status being synced in the wrong direction
when two machines are both synchronizing with a server.

I'm reposting below a message that gives the method for reproducing the
problem.  I can still reproduce it with Ubuntu's 2:2.3.7.2-1~bionic
packages.

I also have problems with only one machine synchronizing with a server,
but those I can't reproduce on demand.

I'm happy to try patches or test other things.

I'm using Maildir storage.

Thanks for any suggestions!

Dan

On Feb 16, 2019, Dan Christensen via dovecot <dovecot at dovecot.org> wrote:

> I'm running dovecot 2.3.4.1 from https://repo.dovecot.org/ on Ubuntu
> 18.04 on three machines that I'll call server, laptop1 and laptop2.
>
> Both laptop1 and laptop2 run dovecot sync against server to keep local
> copies of my imap folders.  Even when I initially had only two machines,
> laptop1 and server, I occasionally noticed that flags were lost, usually
> custom flags used by Gnus, but I couldn't reliably reproduce the
> problem.
>
> Now that I have two laptops syncing against the server, the problem has
> gotten worse and I figured out a way to reproduce it:
>
> - on server: create new IMAP folder test, and put two read messages in it
> - on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
> - on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
>
> At this point, all three machines show the two messages M1 and M2
> as being read.
>
> - on laptop1: mark message M1 unread
> - on laptop2: mark message M2 unread
> - on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
>   Both laptop1 and server have M1 unread, M2 read, as expected.
> - on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
>   Now laptop2 and server have M1 *read*, M2 unread.
> - on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
>   Now laptop1 and the server have both M1 and M2 *read*.
> - on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
>   Now laptop2 has both read as well.
>
> The two lines that say "*read*" are wrong in my opinion.  dsync
> propagated a read mark to an unread message, even though that message
> was marked unread more recently than it was marked read.
>
> I usually use stateful sync, and get many related problems.
> I just did a test in which M1 and M2 started out read, and I
> started with empty files named dstate.test on laptop1 and laptop2.
> Then I did the above procedure, using the command
>
> doveadm sync -u user -l 10 -m test -s "`cat dstate.test`" user at server > dstate.test
>
> At the end, laptop2 and server had both messages unread (which is good),
> but laptop1 had only M1 unread, and repeated runs of the sync command
> did not correct this.  So the stateful sync failed to detect a change.
>
> Are these bugs in dovecot?  Is there more information that I can
> provide?  The output of doveconf -n on one machine is below, and
> the others are almost identical.
>
> Thanks for any help!
>
> Dan
>
> # 2.3.4.1 (3c0b8769e): /etc/dovecot/dovecot.conf
> # OS: Linux 4.15.0-45-generic x86_64 Ubuntu 18.04.1 LTS 
> # Hostname: laptop2
> auth_mechanisms = plain login
> listen = 127.0.0.1
> mail_index_log2_max_age = 10 days
> mail_index_log_rotate_min_age = 1 days
> mail_index_log_rotate_min_size = 300 k
> mail_location = maildir:~/Maildir
> namespace inbox {
>   inbox = yes
>   location = 
>   mailbox Drafts {
>     special_use = \Drafts
>   }
>   mailbox Junk {
>     special_use = \Junk
>   }
>   mailbox Sent {
>     special_use = \Sent
>   }
>   mailbox "Sent Messages" {
>     special_use = \Sent
>   }
>   mailbox Trash {
>     special_use = \Trash
>   }
>   prefix = 
> }
> passdb {
>   args = scheme=CRYPT username_format=%u /etc/dovecot/users
>   driver = passwd-file
> }
> protocols = imap
> service imap-login {
>   inet_listener imap {
>     address = *
>     port = 143
>   }
>   inet_listener imaps {
>     address = *
>     port = 943
>     ssl = yes
>   }
> }
> service imap {
>   process_limit = 25
> }
> ssl_cert = </etc/dovecot/private/dovecot.pem
> ssl_client_ca_dir = /etc/ssl/certs
> ssl_dh = # hidden, use -P to show it
> ssl_key = # hidden, use -P to show it
> userdb {
>   args = username_format=%u /etc/dovecot/users
>   driver = passwd-file
> }
> protocol lda {
>   postmaster_address = [elided]
> }
> protocol imap {
>   mail_max_userip_connections = 20
> }