Hi,
I've recently migrated my two VMs across from Linode (who use KVM) onto a local VPS service (which also uses KVM). Since doing so I have started to see some strange problems with Dovecot relating to indexes and replication.
I have copied the configuration files across from old host to new host. The kernel is the same - as this is Gentoo everything was rebuilt and installed from fresh, but with the same options (use flags). Even the Linux kernel is the same version with the exact same options (as is Dovecot). The filesystem is the same EXT4 with the same options too.
Here is what is logged:
Feb 18 15:20:41 tornado.reub.net dovecot: imap(reuben)<20031><d0yxIEPY3ROfxG3u>: Error: Mailbox INBOX: /home/reuben/Maildir/dovecot.index reset, view is now inconsistent Feb 18 15:20:41 tornado.reub.net dovecot: imap(reuben)<20031><d0yxIEPY3ROfxG3u>: Disconnected: IMAP session state is inconsistent, please relogin. in=146 out=1398 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
The trigger is the move/delete of an email. Just reading seems to be OK. But as soon as you move/delete an email this event occurs and is logged. The move fails on the first attempt (error is logged, client refreshes) but then if attempted again it succeeds.
This error causes the clients to be thrown off and the client view refreshed which seems to require a redownload of everything. It is extremely disruptive to client access.
The odd thing is that as soon as I disable replication by commenting out the mail_replica line, *all* of the index corruption messages and symptoms go away completely and I don't see this problem occur. As soon as I re-add it they come back. So it appears that replication at least is triggering the problem.
Replication is imap1 -> imap2 and on the other host imap2 - imap1
The odd thing is that this never was a problem while the system was on Linode.
I have removed all files on the remote and let dovecot re-replicate everything but this didn't help.
I can go back to the VPS host and talk to them about it as I have an open ticket but from their perspective it looks like an application level problem, so there's not a lot for them to go on especially given everything else is working OK. Could the underlying host be playing a role in this? It seems unlikely given turning replication or or off causes the problem to appear or disappear.
All clients are connecting to only the master of the two replication partners. Many have multiple concurrent sessions (eg my phone, and Thunderbird at the same time) but all are connecting to only one instance at a time. The second instance is only there for a failover scenario (update a DNS entry to failover).
There is no NFS or any sort of sharing set up on the VMs. They are dedicated virtual disks.
Can anyone give me any ideas how to troubleshoot this further or things to try?
Some system details:
# 2.3.18 (9dd8408c18): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.18 (0bc28b32) # OS: Linux 5.16.10-gentoo x86_64 Gentoo Base System release 2.8 # Hostname: tornado.reub.net auth_mechanisms = plain login auth_username_format = %Ln doveadm_password = # hidden, use -P to show it first_valid_uid = 1000 imap_client_workarounds = tb-lsub-flags tb-extra-mailbox-sep last_valid_uid = 1099 login_log_format_elements = user=<%u> auth-method=%m remote=%r local=%l %c %k mail_attribute_dict = file:%h/Maildir/dovecot-attributes mail_location = maildir:~/Maildir mail_plugins = notify replication managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = failure_show_msg=yes %s driver = pam } plugin { mail_replica = tcp:imap2.reub.net:4814 replication_full_sync_interval = 2 hours sieve = file:~/sieve;active=~/.dovecot.sieve } postmaster_address = postmaster@reub.net protocols = imap lmtp sieve submission sieve recipient_delimiter = - service aggregator { fifo_listener replication-notify-fifo { mode = 0666 user = root } unix_listener replication-notify { mode = 0666 user = root } } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { mode = 0777 } } service doveadm { inet_listener { address = 2404:9400:2264:7200::143 port = 4813 ssl = yes } inet_listener { address = 2404:9400:2264:7200::143 port = 4814 ssl = no } user = root } service imap-login { inet_listener imap { port = 143 } } service lmtp { inet_listener lmtp { address = ::1 port = 24 } unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } } service managesieve-login { inet_listener sieve { address = 127.0.0.1 port = 4190 } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } service submission-login { inet_listener submission { address = 103.4.234.81 2404:9400:2264:7200::587 port = 587 } } ssl_cert =
Thanks, Reuben