Size of transaction log file is increasing continuously if dsync is failed

[Duc Anh Do]

Hi Dovecot community,

I would like to share a problem I saw a few days ago. Any comment is highly
appreciated.

*Steps to reproduce*
0. Version of Dovecot I use is v2.3.11.3.
1. Prepare 2 Dovecot backends:

   - One backend is used as the source of dsync command. Other backend is
   used as the destination of the dsync command.
   - Mailbox format is Maildir. Mail dir of two test accounts is empty.

2. Prepare the test account in the source of dsync:

   - Send a test message to the test account. For example, I use swaks:

swaks --server 127.0.0.1 --port 24 --protocol LMTP --to user1 at local

   - Use doveadm expunge command to clean mail dir of the test account:

doveadm expunge -u user1 at local mailbox INBOX all
3. Prepare the test account in the destination of dsync:

   - Log in test account via IMAP then select INBOX first:

telnet 127.0.0.1 imap
...
s login user1 at local 1
...
s select INBOX
...
s logout

   - Then perform dsync:

doveadm -o imapc_host=backend2.local -o imapc_port=143 -o
imapc_user=user1 at local -o imapc_password=1 backup -s '' -R -u user1 at local
imapc:

   - Confirm error occurs when using dsync:

dsync(user1 at local): Info: imapc(backend2.local:143): Connected to
192.168.100.2:143 (local 192.168.100.27:58850)
dsync(user1 at local): Error: Mailbox INBOX sync: mailbox_delete failed: INBOX
can't be deleted.
AQAAAHm4+Jk=
4. Log in the test account in dsync destination then select INBOX. Repeat
many times. I can see the size of the transaction log file is increasing
continuously.
*Before*:
-rw-------. 1 vmail vmail 456 Apr 22 10:54 dovecot.index.log

*After*:
-rw-------. 1 vmail vmail 544 Apr 22 10:59 dovecot.index.log

-rw-------. 1 vmail vmail 632 Apr 22 11:00 dovecot.index.log

-rw-------. 1 vmail vmail 720 Apr 22 11:00 dovecot.index.log

*Problems*

   - Size of the transaction log file is increasing continuously. Do you
   think this is an issue of Dovecot?
   - The value of *mail_index_log_rotate_max_size* is 1M by default.
   However the transaction log file dovecot.index.log is not rotated when its
   size exceeds limit.
   - Because the size of the transaction log file is too big and it is not
   rotated, I think "out of memory" error will occur when I try to use IMAP
   SELECT or POP3 STAT command (if the virtual size limit is small enough).
   - If "size of the transaction log file is increasing continuously" is an
   issue, what should I do to avoid it or is there any workaround? I
   understand I should fix the error *Error: Mailbox INBOX sync:
   mailbox_delete failed: INBOX can't be deleted* first before doing
   anything next. I think removing the transaction log file is an
   understandable workaround.

*Investigaton*

   - About the error *Error: Mailbox INBOX sync: mailbox_delete failed:
   INBOX can't be deleted*, I can understand the cause after reading this
   thread https://dovecot.org/pipermail/dovecot/2014-April/095588.html

2) Maildir + INBOX + backup/sync/replicate
> In the test scenarios where the INBOX on one side was to be completely
> removed, e.g. doveadm backup -R the dsync failed and
> nothing was synced to the target. This is because before moving the source
> mails to the mailbox, dsync cleans out the old ones (
> -R preserves nothing) and in Maildir the INBOX can not be removed. This is
> a feature/not easily solvable, because in Maildir
> INBOX is different from other folders.
>

   - The transaction log file is rotated whenever there's any change in
   index file. I can confirm by sending a test message to the test account in
   dsync destination. I can see that logic in
   *src/lib-index/mail-index-sync.c* and *src/lib-index/mail-index-write.c*.
   After that, the size of the new transaction log file is not increasing
   anymore.


   - When the error *Error: Mailbox INBOX sync: mailbox_delete failed:
   INBOX can't be deleted* occurred, I tried to dump the transaction log
   file and I saw these records:

record: offset=432, type=index-deleted (ext), size=12
record: offset=444, type=index-undeleted (ext), size=12

   - Do you think these records relate to the issue that the size of the
   transaction log file is increasing continuously?
   - Did the error *Error: Mailbox INBOX sync: mailbox_delete failed: INBOX
   can't be deleted* make content of transaction log file
   incomplete/incorrect?


Regards,
Anh Do
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20210422/ae5c4d65/attachment-0001.html>