Hi Dovecot community,
I would like to share a problem I saw a few days ago. Any comment is highly appreciated.
Steps to reproduce
0. Version of Dovecot I use is v2.3.11.3.
1. Prepare 2 Dovecot backends:
- One backend is used as the source of dsync command. Other backend is used as the destination of the dsync command.
- Mailbox format is Maildir. Mail dir of two test accounts is empty.
2. Prepare the test account in the source of dsync:
- Send a test message to the test account. For example, I use swaks:
swaks --server 127.0.0.1 --port 24 --protocol LMTP --to user1@local
- Use doveadm expunge command to clean mail dir of the test account:
doveadm expunge -u user1@local mailbox INBOX all
3. Prepare the test account in the destination of dsync:
- Log in test account via IMAP then select INBOX first:
telnet 127.0.0.1 imap
...
s login user1@local 1
...
s select INBOX
...
s logout
doveadm -o imapc_host=backend2.local -o imapc_port=143 -o imapc_user=user1@local -o imapc_password=1 backup -s '' -R -u user1@local imapc:
- Confirm error occurs when using dsync:
dsync(user1@local): Info: imapc(backend2.local:143): Connected to 192.168.100.2:143 (local 192.168.100.27:58850)
dsync(user1@local): Error: Mailbox INBOX sync: mailbox_delete failed: INBOX can't be deleted.
AQAAAHm4+Jk=
4. Log in the test account in dsync destination then select INBOX. Repeat many times. I can see the size of the transaction log file is increasing continuously.
Before:
-rw-------. 1 vmail vmail 456 Apr 22 10:54 dovecot.index.log
After:
-rw-------. 1 vmail vmail 544 Apr 22 10:59 dovecot.index.log
-rw-------. 1 vmail vmail 632 Apr 22 11:00 dovecot.index.log
-rw-------. 1 vmail vmail 720 Apr 22 11:00 dovecot.index.log
Problems
- Size of the transaction log file is increasing continuously. Do you think this is an issue of Dovecot?
- The value of mail_index_log_rotate_max_size is 1M by default. However the transaction log file dovecot.index.log is not rotated when its size exceeds limit.
- Because the size of the transaction log file is too big and it is not rotated, I think "out of memory" error will occur when I try to use IMAP SELECT or POP3 STAT command (if the virtual size limit is small enough).
- If "size of the transaction log file is increasing continuously" is an issue, what should I do to avoid it or is there any workaround? I understand I should fix the error Error: Mailbox INBOX sync: mailbox_delete failed: INBOX can't be deleted first before doing anything next. I think removing the transaction log file is an understandable workaround.
Investigaton
2) Maildir + INBOX + backup/sync/replicate
In the test scenarios where the INBOX on one side was to be completely removed, e.g. doveadm backup -R the dsync failed and
nothing was synced to the target. This is because before moving the source mails to the mailbox, dsync cleans out the old ones (
-R preserves nothing) and in Maildir the INBOX can not be removed. This is a feature/not easily solvable, because in Maildir
INBOX is different from other folders.
- The transaction log file is rotated whenever there's any change in index file. I can confirm by sending a test message to the test account in dsync destination. I can see that logic in src/lib-index/mail-index-sync.c and src/lib-index/mail-index-write.c. After that, the size of the new transaction log file is not increasing anymore.
- When the error Error: Mailbox INBOX sync: mailbox_delete failed: INBOX can't be deleted occurred, I tried to dump the transaction log file and I saw these records:
record: offset=432, type=index-deleted (ext), size=12
record: offset=444, type=index-undeleted (ext), size=12
- Do you think these records relate to the issue that the size of the transaction log file is increasing continuously?
- Did the error Error: Mailbox INBOX sync: mailbox_delete failed: INBOX can't be deleted make content of transaction log file incomplete/incorrect?