Mail gone when recreating a folder in a second IMAP session
I'm running Dovecot 2.3.21 with the mdbox mailbox format. I lose mails (with an asterisk, see below) when I'm doing the following in 2 IMAP sessions:
Session 1: Create a folder, save a mail and select it:
. create folder . append folder {1} . select folder
- 1 EXISTS
- OK [UIDVALIDITY 1734013510] UIDs valid
Session 2: Delete the folder, create it again, save a mail in it, check the number of messages and the uidvalidity:.
. delete folder . create folder . append folder {1} . status folder (messages uidvalidity)
- STATUS folder (MESSAGES 1 UIDVALIDITY 1734013511)
Everything is as expected until here, the uidvalidity was increased by 1, which makes sense.
Session 1: Just do a noop
. noop
- BYE IMAP session state is inconsistent, please relogin.
Dovecot notices that something is wrong and closes the connection. This error is logged:
Error: Mailbox folder: Corrupted transaction log file /var/spool/dovecot/user/mailboxes/folder/dbox-Mails/dovecot.index.log seq 2: indexid changed: 1734011270 -> 1734011403 (sync_offset=0)
Session 2: Check the folder again:
. status folder (messages uidvalidity)
- STATUS folder (MESSAGES 0 UIDVALIDITY 1734013512)
The mail is gone, uidvalidity was again increased by 1, so this is a new folder.
I think the problem is that the folder's transaction log is deleted in session 1 in mail_transaction_log_file_open() when Dovecot notices discrepancies in what it expects:
if (ret == 0) { /* corrupted */ if (index->readonly) { /* don't delete */ } else { i_unlink_if_exists(file->filepath); } *reason_r = "File is corrupted"; return 0; }
This basically recreates the folder, so now it's empty.
Here's the asterisk: I know that technically the mail isn't lost because it's still in the storage file and can be recovered. But it's still ugly. And when I copy a mail from another folder instead of appending it, I can't recover it for the folder because the information that the mail was copied is definitely lost.
Can this be fixed in Dovecot?
On 12/12/2024 17:00 EET Ewald Dieterich via dovecot <dovecot@dovecot.org> wrote:
I'm running Dovecot 2.3.21 with the mdbox mailbox format. I lose mails (with an asterisk, see below) when I'm doing the following in 2 IMAP sessions:
Session 1: Create a folder, save a mail and select it:
. create folder . append folder {1} . select folder
- 1 EXISTS
- OK [UIDVALIDITY 1734013510] UIDs valid
Session 2: Delete the folder, create it again, save a mail in it, check the number of messages and the uidvalidity:.
. delete folder . create folder . append folder {1} . status folder (messages uidvalidity)
- STATUS folder (MESSAGES 1 UIDVALIDITY 1734013511)
Everything is as expected until here, the uidvalidity was increased by 1, which makes sense.
Session 1: Just do a noop
. noop
- BYE IMAP session state is inconsistent, please relogin.
Dovecot notices that something is wrong and closes the connection. This error is logged:
Error: Mailbox folder: Corrupted transaction log file /var/spool/dovecot/user/mailboxes/folder/dbox-Mails/dovecot.index.log seq 2: indexid changed: 1734011270 -> 1734011403 (sync_offset=0)
Session 2: Check the folder again:
. status folder (messages uidvalidity)
- STATUS folder (MESSAGES 0 UIDVALIDITY 1734013512)
The mail is gone, uidvalidity was again increased by 1, so this is a new folder.
I think the problem is that the folder's transaction log is deleted in session 1 in mail_transaction_log_file_open() when Dovecot notices discrepancies in what it expects:
if (ret == 0) { /* corrupted */ if (index->readonly) { /* don't delete */ } else { i_unlink_if_exists(file->filepath); } *reason_r = "File is corrupted"; return 0; }
This basically recreates the folder, so now it's empty.
Here's the asterisk: I know that technically the mail isn't lost because it's still in the storage file and can be recovered. But it's still ugly. And when I copy a mail from another folder instead of appending it, I can't recover it for the folder because the information that the mail was copied is definitely lost.
Can this be fixed in Dovecot?
Hi!
Thank you for your bug report, we'll take a look. Does the folder recover if you run doveadm force-resync?
Aki
On 13.12.24 13:11, Ewald Dieterich via dovecot wrote:
On 13.12.24 11:42, Aki Tuomi via dovecot wrote:
Thank you for your bug report, we'll take a look. Does the folder recover if you run doveadm force-resync?
Yes, I can recover the mail in the folder with doveadm force-resync.
That's when I append the mail. When I copy the mail from a different folder, it does not recover (of course the mail is still in the source folder).
On 12. Dec 2024, at 17.00, Ewald Dieterich via dovecot <dovecot@dovecot.org> wrote:
I'm running Dovecot 2.3.21 with the mdbox mailbox format. I lose mails (with an asterisk, see below) when I'm doing the following in 2 IMAP sessions:
Thanks, I knew there were these issues when a folder was being recreated with a same name, but I didn't think that it might end up losing emails. The fixes should appear in https://github.com/dovecot/core/ either today or next week, depending on how fast it goes through our review & tests:
Author: Timo Sirainen <timo.sirainen@open-xchange.com> Date: Fri Dec 13 12:26:56 2024 +0200
lib-index: Fix handling indexid change in dovecot.index file
Depending on the situation the dovecot.index should be either deleted or
the mailbox reopened.
Author: Timo Sirainen <timo.sirainen@open-xchange.com> Date: Fri Dec 13 12:26:27 2024 +0200
lib-index: Split off mail_transaction_log_has_changed()
Author: Timo Sirainen <timo.sirainen@open-xchange.com> Date: Fri Dec 13 10:53:16 2024 +0200
lib-index: Don't delete dovecot.index.log when finding that indexid changes
The indexes were recreated, so the new indexes should be used instead of
deleting them.
participants (3)
-
Aki Tuomi
-
Ewald Dieterich
-
Timo Sirainen