Corrupted dbox file/index
Hi
Looking for some insight into mdbox index file management and recovery from corruptions.
I have a two node cluster on NFS with proxy director in front for user stickness. One node (a nominated master) bidirectionally replicates to a 3rd node on a DR site.
We periodically get index file corruptions resulting in rebuilds. However the user experience is poor as messages read/deleted from months/years ago all reappear as unread again.
We've seen corruption because of NFS NTP time sync problems, proxy not being stick, but also the DR node being off line for a while and then tripping corruption within production when it comes back on.
Error message example (1 of):
Error: Corrupted dbox file /mailshare/.. (removed) ../home/mail/storage/m.4 (around offset=993548): EOF reading msg header (got 0/30 bytes) https://wiki2.dovecot.org/MailboxFormat/dbox- i've read up on all the documentation I can find and understand " you must not lose the dbox index files, they can't be regenerated without data loss."
Questions: #1 Any additional tips for avoiding mdbox index corruptions with dsync? Or should I revert to maildir format? I like the performance premise of the mdbox but these index corruptions are a reliability issue.
#2 I'm guessing read status is one of the meta data items lost. But its seems it can't recover it from dovecot.index.backup files either. Any technique to preserve that item as its key to the user experience?
#3 If index/transaction logs are so critical is there some kind of check point backups I can take? Native dovecot feature or do I need to script something.
#4 I've noticed that rebuilding the index does not work if the dovecot.index.log file is lost (deleted as a hard test). The dovecot.index.cache can be but once the log file i gone messages are not automatically (or manually that i can find) recovered from the storage directory.
I've not seen any dovecot.index.log file corruptions but that file seems very high risk. If rebuilding the index only from the log file or a combination process from storage directory?
Is there perhaps an option to just use the transaction log and not the index? Although that doesn't sound wise for performance.
#5 In additional to status UNREAD we also notice files moved to the trash reappear. Is that expected behavior?
Thanks Raymond
On Mon, Oct 15, 2018 at 08:50:21AM +0000, Raymond Sellars wrote:
Hi
Looking for some insight into mdbox index file management and recovery from corruptions.
I have a two node cluster on NFS with proxy director in front for user stickness. One node (a nominated master) bidirectionally replicates to a 3rd node on a DR site.
We periodically get index file corruptions resulting in rebuilds. However the user experience is poor as messages read/deleted from months/years ago all reappear as unread again.
We've seen corruption because of NFS NTP time sync problems, proxy not being stick, but also the DR node being off line for a while and then tripping corruption within production when it comes back on.
Error message example (1 of):
Error: Corrupted dbox file /mailshare/.. (removed) ../home/mail/storage/m.4 (around offset=993548): EOF reading msg header (got 0/30 bytes) https://wiki2.dovecot.org/MailboxFormat/dbox- i've read up on all the documentation I can find and understand " you must not lose the dbox index files, they can't be regenerated without data loss."
Questions: #1 Any additional tips for avoiding mdbox index corruptions with dsync? Or should I revert to maildir format? I like the performance premise of the mdbox but these index corruptions are a reliability issue.
#2 I'm guessing read status is one of the meta data items lost. But its seems it can't recover it from dovecot.index.backup files either. Any technique to preserve that item as its key to the user experience?
#3 If index/transaction logs are so critical is there some kind of check point backups I can take? Native dovecot feature or do I need to script something.
#4 I've noticed that rebuilding the index does not work if the dovecot.index.log file is lost (deleted as a hard test). The dovecot.index.cache can be but once the log file i gone messages are not automatically (or manually that i can find) recovered from the storage directory.
I've not seen any dovecot.index.log file corruptions but that file seems very high risk. If rebuilding the index only from the log file or a combination process from storage directory?
Is there perhaps an option to just use the transaction log and not the index? Although that doesn't sound wise for performance.
#5 In additional to status UNREAD we also notice files moved to the trash reappear. Is that expected behavior?
Thanks Raymond
What version of dovecot and what OS are you running? Is NFS linux/bsd/netapp/etc?
participants (2)
-
Raymond Sellars
-
William Taylor