[Dovecot] Haha wow, thats freakin cool... mdbox
It didnt really click until I had a "ls -ald *" on watch while I moved messages around folders using mdbox... I moved about 200 first to an Archive folder, and scratched my head wondering why i didnt see any new m.* files being created after the move. So then I moved some 5k other messages from other folders to my archive folder...
Then it dawned on me, how damn cool, the data never changes, dovecot just updates its index of what folder the message is in?!
That is awesome!
Also, I ended up answering my own question on the 2m mdbox_rotate_size, if a 10 meg email comes through it will write a 10 meg file, so thats cool too.
Brandon Lamb put forth on 7/18/2010 2:22 AM:
It didnt really click until I had a "ls -ald *" on watch while I moved messages around folders using mdbox... I moved about 200 first to an Archive folder, and scratched my head wondering why i didnt see any new m.* files being created after the move. So then I moved some 5k other messages from other folders to my archive folder...
Then it dawned on me, how damn cool, the data never changes, dovecot just updates its index of what folder the message is in?!
That is awesome!
What happens when these indexes become corrupted, get accidentally deleted, or have the permissions accidentally changed in a way that they become unreadable by Dovecot? Does Dovecot still report the messages as being in the correct folder?
-- Stan
Stan Hoeppner wrote on 18.07.2010:
What happens when these indexes become corrupted, get accidentally deleted, or have the permissions accidentally changed in a way that they become unreadable by Dovecot? Does Dovecot still report the messages as being in the correct folder?
during my tests I've copied and deleted some mails, copied whole mdbox-files etc. and after some time the IMAP session was dropped by Dovecot. In the logfile I've found this:
Jul 18 17:36:18 tux dovecot: imap(testuser@dlutt.de): Error: Corrupted transaction log file /home/vmail/dlutt.de/testuser/mdbox/mailboxes/Papierkorb/dbox-Mails/dovecot.index.log seq 2: file size shrank (sync_offset=14724) Jul 18 17:36:18 tux dovecot: imap(testuser@dlutt.de): Error: Index /home/vmail/dlutt.de/testuser/mdbox/mailboxes/Papierkorb/dbox-Mails/dovecot.index: Lost log for seq=2 offset=14724 Jul 18 17:36:18 tux dovecot: imap(testuser@dlutt.de): Warning: fscking index file /home/vmail/dlutt.de/testuser/mdbox/mailboxes/Papierkorb/dbox-Mails/dovecot.index Jul 18 17:36:18 tux dovecot: imap(testuser@dlutt.de): Warning: mdbox /home/vmail/dlutt.de/testuser/mdbox/storage: rebuilding indexes Jul 18 17:36:18 tux dovecot: imap(testuser@dlutt.de): Error: /home/vmail/dlutt.de/testuser/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index reset, view is now inconsistent Jul 18 17:36:18 tux dovecot: imap(testuser@dlutt.de): Disconnected: IMAP session state is inconsistent, please relogin. bytes=119/850
After a relogin all mails were available and readable without manually fixing the index. Don't know if this always successful but it seems that Dovecot can rebuild the index without problems.
-- Daniel
Daniel Luttermann put forth on 7/18/2010 1:05 PM:
Stan Hoeppner wrote on 18.07.2010:
What happens when these indexes become corrupted, get accidentally deleted, or have the permissions accidentally changed in a way that they become unreadable by Dovecot? Does Dovecot still report the messages as being in the correct folder?
during my tests I've copied and deleted some mails, copied whole mdbox-files etc. and after some time the IMAP session was dropped by Dovecot. In the logfile I've found this:
Jul 18 17:36:18 tux dovecot: imap(testuser@dlutt.de): Error: Corrupted transaction log file /home/vmail/dlutt.de/testuser/mdbox/mailboxes/Papierkorb/dbox-Mails/dovecot.index.log seq 2: file size shrank (sync_offset=14724) Jul 18 17:36:18 tux dovecot: imap(testuser@dlutt.de): Error: Index /home/vmail/dlutt.de/testuser/mdbox/mailboxes/Papierkorb/dbox-Mails/dovecot.index: Lost log for seq=2 offset=14724 Jul 18 17:36:18 tux dovecot: imap(testuser@dlutt.de): Warning: fscking index file /home/vmail/dlutt.de/testuser/mdbox/mailboxes/Papierkorb/dbox-Mails/dovecot.index Jul 18 17:36:18 tux dovecot: imap(testuser@dlutt.de): Warning: mdbox /home/vmail/dlutt.de/testuser/mdbox/storage: rebuilding indexes Jul 18 17:36:18 tux dovecot: imap(testuser@dlutt.de): Error: /home/vmail/dlutt.de/testuser/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index reset, view is now inconsistent Jul 18 17:36:18 tux dovecot: imap(testuser@dlutt.de): Disconnected: IMAP session state is inconsistent, please relogin. bytes=119/850
After a relogin all mails were available and readable without manually fixing the index. Don't know if this always successful but it seems that Dovecot can rebuild the index without problems.
That's not the scenario I described. You manipulated mail files which Dovecot had already indexed, and Dovecot rebuilt the indexes when I saw the file locations had changed. The key here is that you manually, physically, moved files around to different directories. All Dovecot must do at that point is look at where those files now reside, and rebuild the index according to their current location.
What I was asking is what happens when you manually delete the index file out from underneath Dovecot? If files never got moved, but the index file says they're in dir "X" when they physically reside in dir "Y", how does Dovecot recover and correctly recreate the index? At this point Dovecot has no information about the files having been "moved" because the index file which contains this "Y -- X" mapping information is now gone, deleted.
-- Stan
On Sun, 2010-07-18 at 03:44 -0500, Stan Hoeppner wrote:
What happens when these indexes become corrupted, get accidentally deleted, or have the permissions accidentally changed in a way that they become unreadable by Dovecot? Does Dovecot still report the messages as being in the correct folder?
There are multiple layers of safety checks. Mailbox indexes are once in a while written to dovecot.index.backup files. The backup index is also used during index rebuild in case something in the main index is broken.
If mailbox indexes get entirely lost, and
a) mail doesn't exist elsewhere: the original mailbox where the message was saved to is kept in dbox file's metadata, and the message is placed to that mailbox.
b) mail already exists in another mailbox: it's simply left there. it's not copied to the original mailbox.
If there's a permission problem, I think Dovecot just logs errors and fails commands until the permissions are fixed.
participants (4)
-
Brandon Lamb
-
Daniel Luttermann
-
Stan Hoeppner
-
Timo Sirainen