[Dovecot] How to propelry restore a Maildir
Hello,
I would like to know what is the proper way to restore a Maildir :
Assuming someone has got a foobar/[cur|new|tmp] maildir, loses for some reason its content and that some new mail (seen and/or unseen) comes into that mailbox inbetween (that is before the restore procedure takes place, thus creating the need to merge restored messages and new ones).
I'd like to check the following (I don't use deliver but procmail) :
the messages filenames are unique in time (meaning that a new message cannot be assigned a name already existing in the backup (the name of an old message), at least for the same mailbox instance : if that's not the case : how can we merge safely ?
what naming scheme is used (the left part of the name, before the :2,) ?
I've got something like :
1215166123.52887_0.host.dom.ain:2,[...]
what are the 1215166123, the 52887 and 0 ?
what can we deduce from the last modified date of the file as seen in ls -l ?
. when in new/, is it the same as the one in the Received: header or is it when in tmp/ ?
. when in cur/, is it the date the client "saw" the message ? What would "saw" mean in that case ?
should the dovecot-uidlist and dovecot-keywords be updated after the restore or is it better to let dovecot rearrange them itself ?
same question for the index and cache files ?
Thanks
-- Thomas Hummel | Institut Pasteur hummel@pasteur.fr | Pôle informatique - systèmes et réseau
On Tue, 2008-09-09 at 14:16 +0200, Thomas Hummel wrote:
Assuming someone has got a foobar/[cur|new|tmp] maildir, loses for some reason its content and that some new mail (seen and/or unseen) comes into that mailbox inbetween (that is before the restore procedure takes place, thus creating the need to merge restored messages and new ones).
Safest would be to create "restored mails" mailbox and put everything there and let the user figure out how to merge things. Might be annoying enough to merge them so that they'll learn not to make mistakes. :)
I'd like to check the following (I don't use deliver but procmail) :
- the messages filenames are unique in time (meaning that a new message cannot be assigned a name already existing in the backup (the name of an old message), at least for the same mailbox instance : if that's not the case : how can we merge safely ?
I'm not really sure what you're asking. The file names are unique and from that point you should have no problems just keeping them. But Dovecot doesn't really like if messages get "unexpunged", so if it notices that (due to keeping old records in dovecot-uidlist) it'll log some warnings and give the messages new UIDs.
Safest would be to rename the files. Maybe just place an extra letter somewhere in the base file name.
- what naming scheme is used (the left part of the name, before the :2,) ?
You shouldn't need to care.
I've got something like : 1215166123.52887_0.host.dom.ain:2,[...] what are the 1215166123, the 52887 and 0 ?
UNIX timestamp (seconds since 1970), PID and delivery counter. But none of that should matter to you. The file name just has to be unique, it doesn't matter what it contains.
what can we deduce from the last modified date of the file as seen in ls -l ?
. when in new/, is it the same as the one in the Received: header or is it when in tmp/ ?
. when in cur/, is it the date the client "saw" the message ? What would "saw" mean in that case ?
The file's mtime is the same as its IMAP INTERNALDATE, which usually means the time when the message was delivered (as by time() call, not by looking at any headers) or if the message was saved via IMAP APPEND command the timestamp could have also been given as a parameter. In any case it must never change.
should the dovecot-uidlist and dovecot-keywords be updated after the restore or is it better to let dovecot rearrange them itself ?
same question for the index and cache files ?
dovecot-keywords could be a problem if the restored mail used keywords, although it currently isn't a problem since keywords never get removed so the existing dovecot-keywords file produces correct results. That might change some day though.
As for other files, don't worry about them.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Timo Sirainen said the following on 09/09/08 16:54:
Safest would be to create "restored mails" mailbox and put everything there and let the user figure out how to merge things. Might be annoying enough to merge them so that they'll learn not to make mistakes. :)
That's nearly what I do when I restore mailboxes in maildir.
I restore one or more mailbox and I rename them appending a '_restore' to their names and let the user do the dirty work. There is only one problem: often users keep both directories :)
Ciao, luigi
/ +--[Luigi Rosa]-- \
If you tell the boss you were late for work because you had a flat tire, the next morning you will have a flat tire. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFIxo7x3kWu7Tfl6ZQRApVpAJ9Lj8Owys6uuK5u4fVI1jbx8LbgIgCeIRzO ALvseZg+0exTmrmV2nK1wPo= =93OZ -----END PGP SIGNATURE-----
On Tue, Sep 09, 2008 at 05:54:17PM +0300, Timo Sirainen wrote:
I'm not really sure what you're asking. The file names are unique and
let's say someone has a message named
1215166123.52887_0.host.dom.ain:2,
in his mailbox. That he accidentally deletes this mailbox (but it has been backed up somewhere by the administrator or some snapshot mechanisms). Then he receives a new mail.
Can this new mail be named identically, i.e. 1215166123.52887_0.host.dom.ain:2, ? if so, it would be a problem when merging.
UNIX timestamp (seconds since 1970), PID and delivery counter.
I guess that would be answer to the worry expressed above.
Dovecot doesn't really like if messages get "unexpunged"
Why would messages be considered unexpunged (that would concern only messages which were previously tagged as "Deleted", right ?) ? Or do you mean that the index/cache mechanism would notice that messages have disappeared ?
dovecot-keywords could be a problem if the restored mail used keywords, although it currently isn't a problem since keywords never get removed so the existing dovecot-keywords file produces correct results. That might change some day though.
You mean that the following sequence cannot happen since keywords in dovecot-keywords never get deleted :
. message M filename contain keywork b . message M gets deleted accidentaly . keyword b gets deleted from dovecot-keywords . message M is restored but the b keyword isn't known anymore
?
So, apart from this keyword and unexpunged issues (the first one currently working and the second one having the "add a letter" workaround"), nothing prevents to merge (by simply copying messages in cur/) restored and new messages for the user (in the case it wouldn't be an option to let the user do the dirty job ;-)) ?
Thanks.
-- Thomas Hummel | Institut Pasteur hummel@pasteur.fr | Pôle informatique - systèmes et réseau
On Tue, 2008-09-09 at 17:35 +0200, Thomas Hummel wrote:
Can this new mail be named identically, i.e. 1215166123.52887_0.host.dom.ain:2, ? if so, it would be a problem when merging.
Yes, except for Dovecot's potential warning:
Dovecot doesn't really like if messages get "unexpunged"
Why would messages be considered unexpunged (that would concern only messages which were previously tagged as "Deleted", right ?) ? Or do you mean that the index/cache mechanism would notice that messages have disappeared ?
dovecot-uidlist is recreated lazily. So it may list files that have already been deleted. So when Dovecot notices that there exists a new file that's already listed in dovecot-uidlist but that has been expunged from Dovecot's index files, it'll figure out the message is unexpunged and to avoid breaking IMAP clients it'll log a warning and give a new UID to the file.
dovecot-keywords could be a problem if the restored mail used keywords, although it currently isn't a problem since keywords never get removed so the existing dovecot-keywords file produces correct results. That might change some day though.
You mean that the following sequence cannot happen since keywords in dovecot-keywords never get deleted :
. message M filename contain keywork b . message M gets deleted accidentaly . keyword b gets deleted from dovecot-keywords . message M is restored but the b keyword isn't known anymore
?
Right. Keywords can only be added currently.
So, apart from this keyword and unexpunged issues (the first one currently working and the second one having the "add a letter" workaround"), nothing prevents to merge (by simply copying messages in cur/) restored and new messages for the user (in the case it wouldn't be an option to let the user do the dirty job ;-)) ?
Right.
On Tue, Sep 09, 2008 at 06:41:34PM +0300, Timo Sirainen wrote:
On Tue, 2008-09-09 at 17:35 +0200, Thomas Hummel wrote:
Can this new mail be named identically, i.e. 1215166123.52887_0.host.dom.ain:2, ? if so, it would be a problem when merging.
Yes, except for Dovecot's potential warning:
Hmm you mean no ? Can a new message filen the same name as an old message ?
dovecot-uidlist is recreated lazily. So it may list files that have already been deleted. So when Dovecot notices that there exists a new file that's already listed in dovecot-uidlist but that has been expunged from Dovecot's index files, it'll figure out the message is unexpunged and to avoid breaking IMAP clients it'll log a warning and give a new UID to the file.
You mean when a client deletes a message, it is first deleted from index, and then deleted from dovecot-uidlist only if it reappears ? So dovecot-uidlist would grow infinitely ? Or maybe it is cleaned when UIDs have to be rearranged ?
-- Thomas Hummel | Institut Pasteur hummel@pasteur.fr | Pôle informatique - systèmes et réseau
On Sep 9, 2008, at 6:56 PM, Thomas Hummel wrote:
On Tue, Sep 09, 2008 at 06:41:34PM +0300, Timo Sirainen wrote:
On Tue, 2008-09-09 at 17:35 +0200, Thomas Hummel wrote:
Can this new mail be named identically, i.e.
1215166123.52887_0.host.dom.ain:2, ? if so, it would be a problem when merging.Yes, except for Dovecot's potential warning:
Hmm you mean no ? Can a new message filen the same name as an old
message ?
Right. I meant file maildir filenames are always globally unique when
they're created.
dovecot-uidlist is recreated lazily. So it may list files that have already been deleted. So when Dovecot notices that there exists a new file that's already listed in dovecot-uidlist but that has been
expunged from Dovecot's index files, it'll figure out the message is
unexpunged and to avoid breaking IMAP clients it'll log a warning and give a new UID to the file.You mean when a client deletes a message, it is first deleted from
index, and then deleted from dovecot-uidlist only if it reappears ? So dovecot- uidlist would grow infinitely ?
In v1.1 dovecot-uidlist gets recreated (causing deleted records to be
dropped) when more than 25% of it contains deleted records.
Or maybe it is cleaned when UIDs have to be rearranged ?
This should never need to happen in normal operation.
On Tue, Sep 09, 2008 at 07:01:43PM +0300, Timo Sirainen wrote:
Thanks. That's clear now. Except for the unexpunged situation :
when excactly does dovecot notice that a (restored) message reappears ?
You said the message is removed from the index but not (because of lazy expunge) yet from dovecot.uidlist. But what if the index is destroyed for some reason : would that mean that the restoration would be unoticed by dovecot ?
Thanks
-- Thomas Hummel | Institut Pasteur hummel@pasteur.fr | Pôle informatique - systèmes et réseau
On Wed, 2008-09-10 at 14:49 +0200, Thomas Hummel wrote:
On Tue, Sep 09, 2008 at 07:01:43PM +0300, Timo Sirainen wrote:
Thanks. That's clear now. Except for the unexpunged situation :
when excactly does dovecot notice that a (restored) message reappears ?
You said the message is removed from the index but not (because of lazy expunge) yet from dovecot.uidlist.
Right.
But what if the index is destroyed for some reason : would that mean that the restoration would be unoticed by dovecot ?
Yes, Dovecot would happily reassign the old UID for the mail then and possibly cause problems to clients. Unless it reappears while Dovecot is running and the indexes were alive in memory during the whole time, then it would notice it.
participants (3)
-
Luigi Rosa
-
Thomas Hummel
-
Timo Sirainen