[Dovecot] dovecot, mdbox and indexes

[William Blunn]

On 30/06/2011 13:28, Angel L. Mateo wrote:
>     We are now trying to migrate to the third scenario and using mdbox.
>
>     These errors cause a corrupted index and we have to manually 
> delete the user's index so he could read his mail again.
>
> * I connect then to a server that have no index for me. The only 
> problem is that the previously deleted mail appears again.
> * I connect then to a server with an outdated index: this is the 
> problem. I have only the messages in that outdated index. The mail 
> received since then has disappeared (although I could receive new mail).
>
>     How could I solve this problem? Is this really a problem? Should I 
> use a shared storage for indexes? What is the best configuration for a 
> high availability service?
>

!!!!! DANGER, DANGER !!!!!!

Under mdbox, the so-called "indexes" are not indexes in the traditional 
(database) sense of the word (a file maintained alongside the main data 
file which is used to speed up access to the main file, and can be 
regenerated at will), but rather are (binary) database files which 
contain crucial metadata which is not held anywhere else.

These filed cannot be regenerated at will.

For example:

The so-called "map index" (message store database) contains a (flat) 
list of all messages in a (user's) mailstore together with their 
reference counts, which storage file they're stored in.

The "folder index" (folder message metadata database) in each folder 
directory contains a list of the message numbers which are "in" that 
folder. Those message numbers identify a particular message in the 
message store database, which identify a particular m.* storage file, 
and where within that file the message is.

Lose these files, and you destroy the information about:
* which message is supposed to be in which folder,
* which messages are supposed to be deleted and which aren't, and
* all message keywords.

Under mdbox, losing these files buggerises the mailstore.

You may wish to read http://wiki2.dovecot.org/MailboxFormat/dbox

"... with dbox the Index files actually contain significant data which 
is held nowhere else. Index files for both *single-dbox* and 
*multi-dbox* contain message flags and keywords. For *multi-dbox*, the 
index file also contains the map_uids which link (via the "map index") 
to the actual message data. This data cannot be automatically recreated, 
so it is important that Index files are treated with the same care as 
message data files."

Regards,

Bill