[Dovecot] dovecot, mdbox and indexes
Hello everybody,
We are updating our old dovecot 1.1 servers to 2.0. We have a farm os servers, with users in nfs filesystems and indexes in local fs. Mail is stored in maildir format. We don't warrant that a user is always directed to the same server, although our load balancer sends all request from one IP to the same server (during a session time with a inactivity timeout). This is the second scenario described at http://wiki2.dovecot.org/NFS.
We are now trying to migrate to the third scenario and using mdbox.
As an intermediate step, we have configured this second scenario, but with indexes in NFS filesystems. The configuration is:
mmap_disable = yes
dotlock_use_excl = no
mail_nfs_storage = yes
mail_nfs_index = yes
lock_method = fcntl (we have also tried with dotlock)
This configuration works without any problems with few users, but with high load, we have a lot of lock timeouts error like:
Jun 29 10:24:04 myotis34 dovecot: imap(mpm): Error: Timeout (180s) while waiting for lock for transaction log file /home/pas/01/014601/maildir_indexes/.INBOX/dovecot.index.log
These errors cause a corrupted index and we have to manually delete the user's index so he could read his mail again.
So we are planning to configure the third scenario. In this configuration, we don't have any problem during our tests if we use maildir format. But if we use mdbox one, we have problems when we have to change the final server for the server (We allways send the user to the same final server, but sometimes, if the server is down, for example, this is not possible). The tests we have done (with POP) are:
I connect to "my" server. There's no problem. I have my mail. I delete a mail.
I connect then to a server that have no index for me. The only problem is that the previously deleted mail appears again.
I connect then to a server with an outdated index: this is the problem. I have only the messages in that outdated index. The mail received since then has disappeared (although I could receive new mail).
How could I solve this problem? Is this really a problem? Should I use a shared storage for indexes? What is the best configuration for a high availability service?
-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337
On 30/06/2011 13:28, Angel L. Mateo wrote:
We are now trying to migrate to the third scenario and using mdbox. These errors cause a corrupted index and we have to manuallydelete the user's index so he could read his mail again.
I connect then to a server that have no index for me. The only problem is that the previously deleted mail appears again.
I connect then to a server with an outdated index: this is the problem. I have only the messages in that outdated index. The mail received since then has disappeared (although I could receive new mail).
How could I solve this problem? Is this really a problem? Should I use a shared storage for indexes? What is the best configuration for a high availability service?
!!!!! DANGER, DANGER !!!!!!
Under mdbox, the so-called "indexes" are not indexes in the traditional (database) sense of the word (a file maintained alongside the main data file which is used to speed up access to the main file, and can be regenerated at will), but rather are (binary) database files which contain crucial metadata which is not held anywhere else.
These filed cannot be regenerated at will.
For example:
The so-called "map index" (message store database) contains a (flat) list of all messages in a (user's) mailstore together with their reference counts, which storage file they're stored in.
The "folder index" (folder message metadata database) in each folder directory contains a list of the message numbers which are "in" that folder. Those message numbers identify a particular message in the message store database, which identify a particular m.* storage file, and where within that file the message is.
Lose these files, and you destroy the information about:
- which message is supposed to be in which folder,
- which messages are supposed to be deleted and which aren't, and
- all message keywords.
Under mdbox, losing these files buggerises the mailstore.
You may wish to read http://wiki2.dovecot.org/MailboxFormat/dbox
"... with dbox the Index files actually contain significant data which is held nowhere else. Index files for both *single-dbox* and *multi-dbox* contain message flags and keywords. For *multi-dbox*, the index file also contains the map_uids which link (via the "map index") to the actual message data. This data cannot be automatically recreated, so it is important that Index files are treated with the same care as message data files."
Regards,
Bill
El 30/06/11 15:53, William Blunn escribió:
On 30/06/2011 13:28, Angel L. Mateo wrote:
We are now trying to migrate to the third scenario and using mdbox.
These errors cause a corrupted index and we have to manually delete the user's index so he could read his mail again.
- I connect then to a server that have no index for me. The only problem is that the previously deleted mail appears again.
- I connect then to a server with an outdated index: this is the problem. I have only the messages in that outdated index. The mail received since then has disappeared (although I could receive new mail).
How could I solve this problem? Is this really a problem? Should I use a shared storage for indexes? What is the best configuration for a high availability service?
!!!!! DANGER, DANGER !!!!!!
Under mdbox, the so-called "indexes" are not indexes in the traditional (database) sense of the word (a file maintained alongside the main data file which is used to speed up access to the main file, and can be regenerated at will), but rather are (binary) database files which contain crucial metadata which is not held anywhere else.
These filed cannot be regenerated at will.
For example:
The so-called "map index" (message store database) contains a (flat) list of all messages in a (user's) mailstore together with their reference counts, which storage file they're stored in.
The "folder index" (folder message metadata database) in each folder directory contains a list of the message numbers which are "in" that folder. Those message numbers identify a particular message in the message store database, which identify a particular m.* storage file, and where within that file the message is.
Lose these files, and you destroy the information about:
- which message is supposed to be in which folder,
- which messages are supposed to be deleted and which aren't, and
- all message keywords.
Under mdbox, losing these files buggerises the mailstore.
You may wish to read http://wiki2.dovecot.org/MailboxFormat/dbox
"... with dbox the Index files actually contain significant data which is held nowhere else. Index files for both *single-dbox* and *multi-dbox* contain message flags and keywords. For *multi-dbox*, the index file also contains the map_uids which link (via the "map index") to the actual message data. This data cannot be automatically recreated, so it is important that Index files are treated with the same care as message data files."
I know this. That was my question. So, what is the best configuration for a high availabilty environment. An active/passive cluster?
participants (2)
-
Angel L. Mateo
-
William Blunn