[Dovecot] Update indexes with dovecot 1.1
Hello,
I have a farm of dovecot 1.1 servers (debian lenny). Mailboxes are in Maildir format.
Is there any way to manually update indexes?
Does it worth? I mean... Our problem is that mail is delivered via dovecot lda, but, because we have a farm of servers, we don't guarantee that mail is delivered through the same server that the user is using for his imap connection, so the imap server could have this index unupdated. So I want to force an update (or at least nearly updated) in order to have the index updated in all servers at the time the user get to work.
-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337
On Fri, 19 Aug 2011 11:13:21 +0200, "Angel L. Mateo" amateo@um.es wrote:
Is there any way to manually update indexes?
As far as I understood, you just have to delete the index files and Dovecot will regenerate them when a user logs in the next time.
Regards Patrick
El 19/08/11 12:15, Patrick Westenberg escribió:
On Fri, 19 Aug 2011 11:13:21 +0200, "Angel L. Mateo" amateo@um.es wrote:
Is there any way to manually update indexes?
As far as I understood, you just have to delete the index files and Dovecot will regenerate them when a user logs in the next time.
But I want the index to be generated (or updated) before the user login.-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337
On 19.8.2011, at 12.13, Angel L. Mateo wrote:
I have a farm of dovecot 1.1 servers (debian lenny). Mailboxes are in Maildir format.
Is there any way to manually update indexes?
v2.0 has "doveadm index" command to do this. There's no other good way to do this.
Does it worth? I mean... Our problem is that mail is delivered via dovecot lda, but, because we have a farm of servers, we don't guarantee that mail is delivered through the same server that the user is using for his imap connection, so the imap server could have this index unupdated. So I want to force an update (or at least nearly updated) in order to have the index updated in all servers at the time the user get to work.
Doing it via IMAP won't guarantee that everything the user's client needs is indexed/cached. Different clients need different things, Dovecot only indexes stuff that client requests. You'd have to look up from index files what the client wants indexed and then perform IMAP commands requesting those fields. Even the act of doing this may mess up caching decisions, because user may have changed client and now it's indexing unnecessary fields. (Actually now that I think of it, doveadm index has this same problem. Have to get that fixed.)
With v2.0 you could if you use Dovecot proxy (or director) you can also proxy doveadm connections through it, so a "doveadm index" would always go to the correct server. http://wiki2.dovecot.org/Director at the bottom has some info how to set this up (works also with plain proxy, without director).
El 23/08/11 04:40, Timo Sirainen escribió:
On 19.8.2011, at 12.13, Angel L. Mateo wrote:
I have a farm of dovecot 1.1 servers (debian lenny). Mailboxes are in Maildir format.
Is there any way to manually update indexes?
v2.0 has "doveadm index" command to do this. There's no other good way to do this.
Does it worth? I mean... Our problem is that mail is delivered via dovecot lda, but, because we have a farm of servers, we don't guarantee that mail is delivered through the same server that the user is using for his imap connection, so the imap server could have this index unupdated. So I want to force an update (or at least nearly updated) in order to have the index updated in all servers at the time the user get to work.
Doing it via IMAP won't guarantee that everything the user's client needs is indexed/cached. Different clients need different things, Dovecot only indexes stuff that client requests. You'd have to look up from index files what the client wants indexed and then perform IMAP commands requesting those fields. Even the act of doing this may mess up caching decisions, because user may have changed client and now it's indexing unnecessary fields. (Actually now that I think of it, doveadm index has this same problem. Have to get that fixed.)
With v2.0 you could if you use Dovecot proxy (or director) you can also proxy doveadm connections through it, so a "doveadm index" would always go to the correct server. http://wiki2.dovecot.org/Director at the bottom has some info how to set this up (works also with plain proxy, without director).
I'm trying this configuration in a test environment, but we are having lot of problems with director. The main problem is with director and LMTP, because it produces a lot timeout of errors (I have previouslly posted about these problems)
OK. So my question is, does it worth? Our scenario is 8 POP/IMAP servers with almost 70000 users (not all of them are really active), about 8.5 TB in use, with mailboxes in Maildir format over NFS. Our main problem with this is at return of vacations periods (like the one we'll have next 9/1). Our hypothesis is that the first connection of the user is expensive, because he has a lot of unindexed messages in his mailbox. Supposing that doveadm index indexes the mailbox correctly, does it helps to solve our problem?
-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337
On 23.8.2011, at 9.52, Angel L. Mateo wrote:
With v2.0 you could if you use Dovecot proxy (or director) you can also proxy doveadm connections through it, so a "doveadm index" would always go to the correct server. http://wiki2.dovecot.org/Director at the bottom has some info how to set this up (works also with plain proxy, without director). I'm trying this configuration in a test environment, but we are having lot of problems with director. The main problem is with director and LMTP, because it produces a lot timeout of errors (I have previouslly posted about these problems)
Yes, I should look into the LMTP proxy problems.. Those are kind of difficult to debug though since I've never been able to reproduce them. In any case, you could initially move to v2.0 + director without LMTP (i.e. deliver to Maildir directly, then run the doveadm index).
OK. So my question is, does it worth? Our scenario is 8 POP/IMAP servers with almost 70000 users (not all of them are really active), about 8.5 TB in use, with mailboxes in Maildir format over NFS. Our main problem with this is at return of vacations periods (like the one we'll have next 9/1). Our hypothesis is that the first connection of the user is expensive, because he has a lot of unindexed messages in his mailbox. Supposing that doveadm index indexes the mailbox correctly, does it helps to solve our problem?
Yes, if there's a ton of people returning at the same time it'll create a load spike. It's at least partially because mails aren't indexed, so Dovecot has to first read the message headers (and maybe bodies) to produce the initial message list, and afterwards when user actually reads/downloads the message bodies they're re-read from disk, unless the OS still has them cached.
So this kind of preindexing would definitely reduce the CPU load during the spike, but I'm not entirely sure about disk load because of the OS caching (10-50% decrease?). I'd be really interested in seeing actual numbers some day. :)
participants (3)
-
Angel L. Mateo
-
Patrick Westenberg
-
Timo Sirainen