[Dovecot] new configure-option --disable-index
Hello,
I experimented with dovecot the whole evening yesterday and the mailserver worked fine since there haven't been that much users connecting to it. I'm using dovecot 1.0.alpha5 as pop3 and pop3s server. The maildirs are mounted through NFS and since there is no nfslockd running I disabled mmap in the config and told dovecot to use dotlock. Also I experienced, that dovecot was much faster when using one login-process per connection instead of sharing them throughout several connections. Sharings seems to be a greater performance problem (bottleneck) than additional fork()s.
However, under normal load at business time the server load increased to a value that made it impossible to download emails without timeouts, so we switched back to tcpserver/qmail-pop3d for pop3 and only kept dovecot for doing the pop3s part.
Dovecot is really fine, but it still isn't NFS ready. Especially when using dovecot as pop3 only this index-stuff must be disengageable. What is the use of generating a complex index file for hundreds of messages, that will be deleted in the next step after downloading them? But the index is also a problem when using IMAP over NFS. Access to the files is too slow, so the connections keep hanging around and the i/o waiting value is raised to a maximum which slows down the whole machine. It simply doesn't make sense to build and update an index every time the maildir is accessed (which is very cpu and timeconsuming), while search operations (the only moment when the index is actually used and not just heavily updated) don't affect even 1% of all IMAP calls (and such a command doesn't even exist for pop3).
I would be much obliged if you could create an option called "--disable-index" in the configure-script or something like "disable-index = yes" in dovecot.conf that makes dovecot ignoring existing index files and doesn't care for related operations, so NFS should work much faster and IMAP search would simply parse the files directly instead of looking in the index.
I would really appriciate this very much and I'm absolutely sure, that a lot of people beside me are waiting for this.
Regards Marten
On 2006-01-13 18:04:15 +0100, Marten Lehmann wrote:
I experimented with dovecot the whole evening yesterday and the mailserver worked fine since there haven't been that much users connecting to it. I'm using dovecot 1.0.alpha5 as pop3 and pop3s server. The maildirs are mounted through NFS and since there is no nfslockd running I disabled mmap in the config and told dovecot to use dotlock. Also I experienced, that dovecot was much faster when using one login-process per connection instead of sharing them throughout several connections. Sharings seems to be a greater performance problem (bottleneck) than additional fork()s.
However, under normal load at business time the server load increased to a value that made it impossible to download emails without timeouts, so we switched back to tcpserver/qmail-pop3d for pop3 and only kept dovecot for doing the pop3s part.
can we have some numbers? how many concurrent connections? how did you compile dovecot? do you use epoll/inotify?
Dovecot is really fine, but it still isn't NFS ready. Especially when using dovecot as pop3 only this index-stuff must be disengageable. What is the use of generating a complex index file for hundreds of messages, that will be deleted in the next step after downloading them? But the index is also a problem when using IMAP over NFS. Access to the files is too slow, so the connections keep hanging around and the i/o waiting value is raised to a maximum which slows down the whole machine. It simply doesn't make sense to build and update an index every time the maildir is accessed (which is very cpu and timeconsuming), while search operations (the only moment when the index is actually used and not just heavily updated) don't affect even 1% of all IMAP calls (and such a command doesn't even exist for pop3).
for me the index even helps for just opening the folder via imap. having an up2date index speeds up reading the folder via imap drastically.
did you try to store the indexes locally on the disk? you can configure that at runtime.
I would be much obliged if you could create an option called "--disable-index" in the configure-script or something like "disable-index = yes" in dovecot.conf that makes dovecot ignoring existing index files and doesn't care for related operations, so NFS should work much faster and IMAP search would simply parse the files directly instead of looking in the index.
I would really appriciate this very much and I'm absolutely sure, that a lot of people beside me are waiting for this.
hope this helps.
darixHello,
can we have some numbers?
between 09:00 and 09:59 there were 24586 logins, between 10:00 and 10:59 there were 26296.
how many concurrent connections?
Its to laborious to check in detail, but with 7.3 logins per seconds and an average download time of at least 20-30 seconds (usually faster, but the system slowed down) there are a lot and all of them are waiting for NFS-calls to finish.
how did you compile dovecot?
I compiled with:
./configure
--prefix=/pop3/dovecot-%{version}
--with-rundir=/pop3/dovecot-%{version}/var
--disable-ipv6
--without-bsdauth
--without-checkpassword
--without-ldap
--without-pam
--without-passwd
--without-prefetch-userdb
--without-shadow
--without-static-userdb
--without-vpopmail
--with-ssl=openssl
--with-storages=maildir
do you use epoll/inotify?
I guess now (unless its default), but I have never heard of.
for me the index even helps for just opening the folder via imap.
Yes, for IMAP it may be fine. But I'm talking about a plain POP3-dovecot where an index is not required.
having an up2date index speeds up reading the folder via imap drastically.
I'm not sure about this using NFS.
did you try to store the indexes locally on the disk? you can configure that at runtime.
Why do I have to store indexes at all? I don't want a second folder for all the indices. And after all, they wouldn't make sense, because more than one pop3 server could be accessing the NFS-mailboxes at the same time (thats what the outsourcing of the mailboxes to an NFS-share was ment for).
Regards Marten
Marten Lehmann wrote:
I would be much obliged if you could create an option called "--disable-index" in the configure-script or something like "disable-index = yes" in dovecot.conf that makes dovecot ignoring existing index files and doesn't care for related operations, so NFS should work much faster and IMAP search would simply parse the files directly instead of looking in the index.
Either store indexes on a local disk or use nonpersistent memory indexes:
default_mail_env = maildir:/nfs/%d/%u/Maildir:INDEX=MEMORY
Tomi
Hello,
Either store indexes on a local disk or use nonpersistent memory indexes:
default_mail_env = maildir:/nfs/%d/%u/Maildir:INDEX=MEMORY
my default_mail_env is more complex, so its read from a file by passwd and userdb. How can I define the default_mail_env (or at least the index=memory thing) without changing the maildir?
Regards Marten
On Mon, 16 Jan 2006, Marten Lehmann wrote:
Hello,
Either store indexes on a local disk or use nonpersistent memory indexes:
default_mail_env = maildir:/nfs/%d/%u/Maildir:INDEX=MEMORY
my default_mail_env is more complex, so its read from a file by passwd and userdb. How can I define the default_mail_env (or at least the index=memory thing) without changing the maildir?
Hmm, I'm using the storage directory for other processes as well, hence, the Dovecot conf string is not suitable for the "general" maildir attribute (or field or whatever your DB uses as a term for it).
I'm using two attributes, one for Dovecot and one for the others. This, however, has the drawback:
- to manage redundant data (OK, I don't remember changing the mailstorage location manually never),
- Dovecot's string not only specifies the location, but the type as well (maildir, mbox), but this problem you have with other environments as well.
As "default_mail_env" is fixed for all users, I'm not understanding why adding ":INDEX=MEMORY" to whereever its read from would be problematic.
Bye,
-- Steffen Kaiser
Hello,
As "default_mail_env" is fixed for all users, I'm not understanding why adding ":INDEX=MEMORY" to whereever its read from would be problematic.
I don't have a default_mail_env setting, because there is no default. I now tried to set "default_mail_env = INDEX=MEMORY", but this gives me the following in the error-log:
Error: pop3(imaptest@test.de): Ambiguous mail location setting, don't know what to do with it: INDEX=MEMORY (try prefixing it with mbox: or maildir:) Error: pop3(imaptest@test.de): Failed to create storage with data: INDEX=MEMORY
How can I simply set INDEX=MEMORY without defining a default?
Regards Marten
Hello,
The default is autodetection.
I don't think that dovecot will be able to find my mailboxes without giving it a hint, because the location is neither /var/spool/mail/%u nor ~/Maildir :-)
Maybe there's still a way to do this. But you have to tell a little more about your config and post your auth section from dovecot.conf.
It's pretty short:
disable_plaintext_auth = no auth_verbose = yes auth default { mechanisms = plain passdb passwd-file { args = /pop3/conf/pop3-auth.txt }
userdb passwd-file { args = /pop3/conf/pop3-auth.txt } }
On Tue, 17 Jan 2006, Marten Lehmann wrote:
Hello,
The default is autodetection.
I don't think that dovecot will be able to find my mailboxes without giving it a hint, because the location is neither /var/spool/mail/%u nor ~/Maildir :-)
Maybe there's still a way to do this. But you have to tell a little more about your config and post your auth section from dovecot.conf.
It's pretty short:
disable_plaintext_auth = no auth_verbose = yes auth default { mechanisms = plain passdb passwd-file { args = /pop3/conf/pop3-auth.txt }
userdb passwd-file { args = /pop3/conf/pop3-auth.txt } }
First I guess there is a problem with the default: yes vs. no It is not sufficient to change the default mailbox setting, because this setting is used when the userdb does not specify a location. When the majority of your users is reachable with the same template, e.g.:
maildir:/mnt/hd%1Ri/%Ln/%Us:INDEX=MEMORY
(see dovecot_root/doc/variables.txt about the percent variables)
Otherwise, you must appended ":INDEX=MEMORY" to all settings in your userdb, e.g. my default:
default_mail_env = maildir:%h/MailDir:INDEX=/var/spool/dovecot/%i:CONTROL=/var/spool/dovecot/%i
then I maintain a LDAP attribute named "mailDovecot" for those users with a different mailbox, e.g.:
mailDovecot: maildir:/mailcache/%Ln:INDEX=/var/spool/dovecot/%i:CONTROL=/var/spool/dovecot/%i
You see: the "extra" settings must be specified for each entry.
==
Hmm, maybe I grasp your problem now: You use a passwd file, which uses the colon as delimiter for its fields itself, right?
However, I never used passwd, but it's limited to uid/pwd/homedir anyway, right? So you ought to have some fixed template to reach your mailboxes relative to the user's homedirectory, which you can put into default_mail_env and add the ":INDEX=MEMORY" there.
Bye,
-- Steffen Kaiser
Hello,
Hmm, maybe I grasp your problem now: You use a passwd file, which uses the colon as delimiter for its fields itself, right?
thats correct.
However, I never used passwd, but it's limited to uid/pwd/homedir anyway, right? So you ought to have some fixed template to reach your mailboxes relative to the user's homedirectory, which you can put into default_mail_env and add the ":INDEX=MEMORY" there.
As I said before: The Maildir-folder is more complex.
Let's say we have a mailbox called try@test.de, the it's maildir-location would be /pop3/mailboxes/t/te/test.de/_/try/Maildir.
If it would be a subdomain of test.de, lets says try@subdomain.test.de, the maildir-location would be /pop3/mailboxes/t/te/test.de/subdomain/try/Maildir.
dovecot is in no circumstances able to see whats a subdomain and whats the domain. You can't split at the dots, because errors would occur for third-level domains like test.co.uk. So the entry of userdb must be used.
Regards Marten
Marten Lehmann wrote:
The default is autodetection. I don't think that dovecot will be able to find my mailboxes without giving it a hint, because the location is neither /var/spool/mail/%u nor ~/Maildir :-)
You have to give it some hint where they are, of course, e.g. the home directory.
args = /pop3/conf/pop3-auth.txt
What does the file look like (an example line)? And what is the path to your mailboxes, mbox or maildir, ...? Please be a little more verbose.
Marten Lehmann wrote:
You have to give it some hint where they are, of course, e.g. the home directory. is it possible the define default_mail_env so that the homedir given in the userdb is used, e.g.
of course. I have
default_mail_env = maildir:%h/./Maildir/
It's listed in the example config, right above default_mail_env...
On Wed, 18 Jan 2006, Marten Lehmann wrote:
Hello,
Let's put together some posts:
Date: Wed, 18 Jan 2006 15:10:00 +0100 From: Marten Lehmann lehmann@cnm.de Subject: Re: [Dovecot] new configure-option --disable-index
However, I never used passwd, but it's limited to uid/pwd/homedir anyway, right? So you ought to have some fixed template to reach your mailboxes relative to the user's homedirectory, which you can put into default_mail_env and add the ":INDEX=MEMORY" there.
As I said before: The Maildir-folder is more complex.
Let's say we have a mailbox called try@test.de, the it's maildir-location would be /pop3/mailboxes/t/te/test.de/_/try/Maildir.
If it would be a subdomain of test.de, lets says try@subdomain.test.de, the maildir-location would be /pop3/mailboxes/t/te/test.de/subdomain/try/Maildir.
Let me guess another time:
your passwd-file entry looks like so: try:pwd:uid:gid:gecos:/pop3/mailboxes/t/te/test.de/subdomain/try:/bin/false
As Dovecot is concerned the homedirectory of "try" is: /pop3/mailboxes/t/te/test.de/subdomain/try
Date: Wed, 18 Jan 2006 14:56:33 +0100 (CET) From: Jakob Hirsch jh@plonk.de Reply-To: "dovecot@dovecot.org" dovecot@dovecot.org Subject: Re: [Dovecot] new configure-option --disable-index
Marten Lehmann wrote:
The default is autodetection. I don't think that dovecot will be able to find my mailboxes without giving it a hint, because the location is neither /var/spool/mail/%u nor ~/Maildir :-)
~/MailDir would expand to /pop3/mailboxes/t/te/test.de/subdomain/try/MaiDir, because that's try's home directory.
You have to give it some hint where they are, of course, e.g. the home directory.
is it possible the define default_mail_env so that the homedir given in the userdb is used, e.g.
default_mail_env = maildir:$HOME:INDEX=MEMORY
That's a question, correct? :-)
default_mail_env = maildir:%h/MailDir:INDEX=MEMORY -or even, if you don't use the home directory for other stuff than email- default_mail_env = maildir:%h:INDEX=MEMORY
So: as long as your userdb always returns the location of the mail storage location as home directory, you use a fixed template for the mail folders and, hence, can use default_mail_env
Bye,
-- Steffen Kaiser
Hello,
your passwd-file entry looks like so: try:pwd:uid:gid:gecos:/pop3/mailboxes/t/te/test.de/subdomain/try:/bin/false
yes, similar.
default_mail_env = maildir:%h/MailDir:INDEX=MEMORY
Thats what I used now, because the homedirs in the userdb don't end with "/Maildir".
But now the process dies after the login:
dovecot: Jan 18 17:07:34 Info: Dovecot v1.0.alpha5 starting up dovecot: Jan 18 17:07:43 Error: pop3(try@test.de): file mail-index-lock.c: line 247 (mail_index_copy): assertion failed: (!MAIL_INDEX_IS_IN_MEMORY(index)) dovecot: Jan 18 17:07:43 Info: pop3-login: Login: user=try@test.de, method=PLAIN, rip=192.168.1.72, lip=192.168.1.72 dovecot: Jan 18 17:07:43 Error: child 16566 (pop3) killed with signal 6
And in the maildir I file named dovecot-uidlist.lock is left. Any ideas?
Regards Marten
Hello,
any idea, why the process dies?
I have exactly this env in dovecot.conf
default_mail_env = maildir:%h/Maildir:INDEX=MEMORY
And this is the line in the userdb/passwd
try@test.de:$1$tg$ajh.Y16qF15EQXSX/px1k.:99:99::/pop3/mailboxes/t/te/test.de/_/try::
Regards Marten
On Thu, 19 Jan 2006, Marten Lehmann wrote:
Hello,
any idea, why the process dies?
I have exactly this env in dovecot.conf
default_mail_env = maildir:%h/Maildir:INDEX=MEMORY
And this is the line in the userdb/passwd
try@test.de:$1$tg$ajh.Y16qF15EQXSX/px1k.:99:99::/pop3/mailboxes/t/te/test.de/_/try::
I'm tested it in Beta1, but had no problem with it.
Bye,
-- Steffen Kaiser
Hello,
I'm tested it in Beta1, but had no problem with it.
I don't know whats going wrong, but it happened with beta1, too. I enabled all debugging options and this is what the logfile says:
dovecot: Jan 20 18:26:07 Info: Dovecot v1.0.beta1 starting up dovecot: Jan 20 18:26:08 Info: auth(default): passwd-file /mailin/mailboxes/conf/pop3-auth.txt: Read 3 users dovecot: Jan 20 18:26:47 Info: auth(default): client in: AUTH 1 PLAIN service=POP3 secured lip=192.168.33.72 rip=192.168.33.72 resp=AGltYXB0ZXN0QHZhcmlvbWVkaWEuZGUAdGVzdA== dovecot: Jan 20 18:26:47 Info: auth(default): passwd-file /mailin/mailboxes/conf/pop3-auth.txt: Read 3 users dovecot: Jan 20 18:26:47 Info: auth(default): client out: OK 1 user=try@test.de dovecot: Jan 20 18:26:47 Info: auth(default): master in: REQUEST 1 30065 1 dovecot: Jan 20 18:26:47 Info: auth(default): master out: USER 1 try@test.de uid=99 gid=99 home=/mailin/mailboxes/t/te/test.de/_/try dovecot: Jan 20 18:26:47 Info: pop3(try@test.de): Effective uid=99, gid=99 dovecot: Jan 20 18:26:47 Info: pop3(try@test.de): maildir: data=/mailin/mailboxes/t/te/test.de/_/try/Maildir:INDEX=MEMORY dovecot: Jan 20 18:26:47 Info: pop3(try@test.de): maildir: root=/mailin/mailboxes/t/te/test.de/_/try/Maildir, index=, control=, inbox= dovecot: Jan 20 18:26:47 Error: pop3(try@test.de): file mail-index-lock.c: line 252 (mail_index_copy): assertion failed: (!MAIL_INDEX_IS_IN_MEMORY(index)) dovecot: Jan 20 18:26:47 Info: pop3-login: Login: user=try@test.de, method=PLAIN, rip=192.168.33.72, lip=192.168.33.72, TLS dovecot: Jan 20 18:26:47 Error: child 30072 (pop3) killed with signal 6
Did you test with dotlock and did you remove your existing index-files?
Regards Marten
Hello,
this problem is really scary. It once worked with lock_method set to the default and login_process_per_connection=yes. But really only once. I logged in again and then the process died again after the server responded with "+OK Logged in.". But even this doesn't happen all the time. It also happens very often, that after this "+OK Logged in." nothing happens. I can enter whatever I want, but the server doesn't respond any more, but the connection doesn't die. What also might be of interest: Although I'm testing all this for later use with NFS, right now I'm only working on the local harddisk, with no NFS involved. I don't really know whats happening in mail-index-lock.c line 252, I hope Timo has an idea.
Regards Marten
On Wed, 2006-01-18 at 17:15 +0100, Marten Lehmann wrote:
mail-index-lock.c: line 247 (mail_index_copy): assertion failed: (!MAIL_INDEX_IS_IN_MEMORY(index))
Looks like this happens if you have INDEX=MEMORY and lock_method=dotlock. Set lock_method to something else. It's only used for indexes and with in-memory indexes it's not used at all.
Fixed this in CVS anyway.
Marten Lehmann wrote:
Dovecot is really fine, but it still isn't NFS ready. Especially when using dovecot as pop3 only this index-stuff must be disengageable. What is the use of generating a complex index file for hundreds of messages, that will be deleted in the next step after downloading them?
Well, part of the issue is that Dovecot is _primarily_ an IMAP server, where these indexes prove invaluable to its performance. The POP3 side is more of an added bonus.
But the index is also a problem when using IMAP over NFS.
This is why you can configure Dovecot to use indexes on the local disk, or in memory, as I believe someone else has already mentioned.
Access to the files is too slow, so the connections keep hanging around and the i/o waiting value is raised to a maximum which slows down the whole machine. It simply doesn't make sense to build and update an index every time the maildir is accessed (which is very cpu and timeconsuming), while search operations (the only moment when the index is actually used and not just heavily updated) don't affect even 1% of all IMAP calls (and such a command doesn't even exist for pop3).
I was fairly sure Timo has the index doing more than what you'd at first expect.
I would be much obliged if you could create an option called "--disable-index" in the configure-script or something like "disable-index = yes" in dovecot.conf that makes dovecot ignoring existing index files and doesn't care for related operations, so NFS should work much faster and IMAP search would simply parse the files directly instead of looking in the index.
All that being said, I do agree that this option could well have its place. I suspect Timo might even say it's possible to have a "no-index" plug-in to handle this.
-- Curtis Maloney cmaloney@cardgate.net
participants (7)
-
Curtis Maloney
-
Jakob Hirsch
-
Marcus Rueckert
-
Marten Lehmann
-
Steffen Kaiser
-
Timo Sirainen
-
Tomi Hakala