[Dovecot] Dovecot Alpha 4
Hi
We've been using Dovecot as our primary IMAP and POP3 server for around 70,000 users for a couple of months now. On the whole, we've been very pleased with the performance of Dovecot - and are extremely grateful to the developers! However, we have had a few issues, some of which we have yet to resolve.
We chose Dovecot because it was the only IMAP/POP3 server we could find that supported direct mysql authentication and both mboxes and maildirs. With exim as our main SMTP server, we were able to convert the inboxes and imap folders for all 70,000 users from mboxes to maildirs without any downtime whatsoever!
For those who are interested, we have a cluster of 5 web/imap servers (running our own webmail software) and a further 5 pop servers all placed behind 2 Zeus load balancers and running RHEL 4 (actually CentOS 4.1). Storage is provided by an EMC Clariion device, which is mounted via NFS onto all servers in the cluster over gigabit ethernet.
- index files
The main problem has always been the index files becoming corrupted. This seems to have improved with the Alpha 4 release, but still happens for several users each day. Perhaps it would be possible for Dovecot to just delete index files and re-create them when they are corrupted, rather than just erroring? Also, how do you turn off index files altogether? Even when mail_never_cache_fields is set for all available fields, they still get created.
Another issue relating to the index files is that, if a user is deleted and then another user created with the same name, dovecot doesn't have permissions to re-write the index files, because it has a different uid. So, whenever a user is deleted, we have to delete the index files from every machine in the cluster. We could store index files on the NFS device, but this would affect performance (and might cause locking problems). A better solution would be to make the uid one of the variables available in default_mail_env. By naming index files by uid rather than username, this wouldn't be an issue.
- initgroups()
We use an nss-mysql to store all non-administrative system users in a mysql database. We often encounter problems with applications that use the initgroups() function, since this returns all users and groups - which in our case returns masses of data from mysql. When using mysql (or ldap etc) for authentication, it would be useful if there were an option to prevent additional system lookups. At present, we have to comment out the following in /src/lib/restrict-access.c, or the server load goes through the roof:
if (initgroups(env, gid) != 0) { i_fatal("initgroups(%s, %s) failed: %m", env, dec2str(gid)); }
- base_dir permissions bug
Since the alpha 4 release, it seems that the permissions dovecot automatically sets for the base_dir are not sufficient to allow the authentication user to access sockets, unless this user belongs to the same group as the login user - which is contrary to the instructions in the documentation. I'm pretty sure this is a bug, but perhaps someone could confirm.
- authentication caching
Also since the alpha 4 release, we have found that, once the authentication cache is full, all subsequent login attempts for users that haven't been cached return "password mismatch". I though this might be a conflict with nscd, but it happens whether nscd is running or not. So, for the time being, we have had to disable the authentication cache.
I'd be very keen to hear from anyone who has any feedback on any of the points above.
Thanks
Marcus
Marcus Don Applications Development Manager Namesco Limited
Main Line: +44 (0)870 120 8888 Main Fax: +44 (0)870 120 8008 Tech Support: +44 (0)870 162 4950 Email: mdon@names.co.uk Website: http://www.names.co.uk Address: Acton House, Perdiswell Park, Worcester, WR3 7GD
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Finally, the recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.
---------------------------------------
| Free Dial-up - no need to register! |
---------------------------------------Namesco Limited provides Free dialup access to anyone that wishes to use it. You do not even have to be a customer, there is no need to register and is especially handy if you are unable to connect with your current provider. Simply use the following details:
Dialup Number: 0844 535 2100*
Username: namesco
Password: easydialup- Local call rate charges apply, 33% Cheaper than BT's 0845 local call rate.
If you need any help in configuring the Free Dial-up service on your computer, please do not hesitate to call us on 0870 162 4950.
Marcus Don wrote:
Also, how do you turn off index files altogether?
Put "INDEX=MEMORY" on your default_mail_env, by this you make Dovecot to use non-persistent memory indexes, for example:
default_mail_env = maildir:/nfs/%d/%u/Maildir:INDEX=MEMORY
So, whenever a user is deleted, we have to delete the index files from every machine in the cluster. A better solution would be to make the uid one of the variables available in default_mail_env. By naming index files by uid rather than username, this wouldn't be an issue.
If you move to memory indexes you have no use for this but since you use SQL authentication backend you could have very flexible layout for your mail spool and indexes. Just return properly formatted "mail" field on user_query, this will overwrite your default_mail_env you have in dovecot.conf.
For example
user_query = SELECT uid, .., concat('maildir:/nfs/', domain, '/', uid, '/Maildir:INDEX=/var/index/', domain, '/' uid) AS mail, ..
would read mail from /nfs/%domain/%uid/Maildir and store indexes to /var/index/%domain/%uid
Cool but not very neat.
More clean way to do this would be to have templates for environment variables like home, mail and index and use field names from SQL as fill in variables, like:
userdb sql { set home = /nfs/%%spool%%/%d/%%uid%% set mail = maildir:/nfs/%%spool%%/%d/%%uid%%/mail set index = /var/%3RHu/%d/%%uid%% args = /opt/dovecot/etc/sql.conf }
and then get fields like "spool" and "uid" from SQL and let the Dovecot to fill them into template environments you defined.
I know that Timo has more than enough work at the moment so if anyone on the list have some spare time please feel free to take a look on this :)
- authentication caching
Also since the alpha 4 release, we have found that, once the authentication cache is full, all subsequent login attempts for users that haven't been cached return "password mismatch". I though this might be a conflict with nscd, but it happens whether nscd is running or not. So, for the time being, we have had to disable the authentication cache.
Authentication caching was more or less broken in all of alpha series, there are fixes for those in CVS.
I'd be very keen to hear from anyone who has any feedback on any of the points above.
Have you thought of clustered file system to go with your Dovecot hosts and CLARiiON device?
-- Tomi Hakala
On Thursday 27 Oct 2005 17:42, Marcus Don wrote:
Hi
We've been using Dovecot as our primary IMAP and POP3 server for around 70,000 users for a couple of months now. On the whole, we've been very pleased with the performance of Dovecot - and are extremely grateful to the developers! However, we have had a few issues, some of which we have yet to resolve.
We chose Dovecot because it was the only IMAP/POP3 server we could find that supported direct mysql authentication and both mboxes and maildirs. With exim as our main SMTP server, we were able to convert the inboxes and imap folders for all 70,000 users from mboxes to maildirs without any downtime whatsoever!
For those who are interested, we have a cluster of 5 web/imap servers (running our own webmail software) and a further 5 pop servers all placed behind 2 Zeus load balancers and running RHEL 4 (actually CentOS 4.1). Storage is provided by an EMC Clariion device, which is mounted via NFS onto all servers in the cluster over gigabit ethernet.
- index files
The main problem has always been the index files becoming corrupted. This seems to have improved with the Alpha 4 release, but still happens for several users each day. Perhaps it would be possible for Dovecot to just delete index files and re-create them when they are corrupted, rather than just erroring? Also, how do you turn off index files altogether? Even when mail_never_cache_fields is set for all available fields, they still get created.
[snip]
Thanks
Marcus
Hi Marcus,
Good to see another UK ISP using dovecot :) We use an LVS director with 4 pop/imap servers and a master-slave mysql backend for auth. The only problem we are still seeing that I am aware of is dovecot seems to sometimes rely on stale indexes rather than checking to see if they are correct first. This can cause clients to get message not found errors. I would debug properly but every time I get a moment free Manchester's IFL gets too hot and a server goes pop so I have to fix that instead.
Regards Andrew
Andrew Hutchings (A-Wing) - Linux Guru
Netserve Consultants - http://www.domaincity.co.uk/
Linux CDs and DVDs - http://www.linuxiso.co.uk/
Random quote 206: "One World, one web, one program" - Microsoft Promo ad.
"Ein Volk, Ein Reich, Ein Fuhrer" - Adolf Hitler
On Thu, 2005-10-27 at 17:42 +0100, Marcus Don wrote:
- index files
The main problem has always been the index files becoming corrupted. This seems to have improved with the Alpha 4 release, but still happens for several users each day.
Reporting the exact errors here is useful. I do fix them once in a while :)
Perhaps it would be possible for Dovecot to just delete index files and re-create them when they are corrupted, rather than just erroring?
It should do that. If it doesn't do it with some errors, please tell which errors and I'll fix it.
Also, how do you turn off index files altogether? Even when mail_never_cache_fields is set for all available fields, they still get created.
That just means nothing is stored to dovecot.index.cache file. dovecot.index is still used.
Another issue relating to the index files is that, if a user is deleted and then another user created with the same name, dovecot doesn't have permissions to re-write the index files, because it has a different uid. So, whenever a user is deleted, we have to delete the index files from every machine in the cluster. We could store index files on the NFS device, but this would affect performance (and might cause locking problems).
In theory storing the indexes in NFS would be faster, if the mailbox was accessed from multiple computers.. But there are still some problems with it.
Hmm. Wonder if it would be a good idea to delay the index updates and write them only when closing the mailbox. That way there would be less NFS traffic, but the indexes still would be updated globally, and if another computer accessed the mailbox later it would have already up-to-date indexes..
A better solution would be to make the uid one of the variables available in default_mail_env. By naming index files by uid rather than username, this wouldn't be an issue.
I guess.. Added %i to CVS.
- initgroups()
We use an nss-mysql to store all non-administrative system users in a mysql database. We often encounter problems with applications that use the initgroups() function, since this returns all users and groups - which in our case returns masses of data from mysql. When using mysql (or ldap etc) for authentication, it would be useful if there were an option to prevent additional system lookups. At present, we have to comment out the following in /src/lib/restrict-access.c, or the server load goes through the roof:
if (initgroups(env, gid) != 0) { i_fatal("initgroups(%s, %s) failed: %m", env, dec2str(gid)); }
I think it's becuse of login processes, not because of imap/pop3 processes? The initgroups() is used only if userdb returns "system_user" parameter, which by default is only returned if you use passwd userdb. I'll change even that at some point so that auth process can just give a GID-list to be used.
Hmm. And as for login process, it doesn't ever need more than one GID. I guess it shouldn't break anything if I just don't let it do the initgroups() call. Removed.
- base_dir permissions bug
Since the alpha 4 release, it seems that the permissions dovecot automatically sets for the base_dir are not sufficient to allow the authentication user to access sockets, unless this user belongs to the same group as the login user - which is contrary to the instructions in the documentation. I'm pretty sure this is a bug, but perhaps someone could confirm.
You probably mean with mysql passdb/userdb? That's a bug..
- authentication caching
Also since the alpha 4 release, we have found that, once the authentication cache is full, all subsequent login attempts for users that haven't been cached return "password mismatch". I though this might be a conflict with nscd, but it happens whether nscd is running or not. So, for the time being, we have had to disable the authentication cache.
Looks like auth cache was pretty much completely broken in alpha4 for sql/ldap. Fixed.
Timo Sirainen wrote:
Hmm. Wonder if it would be a good idea to delay the index updates and write them only when closing the mailbox. That way there would be less NFS traffic, but the indexes still would be updated globally, and if another computer accessed the mailbox later it would have already up-to-date indexes..
Would be good with POP3 regardless of index storage, not so sure about IMAP.. Perhaps some protocol specific option for users to choose?
-- Tomi Hakala
participants (4)
-
Andrew Hutchings
-
Marcus Don
-
Timo Sirainen
-
Tomi Hakala