Hi,
We have just recently switched from Courier to Dovecot for both IMAAP and POP access for our shared hosting platform, and while the issues we have been seeing with Courier have gone away a new one has popped up. Unfortunately googling for an answer is difficult as it falls into the ownership and permissions category which people seem to have all the time. But in those cases it always happens, whereas what we are seeing is a random ownership change on the dovecot-uidlist file.
Basically what we see is the user is able to login fine for days and then all of a sudden the file has changed ownership to a user under a different domain, what's worse is the other user has not even logged in during that time.
Here's a snippet from a case from a month ago which was escalated to us, there are plenty of other examples but its so random we only know its happened when the customer complains or we happen to watch the logs, the current help desk procedure when the customer calls is to just fix the ownership of the file.
-- 8<-- Mar 25 11:18:05 brio dovecot: pop3-login: Login: user=accounts@DOMAIN1, method=PLAIN, rip=IP, lip=IP, mpid=21739, TLS, session=<mwQLnGH1ZAB2XalI> Mar 25 11:18:08 brio dovecot: pop3(accounts@DOMAIN1): Disconnected: Logged out top=0 (0 b), retr=4 (274142 b), messages=27 (10797760 b), del=0 Mar 25 11:19:24 brio dovecot: pop3(accounts@DOMAIN1): Error: open(/mnt/spool/keepers/t/DOMAIN1/accounts/Maildir/dovecot-uidlist) failed: Permission denied Mar 25 11:19:24 brio dovecot: pop3(accounts@DOAMIN1): Error: open(/mnt/spool/keepers/t/DOMAIN1/accounts/Maildir/dovecot-uidlist) failed: Permission denied -- 8<--
In this case the ownership of the file was: -rw------- 1 jacqui-30747 DOMAIN2 48839 Mar 25 10:31 dovecot-uidlist when it should be: -rw------- 1 accounts-25105 DOMAIN1 48839 Mar 25 10:32 dovecot-uidlist
As you can see both the user and the group have changed to a different user as if that user had recreated the file. Its worth noting that the parent directory does not change: drwx------ 8 accounts-25105 DOMAIN1 4096 Mar 25 10:31 .
so this user should not even be able to access this file.
It would be great if someone can give us some hints where the problem maybe as this has us stumped.
Some more detail if it helps: root@brio:~# dovecot -n # 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.2 auth_mechanisms = plain login cram-md5 apop auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@& default_process_limit = 3000 disable_plaintext_auth = no lock_method = dotlock mail_fsync = always mail_location = maildir:~/Maildir:INDEX=/home/indexes/%1d/%n@%d mail_nfs_index = yes mail_nfs_storage = yes mail_plugins = " quota" maildir_stat_dirs = yes mmap_disable = yes namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = INBOX. separator = . subscriptions = yes } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } passdb { args = /etc/dovecot/dovecot-sql-cram.conf.ext driver = sql } plugin { quota = maildir:User quota } protocols = " imap pop3" service imap-login { inet_listener imap25 { port = 25143 } } service imap-postlogin { executable = script-login -d /usr/local/bin/postlogin.sh } service imap { executable = imap imap-postlogin } service pop3-login { inet_listener pop325 { port = 25110 } } service pop3 { executable = pop3 imap-postlogin } ssl_cert =
root@brio:~# dpkg -l|grep dovecot ii dovecot-core 1:2.1.7-7 amd64 secure mail server that supports mbox, maildir, dbox and mdbox mailboxes ii dovecot-imapd 1:2.1.7-7 amd64 secure IMAP server that supports mbox, maildir, dbox and mdbox mailboxes ii dovecot-mysql 1:2.1.7-7 amd64 MySQL support for Dovecot ii dovecot-pop3d 1:2.1.7-7 amd64 secure POP3 server that supports mbox, maildir, dbox and mdbox mailboxes
Mailbox storage is done over various NFS stores, however each user has a specific NFS store Connections are load balanced using LVS and keepalived over approx 10 real servers, persistant connections are setup so that a single user should only hit a single dovecot server (this is different from our courier setup due to the known NFS limitations with dovecot) We also have IMAP indexes stored on the dovecot servers ext3 file system itself rather than NFS.
Each server is setup and maintained by puppet, and any one of the servers may see the issue. All servers are x86 with at least 2Gb of RAM
User information comes from a MySQL database, and testing the SQL statements only ever return the correct record. Exim is writing to the Maildir with the correct permissions, and after fixing the ownership of the file dovecot can read and change the mailboxes.
We do have a postlogin.sh script which runs after authentication to convert the old courier files over to the dovecot ones, this only happens however when the process has not run before (it appears dovecot deletes the files and recreates them, so we use a state file). Adding debugging to the script shows its running as the correct user at the last successful login. It does not run at all after the ownership has changed.
There are some users who this is happening regularly, and other users who it has happened to only once. It is also not just limited to POP3, although this is where we commonly see it as dovecot accepts the IMAP connection if the ownership on a sub folders uidlist is different.
If you need any further detail, I can provide it. Debugging could be much more difficult because of how random it is, while it can happen daily there have been periods of up to one week before its seen.
Any help in finding this would be greatly appreciated,
Cheers, Bruce