[Dovecot] Odd ownership of the dovecot-uidlist file

[dclist.hook at hook.net.nz]

Hi,

We have just recently switched from Courier to Dovecot for both IMAAP 
and POP access for our shared hosting platform, and while the issues we 
have been seeing with Courier have gone away a new one has popped up. 
Unfortunately googling for an answer is difficult as it falls into the 
ownership and permissions category which people seem to have all the 
time. But in those cases it always happens, whereas what we are seeing 
is a random ownership change on the dovecot-uidlist file.

Basically what we see is the user is able to login fine for days and 
then all of a sudden the file has changed ownership to a user under a 
different domain, what's worse is the other user has not even logged in 
during that time.

Here's a snippet from a case from a month ago which was escalated to us, 
there are plenty of other examples but its so random we only know its 
happened when the customer complains or we happen to watch the logs, the 
current help desk procedure when the customer calls is to just fix the 
ownership of the file.

-- 8<--
Mar 25 11:18:05 brio dovecot: pop3-login: Login: 
user=<accounts at DOMAIN1>, method=PLAIN, rip=IP, lip=IP, mpid=21739, TLS, 
session=<mwQLnGH1ZAB2XalI>
Mar 25 11:18:08 brio dovecot: pop3(accounts at DOMAIN1): Disconnected: 
Logged out top=0 (0 b), retr=4 (274142 b), messages=27 (10797760 b), del=0
Mar 25 11:19:24 brio dovecot: pop3(accounts at DOMAIN1): Error: 
open(/mnt/spool/keepers/t/DOMAIN1/accounts/Maildir/dovecot-uidlist) 
failed: Permission denied
Mar 25 11:19:24 brio dovecot: pop3(accounts at DOAMIN1): Error: 
open(/mnt/spool/keepers/t/DOMAIN1/accounts/Maildir/dovecot-uidlist) 
failed: Permission denied
-- 8<--

In this case the ownership of the file was:
-rw------- 1 jacqui-30747 DOMAIN2 48839 Mar 25 10:31 dovecot-uidlist
when it should be:
-rw------- 1 accounts-25105 DOMAIN1 48839 Mar 25 10:32 dovecot-uidlist

As you can see both the user and the group have changed to a different 
user as if that user had recreated the file. Its worth noting that the 
parent directory does not change:
drwx------ 8 accounts-25105 DOMAIN1  4096 Mar 25 10:31 .

so this user should not even be able to access this file.

It would be great if someone can give us some hints where the problem 
maybe as this has us stumped.


Some more detail if it helps:
root at brio:~# dovecot -n
# 2.1.7: /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.2
auth_mechanisms = plain login cram-md5 apop
auth_username_chars = 
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@&
default_process_limit = 3000
disable_plaintext_auth = no
lock_method = dotlock
mail_fsync = always
mail_location = maildir:~/Maildir:INDEX=/home/indexes/%1d/%n@%d
mail_nfs_index = yes
mail_nfs_storage = yes
mail_plugins = " quota"
maildir_stat_dirs = yes
mmap_disable = yes
namespace inbox {
   inbox = yes
   location =
   mailbox Drafts {
     special_use = \Drafts
   }
   mailbox Junk {
     special_use = \Junk
   }
   mailbox Sent {
     special_use = \Sent
   }
   mailbox "Sent Messages" {
     special_use = \Sent
   }
   mailbox Trash {
     special_use = \Trash
   }
   prefix = INBOX.
   separator = .
   subscriptions = yes
}
passdb {
   args = /etc/dovecot/dovecot-sql.conf.ext
   driver = sql
}
passdb {
   args = /etc/dovecot/dovecot-sql-cram.conf.ext
   driver = sql
}
plugin {
   quota = maildir:User quota
}
protocols = " imap pop3"
service imap-login {
   inet_listener imap25 {
     port = 25143
   }
}
service imap-postlogin {
   executable = script-login -d /usr/local/bin/postlogin.sh
}
service imap {
   executable = imap imap-postlogin
}
service pop3-login {
   inet_listener pop325 {
     port = 25110
   }
}
service pop3 {
   executable = pop3 imap-postlogin
}
ssl_cert = </var/www/mailadmin/ssl/cert.pem
ssl_crypto_device = dynamic
ssl_key = </var/www/mailadmin/ssl/mailx.hosts.net.nz.key
userdb {
   args = /etc/dovecot/dovecot-sql.conf.ext
   driver = sql
}
userdb {
   args = /etc/dovecot/dovecot-sql-cram.conf.ext
   driver = sql
}
verbose_proctitle = yes
protocol imap {
   imap_capability = +XHOST_brio
   imap_logout_format = bytes=%i/%o
   mail_max_userip_connections = 50
   mail_plugins = " quota imap_quota"
}
protocol pop3 {
   mail_max_userip_connections = 10
   pop3_lock_session = no
   pop3_logout_format = top=%t (%p b), retr=%r (%b b), messages=%m (%s 
b), del=%d
   pop3_save_uidl = yes
   pop3_uidl_format = %f
}

root at brio:~# dpkg -l|grep dovecot
ii  dovecot-core                       1:2.1.7-7 amd64        secure 
mail server that supports mbox, maildir, dbox and mdbox mailboxes
ii  dovecot-imapd                      1:2.1.7-7 amd64        secure 
IMAP server that supports mbox, maildir, dbox and mdbox mailboxes
ii  dovecot-mysql                      1:2.1.7-7 amd64        MySQL 
support for Dovecot
ii  dovecot-pop3d                      1:2.1.7-7 amd64        secure 
POP3 server that supports mbox, maildir, dbox and mdbox mailboxes

Mailbox storage is done over various NFS stores, however each user has a 
specific NFS store
Connections are load balanced using LVS and keepalived over approx 10 
real servers, persistant connections are setup so that a single user 
should only hit a single dovecot server (this is different from our 
courier setup due to the known NFS limitations with dovecot)
We also have IMAP indexes stored on the dovecot servers ext3 file system 
itself rather than NFS.

Each server is setup and maintained by puppet, and any one of the 
servers may see the issue. All servers are x86 with at least 2Gb of RAM

User information comes from a MySQL database, and testing the SQL 
statements only ever return the correct record. Exim is writing to the 
Maildir with the correct permissions, and after fixing the ownership of 
the file dovecot can read and change the mailboxes.

We do have a postlogin.sh script which runs after authentication to 
convert the old courier files over to the dovecot ones, this only 
happens however when the process has not run before (it appears dovecot 
deletes the files and recreates them, so we use a state file). Adding 
debugging to the script shows its running as the correct user at the 
last successful login. It does not run at all after the ownership has 
changed.

There are some users who this is happening regularly, and other users 
who it has happened to only once. It is also not just limited to POP3, 
although this is where we commonly see it as dovecot accepts the IMAP 
connection if the ownership on a sub folders uidlist is different.

If you need any further detail, I can provide it. Debugging could be 
much more difficult because of how random it is, while it can happen 
daily there have been periods of up to one week before its seen.

Any help in finding this would be greatly appreciated,

Cheers,
Bruce