[Dovecot] doveadm purge -A via doveadm-proxy director fails after some users
Hi,
we have configured userdb and passdb in the director and try to
iterate all users and pass the "purge" command via doveadm proxy to
port 19000 on the correct director backend host.
A single purge -u username@example.org via doveadm-proxy works correctly, but iterating over some users with -A fails.
Note: users/domains have been anonymized in output:
mail04:~# /usr/bin/doveadm -c
/etc/dovecot-director/dovecot-director.conf -D purge -A 2>&1
doveadm(root): Debug: Loading modules from directory:
/usr/lib/dovecot/modules/doveadm
doveadm(root): Debug: Skipping module doveadm_acl_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so:
undefined symbol: acl_user_module (this is usually intentional, so
just ignore this message)
doveadm(root): Debug: Skipping module doveadm_expire_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so:
undefined symbol: expire_set_lookup (this is usually intentional, so
just ignore this message)
doveadm(root): Debug: Skipping module doveadm_quota_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so:
undefined symbol: quota_user_module (this is usually intentional, so
just ignore this message)
doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so:
undefined symbol: i_stream_create_deflate (this is usually
intentional, so just ignore this message)
doveadm(root): Debug: Skipping module doveadm_fts_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so:
undefined symbol: fts_list_backend (this is usually intentional, so
just ignore this message)
doveadm(user01@domain1.example.org): Debug: auth input:
user=user01@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user02@domain1.example.org): Debug: auth input:
user=user02@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user03@domain1.example.org): Debug: auth input:
user=user03@domain1.example.org proxy host=10.129.3.192
proxy_refresh=86400
doveadm(user04@domain1.example.org): Debug: auth input:
user=user04@domain1.example.org proxy host=10.129.3.192
proxy_refresh=86400
doveadm(user05@domain1.example.org): Debug: auth input:
user=user05@domain1.example.org proxy host=10.129.3.190
proxy_refresh=86400
doveadm(user06@domain1.example.org): Debug: auth input:
user=user06@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user07@domain1.example.org): Debug: auth input:
user=user07@domain1.example.org proxy host=10.129.3.190
proxy_refresh=86400
doveadm(user08@domain1.example.org): Debug: auth input:
user=user08@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user01@domain2.example.org): Debug: auth input:
user=user01@domain2.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user09@domain1.example.org): Debug: auth input:
user=user09@domain1.example.org proxy host=10.129.3.190
proxy_refresh=86400
10 / 94doveadm(user10@domain1.example.org): Debug: auth input:
user=user10@domain1.example.org proxy host=10.129.3.190
proxy_refresh=86400
doveadm(user11@domain1.example.org): Debug: auth input:
user=user11@domain1.example.org proxy host=10.129.3.191
proxy_refresh=86400
doveadm(user12@domain1.example.org): Debug: auth input:
user=user12@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user13@domain1.example.org): Debug: auth input:
user=user13@domain1.example.org proxy host=10.129.3.190
proxy_refresh=86400
doveadm(user14@domain1.example.org): Debug: auth input:
user=user14@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user15@domain1.example.org): Debug: auth input:
user=user15@domain1.example.org proxy host=10.129.3.191
proxy_refresh=86400
doveadm(user16@domain1.example.org): Debug: auth input:
user=user16@domain1.example.org proxy host=10.129.3.191
proxy_refresh=86400
doveadm(user17@domain1.example.org): Debug: auth input:
user=user17@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user18@domain1.example.org): Debug: auth input:
user=user18@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user19@domain1.example.org): Debug: auth input:
user=user19@domain1.example.org proxy host=10.129.3.192
proxy_refresh=86400
20 / 94doveadm(user20@domain1.example.org): Debug: auth input:
user=user20@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user21@domain1.example.org): Debug: auth input:
user=user21@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user22@domain1.example.org): Debug: auth input:
user=user22@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user02@domain2.example.org): Debug: auth input:
user=user02@domain2.example.org proxy host=10.129.3.190
proxy_refresh=86400
doveadm(user23@domain1.example.org): Debug: auth input:
user=user23@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user24@domain1.example.org): Debug: auth input:
user=user24@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user01@domain3.example.org): Debug: auth input:
user=user01@domain3.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user25@domain1.example.org): Debug: auth input:
user=user25@domain1.example.org proxy host=10.129.3.192
proxy_refresh=86400
doveadm(user26@domain1.example.org): Debug: auth input:
user=user26@domain1.example.org proxy host=10.129.3.191
proxy_refresh=86400
doveadm(user27@domain1.example.org): Debug: auth input:
user=user27@domain1.example.org proxy host=10.129.3.190
proxy_refresh=86400
30 / 94doveadm(user28@domain1.example.org): Debug: auth input:
user=user28@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user29@domain1.example.org): Debug: auth input:
user=user29@domain1.example.org proxy host=10.129.3.191
proxy_refresh=86400
doveadm(user30@domain1.example.org): Debug: auth input:
user=user30@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user31@domain1.example.org): Debug: auth input:
user=user31@domain1.example.org proxy host=10.129.3.193
proxy_refresh=86400
doveadm(user31@domain1.example.org): Error: doveadm server failure
doveadm: Error: Failed to iterate through some users
The user "user31@domain1.example.org" is proxied to the correct
backend host according to director status, but the dovecot.log on the
doveadm service
backend host shows the following error:
Jun 29 15:40:31 10.129.3.249 dovecot:
doveadm(user31@domain1.example.org): Error: user
user31@domain1.example.org: Error reading configuration:
net_connect_unix(/var/run/dovecot/config) failed: Permission denied
Jun 29 15:40:31 10.129.3.249 dovecot:
doveadm(user31@domain1.example.org): Error: purge: User lookup failed:
Internal error occurred. Refer to server log for more information.
The wiki http://wiki2.dovecot.org/Services#doveadm states that the
privileges are (temporarily) dropped to the mail user's privileges
after userdb lookup. It seems that from the second purge on which is
passed over a single doveadm connection, the user lookup fails.
It also seems a bit strange, that the "-A" parameter can be observed in the doveadm tcp stream to the backend, since iteration should be already done in the director and the backend should purge only a single user:
D username@example.org purge -A
Is there a bug or have I misconfigured/overlooked something?
Configs of mailbox backend and director are attached.
Kind regards Daniel
On 29.6.2012, at 19.21, Daniel Parthey wrote:
Jun 29 15:40:31 10.129.3.249 dovecot: doveadm(user31@domain1.example.org): Error: user user31@domain1.example.org: Error reading configuration: net_connect_unix(/var/run/dovecot/config) failed: Permission denied
I've noticed a similar problem happening somewhat randomly, but I still haven't looked into why exactly it happens. Anyway the attached patch should fix this specific error, but I'm not sure if there isn't another one. Try and let me know? :)
Timo Sirainen wrote:
On 29.6.2012, at 19.21, Daniel Parthey wrote:
Jun 29 15:40:31 10.129.3.249 dovecot: doveadm(user31@domain1.example.org): Error: user user31@domain1.example.org: Error reading configuration: net_connect_unix(/var/run/dovecot/config) failed: Permission denied
I've noticed a similar problem happening somewhat randomly, but I still haven't looked into why exactly it happens. Anyway the attached patch should fix this specific error, but I'm not sure if there isn't another one. Try and let me know? :)
Unfortunately, the problem still persists with dovecot 2.1.8, which already contains the following code:
enum master_service_flags service_flags = MASTER_SERVICE_FLAG_KEEP_CONFIG_OPEN; const char *error; master_service = master_service_init("doveadm", service_flags, &argc, &argv, NULL); if (master_getopt(master_service) > 0) return FATAL_DEFAULT;
The command /usr/bin/doveadm -c /etc/dovecot-director/dovecot-director.conf -D purge -A still generates the following errors after iterating some dozen users:
doveadm(nagios@metaways.de): Error: doveadm server failure doveadm: Error: Failed to iterate through some users
Which information should I provide to help debugging the problem?
Kind regards Daniel
Daniel Parthey wrote:
Timo Sirainen wrote:
On 29.6.2012, at 19.21, Daniel Parthey wrote:
Jun 29 15:40:31 10.129.3.249 dovecot: doveadm(user31@domain1.example.org): Error: user user31@domain1.example.org: Error reading configuration: net_connect_unix(/var/run/dovecot/config) failed: Permission denied
I've noticed a similar problem happening somewhat randomly, but I still haven't looked into why exactly it happens. Anyway the attached patch should fix this specific error, but I'm not sure if there isn't another one. Try and let me know? :)
Unfortunately, the problem still persists with dovecot 2.1.8, which already contains the following code:
enum master_service_flags service_flags = MASTER_SERVICE_FLAG_KEEP_CONFIG_OPEN; const char *error; master_service = master_service_init("doveadm", service_flags, &argc, &argv, NULL); if (master_getopt(master_service) > 0) return FATAL_DEFAULT;
The command /usr/bin/doveadm -c /etc/dovecot-director/dovecot-director.conf -D purge -A still generates the following errors after iterating some dozen users:
doveadm(nagios@metaways.de): Error: doveadm server failure doveadm: Error: Failed to iterate through some users
Which information should I provide to help debugging the problem?
The change in 2.1.8 didn't fix it and still does not iterate *all* users. My current workaround is to list all users, run a for-loop over the userlist and flush one mailbox after another via the director.
Any hints how to solve this?
Regards Daniel
On 1.8.2012, at 22.32, Daniel Parthey wrote:
Jun 29 15:40:31 10.129.3.249 dovecot: doveadm(user31@domain1.example.org): Error: user user31@domain1.example.org: Error reading configuration: net_connect_unix(/var/run/dovecot/config) failed: Permission denied The change in 2.1.8 didn't fix it and still does not iterate *all* users. My current workaround is to list all users, run a for-loop over the userlist and flush one mailbox after another via the director.
Any hints how to solve this?
The error is still the same "config permission denied" shown above? I found that also from my server and added a debug patch, but it hasn't crashed yet. Could you try the attached patch and getting a gdb backtrace from the resulting core file? (Or at least the raw backtrace - getting a core file might be tricky.)
On 1.8.2012, at 22.56, Timo Sirainen wrote:
On 1.8.2012, at 22.32, Daniel Parthey wrote:
Jun 29 15:40:31 10.129.3.249 dovecot: doveadm(user31@domain1.example.org): Error: user user31@domain1.example.org: Error reading configuration: net_connect_unix(/var/run/dovecot/config) failed: Permission denied The change in 2.1.8 didn't fix it and still does not iterate *all* users. My current workaround is to list all users, run a for-loop over the userlist and flush one mailbox after another via the director.
Any hints how to solve this?
The error is still the same "config permission denied" shown above? I found that also from my server and added a debug patch, but it hasn't crashed yet. Could you try the attached patch and getting a gdb backtrace from the resulting core file? (Or at least the raw backtrace - getting a core file might be tricky.)
Also I wonder if this helps (at least it fixes a crash I managed to cause): http://hg.dovecot.org/dovecot-2.1/rev/476381017ec7
Timo Sirainen wrote:
On 1.8.2012, at 22.32, Daniel Parthey wrote:
Jun 29 15:40:31 10.129.3.249 dovecot: doveadm(user31@domain1.example.org): Error: user user31@domain1.example.org: Error reading configuration: net_connect_unix(/var/run/dovecot/config) failed: Permission denied The change in 2.1.8 didn't fix it and still does not iterate *all* users. My current workaround is to list all users, run a for-loop over the userlist and flush one mailbox after another via the director.
Any hints how to solve this?
The error is still the same "config permission denied" shown above? I found that also from my server and added a debug patch, but it hasn't crashed yet. Could you try the attached patch and getting a gdb backtrace from the resulting core file? (Or at least the raw backtrace - getting a core file might be tricky.)
Running command on a four host setup with mailbox+director instance each: /usr/bin/doveadm -c /etc/dovecot-director/dovecot-director.conf -D purge -A
Output is: doveadm(username@example.org): Error: doveadm server failure doveadm: Error: Failed to iterate through some users
Log says: dovecot: doveadm(username@example.org): Error: userdb lookup: connect(/var/run/dovecot/auth-userdb) failed: Permission denied (euid=501(vmail) egid=123(vmail) missing +r perm: /var/run/dovecot/auth-userdb, we're not in group 122(dovecot), dir owned by 0:0 mode=0755) dovecot: doveadm(username@example.org): Error: purge: User lookup failed: Internal error occurred. Refer to server log for more information.
Here are the directory permissions:
mail01:~# id vmail uid=501(vmail) gid=123(vmail) groups=123(vmail)
mail01:~# id dovecot uid=120(dovecot) gid=122(dovecot) groups=122(dovecot)
mail01:~# ls -ld /var/run/dovecot drwxr-xr-x 4 root root 660 2012-07-11 18:35 /var/run/dovecot
mail01:~# ls -l /var/run/dovecot total 8 srw------- 1 root root 0 2012-07-10 18:29 anvil srw------- 1 root root 0 2012-07-10 18:29 anvil-auth-penalty srw------- 1 root root 0 2012-07-11 18:35 auth-client srw------- 1 dovecot root 0 2012-07-11 18:35 auth-login srw------- 1 root root 0 2012-07-11 18:35 auth-master srw-rw---- 1 dovecot dovecot 0 2012-07-11 18:35 auth-userdb srw------- 1 dovecot root 0 2012-07-11 18:35 auth-worker srw------- 1 root root 0 2012-07-11 18:35 config srw-rw---- 1 root vmail 0 2012-07-11 18:35 dict srw------- 1 root root 0 2012-07-11 18:35 director-admin srw------- 1 root root 0 2012-07-10 10:19 director-userdb srw-rw-rw- 1 root root 0 2012-07-11 18:35 dns-client srw------- 1 root root 0 2012-07-11 18:35 doveadm-server lrwxrwxrwx 1 root root 25 2012-07-10 18:29 dovecot.conf -> /etc/dovecot/dovecot.conf drwxr-xr-x 2 root root 40 2012-07-10 10:19 empty srw------- 1 root root 0 2012-07-11 18:35 imap-postlogin srw-rw-rw- 1 root root 0 2012-07-11 18:35 indexer srw------- 1 dovecot root 0 2012-07-11 18:35 indexer-worker srw------- 1 root root 0 2012-07-11 18:35 ipc srw-rw-rw- 1 root root 0 2012-07-11 18:35 lmtp srw------- 1 root root 0 2012-07-11 18:35 log-errors drwxr-x--- 2 root nogroup 180 2012-07-11 18:35 login -rw------- 1 root root 5 2012-07-10 18:29 master.pid -rw-r--r-- 1 root root 71 2012-07-10 18:29 mounts srw------- 1 root root 0 2012-07-11 18:35 pop3-postlogin srw------- 1 vmail root 0 2012-07-11 18:35 quota-warning srw------- 1 root root 0 2012-07-11 18:35 replication-notify prw------- 1 root root 0 2012-07-11 18:35 replication-notify-fifo srw------- 1 dovecot root 0 2012-07-11 18:35 replicator srw------- 1 root root 0 2012-07-11 18:35 stats prw------- 1 vmail root 0 2012-08-01 22:20 stats-mail
Please see mailbox and director config attached.
Any help is appreciated.
Regards Daniel
On 1.8.2012, at 23.25, Daniel Parthey wrote:
The error is still the same "config permission denied" shown above? I found that also from my server and added a debug patch, but it hasn't crashed yet. Could you try the attached patch and getting a gdb backtrace from the resulting core file? (Or at least the raw backtrace - getting a core file might be tricky.)
Running command on a four host setup with mailbox+director instance each: /usr/bin/doveadm -c /etc/dovecot-director/dovecot-director.conf -D purge -A
dovecot: doveadm(username@example.org): Error: userdb lookup: connect(/var/run/dovecot/auth-userdb) failed: Permission denied (euid=501(vmail) egid=123(vmail) missing +r perm: /var/run/dovecot/auth-userdb, we're not in group 122(dovecot), dir owned by 0:0 mode=0755)
Ah, so the original patch helped! This is a different error.
srw-rw---- 1 dovecot dovecot 0 2012-07-11 18:35 auth-userdb
Simplest solution now would be to make this world-rw, see the auth-userdb socket configuration in http://wiki2.dovecot.org/LDA#Virtual_users
But I guess this should also be fixed by doveadm-server. Although I don't think this should be happening by default anyway. Maybe this is also solved by the http://hg.dovecot.org/dovecot-2.1/rev/476381017ec7 patch?
Hi Timo and list members,
Timo Sirainen wrote:
On 1.8.2012, at 23.25, Daniel Parthey wrote:
The error is still the same "config permission denied" shown above? I found that also from my server and added a debug patch, but it hasn't crashed yet. Could you try the attached patch and getting a gdb backtrace from the resulting core file? (Or at least the raw backtrace - getting a core file might be tricky.)
Running command on a four host setup with mailbox+director instance each: /usr/bin/doveadm -c /etc/dovecot-director/dovecot-director.conf -D purge -A
dovecot: doveadm(username@example.org): Error: userdb lookup: connect(/var/run/dovecot/auth-userdb) failed: Permission denied (euid=501(vmail) egid=123(vmail) missing +r perm: /var/run/dovecot/auth-userdb, we're not in group 122(dovecot), dir owned by 0:0 mode=0755)
Ah, so the original patch helped! This is a different error.
srw-rw---- 1 dovecot dovecot 0 2012-07-11 18:35 auth-userdb
Simplest solution now would be to make this world-rw, see the auth-userdb socket configuration in http://wiki2.dovecot.org/LDA#Virtual_users
But I guess this should also be fixed by doveadm-server. Although I don't think this should be happening by default anyway. Maybe this is also solved by the http://hg.dovecot.org/dovecot-2.1/rev/476381017ec7 patch?
I finally found time to update from 2.1.8 to 2.1.10 and change service auth-user db socket to default mode of 0666.
Unfortunately, the issue is still not solved and I did not manage to get a gdb backtrace, since it does not crash or assert. Current configuration of both mailbox and director is attached.
The error "Permission denied" from the mailbox logs is gone, but the director doveadm command: /usr/bin/doveadm -c /etc/dovecot-director/dovecot-director.conf -D purge -A still throws the error message:
doveadm(username@example.org): Error: doveadm server failure
doveadm: Error: Failed to iterate through some users
Any idea what I could do in addition to making /var/run/dovecot/auth-userdb world-rw?
Regards Daniel
participants (2)
-
Daniel Parthey
-
Timo Sirainen