Hi,
I am one of the people taking over our new suse.de email setup (consisting of dovecot+rspamd+postfix) and wanted to report some issues we experience:
- we use dovecot-director to distribute users between 2 backend servers that share an NFS mount. We found that it proxies lmtp to a different backend than imap of the same user and that caused NFS stale-filehandle errors on the dovecot-uidlist. It then proceeds to re-generate the dovecot-uidlist with new UIDs that creates trouble for users.
a) shouldn't dovecot use locks (fcntl or flock) to protect such files from concurrent updates?
b) could it generate uidlist in a way that re-generating it, assigns the same UIDs again? E.g. via hash over file content
c) how to get dovecot-director to send all traffic for a user to one backend?
- We have 2 backends so that we can do maintenance on one of them while users can still access their emails through the other backend. However, we found that stopping dovecot on one backend left users unable to access their mails. Maybe this is related to how user auth works? How to get this HA setup right, so that we don't have a single point of failure?
grep PRETTY /etc/os-release PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"
rpm -q dovecot23 dovecot23-2.3.15
https://www.zq1.de/~bernhard/temp/dovecot/ has some sysreports.
Ciao Bernhard M.
On 10/09/2021 12:19 Bernhard M. Wiedemann bwiedemann@suse.de wrote:
Hi,
I am one of the people taking over our new suse.de email setup (consisting of dovecot+rspamd+postfix) and wanted to report some issues we experience:
- we use dovecot-director to distribute users between 2 backend servers that share an NFS mount. We found that it proxies lmtp to a different backend than imap of the same user and that caused NFS stale-filehandle errors on the dovecot-uidlist. It then proceeds to re-generate the dovecot-uidlist with new UIDs that creates trouble for users.
a) shouldn't dovecot use locks (fcntl or flock) to protect such files from concurrent updates?
b) could it generate uidlist in a way that re-generating it, assigns the same UIDs again? E.g. via hash over file content
c) how to get dovecot-director to send all traffic for a user to one backend?
- We have 2 backends so that we can do maintenance on one of them while users can still access their emails through the other backend. However, we found that stopping dovecot on one backend left users unable to access their mails. Maybe this is related to how user auth works? How to get this HA setup right, so that we don't have a single point of failure?
grep PRETTY /etc/os-release PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"
rpm -q dovecot23 dovecot23-2.3.15
https://www.zq1.de/~bernhard/temp/dovecot/ has some sysreports.
Ciao Bernhard M.
Can you post your doveconf -n? LMTP should not end up in different backend.
Aki
On 10/09/2021 12:32 Aki Tuomi aki.tuomi@open-xchange.com wrote:
On 10/09/2021 12:19 Bernhard M. Wiedemann bwiedemann@suse.de wrote:
Hi,
I am one of the people taking over our new suse.de email setup (consisting of dovecot+rspamd+postfix) and wanted to report some issues we experience:
- we use dovecot-director to distribute users between 2 backend servers that share an NFS mount. We found that it proxies lmtp to a different backend than imap of the same user and that caused NFS stale-filehandle errors on the dovecot-uidlist. It then proceeds to re-generate the dovecot-uidlist with new UIDs that creates trouble for users.
a) shouldn't dovecot use locks (fcntl or flock) to protect such files from concurrent updates?
b) could it generate uidlist in a way that re-generating it, assigns the same UIDs again? E.g. via hash over file content
c) how to get dovecot-director to send all traffic for a user to one backend?
- We have 2 backends so that we can do maintenance on one of them while users can still access their emails through the other backend. However, we found that stopping dovecot on one backend left users unable to access their mails. Maybe this is related to how user auth works? How to get this HA setup right, so that we don't have a single point of failure?
grep PRETTY /etc/os-release PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"
rpm -q dovecot23 dovecot23-2.3.15
https://www.zq1.de/~bernhard/temp/dovecot/ has some sysreports.
Ciao Bernhard M.
Can you post your
doveconf -n? LMTP should not end up in different backend.Aki
And now I noticed you had those sysreports there... Nvm, I'll look at those...
Aki
On 10/09/2021 11.32, Aki Tuomi wrote:
Can you post your
doveconf -n? LMTP should not end up in different backend.
# 2.3.15 (0503334ab1): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.15 (e6a84e31) # OS: Linux 5.3.18-59.19-default x86_64 # Hostname: dovecot-director2.suse-dmz.suse.de auth_mechanisms = plain login auth_verbose = yes default_client_limit = 8192 default_process_limit = 16384 default_vsz_limit = 2 G director_mail_servers = imap1.suse-dmz.suse.de@nuernberg imap2.suse-dmz.suse.de@nuernberg director_servers = dovecot-director1.suse-dmz.suse.de dovecot-director2.suse-dmz.suse.de doveadm_api_key = # hidden, use -P to show it doveadm_password = # hidden, use -P to show it doveadm_port = 12345 haproxy_trusted_networks = 192.168.254.0/24 hostname = dovecot-director2.suse.de lmtp_proxy = yes login_log_format_elements = user=<%n> application=<%d> method=%m rip=%r lip=%l mpid=%e %c %k mail_plugins = " quota" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql override_fields = proxy=yes director_tag=nuernberg starttls=yes } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap override_fields = proxy=yes director_tag=nuernberg starttls=yes } plugin { sieve = file:~/sieve;active=~/.dovecot.sieve } protocols = imap pop3 lmtp submission sieve service anvil { client_limit = 69636 } service auth-worker { process_limit = 4096 } service auth { client_limit = 98304 } service director { fifo_listener login/proxy-notify { mode = 0666 } inet_listener { address = 192.168.254.65,127.0.0.1,::1 port = 9090 } unix_listener director-userdb { mode = 0600 } unix_listener login/director { mode = 0666 } } service doveadm { inet_listener { address = 192.168.254.65,127.0.0.1,::1 port = 12345 } inet_listener http { address = 192.168.254.65,127.0.0.1,::1 port = 8080 } } service imap-login { executable = imap-login director process_limit = 4096 } service lmtp { inet_listener lmtp { address = 192.168.254.65, 192.168.254.85, 127.0.0.1, ::1 port = 24 } inet_listener lmtp_haproxy { address = 192.168.254.65, 192.168.254.85, 127.0.0.1, ::1 haproxy = yes port = 23 } process_limit = 4096 unix_listener lmtp { mode = 0666 } } service managesieve-login { executable = managesieve-login director inet_listener sieve { address = 192.168.254.65, 192.168.254.85, 127.0.0.1, ::1 } process_limit = 4096 } service pop3-login { executable = pop3-login director } service submission-login { executable = submission-login director inet_listener submission { port = 587 ssl = no } process_limit = 4096 } ssl = required ssl_ca =
---- and some logs ----
Sep 09 08:23:47 dovecot-director2 dovecot[1424]: lmtp(1911): lmtp-server: conn 149.44.160.134:50254 [6]: rcpt msuchanek@imap.suse.de: 6Fr6CJDEOWF3BwAApTUePA: Sent message to <msuchanek> at imap1.suse-dmz.suse.de:24: 250 2.0.0 <msuchanek> IOniGJDEOWHuCwAAGKfGzw Saved (1/1 at 3618 ms) Sep 09 08:24:07 dovecot-director2 dovecot[1424]: lmtp(2196): lmtp-server: conn 149.44.160.134:50360 [4]: rcpt msuchanek@imap.suse.de: KNOyFaTEOWGUCAAApTUePA: Sent message to <msuchanek> at imap1.suse-dmz.suse.de:24: 250 2.0.0 <msuchanek> yHSOJaTEOWFMDwAAGKfGzw Saved (1/1 at 2921 ms) Sep 09 08:27:13 dovecot-director2 dovecot[1424]: imap-login: proxy(msuchanek@offlineimap,192.168.254.74:143): Started proxying to imap2.suse-dmz.suse.de (0.076 secs): user=<msuchanek>, application=<offlineimap>, method=PLAIN, rip=192.168.254.67, lip=192.168.254.85, TLS, TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
Bernhard M. Wiedemann schreef op 2021-09-10 11:19:
Hi,
I am one of the people taking over our new suse.de email setup (consisting of dovecot+rspamd+postfix) and wanted to report some issues we experience:
- we use dovecot-director to distribute users between 2 backend servers that share an NFS mount. We found that it proxies lmtp to a different backend than imap of the same user and that caused NFS stale-filehandle errors on the dovecot-uidlist. It then proceeds to re-generate the dovecot-uidlist with new UIDs that creates trouble for users.
a) shouldn't dovecot use locks (fcntl or flock) to protect such files from concurrent updates?
b) could it generate uidlist in a way that re-generating it, assigns the same UIDs again? E.g. via hash over file content
c) how to get dovecot-director to send all traffic for a user to one backend?
- We have 2 backends so that we can do maintenance on one of them while users can still access their emails through the other backend. However, we found that stopping dovecot on one backend left users unable to access their mails.
Director doesn't do health checks.
Maybe this is related to how user auth works? How to get this HA setup right, so that we don't have a single point of failure?
grep PRETTY /etc/os-release PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"
rpm -q dovecot23 dovecot23-2.3.15
https://www.zq1.de/~bernhard/temp/dovecot/ has some sysreports.
Ciao Bernhard M.
-- With kind regards,
William Edwards
On 10/09/2021 11.33, William Edwards wrote:
- We have 2 backends so that we can do maintenance on one of them while users can still access their emails through the other backend. However, we found that stopping dovecot on one backend left users unable to access their mails.
Director doesn't do health checks.
Is there are recommendation on how to have a director with multiple backends with automatic fail-over? Are there scripts that do health-checks and call doveadm director commands?
Bernhard M. Wiedemann schreef op 2021-09-12 10:34:
On 10/09/2021 11.33, William Edwards wrote:
- We have 2 backends so that we can do maintenance on one of them while users can still access their emails through the other backend. However, we found that stopping dovecot on one backend left users unable to access their mails.
Director doesn't do health checks.
Is there are recommendation on how to have a director with multiple backends with automatic fail-over? Are there scripts that do health-checks and call doveadm director commands?
https://doc.dovecot.org/configuration_manual/dovemon/
-- With kind regards,
William Edwards
On 12/09/2021 14:54 William Edwards wedwards@cyberfusion.nl wrote:
Bernhard M. Wiedemann schreef op 2021-09-12 10:34:
On 10/09/2021 11.33, William Edwards wrote:
- We have 2 backends so that we can do maintenance on one of them while users can still access their emails through the other backend. However, we found that stopping dovecot on one backend left users unable to access their mails.
Director doesn't do health checks.
Is there are recommendation on how to have a director with multiple backends with automatic fail-over? Are there scripts that do health-checks and call doveadm director commands?
https://doc.dovecot.org/configuration_manual/dovemon/
-- With kind regards,
William Edwards
There is also poolmon. https://github.com/brandond/poolmon
Aki
On 10/09/2021 12:19 Bernhard M. Wiedemann bwiedemann@suse.de wrote:
Hi,
I am one of the people taking over our new suse.de email setup (consisting of dovecot+rspamd+postfix) and wanted to report some issues we experience:
- we use dovecot-director to distribute users between 2 backend servers that share an NFS mount. We found that it proxies lmtp to a different backend than imap of the same user and that caused NFS stale-filehandle errors on the dovecot-uidlist. It then proceeds to re-generate the dovecot-uidlist with new UIDs that creates trouble for users.
a) shouldn't dovecot use locks (fcntl or flock) to protect such files from concurrent updates?
b) could it generate uidlist in a way that re-generating it, assigns the same UIDs again? E.g. via hash over file content
c) how to get dovecot-director to send all traffic for a user to one backend?
- We have 2 backends so that we can do maintenance on one of them while users can still access their emails through the other backend. However, we found that stopping dovecot on one backend left users unable to access their mails. Maybe this is related to how user auth works? How to get this HA setup right, so that we don't have a single point of failure?
grep PRETTY /etc/os-release PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"
rpm -q dovecot23 dovecot23-2.3.15
https://www.zq1.de/~bernhard/temp/dovecot/ has some sysreports.
Ciao Bernhard M.
It seems you are normalizing SQL usernames by doing SELECT username, but you are not doing it for LDAP.
See if the problem goes away with
pass_attrs = uid=username
Aki
On 10/09/2021 11.43, Aki Tuomi wrote:
https://www.zq1.de/~bernhard/temp/dovecot/ has some sysreports.
It seems you are normalizing SQL usernames by doing SELECT username, but you are not doing it for LDAP.
See if the problem goes away with
pass_attrs = uid=username
I added it to /etc/dovecot/dovecot-ldap.conf.ext but it did not help. I guess, because we map usernames "bwiedemann@foo" that are in sql, but ldap only has the plain usernames.
On 9/10/21 12:19, Bernhard M. Wiedemann wrote:
I added it to /etc/dovecot/dovecot-ldap.conf.ext but it did not help. I guess, because we map usernames "bwiedemann@foo" that are in sql, but ldap only has the plain usernames.
Why do you have such a mixed SQL-LDAP-setup? Why not just use one or the other?
Ciao, Michael.
On 10/09/2021 13:19 Bernhard M. Wiedemann bwiedemann@suse.de wrote:
On 10/09/2021 11.43, Aki Tuomi wrote:
https://www.zq1.de/~bernhard/temp/dovecot/ has some sysreports.
It seems you are normalizing SQL usernames by doing SELECT username, but you are not doing it for LDAP.
See if the problem goes away with
pass_attrs = uid=username
I added it to /etc/dovecot/dovecot-ldap.conf.ext but it did not help. I guess, because we map usernames "bwiedemann@foo" that are in sql, but ldap only has the plain usernames.
For director to work correctly, your usernames *must* map to same. The redirection is calculated by the full user, so if you bwiedemann and bwiedemann@foo, these two may end up in different backends.
Aki
You can setup the nfs users with NIS over idmapd. Then setup the dovecot server with NIS logins, so the the user login from any server over nis will get uid, gid, userhome etc, It mean it can read the nis users passwd and shadow.
On Fri, 10 Sep, 2021, 2:49 pm Bernhard M. Wiedemann, bwiedemann@suse.de wrote:
Hi,
I am one of the people taking over our new suse.de email setup (consisting of dovecot+rspamd+postfix) and wanted to report some issues we experience:
- we use dovecot-director to distribute users between 2 backend servers that share an NFS mount. We found that it proxies lmtp to a different backend than imap of the same user and that caused NFS stale-filehandle errors on the dovecot-uidlist. It then proceeds to re-generate the dovecot-uidlist with new UIDs that creates trouble for users.
a) shouldn't dovecot use locks (fcntl or flock) to protect such files from concurrent updates?
b) could it generate uidlist in a way that re-generating it, assigns the same UIDs again? E.g. via hash over file content
c) how to get dovecot-director to send all traffic for a user to one backend?
- We have 2 backends so that we can do maintenance on one of them while users can still access their emails through the other backend. However, we found that stopping dovecot on one backend left users unable to access their mails. Maybe this is related to how user auth works? How to get this HA setup right, so that we don't have a single point of failure?
grep PRETTY /etc/os-release PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"
rpm -q dovecot23 dovecot23-2.3.15
https://www.zq1.de/~bernhard/temp/dovecot/ has some sysreports.
Ciao Bernhard M.
participants (5)
-
Aki Tuomi
-
Bernhard M. Wiedemann
-
Michael Ströder
-
Sherin A
-
William Edwards