Re: Auth process sometimes stop responding after upgrade
On 8 Sep 2018, at 15.18, Simone Lazzaris <simone.lazzaris@qcom.it> wrote:
Timo, unfortunately the patch doesn't compile; I've moved the declaration of "conn" one line up to make it work.
Oops, I guess I was too much in a hurry to even compile it. Here's a new patch that compiles and passes our director CI tests.
In data lunedì 10 settembre 2018 09:58:50 CEST, Timo Sirainen ha scritto:
On 8 Sep 2018, at 15.18, Simone Lazzaris <simone.lazzaris@qcom.it> wrote:
Timo, unfortunately the patch doesn't compile; I've moved the declaration of "conn" one line up to make it work.
Oops, I guess I was too much in a hurry to even compile it. Here's a new patch that compiles and passes our director CI tests.
This one is better :) I've compiled and installed the patched version on one VM and it's working. In the next hours, if everything is ok, I'll percolate the change on the whole cluster. Let's see if the issue appears again.
The other cluster have been downgraded (saturday night) to 2.2.30.2 and it's working fine: I couldn't afford to have issues on that one.
-- *Simone Lazzaris* *Qcom S.p.A.*
In data lunedì 10 settembre 2018 09:58:50 CEST, Timo Sirainen ha scritto:
On 8 Sep 2018, at 15.18, Simone Lazzaris <simone.lazzaris@qcom.it> wrote:
Timo, unfortunately the patch doesn't compile; I've moved the declaration of "conn" one line up to make it work.
Oops, I guess I was too much in a hurry to even compile it. Here's a new patch that compiles and passes our director CI tests.
Hi Timo; after 24 hours of field testing, I can say that the issue is mostly gone. I say "mostly" because the service is working as far as the user is concerned, but I see some strange going on in the logs.
Grepping "director" in the log file, I can see that there are some panic and some comunication errors:
Sep 11 03:24:55 imap-front4 dovecot: director: doveadm: Host 192.168.1.143 vhost count changed from 100 to 0 Sep 11 03:24:55 imap-front4 dovecot: director: doveadm: Host 192.168.1.219 vhost count changed from 100 to 0 Sep 11 03:24:55 imap-front4 dovecot: director: doveadm: Host 192.168.1.218 vhost count changed from 100 to 0 Sep 11 03:24:55 imap-front4 dovecot: director: doveadm: Host 192.168.1.216 vhost count changed from 100 to 0 Sep 11 03:24:55 imap-front4 dovecot: director: director(212.183.164.161:9090/right): Host 192.168.1.145 vhost count changed from 100 to 0 Sep 11 03:24:55 imap-front4 dovecot: director: doveadm: Host 192.168.1.217 vhost count changed from 100 to 0 Sep 11 03:24:55 imap-front4 dovecot: director: doveadm: Host 192.168.1.144 vhost count changed from 100 to 0 Sep 11 03:24:55 imap-front4 dovecot: director: doveadm: Host 192.168.1.145 vhost count changed from 0 to 0 Sep 11 03:24:55 imap-front4 dovecot: director: doveadm: Host 192.168.1.142 vhost count changed from 100 to 0 Sep 11 03:25:09 imap-front4 dovecot: director: director(212.183.164.161:9090/right): Host 192.168.1.143 vhost count changed from 0 to 100 Sep 11 03:25:09 imap-front4 dovecot: director: Error: Director 212.183.164.161:9090/right disconnected: Connection closed (bytes in=1116368, bytes out=1182555, 0+27319 USERs received, last input 0.000 s ago, last output 0.000 s ago, connected 4602.589 s ago, 481 peak output buffer size, 1.948 CPU secs since connected) Sep 11 03:25:09 imap-front4 dovecot: director: Connecting to 212.183.164.161:9090 (as 212.183.164.164): Reconnecting after disconnection Sep 11 03:25:09 imap-front4 dovecot: director: Error: Director 212.183.164.161:9090/out disconnected: Connection closed: read(size=968) failed: Connection reset by peer (bytes in=56, bytes out=59143, 0+0 USERs received, 1556 USERs sent in handshake, last input 0.002 s ago, last output 0.002 s ago, connected 0.024 s ago, 8190 peak output buffer size, 0.004 CPU secs since connected, handshake DONE not received) Sep 11 03:25:09 imap-front4 dovecot: director: Connecting to 212.183.164.162:9090 (as 212.183.164.164): Reconnecting after disconnection Sep 11 03:25:09 imap-front4 dovecot: director: director(212.183.164.162:9090/out): Handshake finished in 0.006 secs (bytes in=61, bytes out=59173, 0+0 USERs received, 1556 USERs sent in handshake, last input 0.000 s ago, last output 0.003 s ago, connected 0.006 s ago, 8190 peak output buffer size, 0.000 CPU secs since connected) Sep 11 03:25:10 imap-front4 dovecot: director: Connecting to 212.183.164.161:9090 (as 212.183.164.164): Received CONNECT request from 212.183.164.162:9090/right - replacing current right 212.183.164.162:9090/right Sep 11 03:25:10 imap-front4 dovecot: director: director(212.183.164.161:9090/out): Handshake finished in 0.004 secs (bytes in=61, bytes out=59332, 0+0 USERs received, 1561 USERs sent in handshake, last input 0.000 s ago, last output 0.004 s ago, connected 0.004 s ago, 8190 peak output buffer size, 0.000 CPU secs since connected) Sep 11 03:25:10 imap-front4 dovecot: director: director(212.183.164.161:9090/right): Host 192.168.1.216 vhost count changed from 0 to 100 Sep 11 03:25:10 imap-front4 dovecot: director: Error: Director 212.183.164.161:9090/right disconnected: Connection closed: read(size=558) failed: Connection reset by peer (bytes in=466, bytes out=60271, 0+6 USERs received, 1561 USERs sent in handshake, last input 0.001 s ago, last output 0.000 s ago, connected 0.553 s ago, 8190 peak output buffer size, 0.000 CPU secs since connected) Sep 11 03:25:10 imap-front4 dovecot: director: Connecting to 212.183.164.162:9090 (as 212.183.164.164): Reconnecting after disconnection Sep 11 03:25:10 imap-front4 dovecot: director: director(212.183.164.162:9090/out): Handshake finished in 0.005 secs (bytes in=61, bytes out=59372, 0+0 USERs received, 1562 USERs sent in handshake, last input 0.000 s ago, last output 0.005 s ago, connected 0.005 s ago, 8192 peak output buffer size, 0.000 CPU secs since connected) Sep 11 03:25:10 imap-front4 dovecot: director: Connecting to 212.183.164.161:9090 (as 212.183.164.164): Received CONNECT request from 212.183.164.162:9090/right - replacing current right 212.183.164.162:9090/right Sep 11 03:25:10 imap-front4 dovecot: director: director(212.183.164.161:9090/out): Handshake finished in 0.007 secs (bytes in=61, bytes out=59372, 0+0 USERs received, 1562 USERs sent in handshake, last input 0.000 s ago, last output 0.003 s ago, connected 0.007 s ago, 8516 peak output buffer size, 0.004 CPU secs since connected) Sep 11 03:25:25 imap-front4 dovecot: director: doveadm: Host 192.168.1.144 vhost count changed from 0 to 100 Sep 11 03:25:25 imap-front4 dovecot: director: Panic: file doveadm-connection.c: line 1097 (doveadm_connection_deinit): assertion failed: (conn->to_ring_sync_abort == NULL) Sep 11 03:25:25 imap-front4 dovecot: director: Fatal: master: service(director): child 2237 killed with signal 6 (core dumps disabled) Sep 11 03:25:25 imap-front4 dovecot: director: Connecting to 212.183.164.161:9090 (as 212.183.164.164): Initial connection Sep 11 03:25:25 imap-front4 dovecot: director: Incoming connection from director 212.183.164.163/in Sep 11 03:25:25 imap-front4 dovecot: director: Panic: file doveadm-connection.c: line 1097 (doveadm_connection_deinit): assertion failed: (conn->to_ring_sync_abort == NULL) Sep 11 03:25:25 imap-front4 dovecot: director: Fatal: master: service(director): child 4392 killed with signal 6 (core dumps disabled) Sep 11 03:25:25 imap-front4 dovecot: director: Connecting to 212.183.164.161:9090 (as 212.183.164.164): Initial connection Sep 11 03:25:25 imap-front4 dovecot: director: Incoming connection from director 212.183.164.163/in Sep 11 03:25:25 imap-front4 dovecot: director: Panic: file doveadm-connection.c: line 1097 (doveadm_connection_deinit): assertion failed: (conn->to_ring_sync_abort == NULL) Sep 11 03:25:25 imap-front4 dovecot: director: Fatal: master: service(director): child 4393 killed with signal 6 (core dumps disabled) Sep 11 03:25:25 imap-front4 dovecot: director: Connecting to 212.183.164.161:9090 (as 212.183.164.164): Initial connection Sep 11 03:25:25 imap-front4 dovecot: director: Incoming connection from director 212.183.164.163/in Sep 11 03:25:25 imap-front4 dovecot: director: Panic: file doveadm-connection.c: line 1097 (doveadm_connection_deinit): assertion failed: (conn->to_ring_sync_abort == NULL) Sep 11 03:25:25 imap-front4 dovecot: director: Fatal: master: service(director): child 4394 killed with signal 6 (core dumps disabled) Sep 11 03:25:25 imap-front4 dovecot: director: Connecting to 212.183.164.161:9090 (as 212.183.164.164): Initial connection
On 11 Sep 2018, at 10.57, Simone Lazzaris <s.lazzaris@interactive.eu> wrote:
Sep 11 03:25:55 imap-front4 dovecot: director: Panic: file doveadm-connection.c: line 1097 (doveadm_connection_deinit): assertion failed: (conn->to_ring_sync_abort == NULL) Sep 11 03:25:55 imap-front4 dovecot: director: Fatal: master: service(director): child 4395 killed with signal 6 (core dumps disabled)
It's crashing. Can you get gdb backtrace? First enable core dumps. https://dovecot.org/bugreport.html#coredumps <https://dovecot.org/bugreport.html#coredumps>
In data martedì 11 settembre 2018 10:46:30 CEST, Timo Sirainen ha scritto:
On 11 Sep 2018, at 10.57, Simone Lazzaris <s.lazzaris@interactive.eu> wrote:
Sep 11 03:25:55 imap-front4 dovecot: director: Panic: file doveadm-connection.c: line 1097 (doveadm_connection_deinit): assertion failed: (conn->to_ring_sync_abort == NULL) Sep 11 03:25:55 imap-front4 dovecot: director: Fatal: master: service(director): child 4395 killed with signal 6 (core dumps disabled) It's crashing. Can you get gdb backtrace? First enable core dumps. https://dovecot.org/bugreport.html#coredumps <https://dovecot.org/bugreport.html#coredumps>
Hi all, again;
I've enabled the core dumps and let it go for some day waiting for the issue to reoccur.
Meantime I've also upgraded the poolmon script, as Sami suggested.
It seems that the upgrade has scared the issue away, because it no longer occurred.
Maybe the problem is related to the way the old poolmon talked to the director daemon? I'm not very inclined to downgrade poolmon to catch a traceback, but can do if neccessary.
-- *Simone Lazzaris* *Qcom S.p.A.* simone.lazzaris@qcom.it[1] | www.qcom.it[2]
- LinkedIn[3]* | *Facebook*[4] [5]
[1] mailto:simone.lazzaris@qcom.it [2] https://www.qcom.it [3] https://www.linkedin.com/company/qcom-spa [4] http://www.facebook.com/qcomspa [5] https://www.qcom.it/includes/email-banner.gif
In data martedì 11 settembre 2018 10:46:30 CEST, Timo Sirainen ha scritto:
On 11 Sep 2018, at 10.57, Simone Lazzaris <s.lazzaris@interactive.eu> wrote:
Sep 11 03:25:55 imap-front4 dovecot: director: Panic: file doveadm-connection.c: line 1097 (doveadm_connection_deinit): assertion failed: (conn->to_ring_sync_abort == NULL) Sep 11 03:25:55 imap-front4 dovecot: director: Fatal: master: service(director): child 4395 killed with signal 6 (core dumps disabled) It's crashing. Can you get gdb backtrace? First enable core dumps. https://dovecot.org/bugreport.html#coredumps <https://dovecot.org/bugreport.html#coredumps>
Hi all, again;
I've enabled the core dumps and let it go for some day waiting for the issue to reoccur.
Meantime I've also upgraded the poolmon script, as Sami suggested.
It seems that the upgrade has scared the issue away, because it no longer occurred.
Maybe the problem is related to the way the old poolmon talked to the director daemon? I'm not very inclined to downgrade poolmon to catch a traceback, but can do if neccessary.
-- *Simone Lazzaris* *Qcom S.p.A.* simone.lazzaris@qcom.it[1] | www.qcom.it[2]
- LinkedIn[3]* | *Facebook*[4] [5]
[1] mailto:simone.lazzaris@qcom.it [2] https://www.qcom.it [3] https://www.linkedin.com/company/qcom-spa [4] http://www.facebook.com/qcomspa [5] https://www.qcom.it/includes/email-banner.gif
Hi all, again;
I've enabled the core dumps and let it go for some day waiting for the issue to reoccur.
Meantime I've also upgraded the poolmon script, as Sami suggested.
It seems that the upgrade has scared the issue away, because it no longer occurred.
Maybe the problem is related to the way the old poolmon talked to the director daemon? I'm not very inclined to downgrade poolmon to catch a traceback, but can do if neccessary.
Well, maybe it's not necessary ;) I've performed some maintenance operations on the backends and that triggered the crash. It seems that something goes wrong where one backend come back online.
Unfortunately, the core was not dumped.... And I don't know what to do: the director service was not chrooted, and ulimit -c is unlimited.
From the log file:
Sep 18 12:21:46 imap-front4 dovecot: director: Panic: file doveadm-connection.c: line 1097 (doveadm_connection_deinit): assertion failed: (conn->to_ring_sync_abort == NULL) Sep 18 12:21:46 imap-front4 dovecot: director: Error: Raw backtrace: /usr/local/lib/dovecot/ libdovecot.so.0(+0xa15be) [0xb77345be] -> /usr/local/lib/dovecot/libdovecot.so. 0(+0xa1641) [0xb7734641] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0xb76ba35e] -> dovecot/director() [0x80574f7] -> dovecot/director() [0x8057f03] -> /usr/local/lib/ dovecot/libdovecot.so.0(io_loop_call_io+0x6b) [0xb774d3db] -> /usr/local/lib/dovecot/ libdovecot.so.0(io_loop_handler_run_internal+0xfe) [0xb774ee1e] -> /usr/local/lib/dovecot/ libdovecot.so.0(io_loop_handler_run+0x46) [0xb774d496] -> /usr/local/lib/dovecot/ libdovecot.so.0(io_loop_run+0x48) [0xb774d658] -> /usr/local/lib/dovecot/libdovecot.so. 0(master_service_run+0x2e) [0xb76c645e] -> dovecot/director(main+0x49e) [0x804cf5e] -> /lib/i386-linux-gnu/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0xb753be46] -> dovecot/ director() [0x804d081] Sep 18 12:21:46 imap-front4 dovecot: director: Fatal: master: service(director): child 7941 killed with signal 6 (core not dumped) Sep 18 12:21:46 imap-front4 dovecot: director: Connecting to 212.183.164.161:9090 (as 212.183.164.164): Initial connection Sep 18 12:21:46 imap-front4 dovecot: director: Incoming connection from director 212.183.164.163/in
My current config:
root@imap-front4:~# doveconf -n # 2.2.36 (1f10bfa63): /usr/local/etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-686-pae i686 Debian 7.11 # Hostname: imap-front4 auth_mechanisms = plain login digest-md5 cram-md5 apop scram-sha-1 auth_verbose = yes auth_verbose_passwords = plain base_dir = /var/run/dovecot/ default_login_user = nobody director_doveadm_port = 9091 director_mail_servers = 192.168.1.142 192.168.1.143 192.168.1.216 192.168.1.217 192.168.1.218 192.168.1.219 director_servers = 212.183.164.161 212.183.164.162 212.183.164.163 212.183.164.164 disable_plaintext_auth = no listen = * passdb { args = /usr/local/etc/dovecot/sql.conf driver = sql } protocols = imap pop3 service director { chroot = fifo_listener login/proxy-notify { mode = 0666 } inet_listener { port = 9090 } unix_listener director-userdb { mode = 0600 } unix_listener login/director { mode = 0666 } } service imap-login { executable = imap-login director service_count = 0 vsz_limit = 128 M } service pop3-login { executable = pop3-login director service_count = 0 vsz_limit = 128 M } ssl_cert = </usr/local/etc/dovecot/imapd.pem ssl_key = # hidden, use -P to show it ssl_protocols = !SSlv2 !SSLv3 syslog_facility = local5 userdb { driver = prefetch }
*Simone Lazzaris* *Qcom S.p.A.* simone.lazzaris@qcom.it[1] | www.qcom.it[2]
- LinkedIn[3]* | *Facebook*[4] [5]
On 18 Sep 2018, at 13.29, Simone Lazzaris <s.lazzaris@interactive.eu> wrote:
Hi all, again;
I've enabled the core dumps and let it go for some day waiting for the issue to reoccur.
Meantime I've also upgraded the poolmon script, as Sami suggested.
It seems that the upgrade has scared the issue away, because it no longer occurred.
Maybe the problem is related to the way the old poolmon talked to the director daemon? I'm not very inclined to downgrade poolmon to catch a traceback, but can do if neccessary.
Well, maybe it's not necessary ;) I've performed some maintenance operations on the backends and that triggered the crash. It seems that something goes wrong where one backend come back online.
It's weird how easily you can reproduce the crash. I've ran all kinds of (stress) tests and I can't reproduce this crash. I was able to reproduce the original hang though.
Unfortunately, the core was not dumped.... And I don't know what to do: the director service was not chrooted, and ulimit -c is unlimited.
Do you have: sysctl -w fs.suid_dumpable=2
Alas, I've set fs.suid_dumpable to 2 but the core is not dumped. So far I've checked:
- ulimit -c unlimited, done
- /proc/sys/kernel/core_pattern is set to /var/tmp/core.%p
- /var/tmp is chmod 1777
- daemon is not chrooted
- sysctl -w fs.suid_dumpable=2
- dir /var/tmp is empty and filesystem has 2GB free
This is the logfile:
Sep 18 13:54:22 imap-front4 dovecot: director: doveadm: Host 192.168.1.145 changed down (vhost_count=100 last_updown_change=0) Sep 18 13:54:52 imap-front4 dovecot: director: doveadm: Host 192.168.1.145 changed up (vhost_count=100 last_updown_change=1537271662) Sep 18 13:54:52 imap-front4 dovecot: director: Panic: file doveadm-connection.c: line 1097 (doveadm_connection_deinit): assertion failed: (conn->to_ring_sync_abort == NULL) Sep 18 13:54:52 imap-front4 dovecot: director: Error: Raw backtrace: /usr/local/lib/dovecot/ libdovecot.so.0(+0xa15be) [0xb76fa5be] -> /usr/local/lib/dovecot/libdovecot.so. 0(+0xa1641) [0xb76fa641] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0xb768035e] -> dovecot/director() [0x80574f7] -> dovecot/director() [0x8057f03] -> /usr/local/lib/ dovecot/libdovecot.so.0(io_loop_call_io+0x6b) [0xb77133db] -> /usr/local/lib/dovecot/ libdovecot.so.0(io_loop_handler_run_internal+0xfe) [0xb7714e1e] -> /usr/local/lib/dovecot/ libdovecot.so.0(io_loop_handler_run+0x46) [0xb7713496] -> /usr/local/lib/dovecot/ libdovecot.so.0(io_loop_run+0x48) [0xb7713658] -> /usr/local/lib/dovecot/libdovecot.so. 0(master_service_run+0x2e) [0xb768c45e] -> dovecot/director(main+0x49e) [0x804cf5e] -> /lib/i386-linux-gnu/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0xb7501e46] -> dovecot/ director() [0x804d081] Sep 18 13:54:52 imap-front4 dovecot: director: Fatal: master: service(director): child 8059 killed with signal 6 (core not dumped) Sep 18 13:54:52 imap-front4 dovecot: director: Connecting to 212.183.164.161:9090 (as 212.183.164.164): Initial connection Sep 18 13:54:52 imap-front4 dovecot: director: Incoming connection from director 212.183.164.163/in Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.142 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.143 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.144 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.145 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.216 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.217 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.218 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.219 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.161:9090/out): Handshake finished in 0.001 secs (bytes in=61, bytes out=791, 0+0 USERs received, last input 0.000 s ago, last output 0.001 s ago, connected 0.001 s ago, 408 peak output buffer size, 0.000 CPU secs since connected) Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Handshake finished in 0.006 secs (bytes in=111411, bytes out=56, 2940+0 USERs received, last input 0.000 s ago, last output 0.006 s ago, connected 0.006 s ago, 0 peak output buffer size, 0.004 CPU secs since connected)
I can confirm that I can trigger the issue having one of the backends flapping down/up.
*Simone Lazzaris* *Qcom S.p.A.* simone.lazzaris@qcom.it[1] | www.qcom.it[2]
- LinkedIn[3]* | *Facebook*[4] [5]
[1] mailto:simone.lazzaris@qcom.it [2] https://www.qcom.it [3] https://www.linkedin.com/company/qcom-spa [4] http://www.facebook.com/qcomspa [5] https://www.qcom.it/includes/email-banner.gif
If you are using systemd, create /etc/systemd/system/dovecot.service.d/limits.conf and put [Service]LimitCORE=infinity
and run systemctl daemon-reloadsystemctl restart dovecot ---Aki TuomiDovecot oy -------- Original message --------From: Simone Lazzaris <s.lazzaris@interactive.eu> Date: 18/09/2018 15:01 (GMT+02:00) To: Timo Sirainen <tss@iki.fi> Cc: dovecot@dovecot.org Subject: Re: Auth process sometimes stop responding after upgrade
Alas, I've set fs.suid_dumpable to 2 but the core is not dumped. So far I've checked:
- ulimit -c unlimited, done
- /proc/sys/kernel/core_pattern is set to /var/tmp/core.%p
- /var/tmp is chmod 1777
- daemon is not chrooted
- sysctl -w fs.suid_dumpable=2
- dir /var/tmp is empty and filesystem has 2GB free This is the logfile: Sep 18 13:54:22 imap-front4 dovecot: director: doveadm: Host 192.168.1.145 changed down (vhost_count=100 last_updown_change=0) Sep 18 13:54:52 imap-front4 dovecot: director: doveadm: Host 192.168.1.145 changed up (vhost_count=100 last_updown_change=1537271662) Sep 18 13:54:52 imap-front4 dovecot: director: Panic: file doveadm-connection.c: line 1097 (doveadm_connection_deinit): assertion failed: (conn->to_ring_sync_abort == NULL) Sep 18 13:54:52 imap-front4 dovecot: director: Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0xa15be) [0xb76fa5be] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xa1641) [0xb76fa641] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0xb768035e] -> dovecot/director() [0x80574f7] -> dovecot/director() [0x8057f03] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x6b) [0xb77133db] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xfe) [0xb7714e1e] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x46) [0xb7713496] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x48) [0xb7713658] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x2e) [0xb768c45e] -> dovecot/director(main+0x49e) [0x804cf5e] -> /lib/i386-linux-gnu/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0xb7501e46] -> dovecot/director() [0x804d081] Sep 18 13:54:52 imap-front4 dovecot: director: Fatal: master: service(director): child 8059 killed with signal 6 (core not dumped) Sep 18 13:54:52 imap-front4 dovecot: director: Connecting to 212.183.164.161:9090 (as 212.183.164.164): Initial connection Sep 18 13:54:52 imap-front4 dovecot: director: Incoming connection from director 212.183.164.163/in Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.142 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.143 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.144 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.145 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.216 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.217 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.218 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Host 192.168.1.219 vhost count changed from 100 to 100 Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.161:9090/out): Handshake finished in 0.001 secs (bytes in=61, bytes out=791, 0+0 USERs received, last input 0.000 s ago, last output 0.001 s ago, connected 0.001 s ago, 408 peak output buffer size, 0.000 CPU secs since connected) Sep 18 13:54:52 imap-front4 dovecot: director: director(212.183.164.163/in): Handshake finished in 0.006 secs (bytes in=111411, bytes out=56, 2940+0 USERs received, last input 0.000 s ago, last output 0.006 s ago, connected 0.006 s ago, 0 peak output buffer size, 0.004 CPU secs since connected) I can confirm that I can trigger the issue having one of the backends flapping down/up.
-- Simone Lazzaris Responsabile datacenter
Qcom S.p.A. Via Roggia Vignola, 9 | 24047 Treviglio (BG) T +39036347905 | D +3903631970352| M +393938111237 simone.lazzaris@qcom.it | www.qcom.it
Qcom Official Pages LinkedIn | Facebook
participants (4)
-
Aki Tuomi
-
Simone Lazzaris
-
Simone Lazzaris
-
Timo Sirainen