[Dovecot] dovecot-1.1.13 auth-worker killed
Hi,
I get these errors with dovecot 1.1.13 in my logs:
dovecot: Mar 27 12:00:34 Error: auth-worker(default): *** glibc detected *** dovecot-auth: double free or corruption (!prev): 0x00002b30d5a7dd90 *** dovecot: Mar 27 12:00:34 Error: auth(default): worker-server(jan,147.251.48.141): Aborted: Worker process died unexpectedly dovecot: Mar 27 12:00:34 Error: child 21838 (auth-worker) killed with signal 6 (core not dumped) dovecot: Mar 27 12:10:23 Error: child 22711 (auth-worker) killed with signal 11 (core not dumped) dovecot: Mar 27 12:10:23 Error: auth(default): worker-server(thomas,209.85.146.140): Aborted: Worker process died unexpectedly
Haven't seen this with previous versions.
But somehow I fail to get the core dump. I do 'ulimit -c unlimited' before starting dovecot, I set
soft core 8192
in limits.conf, /proc/sys/kernel/core_pattern is set to /tmp/core.hard core 8192
Regards, Jiri Novosad
My dovecont -n:
1.1.13: /packages/run.64/dovecot-1.1.13//etc/dovecot.conf
OS: Linux 2.6.18-92.1.22.el5 x86_64 Red Hat Enterprise Linux Server release 5.3 (Tikanga) ext3
base_dir: /var/run/dovecot/ log_path: /var/log/dovecot/log info_log_path: /var/log/dovecot/info_log protocols: pop3s imaps pop3 imap listen(default): * listen(imap): * listen(pop3): *:110 ssl_listen(default): ssl_listen(imap): ssl_listen(pop3): *:995 ssl_cert_file: /etc/ssl/cert-anxur.fi.muni.cz.pem ssl_key_file: /etc/ssl/serverkey-anxur.fi.muni.cz.pem ssl_parameters_regenerate: 24 login_dir: /var/run/dovecot/login login_executable(default): /packages/run.64/dovecot-1.1.13/libexec/dovecot/imap-login login_executable(imap): /packages/run.64/dovecot-1.1.13/libexec/dovecot/imap-login login_executable(pop3): /packages/run.64/dovecot-1.1.13/libexec/dovecot/pop3-login login_greeting: HI! login_greeting_capability(default): yes login_greeting_capability(imap): yes login_greeting_capability(pop3): no verbose_proctitle: yes first_valid_uid: 200 mail_location: mbox:/home/%u/mail/:INBOX=/var/mail/%u:INDEX=/home/%u/mail/.imap mail_debug: yes mail_full_filesystem_access: yes maildir_copy_with_hardlinks: no mbox_write_locks: fcntl mbox_dirty_syncs: no mbox_lazy_writes: no dbox_rotate_days: 0 mail_drop_priv_before_exec: yes mail_executable(default): /packages/run.64/dovecot-1.1.13/libexec/dovecot/imap mail_executable(imap): /packages/run.64/dovecot-1.1.13/libexec/dovecot/imap mail_executable(pop3): /packages/run.64/dovecot-1.1.13/libexec/dovecot/pop3 mail_plugin_dir(default): /packages/run.64/dovecot-1.1.13/lib/dovecot/imap mail_plugin_dir(imap): /packages/run.64/dovecot-1.1.13/lib/dovecot/imap mail_plugin_dir(pop3): /packages/run.64/dovecot-1.1.13//lib/dovecot/pop3 mail_log_max_lines_per_sec: 20 imap_max_line_length(default): 131072 imap_max_line_length(imap): 131072 imap_max_line_length(pop3): 65536 imap_client_workarounds(default): delay-newmail outlook-idle tb-extra-mailbox-sep imap_client_workarounds(imap): delay-newmail outlook-idle tb-extra-mailbox-sep imap_client_workarounds(pop3): pop3_uidl_format(default): %08Xu%08Xv pop3_uidl_format(imap): %08Xu%08Xv pop3_uidl_format(pop3): %08Xv%08Xu pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh auth default: executable: /packages/run.64/dovecot-1.1.13/libexec/dovecot/dovecot-auth verbose: yes debug: yes passdb: driver: pam args: dovecot userdb: driver: passwd
On Fri, 2009-03-27 at 12:31 +0100, Jiri Novosad wrote:
dovecot: Mar 27 12:00:34 Error: auth-worker(default): *** glibc detected *** dovecot-auth: double free or corruption (!prev): 0x00002b30d5a7dd90 ***
Heap corruption, not good. How often does this happen?
dovecot: Mar 27 12:00:34 Error: child 21838 (auth-worker) killed with signal 6 (core not dumped) dovecot: Mar 27 12:10:23 Error: child 22711 (auth-worker) killed with signal 11 (core not dumped)
It says it dumped core file, so it did write it somewhere. You just might not be finding it.
in limits.conf, /proc/sys/kernel/core_pattern is set to /tmp/core.
auth-workers are chdired to Dovecot's base_dir (e.g. /var/run/dovecot). The core file is normally written there, do you see it? Try removing path from core_pattern and see if it shows up there then?
What PAM configuration are you using?
On 4/1/2009 5:25 PM, Timo Sirainen wrote:
dovecot: Mar 27 12:00:34 Error: child 21838 (auth-worker) killed with signal 6 (core not dumped) dovecot: Mar 27 12:10:23 Error: child 22711 (auth-worker) killed with signal 11 (core not dumped)
It says it dumped core file, so it did write it somewhere. You just might not be finding it.
? (core not dumped) means it *did* dump it? Ok, so what am I missing?
--
Best regards,
Charles
On Thu, 2009-04-02 at 06:28 -0400, Charles Marcus wrote:
It means I'm blind.
Maybe this patch will cause it to dump core: http://hg.dovecot.org/dovecot-1.1/rev/387fc4f06956
On 4/2/2009 2:01 PM, Timo Sirainen wrote:
? (core not dumped) means it *did* dump it? Ok, so what am I missing?
It means I'm blind.
Heh... well, I'm not at all ok with the idea of you going blind, but I'm really glad to know that 'they' didn't change the meaning of 'not' on me... I'd *really* be in trouble then...
:)
--
Best regards,
Charles
Timo Sirainen wrote:
Thanks, after applying I get the core.
This backtrack is from the signal 11 - killed:
#0 0x0000003d5e60d6fc in ?? () from /lib64/libselinux.so.1 #1 0x0000003d5e60d86b in matchpathcon () from /lib64/libselinux.so.1 #2 0x0000003d64203d3b in ?? () from /usr/lib64/libkrb5support.so.0 #3 0x0000003d64204064 in krb5int_labeled_fopen () from /usr/lib64/libkrb5support.so.0 #4 0x0000003d63679188 in profile_update_file_data () from /usr/lib64/libkrb5.so.3 #5 0x0000003d63679f3e in profile_open_file () from /usr/lib64/libkrb5.so.3 #6 0x0000003d6367d3dc in profile_init () from /usr/lib64/libkrb5.so.3 #7 0x0000003d636715e2 in ?? () from /usr/lib64/libkrb5.so.3 #8 0x0000003d63671742 in krb5_os_init_context () from /usr/lib64/libkrb5.so.3 #9 0x0000003d6365a91e in ?? () from /usr/lib64/libkrb5.so.3 #10 0x00002ba68be0d044 in ?? () from /lib64/security/pam_krb5.so #11 0x00002ba68be096d2 in pam_sm_authenticate () from /lib64/security/pam_krb5.so #12 0x0000003d6a802dc7 in _pam_dispatch () from /lib64/libpam.so.0 #13 0x0000003d6a8026d2 in pam_authenticate () from /lib64/libpam.so.0 #14 0x0000000000419d10 in pam_verify_plain (request=0x42e69d8, password=<value optimized out>, callback=0x411cd0 <verify_plain_callback>) at passdb-pam.c:163 #15 0x000000000041180c in auth_worker_input (client=0x2183cc0) at auth-worker-client.c:178 #16 0x0000000000425d48 in io_loop_handler_run (ioloop=<value optimized out>) at ioloop-epoll.c:203 #17 0x0000000000424e8d in io_loop_run (ioloop=0x2181350) at ioloop.c:336 #18 0x0000000000413dff in main (argc=<value optimized out>, argv=<value optimized out>) at main.c:349
and from the signal 6:
#0 0x0000003d5ce30215 in raise () from /lib64/libc.so.6 #1 0x0000003d5ce31cc0 in abort () from /lib64/libc.so.6 #2 0x0000003d5ce6a7fb in __libc_message () from /lib64/libc.so.6 #3 0x0000003d5ce71ce2 in _int_free () from /lib64/libc.so.6 #4 0x0000003d5ce7590c in free () from /lib64/libc.so.6 #5 0x0000003d5ceb1d56 in re_compile_internal () from /lib64/libc.so.6 #6 0x0000003d5ceb2952 in regcomp () from /lib64/libc.so.6 #7 0x0000003d5e60cb2d in ?? () from /lib64/libselinux.so.1 #8 0x0000003d5e60d0a9 in matchpathcon_init_prefix () from /lib64/libselinux.so.1 #9 0x0000003d5e60d668 in ?? () from /lib64/libselinux.so.1 #10 0x0000003d5e60d86b in matchpathcon () from /lib64/libselinux.so.1 #11 0x0000003d64203d3b in ?? () from /usr/lib64/libkrb5support.so.0 #12 0x0000003d64204064 in krb5int_labeled_fopen () from /usr/lib64/libkrb5support.so.0 #13 0x0000003d63679188 in profile_update_file_data () from /usr/lib64/libkrb5.so.3 #14 0x0000003d63679f3e in profile_open_file () from /usr/lib64/libkrb5.so.3 #15 0x0000003d6367d3dc in profile_init () from /usr/lib64/libkrb5.so.3 #16 0x0000003d636715e2 in ?? () from /usr/lib64/libkrb5.so.3 #17 0x0000003d63671742 in krb5_os_init_context () from /usr/lib64/libkrb5.so.3 #18 0x0000003d6365a91e in ?? () from /usr/lib64/libkrb5.so.3 #19 0x00002ab47f575044 in ?? () from /lib64/security/pam_krb5.so #20 0x00002ab47f5716d2 in pam_sm_authenticate () from /lib64/security/pam_krb5.so #21 0x0000003d6a802dc7 in _pam_dispatch () from /lib64/libpam.so.0 #22 0x0000003d6a8026d2 in pam_authenticate () from /lib64/libpam.so.0 #23 0x0000000000419d10 in pam_verify_plain (request=0x2ab487ff8748, password=<value optimized out>, callback=0x411cd0 <verify_plain_callback>) at passdb-pam.c:163 #24 0x000000000041180c in auth_worker_input (client=0x2ab47f023cc0) at auth-worker-client.c:178 #25 0x0000000000425d48 in io_loop_handler_run (ioloop=<value optimized out>) at ioloop-epoll.c:203 #26 0x0000000000424e8d in io_loop_run (ioloop=0x2ab47f021350) at ioloop.c:336 #27 0x0000000000413dff in main (argc=<value optimized out>, argv=<value optimized out>) at main.c:349
So it looks like a bug in libc. Or selinux? Interesting that it doesn't happen with dovecot version 1.1.12.
Timo Sirainen wrote:
Yes, sorry, my pam setup:
cat /etc/pam.d/dovecot
auth include system-auth auth sufficient pam_krb5.so account required pam_nologin.so account required pam_ldap_fi.so # site-specific, checks host=? account include system-auth
cat /etc/pam.d/system-auth
auth required pam_env.so auth sufficient pam_unix.so nullok try_first_pass auth sufficient pam_krb5.so use_first_pass auth sufficient pam_ldap.so use_first_pass
auth requisite pam_succeed_if.so uid >= 500 quiet auth required pam_deny.so
account sufficient pam_unix.so account sufficient pam_succeed_if.so uid < 500 quiet account sufficient pam_ldap.so account sufficient pam_krb5.so
account required pam_permit.so
password requisite pam_cracklib.so try_first_pass retry=3 password sufficient pam_unix.so md5 shadow nullok try_first_pass use_authtok password sufficient pam_ldap.so use_authtok password sufficient pam_krb5.so use_authtok password required pam_deny.so
session optional pam_keyinit.so revoke session required pam_limits.so session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session sufficient pam_unix.so session sufficient pam_ldap.so session optional pam_krb5.so
I ran:
ldd /export/packages/run.linux.64/dovecot-1.1.13/libexec/dovecot/dovecot-auth
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x0000003d6d800000)
libpam.so.0 => /lib64/libpam.so.0 (0x0000003d6a800000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003d5d600000)
libc.so.6 => /lib64/libc.so.6 (0x0000003d5ce00000)
libaudit.so.0 => /lib64/libaudit.so.0 (0x0000003d65200000)
/lib64/ld-linux-x86-64.so.2 (0x0000003d5ca00000)and then:
ldd /export/packages/run.linux.64/dovecot-1.1.12/libexec/dovecot/dovecot-auth
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x0000003d6d800000)
libpam.so.0 => /lib64/libpam.so.0 (0x0000003d6a800000)
libldap-2.3.so.0 => /usr/lib64/libldap-2.3.so.0 (0x0000003d5d200000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003d5d600000)
libc.so.6 => /lib64/libc.so.6 (0x0000003d5ce00000)
liblber-2.3.so.0 => /usr/lib64/liblber-2.3.so.0 (0x0000003d5da00000)
libaudit.so.0 => /lib64/libaudit.so.0 (0x0000003d65200000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003d61200000)
libsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x0000003d72c00000)
libssl.so.6 => /lib64/libssl.so.6 (0x0000003d67a00000)
libcrypto.so.6 => /lib64/libcrypto.so.6 (0x0000003d62200000)
/lib64/ld-linux-x86-64.so.2 (0x0000003d5ca00000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x0000003d63a00000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x0000003d63600000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000003d61a00000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x0000003d63e00000)
libz.so.1 => /usr/lib64/libz.so.1 (0x0000003d5de00000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x0000003d64200000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x0000003d62600000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x0000003d5e600000)
libsepol.so.1 => /lib64/libsepol.so.1 (0x0000003d5ea00000)which reminded me, that there *were* some changes between the versions :).
This is how I configured 1.1.12: ./configure --without-nss --with-storages=mbox,raw --with-ldap=yes --without-docs --prefix=/packages/run.64/dovecot-1.1.12/
and 1.1.13: ./configure --with-storages=mbox,raw --without-docs --prefix=/packages/run.64/dovecot-1.1.13/
Now when I use the same configuration with 13 as with 12, there are no errors.
Jiri Novosad
participants (3)
-
Charles Marcus
-
Jiri Novosad
-
Timo Sirainen