[Dovecot] dovecot/auth killed with signal 6

[Edward Burr]

I've had this working for over a month now with no problems. Then I
suffered a power outage today, and now dovecot/auth dies as soon as I start
dovecot. I've spent the past three hours searching google, but can't find
even a hint at what's wrong with my server. As far as I can tell, this is
the only thing that has failed after bringing the server back up when power
was restored. Can anyone help me figure out what is wrong?

The maillog records:
May  9 20:27:37 box dovecot: auth: Debug: Loading modules from directory:
/usr/lib64/dovecot/auth
May  9 20:27:37 box dovecot: auth: Debug: Module loaded:
/usr/lib64/dovecot/auth/libauthdb_ldap.so
May  9 20:27:37 box dovecot: auth: Debug: Module loaded:
/usr/lib64/dovecot/auth/libdriver_mysql.so
May  9 20:27:37 box dovecot: auth: Debug: Module loaded:
/usr/lib64/dovecot/auth/libdriver_pgsql.so
May  9 20:27:37 box dovecot: auth: Debug: Module loaded:
/usr/lib64/dovecot/auth/libdriver_sqlite.so
May  9 20:27:37 box dovecot: auth: Debug: Module loaded:
/usr/lib64/dovecot/auth/libmech_gssapi.so
May  9 20:27:37 box dovecot: auth: Panic: io_add(0x1) called twice fd=13,
callback=0x3016833290 -> 0x3016836cd0
May  9 20:27:37 box dovecot: auth: Error: Raw backtrace:
/usr/lib64/dovecot/libdovecot.so.0() [0x301683ca6a] ->
/usr/lib64/dovecot/libdovecot.so.0() [0x301683ca
b6] -> /usr/lib64/dovecot/libdovecot.so.0() [0x3016816dba] ->
/usr/lib64/dovecot/libdovecot.so.0(ioloop_iolist_add+0x7f) [0x301684872f]
-> /usr/lib64/dovecot/li
bdovecot.so.0(io_loop_handle_add+0x39) [0x3016849149] ->
/usr/lib64/dovecot/libdovecot.so.0(io_add+0xa3) [0x3016848593] ->
/usr/lib64/dovecot/libdovecot.so.0(ma
ster_service_io_listeners_add+0x68) [0x3016835b88] ->
/usr/lib64/dovecot/libdovecot.so.0(master_service_init_finish+0x192)
[0x30168363c2] -> dovecot/auth(main+0
x207) [0x415567] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x3016c1ecdd]
-> dovecot/auth() [0x409aa9]
May  9 20:27:37 box dovecot: master: Error: service(auth): child 25615
killed with signal 6 (core dumped)
May  9 20:27:37 box dovecot: master: Error: service(auth): command startup
failed, throttling

A core dump was produced.  The backtrace shows:
# gdb /usr/libexec/dovecot/auth coredump
Core was generated by `dovecot/auth'.
Program terminated with signal 6, Aborted.
#0  0x0000003016c32885 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
dovecot-2.0.9-2.el6_1.1.x86_64
(gdb) bt full
#0  0x0000003016c32885 in raise () from /lib64/libc.so.6
#1  0x0000003016c34065 in abort () from /lib64/libc.so.6
#2  0x000000301683ca78 in ?? () from /usr/lib64/dovecot/libdovecot.so.0
#3  0x000000301683cab6 in ?? () from /usr/lib64/dovecot/libdovecot.so.0
#4  0x0000003016816dba in i_panic () from /usr/lib64/dovecot/libdovecot.so.0
#5  0x000000301684872f in ioloop_iolist_add () from
/usr/lib64/dovecot/libdovecot.so.0
#6  0x0000003016849149 in io_loop_handle_add () from
/usr/lib64/dovecot/libdovecot.so.0
#7  0x0000003016848593 in io_add () from /usr/lib64/dovecot/libdovecot.so.0
#8  0x0000003016835b88 in master_service_io_listeners_add () from
/usr/lib64/dovecot/libdovecot.so.0
#9  0x00000030168363c2 in master_service_init_finish () from
/usr/lib64/dovecot/libdovecot.so.0
#10 0x0000000000415567 in main ()

This is running on CentOS 6.2
# rpm -q dovecot
dovecot-2.0.9-2.el6_1.1.x86_64

# dovecot --version
2.0.9

# dovecot -n
# 2.0.9: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-220.13.1.el6.x86_64 x86_64 CentOS release 6.2 (Final)
auth_debug = yes
auth_mechanisms = plain login
auth_verbose = yes
listen = *
mail_debug = yes
mail_location = mbox:~/mail:INBOX=/var/mail/%u
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date
mbox_write_locks = fcntl
passdb {
  driver = pam
}
plugin {
  sieve = ~/.dovecot.sieve
  sieve_dir = ~/sieve
}
service auth {
  inet_listener {
    port = 12345
  }
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
  }
  user = $default_internal_user
}
ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
ssl_key = </etc/pki/dovecot/private/dovecot.pem
userdb {
  driver = passwd
}