Fatal: master: service(imap): child 28145 killed with signal 11 (core dumped) -- started happening after upgrade last night
Hello,
I upgraded Dovecot last night and have been receiving core-dumps any time I attempt to access a mailbox (via IMAP) ever since. Logging-in seems to work, but any attempt to access the mailbox contents (regardless of folder) causes a core-dump.
Please let me know if I can provide any additional details.
apt Sources Details:
# grep "dovecot" /etc/apt/sources.list deb http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.2 main deb-src http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.2 main
Package Upgrade Details:
Start-Date: 2014-08-14 19:30:57 Commandline: apt-get upgrade Upgrade: dovecot-sieve:amd64 (2.2.13-1~auto+138, 2.2.13-1~auto+150), dovecot-managesieved:amd64 (2.2.13-1~auto+138, 2.2.13-1~auto+150), dovecot-dev:amd64 (2.2.13-1~auto+138, 2.2.13-1~auto+150), dovecot-mysql:amd64 (2.2.13-1~auto+138, 2.2.13-1~auto+150), dovecot-core:amd64 (2.2.13-1~auto+138, 2.2.13-1~auto+150), dovecot-pop3d:amd64 (2.2.13-1~auto+138, 2.2.13-1~auto+150), dovecot-imapd:amd64 (2.2.13-1~auto+138, 2.2.13-1~auto+150) End-Date: 2014-08-14 19:32:39
Filesystem Details
# df -T | grep "xvda1" Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/xvda1 ext3 61796348 8037864 50612756 14% /
================= DOVECONF =================
# doveconf -n # 2.2.13 (c810e288aaed): /etc/dovecot/dovecot.conf # OS: Linux 3.13.0-24-generic x86_64 Ubuntu 14.04.1 LTS auth_mechanisms = plain login disable_plaintext_auth = no listen = *,[::] log_timestamp = "%Y-%m-%d %H:%M:%S " mail_debug = yes mail_max_userip_connections = 20 mail_privileged_group = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave duplicate passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } plugin { antispam_backend = mailtrain antispam_debug_target = syslog antispam_mail_notspam = --ham antispam_mail_sendmail = /bin/bash antispam_mail_sendmail_args = /usr/local/bin/sa-learn-pipe.sh antispam_mail_spam = --spam antispam_spam_pattern_ignorecase = SPAM;JUNK antispam_trash_pattern_ignorecase = trash;Deleted * antispam_verbose_debug = 1 quota = dict:user::file:/var/vmail/%d/%n/.quotausage quota_rule2 = Trash:storage=+100M quota_rule3 = Junk:ignore quota_rule4 = INBOX:storage=+100M quota_warning = storage=100%% quota-reached 100 %u %d quota_warning2 = storage=95%% quota-warning 95 %u %d quota_warning3 = storage=80%% quota-warning 80 %u %d quota_warning4 = -storage=100%% quota-below below %u %d sieve = /var/vmail/%d/%n/.sieve } postmaster_address = postmaster@example.com protocols = imap pop3 sieve service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service config { unix_listener config { group = vmail mode = 0600 user = vmail } } service imap-login { client_limit = 1000 process_limit = 500 } service quota-below { executable = script /usr/local/bin/quota-below.sh unix_listener quota-below { group = vmail mode = 0666 user = vmail } user = vmail } service quota-reached { executable = script /usr/local/bin/quota-reached.sh unix_listener quota-reached { group = vmail mode = 0666 user = vmail } user = vmail } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { group = vmail mode = 0666 user = vmail } user = vmail } ssl_cert =
================= CORE DUMP =================
# gdb /usr/lib/dovecot/imap /var/vmail/example.com/user/core Reading symbols from /usr/lib/dovecot/imap...(no debugging symbols found)...done. [New LWP 27866] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `dovecot/imap'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f034d987a3e in array_idx_modifiable_i () from /usr/lib/dovecot/libdovecot.so.0
================= SAMPLE MAIL LOG ENTRIES =================
Aug 15 19:39:30 xen dovecot: imap-login: Login: user=ben@example.com, method=PLAIN, rip=76.179.182.207, lip=192.64.115.178, mpid=29100, TLS, session=<Ki0NLrAAiwBMs7bP> Aug 15 19:39:30 xen dovecot: imap: Debug: Loading modules from directory: /usr/lib/dovecot/modules Aug 15 19:39:30 xen dovecot: imap: Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so Aug 15 19:39:30 xen dovecot: imap: Debug: Module loaded: /usr/lib/dovecot/modules/lib11_imap_quota_plugin.so Aug 15 19:39:30 xen dovecot: imap: Debug: Module loaded: /usr/lib/dovecot/modules/lib90_antispam_plugin.so Aug 15 19:39:30 xen dovecot: imap: Debug: Added userdb setting: mail=maildir:/var/vmail/example.com/ben/Maildir Aug 15 19:39:30 xen dovecot: imap: Debug: Added userdb setting: plugin/quota_rule=*:storage=0B Aug 15 19:39:30 xen dovecot: imap: Debug: Added userdb setting: plugin/sieve=/var/vmail/example.com/ben/.sieve Aug 15 19:39:30 xen dovecot: imap(ben@example.com): Debug: Effective uid=5000, gid=5000, home=/var/vmail/example.com/ben Aug 15 19:39:30 xen dovecot: imap(ben@example.com): Debug: Quota root: name=user backend=dict args=:file:/var/vmail/example.com/ben/.quotausage Aug 15 19:39:30 xen dovecot: imap(ben@example.com): Debug: Quota rule: root=user mailbox=* bytes=0 messages=0 Aug 15 19:39:30 xen dovecot: imap(ben@example.com): Debug: Quota rule: root=user mailbox=Trash bytes=+104857600 messages=0 Aug 15 19:39:30 xen dovecot: imap(ben@example.com): Debug: Quota rule: root=user mailbox=Junk ignored Aug 15 19:39:30 xen dovecot: imap(ben@example.com): Debug: Quota rule: root=user mailbox=INBOX bytes=+104857600 messages=0 Aug 15 19:39:30 xen dovecot: imap(ben@example.com): Debug: Quota warning: bytes=0 (100%) messages=0 reverse=no command=quota-reached 100 ben@example.com example.com Aug 15 19:39:30 xen dovecot: imap(ben@example.com): Debug: Quota warning: bytes=0 (95%) messages=0 reverse=no command=quota-warning 95 ben@example.com example.com Aug 15 19:39:30 xen dovecot: imap(ben@example.com): Debug: Quota warning: bytes=0 (80%) messages=0 reverse=no command=quota-warning 80 ben@example.com example.com Aug 15 19:39:30 xen dovecot: imap(ben@example.com): Debug: Quota warning: bytes=0 (100%) messages=0 reverse=yes command=quota-below below ben@example.com example.com Aug 15 19:39:30 xen dovecot: imap(ben@example.com): Debug: Quota grace: root=user bytes=0 (10%) Aug 15 19:39:30 xen dovecot: imap(ben@example.com): Debug: dict quota: user=ben@example.com, uri=file:/var/vmail/example.com/ben/.quotausage, noenforcing=0 Aug 15 19:39:30 xen dovecot: imap(ben@example.com): Debug: maildir++: root=/var/vmail/example.com/ben/Maildir, index=, indexpvt=, control=, inbox=/var/vmail/example.com/ben/Maildir, alt= Aug 15 19:39:30 xen dovecot: imap(ben@example.com): Fatal: master: service(imap): child 29100 killed with signal 11 (core dumped)
On 15 Aug 2014, at 21:53, Ben Johnson ben@indietorrent.org wrote:
# gdb /usr/lib/dovecot/imap /var/vmail/example.com/user/core Reading symbols from /usr/lib/dovecot/imap...(no debugging symbols found)...done. [New LWP 27866] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `dovecot/imap'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f034d987a3e in array_idx_modifiable_i () from /usr/lib/dovecot/libdovecot.so.0
What about also giving the command "bt full"? Also having dovecot-dbg package installed before that would make it even more helpful.
On 8/17/2014 5:11 PM, Timo Sirainen wrote:
On 15 Aug 2014, at 21:53, Ben Johnson ben@indietorrent.org wrote:
# gdb /usr/lib/dovecot/imap /var/vmail/example.com/user/core Reading symbols from /usr/lib/dovecot/imap...(no debugging symbols found)...done. [New LWP 27866] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `dovecot/imap'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f034d987a3e in array_idx_modifiable_i () from /usr/lib/dovecot/libdovecot.so.0
What about also giving the command "bt full"? Also having dovecot-dbg package installed before that would make it even more helpful.
Thanks for getting right back on this, Timo. I really appreciate it.
After looking at the output, this seems related to dovecot-antispam plug-in.
Just an observation: I sent another message to the list a few days before this message, describing an unhandled exit scenario where dovecot-antispam plug-in is concerned ("Re: Backtrace io_add(0x1) called twice fd= (was Exit status code 134; what is it, in the context of Dovecot Antispam plug-in?)", and another user submitted a similar message with subject "antispam plugin crash".
Anyway, here's the output with a slightly newer version of the dovecot packages (they wanted to be upgraded again when installing dovecot-dbg):
Reading symbols from /usr/lib/dovecot/imap...Reading symbols from /usr/lib/debug//usr/lib/dovecot/imap...done. done. [New LWP 18372] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `dovecot/imap'. Program terminated with signal SIGSEGV, Segmentation fault. #0 array_idx_modifiable_i (array=0xc8, idx=1) at array.c:12 12 array.c: No such file or directory.
And here is the output of "bt full". I'll include this output without having to be asked in the future; this is my first bug report for dovecot, and I thank you for your patience as I learn what's helpful to you.
Thanks again,
-Ben
On 8/18/2014 8:36 AM, Ben Johnson wrote:
Just an observation: I sent another message to the list a few days before this message, describing an unhandled exit scenario where dovecot-antispam plug-in is concerned ("Re: Backtrace io_add(0x1) called twice fd= (was Exit status code 134; what is it, in the context of Dovecot Antispam plug-in?)", and another user submitted a similar message with subject "antispam plugin crash".
Yep... disabling the antispam plugin stops the crashing.
Ironically, the very reason for which I replaced my distro's source repos with
deb http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.2 main deb-src http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.2 main
and upgraded to the latest available package there is to get antispam working with Ubuntu 14.04 LTS. Out-of-the-box, the plugin does not work on 14.04 (at least with my particular configuration), which includes Dovecot 2.2.9. I started a different thread on that subject, but never followed-up with the appropriate debugging info.
Perhaps the next step is to go back to Dovecot 2.2.9 via the Ubuntu repos and capture a debug backtrace for dovecot's "deliver" executable (since that's where the antispam-related crash occurs on that version of Dovecot).
-Ben
On 18 Aug 2014, at 17:30, Ben Johnson ben@indietorrent.org wrote:
Ironically, the very reason for which I replaced my distro's source repos with
deb http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.2 main deb-src http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.2 main
These are just nightly builds and there's a good chance they break randomly. Maybe the problem in your case is that the antispam plugin just wasn't recompiled after some changes that required recompiling. I'm not sure. In any case a better working repository is here: http://shop.dovecot.fi/home/8-dovecot-ee-repository-access.html (It's free even though it still says $99 in the main page. Everyone's too busy to bother changing the web.)
On 8/20/2014 5:37 AM, Timo Sirainen wrote:
On 18 Aug 2014, at 17:30, Ben Johnson ben@indietorrent.org wrote:
Ironically, the very reason for which I replaced my distro's source repos with
deb http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.2 main deb-src http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.2 main
These are just nightly builds and there's a good chance they break randomly. Maybe the problem in your case is that the antispam plugin just wasn't recompiled after some changes that required recompiling. I'm not sure. In any case a better working repository is here: http://shop.dovecot.fi/home/8-dovecot-ee-repository-access.html (It's free even though it still says $99 in the main page. Everyone's too busy to bother changing the web.)
Thank you for the clarification, Timo.
I was able to get the repository access squared-away via the Dovecot store. The price does in fact reflect $0, so, somebody took the initiative to correct that recently. :)
When I start dovecot, there is a mismatch between ABI versions when the plugin is loaded:
Aug 27 09:22:11 local dovecot: imap: Error: Module is for different ABI version 2.2.ABIv13.2(2.2.13) (we have 2.2.ABIv13(2.2.13.8)): /usr/lib/dovecot/modules/lib90_antispam_plugin.so Aug 27 09:22:11 local dovecot: imap: Fatal: Couldn't load required plugins
In retrospect, this is probably what has been happening all along (with dovecot-deliver exiting with code 134); previously, I failed to notice the message about the ABI mismatch immediately after Dovecot is started.
Does this mean that the plugin source code has not been updated to function with dovecot-2.2.13.8? If so, would I need to downgrade the dovecot version to 2.2.13 for the plugin to work? Or should this message be interpreted in some other way?
For the sake of thoroughness, and to ensure that I didn't botch the procedure in any way, I have included at the bottom of this message the exact steps that I took.
Thank you for any help,
-Ben
1.) Added to /etc/apt/sources.list:
deb https://user:pass@apt.dovecot.fi/ubuntu/precise/2.2 precise main
2.) Installed dovecot from EE package repository:
# apt-get remove dovecot
# apt-get update
# apt-get install dovecot-ee-core dovecot-ee-imapd dovecot-ee-pop3d dovecot-ee-sieve dovecot-ee-managesieved dovecot-ee-mysql dovecot-ee-dbg dovecot-ee-dev
3.) Checked-out latest Antispam plugin source:
# hg clone http://hg.dovecot.org/dovecot-antispam-plugin
4.) Built and installed plugin:
# cd dovecot-antispam-plugin
# sh ./autogen.sh *info* running aclocal (-I m4) *info* running autoheader *info* running autoconf
# ./configure --with-dovecot=/usr/lib/dovecot checking for tput... /usr/bin/tput checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking whether we need an implib... no checking for shared library system... GNU checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking how to run the C preprocessor... gcc -E checking whether ln -s works... yes checking for a BSD-compatible install... /usr/bin/install -c checking for "/usr/lib/dovecot/dovecot-config"... /usr/lib/dovecot/dovecot-config checking whether OS supports plugin dependencies... yes checking for dovecot tree type... installed configure: touching .deps files configure: creating ./config.status config.status: creating buildsys.mk config.status: creating extra.mk config.status: creating config.h
# make Entering directory src. Successfully compiled antispam-plugin.c (plugin). Successfully compiled aux.c (plugin). Successfully compiled backends.c (plugin). crm114.c: In function ‘call_reaver’: crm114.c:73:7: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result] write(pipes[1], signature_hdr, strlen(signature_hdr)); ^ crm114.c:74:7: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result] write(pipes[1], ": ", 2); ^ crm114.c:75:7: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result] write(pipes[1], signature, strlen(signature)); ^ crm114.c:76:7: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result] write(pipes[1], "\r\n\r\n", 4); ^ Successfully compiled crm114.c (plugin). Successfully compiled dspam.c (plugin). Successfully compiled mailbox.c (plugin). Successfully compiled mailtrain.c (plugin). Successfully compiled signature-log.c (plugin). Successfully compiled signature.c (plugin). Successfully compiled spool2dir.c (plugin). Successfully compiled user.c (plugin). Successfully linked lib90_antispam_plugin.so. Leaving directory src. Entering directory doc. Successfully compiled dovecot-antispam.7.txt. Leaving directory doc.
# make install Entering directory src. Leaving directory src. Entering directory doc. Leaving directory doc. Entering directory src. Successfully installed lib90_antispam_plugin.so. Leaving directory src. Entering directory doc. Successfully installed dovecot-antispam.7. Leaving directory doc.
# service dovecot restart
# tail -f -n 50 /var/log/mail.log Aug 27 09:22:11 localhost dovecot: imap: Error: Module is for different ABI version 2.2.ABIv13.2(2.2.13) (we have 2.2.ABIv13(2.2.13.8)): /usr/lib/dovecot/modules/lib90_antispam_plugin.so Aug 27 09:22:11 localhost dovecot: imap: Fatal: Couldn't load required plugins
participants (2)
-
Ben Johnson
-
Timo Sirainen