[Dovecot] lmtp panic in proxy lmtp director
Hi, we are getting some core dumps with signal 6 in ltmp on a dovecot director proxy server, like this:
Dec 5 14:31:51 sproxy1 dovecot: lmtp(29999): Panic: file lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: (proxy->data_input->eof) Dec 5 14:31:51 sproxy1 dovecot: lmtp(29999): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0() [0x363323d99a] -> /usr/lib64/dovecot/libdovecot.so.0() [0x363323d9e6] -> /usr/lib64/dovecot/libdovecot.so.0(i_error+0) [0x3633216f8f] -> dovecot/lmtp() [0x406e57] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xd4) [0x3633248ff4] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x5b) [0x3633249bdb] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x3633248c58] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x3633236fc3] -> dovecot/lmtp(main+0x154) [0x403f84] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x373cc1ec5d] -> dovecot/lmtp() [0x403d69] Dec 5 14:31:51 sproxy1 abrt[30067]: saved core dump of pid 29999 (/usr/libexec/dovecot/lmtp) to /var/spool/abrt/ccpp-1323091911-29999.new/coredump (1368064 bytes) Dec 5 14:31:51 sproxy1 dovecot: master: Error: service(lmtp): child 29999 killed with signal 6 (core dumped)
we have something misconfigured or it's a bug on this dovecot version?
our doveconf -n is: # 2.0.15: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-71.29.1.el6.x86_64 x86_64 CentOS Linux release 6.0 (Final) auth_cache_size = 3 k auth_cache_ttl = 15 mins auth_verbose = yes base_dir = /var/run/dovecot/ default_client_limit = 30000 default_process_limit = 5000 director_doveadm_port = 990 director_mail_servers = 10.80.82.21 10.80.82.22 director_servers = 10.80.82.11 10.80.82.12 doveadm_proxy_port = 24245 lmtp_proxy = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave passdb { args = proxy=y nopassword=y starttls=any-cert driver = static } postmaster_address = xavier.pons@uib.es protocols = imap pop3 lmtp sieve service auth { client_limit = 27048 unix_listener /var/spool/postfix/private/auth { mode = 0666 } unix_listener auth-userdb { group = dovecot mode = 0660 } } service director { fifo_listener login/proxy-notify { mode = 0666 } inet_listener { port = 991 } inet_listener director-doveadm { port = 990 } unix_listener director-userdb { mode = 0660 } unix_listener login/director { mode = 0666 } } service doveadm { inet_listener { port = 24245 } } service imap-login { executable = imap-login director inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } } service lmtp { inet_listener lmtp { port = 30025 } } service managesieve-login { executable = managesieve-login director } service pop3-login { executable = pop3-login director inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 ssl = yes } } ssl = required ssl_cert =
Xavier
-- xavier.pons@uib.es Centre de Tecnologies de la Informació Universitat Illes Balears
On Mon, Dec 05, 2011 at 03:26:54PM +0100, Xavier Pons wrote:
Hi, we are getting some core dumps with signal 6 in ltmp on a dovecot director proxy server, like this:
http://dovecot.org/list/dovecot/2011-September/060830.html
http://www.dovecot.org/list/dovecot/2011-September/061395.htmlwe have something misconfigured or it's a bug on this dovecot version?
I believe there's too short LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS in lmtp-proxy.c.
-jf
This happens when the LMTP proxy doesn't receive feedback from the back-end LMTP process for a certain amount of time.
This typically happens either because of very low I/O performance or NFS locks. Do an strace on your back-end LMTP processes. If you see that these processes are waiting on NFS locks ande you are using NFSv3, you should move over to NFSv4.
On Dec 5, 2011, at 6:26 AM, Xavier Pons wrote:
Hi, we are getting some core dumps with signal 6 in ltmp on a dovecot director proxy server, like this:
Dec 5 14:31:51 sproxy1 dovecot: lmtp(29999): Panic: file lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: (proxy->data_input->eof) Dec 5 14:31:51 sproxy1 dovecot: lmtp(29999): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0() [0x363323d99a] -> /usr/lib64/dovecot/libdovecot.so.0() [0x363323d9e6] -> /usr/lib64/dovecot/libdovecot.so.0(i_error+0) [0x3633216f8f] -> dovecot/lmtp() [0x406e57] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xd4) [0x3633248ff4] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x5b) [0x3633249bdb] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x3633248c58] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x3633236fc3] -> dovecot/lmtp(main+0x154) [0x403f84] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x373cc1ec5d] -> dovecot/lmtp() [0x403d69] Dec 5 14:31:51 sproxy1 abrt[30067]: saved core dump of pid 29999 (/usr/libexec/dovecot/lmtp) to /var/spool/abrt/ccpp-1323091911-29999.new/coredump (1368064 bytes) Dec 5 14:31:51 sproxy1 dovecot: master: Error: service(lmtp): child 29999 killed with signal 6 (core dumped)
we have something misconfigured or it's a bug on this dovecot version?
our doveconf -n is: # 2.0.15: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-71.29.1.el6.x86_64 x86_64 CentOS Linux release 6.0 (Final) auth_cache_size = 3 k auth_cache_ttl = 15 mins auth_verbose = yes base_dir = /var/run/dovecot/ default_client_limit = 30000 default_process_limit = 5000 director_doveadm_port = 990 director_mail_servers = 10.80.82.21 10.80.82.22 director_servers = 10.80.82.11 10.80.82.12 doveadm_proxy_port = 24245 lmtp_proxy = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave passdb { args = proxy=y nopassword=y starttls=any-cert driver = static } postmaster_address = xavier.pons@uib.es protocols = imap pop3 lmtp sieve service auth { client_limit = 27048 unix_listener /var/spool/postfix/private/auth { mode = 0666 } unix_listener auth-userdb { group = dovecot mode = 0660 } } service director { fifo_listener login/proxy-notify { mode = 0666 } inet_listener { port = 991 } inet_listener director-doveadm { port = 990 } unix_listener director-userdb { mode = 0660 } unix_listener login/director { mode = 0666 } } service doveadm { inet_listener { port = 24245 } } service imap-login { executable = imap-login director inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } } service lmtp { inet_listener lmtp { port = 30025 } } service managesieve-login { executable = managesieve-login director } service pop3-login { executable = pop3-login director inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 ssl = yes } } ssl = required ssl_cert =
Xavier
-- xavier.pons@uib.es Centre de Tecnologies de la Informació Universitat Illes Balears
On Mon, 2011-12-05 at 15:26 +0100, Xavier Pons wrote:
Hi, we are getting some core dumps with signal 6 in ltmp on a dovecot director proxy server, like this:
Dec 5 14:31:51 sproxy1 dovecot: lmtp(29999): Panic: file lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: (proxy->data_input->eof)
I think the way I originally planned LMTP proxying to work is simply too complex to work reliably, perhaps even if the code was bug-free. So instead of reading+writing DATA at the same time, this patch changes the DATA to be first read into memory or temp file, and then from there read and sent to the LMTP backends:
http://hg.dovecot.org/dovecot-2.1/raw-rev/51d87deb5c26
The patch applies on top of v2.0 also. I tested that it appears to work, but more testing wouldn't hurt to make sure it won't corrupt large mails when sending them to multiple backends (one mail with multiple RCPT TOs sent to multiple different backends)..
participants (4)
-
Dovecot-GDH
-
Jan-Frode Myklebust
-
Timo Sirainen
-
Xavier Pons