On 08/15/2011 11:17 AM, kzorba@otenet.gr wrote:
Quoting Timo Sirainen tss@iki.fi:
With a quick test I can't reproduce pop3_lock_session=yes causing a crash. I guess it needs something else besides what I tested. It would be helpful if your Dovecot binaries weren't stripped of debug symbols. I could then ask for some more information from the core dumps with gdb.
Timo, all
here is a backtrace with debuging symbols in the executables:
[root@pop08 ]# gdb /opt/dovecot-debug/libexec/dovecot/pop3 core.2929 GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-32.el5_6.2) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /opt/dovecot-debug/libexec/dovecot/pop3...done. BFD: Warning: /var/mail7/folders/P/K/U/amihal/core.2929 is truncated: expected core file size >= 569344, found: 565248. Reading symbols from /opt/dovecot-debug/lib/dovecot/libdovecot-storage.so.0...done. Loaded symbols for /opt/dovecot-debug/lib/dovecot/libdovecot-storage.so.0 Reading symbols from /opt/dovecot-debug/lib/dovecot/libdovecot.so.0...done. Loaded symbols for /opt/dovecot-debug/lib/dovecot/libdovecot.so.0 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libpthread.so.0 Core was generated by `dovecot/pop3'. Program terminated with signal 11, Segmentation fault. #0 istream_raw_mbox_get_start_offset (stream=0x0) at istream-raw-mbox.c:498 498 istream-raw-mbox.c: No such file or directory. in istream-raw-mbox.c (gdb) bt #0 istream_raw_mbox_get_start_offset (stream=0x0) at istream-raw-mbox.c:498 #1 0x00002b4114afc769 in mbox_mail_get_special (_mail=0x1bad8fa0, field=<value optimized out>, value_r=0x7fff9600fa88) at mbox-mail.c:198 #2 0x00002b4114adb2d0 in index_mail_expunge (mail=0x0) at index-mail.c:1503 #3 0x0000000000405eac in client_update_mails (client=0x1bac4a00) at pop3-commands.c:255 #4 0x00000000004061d1 in cmd_quit (client=0x1bac4a00, name=0x1baccbd4 "", args=0x406a0c "") at pop3-commands.c:274 #5 client_command_execute (client=0x1bac4a00, name=0x1baccbd4 "", args=0x406a0c "") at pop3-commands.c:773 #6 0x00000000004045c9 in client_handle_input (client=0x1bac4a00) at pop3-client.c:628 #7 0x00002b4114db0698 in io_loop_call_io (io=0x1bac01d0) at ioloop.c:384 #8 0x00002b4114db19d5 in io_loop_handler_run (ioloop=<value optimized out>) at ioloop-epoll.c:213 #9 0x00002b4114db062d in io_loop_run (ioloop=0x1baa8610) at ioloop.c:405 #10 0x00002b4114d9ef13 in master_service_run (service=0x1baa84e0, callback=0x1baccbe4) at master-service.c:478 #11 0x00000000004039a4 in main (argc=1, argv=0x1baa8370) at main.c:252 (gdb) bt full #0 istream_raw_mbox_get_start_offset (stream=0x0) at istream-raw-mbox.c:498 rstream = <value optimized out> __FUNCTION__ = "istream_raw_mbox_get_start_offset" #1 0x00002b4114afc769 in mbox_mail_get_special (_mail=0x1bad8fa0, field=<value optimized out>, value_r=0x7fff9600fa88) at mbox-mail.c:198 mail = <value optimized out> mbox = 0x1bac97d0 offset = 0 #2 0x00002b4114adb2d0 in index_mail_expunge (mail=0x0) at index-mail.c:1503 value = <value optimized out> guid_128 = "\240\267\252\033\000\000\000\000\001\000\000\000\000\000\000" #3 0x0000000000405eac in client_update_mails (client=0x1bac4a00) at pop3-commands.c:255 search_args = 0x0 ctx = 0x1bace150 mail = 0x1bad8fa0 msgnum = 0 bit = 464309220 ret = <value optimized out> #4 0x00000000004061d1 in cmd_quit (client=0x1bac4a00, name=0x1baccbd4 "", args=0x406a0c "") at pop3-commands.c:274 No locals. #5 client_command_execute (client=0x1bac4a00, name=0x1baccbd4 "", args=0x406a0c "") at pop3-commands.c:773 No locals. #6 0x00000000004045c9 in client_handle_input (client=0x1bac4a00) at pop3-client.c:628 _data_stack_cur_id = 3 line = 0x1baab872 "QUIT" args = 0x406a0c "" ret = <value optimized out> #7 0x00002b4114db0698 in io_loop_call_io (io=0x1bac01d0) at ioloop.c:384 ioloop = 0x1baa8610 t_id = 2 #8 0x00002b4114db19d5 in io_loop_handler_run (ioloop=<value optimized out>) at ioloop-epoll.c:213 ctx = <value optimized out> event = 0x1baa88b0 list = 0x1bac0220 io = 0x0 tv = {tv_sec = 9, tv_usec = 871818} events_count = <value optimized out> msecs = <value optimized out> ---Type <return> to continue, or q <return> to quit--- ret = 1 i = 0 call = false #9 0x00002b4114db062d in io_loop_run (ioloop=0x1baa8610) at ioloop.c:405 No locals. #10 0x00002b4114d9ef13 in master_service_run (service=0x1baa84e0, callback=0x1baccbe4) at master-service.c:478 No locals. #11 0x00000000004039a4 in main (argc=1, argv=0x1baa8370) at main.c:252 service_flags = <value optimized out> storage_service_flags = MAIL_STORAGE_SERVICE_FLAG_DISALLOW_ROOT postlogin_socket_path = 0x0 username = 0x0 c = <value optimized out> set_roots = {0x4072a0, 0x0} (gdb) quit
I have also kept the user's mailbox. I couldn't reproduce the problem by talking POP3 directly to the server by hand.
Any ideas?
Thanks,
Kostas
Hi Timo,
indeed it is a bug that I could not reproduce myself. Having debug symbols and producing the stack trace is the next logical step and I will work on this tomorrow. Since --enable-debug does not work in your configure script, can you direct me as to what is needed? Is there an option in configure or do I need to mess with the makefiles?
On the other hand, I have found two different bugs. Having pop3_lock_session=yes we have the situation described here and also of course delays in local deliveries in case a client has an active pop session. And I can tell you we have a lot of abusing clients that keep hitting our pop servers continuously, or keep connections open for a VERY long time.
To address that, we put pop3_lock_session=no. In this case, there is an fcntl lock leak somewhere. The good news is that we have reproduced that and I will send relevant information in a different mail. I also read the following thread, from a while back:
http://www.dovecot.org/list/dovecot/2009-February/037098.html
Regards,
Kostas