[Dovecot] Dovecot 1.1.1 + zlib plugin + mbox crash
I've tried this on both Solaris 8 and SuSE Enterprise 9 (64-bit).
I get a assert-crash when using a gzipped mbox folder
. OK Logged in. . SELECT test.gz
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Label4 $Label2
$Label1 $Label3 $Label5)
- OK [PERMANENTFLAGS ()] Read-only mailbox.
- 167 EXISTS
- 0 RECENT
- OK [UNSEEN 2] First unseen.
- OK [UIDVALIDITY 1096038620] UIDs valid
- OK [UIDNEXT 42890] Predicted next UID
. OK [READ-ONLY] Select completed. . FETCH 167 FULL
- 167 FETCH (FLAGS () INTERNALDATE "05-Jan-2003 07:05:37 +0000"
RFC822.SIZE 4693 ENVELOPE ("Sat, 04 Jan 2003 21:05:48 -0800" "[xxxxxxx] xx: xxxx xxxxxxxxxxx" (("xxxxx xxxxxxxx" NIL "xxxxxxxxxxxx" "xxxxx.xxx")) ((NIL NIL "xxxxxxx-xxxxxx" "xxxxxxxxxx.xx")) (("xxxxx xxxxxxxx" NIL "xxxxxxxxxxxx" "xxxxx.xxx")) (("xxxxxxx xxxx xxxx" NIL "xxxxxxx" "xxxxxxxxxx.xx")) NIL NIL NIL "xxxxxxxx.xxxxxxx@xxxxx.xxx") BODY ("text" "plain" ("charset" "us-ascii" "format" "flowed") NIL NIL "7bit" 3408 35)) . OK Fetch completed. . FETCH 167 BODY[] Connection closed by foreign host.
dovecot: Jul 23 15:35:39 Info: imap-login: Login: user=<xxxxxxxx>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=xxx.xxx.xxx.xxx dovecot: Jul 23 15:35:39 Info: auth(default): new auth connection: pid=841 dovecot: Jul 23 15:35:54 Panic: IMAP 888 xxxxxxxx xxx.xxx.xxx.xxx : file istream-raw-mbox.c: line 363 (i_stream_create_raw_mbox): assertion failed: (input->v_offset == 0) dovecot: Jul 23 15:35:54 Error: IMAP 888 xxxxxxxx xxx.xxx.xxx.xxx : Raw backtrace: imap [0x4833b8] -> imap [0x483413] -> imap [0x482e46] -> imap [0x43ce47] -> imap(mbox_file_open_stream+0x40) [0x434f40] -> imap [0x4363c1] -> imap [0x436778] -> imap(mail_get_stream+0xa) [0x4547ea] -> imap(index_mail_set_seq+0x144) [ 0x44b444] -> imap [0x436a79] -> imap(mail_set_seq+0x7) [0x454697] -> imap(index_storage_search_next_nonblock+0xd7) [0x44ef07] -> imap(mailbox_search_next_non block+0x10) [0x455ef0] -> imap(mailbox_search_next+0x20) [0x455f20] -> imap(imap_fetch+0x87) [0x41d457] -> imap(cmd_fetch+0x1f4) [0x4174b4] -> imap [0x41b2e4 ] -> imap [0x41b2d8] -> imap [0x41bc91] -> imap(client_input+0x5f) [0x41bdaf] -> imap(io_loop_handler_run+0xf7) [0x489d97] -> imap(io_loop_run+0x28) [0x48906 8] -> imap(main+0x462) [0x4233b2] -> /lib64/tls/libc.so.6(__libc_start_main+0xea) [0x2a9578caaa] -> imap(__ctype_b_loc+0x4a) [0x415b3a] dovecot: Jul 23 15:35:54 Error: child 888 (imap) killed with signal 6
dovecot -n output:
# 1.1.1: /opt/RDGdovect/etc/dovecot.conf base_dir: /opt/RDGdovect/var/run/ log_path: /opt/RDGdovect/var/log/dovecot.log info_log_path: /opt/RDGdovect/var/log/dovecot.log protocols: imap listen: *:143 ssl_disable: yes disable_plaintext_auth: no shutdown_clients: no login_dir: /opt/RDGdovect/var/run/login login_executable: /opt/RDGdovect/libexec/dovecot/imap-login login_greeting: University of Reading IMAP test1.1 ready. login_process_per_connection: no verbose_proctitle: yes mail_location: mbox:/export/folders/%13.h/mail/:INBOX=/export/mail/%13.h/INBOX:INDEX=/export/indexes/%13.h mbox_very_dirty_syncs: yes mail_drop_priv_before_exec: yes mail_plugins: zlib mail_log_prefix: %Us %p %u %r : imap_client_workarounds: outlook-idle delay-newmail auth default: mechanisms: plain login username_format: %Ln verbose: yes debug: yes passdb: driver: ldap args: /opt/RDGdovect/etc/dovecot-ldap.conf userdb: driver: passwd-file args: /opt/RDGdovect/etc/userdb
I've attached the relevant test.gz file (anonymised for the test)
Has anybody else got it working?
Best Wishes, Chris
On Wed, 2008-07-23 at 15:58 +0100, Chris Wakelin wrote:
I've tried this on both Solaris 8 and SuSE Enterprise 9 (64-bit).
I get a assert-crash when using a gzipped mbox folder
..
istream-raw-mbox.c: line 363 (i_stream_create_raw_mbox): assertion failed: (input->v_offset == 0)
Timo Sirainen wrote:
On Wed, 2008-07-23 at 15:58 +0100, Chris Wakelin wrote:
I've tried this on both Solaris 8 and SuSE Enterprise 9 (64-bit).
I get a assert-crash when using a gzipped mbox folder
..
istream-raw-mbox.c: line 363 (i_stream_create_raw_mbox): assertion failed: (input->v_offset == 0)
Well, now it doesn't assert crash, but (on the Solaris 8 machine):
dovecot: Jul 24 00:22:22 Error: IMAP 13458 xxxxxxxx xxx.xxx.xxx.xxx : Cached message offset 75979 is invalid for mbox file (read-only mbox stream) dovecot: Jul 24 00:22:22 Error: IMAP 13458 xxxxxxxx xxx.xxx.xxx.xxx : Cached message offset 75979 is invalid for mbox file (read-only mbox stream) dovecot: Jul 24 00:22:22 Error: IMAP 13458 xxxxxxxx xxx.xxx.xxx.xxx : Losing sync for mail uid=39 in mbox file (read-only mbox stream) dovecot: Jul 24 00:22:22 Error: IMAP 13458 xxxxxxxx xxx.xxx.xxx.xxx : Unexpectedly lost From-line at 75979 dovecot: Jul 24 00:22:22 Error: IMAP 13458 xxxxxxxx xxx.xxx.xxx.xxx : FETCH for mailbox test.gz UID 39 got too little data: 0 vs 1671 dovecot: Jul 24 00:22:22 Error: IMAP 13458 xxxxxxxx xxx.xxx.xxx.xxx : Corrupted index cache file /export/indexes/xxxxxxxx/.imap/test.gz/dovecot.index.cache: Broken virtual size for mail UID 39 dovecot: Jul 24 00:22:22 Info: IMAP 13458 xxxxxxxx xxx.xxx.xxx.xxx : Disconnected: Disconnected bytes=135/88800
after (I think - I was picking messages at "random"),
. SELECT test.gz (ok) . FETCH 1:167 FULL (ok) . FETCH 167 BODY[] (ok) . FETCH 123 BODY[]
- 123 FETCH (BODY[] {1769} (ok)
. FETCH 39 BODY[]
- 39 FETCH (BODY[] {1671}
Connection closed by foreign host.
I'd deleted the index files to be sure it wasn't left over from the previous problems.
I'll try on the SuSE box as well.
Best Wishes, Chris
On Thu, 2008-07-24 at 00:34 +0100, Chris Wakelin wrote:
Well, now it doesn't assert crash, but (on the Solaris 8 machine):
dovecot: Jul 24 00:22:22 Error: IMAP 13458 xxxxxxxx xxx.xxx.xxx.xxx : Cached message offset 75979 is invalid for mbox file (read-only mbox stream)
I'm not able to reproduce this at least on Linux..
Timo Sirainen wrote:
On Thu, 2008-07-24 at 00:34 +0100, Chris Wakelin wrote:
Well, now it doesn't assert crash, but (on the Solaris 8 machine):
dovecot: Jul 24 00:22:22 Error: IMAP 13458 xxxxxxxx xxx.xxx.xxx.xxx : Cached message offset 75979 is invalid for mbox file (read-only mbox stream)
I'm not able to reproduce this at least on Linux..
Sorry, neither am I on Linux :p
I guess I was too impatient and didn't do a "make clean" first on the Solaris box (it takes about 10-15 minutes to compile otherwise; the SuSE machine is *much* faster). It's chugging through it now and I'll let you know if it doesn't fix the problem.
Chris
On Thu, 2008-07-24 at 00:47 +0100, Chris Wakelin wrote:
Timo Sirainen wrote:
On Thu, 2008-07-24 at 00:34 +0100, Chris Wakelin wrote:
Well, now it doesn't assert crash, but (on the Solaris 8 machine):
dovecot: Jul 24 00:22:22 Error: IMAP 13458 xxxxxxxx xxx.xxx.xxx.xxx : Cached message offset 75979 is invalid for mbox file (read-only mbox stream)
I'm not able to reproduce this at least on Linux..
Sorry, neither am I on Linux :p
I guess I was too impatient and didn't do a "make clean" first on the Solaris box (it takes about 10-15 minutes to compile otherwise; the SuSE machine is *much* faster). It's chugging through it now and I'll let you know if it doesn't fix the problem.
make clean shouldn't ever be necessary, since automake does dependency tracking..
I can try this in Solaris 10, but I don't really see why the OS/architecture should matter.
On Thu, 2008-07-24 at 02:51 +0300, Timo Sirainen wrote:
dovecot: Jul 24 00:22:22 Error: IMAP 13458 xxxxxxxx xxx.xxx.xxx.xxx : Cached message offset 75979 is invalid for mbox file (read-only mbox stream)
I'm not able to reproduce this at least on Linux..
Sorry, neither am I on Linux :p
I guess I was too impatient and didn't do a "make clean" first on the Solaris box (it takes about 10-15 minutes to compile otherwise; the SuSE machine is *much* faster). It's chugging through it now and I'll let you know if it doesn't fix the problem.
make clean shouldn't ever be necessary, since automake does dependency tracking..
I can try this in Solaris 10, but I don't really see why the OS/architecture should matter.
Oh, unless there are some bugs in your zlib version, such as related to seeking..
Timo Sirainen wrote:
make clean shouldn't ever be necessary, since automake does dependency tracking..
I thought so too, but it was fine on SuSE when I did a "make clean".
Anyway, it didn't make any difference on Solaris 8 :(
I can try this in Solaris 10, but I don't really see why the OS/architecture should matter.
Oh, unless there are some bugs in your zlib version, such as related to seeking..
Could be, I imagine it's as prehistoric as the bzlib :)
I'll look for patches on SunSolve tomorrow. Unfortunately, I'm stuck with Solaris 8 for the production server (it's a nice old machine given an extra 3+ years of life by the performance gains of Dovecot over UW-IMAP).
Best Wishes, Chris
Chris Wakelin wrote:
I can try this in Solaris 10, but I don't really see why the OS/architecture should matter.
Oh, unless there are some bugs in your zlib version, such as related to seeking..
Could be, I imagine it's as prehistoric as the bzlib :)
Right, I've tried dovecot-1.1.2 using my own zlib-1.2.3 (and bzlib-1.0.5) build on Solaris 8:
ldd lib20_zlib_plugin.so gives libz.so => /opt/RDGzlib/lib/libz.so libbz2.so => /opt/RDGbzlib/lib/libbz2.so ...
and get the same problems.
I've tried building on Solaris 10 (using Sun's zlib and bzlib2) and get a similar problem
ldd lib20_zlib_plugin.so libz.so.1 => /usr/lib/libz.so.1 libbz2.so.1 => /usr/lib/libbz2.so.1 ...
After FETCH 1:167 FULL, FETCH 167 BODY[], FETCH 123 BODY[], FETCH 39 BODY[], FETCH 23 BODY[] :-
dovecot: Jul 24 16:02:51 Error: IMAP 28535 xxxxxxxx xxx.xxx.xxx.xxx : Cached message offset 45576 is invalid for mbox file (read-only mbox stream) dovecot: Jul 24 16:02:51 Error: IMAP 28535 xxxxxxxx xxx.xxx.xxx.xxx : Cached message offset 45576 is invalid for mbox file (read-only mbox stream) dovecot: Jul 24 16:02:51 Error: IMAP 28535 xxxxxxxx xxx.xxx.xxx.xxx : Losing sync for mail uid=23 in mbox file (read-only mbox stream) dovecot: Jul 24 16:02:51 Error: IMAP 28535 xxxxxxxx xxx.xxx.xxx.xxx : Unexpectedly lost From-line at 45576 dovecot: Jul 24 16:02:51 Error: IMAP 28535 xxxxxxxx xxx.xxx.xxx.xxx : Unexpectedly lost From-line at 45576 dovecot: Jul 24 16:02:51 Error: IMAP 28535 xxxxxxxx xxx.xxx.xxx.xxx : Couldn't get mbox size dovecot: Jul 24 16:02:51 Info: IMAP 28535 xxxxxxxx xxx.xxx.xxx.xxx : Disconnected: Internal error occurred. Refer to server log for more information. [2008-07-24 16:02:51] bytes=114/90517
On the SuSE box, it's fine. However, on all three boxes when I use bzip2 instead (i.e. same folder, bzipped) I get a segfault crash after just "FETCH 1:167 FULL" :-
dovecot: Jul 24 16:17:29 Error: child 28544 (imap) killed with signal 11
backtrace on the Solaris 8 version:
#0 0xff2505a0 in memmove () from /usr/platform/SUNW,Ultra-250/lib/libc_psr.so.1 #1 0xa6d60 in i_stream_compress (stream=0xfc930) at istream.c:301 #2 0xff1414e4 in i_stream_zlib_seek (stream=0xfc930, v_offset=0, mark=true) at istream-zlib.c:173 #3 0xa6920 in i_stream_seek_mark (stream=0xfc958, v_offset=4443332646273026) at istream.c:139 #4 0x4f654 in istream_raw_mbox_seek (stream=0x1142f0, offset=0) at istream-raw-mbox.c:625 #5 0x45cd8 in mbox_file_seek (mbox=0x10e6b8, view=0x4868b0, seq=1, deleted_r=0xffbeee37) at mbox-file.c:171 #6 0x4739c in mbox_mail_seek (mail=0x10edd8) at mbox-mail.c:70 #7 0x474f4 in mbox_mail_get_received_date (_mail=0xffffffff, date_r=0xffbeef8c) at mbox-mail.c:103 #8 0x6acf4 in mail_get_received_date (mail=0xffffffff, date_r=0xffbeef8c) at mail.c:79 #9 0x2aeb0 in fetch_internaldate (ctx=0x10edd8, mail=0x10edd8, context=0x0) at imap-fetch.c:552 #10 0x2a4fc in imap_fetch_more (ctx=0xf6180) at imap-fetch.c:309 #11 0x2a744 in imap_fetch (ctx=0xf6180) at imap-fetch.c:361 #12 0x23628 in cmd_fetch (cmd=0xf60f0) at cmd-fetch.c:152 #13 0x28784 in client_command_input (cmd=0xf60f0) at client.c:580 #14 0x28628 in client_command_input (cmd=0xf60f0) at client.c:629 #15 0x2880c in client_handle_next_command (client=0xf5e90, remove_io_r=0xffbef2bf) at client.c:670 #16 0x288f4 in client_handle_input (client=0xf5e90) at client.c:680 #17 0x28a40 in client_input (client=0xf5e90) at client.c:725 #18 0xaa1a4 in io_loop_handler_run (ioloop=0xf29e8) at ioloop-poll.c:200 #19 0xa99d4 in io_loop_run (ioloop=0xf29e8) at ioloop.c:308 #20 0x31cf4 in main (argc=0, argv=0xffbef504, envp=0xffbef514) at main.c:293
I could send you the dovecot index files at each stage of the zlib test if that would help?
Chris
On Thu, Jul 24, 2008 at 04:30:28PM +0100, Chris Wakelin wrote:
Chris Wakelin wrote:
I can try this in Solaris 10, but I don't really see why the OS/architecture should matter.
Oh, unless there are some bugs in your zlib version, such as related to seeking..
Could be, I imagine it's as prehistoric as the bzlib :)
Right, I've tried dovecot-1.1.2 using my own zlib-1.2.3 (and bzlib-1.0.5) build on Solaris 8:
ldd lib20_zlib_plugin.so gives libz.so => /opt/RDGzlib/lib/libz.so libbz2.so => /opt/RDGbzlib/lib/libbz2.so ...
and get the same problems.
I cannot get Dovecot 1.1.2 to compile at all on Solaris8 with gcc 3.3.4 and newly recompiled versions of bzip2 and zlib libraries. This is the first time I've ever had a problem getting Dovecot to compile under Solaris 8.
Making all in zlib make[4]: Entering directory `/usr/local/src/dovecot/dovecot-1.1.2/src/plugins/zlib' /bin/bash ../../../libtool --tag=CC --mode=link gcc -std=gnu99 -O0 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -module -avoid-version -o lib20_zlib_plugin.la -rpath /usr/local/adm/dovecot/lib/dovecot istream-bzlib.lo istream-zlib.lo zlib-plugin.lo -lz -lbz2 -lrt -lnsl -lsocket gcc -shared -Wl,-h -Wl,lib20_zlib_plugin.so -o .libs/lib20_zlib_plugin.so .libs/istream-bzlib.o .libs/istream-zlib.o .libs/zlib-plugin.o -lz -lbz2 -lrt -lnsl -lsocket -lc Text relocation remains referenced against symbol offset in file <unknown> 0x3ab8 /usr/local/lib/libbz2.a(decompress.o) <unknown> 0x3abc /usr/local/lib/libbz2.a(decompress.o) <unknown> 0x3ac0 /usr/local/lib/libbz2.a(decompress.o) fgetc 0x20e8 /usr/local/lib/libbz2.a(bzlib.o) exit 0x3c /usr/local/lib/libbz2.a(bzlib.o) __ctype 0x2f60 /usr/local/lib/libbz2.a(bzlib.o) __ctype 0x2f68 /usr/local/lib/libbz2.a(bzlib.o) fopen64 0x3104 /usr/local/lib/libbz2.a(bzlib.o) ungetc 0x2100 /usr/local/lib/libbz2.a(bzlib.o) .div 0x8a8 /usr/local/lib/libbz2.a(compress.o) .div 0xabc /usr/local/lib/libbz2.a(blocksort.o) .div 0xe48 /usr/local/lib/libbz2.a(blocksort.o) ld: fatal: relocations remain against allocatable but non-writable sections collect2: ld returned 1 exit status make[4]: *** [lib20_zlib_plugin.la] Error 1
-- Dean Brooks dean@iglou.com
On Jul 24, 2008, at 1:11 PM, Dean Brooks wrote:
I cannot get Dovecot 1.1.2 to compile at all on Solaris8 with gcc 3.3.4 and newly recompiled versions of bzip2 and zlib libraries. This is the first time I've ever had a problem getting Dovecot to compile under Solaris 8.
..
<unknown> 0x3ab8 /usr/local/lib/ libbz2.a(decompress.o)
I think the problem is you're trying to link libbz2.a instead of
libbz2.so (you didn't build the .so at all?).
On Mon, Aug 04, 2008 at 02:26:50PM -0400, Timo Sirainen wrote:
On Jul 24, 2008, at 1:11 PM, Dean Brooks wrote:
I cannot get Dovecot 1.1.2 to compile at all on Solaris8 with gcc 3.3.4 and newly recompiled versions of bzip2 and zlib libraries. This is the first time I've ever had a problem getting Dovecot to compile under Solaris 8.
..
<unknown> 0x3ab8 /usr/local/lib/ libbz2.a(decompress.o)
I think the problem is you're trying to link libbz2.a instead of
libbz2.so (you didn't build the .so at all?).
When building the bz2 library from source, the .so shared library is not built by default. It also appears that when forcing the shared library of libbz2 to be built, the compile process fails under Solaris8 without some sort of modifications. Ugh.
Does dovecot require a shared version of the libbz2 library to compile correctly? Is there no way to statically link this?
Or, alternatively, is there a way to shut off this libbz2 plugin completely in the configure script?
-- Dean Brooks dean@iglou.com
On Aug 4, 2008, at 2:38 PM, Dean Brooks wrote:
Does dovecot require a shared version of the libbz2 library to compile correctly? Is there no way to statically link this?
You could link the libbz2.a into the imap binary (and hope that linker
doesn't optimize it away), but you can't link .a libraries to shared
libraries (plugins).
Or, alternatively, is there a way to shut off this libbz2 plugin completely in the configure script?
You could afterwards remove HAVE_BZLIB from config.h and remove it
from Makefile.
Wonder if there's a way in autoconf to detect if a library is shared..
On Mon, Aug 04, 2008 at 04:24:09PM -0400, Timo Sirainen wrote:
On Aug 4, 2008, at 2:38 PM, Dean Brooks wrote:
Does dovecot require a shared version of the libbz2 library to compile correctly? Is there no way to statically link this?
You could link the libbz2.a into the imap binary (and hope that linker
doesn't optimize it away), but you can't link .a libraries to shared
libraries (plugins).Or, alternatively, is there a way to shut off this libbz2 plugin completely in the configure script?
You could afterwards remove HAVE_BZLIB from config.h and remove it
from Makefile.Wonder if there's a way in autoconf to detect if a library is shared..
Is there a way to implement a --without-bzlib style option so people can disable compilation of some of these plugins?
It seems like a bad idea to make Dovecot completely contingent on having a specific .so available on a system. Especially given that the .so version of bzlib isn't even installed by default when compiling the bzlib library.
-- Dean Brooks dean@iglou.com
On Aug 5, 2008, at 11:44 AM, Dean Brooks wrote:
Is there a way to implement a --without-bzlib style option so people can disable compilation of some of these plugins?
Chris Wakelin wrote:
Oh, unless there are some bugs in your zlib version, such as related to seeking..
Could be, I imagine it's as prehistoric as the bzlib :)
Right, I've tried dovecot-1.1.2 using my own zlib-1.2.3 (and bzlib-1.0.5) build on Solaris 8:
ldd lib20_zlib_plugin.so gives libz.so => /opt/RDGzlib/lib/libz.so libbz2.so => /opt/RDGbzlib/lib/libbz2.so ...
and get the same problems.
I've been testing zlib again in Dovecot 1.1.1 (plus assert-crash fix) and 1.1.2 and I'm beginning to think it may be some sort of race condition (the Solaris 8 box is significantly slower than the SuSE one!).
If I leave a few seconds pause between the "FETCH <uid> BODY[]" commands, it seems not to crash, even with the same sequence. Mind you, I wouldn't have expected to be able to type quickly enough to cause problems!
Is that plausible?
Best Wishes, Chris
Chris Wakelin wrote:
I've been testing zlib again in Dovecot 1.1.1 (plus assert-crash fix) and 1.1.2 and I'm beginning to think it may be some sort of race condition (the Solaris 8 box is significantly slower than the SuSE one!).
If I leave a few seconds pause between the "FETCH <uid> BODY[]" commands, it seems not to crash, even with the same sequence. Mind you, I wouldn't have expected to be able to type quickly enough to cause problems!
Is that plausible?
Best Wishes, Chris
Still a problem with 1.1.3rc. I've got some truss output (attached) which shows some differences depending on whether I wait a few seconds between FETCHes. The sequence of events is
(remove .imap/test.gz directory)
LOGIN ... SELECT test.gz
(start truss on imap process)
FETCH 1:167 FULL FETCH 167 BODY[] FETCH 37 BODY[] FETCH 123 BODY[]
With a pause it works; without, the FETCH 37 bombs out.
Any idea what's going wrong?
Best Wishes, Chris
participants (3)
-
Chris Wakelin
-
Dean Brooks
-
Timo Sirainen