Hi,
I'm trying to install dovecot-0.99.9-test6 with gcc-3.2.2 under AIX (4.3.3) and found some problems.
First, I need to define _XOPEN_SOURCE_EXTENDED to 1 in src/lib/fdpass.c and src/auth/mycrypt.c or undefine it to compile successfully.
When I define _XOPEN_SOURCE_EXTENDED to 1, I get the following error while connecting:
dovecot: Apr 21 17:56:44 Info: Dovecot starting up imap-login: Apr 21 17:57:19 Fatal: fd_send(5) failed: Bad file number dovecot: Apr 21 17:57:19 Error: child 39504 (login) returned error 89
This happens after a successful authentication (i.e. wrong authentications are correctly denied). Something similar happens if I define _XOPEN_SOURCE to 500 in both files.
When I undefine the _XOPEN_SOURCE_EXTENDEDs without modifying the original _XOPEN_SOURCE defines, it seems to come further:
imap-login: Apr 21 18:24:23 Info: Login: xxx [xxx.xxx.xxx.xxx] imap(xxx): Apr 21 18:24:23 Panic: file mail-index-data.c: line 439 (mail_index_data_append): assertion failed: ((size & (MEM_ALIGN_SIZE-1)) == 0) dovecot: Apr 21 18:24:23 Error: child 95800 (imap) killed with signal 6
but this doesn't work either.
Anything other I could try?
Thanks, Adam
Adam adam@os.inf.tu-dresden.de Lackorzynski http://os.inf.tu-dresden.de/~adam/
On Mon, 2003-04-21 at 21:54, Adam Lackorzynski wrote:
Hi,
I'm trying to install dovecot-0.99.9-test6 with gcc-3.2.2 under AIX (4.3.3) and found some problems.
First, I need to define _XOPEN_SOURCE_EXTENDED to 1 in src/lib/fdpass.c and src/auth/mycrypt.c or undefine it to compile successfully. .. When I undefine the _XOPEN_SOURCE_EXTENDEDs without modifying the original _XOPEN_SOURCE defines, it seems to come further:
I guess leaving it 1 for mycrypt.c is fine. Does AIX have some "AIX" macro defined? So I could change fdpass.c to have: #ifndef AIX # define _XOPEN_SOURCE_EXTENDED /* for Tru64, breaks AIX */ #endif
imap(xxx): Apr 21 18:24:23 Panic: file mail-index-data.c: line 439 (mail_index_data_append): assertion failed: ((size & (MEM_ALIGN_SIZE-1)) == 0)
Hmm.. This is quite likely because struct mail_index_data_record_header isn't padded to take 8 bytes with 32bit off_t. Has all systems until now really had 64bit off_t? This should fix it: Index: mail-index.h =================================================================== RCS file: /home/cvs/dovecot/src/lib-index/mail-index.h,v retrieving revision 1.73 diff -u -r1.73 mail-index.h --- mail-index.h 21 Apr 2003 13:42:00 -0000 1.73 +++ mail-index.h 21 Apr 2003 19:58:55 -0000 @@ -167,6 +167,7 @@ unsigned int data_size; /* including this header */ time_t internal_date; + time_t filler; uoff_t virtual_size; uoff_t header_size; But I'm not sure if I want to commit this change. Maybe some systems have added the padding elsewhere? Changing the location could break things. After next release I'll break the index files anyway, I think I'll leave this fix until then.
On Mon Apr 21, 2003 at 23:02:13 +0300, Timo Sirainen wrote:
Hmm.. This is quite likely because struct mail_index_data_record_header isn't padded to take 8 bytes with 32bit off_t. Has all systems until now really had 64bit off_t? This should fix it:
+ time_t filler;
Now, the error message is different:
imap(xxx): Apr 21 20:42:14 Error: Corrupted index data file .../Mail/.imap/INBOX/.imap.index.data: Invalid location field for record 1
imap(xxx): Apr 21 20:42:14 Error: Corrupted index data file .../Mail/.imap/INBOX/.imap.index.data: Invalid location field for record 1
dovecot: Apr 21 20:42:14 Error: child 73232 (imap) killed with signal 11
I deleted the .imap dir before connecting and the mbox file is empty in
this case here.
If the mbox contains some mails (7 in this case), I'm getting this
(.imap dir also didn't exist before):
imap(xxx): Apr 21 20:52:52 Error: Corrupted index data file .../Mail/test/.imap/box2/.imap.index.data: Invalid location field for record 7
That's what I have for now. The AC_AIX is probably not needed...
diff -ur dovecot-0.99.9-test6.orig/configure.in dovecot-0.99.9-test6/configure.in
--- dovecot-0.99.9-test6.orig/configure.in Mon Apr 21 17:54:11 2003
+++ dovecot-0.99.9-test6/configure.in Mon Apr 21 22:06:09 2003
@@ -5,6 +5,7 @@
AM_MAINTAINER_MODE
+AC_AIX
AC_ISC_POSIX
AC_PROG_CC
AC_PROG_CPP
diff -ur dovecot-0.99.9-test6.orig/src/auth/mycrypt.c dovecot-0.99.9-test6/src/auth/mycrypt.c
--- dovecot-0.99.9-test6.orig/src/auth/mycrypt.c Tue Nov 26 19:39:57 2002
+++ dovecot-0.99.9-test6/src/auth/mycrypt.c Mon Apr 21 22:15:16 2003
@@ -1,5 +1,5 @@
#define _XOPEN_SOURCE 4
-#define _XOPEN_SOURCE_EXTENDED
+#define _XOPEN_SOURCE_EXTENDED 1
#define _XOPEN_VERSION 4
#define _XPG4_2
#include
On Mon, 2003-04-21 at 23:57, Adam Lackorzynski wrote:
On Mon Apr 21, 2003 at 23:02:13 +0300, Timo Sirainen wrote:
Hmm.. This is quite likely because struct mail_index_data_record_header isn't padded to take 8 bytes with 32bit off_t. Has all systems until now really had 64bit off_t? This should fix it:
time_t filler;Now, the error message is different:
imap(xxx): Apr 21 20:42:14 Error: Corrupted index data file .../Mail/.imap/INBOX/.imap.index.data: Invalid location field for record 1
Umm. Are you sure the sources got fully rebuilt after you added the filler? That looks like it..
On Tue Apr 22, 2003 at 22:50:01 +0300, Timo Sirainen wrote:
imap(xxx): Apr 21 20:42:14 Error: Corrupted index data file .../Mail/.imap/INBOX/.imap.index.data: Invalid location field for record 1
Umm. Are you sure the sources got fully rebuilt after you added the filler? That looks like it..
Yes, "make clean"'ed it before.
Adam
Adam adam@os.inf.tu-dresden.de Lackorzynski http://os.inf.tu-dresden.de/~adam/
On Wed, 2003-04-23 at 01:45, Adam Lackorzynski wrote:
On Tue Apr 22, 2003 at 22:50:01 +0300, Timo Sirainen wrote:
imap(xxx): Apr 21 20:42:14 Error: Corrupted index data file .../Mail/.imap/INBOX/.imap.index.data: Invalid location field for record 1
Umm. Are you sure the sources got fully rebuilt after you added the filler? That looks like it..
Yes, "make clean"'ed it before.
Something's really messed up then. Maybe there's some mmap() problems, try if setting "default_mail_env = mbox:~/Mail:INDEX=MEMORY" helps?
On Wed Apr 23, 2003 at 02:50:54 +0300, Timo Sirainen wrote:
On Wed, 2003-04-23 at 01:45, Adam Lackorzynski wrote:
On Tue Apr 22, 2003 at 22:50:01 +0300, Timo Sirainen wrote:
imap(xxx): Apr 21 20:42:14 Error: Corrupted index data file .../Mail/.imap/INBOX/.imap.index.data: Invalid location field for record 1
Umm. Are you sure the sources got fully rebuilt after you added the filler? That looks like it..
Yes, "make clean"'ed it before.
Something's really messed up then. Maybe there's some mmap() problems, try if setting "default_mail_env = mbox:~/Mail:INDEX=MEMORY" helps?
This results in
dovecot: Apr 23 11:16:05 Error: child 86116 (imap) killed with signal 11
Is it worth finding out where the sigsegv in imap happens?
Adam
Adam adam@os.inf.tu-dresden.de Lackorzynski http://os.inf.tu-dresden.de/~adam/
On Wed, 2003-04-23 at 14:43, Adam Lackorzynski wrote:
Something's really messed up then. Maybe there's some mmap() problems, try if setting "default_mail_env = mbox:~/Mail:INDEX=MEMORY" helps?
This results in
dovecot: Apr 23 11:16:05 Error: child 86116 (imap) killed with signal 11
Is it worth finding out where the sigsegv in imap happens?
Yes, I'd like to know why it crashes. I've tried to prevent that everywhere. Two ways to do this:
a) Run the imap binary directly: MAIL=mbox:~/Mail:INDEX=MEMORY gdb /usr/local/libexec/dovecot/imap r x select inbox .. crash? bt
b) Set mail_drop_priv_before_exec = yes in config file, and core dump should be written to your home directory.
I think I should write this down somewhere in web :)
On Wed Apr 23, 2003 at 15:13:27 +0300, Timo Sirainen wrote:
Yes, I'd like to know why it crashes. I've tried to prevent that everywhere. Two ways to do this:
Program received signal SIGSEGV, Segmentation fault. 0xd017f528 in memset () from /usr/lib/libc.a(shr.o) (gdb) bt #0 0xd017f528 in memset () from /usr/lib/libc.a(shr.o) #1 0x1001e5a4 in mail_index_init_header (index=0x2000ef58, hdr=0xffffffff) at mail-index-open.c:231 #2 0x1001e820 in mail_index_create_memory (index=0x2000ef58, flags=19) at mail-index-open.c:289 #3 0x1001ec24 in mail_index_open (index=0x2000ef58, flags=3) at mail-index-open.c:411 #4 0x1000ded0 in index_storage_init (storage=0x200096a8, box=0x20000cb0, index=0x2000ef58, name=0x10065ae8 "INBOX", readonly=0, fast=0) at index-storage.c:259 #5 0x100401e0 in mbox_open (storage=0x200096a8, name=0x10065ae8 "INBOX", readonly=0, fast=0) at mbox-storage.c:360 #6 0x100402f0 in mbox_open_mailbox (storage=0x200096a8, name=0x20009a78 "inbox", readonly=0, fast=0) at mbox-storage.c:380 #7 0x1004e23c in _cmd_select_full (client=0x200097a8, readonly=0) at cmd-select.c:23 #8 0x1004e4c4 in cmd_select (client=0xffffffff) at cmd-select.c:87 #9 0x1004b9ec in client_handle_input (client=0x200097a8) at client.c:306 #10 0x1004bb5c in _client_input (context=0x200097a8) at client.c:342 #11 0x10001ec0 in io_loop_handler_run (ioloop=0x20008fe8) at ioloop-poll.c:212 #12 0x1000148c in io_loop_run (ioloop=0x20008fe8) at ioloop.c:295 #13 0x10000964 in main (argc=-1, argv=0x20004de4, envp=0x2ff22b80) at main.c:166 ---Type <return> to continue, or q <return> to quit--- #14 0x100001dc in __start () (gdb)
The problem seems to be that
index->mmap_base = mmap_anon(index->mmap_full_length);
in lib-index/mail-index-open.c(287) returns -1 (MAP_FAILED) which is then fed to mail_index_init_header(..., 0xffffffff) without being checked.
The mmap in anon_mmap_fixed in lib/mmap-anon.c returns "Not enough space" in errno. The parameter values are "address=0x00002000 length=0x00002000".
Adam
Adam adam@os.inf.tu-dresden.de Lackorzynski http://os.inf.tu-dresden.de/~adam/
On Wed, 2003-04-23 at 15:56, Adam Lackorzynski wrote:
The problem seems to be that
index->mmap_base = mmap_anon(index->mmap_full_length);
in lib-index/mail-index-open.c(287) returns -1 (MAP_FAILED) which is then fed to mail_index_init_header(..., 0xffffffff) without being checked.
Right, I've missed all of them for some reason. Added to CVS.
The mmap in anon_mmap_fixed in lib/mmap-anon.c returns "Not enough space" in errno. The parameter values are "address=0x00002000 length=0x00002000".
Linux and *BSD returns only EINVAL if address and/or length is invalid, but UNIX98 said ENOMEM can be returned too then. This is a bit annoying in that I can't now know if the mmap() failed because there's really no memory or because the address was just invalid. If it's because there's no memory, Dovecot will try a bit too long before failing. This should anyway fix: RCS file: /home/cvs/dovecot/src/lib/mmap-anon.c,v retrieving revision 1.10 diff -u -r1.10 mmap-anon.c --- mmap-anon.c 29 Dec 2002 19:33:04 -0000 1.10 +++ mmap-anon.c 23 Apr 2003 13:23:41 -0000 @@ -176,7 +176,7 @@ if (ret == 0) break; - if (ret < 0 && errno != EINVAL) + if (ret < 0 && errno != EINVAL && errno != ENOMEM) return MAP_FAILED; } @@ -200,7 +200,7 @@ } if (anon_mmap_fixed(grow_base, new_size - hdr->size) < 0) { - if (errno == EINVAL) { + if (errno == EINVAL || errno == ENOMEM) { /* can't grow, wanted address space is already in use */ return 0; }
On Wed Apr 23, 2003 at 16:29:55 +0300, Timo Sirainen wrote:
Linux and *BSD returns only EINVAL if address and/or length is invalid, but UNIX98 said ENOMEM can be returned too then. This is a bit annoying in that I can't now know if the mmap() failed because there's really no memory or because the address was just invalid. If it's because there's no memory, Dovecot will try a bit too long before failing. This should anyway fix:
Hmm, mmap seems broken somehow or something:
x select inbox address=00002000 length=00002000 mmap: : Not enough space address=04002000 length=00002000 mmap: : Not enough space address=08002000 length=00002000 mmap: : Not enough space address=0c002000 length=00002000 mmap: : Not enough space address=10002000 length=00002000 mmap: : Not enough space address=14002000 length=00002000 mmap: : Not enough space address=18002000 length=00002000 mmap: : Not enough space address=1c002000 length=00002000 mmap: : Not enough space address=00002000 length=00002000 mmap: : Not enough space ....
address=1fed2000 length=00002000 mmap: : Not enough space address=1fee2000 length=00002000 mmap: : Not enough space address=1fef2000 length=00002000 mmap: : Not enough space address=1ff02000 length=00002000 mmap: : Not enough space address=1ff12000 length=00002000 mmap: : Not enough space
Program received signal SIGSEGV, Segmentation fault. 0xd017f528 in memset () from /usr/lib/libc.a(shr.o) (gdb) bt #0 0xd017f528 in memset () from /usr/lib/libc.a(shr.o) #1 0x1001e64c in mail_index_init_header (index=0x2000f038, hdr=0xffffffff) at mail-index-open.c:231 #2 0x1001e8c8 in mail_index_create_memory (index=0x2000f038, flags=19) at mail-index-open.c:289 #3 0x1001eccc in mail_index_open (index=0x2000f038, flags=3) at mail-index-open.c:411 ...
The output is from anon_mmap_fixed.
I don't have the return checks in yet, that's why the sigsegv.
Adam
Adam adam@os.inf.tu-dresden.de Lackorzynski http://os.inf.tu-dresden.de/~adam/
On Wed, 2003-04-23 at 18:03, Adam Lackorzynski wrote:
Hmm, mmap seems broken somehow or something:
x select inbox address=00002000 length=00002000 .. address=00002000 length=00002000 mmap: : Not enough space
Hmm. It wraps around to original location? It shouldn't do that. Or maybe that's just different mmap_anon() call.
Anyway, I changed it now so that it fallbacks to non-fixed mmap() after 100 tries. mremap_anon() was also somewhat broken. I put 0.99.9-test7 to web page which fixes these and other stuff.
On Wed Apr 23, 2003 at 18:58:43 +0300, Timo Sirainen wrote:
Anyway, I changed it now so that it fallbacks to non-fixed mmap() after 100 tries. mremap_anon() was also somewhat broken. I put 0.99.9-test7 to web page which fixes these and other stuff.
The define in fdpass.c has to be _AIX not just AIX.
In mail-index-open.c:289(mail_index_create_memory) you use a path variable which is undefined.
Now, with the filler still in, it works for an empty inbox. If a mailbox contains something (7 mails in this case), I get:
imap-login: Apr 23 16:22:16 Info: Login: xxx [x.x.x.x] imap(xxx): Apr 23 16:22:16 Error: Corrupted index data file HOME/Mail/f/.imap/sf1/.imap.index.data: Invalid location field for record 7
I deleted the .imap dirs before.
And for "default_mail_env = mbox:~/Mail:INDEX=MEMORY", I'm getting:
imap(xxx): Apr 23 16:16:54 Error: msync() failed with index file (in-memory index for HOME/Mail/f/sf1): Invalid argument
Adam
Adam adam@os.inf.tu-dresden.de Lackorzynski http://os.inf.tu-dresden.de/~adam/
On Wed, 2003-04-23 at 19:32, Adam Lackorzynski wrote:
In mail-index-open.c:289(mail_index_create_memory) you use a path variable which is undefined.
Whops. Should have compiled it first :)
Now, with the filler still in, it works for an empty inbox. If a mailbox contains something (7 mails in this case), I get:
imap-login: Apr 23 16:22:16 Info: Login: xxx [x.x.x.x] imap(xxx): Apr 23 16:22:16 Error: Corrupted index data file HOME/Mail/f/.imap/sf1/.imap.index.data: Invalid location field for record 7
Yes, what I'd like to know is if INDEX=MEMORY gives the same error once it actually works.
And for "default_mail_env = mbox:~/Mail:INDEX=MEMORY", I'm getting:
imap(xxx): Apr 23 16:16:54 Error: msync() failed with index file (in-memory index for HOME/Mail/f/sf1): Invalid argument
OK, 0.99.9-test8 now doesn't call msync() for anon-mmaped memory.
On Wed Apr 23, 2003 at 20:10:20 +0300, Timo Sirainen wrote:
On Wed, 2003-04-23 at 19:32, Adam Lackorzynski wrote:
Now, with the filler still in, it works for an empty inbox. If a mailbox contains something (7 mails in this case), I get:
imap-login: Apr 23 16:22:16 Info: Login: xxx [x.x.x.x] imap(xxx): Apr 23 16:22:16 Error: Corrupted index data file HOME/Mail/f/.imap/sf1/.imap.index.data: Invalid location field for record 7
Yes, what I'd like to know is if INDEX=MEMORY gives the same error once it actually works.
Without "INDEX=MEMORY", I'm getting the "Corrupted index data file...Invalid location field for record 7" error as above.
With "INDEX=MEMORY", I'm now getting:
imap-login: Apr 23 18:46:44 Info: Login: xxx [x.x.x.x] imap(xxx): Apr 23 18:46:44 Error: Corrupted index data file (in-memory index data index for HOME/Mail/f/sf1): Invalid location field for record 7 imap(xxx): Apr 23 18:46:44 Error: Corrupted index data file (in-memory index data index for HOME/Mail/f/sf1): Invalid location field for record 1 imap(xxx): Apr 23 18:46:44 Error: Corrupted index data file (in-memory index data index for HOME/Mail/f/sf1): Invalid location field for record 7 imap(xxx): Apr 23 18:46:44 Error: Corrupted index data file (in-memory index data index for HOME/Mail/f/sf1): Invalid location field for record 1
Adam
Adam adam@os.inf.tu-dresden.de Lackorzynski http://os.inf.tu-dresden.de/~adam/
On Wed, 2003-04-23 at 21:53, Adam Lackorzynski wrote:
imap(xxx): Apr 23 18:46:44 Error: Corrupted index data file (in-memory index data index for HOME/Mail/f/sf1): Invalid location field for record 7
Got it, the struct sizes aren't padded to 8 bytes but Dovecot wants that. So it pads them while writing, but the reading part is a bit too pedantic and wants exact sizes. For now the easiest fix is to configure --with-mem-align=4. I changed configure to do this automatically now with 32bit file offsets. Have to fix this properly later..
I just compiled and tested Dovecot under AIX 4.3, compiled with xlc. Seems to be working.
On Fri, 2003-04-25 at 04:34, Timo Sirainen wrote:
Got it, the struct sizes aren't padded to 8 bytes but Dovecot wants that. So it pads them while writing, but the reading part is a bit too pedantic and wants exact sizes. For now the easiest fix is to configure --with-mem-align=4. I changed configure to do this automatically now with 32bit file offsets. Have to fix this properly later..
Or actually maybe I should fix it properly now. This breaks if pointers are 64bit but off_t is 32bit, and CPU doesn't want to do unaligned memory accesses..
Index files should probably have separate alignment behaviour since it can't contain pointers. off_t is likely to be the only possible 64bit value in it.
On Fri Apr 25, 2003 at 04:34:56 +0300, Timo Sirainen wrote:
Got it, the struct sizes aren't padded to 8 bytes but Dovecot wants that. So it pads them while writing, but the reading part is a bit too pedantic and wants exact sizes. For now the easiest fix is to configure --with-mem-align=4. I changed configure to do this automatically now with 32bit file offsets. Have to fix this properly later..
I just compiled and tested Dovecot under AIX 4.3, compiled with xlc. Seems to be working.
Works for me as well now (with the filler still in and with gcc). Thanks a lot!
Adam
Adam adam@os.inf.tu-dresden.de Lackorzynski http://os.inf.tu-dresden.de/~adam/
participants (2)
-
Adam Lackorzynski
-
Timo Sirainen