[Dovecot] 1.0rc8: another problem? Possibly 64-bit index?
Yesterday I gave a status report about 10.rc8 in production, which mentioned a problem about "Login process died too early..."
Timo suggested a patch for logging error messages. I've applied this.
Others suggested increasing "login_max_processes_count". That was already way above our likely maximum, but I've doubled it anyway.
Today, I've just repeated the trial, but with one significant difference and one significant new problem. The significant difference is that today's machine is not a 32-bit i686 but a 64-bit (and faster) x86_64.
The new problem is lots of occurences of:
[...]: IMAP([...]): file ../../../src/lib-index/mail-cache-transaction.c: line 709 (mail_cache_add): assertion failed: (fixed_size == (unsigned int)-1 || fixed_size == data_size) [...]: child 4365 (imap) killed with signal 6
and:
[...]: IMAP([...]): file ../../../../src/lib-storage/index/index-mail.c: line 105 (index_mail_get_fixed_field): assertion failed: (buffer_get_used_size(buf) == data_size) [...]: child 15009 (imap) killed with signal 6
Naturally, I've backed off quickly.
Theoretically, this problem could be attributed to any of the three changes: logging, max processes, different (64-bit) machine.
I suspect it's not the "login_max_processes_count" increase.
Is it a side-effect of the better logging? (That is, was it always happening but the information simply not getting logged?)
Note that our index files (and mail spool (INBOX, folder) areas) are on an NFS (NetApp) shared area accessed by all three classes of our machines: o existing UW-IMAP (which doesn't use the indexes) o yesterday-dovecot (32-bit) o today-dovecot (64-bit)
My sneaky suspicion is that this is a subtle 64-bit-ism, either on its own, or in some sort of interaction with an indexes created from yesterday's 32-bit client.
(Yes, theoretically, I could try another test to attempt to confirm or eliminate the 32/64 difference. But these issues didn't show up under testing, but only when we began trial service; I can't experiment on the users!)
--
: David Lee I.T. Service : : Senior Systems Programmer Computer Centre : : Durham University : : http://www.dur.ac.uk/t.d.lee/ South Road : : Durham DH1 3LE : : Phone: +44 191 334 2752 U.K. :
David Lee wrote:
My sneaky suspicion is that this is a subtle 64-bit-ism, either on its own, or in some sort of interaction with an indexes created from yesterday's 32-bit client.
(Yes, theoretically, I could try another test to attempt to confirm or eliminate the 32/64 difference. But these issues didn't show up under testing, but only when we began trial service; I can't experiment on the users!)
When you move from 32-bit to 64-bit you need to rebuild the indexes (known problem). See, for example http://dovecot.org/list/dovecot/2006-October/016595.html
Best Wishes, Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
On Thu, 2006-10-12 at 15:58 +0100, David Lee wrote:
[...]: IMAP([...]): file ../../../src/lib-index/mail-cache-transaction.c: line 709 (mail_cache_add): assertion failed: (fixed_size == (unsigned int)-1 || fixed_size == data_size) [...]: child 4365 (imap) killed with signal 6
Yep, this is a problem with dovecot.index.cache file created with 32bit machine not working in 64bit machine. It should automatically detect this and delete it, but for some reason it doesn't seem to work always. You could do the exact same thing manually anyway by deleting them.
Timo Sirainen wrote:
Yep, this is a problem with dovecot.index.cache file created with 32bit machine not working in 64bit machine. It should automatically detect this and delete it, but for some reason it doesn't seem to work always. You could do the exact same thing manually anyway by deleting them.
Yes, but the OP was reporting trying to run *both* a 32-bit and a 64-bit server sharing the same NFS-mounted filesystem. In other words, either you (Timo) need to change the indexes to be platform independent (i.e. no 64-bit-isms) or you need to put a BIG WARNING to not share the message indexes between different width platforms. The performance hit of repeated nuking of the index files is going to be huge.
Obviously, the OP could also configure the index files to be located on a local filesystem (i.e. not shared in the message store), which should also be higher performance under all circumstances with networked message stores...
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748
On Thu, 12 Oct 2006, John Peacock wrote:
Timo Sirainen wrote:
Yep, this is a problem with dovecot.index.cache file created with 32bit machine not working in 64bit machine. It should automatically detect this and delete it, but for some reason it doesn't seem to work always. You could do the exact same thing manually anyway by deleting them.
Yes, but the OP was reporting trying to run *both* a 32-bit and a 64-bit server sharing the same NFS-mounted filesystem. In other words, either you (Timo) need to change the indexes to be platform independent (i.e. no 64-bit-isms) or you need to put a BIG WARNING to not share the message indexes between different width platforms. The performance hit of repeated nuking of the index files is going to be huge.
Indeed. A fix, or a warning, would be very useful
Obviously, the OP could also configure the index files to be located on a local filesystem (i.e. not shared in the message store), which should also be higher performance under all circumstances with networked message stores...
In such a set-up (set of front end IMAP/LDA machines, accessing shared inbox/folder stores over NFS), what is the recommended place for the indexes? Shared (logically alongside the shared inbox/folder areas)? Or privately within each IMAP/LDA machine?
What are the locking implications on a busy mailstore?(*) Although we try to keep all activity for a given user on one machine, we cannot guarantee this. Suppose an email LDA/delivery occurs on one machine concurrently with an IMAP update to the same folder from a different machine? Or similarly multiple simultaneous deliveries on different machines? Etc. What are the locking considerations for indexes then?
(*) We use trad. UNIX format. Not ideal. But we're not in a position to migrate to (say) maildir.
--
: David Lee I.T. Service : : Senior Systems Programmer Computer Centre : : Durham University : : http://www.dur.ac.uk/t.d.lee/ South Road : : Durham DH1 3LE : : Phone: +44 191 334 2752 U.K. :
Yes, but the OP was reporting trying to run *both* a 32-bit and a 64-bit server sharing the same NFS-mounted filesystem. In other words, either you (Timo) need to change the indexes to be platform independent (i.e. no 64-bit-isms) or you need to put a BIG WARNING to not share the message indexes between different width platforms. The performance hit of repeated nuking of the index files is going to be huge.
I thought that would be an obvious thing as a DON'T - I think many applications would choke on that one.
Yours, JC
(06.10.12 kl.19:06) Jakob Curdes skrev följande till :
Yes, but the OP was reporting trying to run *both* a 32-bit and a 64-bit server sharing the same NFS-mounted filesystem. In other words, either you (Timo) need to change the indexes to be platform independent (i.e. no 64-bit-isms) or you need to put a BIG WARNING to not share the message indexes between different width platforms. The performance hit of repeated nuking of the index files is going to be huge.
I thought that would be an obvious thing as a DON'T - I think many applications would choke on that one.
Or you can separate the indexes with a little loss of storage space. I guess it could even be directly supported by dovecot. Ex .imap32, .imap64
Cheers, Jens
Yours, JC
'In theory, there is no difference between theory and practice.
But, in practice, there is.' Jens Låås Email: jens.laas@data.slu.se
Department of Computer Services, SLU Phone: +46 18 67 35 15
Vindbrovägen 1
P.O. Box 7079
S-750 07 Uppsala
SWEDENJens Laas wrote:
Or you can separate the indexes with a little loss of storage space. I guess it could even be directly supported by dovecot. Ex .imap32, .imap64
It shouldn't be hard to make the indexes machine independent (e.g. 32/64 bit and even little-endian/big-endian - see htonl(3)).
I haven't looked at the code. Perhaps that's what it intends to do.
Jonathan.
On Fri, 2006-10-13 at 12:53 +0100, Jonathan wrote:
Jens Laas wrote:
Or you can separate the indexes with a little loss of storage space. I guess it could even be directly supported by dovecot. Ex .imap32, .imap64
It shouldn't be hard to make the indexes machine independent (e.g. 32/64 bit
I guess I'll do this. The problem is only these time_t fields that I'm using directly. I think we can manage for a while with 32bit timestamps. :) Although this means that in next release the dovecot.index.cache files will be rebuilt in 64bit systems.
Also uoff_t is used directly in there, but I'll leave them since they're 64bit by default also with 32bit machines.
and even little-endian/big-endian - see htonl(3)).
This isn't that simple. The index files are directly mmap()ed to memory and the data is used all over the place. It'd slow down and bloat everything.
On Fri, October 13, 2006 2:41 am, Jens Laas wrote:
(06.10.12 kl.19:06) Jakob Curdes skrev följande till :
Yes, but the OP was reporting trying to run *both* a 32-bit and a 64-bit server sharing the same NFS-mounted filesystem. In other words, either you (Timo) need to change the indexes to be platform independent (i.e. no 64-bit-isms) or you need to put a BIG WARNING to not share the message indexes between different width platforms. The performance hit of repeated nuking of the index files is going to be huge.
I thought that would be an obvious thing as a DON'T - I think many applications would choke on that one.
Or you can separate the indexes with a little loss of storage space. I guess it could even be directly supported by dovecot. Ex .imap32, .imap64
Probably better would be to use the machine hardware name (eg "i686") rather than bits. Obviously truly platform-independent indexes would be best.
Jim
On Thu, 12 Oct 2006, Timo Sirainen wrote:
On Thu, 2006-10-12 at 15:58 +0100, David Lee wrote:
[...]: IMAP([...]): file ../../../src/lib-index/mail-cache-transaction.c: line 709 (mail_cache_add): assertion failed: (fixed_size == (unsigned int)-1 || fixed_size == data_size) [...]: child 4365 (imap) killed with signal 6
Yep, this is a problem with dovecot.index.cache file created with 32bit machine not working in 64bit machine. It should automatically detect this and delete it, but for some reason it doesn't seem to work always. You could do the exact same thing manually anyway by deleting them.
OK. Thanks. (And to Chris Wakelin for similar information.)
Does that mean that a set of front-end machines, all NFS-mounting from a common area, need (pragmatically) to be all 32-bit or all 64-bit? (Is this documented?)
--
: David Lee I.T. Service : : Senior Systems Programmer Computer Centre : : Durham University : : http://www.dur.ac.uk/t.d.lee/ South Road : : Durham DH1 3LE : : Phone: +44 191 334 2752 U.K. :
participants (8)
-
Chris Wakelin
-
David Lee
-
Jakob Curdes
-
Jens Laas
-
Jim Trigg
-
John Peacock
-
Jonathan
-
Timo Sirainen