[Dovecot] record points outside file error with dovecot revision 9116:9ae55b68cf61
I use mbox with mbox-snarf plugin.
I get errors like:
dovecot: IMAP(testuser): Corrupted index cache file /full/path/ dovecot.index.cache: record points outside file
The errors can be in either folders or INBOX.
I put some syslog statements in src/lib-index/mail-cache-lookup.c to
help understand this.
if (offset + sizeof(*rec) > cache->mmap_length) {
mail_cache_set_corrupted(cache, "record points
outside file");
syslog(LOG_DEBUG,"mail_cache_get_record rec->size is
%d ", rec->size);
syslog(LOG_DEBUG,"cache->mmap_length is %d ", cache-
mmap_length); syslog(LOG_DEBUG,"offset is %d ", offset); return -1; }
syslog: mail_cache_get_record rec->size is 268595472 syslog: cache->mmap_length is 4096 syslog: offset is 1630760037
Oh I forgot to print off sizeof(*rec). Would that help?
Thanks, Jonathan
On Tue, 2009-06-02 at 09:51 -0400, Jonathan Siegle wrote:
I use mbox with mbox-snarf plugin.
I get errors like:
dovecot: IMAP(testuser): Corrupted index cache file /full/path/ dovecot.index.cache: record points outside file
So you're using AIX? Do you also happen to use NFS? Can you reproduce this error by running imaptest for a while? http://imapwiki.org/ImapTest
On Tue, 2 Jun 2009, Timo Sirainen wrote:
On Tue, 2009-06-02 at 09:51 -0400, Jonathan Siegle wrote:
I use mbox with mbox-snarf plugin.
I get errors like:
dovecot: IMAP(testuser): Corrupted index cache file /full/path/ dovecot.index.cache: record points outside file
So you're using AIX? Do you also happen to use NFS? Can you reproduce this error by running imaptest for a while? http://imapwiki.org/ImapTest
No NFS here. The filesystem is called GPFS. It is a clustered FS. I'll work on reproducing with imaptest. Once I reproduce, I'll give more details about how I run dovecot.
thanks, Jonathan
On Tue, 2009-06-02 at 13:24 -0400, jsiegle@psu.edu wrote:
dovecot: IMAP(testuser): Corrupted index cache file /full/path/ dovecot.index.cache: record points outside file
So you're using AIX? Do you also happen to use NFS? Can you reproduce this error by running imaptest for a while? http://imapwiki.org/ImapTest
No NFS here. The filesystem is called GPFS. It is a clustered FS.
Ah, that probably explains it. Can multiple different servers modify the same mailbox? Cache file is the part of Dovecot that demands the most from the OS/filesystem. The most difficult part is probably that it writes to the file without locking. It first reserves a space and then starts writing there. Multiple processes can write to the same file at the same time. This should probably be changed at some point, since it could just buffer more data to memory and then lock, write, unlock. That would also make the code simpler, since it can currently leave holes to the file because it has to guess initially how much space to reserve..
On Tue, 2 Jun 2009, Timo Sirainen wrote:
On Tue, 2009-06-02 at 13:24 -0400, jsiegle@psu.edu wrote:
dovecot: IMAP(testuser): Corrupted index cache file /full/path/ dovecot.index.cache: record points outside file
So you're using AIX? Do you also happen to use NFS? Can you reproduce this error by running imaptest for a while? http://imapwiki.org/ImapTest
No NFS here. The filesystem is called GPFS. It is a clustered FS.
Ah, that probably explains it. Can multiple different servers modify the same mailbox?
Yes. I have N+1 machines that access the same filesystem and it looks local to both machines.
Cache file is the part of Dovecot that demands the most from the OS/filesystem. The most difficult part is probably that it writes to the file without locking. It first reserves a space and then starts writing there. Multiple processes can write to the same file at the same time. This should probably be changed at some point, since it could just buffer more data to memory and then lock, write, unlock. That would also make the code simpler, since it can currently leave holes to the file because it has to guess initially how much space to reserve..
Ok. Should I disable Cache file or what would be the best fix?
Thanks, Jonathan
On Tue, 2009-06-02 at 13:40 -0400, jsiegle@psu.edu wrote:
Cache file is the part of Dovecot that demands the most from the OS/filesystem. The most difficult part is probably that it writes to the file without locking. It first reserves a space and then starts writing there. Multiple processes can write to the same file at the same time. This should probably be changed at some point, since it could just buffer more data to memory and then lock, write, unlock. That would also make the code simpler, since it can currently leave holes to the file because it has to guess initially how much space to reserve..
Ok. Should I disable Cache file or what would be the best fix?
If you do that, you'll lose all the performance advantages of Dovecot. Do you already have mmap_disable=yes? You could also try if setting mail_nfs_index=yes happens to help.
On Jun 2, 2009, at 1:31 PM, Timo Sirainen wrote:
On Tue, 2009-06-02 at 13:24 -0400, jsiegle@psu.edu wrote:
dovecot: IMAP(testuser): Corrupted index cache file /full/path/ dovecot.index.cache: record points outside file
So you're using AIX? Do you also happen to use NFS? Can you
reproduce this error by running imaptest for a while? http://imapwiki.org/ImapTestNo NFS here. The filesystem is called GPFS. It is a clustered FS.
Ah, that probably explains it. Can multiple different servers modify
the same mailbox? Cache file is the part of Dovecot that demands the most from the OS/filesystem. The most difficult part is probably that it writes to the file without locking. It first reserves a space and then starts writing there. Multiple processes can write to the same file at the same time.
Timo,
Are you saying that multiple processes on the same folder(INBOX) on
the same IMAP server can cause this collision as well? Is there a
difference between running multiple processes on the same
folder(INBOX) on multiple IMAP servers vs running multiple processes
on the same folder on a single IMAP server?
This should probably be changed at some point, since it could just buffer more data to memory and then lock, write, unlock.
That would also make the code simpler, since it can currently leave holes
to the file because it has to guess initially how much space to reserve..
This is definitely on my wish list.
Thanks, Jonathan
On Wed, 2009-06-03 at 09:31 -0400, Jonathan Siegle wrote:
Are you saying that multiple processes on the same folder(INBOX) on
the same IMAP server can cause this collision as well? Is there a
difference between running multiple processes on the same
folder(INBOX) on multiple IMAP servers vs running multiple processes
on the same folder on a single IMAP server?
I don't know. That depends on how GPFS is implemented.
On Jun 3, 2009, at 9:35 AM, Timo Sirainen wrote:
On Wed, 2009-06-03 at 09:31 -0400, Jonathan Siegle wrote:
Are you saying that multiple processes on the same folder(INBOX) on the same IMAP server can cause this collision as well? Is there a difference between running multiple processes on the same folder(INBOX) on multiple IMAP servers vs running multiple processes on the same folder on a single IMAP server?
I don't know. That depends on how GPFS is implemented.
Pick a local filesystem, say ext3?
On Wed, 2009-06-03 at 10:14 -0400, Jonathan Siegle wrote:
On Jun 3, 2009, at 9:35 AM, Timo Sirainen wrote:
On Wed, 2009-06-03 at 09:31 -0400, Jonathan Siegle wrote:
Are you saying that multiple processes on the same folder(INBOX) on the same IMAP server can cause this collision as well? Is there a difference between running multiple processes on the same folder(INBOX) on multiple IMAP servers vs running multiple processes on the same folder on a single IMAP server?
I don't know. That depends on how GPFS is implemented.
Pick a local filesystem, say ext3?
But with ext3 you can't have multiple servers accessing the same filesystem.
On Wed, 2009-06-03 at 12:11 -0400, Timo Sirainen wrote:
On Wed, 2009-06-03 at 10:14 -0400, Jonathan Siegle wrote:
On Jun 3, 2009, at 9:35 AM, Timo Sirainen wrote:
On Wed, 2009-06-03 at 09:31 -0400, Jonathan Siegle wrote:
Are you saying that multiple processes on the same folder(INBOX) on the same IMAP server can cause this collision as well? Is there a difference between running multiple processes on the same folder(INBOX) on multiple IMAP servers vs running multiple processes on the same folder on a single IMAP server?
I don't know. That depends on how GPFS is implemented.
Pick a local filesystem, say ext3?
But with ext3 you can't have multiple servers accessing the same filesystem.
But of course there are no problems (well, some very rare random ones maybe) having multiple processes accessing the same mailbox on the same server. Ever since I wrote my imaptest tool (a few years ago?) I've been heavily stress testing multiple connections modifying the mailbox at the same time.
On Jun 3, 2009, at 12:17 PM, Timo Sirainen wrote:
On Wed, 2009-06-03 at 12:11 -0400, Timo Sirainen wrote:
On Wed, 2009-06-03 at 10:14 -0400, Jonathan Siegle wrote:
On Jun 3, 2009, at 9:35 AM, Timo Sirainen wrote:
On Wed, 2009-06-03 at 09:31 -0400, Jonathan Siegle wrote:
Are you saying that multiple processes on the same
folder(INBOX) on the same IMAP server can cause this collision as well? Is there a difference between running multiple processes on the same folder(INBOX) on multiple IMAP servers vs running multiple
processes on the same folder on a single IMAP server?I don't know. That depends on how GPFS is implemented.
Pick a local filesystem, say ext3?
But with ext3 you can't have multiple servers accessing the same filesystem.
But of course there are no problems (well, some very rare random ones maybe) having multiple processes accessing the same mailbox on the
same server. Ever since I wrote my imaptest tool (a few years ago?) I've
been heavily stress testing multiple connections modifying the mailbox at
the same time.
Timo,
Can you tell me what platform/filesystem you use for testing? Oh and
can you hint at what "bad things" may happen when I get the error
"record points outside file"?
Thanks, Jonathan
On Jun 9, 2009, at 6:40 AM, Jonathan Siegle wrote:
But of course there are no problems (well, some very rare random ones maybe) having multiple processes accessing the same mailbox on the
same server. Ever since I wrote my imaptest tool (a few years ago?) I've
been heavily stress testing multiple connections modifying the mailbox
at the same time.Timo, Can you tell me what platform/filesystem you use for testing?
Linux/ext3 mainly. I also use Solaris/ufs for my own mails. Once in a
while I also try OS X/HFS+.
Oh and can you hint at what "bad things" may happen when I get the
error "record points outside file"?
When you get that error it's not that bad. Dovecot just deletes the
cache file which can cause small slowdowns. More problematic is when
there's a corruption that Dovecot doesn't detect and sends garbage to
client.
participants (3)
-
Jonathan Siegle
-
jsiegle＠psu.edu
-
Timo Sirainen