[Dovecot] Users with large (4GB) inboxes crippling dovecot
Hi all,
I'm new here and would very much appreciate any help you can give me.
We are running a rather outdated mail server that until recently has been running beautifully. Under the pretense of "it is ain't broke, don't fix it" it hasn't been updated so is running Fedora Core 4 and dovecot v0.99.14.
What is happening is that as users log in (via thunderbird), they are shown the contents of their mailboxes. However, when they try and access a given email, they get no response even if they wait upwards of 10 minutes. I believe that the problem has arisen from their huge inboxes, several users have inboxes in excess of 4GB.
Do any of you know how I could split these large mbox files down in to manageable chunks of 1GB?
Please be aware that I only have remote access to the server over a modest internet connection.
Thanks again for any suggestions.
Rob
View this message in context: http://www.nabble.com/Users-with-large-%284GB%29-inboxes-crippling-dovecot-t... Sent from the Dovecot mailing list archive at Nabble.com.
On Thu, 28 May 2009 02:07:01 -0700 (PDT) robforrest wrote:
Hi all,
I'm new here and would very much appreciate any help you can give me.
We are running a rather outdated mail server that until recently has been running beautifully. Under the pretense of "it is ain't broke, don't fix it" it hasn't been updated so is running Fedora Core 4 and dovecot v0.99.14.
What is happening is that as users log in (via thunderbird), they are shown the contents of their mailboxes. However, when they try and access a given email, they get no response even if they wait upwards of 10 minutes. I believe that the problem has arisen from their huge inboxes, several users have inboxes in excess of 4GB.
Do any of you know how I could split these large mbox files down in to manageable chunks of 1GB?
I don't know. I know that maildir is far better for this situation. You should - Upgrade dovecot - Convert to maildir format
For the conversion to maildir refer to the discussion on this list.
--Frank Elsner
on 5-28-2009 2:07 AM robforrest spake the following:
Hi all,
I'm new here and would very much appreciate any help you can give me.
We are running a rather outdated mail server that until recently has been running beautifully. Under the pretense of "it is ain't broke, don't fix it" it hasn't been updated so is running Fedora Core 4 and dovecot v0.99.14.
It's broke, you need to fix it. This system is an accident waiting to happen (or a bot waiting to be owned).
What is happening is that as users log in (via thunderbird), they are shown the contents of their mailboxes. However, when they try and access a given email, they get no response even if they wait upwards of 10 minutes. I believe that the problem has arisen from their huge inboxes, several users have inboxes in excess of 4GB.
Do any of you know how I could split these large mbox files down in to manageable chunks of 1GB?
Please be aware that I only have remote access to the server over a modest internet connection.
Thanks again for any suggestions.
Rob Why not try a current version of dovecot. 1.0 would help a lot, and 1.1 would be even better. 0.99.14 has been improved so much, you are just not going to fix it easily.
You can get more current rpms of dovecot from atrpms.net, but you might have to either try a RHEL4 version (loosely based on Core 3) or rebuild a src rpm on a core 4 machine. That is one of the problems with running core on a server. If you let it get too far out of service, you have more trouble updating it.
On Thu, 2009-05-28 at 02:07 -0700, robforrest wrote:
What is happening is that as users log in (via thunderbird), they are shown the contents of their mailboxes. However, when they try and access a given email, they get no response even if they wait upwards of 10 minutes. I believe that the problem has arisen from their huge inboxes, several users have inboxes in excess of 4GB.
mbox_very_dirty_syncs = yes would probably make these mailboxes usable without having to split anything or convert to Maildir. That requires Dovecot v1.0+ of course.
on 5-28-2009 9:36 AM Timo Sirainen spake the following:
On Thu, 2009-05-28 at 02:07 -0700, robforrest wrote:
What is happening is that as users log in (via thunderbird), they are shown the contents of their mailboxes. However, when they try and access a given email, they get no response even if they wait upwards of 10 minutes. I believe that the problem has arisen from their huge inboxes, several users have inboxes in excess of 4GB.
mbox_very_dirty_syncs = yes would probably make these mailboxes usable without having to split anything or convert to Maildir. That requires Dovecot v1.0+ of course.
The indexing improvements in 1.0 would probably help a lot also. I have several large mboxes and also some gzipped log storage boxes and 1.1 is fairly speedy with them. A full re-index in thunderbird takes right at a minute with a 6gb gzipped box I just tested.
On May 28, 2009, at 10:39 AM, Scott Silva wrote:
on 5-28-2009 9:36 AM Timo Sirainen spake the following:
On Thu, 2009-05-28 at 02:07 -0700, robforrest wrote:
What is happening is that as users log in (via thunderbird), they
are shown the contents of their mailboxes. However, when they try and
access a given email, they get no response even if they wait upwards of 10
minutes. I believe that the problem has arisen from their huge inboxes,
several users have inboxes in excess of 4GB.mbox_very_dirty_syncs = yes would probably make these mailboxes
usable without having to split anything or convert to Maildir. That requires Dovecot v1.0+ of course.The indexing improvements in 1.0 would probably help a lot also. I
have several large mboxes and also some gzipped log storage boxes and 1.1
is fairly speedy with them. A full re-index in thunderbird takes right at a
minute with a 6gb gzipped box I just tested.
Can you state what the size of your "several large mboxes" is? I have
been curious about this. My current email server forces me to manage
my IMAP boxes not based on size, though I believe there is a less than
2GB limit, but on message count. Too many messages kills me, the
storage is one file for each mailbox.
As I am migrating to Dovecot, it would be nice to know what I am in
for, and if I can simply stop managing this? My users are going to
use the server like it is gmail, and never file a darn thing.
One thing I wanted to do, was look at the inbox, and if it is over
xMB's, rename it to inbox.date which will force the user to deal with
it. However, if there is no need to do this, and Dovecot can handle
it, then I would prefer to skip the intrusion on my users.
What happens in Dovecot if one inbox is massive and in fact hurting
performance. Does that performance hit trickle down to all users, or
just that one user?
Thank you, and looking forward to being 100% Doevcot here in a few
weeks.
Scott * If you contact me off list replace talklists@ with scott@ *
on 5-28-2009 12:24 PM Scott Haneda spake the following:
On May 28, 2009, at 10:39 AM, Scott Silva wrote:
on 5-28-2009 9:36 AM Timo Sirainen spake the following:
On Thu, 2009-05-28 at 02:07 -0700, robforrest wrote:
What is happening is that as users log in (via thunderbird), they are shown the contents of their mailboxes. However, when they try and access a given email, they get no response even if they wait upwards of 10 minutes. I believe that the problem has arisen from their huge inboxes, several users have inboxes in excess of 4GB.
mbox_very_dirty_syncs = yes would probably make these mailboxes usable without having to split anything or convert to Maildir. That requires Dovecot v1.0+ of course.
The indexing improvements in 1.0 would probably help a lot also. I have several large mboxes and also some gzipped log storage boxes and 1.1 is fairly speedy with them. A full re-index in thunderbird takes right at a minute with a 6gb gzipped box I just tested.
Can you state what the size of your "several large mboxes" is? I have been curious about this. My current email server forces me to manage my IMAP boxes not based on size, though I believe there is a less than 2GB limit, but on message count. Too many messages kills me, the storage is one file for each mailbox.
As I am migrating to Dovecot, it would be nice to know what I am in for, and if I can simply stop managing this? My users are going to use the server like it is gmail, and never file a darn thing.
One thing I wanted to do, was look at the inbox, and if it is over xMB's, rename it to inbox.date which will force the user to deal with it. However, if there is no need to do this, and Dovecot can handle it, then I would prefer to skip the intrusion on my users.
What happens in Dovecot if one inbox is massive and in fact hurting performance. Does that performance hit trickle down to all users, or just that one user?
Thank you, and looking forward to being 100% Doevcot here in a few weeks. My largest is about 5+ GB gzipped (I thought it was 6gb until I just checked), but I am on a 64 bit server and don't have a 2 gb filesize limit. I have several users with 2 to 3 GB inboxes on IMAP, and don't get any complaints. I have been considering going to maildir, and scripting some stuff like moving messages older than 30 days out of the inbox, and also purging any messages marked as deleted, and also older stuff in the trash. Maildir is much easier with this since it is one file per message instead of one file per folder.
If one user does manage to really hose a box, the re-indexing that dovecot can slow the system for a little while, but it clears up pretty fast. Only rarely have I had to intervene and kill a process.
The one place where dovecot might be less than stellar is with pop3 access from outlook if they leave mail on the server. It can get confused sometimes and re-download everything, but this is Outlook's problem, not dovecot's.
On May 28, 2009, at 1:12 PM, Scott Silva wrote:
Can you state what the size of your "several large mboxes" is? I
have been curious about this. My current email server forces me to
manage my IMAP boxes not based on size, though I believe there is a less than
2GB limit, but on message count. Too many messages kills me, the
storage is one file for each mailbox.As I am migrating to Dovecot, it would be nice to know what I am in
for, and if I can simply stop managing this? My users are going to use
the server like it is gmail, and never file a darn thing.One thing I wanted to do, was look at the inbox, and if it is over xMB's, rename it to inbox.date which will force the user to deal with it. However, if there is no need to do this, and Dovecot can
handle it, then I would prefer to skip the intrusion on my users.What happens in Dovecot if one inbox is massive and in fact hurting performance. Does that performance hit trickle down to all users, or just that one user?
Thank you, and looking forward to being 100% Doevcot here in a few
weeks.
Thanks for the reply..
My largest is about 5+ GB gzipped (I thought it was 6gb until I just
checked), but I am on a 64 bit server and don't have a 2 gb filesize limit. I
have several users with 2 to 3 GB inboxes on IMAP, and don't get any
complaints. I
Are you saying that you can gzip a mailbox, and Dovecot will then
somehow uncompress that and send the data back out to the Desktop
Client as a normal looking IMAP box?
I have never heard of this tactic, and I am not finding any references
to it on google, though I am probably just not understanding it and
looking at the wrong search terms.
have been considering going to maildir, and scripting some stuff
like moving messages older than 30 days out of the inbox, and also purging any
messages marked as deleted, and also older stuff in the trash. Maildir is
much easier with this since it is one file per message instead of one file per
folder.If one user does manage to really hose a box, the re-indexing that
dovecot can slow the system for a little while, but it clears up pretty fast.
Only rarely have I had to intervene and kill a process.The one place where dovecot might be less than stellar is with pop3
access from outlook if they leave mail on the server. It can get confused
sometimes and re-download everything, but this is Outlook's problem, not
dovecot's.
I used to keep an eye on these clients, and set a "never leave mail on
server" setting on my old server. Not sure how to do this in Dovecot,
though if I have my way, everyone is going to be on IMAP, though they
are free to use it like POP if they want.
Thanks again for your reply, impressive stats to me, considering the
stats of where I am coming from.
Scott * If you contact me off list replace talklists@ with scott@ *
On Thu, 2009-05-28 at 13:35 -0700, Scott Haneda wrote:
My largest is about 5+ GB gzipped (I thought it was 6gb until I just
checked), but I am on a 64 bit server and don't have a 2 gb filesize limit. I
have several users with 2 to 3 GB inboxes on IMAP, and don't get any
complaints. IAre you saying that you can gzip a mailbox, and Dovecot will then
somehow uncompress that and send the data back out to the Desktop
Client as a normal looking IMAP box?I have never heard of this tactic, and I am not finding any references
to it on google, though I am probably just not understanding it and
looking at the wrong search terms.
On Thu, May 28, 2009 at 3:47 PM, Timo Sirainen tss@iki.fi wrote:
On Thu, 2009-05-28 at 13:35 -0700, Scott Haneda wrote:
My largest is about 5+ GB gzipped (I thought it was 6gb until I just checked), but I am on a 64 bit server and don't have a 2 gb filesize limit. I have several users with 2 to 3 GB inboxes on IMAP, and don't get any complaints. I
Are you saying that you can gzip a mailbox, and Dovecot will then somehow uncompress that and send the data back out to the Desktop Client as a normal looking IMAP box?
I have never heard of this tactic, and I am not finding any references to it on google, though I am probably just not understanding it and looking at the wrong search terms.
What advantage is compression rather then disk space on server?
Matt
On Thu, 2009-05-28 at 16:00 -0500, Matt wrote:
What advantage is compression rather then disk space on server?
For mbox, there's no other advantage than less disk space usage. Instead there are disadvantages, because the mbox is then read-only.
For Maildir, it might also make disk I/O usage slightly smaller but probably not much unless you have large mails.
<snip> > > > I used to keep an eye on these clients, and set a "never leave mail on > server" setting on my old server. Not sure how to do this in Dovecot, > though if I have my way, everyone is going to be on IMAP, though they > are free to use it like POP if they want. > > Thanks again for your reply, impressive stats to me, considering the > stats of where I am coming from.
With many kudos to Timo, dovecot gets better with each release. When I went from UWImap to 0.99 the improvements were great. From 0.99 to 1.0, while not as dramatic, was even better. 1.0 to 1.1 even better. The improvements aren't as big from release to release, but they are noticeable. I would never go back to UWImap for any reason.
Hi Scott,
Are you saying that you can gzip a mailbox, and Dovecot will then somehow uncompress that and send the data back out to the Desktop Client as a normal looking IMAP box?
I have never heard of this tactic, and I am not finding any references to it on google, though I am probably just not understanding it and looking at the wrong search terms. I use this to save/backup older emails. zipped mbox = Fast, small, only one file to handle, not changeable = optimal for backup
Stefan
robforrest schrieb:
Hi all,
I'm new here and would very much appreciate any help you can give me.
We are running a rather outdated mail server that until recently has been running beautifully. Under the pretense of "it is ain't broke, don't fix it" it hasn't been updated so is running Fedora Core 4 and dovecot v0.99.14.
What is happening is that as users log in (via thunderbird), they are shown the contents of their mailboxes. However, when they try and access a given email, they get no response even if they wait upwards of 10 minutes. I believe that the problem has arisen from their huge inboxes, several users have inboxes in excess of 4GB.
Do any of you know how I could split these large mbox files down in to manageable chunks of 1GB?
Please be aware that I only have remote access to the server over a modest internet connection.
Thanks again for any suggestions.
Rob
Hi Rob, cant say something about 4 G Mailboxes TB and dovecot, but i can see that TB gets into troubles with courier maildir when going over 0,5 GB Mark and thousends of emails typical with mail lists so i speculate the problem goes more to TB then to dovecot Regards
robforrest wrote:
What is happening is that as users log in (via thunderbird), they are shown the contents of their mailboxes. However, when they try and access a given email, they get no response even if they wait upwards of 10 minutes. I believe that the problem has arisen from their huge inboxes, several users have inboxes in excess of 4GB.
FC4 is quite old... can the kernel and filesystem you're using properly handle files over 4GB?
This is certainly one advantage dbox and maildir have -- not being limited to the FS file size limit per folder.
But what sort of user puts 4GB of mail into a _single_ folder??
-- Curtis Maloney cmaloney@cardgate.net
At 07:46 PM 5/28/2009, Curtis Maloney wrote: FC4 is quite old... can the kernel and filesystem you're using properly handle files over 4GB?
This is certainly one advantage dbox and maildir have -- not being limited to the FS file size limit per folder.
But what sort of user puts 4GB of mail into a _single_ folder??
One that's been using the mail server since FC4? :-)
Rick
On Friday, May 29 at 09:46 AM, quoth Curtis Maloney:
This is certainly one advantage dbox and maildir have -- not being limited to the FS file size limit per folder.
That's not *entirely* accurate. Certainly no single message can exceed the 2GB limit even with maildir, and the other issue that begins to come up is the impact/effect of large numbers of files. Depending on the filesystem (I'm assuming ext2?), there's probably a hard limit on the number of files per directory, and almost certainly there's a big performance penalty for that many files. To get good performance with Maildir and really large folders, you need a filesystem that can handle large numbersof files. Ext3 has directory hashing, ReiserFS is good... I believe XFS and several others have tackled the problem as well (I don't know about FFS).
That said... eGADS - a real life FC4 in the wild?!?! According to fedoraproject.org:
For 20030101-20050607 there are a potential 863 CVE named
vulnerabilities that could have affected FC4 packages. 759 (88%)
of those are fixed because FC4 includes an upstream version that
includes a fix, 10 (1%) are still outstanding, and 94 (11%) are
fixed with a backported patch.That would make me a little nervous.... that's just the issues over the course of two years, ending in 2005 (FOUR years ago).
~Kyle
I would therefore like to posit that computingâs central challenge, how not to make a mess of it, has not yet been met. -- Edsger Dijkstra
On May 28, 2009, at 10:28 PM, Kyle Wheeler wrote:
On Friday, May 29 at 09:46 AM, quoth Curtis Maloney:
This is certainly one advantage dbox and maildir have -- not being
limited to the FS file size limit per folder.That's not *entirely* accurate. Certainly no single message can
exceed the 2GB limit even with maildir,
Dovecot is built with large file support, so it can handle >4GB mails
and >4GB mboxes just fine.
On Thu, 2009-05-28 at 21:28 -0500, Kyle Wheeler wrote:
On Friday, May 29 at 09:46 AM, quoth Curtis Maloney:
This is certainly one advantage dbox and maildir have -- not being limited to the FS file size limit per folder.
That's not *entirely* accurate. Certainly no single message can exceed the 2GB limit even with maildir, and the other issue that begins to come up is the impact/effect of large numbers of files. Depending on the filesystem (I'm assuming ext2?), there's probably a hard limit on
FC4 had ext3 (unless my memory is totally mistaken).
the number of files per directory, and almost certainly there's a big
Subdirectories, yes (because the link count in the inode is of quite finite size). But there never was TTBOMK a limit on the number of files (!= Directories) in ext2 (except the trivial one: The directory is as large as the largest file. But that applies probably to all filesystems - though more recent ones allow for much larger files). Let alone ext3.
performance penalty for that many files. To get good performance with Maildir and really large folders, you need a filesystem that can handle large numbersof files. Ext3 has directory hashing, ReiserFS is
Make sure you have the "dir_index" option set on that filesystem (which
is probably set per default anyways these days. Otherwise you can change
it on the fly with tune2fs).
The back of my head suggests that one has to recreate the directory
after changing that option (read: mkdir new; mv old/* new; rmdir old; mv new old. Solving the "command line too long" problem is left to the
reader;-).
good... I believe XFS and several others have tackled the problem as well (I don't know about FFS).
That said... eGADS - a real life FC4 in the wild?!?! According to fedoraproject.org:
For 20030101-20050607 there are a potential 863 CVE named vulnerabilities that could have affected FC4 packages. 759 (88%) of those are fixed because FC4 includes an upstream version that includes a fix, 10 (1%) are still outstanding, and 94 (11%) are fixed with a backported patch.That would make me a little nervous.... that's just the issues over the course of two years, ending in 2005 (FOUR years ago).
I don't know what/how others do but many servers are not really "in the wild" but behind more recent firewalls and/or loadbalancers and/or similar equipment (like running database servers behind webservers).
And "running FC4" doesn't mean that that certain/some/several/many packages aren't replaced by more recent ones - for whatever reason (security, performance, newer version, newer drivers in the kernel, ...).
For a "pristine FC4 with lots of services directly at the Internet": I totally agree with you.
Apart from the basic question if one shouldn't better run a more conservative distribution (like RHEL) in the first place where the support cycle is much longer.
Bernd-- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services
on 5-29-2009 1:42 AM Bernd Petrovitsch spake the following:
On Thu, 2009-05-28 at 21:28 -0500, Kyle Wheeler wrote:
This is certainly one advantage dbox and maildir have -- not being limited to the FS file size limit per folder. That's not *entirely* accurate. Certainly no single message can exceed
On Friday, May 29 at 09:46 AM, quoth Curtis Maloney: the 2GB limit even with maildir, and the other issue that begins to come up is the impact/effect of large numbers of files. Depending on the filesystem (I'm assuming ext2?), there's probably a hard limit on
FC4 had ext3 (unless my memory is totally mistaken).
the number of files per directory, and almost certainly there's a big
Subdirectories, yes (because the link count in the inode is of quite finite size). But there never was TTBOMK a limit on the number of files (!= Directories) in ext2 (except the trivial one: The directory is as large as the largest file. But that applies probably to all filesystems - though more recent ones allow for much larger files). Let alone ext3.
performance penalty for that many files. To get good performance with Maildir and really large folders, you need a filesystem that can handle large numbersof files. Ext3 has directory hashing, ReiserFS is
Make sure you have the "dir_index" option set on that filesystem (which is probably set per default anyways these days. Otherwise you can change it on the fly with
tune2fs).
I don't think you can do it on the fly. Any directories created before the option was set will not have directory indexing until a fsck is done.
<quote>
Using Directory Indexing
This feature improves file access in large directories or directories containing many files by using hashed binary trees to store the directory information. It's perfectly safe to use, and it provides a fairly substantial improvement in most cases; so it's a good idea to enable it:
# tune2fs -O dir_index /dev/hdXY
This will only take effect with directories created on that filesystem after tune2fs is run. In order to apply this to currently existing directories, we must run the e2fsck utility to optimize and reindex the directories on the filesystem:
# e2fsck -D -f /dev/hdXY
Note: This should work with both ext2 and ext3 filesystems. Depending on the size of your filesystem, this could take a long time. Perhaps you should go get some coffee...
</quote>
The back of my head suggests that one has to recreate the directory after changing that option (read:
mkdir new; mv old/* new; rmdir old; mv new old. Solving the "command line too long" problem is left to the reader;-).
On 30. mai. 2009, at 00.03, Scott Silva wrote:
# e2fsck -D -f /dev/hdXY
Note: This should work with both ext2 and ext3 filesystems.
Depending on the size of your filesystem, this could take a long time. Perhaps you
should go get some coffee...
AFAIK an "updatedb" (as in locate/slocate), or perhaps just a "find /"
should access all directories and make and index in each of them. That
way you won't need to umount the fs.
roy
Roy Sigurd Karlsbakk (+47) 97542685 / 98013356 roy@karlsbakk.net http://blogg.karlsbakk.net/
I all pedagogikk er det essensielt at pensum presenteres
intelligibelt. Det er et elementært imperativ for alle pedagoger å
unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de
fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Sat, May 30, 2009 at 11:25:58PM +0200, Roy Sigurd Karlsbakk wrote:
On 30. mai. 2009, at 00.03, Scott Silva wrote: [...] AFAIK an "updatedb" (as in locate/slocate) [...]
This won't help with the original poster's problem (which is in helping the kernel to cope with huge directories). The OP's problem is at the kernel level, locate is an application (albeit a nice one).
Regards
- -- tomás -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFKIgD1Bcgs9XrR2kYRAutsAJ46nRL6gV1qCejXvL2jORYK0jGcWgCfeTsk iox6dAFCXBRbuUpuMQQL4r8= =EsWv -----END PGP SIGNATURE-----
On 31. mai. 2009, at 06.00, tomas@tuxteam.de wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Sat, May 30, 2009 at 11:25:58PM +0200, Roy Sigurd Karlsbakk wrote:
On 30. mai. 2009, at 00.03, Scott Silva wrote: [...] AFAIK an "updatedb" (as in locate/slocate) [...]
This won't help with the original poster's problem (which is in
helping the kernel to cope with huge directories). The OP's problem is at the kernel level, locate is an application (albeit a nice one).
My reply was about how to update the directory index after adding it
to a ext3 filesystem.
The original poster's problem was about large _files_, not
directories. ext3 can handle large directories without problems
(except that the maximum number of subdirectories in one directory is
fixed to 32000). Also, files opened with O_LARGEFILE, can be up to 2TB
on ext3.
roy
Roy Sigurd Karlsbakk (+47) 97542685 / 98013356 roy@karlsbakk.net http://blogg.karlsbakk.net/
I all pedagogikk er det essensielt at pensum presenteres
intelligibelt. Det er et elementært imperativ for alle pedagoger å
unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de
fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
on 5-30-2009 2:25 PM Roy Sigurd Karlsbakk spake the following:
On 30. mai. 2009, at 00.03, Scott Silva wrote:
# e2fsck -D -f /dev/hdXY
Note: This should work with both ext2 and ext3 filesystems. Depending on the size of your filesystem, this could take a long time. Perhaps you should go get some coffee...
AFAIK an "updatedb" (as in locate/slocate), or perhaps just a "find /" should access all directories and make and index in each of them. That way you won't need to umount the fs.
If you look at the quoted text, an fsck is required on pre-existing directories. They don't have the proper extents or index structures if created before directory_index is turned on. No other program is going to update the directory indexes. You can read those directories as many times as you want, it won't change it.
Please find below details of a crash of deliver. The dovecot version is still at 1.2.rc3 as we use Stephan Bosch's sieve, for which there is no patch for rc4. Sieve is at 0.1.4 and managesieve at 0.11.3 if that matters.
The mail had 4 recipients at the local mail server, the first seems to have triggered the crash, the other three mails were delivered: May 31 02:07:25 apollo postfix/qmgr[6437]: AF24F3F81D7: from=root@triton.ifh.de, size=765407, nrcpt=4 (queue active)
This is the first and only crash I could find in the logs (approx 500k Lines of postfix/dovecot logs)
The following information could be extracted from the logfile (no core dump unfortunately)
May 31 02:07:25 apollo dovecot: deliver(user1): Panic: file istream.c: line 99 (i_stream_read): assertion failed: ((size_t)ret+old_size == _stream->pos - _stream->skip)
May 31 02:07:25 apollo dovecot: deliver(user1): Raw backtrace: /usr/libexec/dovecot/deliver [0x48e920] -> /usr/libexec/dovecot/deliver(i_syslog_fatal_handler+0x2b) [0x48e9bb] -> /usr/libexec/dovecot/deliver [0x48e006] -> /usr/libexec/dovecot/deliver [0x49305e] -> /usr/libexec/dovecot/deliver [0x4959f1] -> /usr/libexec/dovecot/deliver(i_stream_read+0x48) [0x492f28] -> /usr/libexec/dovecot/deliver(i_stream_read_data+0x28) [0x493088] -> /usr/libexec/dovecot/deliver [0x48764e] -> /usr/libexec/dovecot/deliver [0x487afe] -> /usr/libexec/dovecot/deliver(message_parser_parse_next_block+0x29) [0x487769] -> /usr/libexec/dovecot/deliver(index_mail_cache_parse_continue+0x1f) [0x446cff] -> /usr/libexec/dovecot/deliver(maildir_save_continue+0x48) [0x42c9c8] -> /usr/libexec/dovecot/deliver(mail_storage_copy+0x6a) [0x454c7a] -> /usr/libexec/dovecot/deliver(maildir_copy+0x5e) [0x4295fe] -> /usr/lib64/dovecot/lda/lib10_quota_plugin.so [0x2b5bf2754e4c] -> /usr/lib64/dovecot/lda/lib90_sieve_plugin.so [0x2b5bf2983b25] -> /usr/lib6
The output of dovecot -n:
# 1.2.rc3: /etc/dovecot.conf # OS: Linux 2.6.18-128.1.6.el5 x86_64 Scientific Linux SL release 5.3 (Boron) protocols: imap imaps managesieve listen(default): * listen(imap): * listen(managesieve): *:2009 disable_plaintext_auth: yes shutdown_clients: no login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(managesieve): /usr/libexec/dovecot/managesieve-login login_processes_count: 1 login_max_processes_count: 256 verbose_proctitle: yes mmap_disable: yes fsync_disable: yes lock_method: dotlock maildir_copy_with_hardlinks: no mbox_read_locks: dotlock flock mbox_write_locks: dotlock flock mail_drop_priv_before_exec: yes mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(managesieve): /usr/libexec/dovecot/managesieve mail_plugins(default): quota imap_quota zlib acl imap_acl mail_plugins(imap): quota imap_quota zlib acl imap_acl mail_plugins(managesieve): mail_plugin_dir(default): /usr/lib64/dovecot/imap mail_plugin_dir(imap): /usr/lib64/dovecot/imap mail_plugin_dir(managesieve): /usr/lib64/dovecot/managesieve imap_client_workarounds(default): delay-newmail netscape-eoh tb-extra-mailbox-sep outlook-idle imap_client_workarounds(imap): delay-newmail netscape-eoh tb-extra-mailbox-sep outlook-idle imap_client_workarounds(managesieve): namespace: type: private separator: / location: maildir:~/Maildir inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: #mbox/ location: mbox:~/mail hidden: yes list: yes subscriptions: yes auth default: mechanisms: gssapi gss-spnego login plain realms: IFH.DE DESY.DE passdb: driver: pam args: max_requests=1 userdb: driver: passwd-file args: /etc/userdb socket: type: listen client: path: /var/run/dovecot/auth-client mode: 432 master: path: /var/run/dovecot/auth-master mode: 438 plugin: quota: maildir:User Quota quota_rule: *:storage=1000MB quota_rule2: INBOX:storage=1000MB quota_rule3: Trash:ignore acl: vfile:/etc/dovecot-acls:cache_secs=300 sieve_storage: /var/spool/sieve/%u sieve: /var/spool/sieve/%u.sieve quota_warning: storage=95%% /usr/local/bin/quota-warning.sh 95 quota_warning2: storage=80%% /usr/local/bin/quota-warning.sh 80 quota_warning3: storage=60%% /usr/local/bin/quota-warning.sh 60
-- Wolfgang Friebel Deutsches Elektronen-Synchrotron DESY Phone/Fax: +49 33762 77372/216 Platanenallee 6 Mail: Wolfgang.Friebel AT desy.de D-15738 Zeuthen Germany
On Tue, 2009-06-02 at 17:07 +0200, Wolfgang.Friebel@desy.de wrote:
This is the first and only crash I could find in the logs (approx 500k Lines of postfix/dovecot logs)
The following information could be extracted from the logfile (no core dump unfortunately)
May 31 02:07:25 apollo dovecot: deliver(user1): Panic: file istream.c: line 99 (i_stream_read): assertion failed: ((size_t)ret+old_size == _stream->pos - _stream->skip)
I suppose this hasn't happened since? Core dump from this would be really interesting. I've also attached a patch that could at least show if my guess it right where it probably crashed..
On Mon, 15 Jun 2009, Timo Sirainen wrote:
On Tue, 2009-06-02 at 17:07 +0200, Wolfgang.Friebel@desy.de wrote:
This is the first and only crash I could find in the logs (approx 500k Lines of postfix/dovecot logs)
The following information could be extracted from the logfile (no core dump unfortunately)
May 31 02:07:25 apollo dovecot: deliver(user1): Panic: file istream.c: line 99 (i_stream_read): assertion failed: ((size_t)ret+old_size == _stream->pos - _stream->skip)
I suppose this hasn't happened since? Core dump from this would be really interesting. I've also attached a patch that could at least show if my guess it right where it probably crashed..
No, it did not happen again. I would get now a coredump if dovecot was crashing another time. In addition I watch the mail log file, there are no errors recorded (for about 100k messages received) since that crash.
See you in Berlin!
On Qui, 28 Mai 2009, Curtis Maloney wrote:
But what sort of user puts 4GB of mail into a _single_ folder??
Those that never delete the emails nor move them into folders.
Everything they ever received is in the INBOX.
And yes, this happens a lot.
-- Eduardo M KALINOWSKI eduardo@kalinowski.com.br
on 5-28-2009 4:46 PM Curtis Maloney spake the following:
robforrest wrote:
What is happening is that as users log in (via thunderbird), they are shown the contents of their mailboxes. However, when they try and access a given email, they get no response even if they wait upwards of 10 minutes. I believe that the problem has arisen from their huge inboxes, several users have inboxes in excess of 4GB.
FC4 is quite old... can the kernel and filesystem you're using properly handle files over 4GB?
This is certainly one advantage dbox and maildir have -- not being limited to the FS file size limit per folder.
But what sort of user puts 4GB of mail into a _single_ folder??
(l)users!
;-P
participants (16)
-
Bernd Petrovitsch
-
Curtis Maloney
-
dovecot＠corwyn.net
-
Eduardo M KALINOWSKI
-
Frank Elsner
-
Kyle Wheeler
-
Matt
-
Robert Schetterer
-
robforrest
-
Roy Sigurd Karlsbakk
-
Scott Haneda
-
Scott Silva
-
Stefan Klatt
-
Timo Sirainen
-
tomas＠tuxteam.de
-
Wolfgang.Friebel＠desy.de