[Dovecot] Vanishing maildirsize files with 1.0.7 using 'deliver' via postfix
Aloha,
I've searched the list and have found no similar reports, so I'm asking here.
We have 3 RH5.1 boxes running the stock Postfix 2.3.3, but due to issues with the stock Dovecot 1.0rc15, Redhat supplied us with the version of Dovecot 1.0.7 that will be in Redhat 5.2.
The mail store is automounted per-user from a Sun box via NFS.
We were using it with no problems until last Friday, when we attempted to implement quotas. I had done quite a lot of testing on a single system, so I didn't anticipate any issues here. I created maildirsize files for each user in the NFS-mounted /var/spool/mail/$USER and everything seemed fine. I changed mailbox_command to /usr/libexec/dovecot/deliver in postfix main.cf. Mail was delivered, quota files updated, all good.
However almost immediately a maildirsize folder vanished. Soon after another, then another, and now we're missing nearly 40 of them.
Any ideas as to why they're vanishing? Something to do with NFS? I've tried to trigger it by massively sending to the same user on all three systems but have yet to reproduce it at will.
Any clues would be appreciated. For reasons of support I cannot upgrade to a later dovecot at this time.
Best Regards,
- jason
Jason Forester wrote:
We have 3 RH5.1 boxes running the stock Postfix 2.3.3, but due to issues with the stock Dovecot 1.0rc15, Redhat supplied us with the version of Dovecot 1.0.7 that will be in Redhat 5.2.
The mail store is automounted per-user from a Sun box via NFS.
Timo strongly recommends to use 1.1 if you are using NFS...
1.1 is available via atrpms.net
Charles Marcus wrote:
Jason Forester wrote: We have 3 RH5.1 boxes running the stock Postfix 2.3.3, but due to issues with the stock Dovecot 1.0rc15, Redhat supplied us with the version of Dovecot 1.0.7 that will be in Redhat 5.2.
The mail store is automounted per-user from a Sun box via NFS.
Timo strongly recommends to use 1.1 if you are using NFS...
1.1 is available via atrpms.net
I do understand, but as I stated fairly clearly I thought "For reasons of support I cannot upgrade to a later dovecot at this time."
Of course I have a ticket open with RedHat, but I thought I might come to the source, find out if it's some form of a known bug, and then perhaps I *could* use that as leverage to get another version installed. So the question still stands:
Is this a known bug? Is maildirsize not NFS safe?
Knowing that would be a huge first step for me, so thanks in advance for any assistance.
Best Regards,
-jason
On Wed, May 21, 2008 at 12:21 PM, Jason Forester hilobird@gmail.com wrote:
Aloha,
I've searched the list and have found no similar reports, so I'm asking here.
We have 3 RH5.1 boxes running the stock Postfix 2.3.3, but due to issues with the stock Dovecot 1.0rc15, Redhat supplied us with the version of Dovecot 1.0.7 that will be in Redhat 5.2.
The mail store is automounted per-user from a Sun box via NFS.
We were using it with no problems until last Friday, when we attempted to implement quotas. I had done quite a lot of testing on a single system, so I didn't anticipate any issues here. I created maildirsize files for each user in the NFS-mounted /var/spool/mail/$USER and everything seemed fine. I changed mailbox_command to /usr/libexec/dovecot/deliver in postfix main.cf. Mail was delivered, quota files updated, all good.
However almost immediately a maildirsize folder vanished. Soon after another, then another, and now we're missing nearly 40 of them.
Any ideas as to why they're vanishing? Something to do with NFS? I've tried to trigger it by massively sending to the same user on all three systems but have yet to reproduce it at will.
Any clues would be appreciated. For reasons of support I cannot upgrade to a later dovecot at this time.
Best Regards,
- jason
On Wed, 2008-05-21 at 12:21 -1000, Jason Forester wrote:
We have 3 RH5.1 boxes running the stock Postfix 2.3.3, but due to issues with the stock Dovecot 1.0rc15, Redhat supplied us with the version of Dovecot 1.0.7 that will be in Redhat 5.2.
There is one change to Maildir++ quota since v1.0.7: http://hg.dovecot.org/dovecot-1.0/rev/288fd70ac258
But that's mostly an optimization and doesn't cause files to be deleted.
However almost immediately a maildirsize folder vanished. Soon after another, then another, and now we're missing nearly 40 of them.
Maildir++ quota recalculation works like:
Go through all files and sum up their sizes. Keep track of the highest directory mtime. Once everything is calculated, again go through all directories and find the highest mtime. If the highest mtime had increased, something had changed, so delete maildirsize file.
So could it be simply that the recalculation is slow enough that the maildir changes during it and causes the file to be deleted? Another possibility is that some error happens, but then Dovecot should have logged something.
Do you keep your quota limits stored only in the maildirsize file? If not, having the files deleted shouldn't affect functionality, but of course the performance is worse.
The rebuild performance could be improved if maildir filenames contain ",S=xxx" containing the file's size so stat() calls can be avoided. But you can't really change existing filenames without having them show up as new mails.
Hi Timo, thanks for replying. My comments are inline:
On Sun, May 25, 2008 at 2:25 AM, Timo Sirainen tss@iki.fi wrote:
<snip>
However almost immediately a maildirsize folder vanished. Soon after another, then another, and now we're missing nearly 40 of them.
Maildir++ quota recalculation works like:
Go through all files and sum up their sizes. Keep track of the highest directory mtime. Once everything is calculated, again go through all directories and find the highest mtime. If the highest mtime had increased, something had changed, so delete maildirsize file.
So could it be simply that the recalculation is slow enough that the maildir changes during it and causes the file to be deleted? Another possibility is that some error happens, but then Dovecot should have logged something.
I'm not seeing anything in my logs. It's certainly possible that multiple mails are delivered during a recalculation. I'm not quite sure I understand the mtime explanation above, are you saying that if dovecot thinks that the maildirsize file is inaccurate it deletes it and goes on?
Do you keep your quota limits stored only in the maildirsize file? If not, having the files deleted shouldn't affect functionality, but of course the performance is worse.
Basically yes, we have many users with different quotas and as such we cannot use a single quota in the config file.
So, it sounds like I'm the only person who has encountered this problem? If it's really just a consequence of the maildir changing, wouldn't it be more common?
Thanks again Timo,
- jason
On May 25, 2008, at 10:09 PM, Jason Forester wrote:
Maildir++ quota recalculation works like:
Go through all files and sum up their sizes. Keep track of the
highest directory mtime. Once everything is calculated, again go through all directories and find the highest mtime. If the highest mtime had increased, something had changed, so delete maildirsize file.So could it be simply that the recalculation is slow enough that the maildir changes during it and causes the file to be deleted? Another possibility is that some error happens, but then Dovecot should have logged something.
I'm not seeing anything in my logs. It's certainly possible that multiple mails are delivered during a recalculation. I'm not quite sure I understand the mtime explanation above, are you saying that if dovecot thinks that the maildirsize file is inaccurate it deletes it and goes on?
Pretty much, yes. If any mailbox's new/ or cur/ directory changes in
some way (flag changes also cause this..) during recalculation, the
result is just discarded.
Do you keep your quota limits stored only in the maildirsize file? If not, having the files deleted shouldn't affect functionality, but of course the performance is worse.
Basically yes, we have many users with different quotas and as such we cannot use a single quota in the config file.
That's a bug in v1.0 actually, it shouldn't be deleting the file if it
can't be regenerated. v1.1 doesn't delete it and just hopes it gets
eventually recalculated and fixed.
So, it sounds like I'm the only person who has encountered this problem? If it's really just a consequence of the maildir changing, wouldn't it be more common?
I think there are two issues:
Your recalculation is probably quite slow, assuming you don't have
the S=sizes in filenames. deliver adds these to new mails, but it
doesn't help for old ones.Most people specify the limits in Dovecot's userdb, so this isn't
an issue.
What userdb do you use?
On Sun, May 25, 2008 at 9:18 AM, Timo Sirainen tss@iki.fi wrote:
<snip>
I think there are two issues:
- Your recalculation is probably quite slow, assuming you don't have the S=sizes in filenames. deliver adds these to new mails, but it doesn't help for old ones.
We just changed to Deliver, so that's probably true.
- Most people specify the limits in Dovecot's userdb, so this isn't an issue.
What userdb do you use?
We are using ldap for auth/user. I will take a look at specifying it there so that the quotas don't vanish, but won't reporting still fail? The main reason the customer wants IMAP quotas instead of filesystem quotas is for reporting quota utilization via mail client and webmail. If I can't get stable reporting I'll probably just go back to fs quotas and be done with it.
Thanks,
- jason
On May 26, 2008, at 4:30 AM, Jason Forester wrote:
- Most people specify the limits in Dovecot's userdb, so this
isn't an issue.What userdb do you use?
We are using ldap for auth/user. I will take a look at specifying it there so that the quotas don't vanish, but won't reporting still fail?
It doesn't fail, it still reports the calculated quota size, but it
just doesn't remember it for future calculations.
The main reason the customer wants IMAP quotas instead of filesystem quotas is for reporting quota utilization via mail client and webmail. If I can't get stable reporting I'll probably just go back to fs quotas and be done with it.
Filesystem quota can be reported just as well. Although for NFS RPC
quota you'd have to use a separate plugin with v1.0: http://dovecot.org/patches/1.0/quota-rquotad.c
On Sun, May 25, 2008 at 3:30 PM, Jason Forester hilobird@gmail.com wrote:
On Sun, May 25, 2008 at 9:18 AM, Timo Sirainen tss@iki.fi wrote:
<snip>
I think there are two issues:
- Your recalculation is probably quite slow, assuming you don't have the S=sizes in filenames. deliver adds these to new mails, but it doesn't help for old ones.
We just changed to Deliver, so that's probably true.
- Most people specify the limits in Dovecot's userdb, so this isn't an issue.
What userdb do you use?
We are using ldap for auth/user. I will take a look at specifying it there so that the quotas don't vanish
Just wanted to follow up and say that specifying the quotas in the userdb has resolved the issue; maildirsize files are created if they're missing and quotas appear to be handled as designed.
So thanks very much for the solution Timo. This is my last week at this job and that was my last major ticket. Much appreciated!
- jason
participants (3)
-
Charles Marcus
-
Jason Forester
-
Timo Sirainen