[Dovecot] possible nfs issue
Hi all, we've started receiving complaints from users that seemingly use more quota than they actually have. We noticed that these users have (in some cases many) .nfs files in their mailspool. Some of our admins checked their own dirs, and noticed them there as well. This could of course be unrelated to dovecot (kernel issue, netapp issue) but maybe somehow has an idea about if dovecot could cause this. This has been going on for at least a year, not really enough to notice before now.
root@userimap1# find . -type f|grep -i .nfs ./cur/.nfs00000000003967ad003c0603 ./cur/.nfs000000000757b44b003be609 ./cur/.nfs00000000035e89bd003be60b ./cur/.nfs000000000796251c003be60c ./cur/.nfs000000000796251f003be60e ./cur/.nfs000000000262f9a1003be33a ./cur/.nfs00000000096513f3003be524 ./cur/.nfs0000000007962525003be60f ./cur/.nfs0000000003e7d8ab003be62b ./cur/.nfs00000000026f4fad003be50d ./cur/.nfs0000000000bdaeab003c0611 ./cur/.nfs0000000005da42c7003be525 ./cur/.nfs0000000003d74729003be526 ./cur/.nfs000000000229769e003be535 ./cur/.nfs000000000440969e003be516
With NFS these files are created when a file gets unlinked, but another process still has it open. It disappears as soon as the other process closes it. For some reason they dont disappear. As far as I can tell we've had no server crashes that could explain this. One possible theory is that a rename happens after an unlink. In that case the file remains. This could possibly be a dovecot issue.
Anyone else with NFS mailspools seeing this?
Cor
On 3.10.2012, at 0.39, Cor Bosman wrote:
With NFS these files are created when a file gets unlinked, but another process still has it open. It disappears as soon as the other process closes it. For some reason they dont disappear. As far as I can tell we've had no server crashes that could explain this. One possible theory is that a rename happens after an unlink. In that case the file remains. This could possibly be a dovecot issue.
How can a rename happen after unlink? The rename should fail. (Unless doing rename(.nfs1234, something), but Dovecot definitely isn't doing that.)
On 3.10.2012, at 0.45, Timo Sirainen wrote:
On 3.10.2012, at 0.39, Cor Bosman wrote:
With NFS these files are created when a file gets unlinked, but another process still has it open. It disappears as soon as the other process closes it. For some reason they dont disappear. As far as I can tell we've had no server crashes that could explain this. One possible theory is that a rename happens after an unlink. In that case the file remains. This could possibly be a dovecot issue.
How can a rename happen after unlink? The rename should fail. (Unless doing rename(.nfs1234, something), but Dovecot definitely isn't doing that.)
You could see if this old test program leaves .nfs files lying around:
http://dovecot.org/tmp/readdir.c
Just comment out the line:
close(fd);On Oct 3, 2012, at 12:35 AM, Timo Sirainen <tss@iki.fi> wrote:
On 3.10.2012, at 0.45, Timo Sirainen wrote:
On 3.10.2012, at 0.39, Cor Bosman wrote:
With NFS these files are created when a file gets unlinked, but another process still has it open. It disappears as soon as the other process closes it. For some reason they dont disappear. As far as I can tell we've had no server crashes that could explain this. One possible theory is that a rename happens after an unlink. In that case the file remains. This could possibly be a dovecot issue.
How can a rename happen after unlink? The rename should fail. (Unless doing rename(.nfs1234, something), but Dovecot definitely isn't doing that.)
You could see if this old test program leaves .nfs files lying around:
http://dovecot.org/tmp/readdir.c
Just comment out the line:
close(fd);
I meant the .nfs1234 indeed, but it seemed very unlikely. Thanks for clarifying. The readdir program leaves no .nfs files. We'll have to explore other possibilities.
Cor
Maybe it's a cross program issue?
We used to randomly have this happen a long time ago, when using
postfix and dovecot.
Since switching to using the dovecot lda/lmtp instead of postfix for
mailbox delievery, I haven't seen this happen at all anymore.
I'm not saying that postfix is at fault for this, but could be a
timing/race issue between postfix/dovecot accesses to the mailbox.
Quoting Cor Bosman <cor@xs4all.nl>:
On Oct 3, 2012, at 12:35 AM, Timo Sirainen <tss@iki.fi> wrote:
On 3.10.2012, at 0.45, Timo Sirainen wrote:
On 3.10.2012, at 0.39, Cor Bosman wrote:
With NFS these files are created when a file gets unlinked, but
another process still has it open. It disappears as soon as the
other process closes it. For some reason they dont disappear. As
far as I can tell we've had no server crashes that could explain
this. One possible theory is that a rename happens after an
unlink. In that case the file remains. This could possibly be a
dovecot issue.How can a rename happen after unlink? The rename should fail.
(Unless doing rename(.nfs1234, something), but Dovecot definitely
isn't doing that.)You could see if this old test program leaves .nfs files lying around:
http://dovecot.org/tmp/readdir.c
Just comment out the line:
close(fd);I meant the .nfs1234 indeed, but it seemed very unlikely. Thanks for
clarifying. The readdir program leaves no .nfs files. We'll have to
explore other possibilities.Cor
On 10/2/2012 4:39 PM, Cor Bosman wrote:
Anyone else with NFS mailspools seeing this?
Cor
I haven't seen them yet, however, to help troubleshoot, see this link and follow it's links for more details on .nfs files
http://wordpress.org/support/topic/how-can-i-prevent-unwanted-nfs-files-from...
Jack
On 10/2/2012 2:39 PM, Cor Bosman wrote:
Anyone else with NFS mailspools seeing this?
Yes, it is like 1999 all over again. I haven't had a chance to track them down or setup a cron job to rm them all. All of the ones I'm seeing are ex dovecot.index files but it looks like yours are ex messages?
I figured this was a probably a regression in the RHEL6.3 (Sl6.3) (2.6.32-279.9.1.el6.x86_64) kernel. What are you running Cor?
-- Kelsey Cummings - kgc@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
On Oct 4, 2012, at 3:55 AM, Kelsey Cummings <kgc@corp.sonic.net> wrote:
On 10/2/2012 2:39 PM, Cor Bosman wrote:
Anyone else with NFS mailspools seeing this?
Yes, it is like 1999 all over again. I haven't had a chance to track them down or setup a cron job to rm them all. All of the ones I'm seeing are ex dovecot.index files but it looks like yours are ex messages?
I figured this was a probably a regression in the RHEL6.3 (Sl6.3) (2.6.32-279.9.1.el6.x86_64) kernel. What are you running Cor?
We're running debian with a 3.2.2 kernel. Just yesterday one of my colleagues had a few new ones in his mailspool. Definitely no server crash or anything. Something is creating these outside the 'normal' parameters for .nfs files. My colleague said these were emails he deleted that day. We've set up a cleaning run, and are probably going to ignore it for now. These things are near impossible to track down without a lot of debugging.
Cor
participants (5)
-
Cor Bosman
-
Jack Bates
-
Kelsey Cummings
-
Patrick Domack
-
Timo Sirainen