[Dovecot] Multiple instances of .customflags staying open
# cat /proc/version Linux version 2.4.20-18.9 (bhcompile@daffy.perf.redhat.com) (gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)) #1 Thu May 29 07:09:43 EDT 2003 # dovecot --version 0.99.10
I've just recently noticed a serious problem with my system. After about
an hour or so I get errors in /var/log/maillog such as this;
Sep 21 06:40:16 the-system imap(ray): opendir() failed with file
/home/ray/Maildir//new: Too many open files
I investigated with 'lsof' and found that many copies of '.customflags' were open for each and every folder. ie; Each folder's '.customflags' file was open 15 times!
I suspect that postfix is not able to store newly collected email while this is occuring, therefore I am likely losing messages.
I've tried with multiple mail clients (Sylpheed & Evolution) and multiple users, always with the same result.
Further investigation indicates that each time the client scans for new mail another instance of each and every '.customflags' is opened. As I have the client set for checking at five minute intervals then it was probably 75 minutes until the client barfed. Less, if I manualy scan for new mail a few times.
If I shutdown the client and restart it then everything is fine for about an hour. Running 'lsof /home/ray/Maildir/.customflags' on the server I can see the list of files increasing each time the client checks for new mail.
Any help would be appreciated!
On Mon, 2003-09-22 at 16:13, Ron wrote:
I've just recently noticed a serious problem with my system. After about an hour or so I get errors in /var/log/maillog such as this; Sep 21 06:40:16 the-system imap(ray): opendir() failed with file
/home/ray/Maildir//new: Too many open filesI investigated with 'lsof' and found that many copies of '.customflags' were open for each and every folder. ie; Each folder's '.customflags' file was open 15 times!
Could it be that your .customflags files are not writable to users? That's the only code path that I can see it leaking file descriptors for .customflags. Attached patch should also fix it.
On Mon, 22 Sep 2003 17:28:48 +0300 Timo Sirainen (Timo Sirainen tss@iki.fi) said:
On Mon, 2003-09-22 at 16:13, Ron wrote:
I've just recently noticed a serious problem with my system. After about an hour or so I get errors in /var/log/maillog such as this; Sep 21 06:40:16 the-system imap(ray): opendir() failed with file
/home/ray/Maildir//new: Too many open filesI investigated with 'lsof' and found that many copies of '.customflags' were open for each and every folder. ie; Each folder's '.customflags' file was open 15 times!
Could it be that your .customflags files are not writable to users?
No, they're all writable to the user.
That's the only code path that I can see it leaking file descriptors for.customflags. Attached patch should also fix it.
This is likely a dumb question, but how do I apply the patch? I'm no programmer, just an admin/user.
btw, the situation got worse today. I couldn't keep the mail client running for more than a minute at a time. This is a real nuicence.
Thanks for the speedy reply!
On Tue, 2003-09-23 at 14:16, Ron wrote:
I investigated with 'lsof' and found that many copies of '.customflags' were open for each and every folder. ie; Each folder's '.customflags' file was open 15 times!
Try manually running:
MAIL=~Maildir /usr/local/libexec/dovecot/imap x select inbox
and check if that process also is leaking .customflags files?
And is it really the same .customflags file that is open 15 times?
Do you use INDEX=MEMORY?
Could it be that your .customflags files are not writable to users?
No, they're all writable to the user.
Then that patch probably doesn't help. Hmm. There's another way actually for leaks to happen but I think it also leaks .data files. Happens when opening indexes fails because it has to be rebuilt. Shouldn't happen normally so I'm not sure if that's your problem either. I'll fix that later today anyway.
That's the only code path that I can see it leaking file descriptors for.customflags. Attached patch should also fix it.
This is likely a dumb question, but how do I apply the patch? I'm no programmer, just an admin/user.
cd dovecot-0.99.10 patch -p0 < file.diff
(or -p1 sometimes)
On Tue, 23 Sep 2003 15:27:10 +0300 Timo Sirainen (Timo Sirainen tss@iki.fi) said:
Try manually running: MAIL=~Maildir /usr/local/libexec/dovecot/imap x select inbox and check if that process also is leaking .customflags files?
Two copies of the .customflags file get opened, and stay opened. I execute the commands (1 select INBOX, 2 select INBOX, etc) and only the two are opened;
# lsof|grep customflag imap 1703 ron mem REG 3,5 5 5488711 /home/ron/Maildir/.customflags 1703 ron 5u REG 3,5 5 5488711 /home/ron/Maildir/.customflags imap
And is it really the same .customflags file that is open 15 times?
Yes, absolutely.
Do you use INDEX=MEMORY?
Do you mean this?; default_mail_env = maildir:%h/Maildir/:INDEX=MEMORY
Thanks again!
On Tue, 23 Sep 2003 20:40:23 +0300 Timo Sirainen (Timo Sirainen tss@iki.fi) said:
On Tue, 2003-09-23 at 16:08, Ron wrote:
Do you use INDEX=MEMORY?
Do you mean this?; default_mail_env = maildir:%h/Maildir/:INDEX=MEMORY
Yes. OK, got it :) It happens with INDEX=MEMORY. Included patch should fix it.
Beaut, thanks. I'll apply it later today and report back.
One more piece of information I've discovered which may help though; I changed the mail checking interval from five minutes to thirty minutes and then left the mail client running overnight. It did _not_ die. There was only five occurences of any one .customflags file open, whereas there should have been around fifteen (and therefore it would have locked up).
Maybe the opened files are timing out after a period of maybe two or three hours? Therefore the problem had been there all along (on my system, anyway) until a week or so ago when I changed from ten minute intervals to five minute intervals. ie; I'm opening the files faster than they timeout and close.
What do you think? Or have I gone completely mad? ;-)
cheers!
On Wed, 24 Sep 2003 07:09:01 +1000 Ron (Ron dovecot@fabre.id.au) said:
On Tue, 23 Sep 2003 20:40:23 +0300 Timo Sirainen (Timo Sirainen tss@iki.fi) said:
Yes. OK, got it :) It happens with INDEX=MEMORY. Included patch should fix it.
Beaut, thanks. I'll apply it later today and report back.
ok, I applied the patch and things are much better. When I check for open customflags files I find that there are only a couple of unique ones, and only for a short time (probably while the client is scanning the folders).
However, a new bug has appeared. When I press send on a message I sometimes get this error;
The message was queued but could not be sent. Use "Send queued messages" from the main window to retry.
The message does not actualy appear in the Queue folder unless I quit the client and restart it (this bit could be a client issue). When I check maillog I see this entry;
Sep 24 19:03:53 the-system imap(ron): Corrupted index file
(in-memory index for /home/ron/Maildir//.Queue): UIDVALIDITY
changed in uidlist
Any (new) ideas?
cheers!
On Wed, 2003-09-24 at 12:16, Ron wrote:
The message does not actualy appear in the Queue folder unless I quit the client and restart it (this bit could be a client issue). When I check maillog I see this entry;
Sep 24 19:03:53 the-system imap(ron): Corrupted index file
(in-memory index for /home/ron/Maildir//.Queue): UIDVALIDITY
changed in uidlistAny (new) ideas?
Hmh. I guess INDEX=MEMORY doesn't work too well. I can't reproduce this myself though. Do you use multiple clients, or does your client use multiple IMAP connections?
Can you check if the UIDVALIDITY really changes in the file? It's the middle number in first line of dovecot-uidlist file.
On Thu, 25 Sep 2003 02:57:03 +0300 Timo Sirainen (Timo Sirainen tss@iki.fi) said:
Hmh. I guess INDEX=MEMORY doesn't work too well. I can't reproduce this myself though. Do you use multiple clients, or does your client use multiple IMAP connections?
Can you check if the UIDVALIDITY really changes in the file? It's the middle number in first line of dovecot-uidlist file.
Before creating the new message; 1 1061811585 283 282 1064450999.P20289Q9M323521.the-system
After sending, but failing; 1 1061811585 283 282 1064450999.P20289Q9M323521.the-system
After the message appeared in the queue, and was successfully sent; 1 1061811585 284 283 1064452695.P20289Q14M935913.the-system
btw, it doesn't require a restart of the client, if I wait for about 30 seconds then the message appears in the Queue folder ready for me to force the send again.
cheers!
Hi Ron!
On Thu, 25 Sep 2003, Ron wrote:
On Thu, 25 Sep 2003 02:57:03 +0300 Timo Sirainen (Timo Sirainen tss@iki.fi) said:
Hmh. I guess INDEX=MEMORY doesn't work too well. I can't reproduce this myself though. Do you use multiple clients, or does your client use multiple IMAP connections?
I wonder why INDEX=MEMORY is being used at all. IIRC, Ron said it was RH standard config.
Can you check if the UIDVALIDITY really changes in the file? It's the middle number in first line of dovecot-uidlist file.
Before creating the new message; 1 1061811585 283 282 1064450999.P20289Q9M323521.the-system
After sending, but failing; 1 1061811585 283 282 1064450999.P20289Q9M323521.the-system
I don't see how swnding mail has anything to do with dovecot. That's a transaction between your mail client and your SMTP daemon.
Timo, Ron has an strace of the imap process if you need any further confirmation of the location of the fd leak. But it looks like you've found it anyway.
-- Charlie
On 2003-09-25 05:51:33 -0400, Charlie Brady wrote:
I don't see how swnding mail has anything to do with dovecot. That's a transaction between your mail client and your SMTP daemon.
i think is "sentmail" folder is on the imap server.
darix
On Thu, 25 Sep 2003 13:03:50 +0200 Marcus Rueckert (Marcus Rueckert rueckert@informatik.uni-rostock.de) said:
On 2003-09-25 05:51:33 -0400, Charlie Brady wrote:
I don't see how swnding mail has anything to do with dovecot. That's a transaction between your mail client and your SMTP daemon.
i think is "sentmail" folder is on the imap server.
Yes, my Sent and Queue folders are on the IMAP server. This problem normaly appears when sending mail, and therefore in the Queue folder, however I've occasionaly noticed it in other folders.
cheers!
On Thu, 25 Sep 2003 05:51:33 -0400 (EDT) Charlie Brady (Charlie Brady charlieb-dovecot@e-smith.com) said:
Hi Ron!
Howdy stranger! ;-)
I don't see how swnding mail has anything to do with dovecot. That's a transaction between your mail client and your SMTP daemon.
Except that the messages get stored in both the Queue and Sent folders. It's the Queue folder that is generating the error.
Timo, Ron has an strace of the imap process if you need any further confirmation of the location of the fd leak. But it looks like you've found it anyway.
heh. I almost forgot about that.
cheers! Ron
On Wed, 2003-09-24 at 00:09, Ron wrote:
Maybe the opened files are timing out after a period of maybe two or three hours? Therefore the problem had been there all along (on my system, anyway) until a week or so ago when I changed from ten minute intervals to five minute intervals. ie; I'm opening the files faster than they timeout and close.
What do you think? Or have I gone completely mad? ;-)
It leaked one file descriptor every time a client did STATUS or SELECT command, so it was there before too. I don't think it ever closed them.. Maybe your client reconnected. Dunno :)
participants (4)
-
Charlie Brady
-
Marcus Rueckert
-
Ron
-
Timo Sirainen