[Dovecot] doveot reporting "No space left on device" - yet df show plenty of space / inodes.
Hi,
I am running dovecot 1.0.rc7 on a Suse Linux server. The server has approx 200+ mailboxes.
Last week the filesystem (/dev/mapper/datavg/dat2lv) ran out of space - causing it to do into read-only mode. When I realised this I allocated some more space and re-booted the machine...
Strangely it seems that dovecot is still having problems... It's like dovecot doesn't realise that there is enough space on the file system.
Also - it only is happening to a few users at this stage - not all users are having problems sending/receiving emails...
The dovecotstderr.log file contains:
dovecot: Aug 05 09:47:38 Error: IMAP(rebecca): file_dotlock_open(/data/mail/rebecca/dovecot-uidlist) failed: No space left on device
When I do a df -h
mail:/var/log # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda4 9.0G 5.7G 3.3G 64% / tmpfs 491M 8.0K 491M 1% /dev/shm /dev/mapper/datavg-dat2lv 300G 130G 171G 44% /data
When I do a df -i
mail:/var/log # df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda4 0 0 0 - / tmpfs 125505 5 125500 1% /dev/shm /dev/mapper/datavg-dat2lv 357753312 1030354 356722958 1% /data
I manually created a file in /data/mail/rebecca - which worked fine, and didn't cause any problems.
HELP!!!
I am stumped.... I can't understand why Dovecot is complaining..
I have manually stopped postfix and dovecot, and then re-started them.. But to no avail.
HELP - I am desperate.... My previously very reliable email server seems to be sick..
Meet singles near you. Try ninemsn dating now! http://a.ninemsn.com.au/b.aspx?URL=http%3A%2F%2Fdating%2Eninemsn%2Ecom%2Eau%2Fchannel%2Findex%2Easpx%3Ftrackingid%3D1046247&_t=773166080&_r=WL_TAGLINE&_m=EXT
On Aug 4, 2008, at 8:30 PM, Tom Brownlie wrote:
dovecot: Aug 05 09:47:38 Error: IMAP(rebecca): file_dotlock_open(/ data/mail/rebecca/dovecot-uidlist) failed: No space left on device
Some syscall is failing here with ENOSPC. Unfortunately the log
doesn't show which syscall it is.. You could check this with strace.
For example change:
mail_executable = /usr/bin/strace -o /tmp/dovecot.log /usr/local/ libexec/dovecot/imap
Then grep for ENOSPC from the /tmp/dovecot.log and show what it says?
I manually created a file in /data/mail/rebecca - which worked fine,
and didn't cause any problems.
Can you create hard links (ln file file2)?
I have manually stopped postfix and dovecot, and then re-started
them.. But to no avail.
Did you try rebooting the entire machine? Maybe the kernel is still
confused.
Hi Timo,
Thanks - but no luck....
#1 I changed dovecot.conf as suggested, stopped it and then restarted it. Tried this users account - and it failed again. There is no ENOSPC in the /tmp/dovecot.log file!! This file is growing rapidly, but no ENOSPC...
#2 Yes I can create hard links. I created a file in /data/mail/rebecca called toms.test I then ln toms.test toms.test.2 It worked fine.
#3 I haven't rebooted the server yet, because there are 200 users on it, and it is located remotely from me....
Any other suggestions???> From: tss@iki.fi> To: tomb1971@hotmail.com> Date: Mon, 4 Aug 2008 20:43:47 -0400> CC: dovecot@dovecot.org> Subject: Re: [Dovecot] doveot reporting "No space left on device" - yet df show plenty of space / inodes.> > On Aug 4, 2008, at 8:30 PM, Tom Brownlie wrote:> > > dovecot: Aug 05 09:47:38 Error: IMAP(rebecca): file_dotlock_open(/ > > data/mail/rebecca/dovecot-uidlist) failed: No space left on device> > Some syscall is failing here with ENOSPC. Unfortunately the log > doesn't show which syscall it is.. You could check this with strace. > For example change:> > mail_executable = /usr/bin/strace -o /tmp/dovecot.log /usr/local/ > libexec/dovecot/imap> > Then grep for ENOSPC from the /tmp/dovecot.log and show what it says?> > > I manually created a file in /data/mail/rebecca - which worked fine, > > and didn't cause any problems.> > Can you create hard links (ln file file2)?> > > I have manually stopped postfix and dovecot, and then re-started > > them.. But to no avail.> > Did you try rebooting the entire machine? Maybe the kernel is still > confused.>
Are you paid what you're worth? Find out: SEEK Salary Centre http://a.ninemsn.com.au/b.aspx?URL=http%3A%2F%2Fninemsn%2Eseek%2Ecom%2Eau%2Fcareer%2Dresources%2Fsalary%2Dcentre%2F%3Ftracking%3Dsk%3Ahet%3Asc%3Anine%3A0%3Ahot%3Atext&_t=764565661&_r=OCT07_endtext_salary&_m=EXT
On Aug 4, 2008, at 9:04 PM, Tom Brownlie wrote:
#1 I changed dovecot.conf as suggested, stopped it and then restarted it. Tried this users account - and it failed again. There is no ENOSPC in the /tmp/dovecot.log file!! This file is
growing rapidly, but no ENOSPC...
The file is now being recreated for each user that logs in. Ideally
you should find a way to get it written only for the one user that has
the problem and then reproduce the error only once so that you know
exactly what operations were in the log file. So perhaps something like:
mail_executable = /tmp/imap.sh
and imap.sh contains:
#!/bin/sh
if [ $USER = problemuser ]; then
exec strace -o /tmp/dovecot.log.$$.date +%s /usr/local/libexec/
dovecot/imap
else
exec /usr/local/libexec/dovecot/imap
fi
Hi Timo,
That got better results.... I did as requested, tried the problem user and:
- 3 log files were created
mail:/tmp # ls -l /tmp/dove* -rw------- 1 root root 8601 Aug 5 11:06 /tmp/dovecot.log.24160.1217898384 -rw------- 1 root root 68107 Aug 5 11:06 /tmp/dovecot.log.24166.1217898385 -rw------- 1 root root 10067 Aug 5 11:06 /tmp/dovecot.log.24170.1217898386
mail:/tmp # grep ENOSPC dovecot* dovecot.log.24166.1217898385:open("/data/mail/rebecca/temp.mail.24168.891513683e680a10", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = -1 ENOSPC (No space left on device) dovecot.log.24170.1217898386:open("/data/mail/rebecca/temp.mail.24172.9054608ba8d51503", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = -1 ENOSPC (No space left on device)
Does that help you in any way?
Tom.> From: tss@iki.fi> To: tomb1971@hotmail.com> Date: Mon, 4 Aug 2008 21:10:34 -0400> CC: dovecot@dovecot.org> Subject: Re: [Dovecot] doveot reporting "No space left on device" - yet df show plenty of space / inodes.> > On Aug 4, 2008, at 9:04 PM, Tom Brownlie wrote:> > > #1> > I changed dovecot.conf as suggested, stopped it and then restarted it.> > Tried this users account - and it failed again.> > There is no ENOSPC in the /tmp/dovecot.log file!! This file is > > growing rapidly, but no ENOSPC...> > The file is now being recreated for each user that logs in. Ideally > you should find a way to get it written only for the one user that has > the problem and then reproduce the error only once so that you know > exactly what operations were in the log file. So perhaps something like:> > mail_executable = /tmp/imap.sh> > and imap.sh contains:> > #!/bin/sh> > if [ $USER = problemuser ]; then> exec strace -o /tmp/dovecot.log.$$.date +%s /usr/local/libexec/ > dovecot/imap> else> exec /usr/local/libexec/dovecot/imap> fi>
Win a Nokia E51 with mobile Hotmail SMS alerts http://www.livelife.ninemsn.com.au/compIntro.aspx?compId=4589
On Tuesday of August 5 2008, Tom Brownlie wrote:
Also - it only is happening to a few users at this stage - not all users are having problems sending/receiving emails...
this suggests that problem could be quota-related, did you check? regards
Marcin Gryszkalis, PGP 0x9F183FA3 jabber jid:mg@fork.pl, gg:2532994 http://the.fork.pl
i do not believe that we have a quota system, but how do I check?> From: mg@fork.pl> To: dovecot@dovecot.org> Date: Tue, 5 Aug 2008 03:59:57 +0200> Subject: Re: [Dovecot] doveot reporting "No space left on device" - yet df show plenty of space / inodes.> > On Tuesday of August 5 2008, Tom Brownlie wrote:> > Also - it only is happening to a few users at this stage - not all users> > are having problems sending/receiving emails...> > this suggests that problem could be quota-related, did you check?> regards> -- > Marcin Gryszkalis, PGP 0x9F183FA3 > jabber jid:mg@fork.pl, gg:2532994> http://the.fork.pl
It's simple! Sell your car for just $40 at CarPoint.com.au http://a.ninemsn.com.au/b.aspx?URL=http%3A%2F%2Fsecure%2Dau%2Eimrworldwide%2Ecom%2Fcgi%2Dbin%2Fa%2Fci%5F450304%2Fet%5F2%2Fcg%5F801459%2Fpi%5F1004813%2Fai%5F859641&_t=762955845&_r=tig_OCT07&_m=EXT
On Tuesday of August 5 2008, Tom Brownlie wrote:
i do not believe that we have a quota system, but how do I check?
mount would show something like /dev/hda6 on /home type ext3 (rw,nosuid,nodev,noatime,usrquota)
usrquota is the one, could be also grpquota ro something like that...
regards
Marcin Gryszkalis, PGP 0x9F183FA3 jabber jid:mg@fork.pl, gg:2532994 http://the.fork.pl
participants (3)
-
Marcin Gryszkalis
-
Timo Sirainen
-
Tom Brownlie