[Dovecot] (1.0.13) fsync failed: Disk quota exceeded for some accounts
Hello all,
We've recently migrated the mail server used by our 5000 students, from Tru64/UW-IMAP/Procmail/Postfix/mbox to Debian Etch/Dovecot/Deliver/Postfix/Maildir
E-mails are not stored directly on the server (except for index and control files), but on an NAS that exports the students' homedirs on NFS. Each student has a 100MB quota on the NAS.
Here is the configuration : vega:~# dovecot -n # 1.0.13: /etc/dovecot/dovecot.conf log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap pop3 listen: [::] ssl_disable: yes disable_plaintext_auth: no verbose_ssl: yes login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_greeting_capability(default): yes login_greeting_capability(imap): yes login_greeting_capability(pop3): no login_process_size: 32 login_max_processes_count: 1024 mail_privileged_group: mail mail_location: maildir:~/Maildir:INDEX=/var/mail/index/%u:CONTROL=/var/mail/control/%u maildir_copy_with_hardlinks: yes maildir_copy_preserve_filename: yes mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xu%08Xv namespace: type: private separator: / inbox: yes namespace: type: private separator: / prefix: mail/ hidden: yes auth default: cache_size: 1024 passdb: driver: pam args: blocking=yes cache_key=%u dovecot userdb: driver: passwd args: blocking=yes
When one mails a student that has filled his quota, one immediately receives this message : "Your message to <studentx> was automatically rejected: Not enough disk space
Reporting-UA: vega; Dovecot Mail Delivery Agent Final-Recipient: rfc822; studentx Original-Message-ID: 490F0A9A.8030109@utc.fr Disposition: automatic-action/MDN-sent-automatically; deleted"
And in /var/log/mail.log : (...) save failed to INBOX: Not enough disk space (...) Rejected: Not enough disk space
So far, so good, except that for *some* users (and I can't seem to find anything unusual about their account), one gets this in /var/log/mail.log : Nov 6 10:42:24 vega deliver(studentx): fsync(/voletu/users/studentx/Maildir/tmp/1225964544.P25878Q0M80993.vega) failed: Disk quota exceeded Nov 6 10:42:24 vega deliver(studentx): msgid=2254E7BC-665D-4599-B719-825E3A58DC1D@utc.fr: save failed to INBOX: Internal error occurred. Refer to server log for more information. [2008-11-06 10:42:24] Nov 6 10:42:24 vega postfix/local[25827]: 010F45EC18: to=studentx@vega.utc.fr, orig_to=studentx@etu.utc.fr, relay=local, delay=0.2, delays=0.02/0/0/0.17, dsn=4.3.0, status=deferred (temporary failure)
So for *some* users, it seems that Deliver doesn't detect that there isn't enough space, it tries to write the e-mail and of course fails, then reports an error, and Postfix interprets this as a temporary error and retries later.
After about a week : vega:~# qshape deferred T 5 10 20 40 80 160 320 640 1280 1280+ TOTAL 500 1 4 3 9 12 6 8 17 140 300 etu.utc.fr 499 1 4 3 9 12 6 8 17 139 300 vega.utc.fr 1 0 0 0 0 0 0 0 0 1 0
For the students that have this problem (about 30 for now), this happens *even* if I delete ~studentx/Maildir, /var/mail/index/studentx and /var/mail/control/studentx, let Deliver recreate the necessary files (by sending the user a mail) so that everything is clean, artificially fill the quota with a big file in the homedir, then send a mail again. It gets deferred...
I have looked at the changelog for Dovecot, but have not found a clear clue that is a bug (if it is !) that has already been corrected in a more recent version... Though if possible, I'd like to keep the Debian package (from etch-backports) instead of compiling it from source.
Also, while I've found Dovecot's wiki to be surprisingly informative, there are still a few points that did not seem very clear :
- should I use the quota:fs plugin in this case (it seems to work out well without it for most users) ?
- would it work in v1.0.13 on NFS ?
- if it worked, and I enabled it, what would change for me ?
- what is *supposed* to happen without the quota plugin when a user has filled his quota : mail is rejected, or mail is deferred ?
- and what about imap_quota ?
Best regards, Eric
Hi,
El Jueves, 6 de Noviembre de 2008 a las 11:30, Eric Marin escribió:
- should I use the quota:fs plugin in this case (it seems to work out well without it for most users) ?
quota:fs is only about reporting quota status using IMAP. So, it should do no diference in this case.
- would it work in v1.0.13 on NFS ?
No, you have to apply a patch to get 1.0 quota over NFS
- if it worked, and I enabled it, what would change for me ?
You can offer that information to your imap users. I think it's good offering this info to my users, but this is another issue.
Maybe the problem is with your NAS quotas. Have you tried deleting one of these users' quota, and reapplying it?
HTH
Joseba Torre. CIDIR Bizkaia.
Hi,
Joseba Torre a écrit :
Hi,
El Jueves, 6 de Noviembre de 2008 a las 11:30, Eric Marin escribió:
- should I use the quota:fs plugin in this case (it seems to work out well without it for most users) ?
quota:fs is only about reporting quota status using IMAP. So, it should do no diference in this case.
- would it work in v1.0.13 on NFS ?
No, you have to apply a patch to get 1.0 quota over NFS
- if it worked, and I enabled it, what would change for me ?
You can offer that information to your imap users. I think it's good offering this info to my users, but this is another issue.
Yes that would be great, i'll see about that, especially for Horde/IMP. But then, what does imap_quota do that quota itself doesn't ?
Maybe the problem is with your NAS quotas. Have you tried deleting one of these users' quota, and reapplying it?
HTH
Unfortunately, I don't think this can be the cause because we defined quotas globally on the NAS : only one line defines the quota for all students...
Eric
El Jueves, 6 de Noviembre de 2008 a las 13:20, Eric Marin escribió:
- if it worked, and I enabled it, what would change for me ?
You can offer that information to your imap users. I think it's good offering this info to my users, but this is another issue.
Yes that would be great, i'll see about that, especially for Horde/IMP. But then, what does imap_quota do that quota itself doesn't ?
I'm not sure, but I think quota:fs enables quota support for dovecot, and tells how it has to handle it. imap_quota enable the IMAP4 Quota extension. So, in the real world you need both.
Maybe the problem is with your NAS quotas. Have you tried deleting one of these users' quota, and reapplying it?
HTH
Unfortunately, I don't think this can be the cause because we defined quotas globally on the NAS : only one line defines the quota for all students...
And it can't handle per user exceptions?
Aaaaaaaaagur.
Joseba Torre. CIDIR Bizkaia.
Joseba Torre a écrit :
HTH
Unfortunately, I don't think this can be the cause because we defined quotas globally on the NAS : only one line defines the quota for all students...
And it can't handle per user exceptions?
Aaaaaaaaagur.
Well, yes it can, of course :-) What I meant was that we had made no exception for this user, so the user itself is just using the global quota... Besides, when we migrated our server, we already disabled all quotas, then re-enabled them once everything was completed.
Eric
Eric Marin wrote:
So for *some* users, it seems that Deliver doesn't detect that there isn't enough space, it tries to write the e-mail and of course fails, then reports an error, and Postfix interprets this as a temporary error and retries later.
Without knowing anything about the problem I would guess this is a rounding issue. Probably you are off by the number of bytes in your short message saying you are over-quota. Sometimes you are just inside and sometimes just outside?
Ed
Hi,
unfortunately, for those users that cause problem, even if the user is completely over-quota (say 900MB of files for a 100MB quota) *before* adding the small mail, Deliver still reports an fsync failure.
Eric
Ed W a écrit :
Eric Marin wrote:
So for *some* users, it seems that Deliver doesn't detect that there isn't enough space, it tries to write the e-mail and of course fails, then reports an error, and Postfix interprets this as a temporary error and retries later.
Without knowing anything about the problem I would guess this is a rounding issue. Probably you are off by the number of bytes in your short message saying you are over-quota. Sometimes you are just inside and sometimes just outside?
Ed
Try a script of quota warning like this:
#!/bin/bash PERCENT=$1 cat << EOF | /usr/lib/dovecot/deliver -d "$USER" -c /usr/local/etc/dovecot-nowarning.conf From: postmaster@domain.com Subject: Quota warning $PERCENT% full ........ EOF
Where /usr/local/etc/dovecot-nowarning.conf is a equal dovecot.conf without quota check.
With this, the users have the quota warning also with quota full, with no problems.
Cla.
Ed W ha scritto:
Eric Marin wrote:
So for *some* users, it seems that Deliver doesn't detect that there isn't enough space, it tries to write the e-mail and of course fails, then reports an error, and Postfix interprets this as a temporary error and retries later.
Without knowing anything about the problem I would guess this is a rounding issue. Probably you are off by the number of bytes in your short message saying you are over-quota. Sometimes you are just inside and sometimes just outside?
Ed
--
Claudio Prono Systems Development @ Atpss.net Srl, Divisione Implementazione Sistemi Via San Bernardino, 17 - 10137 Torino (TO) - IT Tel +39-011.32.72.100 Fax +39-011.32.46.497 PGP Fingerprint: 75C2 4049 E23D 2FBF A65F 40DB EA5C 11AC C2B0 3647 Disclaimer: http://atpss.net/disclaimer
On Thu, 2008-11-06 at 11:30 +0100, Eric Marin wrote:
So far, so good, except that for *some* users (and I can't seem to find anything unusual about their account), one gets this in /var/log/mail.log : Nov 6 10:42:24 vega deliver(studentx): fsync(/voletu/users/studentx/Maildir/tmp/1225964544.P25878Q0M80993.vega) failed: Disk quota exceeded
Strange that no-one had complained about this before. The problem is that your kernel is caching writes and it doesn't know the user is over quota. So then when fsync() finally gets called the kernel writes the data to NFS server, which replies that quota exceeded.
This fixes it: http://hg.dovecot.org/dovecot-1.0/rev/9879434a6339
It works, thank you very much !
What can I say, this is a shining example of what's good with open-source.
Eric
Timo Sirainen a écrit :
On Thu, 2008-11-06 at 11:30 +0100, Eric Marin wrote:
So far, so good, except that for *some* users (and I can't seem to find anything unusual about their account), one gets this in /var/log/mail.log : Nov 6 10:42:24 vega deliver(studentx): fsync(/voletu/users/studentx/Maildir/tmp/1225964544.P25878Q0M80993.vega) failed: Disk quota exceeded
Strange that no-one had complained about this before. The problem is that your kernel is caching writes and it doesn't know the user is over quota. So then when fsync() finally gets called the kernel writes the data to NFS server, which replies that quota exceeded.
This fixes it: http://hg.dovecot.org/dovecot-1.0/rev/9879434a6339
participants (5)
-
Claudio Prono
-
Ed W
-
Eric Marin
-
Joseba Torre
-
Timo Sirainen