[Dovecot] Well, "Hard filesystem quota can also be problematic"
"Hard filesystem quota can also be problematic", quoth http://www.dovecot.org/.
Since we actually use quota for our userbasg, this merited further investigation.
Using 1.0-stable, we enabled quota on /var and /users filesystems, and set very low quota for a test account. Then we had some mail delivered on it, and we moved some messages about.
At some point, Thunderbird started to croak about
"The current command did not succeed. The mail server responded:
Internal error occured. Refer to server log for more information.
[<timestamp>].",
"The current command did not succeed. The mail server responded: .",
"The current command did not succeed. The mail server responded:
Out of disk space."This I can live with. Hoever, at some point in our experiments we also found that a mailbox (mbox format) that did contain a few dozen messages suddenly was truncated to 0 bytes. Not so hot.
Can anyone tell me whether or not there is, or will be, some way of using dovecot on a system with quota without risk of data loss when quota limits are reached? How do people cope with this?
Regards, Jeroen.
p.s. That first error message gave rise to the enclosed patch.
diff -r -c1 dovecot-1.0-stable-patched/doc/auth-protocol.txt dovecot-1.0-stable/doc/auth-protocol.txt *** dovecot-1.0-stable-patched/doc/auth-protocol.txt Wed Jul 27 08:08:35 2005 --- dovecot-1.0-stable/doc/auth-protocol.txt Mon Jan 31 17:37:54 2005
*** 171,173 ****
! FAIL reply means an internal error occurred. Usually either a configuration mistake or temporary error caused by lost resource (eg. database down). --- 171,173 ----
! FAIL reply means an internal error occured. Usually either a configuration mistake or temporary error caused by lost resource (eg. database down). diff -r -c1 dovecot-1.0-stable-patched/doc/index.txt dovecot-1.0-stable/doc/index.txt *** dovecot-1.0-stable-patched/doc/index.txt Wed Jul 27 08:08:39 2005 --- dovecot-1.0-stable/doc/index.txt Mon Jan 31 17:37:54 2005
*** 89,91 **** processes has to scan the mailbox, other processes simply check from the ! log file what changes occurred.
--- 89,91 ---- processes has to scan the mailbox, other processes simply check from the ! log file what changes occured.
diff -r -c1 dovecot-1.0-stable-patched/src/imap/client.h dovecot-1.0-stable/src/imap/client.h *** dovecot-1.0-stable-patched/src/imap/client.h Wed Jul 27 08:08:26 2005 --- dovecot-1.0-stable/src/imap/client.h Mon Jan 31 17:37:54 2005
*** 68,70 **** /* Read a number of arguments. Returns TRUE if everything was read or ! FALSE if either needs more data or error occurred. */ int client_read_args(struct client *client, unsigned int count, --- 68,70 ---- /* Read a number of arguments. Returns TRUE if everything was read or ! FALSE if either needs more data or error occured. */ int client_read_args(struct client *client, unsigned int count, diff -r -c1 dovecot-1.0-stable-patched/src/imap/cmd-idle.c dovecot-1.0-stable/src/imap/cmd-idle.c *** dovecot-1.0-stable-patched/src/imap/cmd-idle.c Wed Jul 27 08:08:30 2005 --- dovecot-1.0-stable/src/imap/cmd-idle.c Mon Jan 31 17:37:54 2005
*** 159,161 **** } else if (ctx->sync_pending) { ! /* more changes occurred while we were sending changes to client */ --- 159,161 ---- } else if (ctx->sync_pending) { ! /* more changes occured while we were sending changes to client */ diff -r -c1 dovecot-1.0-stable-patched/src/lib/file-cache.h dovecot-1.0-stable/src/lib/file-cache.h *** dovecot-1.0-stable-patched/src/lib/file-cache.h Wed Jul 27 08:06:28 2005 --- dovecot-1.0-stable/src/lib/file-cache.h Mon Jan 31 17:37:55 2005
*** 12,14 **** /* Read data from file, returns how many bytes was actually read or -1 if ! error occurred. */ ssize_t file_cache_read(struct file_cache *cache, uoff_t offset, size_t size); --- 12,14 ---- /* Read data from file, returns how many bytes was actually read or -1 if ! error occured. */ ssize_t file_cache_read(struct file_cache *cache, uoff_t offset, size_t size); diff -r -c1 dovecot-1.0-stable-patched/src/lib/istream.h dovecot-1.0-stable/src/lib/istream.h *** dovecot-1.0-stable-patched/src/lib/istream.h Wed Jul 27 08:06:32 2005 --- dovecot-1.0-stable/src/lib/istream.h Mon Jan 31 17:37:54 2005
*** 60,62 **** /* Like i_stream_next_line(), but reads for more data if needed. Returns NULL ! if more data is needed or error occurred. */ char *i_stream_read_next_line(struct istream *stream); --- 60,62 ---- /* Like i_stream_next_line(), but reads for more data if needed. Returns NULL ! if more data is needed or error occured. */ char *i_stream_read_next_line(struct istream *stream); diff -r -c1 dovecot-1.0-stable-patched/src/lib/read-full.h dovecot-1.0-stable/src/lib/read-full.h *** dovecot-1.0-stable-patched/src/lib/read-full.h Wed Jul 27 08:06:38 2005 --- dovecot-1.0-stable/src/lib/read-full.h Mon Jan 31 17:37:54 2005
*** 3,5 ****
! /* Read data from file. Returns -1 if error occurred, or 0 if EOF came before everything was read, or 1 if all was ok. */ --- 3,5 ----
! /* Read data from file. Returns -1 if error occured, or 0 if EOF came before everything was read, or 1 if all was ok. */ diff -r -c1 dovecot-1.0-stable-patched/src/lib/strfuncs.c dovecot-1.0-stable/src/lib/strfuncs.c *** dovecot-1.0-stable-patched/src/lib/strfuncs.c Wed Jul 27 08:06:47 2005 --- dovecot-1.0-stable/src/lib/strfuncs.c Mon May 16 23:15:10 2005
*** 100,102 **** if (len < 0) { ! /* some error occurred */ len = 0; --- 100,102 ---- if (len < 0) { ! /* some error occured */ len = 0; diff -r -c1 dovecot-1.0-stable-patched/src/lib/write-full.h dovecot-1.0-stable/src/lib/write-full.h *** dovecot-1.0-stable-patched/src/lib/write-full.h Wed Jul 27 08:06:43 2005 --- dovecot-1.0-stable/src/lib/write-full.h Mon Jan 31 17:37:55 2005
*** 3,5 ****
! /* Write data into file. Returns -1 if error occurred, or 0 if all was ok. If there's not enough space in device, -1 with ENOSPC is returned, and --- 3,5 ----
! /* Write data into file. Returns -1 if error occured, or 0 if all was ok. If there's not enough space in device, -1 with ENOSPC is returned, and diff -r -c1 dovecot-1.0-stable-patched/src/lib-imap/imap-parser.h dovecot-1.0-stable/src/lib-imap/imap-parser.h *** dovecot-1.0-stable-patched/src/lib-imap/imap-parser.h Wed Jul 27 08:07:00 2005 --- dovecot-1.0-stable/src/lib-imap/imap-parser.h Mon Jan 31 17:37:55 2005
*** 88,90 **** need to do that. Returns number of arguments read (may be less than count ! in case of EOL), -2 if more data is needed or -1 if error occurred.
--- 88,90 ---- need to do that. Returns number of arguments read (may be less than count ! in case of EOL), -2 if more data is needed or -1 if error occured.
diff -r -c1 dovecot-1.0-stable-patched/src/lib-mail/message-parser.h dovecot-1.0-stable/src/lib-mail/message-parser.h *** dovecot-1.0-stable-patched/src/lib-mail/message-parser.h Wed Jul 27 08:06:56 2005 --- dovecot-1.0-stable/src/lib-mail/message-parser.h Mon Jan 31 17:37:55 2005
*** 112,114 **** input stream is non-blocking and more data needs to be read, -1 when all is ! done or error occurred (see stream's error status). */ int message_parse_header_next(struct message_header_parser_ctx *ctx, --- 112,114 ---- input stream is non-blocking and more data needs to be read, -1 when all is ! done or error occured (see stream's error status). */ int message_parse_header_next(struct message_header_parser_ctx *ctx, diff -r -c1 dovecot-1.0-stable-patched/src/lib-sql/sql-api.h dovecot-1.0-stable/src/lib-sql/sql-api.h *** dovecot-1.0-stable-patched/src/lib-sql/sql-api.h Wed Jul 27 08:06:51 2005 --- dovecot-1.0-stable/src/lib-sql/sql-api.h Mon Jan 31 17:37:55 2005
*** 23,25 **** /* Go to next row, returns 1 if ok, 0 if this was the last row or -1 if error ! occurred. This needs to be the first call for result. */ int sql_result_next_row(struct sql_result *result); --- 23,25 ---- /* Go to next row, returns 1 if ok, 0 if this was the last row or -1 if error ! occured. This needs to be the first call for result. */ int sql_result_next_row(struct sql_result *result); diff -r -c1 dovecot-1.0-stable-patched/src/lib-storage/index/maildir/maildir-sync.c dovecot-1.0-stable/src/lib-storage/index/maildir/maildir-sync.c *** dovecot-1.0-stable-patched/src/lib-storage/index/maildir/maildir-sync.c Wed Jul 27 08:08:03 2005 --- dovecot-1.0-stable/src/lib-storage/index/maildir/maildir-sync.c Mon Jan 31 17:37:55 2005
*** 7,9 **** We want to be as efficient as we can. The most efficient way to ! check if changes have occurred is to stat() the new/ and cur/ directories and uidlist file - if their mtimes haven't changed, --- 7,9 ---- We want to be as efficient as we can. The most efficient way to ! check if changes have occured is to stat() the new/ and cur/ directories and uidlist file - if their mtimes haven't changed,
*** 14,16 **** then make a modification. Such modifications wouldn't get noticed ! until a new modification occurred later.
--- 14,16 ---- then make a modification. Such modifications wouldn't get noticed ! until a new modification occured later.
diff -r -c1 dovecot-1.0-stable-patched/src/lib-storage/mail-storage.c dovecot-1.0-stable/src/lib-storage/mail-storage.c *** dovecot-1.0-stable-patched/src/lib-storage/mail-storage.c Wed Jul 27 08:07:58 2005 --- dovecot-1.0-stable/src/lib-storage/mail-storage.c Mon Jan 31 17:37:55 2005
*** 12,14 ****
#define CRITICAL_MSG
! "Internal error occurred. Refer to server log for more information."
#define CRITICAL_MSG_STAMP CRITICAL_MSG " [%Y-%m-%d %H:%M:%S]"
--- 12,14 ----
#define CRITICAL_MSG
! "Internal error occured. Refer to server log for more information."
#define CRITICAL_MSG_STAMP CRITICAL_MSG " [%Y-%m-%d %H:%M:%S]"
diff -r -c1 dovecot-1.0-stable-patched/src/lib-storage/mail-storage.h dovecot-1.0-stable/src/lib-storage/mail-storage.h
*** dovecot-1.0-stable-patched/src/lib-storage/mail-storage.h Wed Jul 27 08:07:25 2005
--- dovecot-1.0-stable/src/lib-storage/mail-storage.h Mon Jan 31 17:37:55 2005
*** 225,227 **** /* Deinitialize mailbox list request. Returns FALSE if some error ! occurred while listing. */ int mail_storage_mailbox_list_deinit(struct mailbox_list_context *ctx); --- 225,227 ---- /* Deinitialize mailbox list request. Returns FALSE if some error ! occured while listing. */ int mail_storage_mailbox_list_deinit(struct mailbox_list_context *ctx);
*** 239,241 ****
! /* Returns the error message of last occurred error. */ const char *mail_storage_get_last_error(struct mail_storage *storage, --- 239,241 ----
! /* Returns the error message of last occured error. */ const char *mail_storage_get_last_error(struct mail_storage *storage,
*** 252,254 **** const char *name, enum mailbox_open_flags flags); ! /* Close the box. Returns FALSE if some cleanup errors occurred, but the mailbox was closed anyway. */ --- 252,254 ---- const char *name, enum mailbox_open_flags flags); ! /* Close the box. Returns FALSE if some cleanup errors occured, but the mailbox was closed anyway. */
*** 373,378 **** /* Get the time message was received (IMAP INTERNALDATE). ! Returns (time_t)-1 if error occurred. */ time_t (*get_received_date)(struct mail *mail); /* Get the Date-header in mail. Timezone is in minutes. ! Returns (time_t)-1 if error occurred, 0 if field wasn't found or couldn't be parsed. */ --- 373,378 ---- /* Get the time message was received (IMAP INTERNALDATE). ! Returns (time_t)-1 if error occured. */ time_t (*get_received_date)(struct mail *mail); /* Get the Date-header in mail. Timezone is in minutes. ! Returns (time_t)-1 if error occured, 0 if field wasn't found or couldn't be parsed. */
*** 380,385 **** /* Get the full virtual size of mail (IMAP RFC822.SIZE). ! Returns (uoff_t)-1 if error occurred */ uoff_t (*get_virtual_size)(struct mail *mail); /* Get the full physical size of mail. ! Returns (uoff_t)-1 if error occurred */ uoff_t (*get_physical_size)(struct mail *mail); --- 380,385 ---- /* Get the full virtual size of mail (IMAP RFC822.SIZE). ! Returns (uoff_t)-1 if error occured */ uoff_t (*get_virtual_size)(struct mail *mail); /* Get the full physical size of mail. ! Returns (uoff_t)-1 if error occured */ uoff_t (*get_physical_size)(struct mail *mail); diff -r -c1 dovecot-1.0-stable-patched/src/lib-storage/subscription-file/subscription-file.h dovecot-1.0-stable/src/lib-storage/subscription-file/subscription-file.h *** dovecot-1.0-stable-patched/src/lib-storage/subscription-file/subscription-file.h Wed Jul 27 08:08:08 2005 --- dovecot-1.0-stable/src/lib-storage/subscription-file/subscription-file.h Mon Jan 31 17:37:55 2005
*** 9,11 ****
! /* Deinitialize subscription file listing. Returns FALSE if some error occurred while listing. */ --- 9,11 ----
! /* Deinitialize subscription file listing. Returns FALSE if some error occured while listing. */ diff -r -c1 dovecot-1.0-stable-patched/src/login-common/ssl-proxy-gnutls.c dovecot-1.0-stable/src/login-common/ssl-proxy-gnutls.c *** dovecot-1.0-stable-patched/src/login-common/ssl-proxy-gnutls.c Wed Jul 27 08:08:22 2005 --- dovecot-1.0-stable/src/login-common/ssl-proxy-gnutls.c Mon Jan 31 17:37:55 2005
*** 82,84 **** if (verbose_ssl) { ! /* fatal error occurred */ if (error == GNUTLS_E_FATAL_ALERT_RECEIVED) { --- 82,84 ---- if (verbose_ssl) { ! /* fatal error occured */ if (error == GNUTLS_E_FATAL_ALERT_RECEIVED) { diff -r -c1 dovecot-1.0-stable-patched/src/login-common/ssl-proxy.h dovecot-1.0-stable/src/login-common/ssl-proxy.h *** dovecot-1.0-stable-patched/src/login-common/ssl-proxy.h Wed Jul 27 08:08:14 2005 --- dovecot-1.0-stable/src/login-common/ssl-proxy.h Mon Jan 31 17:37:55 2005
*** 9,11 **** /* establish SSL connection with the given fd, returns a new fd which you ! must use from now on, or -1 if error occurred. Unless -1 is returned, the given fd must be simply forgotten. */ --- 9,11 ---- /* establish SSL connection with the given fd, returns a new fd which you ! must use from now on, or -1 if error occured. Unless -1 is returned, the given fd must be simply forgotten. */
Jeroen Scheerder wrote:
"Hard filesystem quota can also be problematic", quoth http://www.dovecot.org/.
[...]
Can anyone tell me whether or not there is, or will be, some way of using dovecot on a system with quota without risk of data loss when quota limits are reached? How do people cope with this?
[...]
Nobody uses quota on their servers anymore these days, I presume? :-)
We do...otherwise, there's the inevitable 'Tragedy of the Commons'...to say nothing of idiots mailing MB files to everybody in the community. And we're still on UW IMAP and the one-big-file INBOX which is a horror show when it gets big.
lack of quota compatibility is a migration to Dovecote show-stopper for us.
Jeroen Scheerder wrote:
Jeroen Scheerder wrote:
"Hard filesystem quota can also be problematic", quoth http://www.dovecot.org/.
[...]
Can anyone tell me whether or not there is, or will be, some way of using dovecot on a system with quota without risk of data loss when quota limits are reached? How do people cope with this?
[...]
Nobody uses quota on their servers anymore these days, I presume? :-)
--
Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean@bard.edu voice: 845-758-7475, fax: 845-758-7035
I'm just wondering about a quota system. How would that work?
I want to limit mail box sizes - but I don't want users losing any email. Seems mutually exclusive.
So - how is this done in the real world? Automatic warnings?
We have our 'incoming' mail areas and the users mailstore on seperate disks.
New mail is stored in an unquotad spool on one disk (meaning they can have as much mail as they like delivered within the systems hard disk capacity) and on login its moved onto a quotad disk along with the rest of their mail.
So they're quota'd on the mail they keep hanging about, but if they go over quota any new mail is held in the spool until they sort themselves out. So no mail should be lost.
We use mbox but I guess you could look at it in terms of maildir as having the new/ folder symlinked pr mounted on an unquotad disk for spooling and all other folders (including cur/) on the normal quota'd mail area.
Not perfect, but works reasonably well for us. Only problem being when people ignore the soft limit warnings and hit their hard limit / expiry period. They then can't even delete mails to clean their mail up so have to contact us.
When we can get users to reliably read instructions we'll be happy :P
Kev
Marc Perkel wrote:
I'm just wondering about a quota system. How would that work?
I want to limit mail box sizes - but I don't want users losing any email. Seems mutually exclusive.
So - how is this done in the real world? Automatic warnings?
Marc Perkel wrote:
I'm just wondering about a quota system. How would that work?
I want to limit mail box sizes - but I don't want users losing any email. Seems mutually exclusive.
So - how is this done in the real world? Automatic warnings?
In my case, Postfix seems to handle it fine, bouncing the message with a "user's mailbox is full" reply. I didn't set up anything explicitly to handle this.
However, Kev's suggestion of inbox in a separate spool is possibly a preferable solution.
-- Curtis Maloney
Stewart Dean wrote:
We do...otherwise, there's the inevitable 'Tragedy of the Commons'...to say nothing of idiots mailing MB files to everybody in the community. And we're still on UW IMAP and the one-big-file INBOX which is a horror show when it gets big.
I also run Dovecot on a box with filesystem quotas, but just like Alan Premselaar, I also use Maildir. When people have hit their quota, there's been no evidence of lost mail, as Postfix simply refuses the message.
Of course, if you're going to stick with mbox, well, hope you enjoy the headaches :)
-- Curtis Maloney
Curtis Maloney wrote:
Stewart Dean wrote:
We do...otherwise, there's the inevitable 'Tragedy of the Commons'...to say nothing of idiots mailing MB files to everybody in the community. And we're still on UW IMAP and the one-big-file INBOX which is a horror show when it gets big.
I also run Dovecot on a box with filesystem quotas, but just like Alan Premselaar, I also use Maildir. When people have hit their quota, there's been no evidence of lost mail, as Postfix simply refuses the message.
This is a non sequitur. Mail storage is more than just the inbox.
Postfix does honour its "mailbox_size_limit", true. But there's more to store than just a single mailbox, of which the maximum size can be set to enforce quota, in some or other way.
Or am I misunderstanding, and does Postfix (or procmail, which is typically used as Postfix' local delivery agent) have explicit quota handling that I've been overlooking?
Perhaps you mean to say that a write error occurs upon hitting hard quota, or upon exceeding the grace period after a soft quotum has been surpassed, and that the local delivery agent seems to handle this gracefully.
That may be true. However, I think this fact alone, although it lets one sleep better, is not sufficient. Dovecot especially may require even more storage after delivery. Furthermore, mail delivery isn't the only way mailboxes get modified: the IMAP server also needs to, and to make it really work it seems necessary for the IMAP server to report filesystem quota to the IMAP clients.
Apparently, v4 of the IMAP protocol has a QUOTA extension for this; it's reported as a CAPABILITY. I think some of that is documented in [1]. Am I wrong in suspecting that this is a higly desirable, or even a required, piece of the puzzle?
-- J$
[1] http://www.melnikov.ca/mel/Drafts/draft-cridland-imap-quota-00.txt
On 16.8.2005, at 09:16, Jeroen Scheerder wrote:
Apparently, v4 of the IMAP protocol has a QUOTA extension for this; it's reported as a CAPABILITY. I think some of that is documented in [1]. Am I wrong in suspecting that this is a higly desirable, or even a required, piece of the puzzle?
I don't think it's exactly needed as few clients use it. But Dovecot has a quota plugin available for it and it kind of works. Currently it has implemented only one way to track quota though: just read all files and sum their sizes. This should work fast enough with mboxes, but not a good idea with maildir. Other implementations will come later..
Tomi has written something about how to use the quota plugin in http://wiki.dovecot.org/LDA
Hi
I want a 50MB quota for each user (using mbox files under var/spool/mail)
I guess first i rebuild 1.0 stable using the wiki info but am confused by the syntax of the args line why uid=5000 gid=5000
I am using default_mail_env = mbox:/var/spool/mail/%u
my users home dir are under /home but mail is under /var/spool/mail which is where i want the quota = 52428800 (50MB)
auth default { .. userdb static { args = uid=5000 gid=5000 home=/var/vmail/%d/%n quota=/var/vmail/%d/%n/mail:storage=10485760 } .. }
would my args be
args = uid=5000 gid=5000 home=/var/spool/mail/%u quota=/var/spool/mail/%u/mail:storage=52428800
Mark
----- Original Message ----- From: "Timo Sirainen" tss@iki.fi To: "Jeroen Scheerder" Jeroen.Scheerder@phil.uu.nl Cc: dovecot@dovecot.org Sent: Tuesday, August 16, 2005 9:59 AM Subject: Re: [Dovecot] Well, "Hard filesystem quota can also be problematic"
Obantec Support wrote:
I guess first i rebuild 1.0 stable using the wiki info but am confused by
I'm not so sure if quota plugin is going to work with 1.0 stable release..
the syntax of the args line why uid=5000 gid=5000
That's because it is for virtual domain system where you have single uid and gid for all mail users, you cannot use static userdb on a "normal" mail host where you have unique user id's.
my users home dir are under /home but mail is under /var/spool/mail which is where i want the quota = 52428800 (50MB)
This is a bit tricky one, see at this time quota plugin works by summing up sizes of all files under a single directory structure you provide on a quota= parameter, so you'd have to have inbox for example at "/home/user/mail/inbox" and then folders at "/home/user/mail/folders" and set: "quota=/home/%u/mail:storage=52428800"
But I think it's quite easy to enhance quota plugin so that it looks size of mailbox at /var/mail/ also.
-- Tomi Hakala
Timo Sirainen wrote:
On 16.8.2005, at 09:16, Jeroen Scheerder wrote:
Apparently, v4 of the IMAP protocol has a QUOTA extension for this; it's reported as a CAPABILITY. I think some of that is documented in [1]. Am I wrong in suspecting that this is a higly desirable, or even a required, piece of the puzzle?
I don't think it's exactly needed as few clients use it. But Dovecot has a quota plugin available for it and it kind of works. Currently it has implemented only one way to track quota though: just read all files and sum their sizes. This should work fast enough with mboxes, but not a good idea with maildir. Other implementations will come later..
Tomi has written something about how to use the quota plugin in http://wiki.dovecot.org/LDA
That, if I'm reading correctly, is for systems on which for some or other reason one cannot use file system quota's, but still wants to make Dovecot handle quota enforcement. The page you're referring to (also) discusses a plugin for that.
What I'm looking for, though, is to actually use file system quota. I would have thought reporting file system quota in the way I described above would be helpful, but reading your words I'm having second thoughts about that.
Which kind of leads me to think that I seem to be looking for problems that may not be there; perhaps file system quota's, along with a solid delivery agent (such as procmail) and a strategy of making sure there's sufficient storage for mailbox writes by the imap server, as sketched before, will suffice?
Jeroen Scheerder wrote:
that may not be there; perhaps file system quota's, along with a solid delivery agent (such as procmail) and a strategy of making sure there's sufficient storage for mailbox writes by the imap server, as sketched before, will suffice?
Don't know if it helps you (didn't even read the whole thread), but there are MTAs (at least Exim) with builtin delivery and quota handling.
On Tuesday 2005-August-16 01:16, Jeroen Scheerder wrote:
I also run Dovecot on a box with filesystem quotas, but just like Alan Premselaar, I also use Maildir. When people have hit their quota, there's been no evidence of lost mail, as Postfix simply refuses the message.
This is a non sequitur. Mail storage is more than just the inbox.
Postfix does honour its "mailbox_size_limit", true. But there's more to store than just a single mailbox, of which the maximum size can be set to enforce quota, in some or other way.
Or am I misunderstanding, and does Postfix (or procmail, which is typically used as Postfix' local delivery agent) have explicit quota handling that I've been overlooking?
First, one thing you are wrong about is that Postfix typically does not use procmail. Postfix includes its own LDA, local(8). Distributors can do what they will, and they might set it up to use procmail, but if you install Postfix from source, the default LDA is local(8).
Perhaps you mean to say that a write error occurs upon hitting hard quota, or upon exceeding the grace period after a soft quotum has been surpassed, and that the local delivery agent seems to handle this gracefully.
I don't use quota, but here's my understanding of the sequence:
- Mail arrives for valid user "packrat", who happens to be over quota.
- Postfix accepts and queues the mail (as user "postfix".)
- Postfix passes to MDA/LDA. That write would be as user "packrat", who, per above, is over quota. FS write fails.
- Postfix generates a bounce for the envelope sender.
If "packrat" is not quite to the quota, say, 1MB below it, and gets a 2MB mail, that would bounce. But a later 2KB mail would be delivered, and so on, until the quota is reached.
That may be true. However, I think this fact alone, although it lets one sleep better, is not sufficient. Dovecot especially may require even more storage after delivery. Furthermore, mail delivery isn't
Index and subscription files, yes.
mail to this address is discarded unless "/dev/rob0"
or "not-spam" is in Subject: header/dev/rob0 wrote:
[...]
Or am I misunderstanding, and does Postfix (or procmail, which is typically used as Postfix' local delivery agent) have explicit quota handling that I've been overlooking?
First, one thing you are wrong about is that Postfix typically does not use procmail. Postfix includes its own LDA, local(8). Distributors can do what they will, and they might set it up to use procmail, but if you install Postfix from source, the default LDA is local(8).
I did not claim procmail to be the default delivery agent for Postfix; I called it the typical delivery agent, since most Postfix installations I've seen (and I've seen quite a few) use procmail as the LDA.
Either way, it doesn't matter. Both don't bother with any quota support; they just write stuff and handle write failures soundly, in a generic way.
Perhaps you mean to say that a write error occurs upon hitting hard quota, or upon exceeding the grace period after a soft quotum has been surpassed, and that the local delivery agent seems to handle this gracefully.
I don't use quota, but here's my understanding of the sequence:
- Mail arrives for valid user "packrat", who happens to be over quota.
- Postfix accepts and queues the mail (as user "postfix".)
- Postfix passes to MDA/LDA. That write would be as user "packrat", who, per above, is over quota. FS write fails.
- Postfix generates a bounce for the envelope sender.
If "packrat" is not quite to the quota, say, 1MB below it, and gets a 2MB mail, that would bounce. But a later 2KB mail would be delivered, and so on, until the quota is reached.
That is exactly what happens. Recipients get bounces. The reason of delivery failure that is reported, though, may not necessarily make much sense.
What does *not* happen, is that Postfix checks for sufficient storage quota before delivery before the delivery attempt. The author I responded to, though, wrote "When people have hit their quota [..] Postfix simply refuses the message." Now what I tried to write, and what you seem to be confirming, is that this is not what happens. Postfix doesn't refuse the message, it accepts it, passes it on to local delivery -- and then hits quota-induced I/O errors.
Jeroen Scheerder wrote:
Can anyone tell me whether or not there is, or will be, some way of using dovecot on a system with quota without risk of data loss when quota limits are reached? How do people cope with this?
I use Dovecot 0.9.14 with disk quotas.
I use maildir format however. you can't expect to use mbox format and not get data loss/corruption when you hit a filesystem quota. I've fairly certain that even UW-IMAP will corrupt your mbox files in the case of exceeding your quota as well.
so, with the maildir storage format, procmail and mimedefang rules to reject mail for users that are over-quota, and dovecot configured to store indexes and control files on a non-quota'd filesystem, I've got a successful installation with filesystem quotas.
alan
On Fri, 2005-07-29 at 18:24 +0900, Alan Premselaar wrote:
I use maildir format however. you can't expect to use mbox format and not get data loss/corruption when you hit a filesystem quota. I've fairly certain that even UW-IMAP will corrupt your mbox files in the case of exceeding your quota as well.
Actually neither should be corrupting it. Both first grow the file size by the needed amount and only after that start writing to it. So if the growing fails, they revert back to original size and give some error message.
Alan Premselaar wrote:
Jeroen Scheerder wrote:
Can anyone tell me whether or not there is, or will be, some way of using dovecot on a system with quota without risk of data loss when quota limits are reached? How do people cope with this?
I use Dovecot 0.9.14 with disk quotas.
I use maildir format however. you can't expect to use mbox format and not get data loss/corruption when you hit a filesystem quota. I've fairly certain that even UW-IMAP will corrupt your mbox files in the case of exceeding your quota as well.
so, with the maildir storage format, procmail and mimedefang rules to reject mail for users that are over-quota, and dovecot configured to store indexes and control files on a non-quota'd filesystem, I've got a successful installation with filesystem quotas.
Thanks, that's insightful. I'lll probably add a quota checker to my procmail setup, and configure the indexes to get stored in a tmp dir sans quota. What are these control files? I've not seen them appear in any dovecot config file...
On Wed, 2005-07-27 at 08:12 +0200, Jeroen Scheerder wrote:
"The current command did not succeed. The mail server responded: Out of disk space."
This I can live with. Hoever, at some point in our experiments we also found that a mailbox (mbox format) that did contain a few dozen messages suddenly was truncated to 0 bytes. Not so hot.
This shouldn't happen. There is really only one place where mbox file is truncated in the code, and it shouldn't be done unless client was actually expunging messages. If you can reproduce it with strace output I'd like to see that (set mail_executable = strace -o log .../dovecot/imap).
p.s. That first error message gave rise to the enclosed patch.
Thanks. I must have been writing occurred wrong forever :)
participants (12)
-
/dev/rob0
-
Alan Premselaar
-
Curtis Maloney
-
Jakob Hirsch
-
Jeroen Scheerder
-
Kev
-
Marc Perkel
-
Obantec Support
-
Steffen Kaiser
-
Stewart Dean
-
Timo Sirainen
-
Tomi Hakala