Thank you Timo and Lorens for your quick responses. Timo, I thought as much in regards to the rename()s, but I was hoping.

Lorens, the INBOX in /var/spool/mail/%u would keep the INBOX on a non-quota file system. This means Dovecot reads the user's INBOX from /var/spool/mail/%u which is not what we want. We want the INBOX to be in the home directory. Users could leave mail in /var/spool/mail/%u and it would never go over quota. That is, obviously, not what we want. We need to never bounce mail yet deliver the mail that has built up for an over quota user as quickly as we can once that user goes below quota. If Dovecot moved the mail from /var/spool/mail/%u to the user's homedir then that is exactly what I want. As long as it fails gracefully if the user is over quota.

Other reasons keep us from creating separate mail queues and attempting to redeliver at intervals. Mainly, volume of mail.

Regards,

| Todd Piket | Email: todd@mtu.edu | | Programmer/Analyst | Phone: (906) 487-1720 | | Distributed Computing Services | | | Michigan Technological University | |

Lorens wrote:

On Wed, Feb 08, 2006 at 02:18:06PM -0500, Todd Piket wrote:

...

I already wrote to Timo, but I thought some others on this list may have had a similar situation. So here goes:

The scenario at Michigan Technological University stems from long standing traditions and policies that probably won't change. That said, two of the policies are 1.)we don't bounce mail and 2.)we guarantee delivery when we accept a piece of mail. The UW IMAP "snarf" capability was nice because it allowed us to introduce file system, per user quotas w/o worrying about the "INBOX" in an over quota situation. This is because the user could still read mail. The delivered mail would get "snarfed" from /var/mail/user when the user went back under quota.

In our test cases with switching to Maildir with Dovecot, we deliver directly to the user's INBOX instead of /var/mail/user so what do we do when the user is over quota because we can't bounce the mail?

Why do you do this? What would be the problem with something like

default_mail_env = maildir:~/Maildir:INBOX=/var/spool/mail/%u

which if I've understood correctly means that the user's stored mail is in his home directory Maildir, while new mail is delivered into /var/spool/mail/%u.

I've never had reason to try this, but these are obviously on different filesystems, and it seems to me as if it would be functionally equivalent to having Maildir/new on a different file system.

...

After many lengthy discussions our thought was this. Could we symbolically link the user's "Maildir/new" to a non-quota file system such that the move from "new" to "cur" would fail if the user is over quota?

See Timo's response.

In a more general way, how do you deal with people keeping all their mail in the INBOX? Is it only that it's annoying to have your mail marked "new"?

HTH.

The perfect scenario would be if Dovecot could check /var/mail/%u to see if there is new mail and if so, *attempt* to move it to ~/Maildir. If Dovecot gets an over quota message it can stop and just return "No new mail" to the client and allow the user to see existing mail, but do nothing more. This should happen every time the user "checks" for new mail such that an under quota scenario (user deletes Sent folder messages, for instance) results in mail from /var/mail/%u to be "delivered" to ~/Maildir.

This is bascially what UW-IMAP does. It is quite handy in our situation. The problem with doing this with Dovecot and/or Maildir is, I believe, you must introduce some kind of locking mechanism in /var/mail/%u because the delivery agent and Dovecot may step on each other's toes otherwise. Since locking is "bad" in maildir this is not ideal.

Regards,

| Todd Piket | Email: todd@mtu.edu | | Programmer/Analyst | Phone: (906) 487-1720 | | Distributed Computing Services | | | Michigan Technological University | |

Lorens wrote:

On Wed, Feb 08, 2006 at 03:50:17PM -0500, Todd Piket wrote:

...

Thank you Timo and Lorens for your quick responses. Timo, I thought as much in regards to the rename()s, but I was hoping.

Lorens, the INBOX in /var/spool/mail/%u would keep the INBOX on a non-quota file system. This means Dovecot reads the user's INBOX from /var/spool/mail/%u which is not what we want. We want the INBOX to be in the home directory. Users could leave mail in /var/spool/mail/%u and it would never go over quota. That is, obviously, not what we want.

OK... quick check, yes, in a pure maildir environment even when I keep my mails "new", they are still moved from new to cur. I wasn't certain :-)

...

We need to never bounce mail yet deliver the mail that has built up for an over quota user as quickly as we can once that user goes below quota. If Dovecot moved the mail from /var/spool/mail/%u to the user's homedir then that is exactly what I want. As long as it fails gracefully if the user is over quota.

I suppose that your "perfect" scenario would be if dovecot moved mail from /var/spool/mail/%u to ~/Maildir before sending it to the client, and failed with an "over quota" error message if the move to the maildir failed because it was over quota? Of course that means that mail is only moved when the client reads it, not as soon as he goes under quota, but you have the same behaviour with the other solutions.

Curtis,

Point 1 is/was taken into consideration, but we don't actually want the INBOX to remain in /var/mail/%u so using that feature of Dovecot doesn't get us anywhere at MTU.

Point 2 however has definitely piqued my interest. I will be exploring both options with enthusiasm starting tomorrow. Thank you so much for your ideas in this regard. As a Dovecot newbie I am not yet as familiar with it as you and others on this list are. I forgot that it does indeed spawn a binary that I can easily wrap.

Again I thank you. This will probably do the trick though if anyone else would like to chime in I'm still all ears.

Regards,

| Todd Piket | Email: todd@mtu.edu | | Programmer/Analyst | Phone: (906) 487-1720 | | Distributed Computing Services | | | Michigan Technological University | |

Curtis Maloney wrote:

Todd Piket wrote:

...

This is bascially what UW-IMAP does. It is quite handy in our situation. The problem with doing this with Dovecot and/or Maildir is, I believe, you must introduce some kind of locking mechanism in /var/mail/%u because the delivery agent and Dovecot may step on each other's toes otherwise. Since locking is "bad" in maildir this is not ideal.

Two points.

1. There's nothing to say /var/mail/%u can't be mbox. Dovecot already supports the INBOX being a different format to the rest of the mail.
   Handy if you don't run a Maildir capable LDA (but who does that? :)

2. Maildir was designed to not need locking. If dovecot tries to move a mail out of /new, it knows implicitly that the LDA is finished with it.
   This is because the LDA doesn't rename() new mail into /new from /tmp until it's finished writing it.

For some reason I recall there being an "on login" script hook possible with dovecot - if nothing else, just wrap the imap binary. This script could iterate over the /var/mail/%u/new/ directory, moving each file to ~/Maildir/tmp, then /new, and bailing out when it hits quota full.

Or am I missing some complexity here?

-- Curtis Maloney cmaloney@cardgate.net

On Thu, 9 Feb 2006, Curtis Maloney wrote:

1. There's nothing to say /var/mail/%u can't be mbox. Dovecot already supports the INBOX being a different format to the rest of the mail. Handy if you don't run a Maildir capable LDA (but who does that? :)

The actual point is that this INBOX is _writable_. The UW-Snarf feature will move the mail from a non-quota /spooldir/%u into the user's filesystem-quota limited INBOX. When that storage becomes full, this copying stops, waits till there is new available space, then moves further mails. So, the mail is deliviered to /spooldir/%u, and won't be bounced, because of "user over quota"; but the user cannot _use_ endless storage.

We have been using this UW-Imap feature, too, long time. This was on some Dovecot TODO-list some time ago -- I think it was called UW-slurp feature -- , but I don't found it in the CVS TODO file unfortunately. I will need some work-around for this too.

For some reason I recall there being an "on login" script hook possible with dovecot - if nothing else, just wrap the imap binary. This script could iterate over the /var/mail/%u/new/ directory, moving each file to ~/Maildir/tmp, then /new, and bailing out when it hits quota full.

See above, the spool file must be read regularily.

Bye,

-- Steffen Kaiser

On Wed, 2006-02-08 at 16:42 -0500, Todd Piket wrote:

The perfect scenario would be if Dovecot could check /var/mail/%u to see if there is new mail and if so, *attempt* to move it to ~/Maildir. If Dovecot gets an over quota message it can stop and just return "No new mail" to the client and allow the user to see existing mail, but do nothing more. This should happen every time the user "checks" for new mail such that an under quota scenario (user deletes Sent folder messages, for instance) results in mail from /var/mail/%u to be "delivered" to ~/Maildir.

Oh, and this could probably be done best using a plugin. You could check plugins/convert code as an example how to copy mails from one storage format to another at login. You could even hook into Dovecot's expunge operation so that immediately after expunging it tries to move more mails into INBOX. Doing it only while logging in isn't probably enough since IMAP connections can last a long time.

BTW. if you haven't already, you should also place Dovecot's indexes and maildir control dir to non-quota partition or things break.

Again, thanks all who responded. I'm still interested in ideas. As for the plugin, I am loving that idea! So, on to my favorite editor and some C code. Quick question: are plugins only in 1.0?

It is going to be difficult, at best, to get my institution to let me run "non-production" code in our production environment.

Is dovecot-stable still around and, if so, is that considered production level? Does that have the plugin architecture in it? I have until March 5th so if 1.0 is out by then I'm good anyway ;).

Regards,

| Todd Piket | Email: todd@mtu.edu | | Programmer/Analyst | Phone: (906) 487-1720 | | Distributed Computing Services | | | Michigan Technological University | |

Timo Sirainen wrote:

On Wed, 2006-02-08 at 16:42 -0500, Todd Piket wrote:

...

The perfect scenario would be if Dovecot could check /var/mail/%u to see if there is new mail and if so, *attempt* to move it to ~/Maildir. If Dovecot gets an over quota message it can stop and just return "No new mail" to the client and allow the user to see existing mail, but do nothing more. This should happen every time the user "checks" for new mail such that an under quota scenario (user deletes Sent folder messages, for instance) results in mail from /var/mail/%u to be "delivered" to ~/Maildir.

Oh, and this could probably be done best using a plugin. You could check plugins/convert code as an example how to copy mails from one storage format to another at login. You could even hook into Dovecot's expunge operation so that immediately after expunging it tries to move more mails into INBOX. Doing it only while logging in isn't probably enough since IMAP connections can last a long time.

BTW. if you haven't already, you should also place Dovecot's indexes and maildir control dir to non-quota partition or things break.

On Wed, 2006-02-08 at 22:18 +0100, Lorens wrote:

...

We need to never bounce mail yet deliver the mail that has built up for an over quota user as quickly as we can once that user goes below quota. If Dovecot moved the mail from /var/spool/mail/%u to the user's homedir then that is exactly what I want. As long as it fails gracefully if the user is over quota.

This sounds like an Extremely Complicated[tm] solution is desired where one doesn't appear to be necessary, just a bunch of little steps:

1. Have your MTA deliver messages into $HOME/Maildir

Your MTA should tempfail if delivery fails because write() failed [which it would if the user went over quota]. Temporary failures should be retried later.

2. If you really want to "never bounce [local] messages", consider configuring your MTA to simply never expire local mail- although some users consider bounces useful as it means the recipient didn't receive the mail in a [reasonable] amount of time.

In any event, on qmail this is trivial:

(cd /var/qmail/queue/local && find . -type f)
| awk '{ print "touch -c /var/qmail/queue/info/"$0 }' | sh

could go into your crontab and run (say) once a day or so. If you simply wanted to rejuvenate messages that were deferred due to quotas, you could hook something into repquota.

3. Finally, to "try as quickly as possible", consider using ngrep

ngrep -s 65535 EXPUNGE 'tcp dst port 143' 2>&1 | awk '{ if ($0 ~ /^T/) { print "pkill -ALRM qmail-send" } }' | sh

Substituting the "pkill -ALRM qmail-send" for whatever you use to tell your MTA to retry deliveries as soon as possible.

For the record, you'll note that dovecot isn't involved in any of this, but your MTA needs to do three things:

1. The Right Thing when write() fails and we're updating the INBOX
2. Provide a way to restart at least local delivery as soon as possible
3. Provide a way to rejuvenate messages

-- Internet Connection High Quality Web Hosting http://www.internetconnection.net/

On Thu, 9 Feb 2006, Todd Piket wrote:

A scenario similar to what you describe has been discussed in the group and is still a possibility. The plugin for Dovecot is presenting itself as the simplest and cleanest solution so far. The MTA will, of course, "do the

When there is "just" the INBOX, that's OK, but when you allow mail filtering, this can be problematic:

a) you need to keep both sets of folders in sync and scan all of them for new mails,& b) probably is possible to filter in condition of already present mails.

There would be another way to overcome the quota problem by having the file operations be performed by root. E.g. the LDA spools the file into MailDir's tmp/, then drops to user privilges, then performs filtering, spooling, bouncing a.s.o. This has security implications, probably, but because of the naming style of the files, there should be no clashes, e.g. when tmp/ is symlink into other areas of the filesystem.

Bye,

-- Steffen Kaiser