Re: [Dovecot] Auto create folder(s)
Fábio M. Catunda wrote:
Bill Landry escreveu:
Lars Stavholm wrote:
Charles Marcus wrote:
On 9/26/2007, Lars Stavholm (stava@telcotec.se) wrote:
if not -> maybe this is for you:http://wiki.dovecot.org/PostLoginScripting
Even better.
But then this would happen for every user, at every login, when all you need is to do it once, when the new user is created...
Very true.
Postfixadmin does this automatically when the account is created, and this is the ideal time to do such things, no?
It is the ideal time, you're right about that. However, to my understanding, Postfixadmin is a web interface for managing postfix config and users in a mysql db. Not very automated. Our users are normal unix users (we do use LDAP) and are not stored in mysql. We use webmin/LDAP for user maintenance.
These folders could easily be create dynamically by maildrop, if used as your MDA. I use it as my MDA between Postfix and Dovecot to automatically create spam folders for new accounts and also for delivering mail that exceeds our spam thresholds into the spam folder for end user review.
Bill
But with maildrop you still run the process of creating/checking on every delivery, same thing as dovecot, right?
Correct. If the folder does not exist, maildrop will create it on first delivery. If the end user deletes the folder, maildrop will recreate the folder on the next message delivery to that account.
Bill
On 9/26/2007, Bill Landry (bill@inetmsg.com) wrote:
But with maildrop you still run the process of creating/checking on every delivery, same thing as dovecot, right?
Correct. If the folder does not exist, maildrop will create it on first delivery. If the end user deletes the folder, maildrop will recreate the folder on the next message delivery to that account.
Could it could be written to simply create the folder if the target folder doesn't exist (ie, if the initial save fails due to non-existent folder)? Then there would be no performance hit...
--
Best regards,
Charles
Charles Marcus escreveu:
On 9/26/2007, Bill Landry (bill@inetmsg.com) wrote:
But with maildrop you still run the process of creating/checking on every delivery, same thing as dovecot, right?
Correct. If the folder does not exist, maildrop will create it on first delivery. If the end user deletes the folder, maildrop will recreate the folder on the next message delivery to that account.
Could it could be written to simply create the folder if the target folder doesn't exist (ie, if the initial save fails due to non-existent folder)? Then there would be no performance hit...
There will be a performance hit couse you have to check if the folder exists or not. With maildrop its pretty easy to do that, but you will have an extra access to your HD on every message delivery. Maildrop can run shell commands it the user that runs it have a valid shell (Debian-exim do NOT have a valid shell, I created another user just to make the delivery), so, you can run something like
/^Envelope-to:.*/
getaddr($MATCH) =~ /^.*/;
DEST = $MATCH
USER = /bin/echo $DEST | /usr/bin/cut -f1 -d'@'
DOMAIN = /bin/echo $DEST | /usr/bin/cut -f2 -d'@'
DOEXIST=[ -d /var/mail/$DOMAIN/$USER/Maildir ]; echo $?
if ($DOEXIST == 1){ <RUN SOMETHING HERE TO CREATE MAILDIRS> }
Thats it!
[]'s, FMC!
!DSPAM:46fabb75183501503319954!
Fábio M. Catunda wrote:
Charles Marcus escreveu:
On 9/26/2007, Bill Landry (bill@inetmsg.com) wrote:
But with maildrop you still run the process of creating/checking on every delivery, same thing as dovecot, right?
Correct. If the folder does not exist, maildrop will create it on first delivery. If the end user deletes the folder, maildrop will recreate the folder on the next message delivery to that account.
Could it could be written to simply create the folder if the target folder doesn't exist (ie, if the initial save fails due to non-existent folder)? Then there would be no performance hit...
There will be a performance hit couse you have to check if the folder exists or not. With maildrop its pretty easy to do that, but you will have an extra access to your HD on every message delivery. Maildrop can run shell commands it the user that runs it have a valid shell (Debian-exim do NOT have a valid shell, I created another user just to make the delivery), so, you can run something like
/^Envelope-to:.*/ getaddr($MATCH) =~ /^.*/; DEST = $MATCH USER =
/bin/echo $DEST | /usr/bin/cut -f1 -d'@'DOMAIN =/bin/echo $DEST | /usr/bin/cut -f2 -d'@'DOEXIST=
[ -d /var/mail/$DOMAIN/$USER/Maildir ]; echo $?if ($DOEXIST == 1){ <RUN SOMETHING HERE TO CREATE MAILDIRS> }
I simply use this in the maildroprc:
# Auto-Create new mailbox
if (/^Received:/)
{
test -d $DEFAULT
if ($RETURNCODE == 1)
{
/usr/local/bin/maildirmake $DEFAULT
/usr/local/bin/maildirmake -f Spam $DEFAULT
echo "Spam" >> $DEFAULT/subscriptions
}
}
# Deliver to Inbox or Spam box (create spam box if it does not exist)
if (/^X-Spam-Flag: YES/:h)
{
test -d $DEFAULT/.Spam
if ($RETURNCODE == 1)
{
/usr/local/bin/maildirmake -f Spam $DEFAULT
echo "Spam" >> $DEFAULT/subscriptions
}
exception {
to "$DEFAULT/.Spam"
}
# if all else fails, do regular delivery
exception {
to "$DEFAULT"
}
}
else
{
exception {
to "$DEFAULT"
}
}
Maildrop automatically creates the maildir for new accounts if a valid e-mail address is found and the account has not already been created when the first message arrives for the new user.
It also tests that the spam folder exists whenever a message is received that contains the spam x-header, so that the folder can be created prior to the spam message delivery (this test is very fast and does not seem to hinder performance since maildrop written in C++ and is highly optimized to support these kinds of functions).
Bill
Bill Landry wrote:
Fábio M. Catunda wrote:
Charles Marcus escreveu:
On 9/26/2007, Bill Landry (bill@inetmsg.com) wrote:
But with maildrop you still run the process of creating/checking on every delivery, same thing as dovecot, right?
Correct. If the folder does not exist, maildrop will create it on first delivery. If the end user deletes the folder, maildrop will recreate the folder on the next message delivery to that account.
Could it could be written to simply create the folder if the target folder doesn't exist (ie, if the initial save fails due to non-existent folder)? Then there would be no performance hit...
There will be a performance hit couse you have to check if the folder exists or not. With maildrop its pretty easy to do that, but you will have an extra access to your HD on every message delivery. Maildrop can run shell commands it the user that runs it have a valid shell (Debian-exim do NOT have a valid shell, I created another user just to make the delivery), so, you can run something like
/^Envelope-to:.*/ getaddr($MATCH) =~ /^.*/; DEST = $MATCH USER =
/bin/echo $DEST | /usr/bin/cut -f1 -d'@'DOMAIN =/bin/echo $DEST | /usr/bin/cut -f2 -d'@'DOEXIST=
[ -d /var/mail/$DOMAIN/$USER/Maildir ]; echo $?if ($DOEXIST == 1){ <RUN SOMETHING HERE TO CREATE MAILDIRS> }
I simply use this in the maildroprc:
# Auto-Create new mailbox if (/^Received:/) {
test -d $DEFAULTif ($RETURNCODE == 1) {/usr/local/bin/maildirmake $DEFAULT/usr/local/bin/maildirmake -f Spam $DEFAULTecho "Spam" >> $DEFAULT/subscriptions} }# Deliver to Inbox or Spam box (create spam box if it does not exist) if (/^X-Spam-Flag: YES/:h) {
test -d $DEFAULT/.Spamif ($RETURNCODE == 1) {/usr/local/bin/maildirmake -f Spam $DEFAULTecho "Spam" >> $DEFAULT/subscriptions} exception { to "$DEFAULT/.Spam" } # if all else fails, do regular delivery exception { to "$DEFAULT" } } else { exception { to "$DEFAULT" } }Maildrop automatically creates the maildir for new accounts if a valid e-mail address is found and the account has not already been created when the first message arrives for the new user.
It also tests that the spam folder exists whenever a message is received that contains the spam x-header, so that the folder can be created prior to the spam message delivery (this test is very fast and does not seem to hinder performance since maildrop written in C++ and is highly optimized to support these kinds of functions).
Bill
I didnt know that maildrop generates a variable called RETURNCODE when you run some external commands, thanks for the advise!
Anyway, even if maildrop is written in C++, it have to start a bash and run the external command. It's realy fast, but if you take a heavy loaded system it might put you in a lot problems!
There are some tools to test a mailsystem, it generates a lot of messages and keep counting the mailqueue, in such situation you willl see hundreds of bashs started running what you have in maildroprc.
The case here is not courier-imap, but in this case maildirmake it not enought to create a user maildir, you have to install the quota file too, so, maildrop might do the wrong job in such situation.
I'm new with dovecot, still dont know how to use quota, but if you need to read some aditional information from some db at the time of create the maildir, you probably will have troubles with maildrop too!
I realy prefer to create mailboxes at the time of the account creation.
[]'s, FMC!
!DSPAM:46fad644211981503319954!
Fábio M. Catunda wrote the following on 9/26/2007 3:01 PM -0800:
I didnt know that maildrop generates a variable called RETURNCODE when you run some external commands, thanks for the advise!
Anyway, even if maildrop is written in C++, it have to start a bash and run the external command. It's realy fast, but if you take a heavy loaded system it might put you in a lot problems!
With maildrop we are talking milliseconds in comparison to the time it takes to do e-mail virus scanning and spam filtering. From monitoring the Postfix list, there appear to be some fairly large sites using maildrop, and I never hear of any complaints with it.
There are some tools to test a mailsystem, it generates a lot of messages and keep counting the mailqueue, in such situation you willl see hundreds of bashs started running what you have in maildroprc.
The case here is not courier-imap, but in this case maildirmake it not enought to create a user maildir, you have to install the quota file too, so, maildrop might do the wrong job in such situation.
Maildrop can probably be configure to install the quota file. Maildrop also supports quota:
-w N
The -w N option places a warning message into the maildir if the
maildir has a quota setting, and after the message wassuccessfully delivered the maildir was at least N percent full. The warning message is copied verbatim from /usr/local/etc/quotawarnmsg with the addition of the "Date:" and "Message-Id:" headers. The warning is repeated every 24 hours (at least), until the maildir drops below N percent full.
I'm new with dovecot, still dont know how to use quota, but if you need to read some aditional information from some db at the time of create the maildir, you probably will have troubles with maildrop too!
I realy prefer to create mailboxes at the time of the account creation.
That's fine, you can always add maildirmake to some script that creates the accounts and the mailboxes at the same time.
Bill
On 9/26/2007, Fábio M. Catunda (catunda@contactnet.com.br) wrote:
Could it could be written to simply create the folder if the target
folder doesn't exist (ie, if the initial save fails due to non-existent folder)? Then there would be no performance hit...
There will be a performance hit couse you have to check if the folder exists or not.
This is not good... but not what I wrote either.
I'm not a programmer, so this may not be possible, but...
What I wrote was, do NOT test BEFORE delivery is attempted - only create the folder IF delivery attempt FAILS *DUE* to non-existent folder.
Ie:
Deliver to user+foo Error: Folder 'foo' doesn't exist for user Ok, Do: Create folder 'foo' for user ReDo: Deliver to user+foo
Maybe this is not reliable - and if not, definitely shouldn't be done. But if there is a way to catch the error 'destination folder doesn't exist', then it would NOT be a performance hit, because no test is performed, only an error handled differently.
Again, this may not be doable, if not, sorry for the noise...
--
Best regards,
Charles
At 4:49 PM -0400 9/26/07, Charles Marcus imposed structure on a stream of electrons, yielding:
On 9/26/2007, Fábio M. Catunda (catunda@contactnet.com.br) wrote:
Could it could be written to simply create the folder if the target folder doesn't exist (ie, if the initial save fails due to non-existent folder)? Then there would be no performance hit...
There will be a performance hit couse you have to check if the folder exists or not.
This is not good... but not what I wrote either.
I'm not a programmer, so this may not be possible, but...
What I wrote was, do NOT test BEFORE delivery is attempted - only create the folder IF delivery attempt FAILS *DUE* to non-existent folder.
Ie:
Deliver to user+foo Error: Folder 'foo' doesn't exist for user Ok, Do: Create folder 'foo' for user ReDo: Deliver to user+foo
Maybe this is not reliable - and if not, definitely shouldn't be done. But if there is a way to catch the error 'destination folder doesn't exist', then it would NOT be a performance hit, because no test is performed, only an error handled differently.
I don't think there would be much real benefit on most OS's and filesystems given what is going to actually need to happen underneath the surface logic. It is simpler in English, and maybe in C, but the difference in what actually gets done is trivial, because whether the code says 'check for existence' or 'use this' the OS is still going to have to do roughly the same work.
More significantly, I think such a feature as you describe would have to be used with great caution. I certainly don't want to find that some spammer has created a bazillion new IMAP folders by sending to random tags.
--
Bill Cole
bill@scconsult.com
On Wed, 2007-09-26 at 16:49 -0400, Charles Marcus wrote:
What I wrote was, do NOT test BEFORE delivery is attempted - only create the folder IF delivery attempt FAILS *DUE* to non-existent folder.
This is how Dovecot's deliver works. Or v1.0 deliver tries to create it if mailbox opening fails for any reason (won't cause problems if mailbox does already exist). v1.1 has better error handling and tries to create the mailbox only if the mailbox doesn't exist.
Fábio M. Catunda wrote:
Charles Marcus escreveu:
On 9/26/2007, Bill Landry (bill@inetmsg.com) wrote:
But with maildrop you still run the process of creating/checking on every delivery, same thing as dovecot, right?
Correct. If the folder does not exist, maildrop will create it on first delivery. If the end user deletes the folder, maildrop will recreate the folder on the next message delivery to that account.
Could it could be written to simply create the folder if the target folder doesn't exist (ie, if the initial save fails due to non-existent folder)? Then there would be no performance hit...
There will be a performance hit couse you have to check if the folder exists or not.
Charles did not say you need to check if the folder exists or not.
With maildrop its pretty easy to do that, but you will have an extra access to your HD on every message delivery. Maildrop can run shell commands it the user that runs it have a valid shell (Debian-exim do NOT have a valid shell, I created another user just to make the delivery), so, you can run something like
/^Envelope-to:.*/ getaddr($MATCH) =~ /^.*/; DEST = $MATCH USER =
/bin/echo $DEST | /usr/bin/cut -f1 -d'@'DOMAIN =/bin/echo $DEST | /usr/bin/cut -f2 -d'@'DOEXIST=
[ -d /var/mail/$DOMAIN/$USER/Maildir ]; echo $?if ($DOEXIST == 1){ <RUN SOMETHING HERE TO CREATE MAILDIRS> }
Thats it!
There's a pattern here. instead of
if (file|dir) (exists|is writable) ... write to ...use
write to ...
if (error) handle itThis has multiple benefits:
- optimized for the common case
- less vulnerable to race conditions (things happen between the time you check and the time you do).
- if the file/dir exists/writable, no check to do.
with maildrop, this is done with exceptions (otherwise trying to deliver would result in an error reported to the MTA).
# try writing
exception {
TO $folder
}
# failed. maybe folder doesn't exist. try creating it
# add whatever checks or actins inside this...
$maildirmake $folder
# try writing again
TO $folder
you can adapat this to delivering spam to the Inbox if the user has deleted his Junk folder (a method to opt-out).
mouss wrote the following on 9/30/2007 3:45 PM -0800:
Fábio M. Catunda wrote:
Charles Marcus escreveu:
On 9/26/2007, Bill Landry (bill@inetmsg.com) wrote:
But with maildrop you still run the process of creating/checking on every delivery, same thing as dovecot, right?
Correct. If the folder does not exist, maildrop will create it on first delivery. If the end user deletes the folder, maildrop will recreate the folder on the next message delivery to that account.
Could it could be written to simply create the folder if the target folder doesn't exist (ie, if the initial save fails due to non-existent folder)? Then there would be no performance hit...
There will be a performance hit couse you have to check if the folder exists or not.
Charles did not say you need to check if the folder exists or not.
With maildrop its pretty easy to do that, but you will have an extra access to your HD on every message delivery. Maildrop can run shell commands it the user that runs it have a valid shell (Debian-exim do NOT have a valid shell, I created another user just to make the delivery), so, you can run something like
/^Envelope-to:.*/ getaddr($MATCH) =~ /^.*/; DEST = $MATCH USER =
/bin/echo $DEST | /usr/bin/cut -f1 -d'@'DOMAIN =/bin/echo $DEST | /usr/bin/cut -f2 -d'@'DOEXIST=
[ -d /var/mail/$DOMAIN/$USER/Maildir ]; echo $?if ($DOEXIST == 1){ <RUN SOMETHING HERE TO CREATE MAILDIRS> }
Thats it!
There's a pattern here. instead of
if (file|dir) (exists|is writable) ... write to ...
use
write to ... if (error) handle it
This has multiple benefits:
- optimized for the common case
- less vulnerable to race conditions (things happen between the time you check and the time you do).
- if the file/dir exists/writable, no check to do.
with maildrop, this is done with exceptions (otherwise trying to deliver would result in an error reported to the MTA).
# try writing exception { TO $folder } # failed. maybe folder doesn't exist. try creating it # add whatever checks or actins inside this...
$maildirmake $folder# try writing again TO $folderyou can adapat this to delivering spam to the Inbox if the user has deleted his Junk folder (a method to opt-out).
Mouss, your post got me thinking, and after a bit of struggle and testing, I was finally able to come up with a working maildroprc configuration that would first attempt message delivery and then fall-back to creating the maildir as an exception if the maildir did not exist. The issue I was having was that maildrop was creating mbox files if the maildir directory structure was not already existing before sending a message to a new account. I finally found a hint in an e-mail exchange on the Courier maildrop list:
http://osdir.com/ml/mail.maildrop/2004-10/msg00031.html
The trick was to terminate the path with a forward slash "/". I know this trailing "/" is a maildir standard, but even a follow up review of the maildrop "maildropfilter" documentation didn't reveal this little tidbit. There was just one reference to maildir format vs. "file" (mbox) format in the official maildrop documentation:
http://www.courier-mta.org/maildrop/maildropfilter.html "If /|expression|/ is a directory, *maildrop* assumes that the directory is a maildir directory. Otherwise, *maildrop* will deliver the message to a file, formatted in traditional mailbox format. *maildrop* will use either dot-locking, or flock()-locking when delivering the message to the file."
However, is the maildir directory structure does not exist, then maildrop creates an mbox file. With the trailing forward slash, maildrop will then assume maildir, and if the directory does not exist, the error condition will trigger and the exception to create the maildir will be followed.
Hope this help someone else who has been trying to figure this out. And thanks for the hint, Mouss!
Bill
Bill Landry wrote:
mouss wrote the following on 9/30/2007 3:45 PM -0800:
Fábio M. Catunda wrote:
Charles Marcus escreveu:
On 9/26/2007, Bill Landry (bill@inetmsg.com) wrote:
But with maildrop you still run the process of creating/checking on every delivery, same thing as dovecot, right?
Correct. If the folder does not exist, maildrop will create it on first delivery. If the end user deletes the folder, maildrop will recreate the folder on the next message delivery to that account.
Could it could be written to simply create the folder if the target folder doesn't exist (ie, if the initial save fails due to non-existent folder)? Then there would be no performance hit...
There will be a performance hit couse you have to check if the folder exists or not.
Charles did not say you need to check if the folder exists or not.
With maildrop its pretty easy to do that, but you will have an extra access to your HD on every message delivery. Maildrop can run shell commands it the user that runs it have a valid shell (Debian-exim do NOT have a valid shell, I created another user just to make the delivery), so, you can run something like
/^Envelope-to:.*/ getaddr($MATCH) =~ /^.*/; DEST = $MATCH USER =
/bin/echo $DEST | /usr/bin/cut -f1 -d'@'DOMAIN =/bin/echo $DEST | /usr/bin/cut -f2 -d'@'DOEXIST=
[ -d /var/mail/$DOMAIN/$USER/Maildir ]; echo $?if ($DOEXIST == 1){ <RUN SOMETHING HERE TO CREATE MAILDIRS> }
Thats it!
There's a pattern here. instead of
if (file|dir) (exists|is writable) ... write to ...
use
write to ... if (error) handle it
This has multiple benefits:
- optimized for the common case
- less vulnerable to race conditions (things happen between the time you check and the time you do).
- if the file/dir exists/writable, no check to do.
with maildrop, this is done with exceptions (otherwise trying to deliver would result in an error reported to the MTA).
# try writing exception { TO $folder } # failed. maybe folder doesn't exist. try creating it # add whatever checks or actins inside this...
$maildirmake $folder# try writing again TO $folderyou can adapat this to delivering spam to the Inbox if the user has deleted his Junk folder (a method to opt-out).
Mouss, your post got me thinking, and after a bit of struggle and testing, I was finally able to come up with a working maildroprc configuration that would first attempt message delivery and then fall-back to creating the maildir as an exception if the maildir did not exist.
you could have just asked. I didn't want to post mine since this is not a maildrop list ;-p
[snip]
mouss wrote:
Mouss, your post got me thinking, and after a bit of struggle and testing, I was finally able to come up with a working maildroprc configuration that would first attempt message delivery and then fall-back to creating the maildir as an exception if the maildir did not exist.
you could have just asked. I didn't want to post mine since this is not a maildrop list ;-p
Yes, of course, but then I would have missed out on all of the fun and frustration of trying to figure this out... ;-)
Actually, it was a good exercise, as I had not reviewed my maildroprc configuration file since I originally created it almost 1.5 years ago, and learned some new stuff in the process.
Bill
participants (6)
-
"Fábio M. Catunda"
-
Bill Cole
-
Bill Landry
-
Charles Marcus
-
mouss
-
Timo Sirainen