[Dovecot] processing Dovecot maildirs with shell scripts
Hello,
I have a postfix server which uses procmail as its delivery agent to deliver incoming messages to several maildirs. For several reasons, not really relevant here, I need to process/refilter/sort again the content of these mailboxes before delivering the messages to the _other_ maildirs which Dovecot sees and serves to local or remote imap users. Basically, I need to write some shell scripts that, from cron jobs, regularly do this:
foreach file in some_incoming_mailbox/new/ do #process the file and/or set some variable according to its content delete the file or mv it to some_other_maildir/new/
where the actual action and value of some_other_maildir is set every time by the "process the file" part of the script.
The part that I still miss is guidelines to do these things safely, that is without losing or corrupting messages or doing anything else that would confuse dovecot.
For example, under which conditions it is sure to move or delete (the Unix way, with mv or rm) a message from mailbox1/new/ to mailbox2/new, when the latter is served via dovecot to remote users?
What is advisable and most efficient? Use locks (how?), only work on file which are at least one minute old? Should I move the messages to tmp instead of new? What else?
Tips or pointers about this are very welcome.
Ciao, Marco
On Mon, Aug 27, 2007 21:21:31 PM +0200, Marcus Rueckert (darix@opensu.se) wrote:
formail
No, sorry.
I just come from a thread in the procmail mailing list whose result is that formail is either unusable or unnecessary in cases like this:
http://www.xray.mpe.mpg.de/mailing-lists/procmail/2007-08/msg00117.html
-- Help *everybody* love Free Standards and Free Software! http://digifreedom.net
On Mon, 2007-08-27 at 21:23 +0200, M. Fioretti wrote:
Hello,
I have a postfix server which uses procmail as its delivery agent to deliver incoming messages to several maildirs. For several reasons, not really relevant here, I need to process/refilter/sort again the content of these mailboxes before delivering the messages to the _other_ maildirs which Dovecot sees and serves to local or remote imap users. Basically, I need to write some shell scripts that, from cron jobs, regularly do this:
foreach file in some_incoming_mailbox/new/ do #process the file and/or set some variable according to its content delete the file or mv it to some_other_maildir/new/
So what happens if Dovecot sees the mail and moves it to cur/ before you had a chance to do that? If you really need to see all the new mails, then you'll need to figure out some way to handle that, such as add a new private flag for all processed messages.
Or you could deliver mails to some private maildir that Dovecot doesn't see, and process the messages from there to maildirs that Dovecot sees.
The part that I still miss is guidelines to do these things safely, that is without losing or corrupting messages or doing anything else that would confuse dovecot.
For example, under which conditions it is sure to move or delete (the Unix way, with mv or rm) a message from mailbox1/new/ to mailbox2/new, when the latter is served via dovecot to remote users?
mv, rm and ln are always safe. Just don't ever write to new mail files in new/ or cur/ directory, because then Dovecot might read partially written files. That means that "cp file new/" isn't safe, but "ln file new/" is.
Also make sure the filename is always unique.
On Tue, Aug 28, 2007 01:03:13 AM +0300, Timo Sirainen (tss@iki.fi) wrote:
Basically, I need to write some shell scripts that, from cron jobs, regularly do this:
foreach file in some_incoming_mailbox/new/ do #process the file and/or set some variable according to its content delete the file or mv it to some_other_maildir/new/
So what happens if Dovecot sees the mail and moves it to cur/ before you had a chance to do that? If you really need to see all the new mails, then you'll need to figure out some way to handle that, such as add a new private flag for all processed messages.
Or you could deliver mails to some private maildir that Dovecot doesn't see, and process the messages from there to maildirs that Dovecot sees.
Yes, this is exactly what I want to do, sorry if it wasn't clear from the beginning. I am not going to _create_ messages, or _remove_ them under Dovecot's nose.
I have procmail delivering to:
aux_maildir_1/ aux_maildir_2/ ... aux_maildir_N/ INBOX
these scripts must only delete or move (not create or rename!) files/messages, with certain criteria and at certain times, from the aux_maildirs only to INBOX and INBOX.one, INBOX.two, INBOX.number
Dovecot only sees the INBOX* folders. Under these constraints I'm safe, am I not? Also, since all the file names are created by procmail (there is nobody else injecting email from the outside) I can count on them being always unique, right?
mv, rm and ln are always safe. Just don't ever write to new mail files in new/ or cur/ directory, because then Dovecot might read partially written files. That means that "cp file new/" isn't safe, but "ln file new/" is.
Er.. sorry, what do you mean exactly here? where is "file" where I link to new? What is the complete, exact sequence of Unix commands to safely go from:
aux_maildir_1/new/some_file
to, say,
INBOX.one/new/some_file
TIA, Marco
-- Help *everybody* love Free Standards and Free Software http://digifreedom.net/
On Thu, 2007-08-30 at 23:52 +0200, M. Fioretti wrote:
Dovecot only sees the INBOX* folders. Under these constraints I'm safe, am I not? Also, since all the file names are created by procmail (there is nobody else injecting email from the outside) I can count on them being always unique, right?
Right.
mv, rm and ln are always safe. Just don't ever write to new mail files in new/ or cur/ directory, because then Dovecot might read partially written files. That means that "cp file new/" isn't safe, but "ln file new/" is.
Er.. sorry, what do you mean exactly here? where is "file" where I link to new?
Whatever file. :)
What is the complete, exact sequence of Unix commands to safely go from:
aux_maildir_1/new/some_file
to, say,
INBOX.one/new/some_file
If you want to move, then it's simply
mv aux_maildir_1/new/some_file INBOX.one/new/some_file
The important thing is that you don't use "cp".
participants (3)
-
M. Fioretti
-
Marcus Rueckert
-
Timo Sirainen