Hello everybody,
I am in the process to install dovecot as an IMAP server.
Currently, this system runs postfix and uses procmail for local (mbox style) delivery.
Procmail ist started by postfix like this:
/etc/postfix/main.cf:
mailbox_command = /usr/bin/procmail
~/.forward:
"|IFS=' ' && exec /usr/bin/procmail -f- || exit 75 # jw"Before going into IMAP details, I decided to convert the system to Maildir as a first step. So I created a procmail-entryrecipe to populate a test folder:
~/.procmailrc:
:0
* ^Subject:.*dovecot-test
{
ORGMAIL="${HOME}/Maildir/"
DEFAULT=${ORGMAIL}
DELIVER="/usr/lib/dovecot/deliver -o mail_location=maildir:~/Maildir -o lda_mailbox_autocreate=yes"
:0
| $DELIVER -m dovecot-test
}This setup seems to work.
But there is one point that bugs me: in the first tests, dovecot was mis-configured (the two -o options were missing) and the test mails were silently lost. No errors in the procmail logfile. Only dovecot reports errors in its logfile. But the error seems not to be noticed by procmail. Therefore procmail assumes deliver could store the mail successfully and won't keep a backup of it.
I am sure, I'm doing something stupid here. I have never seen such behavior from procmail before. Normally procmail would report failure of sub-commands.
Now I hesitate to go forward in this conversion process. I don't want to loose mails if something is wrong with the dovcot setup.
Besides: I wonder whether the setting
DEFAULT=${ORGMAIL}is a good idea. Wouldn't it be better to have
DEFAULT=/var/spool/mail/$USERso there would be a fallback in case there are delivery problems to ${ORGMAIL}
Any thougths?
-- Josef Wolf jw@raven.inka.de
Hi Josef,
Just went through this one recently using CentOS with SendMail, Procmail and IMAP conversion from mbox to Maildir - I needed to make sure I could run both formats while I migrated users one-by-one.
Not sure if what I did helps, but I created a 'how-to' page on it on my website rather than post an essay on the e-mail forum:
http://dmc1961.id.au/howto_17101101.php
In essence I am using /etc/procmailrc and the users_db approach to get e-mail delivered correctly after I have converted an account to Maildir format. I also never trust anything in my IT life, so I backup the mail directories under $HOME to mail_old.
Not sure if this is any use to you but works for me now.
On 13/10/17 07:52, Josef Wolf wrote:
Hello everybody,
I am in the process to install dovecot as an IMAP server.
Currently, this system runs postfix and uses procmail for local (mbox style) delivery.
Procmail ist started by postfix like this:
/etc/postfix/main.cf: mailbox_command = /usr/bin/procmail ~/.forward: "|IFS=' ' && exec /usr/bin/procmail -f- || exit 75 # jw"Before going into IMAP details, I decided to convert the system to Maildir as a first step. So I created a procmail-entryrecipe to populate a test folder:
~/.procmailrc: :0 * ^Subject:.*dovecot-test { ORGMAIL="${HOME}/Maildir/" DEFAULT=${ORGMAIL} DELIVER="/usr/lib/dovecot/deliver -o mail_location=maildir:~/Maildir -o lda_mailbox_autocreate=yes" :0 | $DELIVER -m dovecot-test }This setup seems to work.
But there is one point that bugs me: in the first tests, dovecot was mis-configured (the two -o options were missing) and the test mails were silently lost. No errors in the procmail logfile. Only dovecot reports errors in its logfile. But the error seems not to be noticed by procmail. Therefore procmail assumes deliver could store the mail successfully and won't keep a backup of it.
I am sure, I'm doing something stupid here. I have never seen such behavior from procmail before. Normally procmail would report failure of sub-commands.
Now I hesitate to go forward in this conversion process. I don't want to loose mails if something is wrong with the dovcot setup.
Besides: I wonder whether the setting
DEFAULT=${ORGMAIL}is a good idea. Wouldn't it be better to have
DEFAULT=/var/spool/mail/$USERso there would be a fallback in case there are delivery problems to ${ORGMAIL}
Any thougths?
--
As always, I remain at your service.
Kindest Regards, David.M.Clark (Director - Senior Linux/UNIX Consultant)
Davrom Consulting Pty Ltd Mobile: 0418763124 PO Box 1644, Sunnybank Hills, 4109 E-mail (Work): david@davrom.com ABN: 81 096 990 804 E-mail (Priv): dmc1961@dmc1961.id.au Website: http://davrom.com Skype: dmc1961 Podcast: http://ldup.com.au Google: dmc1961@gmail.com
Specialising in: Linux (Fedora/RedHat/CentOS), UNIX, SCO, MikroTik, Networking/Internet, E-mail/Web Technologies
Please note: Any e-mail communication bearing this signature is for the exclusive purpose of the sender and is not for publication without the expressed permission of the sender or respective sender's organisation.
On Fri, Oct 13, 2017 at 11:34:22AM +1000, David.M.Clark wrote:
Just went through this one recently using CentOS with SendMail, Procmail and IMAP conversion from mbox to Maildir - I needed to make sure I could run both formats while I migrated users one-by-one.
Thanks for your suggestion, David!
My approach is similar. But I want to migrate not only users one-by-one but also their mailboxes one-by-one.
I'm going the route that procmail will deliver to mbox folders directly. So dovecot-lda won't even see mails for mbox folders.
But that's not the point that bothers me.
The problem is that in case of problems (e.g. broken/bad dovecot configuration) mail is silently dropped without passing back the error indication back to procmail. Therefore procmail assumes the delivery was successful and won't bother to store it into some safe place. This effectively results in lost mail.
-- Josef Wolf jw@raven.inka.de
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
Greetings,
On Fri, 2017-10-13 at 11:34 +1000, David.M.Clark wrote:
Just went through this one recently using CentOS with SendMail, Procmail and IMAP conversion from mbox to Maildir - I needed to make sure I could run both formats while I migrated users one-by-one.
When I have migrated solutions from mbox to maildir (I have done more than one) I found it easier to get a mail client to do the hard work for me (if you're on a LAN with the server it's faster). I created new usernames with maildir as their base, and copied the folder over. This approach could be automated, probably easier than going via a script to open each mbox and push each message through an LDA. There are a heap of examples around for connecting to IMAP via PERL (https://www.example-code.com/perl/imap.asp is good). If you write your perl script properly you could migrate multiple users at once from the cli ... xargs is a wonderful tool :). Unless you are hosting a mail service that needs to be up 24/7 you should be able to migrate large amounts of mail out of hours. If you have users that have been using POP to get their email and want to move them to IMAP it's also easier to have the MUA do the work (I have migrated users from using POP and local storage, to remote IMAP this way - especially users of MS Outlook).
In essence I am using /etc/procmailrc and the users_db approach to get e-mail delivered correctly after I have converted an account to Maildir format. I also never trust anything in my IT life, so I backup the mail directories under $HOME to mail_old.
Yeah, safe bet there. Creating copies before starting the migration process is the best way (depending on the underlying filesystem also probably faster if you need to restore something). Even using a second copy as the base to push things from mbox into a new maildir.
While I like procmail (it has some nice features), I also migrated things to sieve a while back. There are a couple of things I can do with procmail that I have found it hard to match with sieve, but properly nested if statements mean I can traverse my rules faster. Most people will end up using MUA filters on an inbox rather than a procmail/sieve setup (the number of MUA's that support either of these server side filters is minimal, and the number that support them properly is even smaller - I did have a webmail client destroy my nested "if" statements). Using procmail or sieve as an intermediate step for initial transition from mbox to maildir is a safe bet, then it comes down to having dovecot setup properly to deliver into maildir.
Nikolai Lusan nikolai@lusan.id.au -----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEVfd4GW6z4nsBxdLo4ZaDRV2VL6QFAlngpycACgkQ4ZaDRV2V L6SDsxAAn65oBEhOb4HxOzL5oAg+GREziMEcav0iOkXwjkGi6ithX3xc5T9vgA36 dJWDgkTdTVN82vaZKu3Eg7V8baqld5l6alFRJv5nhU3G37m287SapxCG3smwT8OA 2y3SAkPJO905cOcINYGESIMLtBtRSympDcVtM8+gHjy7+xSWiiaxEt3wSojbmrAT NqunbuBCrDQXK5GeWYBcOPTHj+7Qb3CEGCEo3mTDtgIBsNRpw07PIc+px0KbRJT2 4h5Gft1IzHUC9b63yRXUK7O00SAxxjAQ2fJAmjq5kBxPuwmW70pbcQMPrTQd4HAL X8QtzoqoLPCLHu3Ptx5eRLpw3IUSiHWFeBZGG/5RWx/GPyzKFMMFmSURKjxmVbkG imo+2vR8LgBFsE1KXc7cRPjkHaTJNus9N7LvWepR/uSDthXJGWpYttCilAm9wmRG usflda8Qqq+AJxiUOYP4n+Qu130yNaW0K5T4LJRVJ1DHiQcRRrk6D+r8wpmG6nBi jrJR7/ge2mk+axl1PBJklAycTxdjEFwdXA3/NmBA0nN1NLxMMdVMneipGuAYQVP7 JJA6mMySMahj1c9DSDj8O8x3cbc2uSXSB7AzKogRnnWeKewh1DoEIeVSiMwJysvS 0lNEWGIzE/cXG72JQniDGrKLZZN5yJEuKZPiSmEbJB/rYOyGpGE= =JJgP -----END PGP SIGNATURE-----
Zitat von Josef Wolf jw@raven.inka.de:
Any thougths?
Deliver IMHO should be run directly by the MTA, if possible, because
being run by procmail you loose the capability of using SIEVE.
The dovecot way to convert mailbox formats is the dsync (doveadm sync)
command, which is the Swiss army knife for this kind of purpose.
And if you are in need of converting a bunch of inboxes, one after
another, keep in mind that we got the mail_location = auto: command,
where dovecot decides on runtime by the name of the underlyding
subfolder about the mail format to be used.
So for example, dovecot can distinguish between ~/Maildir, ~/Mail,
~/mdbox and others. This gives you the tools at hand to migrate
literally tens of thousands of mailboxes, while your services are
running normally, one after another.
On Sat, Oct 14, 2017 at 03:56:17PM +0200, Marc Stürmer wrote:
Zitat von Josef Wolf jw@raven.inka.de:
Deliver IMHO should be run directly by the MTA, if possible, because being run by procmail you loose the capability of using SIEVE.
I have not looked into SIEVE yet. From some ramblings on the mailing list, SIEVE seems not to be as powerful as procmail?
And if you are in need of converting a bunch of inboxes, one after another, keep in mind that we got the mail_location = auto: command, where dovecot decides on runtime by the name of the underlyding subfolder about the mail format to be used.
So for example, dovecot can distinguish between ~/Maildir, ~/Mail, ~/mdbox and others. This gives you the tools at hand to migrate literally tens of thousands of mailboxes, while your services are running normally, one after another.
My problem is NOT the conversion process (yet). For conversion, I have created procmail recipes like this:
:0
- ^TOgimp-user
{
:0:
- ? test -f $HOME/Mail/ML.gimp-user ML.gimp-user :0 { MAILDIR=$HOME/Maildir/ DEFAULT=$HOME/Maildir/ :0 | $DELIVER -m ML.gimp-user } }
With this, procmail will file into mbox when an mbox folder exists, and into Maildir if no mbox folder exists. That way I can simply remove mbox files one by one. No changes to any config files are needed when folders are converted.
But as I stated before, my REAL concern is that mail is lost when dovecot fails to store the mail. Reporting the failure from dovecot-lda back to procmail doesn't work properly. Any thougths on THIS topic?
-- Josef Wolf jw@raven.inka.de
participants (4)
-
David.M.Clark
-
Josef Wolf
-
Marc Stürmer
-
Nikolai Lusan