[Dovecot] conversion to maildir
is really a royal pain in the butt. most of the conversion tools only do a partial job. mb2md seems to be the best but it does not convert the from line correctly unless you are using exim for an MTA. It also does not get the file status right so everything is written to the new maildir directory. something else that makes the process painful is when you have folders containing mailboxes. And the process is *slow* when you have 2 GB of mail (yes, I will be deleting some soon).
looks like I will be debugging and rewrapping this package real soon now..
to relate this to dovecot: this is probably not the job of the imap server but it would be real nice to be able to fit in some sort of a plug-in that could automatically do the conversion on a mailbox as it is opened.
it would make migrations trivial easy. give me the ability to use python for plug-ins and I will write such a tool.
---eric
On 26 August 2003 at 13:16, "Eric S. Johansson" esj@harvee.org wrote:
to relate this to dovecot: this is probably not the job of the imap server but it would be real nice to be able to fit in some sort of a plug-in that could automatically do the conversion on a mailbox as it is opened.
it would make migrations trivial easy. give me the ability to use python for plug-ins and I will write such a tool.
I wouldn't mind seeing such a tool also :)
On Tue, 2003-08-26 at 20:16, Eric S. Johansson wrote:
is really a royal pain in the butt. most of the conversion tools only do a partial job. mb2md seems to be the best but it does not convert the from line correctly unless you are using exim for an MTA. It also does not get the file status right so everything is written to the new maildir directory. something else that makes the process painful is when you have folders containing mailboxes. And the process is *slow* when you have 2 GB of mail (yes, I will be deleting some soon).
Next Dovecot version will support both mboxes and maildir. That could do it at least one mailbox at a time.
to relate this to dovecot: this is probably not the job of the imap server but it would be real nice to be able to fit in some sort of a plug-in that could automatically do the conversion on a mailbox as it is opened.
Yes, that should be possible.
it would make migrations trivial easy. give me the ability to use python for plug-ins and I will write such a tool.
We'd need a Python plugin :) I don't think it was difficult to export C stuff from .h files to Python using some tool .. which I can't remember anymore.
Timo Sirainen explained:
Next Dovecot version will support both mboxes and maildir. That could do it at least one mailbox at a time.
I wonder if we're both envisioning the same thing? I'm envisioning a sequence where the maildir directory is looked for first and failing that, trying for a mbox one. for example, if you have a mailbox called plugh it would look for a maildir directory called plugh and if it did not exist, look for a mbox file called plugh. Granted it would make the subscription to semantics a little funky but not unmanageable.
then, if the mbox file was the only mailbox, its contents would be converted to maildir and the mbox file would be moved to a "junk" directory
to relate this to dovecot: this is probably not the job of the imap server but it would be real nice to be able to fit in some sort of a plug-in that could automatically do the conversion on a mailbox as it is opened.
Yes, that should be possible.
in addition to my ramblings above, seems to me that might be more practical to specify a "magic" directory addition to the imap directory representation which contains the hierarchy of mbox files which would allow users to copy and then delete mboxes to convert formats.
I personally prefer the automatic invisible conversion if at all possible because it makes the customer experience a dream (if it works right).
We'd need a Python plugin :) I don't think it was difficult to export C stuff from .h files to Python using some tool .. which I can't remember anymore.
but it's not all that you need. You want to embed python in dovecot and find a way to invoke dovecot functions mailbox from python.
---eric
On Tue, 2003-08-26 at 22:55, Eric S. Johansson wrote:
Next Dovecot version will support both mboxes and maildir. That could do it at least one mailbox at a time.
I wonder if we're both envisioning the same thing?
I think you were talking about two things - mbox2maildir script first and then the automatic thingy.
in addition to my ramblings above, seems to me that might be more practical to specify a "magic" directory addition to the imap directory representation which contains the hierarchy of mbox files which would allow users to copy and then delete mboxes to convert formats.
That's what I meant with being possible with next version:
namespace private { separator = / prefix = #mbox/ location = mbox:~/mail }
We'd need a Python plugin :) I don't think it was difficult to export C stuff from .h files to Python using some tool .. which I can't remember anymore.
but it's not all that you need. You want to embed python in dovecot and find a way to invoke dovecot functions mailbox from python.
I was thinking about SWIG. http://www.swig.org/
You'd mostly just need to convert src/lib-storage/mail-storage.h, imap/client.h and imap/commands-util.h. Maybe some of lib-imap/*.h and lib-mail/*.h
On Wed, 2003-08-27 at 04:12, Timo Sirainen wrote:
We'd need a Python plugin :) I don't think it was difficult to export C stuff from .h files to Python using some tool .. which I can't remember anymore.
but it's not all that you need. You want to embed python in dovecot and find a way to invoke dovecot functions mailbox from python.
I was thinking about SWIG. http://www.swig.org/
You'd mostly just need to convert src/lib-storage/mail-storage.h, imap/client.h and imap/commands-util.h. Maybe some of lib-imap/*.h and lib-mail/*.h
Do dovecot plugins run as seperate processes and communicate with some sort of IPC, or do they run within the same process as the main server? If they run within a plugin would need to embed the actual interpreter.
By the way, I converted over UWA's student email server to dovecot from UW-IMAP and cucipop last night. Relatively few problems, which I've now fixed with some tweaks to the config. The only outstanding problem is these assertions I keep getting in the logs: Aug 28 17:03:51 tartarus imap(mpienaar): file mbox-rewrite.c: line 429 (mbox_write_header): assertion failed: (hdr_parsed_size.physical_size == hdr_size)
Does the assertion indicate a corrupted mailbox file? I can open that user's mailbox fine with mutt. It's dovecot 0.99.10-5 from Debian.
Load has dropped a lot on the server (we have 20k accounts), so thanks for a useful program :)
Cheers Grahame
-- Grahame Bowland Email: grahame@ucs.uwa.edu.au University Communications Services Phone: +61 8 9380 1175 The University of Western Australia Fax: +61 8 9380 1109 CRICOS: 00126G
On Thu, 2003-08-28 at 12:26, Grahame Bowland wrote:
Do dovecot plugins run as seperate processes and communicate with some sort of IPC, or do they run within the same process as the main server? If they run within a plugin would need to embed the actual interpreter.
They run within the server. I don't think there's much point in separating them into other processes. So yes, you'd need embedded interpreter but I don't think that's much of a problem.
By the way, I converted over UWA's student email server to dovecot from UW-IMAP and cucipop last night. Relatively few problems, which I've now fixed with some tweaks to the config.
Nice :)
The only outstanding problem is these assertions I keep getting in the logs: Aug 28 17:03:51 tartarus imap(mpienaar): file mbox-rewrite.c: line 429 (mbox_write_header): assertion failed: (hdr_parsed_size.physical_size == hdr_size)
Does the assertion indicate a corrupted mailbox file? I can open that user's mailbox fine with mutt. It's dovecot 0.99.10-5 from Debian.
I heard about similiar problem before too.. I couldn't really figure out what it was that caused the problem. I've never seen it myself and I've been using mboxes all the time.
What that error message means is that message header size is different when rewriting mbox than what it was at last synchronization. But since the syncing is done just before the rewriting, that should never happen. Except if there's a bug somewhere..
If it's possible I would like to see that mbox file. Actually it could be enough if you just hide the actual content in it. Here's a pretty perl script for that:
perl -e 'while (<>) { if (/^From ([^ ]*) *(.*)$/) { $hdr = 1; print "From foo\@bar.org $2\n"; } elsif ($hdr) { if (/^$/) { $hdr = 0; print "\n"; } elsif (/^Date:/i || /^Status:/i || /^X-Status:/i || /^X-Keywords:/i || /^X-UID:/i || /^X-IMAPbase:/i || /^Content-Length/i) { print $_; } else { s/^([^:]+:) (.*)/("$1 "."X"x length($2))/e; print $_; } } else { s/./B/g; print $_; } }' < inbox
On Thu, Aug 28, 2003 at 17:26:17, Grahame Bowland wrote:
Hi Grahame,
By the way, I converted over UWA's student email server to dovecot from UW-IMAP and cucipop last night. Relatively few problems, which I've now fixed with some tweaks to the config. The only outstanding problem is these assertions I keep getting in the logs: Aug 28 17:03:51 tartarus imap(mpienaar): file mbox-rewrite.c: line 429 (mbox_write_header): assertion failed: (hdr_parsed_size.physical_size == hdr_size)
We are also considering leaving UW-IMAP in favour of dovecot, I would be very gratefull if you would care to share your tweaks to the config with us?
Best wishes Lars
Lars Anderson mailto:lsa@business.auc.dk Department of Business Studies http://www.business.auc.dk/~lsa/ Aalborg University Voice: +45 96358225, Fax: +45 98153505 Denmark Office: Fib4-117
On Sat, Aug 30, 2003 at 01:33:28PM -0400, seth vidal wrote:
We are also considering leaving UW-IMAP in favour of dovecot, I would be very gratefull if you would care to share your tweaks to the config with us?
At the risk of sounding too ridiculous:
ditto, I'd like to see those tweaks too.
yea, me too ;) there shouldn't be much tweaks needed really. Just set the default_mail_env and mbox_locks correctly.
If you want to have backwards compatibility for clients so that their ~/mail and ~user/mail prefixes still work, set full_filesystem_access = yes.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Am Dienstag, 26. August 2003 21:55 schrieb Eric S. Johansson:
then, if the mbox file was the only mailbox, its contents would be converted to maildir and the mbox file would be moved to a "junk" directory
Why don't you convert your mbox files using formail and procmail?
Greetings,
Gunter
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Dummheit ist die seltsamste aller Krankheiten. Der Kranke leidet niemals unter ihr. Aber die anderen. -- Paul-Henri Spaak +-+-+-+-+-+-+-+-+-+-+-+-+ http://www.lspace.org +-+-+-+-+-+-+-+-+-+-+-+-+ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux)PGP-verschlüsselte Mails bevorzugt! +
iD8DBQE/S8AX0ORHvREo8l8RApZuAJ91tUsxWd6bLW67Q3NsuTDbzfMlqgCgi98I deJuqLCTcnXX0DSFaUm+zho= =caTN -----END PGP SIGNATURE-----
On Tue, Aug 26, 2003 at 10:16:17PM +0200, Gunter Ohrner wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Am Dienstag, 26. August 2003 21:55 schrieb Eric S. Johansson:
then, if the mbox file was the only mailbox, its contents would be converted to maildir and the mbox file would be moved to a "junk" directory
Why don't you convert your mbox files using formail and procmail?
Because I don't know how. Can you post a script? or guidelines and suggestions?
Bob Hall
Bob Hall wrote:
Why don't you convert your mbox files using formail and procmail?
Because I don't know how. Can you post a script? or guidelines and suggestions?
It's mind-numbingly easy.
As for getting the mail into Procmail, that's where Formail comes in. You tell it to parse the mailbox, spitting out one message at a time, which you then feed back into Procmail which goes by your rules. As it goes, it'll then re-deliver based on your rules.
This example is based directly from the set of examples from Formail's man page:
cat mailbox | formail -ds procmail
With your Procmail recipes, follow your mailbox name with a slant, e.g.:
:0:
- 1^0 ^Return-Path:.*dovecot-bounces@procontrol.fi
- 1^0 ^Return-Path:.*dovecot-bounce@procontrol.fi /home/ianj/maildir/dovecot/
This one is based on a real recipe which on my system does a mbox by month, augmented to deliver to a maildir.
and bingo! Any subsequent mail will be delivered to a maildir. The cool part is that you don't have to do anything special beforehand.
For a main inbox, you'd remove the condition lines, and of course, specify the path to the directory (followed by a slant to tell Procmail that you want maildir delivery).
Though one of the other people on the list mentioned wanting to be able to make maildir the system-wide default mailbox format for users' inboxes, which of course can be overridden in ~/.procmailrc. Any thoughts on this one? I may move my inbox to a maildir format while all my mailing list traffic will be filed in monthly mbox files since I rarely make any writes to those files from my MUA to begin with.
--Ian.
Le 26/08/2003 21:18, Timo Sirainen a écrit :
Next Dovecot version will support both mboxes and maildir. That could do it at least one mailbox at a time.
Do you mean that with the actual stable version (99.10) it is not possible to handle both maildir and mailbox imap folders for the same account ?
-- Nico
On Thu, 2003-08-28 at 13:01, Nicolas STRANSKY wrote:
Next Dovecot version will support both mboxes and maildir. That could do it at least one mailbox at a time.
Do you mean that with the actual stable version (99.10) it is not possible to handle both maildir and mailbox imap folders for the same account ?
Yep. It didn't need that much changes, but I hadn't bothered to do that until now.
On Tue, 26 Aug 2003, Eric S. Johansson wrote:
to relate this to dovecot: this is probably not the job of the imap server
I disagree. An imap server can be expected to copy messages from one folder to another perfectly.
Here's one tool - http://home.arcor.de/armin.diehl/imapcopy/imapcopy.html
- I expect there are others.
-- Charlie
participants (11)
-
Bob Hall
-
Charlie Brady
-
Eric S. Johansson
-
Grahame Bowland
-
Gunter Ohrner
-
Ian R. Justman
-
Lars Anderson
-
Nicolas STRANSKY
-
seth vidal
-
Timo Sirainen
-
William Yodlowsky