[Dovecot] mbox vs maildir
I'm using Dovecot 1.0.1-12 on Linux/Fedora 7 along with sendmail and procmail all running on the same box mail is stored in mbox format
It's a small system with a half dozen or so e-mail "accounts". Each with 40-60MB of messages in various folders.
I keep seeing messages about how mbox is antiquated and anybody with more than 100 messages etc should not use mbox, but use maildir instead.
I'm not entirely convinced.... there seem to be pros and cons for each. Is there a discussion somewhere that really highlights why one format is so much better than the other?
The last time I tried to convert from mbox to maildir, things got pretty botched up, no data loss, but it wasn't pretty. :-)
Can Dovecot handle mbox for some users and maildir for others? I'd like to try a conversion for one user... I'll probably create a new user, then have procmail copy (via ! action code) all mail for one user to that new user.
Thank you
Wow this is weird because I'm about to make this same jump next week!
From what I'm reading so far the big draw back with mbox is the single file with all the emails in it. When you delete a message from that file the whole file has to be rewritten without that email in it. If the box is big enough that can be a serious drag on the server. We have been using Dovecot here all school year for Imap & Pop3 with the Mbox format and when two or more people delete at the same time the utilization on my 3ware card shoots up. We bought the BBU unit for the 3ware so I could enable WRITE cache and that has helped tremendously.
I thought this study in regards to speed was quite interesting: http://www.courier-mta.org/mbox-vs-maildir/ http://www.courier-mta.org/mbox-vs-maildir/
So far my testing conversion process has gone really well. I am surprised how easy it was to tell procmail to do MailDir instead and even the conversion process was super easy. For converting the old inbox and folders I am using the tool mb2md.pl from http://batleth.sapienti-sat.org/projects/mb2md/
I was having a really hard time figuring all of this out until I ran into this webpage: http://adam.rosi-kessel.org/weblog/2007/04/18/adams-super-simple-guide-to-mb...
I know through namespaces you can do inbox in one type and other boxes in another type. I was initially thinking about doing all new stuff in maildir and still support the old ~/mail format. The setup seemed easy enough, but I figured in the long run I am shutting down the server for a few hours to do this so I mis well go all the way.
The only thing I'm not sure of is what the best file system to keep this on. I have been keeping my home directories on ReiserFS for quite a while, but one of our tech thinks XFS would be good. All data I have right now tells me to stay ReiserFS though. Even Dovecot's own page says XFS may not be a wise choice.
Hope some of this stuff helps you. My server BTW is: Slackware Slamd64 11 (Added Kerberos, Dovecot, etc after the fact) Dual AMD Opteron 242s 4 Gigs RAM 800 Gig RAID 5 3G SATA array ReiserFS on /home /var/spool/mail
-Jesse C. Smillie
"Insert inspirational or witty comment here...."
Don Russell wrote:
I'm using Dovecot 1.0.1-12 on Linux/Fedora 7 along with sendmail and procmail all running on the same box mail is stored in mbox format
It's a small system with a half dozen or so e-mail "accounts". Each with 40-60MB of messages in various folders.
I keep seeing messages about how mbox is antiquated and anybody with more than 100 messages etc should not use mbox, but use maildir instead.
I'm not entirely convinced.... there seem to be pros and cons for each. Is there a discussion somewhere that really highlights why one format is so much better than the other?
The last time I tried to convert from mbox to maildir, things got pretty botched up, no data loss, but it wasn't pretty. :-)
Can Dovecot handle mbox for some users and maildir for others? I'd like to try a conversion for one user... I'll probably create a new user, then have procmail copy (via ! action code) all mail for one user to that new user.
Thank you
This mail was scanned by BitDefender For more informations please visit http://www.bitdefender.com
On Fri, 2007-06-29 at 13:56 -0400, Jesse C. Smillie wrote:
I thought this study in regards to speed was quite interesting: http://www.courier-mta.org/mbox-vs-maildir/ http://www.courier-mta.org/mbox-vs-maildir/
It probably doesn't have much relevance to Dovecot+mbox though. Maildir is faster with expunges and with concurrent mailbox access, but I think Dovecot+mbox (without external changes) is faster for pretty much everything else.
Anyway I think the main reason to use maildir is that it's much more difficult to corrupt mailboxes.
The upcoming dbox and cydir formats of course beat everything in performance :)
Timo Sirainen tss@iki.fi writes:
The upcoming dbox and cydir formats of course beat everything in performance :)
cydir ? Does this mean there is a cyrus-like storage coming soon ?
-- Nicolas
On Fri, 2007-06-29 at 20:26 +0200, Nicolas KOWALSKI wrote:
Timo Sirainen tss@iki.fi writes:
The upcoming dbox and cydir formats of course beat everything in performance :)
cydir ? Does this mean there is a cyrus-like storage coming soon ?
Already in v1.1 tree. It's what I'm using for my index file stress tests, because the format is practically just the index files and "<uid>." named files.
Timo Sirainen tss@iki.fi writes:
On Fri, 2007-06-29 at 20:26 +0200, Nicolas KOWALSKI wrote:
Timo Sirainen tss@iki.fi writes:
The upcoming dbox and cydir formats of course beat everything in performance :)
cydir ? Does this mean there is a cyrus-like storage coming soon ?
Already in v1.1 tree. It's what I'm using for my index file stress tests, because the format is practically just the index files and "<uid>." named files.
I just read http://www.dovecot.org/list/dovecot/2007-May/022772.html, and the results look really impressive. 10 times faster than Maildir...
Time to upgrade and test ! :-)
Thanks Timo,
Nicolas
On Fri, 2007-06-29 at 21:49 +0200, Nicolas KOWALSKI wrote:
Timo Sirainen tss@iki.fi writes:
On Fri, 2007-06-29 at 20:26 +0200, Nicolas KOWALSKI wrote:
Timo Sirainen tss@iki.fi writes:
The upcoming dbox and cydir formats of course beat everything in performance :)
cydir ? Does this mean there is a cyrus-like storage coming soon ?
Already in v1.1 tree. It's what I'm using for my index file stress tests, because the format is practically just the index files and "<uid>." named files.
I just read http://www.dovecot.org/list/dovecot/2007-May/022772.html, and the results look really impressive. 10 times faster than Maildir...
Time to upgrade and test ! :-)
Just remember that if you lose the index files there's no easy way to recover the mailbox. Well, except by copying the files to maildir..
I'm not sure if I should try to make cydir anything else than a benchmark format or a simple example for writing mail storage backends. I'm hoping that dbox will be practically as fast in all situations.
Timo Sirainen tss@iki.fi writes:
On Fri, 2007-06-29 at 21:49 +0200, Nicolas KOWALSKI wrote:
Timo Sirainen tss@iki.fi writes:
On Fri, 2007-06-29 at 20:26 +0200, Nicolas KOWALSKI wrote:
Timo Sirainen tss@iki.fi writes:
The upcoming dbox and cydir formats of course beat everything in performance :)
cydir ? Does this mean there is a cyrus-like storage coming soon ?
Already in v1.1 tree. It's what I'm using for my index file stress tests, because the format is practically just the index files and "<uid>." named files.
I just read http://www.dovecot.org/list/dovecot/2007-May/022772.html, and the results look really impressive. 10 times faster than Maildir...
Time to upgrade and test ! :-)
Just remember that if you lose the index files there's no easy way to recover the mailbox. Well, except by copying the files to maildir..
Good to know, Thanks.
I'm not sure if I should try to make cydir anything else than a benchmark format or a simple example for writing mail storage backends. I'm hoping that dbox will be practically as fast in all situations.
If performance of dbox is as fast as cydir, and if cydir is not easily recoverable (a cyrus reconstruct-like tool would help a lot here), well, cydir could stay as benchmark format. But...
As I noticed almost no difference in performance (no real numbers here, just a usage feeling with 5-10k messages mailboxes) between dovecot 1.0 + maildir and cyrus 2.2.13/2.3.8, having dovecot use a storage format 10 times faster than Maildir is really attractive.
-- Nicolas
On Fri, 2007-06-29 at 22:16 +0200, Nicolas KOWALSKI wrote:
As I noticed almost no difference in performance (no real numbers here, just a usage feeling with 5-10k messages mailboxes) between dovecot 1.0 + maildir and cyrus 2.2.13/2.3.8, having dovecot use a storage format 10 times faster than Maildir is really attractive.
The 10 times was only when appending new messages. I haven't benchmarked other operations. I doubt there is any noticeable performance gain for single users, unless your mailbox has hundreds of thousands of messages.
Just remember that if you lose the index files there's no easy way to recover the mailbox. Well, except by copying the files to maildir..
I'm not sure if I should try to make cydir anything else than a benchmark format or a simple example for writing mail storage backends. I'm hoping that dbox will be practically as fast in all situations.
One advantage of cydir over dbox was mentioned by Mark above re incremental backups - with dbox, you'd still have to backup the entire mailbox file, while with cydir, you'd only have to copy newer messages.
I think making cydir a real, viable replacement for standard maildir would be a good thing...
:)
--
Best regards,
Charles
On 6/29/07, Charles Marcus CMarcus@media-brokers.com wrote:
Just remember that if you lose the index files there's no easy way to recover the mailbox. Well, except by copying the files to maildir..
I'm not sure if I should try to make cydir anything else than a benchmark format or a simple example for writing mail storage backends. I'm hoping that dbox will be practically as fast in all situations.
One advantage of cydir over dbox was mentioned by Mark above re incremental backups - with dbox, you'd still have to backup the entire mailbox file, while with cydir, you'd only have to copy newer messages.
True. In my old job, this was the main reason I switched users mailboxes from UW-imap MBX to Cyrus. Performance was also much better after the switch.
-- Nicolas
On Fri, 2007-06-29 at 16:37 -0400, Charles Marcus wrote:
Just remember that if you lose the index files there's no easy way to recover the mailbox. Well, except by copying the files to maildir..
I'm not sure if I should try to make cydir anything else than a benchmark format or a simple example for writing mail storage backends. I'm hoping that dbox will be practically as fast in all situations.
One advantage of cydir over dbox was mentioned by Mark above re incremental backups - with dbox, you'd still have to backup the entire mailbox file, while with cydir, you'd only have to copy newer messages.
I was thinking about making dbox configurable. If it is run in one-mail-per-file mode there's no need for locking either.
One advantage of cydir over dbox was mentioned by Mark above re incremental backups - with dbox, you'd still have to backup the entire mailbox file, while with cydir, you'd only have to copy newer messages.
I was thinking about making dbox configurable. If it is run in one-mail-per-file mode there's no need for locking either.
Ahhh... ! Ok, that would be 'a good thing'... :)
Curious, though - why *not* make cydir a real usable format, if its performance is so good? Is it only/because there is no good solution to the extra risk of data loss if indexes get lost/corrupted?
--
Best regards,
Charles
On Sat, 2007-06-30 at 13:01 -0400, Charles Marcus wrote:
One advantage of cydir over dbox was mentioned by Mark above re incremental backups - with dbox, you'd still have to backup the entire mailbox file, while with cydir, you'd only have to copy newer messages.
I was thinking about making dbox configurable. If it is run in one-mail-per-file mode there's no need for locking either.
Ahhh... ! Ok, that would be 'a good thing'... :)
Curious, though - why *not* make cydir a real usable format, if its performance is so good?
What if dbox's performance will be even better? We'll see.
Is it only/because there is no good solution to the extra risk of data loss if indexes get lost/corrupted?
That, and dbox will have some other features such as single instance attachment storage, which can't be implemented to cydir without making the format more complex (and then it's practically the same as dbox).
Curious, though - why *not* make cydir a real usable format, if its performance is so good?
What if dbox's performance will be even better? We'll see.
Heh... don't know why I even bothered asking - you are always about 357 steps ahead of me... ;)
Is it only/because there is no good solution to the extra risk of data loss if indexes get lost/corrupted?
That, and dbox will have some other features such as single instance attachment storage, which can't be implemented to cydir without making the format more complex (and then it's practically the same as dbox).
Bingo - dbox wins... :)
Thanks for all you do Timo...
--
Best regards,
Charles
- Jesse C. Smillie jsmillie@gatewayk12.org:
The only thing I'm not sure of is what the best file system to keep this on. I have been keeping my home directories on ReiserFS for quite a while, but one of our tech thinks XFS would be good.
XFS is lousy for many small files. We tried XFS for our 9000 Users (Maildir) and swithced back to ext3.
All data I have right now tells me to stay ReiserFS though. Even Dovecot's own page says XFS may not be a wise choice.
My experience tells me to stay away from ReiserFS as well.
-- Ralf Hildebrandt (Ralf.Hildebrandt@charite.de) plonk@charite.de Postfix - Einrichtung, Betrieb und Wartung Tel. +49 (0)30-450 570-155 http://www.arschkrebs.de Unix is the answer, but only if you phrase the question very carefully.
On Fri, 2007-06-29 at 20:34 +0200, Ralf Hildebrandt wrote:
- Jesse C. Smillie jsmillie@gatewayk12.org:
The only thing I'm not sure of is what the best file system to keep this on. I have been keeping my home directories on ReiserFS for quite a while, but one of our tech thinks XFS would be good.
XFS is lousy for many small files. We tried XFS for our 9000 Users (Maildir) and swithced back to ext3.
All data I have right now tells me to stay ReiserFS though. Even Dovecot's own page says XFS may not be a wise choice.
My experience tells me to stay away from ReiserFS as well.
How about NSS? I was considering using Netware's NSS for my backend server - either from Netware or Linux OES, but I'm not sure how OES actually handles the filesystem.. Anyways, in addition to journaling, you get deleted file salvage, open file backup/snapshot, and easily expandable volumes... I know you can do the last with LVM on Linux, and I recall something similar on FreeBSD - but I have no experience with either, and they're both missing salvage and snapshot.
Ok ok, so I really just love salvage. ;)
- Rick Romero rick@havokmon.com:
My experience tells me to stay away from ReiserFS as well.
How about NSS? I was considering using Netware's NSS for my backend server - either from Netware or Linux OES, but I'm not sure how OES actually handles the filesystem.. Anyways, in addition to journaling, you get deleted file salvage, open file backup/snapshot, and easily expandable volumes...
Sounds like a job for ZFS :)
-- Ralf Hildebrandt (Ralf.Hildebrandt@charite.de) plonk@charite.de Postfix - Einrichtung, Betrieb und Wartung Tel. +49 (0)30-450 570-155 http://www.arschkrebs.de Ballmer should step down in favour of Mr T, because he pity the fool who don't got high-end video cards and 4GB RAM for Vista Aero!
On Fri, 2007-06-29 at 20:53 +0200, Ralf Hildebrandt wrote:
- Rick Romero rick@havokmon.com:
My experience tells me to stay away from ReiserFS as well.
How about NSS? I was considering using Netware's NSS for my backend server - either from Netware or Linux OES, but I'm not sure how OES actually handles the filesystem.. Anyways, in addition to journaling, you get deleted file salvage, open file backup/snapshot, and easily expandable volumes...
Sounds like a job for ZFS :)
Yeah, but ZFS seems too new compared to 10 year old NSS ;)
On 6/29/2007, Rick Romero (rick@havokmon.com) wrote:
I know you can do the last with LVM on Linux, and I recall something similar on FreeBSD - but I have no experience with either, and they're both missing salvage and snapshot.
Eh? Guess I've just been dreaming then every time I do a snapshot on one of my LVM volumes......
;)
--
Best regards,
Charles
On Fri, 2007-06-29 at 16:32 -0400, Charles Marcus wrote:
On 6/29/2007, Rick Romero (rick@havokmon.com) wrote:
I know you can do the last with LVM on Linux, and I recall something similar on FreeBSD - but I have no experience with either, and they're both missing salvage and snapshot.
Eh? Guess I've just been dreaming then every time I do a snapshot on one of my LVM volumes......
I see that now, Wikipedia has a decent comparison page: http://en.wikipedia.org/wiki/Comparison_of_file_systems
Snapshot is a start, but still no salvage that I can find :(
OTOH, Being 10 years old means NSS is 'only' limited to 8TB ....
All data I have right now tells me to stay ReiserFS though. Even Dovecot's own page says XFS may not be a wise choice.
My experience tells me to stay away from ReiserFS as well.
Please, lets not start that war up again! ;)
Reiser has worked fine for me for many years, but I think the next time I rebuild my servers I'll be using ext3, in anticipation of ext4 (since it should be a fairly seamless switch)...
--
Best regards,
Charles
- Charles Marcus CMarcus@Media-Brokers.com:
Please, lets not start that war up again! ;)
Reiser has worked fine for me for many years, but I think the next time I rebuild my servers I'll be using ext3, in anticipation of ext4 (since it should be a fairly seamless switch)...
I did the ext3 -> ext4 switch on two of our proxyservers a few months ago. Then we forgot (!) about that test and the boxes just kept running and running and running ...
-- Ralf Hildebrandt (Ralf.Hildebrandt@charite.de) plonk@charite.de Postfix - Einrichtung, Betrieb und Wartung Tel. +49 (0)30-450 570-155 http://www.arschkrebs.de A bus station is where buses stop. A train station is where trains stop. On my desk, there is a workstation...
- Charles Marcus CMarcus@Media-Brokers.com:
I did the ext3 -> ext4 switch on two of our proxyservers a few months ago. Then we forgot (!) about that test and the boxes just kept running and running and running ...
Interesting... have you noticed any differences in performance?
No. But at least it didn't explode in my face :)
-- Ralf Hildebrandt (Ralf.Hildebrandt@charite.de) plonk@charite.de Postfix - Einrichtung, Betrieb und Wartung Tel. +49 (0)30-450 570-155 http://www.arschkrebs.de Am I advocating HTML email? No. I think it sucks. I'm really not interested in your artistic whims of yellow text on funky purple wallpaper.
I did the ext3 -> ext4 switch on two of our proxyservers a few months ago. Then we forgot (!) about that test and the boxes just kept running and running and running ...
Interesting... have you noticed any differences in performance?
No. But at least it didn't explode in my face :)
Heh.. yeah, thats always a GoodThing(TM)
--
Best regards,
Charle
Ralf Hildebrandt wrote:
XFS is lousy for many small files. We tried XFS for our 9000 Users (Maildir) and swithced back to ext3.
Properly tuned XFS is supposedly very nice. Check out:
http://everything2.com/index.pl?node_id=1479435
for more info.
On Fri, 2007-06-29 at 13:56 -0400, Jesse C. Smillie wrote:
Wow this is weird because I'm about to make this same jump next week!
From what I'm reading so far the big draw back with mbox is the single file with all the emails in it. When you delete a message from that file the whole file has to be rewritten without that email in it. If the box is big enough that can be a serious drag on the server. We have been using Dovecot here all school year for Imap & Pop3 with the Mbox format and when two or more people delete at the same time the utilization on my 3ware card shoots up. We bought the BBU unit for the 3ware so I could enable WRITE cache and that has helped tremendously.
I thought this study in regards to speed was quite interesting: http://www.courier-mta.org/mbox-vs-maildir/ http://www.courier-mta.org/mbox-vs-maildir/
So far my testing conversion process has gone really well. I am surprised how easy it was to tell procmail to do MailDir instead and even the conversion process was super easy. For converting the old inbox and folders I am using the tool mb2md.pl from http://batleth.sapienti-sat.org/projects/mb2md/
I was having a really hard time figuring all of this out until I ran into this webpage: http://adam.rosi-kessel.org/weblog/2007/04/18/adams-super-simple-guide-to-mb...
I know through namespaces you can do inbox in one type and other boxes in another type. I was initially thinking about doing all new stuff in maildir and still support the old ~/mail format. The setup seemed easy enough, but I figured in the long run I am shutting down the server for a few hours to do this so I mis well go all the way.
The only thing I'm not sure of is what the best file system to keep this on. I have been keeping my home directories on ReiserFS for quite a while, but one of our tech thinks XFS would be good. All data I have right now tells me to stay ReiserFS though. Even Dovecot's own page says XFS may not be a wise choice.
We've had good experiences with XFS, hosting ~40k mailboxes totalling ~400 GiB across 2 NFS fileservers (although we're not using dovecot for those users, yet) on hardware that's not particularly beefy by today's standards (and well below the specs you provided for your server.)
I've also personally had terrible experiences on a couple of other systems with reiserfs -- especially when the FS became slightly corrupt (due to failing hardware or the power going out at *just* the right time), though performance wasn't that great either.
All the systems I mention were/are using Maildir, and this is somewhat contrary to many benchmarks, but it's been working well for me. YMMV and all that rot.
-- Ben Winslow rain@bluecherry.net
On Fri, 29 Jun 2007 10:14:42 -0700 Don Russell russell.don@gmail.com wrote:
Can Dovecot handle mbox for some users and maildir for others?
Yes, if you don't have the mail_location variable set, then Dovecot will look in ~/Maildir /var/mail/username ~/mail ~/Mail in that order.
See http://wiki.dovecot.org/MailLocation
j
A suggestion: along with mail_location, create a mail_directories variable that contains the search order, so that it can be specified. It would default to something like mail_directories = maildir:~/Maildir|mbox:~/mail:INBOX=/var/mail/%u|...
I could use this. On one system I manage, I've converted a couple of us to Maildirs, but the rest of the users are still on mbox in a non-standard place: mbox:~/Mail:INBOX=~/mbox I don't want to do anything special, because eventually all the users will be switched to Maildirs, but in the meantime it would be nice to be able to use Dovecot for all of them without doing a "ln -s ~user/mbox ~user/Mail/inbox" for each of them.
j
On Fri, 29 Jun 2007 13:39:11 -0500 John Gateley dovecot@jfoo.net wrote:
On Fri, 29 Jun 2007 10:14:42 -0700 Don Russell russell.don@gmail.com wrote:
Can Dovecot handle mbox for some users and maildir for others?
Yes, if you don't have the mail_location variable set, then Dovecot will look in ~/Maildir /var/mail/username ~/mail ~/Mail in that order.
See http://wiki.dovecot.org/MailLocation
j
On Fri, 2007-06-29 at 14:41 -0500, John Gateley wrote:
A suggestion: along with mail_location, create a mail_directories variable that contains the search order, so that it can be specified. It would default to something like mail_directories = maildir:~/Maildir|mbox:~/mail:INBOX=/var/mail/%u|...
How about a post-login script like http://wiki.dovecot.org/MailLocation explains?
On Fri, 29 Jun 2007 22:50:46 +0300 Timo Sirainen tss@iki.fi wrote:
On Fri, 2007-06-29 at 14:41 -0500, John Gateley wrote:
A suggestion: along with mail_location, create a mail_directories variable that contains the search order, so that it can be specified. It would default to something like mail_directories = maildir:~/Maildir|mbox:~/mail:INBOX=/var/mail/%u|...
How about a post-login script like http://wiki.dovecot.org/MailLocation explains?
I didn't read carefully enough - that looks like it does the job. Thanks. I still think if you have a default search order, a variable controlling that would be kind of cool...
j
FWIW, I used imapsync for the data migrations of a) an old sendmail server with mbox format to a new server running postfix+dovecot, and b) and old and busted Microsoft SBS2000 Exchange instance to a new server running postfix+dovecot.
Worked well in both cases, and the process left the original servers intact in case the migration hadn't gone as well.
Completely unscientific observation, but the Maildir format seems quite responsive in the common email pattern of open the inbox, fetch the message headers, let the filter process spam, finally read ham messages. That has a lot to do with Dovecot's speed, of course.
Don Russell wrote:
I'm using Dovecot 1.0.1-12 on Linux/Fedora 7 along with sendmail and procmail all running on the same box mail is stored in mbox format
It's a small system with a half dozen or so e-mail "accounts". Each with 40-60MB of messages in various folders.
I keep seeing messages about how mbox is antiquated and anybody with more than 100 messages etc should not use mbox, but use maildir instead.
I'm not entirely convinced.... there seem to be pros and cons for each. Is there a discussion somewhere that really highlights why one format is so much better than the other?
One factor not often discussed: When I switched from mbox to maildir the size of incremental backups went down to a fraction of its previous size (only the new messages are backed up, not the entire mailbox). It allowed me to do incremental backups to disk instead of tape, which I still use for full backups.
P.S. consider maildrop instead of procmail if you switch to maildir. See info in the dovecot wiki.
Mark
Don Russell wrote:
I'm using Dovecot 1.0.1-12 on Linux/Fedora 7 along with sendmail and procmail all running on the same box mail is stored in mbox format
It's a small system with a half dozen or so e-mail "accounts". Each with 40-60MB of messages in various folders.
I keep seeing messages about how mbox is antiquated and anybody with more than 100 messages etc should not use mbox, but use maildir instead.
I'm not entirely convinced.... there seem to be pros and cons for each. Is there a discussion somewhere that really highlights why one format is so much better than the other?
mbox is broken by design. Look at the next line. From what I can tell, mbox will convert the first word of this line to ">From". This means the message is modified, which is ok for raw text, but is not ok for structure text such as TeX or XML.
I've also had to edit mbox files with via to remove garbage, probably caused by lock issues.
Regarding performances: while a single file should be faster to parse than loading N files, how about:
- moving messages between folders (including folders for different accounts)
- rsync-like backup is simple and fast with maildir
- using messages to retrain a spam filter ("mv" is all that is needed!).
The last time I tried to convert from mbox to maildir, things got pretty botched up, no data loss, but it wasn't pretty. :-)
just because you got it wrong doesn't make it's hard. you probably didn't take enough time to get it right.
Can Dovecot handle mbox for some users and maildir for others? I'd like to try a conversion for one user... I'll probably create a new user, then have procmail copy (via ! action code) all mail for one user to that new user.
Why not use one of the available mbox 2 maildir utilities.
mouss wrote:
mbox is broken by design. Look at the next line. From what I can tell, mbox will convert the first word of this line to ">From". This means the message is modified, which is ok for raw text, but is not ok for structure text such as TeX or XML.
argh. the example doesn't even work since my mailer added a space before the From! anyway, the issue is real.
mouss wrote:
Can Dovecot handle mbox for some users and maildir for others? I'd like to try a conversion for one user... I'll probably create a new user, then have procmail copy (via ! action code) all mail for one user to that new user.
Why not use one of the available mbox 2 maildir utilities.
in case you would have problems with this for whatever reason, another option is to simply start copying mail-folders inside a mail-client which connects to both old and new imap-server (or old mailboxes vs. new imap-server)
with that option you can even easily convert from mailboxes with "weird" formats like native pegasus mail mailboxes to maildir-format, assuming that you have set maildir-format as default on your new imap-server
just connect to your new imap-server with maildir as default inside your email-client and start copying etc.
Can Dovecot handle mbox for some users and maildir for others? I'd like to try a conversion for one user... I'll probably create a new user, then have procmail copy (via ! action code) all mail for one user to that new user.
I recently inherited a sendmail + UW IMAP installation (where hundred megabytes mbox files were becoming issues) and have since migrated to Postfix + Dovecot and have been very pleased. During the migration to the new server, I set up the old UW IMAP/POP3 on 127.0.0.1 and Perdition proxies on the original IP address. From Perdition you can specify which destination server to use via regex so it was easy to transition on a per user (or domain or anything else that you can specify via regex) basis. As a matter of fact we migrated the users over a one month period and had zero issues. The final touch was changing the DNS to the new server. Most users did not even know about the change and gave us plenty of opportunities for live testing. The only thing to watch out for is that Perdition doesn't know what capabilities to advertise for IMAP (especially if the two servers are very different) so you have to use the least common eliminator initially.
If getting a new server isn't an option, Xen and VPS environments have made provisioning a new server very cheap nowadays...
Tim
The last time I tried to convert from mbox to maildir, things got pretty botched up, no data loss, but it wasn't pretty. :-)
just because you got it wrong doesn't make it's hard. you probably didn't take enough time to get it right.
Well, if you know the RIGHT way - just share it with the rest of us. I was asking Timo before and as far as I understood - there is no way of converting mbox to Maildir without losing message UIDs. And given that we already have huge mailboxes, that means that every user, connecting to the server after the conversion will need to redownload all the messages and delete all the messages in the local cache. Just imagine that user with ~5Gb of mail across 5-10 folders connect from home not-so-fast DSL line to check email and has to wait until the client rechecks all the mail.... No, this is not right. Exactly as Don Russell wrote - "no data loss, but it isn't pretty". And this is the only reason I don't migrate to Maildir. I really want to do it, but this is not something I can do to my users.
FiL
On Tue, 2007-07-10 at 13:57 -0400, FiL @ Kpoxa wrote:
The last time I tried to convert from mbox to maildir, things got pretty botched up, no data loss, but it wasn't pretty. :-)
just because you got it wrong doesn't make it's hard. you probably didn't take enough time to get it right.
Well, if you know the RIGHT way - just share it with the rest of us. I was asking Timo before and as far as I understood - there is no way of converting mbox to Maildir without losing message UIDs.
With existing tools. It would be pretty easy to write such a tool that creates dovecot-uidlist file with correct UIDs.
Don Russell wrote:
I'm using Dovecot 1.0.1-12 on Linux/Fedora 7 along with sendmail and procmail all running on the same box mail is stored in mbox format
[snip]
Thanks to all who replied. This seems to have sparked quite a discussion, and given me quite a bit to read/look into.
Sounds like dbox is worth waiting for. I'd I'm going to convert from mbox, I'd rather convert once. :-)
participants (16)
-
albinootje
-
Ben Winslow
-
Charles Marcus
-
Don Russell
-
FiL ＠ Kpoxa
-
Jeff Kowalczyk
-
Jesse C. Smillie
-
John Gateley
-
Mark Nienberg
-
Michal Soltys
-
mouss
-
Nicolas KOWALSKI
-
Ralf Hildebrandt
-
Rick Romero
-
Tim Tsai
-
Timo Sirainen