[Dovecot] dbox vs. mdbox
What are the pros and cons of both? Especially in regards to performance, stability, management & maintenance?
I really appreciate feedback. We're on a time-crunch to migrate from a debian 5 box w/ dovecot 1.1 to a debian 6 box w/ dovecot 2.0.9 (built from source).
Thanks,
Doug Mortensen Network Consultant Impala Networks Inc CCNA, MCSA, Security+, A+ Linux+, Network+, Server+ . www.impalanetworks.com P: (505) 327-7300 F: (505) 327-7545
Hi
I really appreciate feedback. We're on a time-crunch to migrate from a debian 5 box w/ dovecot 1.1 to a debian 6 box w/ dovecot 2.0.9 (built from source).
I would have thought if time is tight then stick with what you know and migrate later? If you aren't building your new box as a virtualised container of some sort then stop and consider that to be much more desirable... With a virtualised install you can easily spin up an alternative config and more easily migrate the format of the data separately to the config of the server?
Oh, I'm also a huge fan of the "proxy" feature which allows you to spin up a new machine/config, then migrate users across slowly rather than one big bang. You can use it with your current migration to test the upgrade for example?
According to the wiki, your answer to the dbox question is probably best given here: http://wiki2.dovecot.org/MailboxFormat/dbox
Good luck
Ed W
Thanks for the input Ed. The evening before you sent the message, I actually had decided to do just what you recommended (stick with what we know). We have since put the server into production, and things are working well. We built to a physical box, rather than a VM. I do like xen VMs myself. We didn't do the proxy, but I appreciate the tip.
I actually read the wiki before posting.
I guess to get more specific, some of the questions I have regarding dbox vs. mdbox are:
- What is the advantage to using multiple files?
- What is the advantage to using a single sdbox file for each user?
- Is this a binary format, or txt (UTF?)?
- Are there real-world benchmarks showing measurable differences between maildir, sdbox, mdbox?
- Are sdbox & mdbox equally stable to Maildir? Are they recommended for production systems?
This is the kind of info that I'm after. I didn't really see that in the wiki. In the wiki, I only saw that the different formats exist, and basics on how they work.
Any feedback here would be appreciated.
Thanks,
Doug Mortensen Network Consultant Impala Networks P: 505.327.7300
On 4.3.2011, at 23.05, Douglas Mortensen wrote:
I guess to get more specific, some of the questions I have regarding dbox vs. mdbox are:
- What is the advantage to using multiple files?
mdbox in theory uses less disk I/O for "normal users".
- What is the advantage to using a single sdbox file for each user?
It's simpler. More difficult to get corrupted. Also if in future there exists a filesystem that supports smaller files better, it's then faster than mdbox. Probably unlikely that it will happen anytime soon.
- Is this a binary format, or txt (UTF?)?
dbox headers/metadata is ASCII. The message bodies can of course be anything.
- Are there real-world benchmarks showing measurable differences between maildir, sdbox, mdbox?
Not that I'm aware of. So far everyone I've tried to ask have replaced their whole mail system and their storage, so the before/after numbers can't be compared. I'm very interested in knowing myself too.
- Are sdbox & mdbox equally stable to Maildir? Are they recommended for production systems?
sdbox is so simple that I doubt anyone will find any kind of corruption bugs. mdbox is more complex, but people are using it in production and I haven't heard of any problems recently. Although there have been bugs in how mdbox handles already corrupted files, v2.0.10 had several fixes related to that.
Hi,
Just giving my own experience:
I am using dovecot 2.0.9 (well, now 2.0.10 since today) in production without problems
Sdbox is using far too much I/O on a busy server, I had to switch to mdbox
Mdbox is running well so far, and resources (IO or CPU) are not an issue anymore.
Converting from Maildir to s/mdbox is easy
Converting from sdbox to mdbox has been a complete nightmare. I have never managed to make it completely, finally made it through imap protocol between 2 instance of dovecot. You better choose before sd or md, but not try to convert between the 2
my 2 cents
JM
On Fri, 4 Mar 2011 23:19:21 +0200, Timo Sirainen wrote:
- What is the advantage to using a single sdbox file for each user?
On 4.3.2011, at 23.05, Douglas Mortensen wrote:
I guess to get more specific, some of the questions I have regarding dbox vs. mdbox are: 1. What is the advantage to using multiple files? mdbox in theory uses less disk I/O for "normal users".
It's simpler. More difficult to get corrupted. Also if in future there exists a filesystem that supports smaller files better, it's then faster than mdbox. Probably unlikely that it will happen anytime soon. 3.
e="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%">4. Are there real-world benchmarks showing measurable differences between maildir, sdbox, mdbox? torage, so
an of course be anything. the before/after numbers can't be compared. I'm very interested in knowing myself too.
e type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%">5. Are sdbox & mdbox equally stable to Maildir? Are they recommended for production systems? sdbox is so simple that I doubt anyone will find any kind of corruption bugs. mdbox is more complex, but people are using
dles already corrupted files, v2.0.10 had several fixes related to that.
On 5.3.2011, at 4.14, Joan Moreau wrote:
- Sdbox is using far too much I/O on a busy server, I had to switch to mdbox ..
- Converting from Maildir to s/mdbox is easy
Are you saying sdbox uses more disk I/O than maildir? That's unexpected.
- Converting from sdbox to mdbox has been a complete nightmare. I have never managed to make it completely, finally made it through imap protocol between 2 instance of dovecot. You better choose before sd or md, but not try to convert between the 2
What was difficult about it? You should have been able to do it really easily with dsync.
- Sdbox is using far too much I/O on a busy server, I had to switch to mdbox
- Converting from Maildir to s/mdbox is easy
Are you saying sdbox uses more disk I/O than maildir? That's unexpected.
No. I am saying that sbbox is not sustainable when having very large mailbox, IO becomes too high (even with high-end storage devices)
erting from sdbox to mdbox has been a complete nightmare. I have never
managed to make it completely, finally made it through imap protocol between 2 instance of dovecot. You better choose before sd or md, but not try to convert between the 2
was difficult about it? You should have been able to do it really easily with dsync
No, as you know very well: I sent plenty of emails about this matter at that time on this mailng list, but nobody found any solutions, so, I gave up. Dsync is too buggy to convert sdbox to mdbox.
On Sun, 2011-03-06 at 15:46 -0500, Joan Moreau wrote:
No, as you know very well: I sent plenty of emails about this matter at that time on this mailng list, but nobody found any solutions, so, I gave up. Dsync is too buggy to convert sdbox to mdbox.
I think I fixed all the crashes you reported. The only two problems left were 1) the "Too many open files" error, which I'm pretty sure is caused by 2) m.* file sizes are too small.
http://hg.dovecot.org/dovecot-2.0/rev/d1f30e375f40 should fix 1), but I don't understand the cause of 2), since your config looked ok. Did you solve it somehow or are your m.* files still too small?
The only solution I found was to use IMAP protocol to read from sdbox and write as mdbox.
Dsync was NOT a solution at all. Sorry about that.
On Sun, 06 Mar 2011 23:58:35 +0200, Timo Sirainen wrote:
On Sun, 2011-03-06 at 15:46 -0500, Joan Moreau wrote:
No, as you know very well: I sent plenty of emails about this matter at that time on this mailng list, but nobody found any solutions, so, I gave up. Dsync is too buggy to convert sdbox to mdbox. I think I fixed all the crashes you reported. The only two problems left were 1) the "Too many open files" error, which I'm pretty sure is caused by 2) m.* file sizes are too small. http://hg.dovecot.org/dovecot-2.0/rev/d1f30e375f40 [1] should fix 1), but I don't understand the cause of 2), since your config looked ok. Did you solve it somehow or are your m.* files still too small?
Links:
Timo Sirainen
- Are there real-world benchmarks showing measurable differences between maildir, sdbox mdbox?
Not that I'm aware of. So far everyone I've tried to ask have replaced their whole mail system and their storage, so the before/after numbers can't be compared. I'm very interested in knowing myself too.
I think I can give my contribution here. I'm planning to migrate from dovecot 1.x to 2.x. Currently, on 1.x I'm using Maildir as this was my best choice at the time, but now I'm trying to decide the mailbox format for a 2.x fresh install.
The environment will be virtually the same as this is running in a virtualized
environment. I'm keeping the same storage and storage network, same host
hardware and about the same VM specs.
Its the data from about 100 users, sizing about 300GB in size spread over about 2 million files in Maildir format. So I think this could provide statistically relevant information.
My ideia is to install a fresh server and replicate the production maildir on it to build a test "source disk" that I'll use then to experiment the conversions to sdbox and mdbox. I then plan to test the performance of the dbox formats, but I can include Maildir measurements just for the record.
I'm open to suggestions on how to test this properly
On 28.3.2012, at 13.13, Hangas wrote:
Timo Sirainen
writes:
- Are there real-world benchmarks showing measurable differences between maildir, sdbox mdbox?
Not that I'm aware of. So far everyone I've tried to ask have replaced their whole mail system and their storage, so the before/after numbers can't be compared. I'm very interested in knowing myself too.
I think I can give my contribution here. I'm planning to migrate from dovecot 1.x to 2.x. Currently, on 1.x I'm using Maildir as this was my best choice at the time, but now I'm trying to decide the mailbox format for a 2.x fresh install. .. My ideia is to install a fresh server and replicate the production maildir on it to build a test "source disk" that I'll use then to experiment the conversions to sdbox and mdbox. I then plan to test the performance of the dbox formats, but I can include Maildir measurements just for the record.
I'm open to suggestions on how to test this properly
The main problem is that it's difficult to do any "real world" tests with IMAP, especially when users are using many different kinds of IMAP clients. So I'm very interested in hearing some numbers (and disk IO graphs for a few weeks would be great) before your migration and after your migration, but the numbers for your tests might not mean all that much.
On Thu, Mar 29, 2012 at 12:30 AM, Timo Sirainen tss@iki.fi wrote:
The main problem is that it's difficult to do any "real world" tests with IMAP, especially when users are using many different kinds of IMAP clients. So I'm very interested in hearing some numbers (and disk IO graphs for a few weeks would be great) before your migration and after your migration, but the numbers for your tests might not mean all that much.
I was considering using the imaptest tool to simulate IMAP activity. I would keep the same machine configuration, only varying the mailbox format while running imaptest against each setup for a few hours/days.
I'm now converting the original Maildir format to both dbox formats and I'll give it a try. I'll share some graphs afterwards.
Hi,
I finally deployed a migration test plant and done some conversions last night, so I took the chance to collect some data of these tests.
I've set up a virtual machine with a dovecot 2.0.13 installation in order test the mailbox conversion.
This machine has two adicional disks, one with the source data and other to store the conversion results. The source data is a restore from the latest production Maildir backup and I did a conversion to sdbox, and then to mdbox.
I attach DiskIO and CPU plots for both conversions
orange Line is read from source disk, blue line is write to destination. sdbox on the left and mdbox on the right.
Ok, this is a conversion process and not actual mailbox usage. And lacks info on how does dbox behaves in read operations.. but this is what I got for now.
[image: Inline image 1]
[image: Inline image 2] Miguel
On Thu, Mar 29, 2012 at 12:16 PM, Miguel Afonso mafonso@hangas.net wrote:
On Thu, Mar 29, 2012 at 12:30 AM, Timo Sirainen tss@iki.fi wrote:
The main problem is that it's difficult to do any "real world" tests with IMAP, especially when users are using many different kinds of IMAP clients. So I'm very interested in hearing some numbers (and disk IO graphs for a few weeks would be great) before your migration and after your migration, but the numbers for your tests might not mean all that much.
I was considering using the imaptest tool to simulate IMAP activity. I would keep the same machine configuration, only varying the mailbox format while running imaptest against each setup for a few hours/days.
I'm now converting the original Maildir format to both dbox formats and I'll give it a try. I'll share some graphs afterwards.
participants (6)
-
Douglas Mortensen
-
Ed W
-
Hangas
-
Joan Moreau
-
Miguel Afonso
-
Timo Sirainen