[Dovecot] Questions about dbox (sdbox?), mdbox
Hello,
So ive spent the last 45 minutes or so searching the list for dbox stuff, but Im not seeing anything real definitive.
My question is basically whether the dbox format is ready for production environment, is sdbox or mdbox more stable/ready than the other or does it matter? What are people's experience/review with running this format (and if you reply can you tell me which one you're using).
Is there any docs/wiki page for migrating from maildir -> dbox? Searching for "dbox" on the wiki only returns two results, and only one is actually for dbox.
What I have (maildir format) is working, but who doesnt like shiny new toys? =P
I have ~450gigs / 14k pop accounts I would have to migrate. Oh thats another question, I have seen "hybrid" throw around, can I take this to mean dbox/mdbox/both support both formats, maybe only new mail goes into the dbox stuff? And if so (or even if not) is there any way to actually convert from maildir to dbox?
If anyone has any links that I cant seem to find by consulting the oracle (google) that would be fantastic
=)
On Fri, Jul 16, 2010 at 2:19 PM, Brandon Lamb <brandonlamb@gmail.com> wrote:
Hello,
So ive spent the last 45 minutes or so searching the list for dbox stuff, but Im not seeing anything real definitive.
My question is basically whether the dbox format is ready for production environment, is sdbox or mdbox more stable/ready than the other or does it matter? What are people's experience/review with running this format (and if you reply can you tell me which one you're using).
Is there any docs/wiki page for migrating from maildir -> dbox? Searching for "dbox" on the wiki only returns two results, and only one is actually for dbox.
What I have (maildir format) is working, but who doesnt like shiny new toys? =P
I have ~450gigs / 14k pop accounts I would have to migrate. Oh thats another question, I have seen "hybrid" throw around, can I take this to mean dbox/mdbox/both support both formats, maybe only new mail goes into the dbox stuff? And if so (or even if not) is there any way to actually convert from maildir to dbox?
If anyone has any links that I cant seem to find by consulting the oracle (google) that would be fantastic
=)
Another question, if there is such a conversion tool, does it break UIDLs and require pop3 clients to redownload messages?
On 16.7.2010, at 22.19, Brandon Lamb wrote:
My question is basically whether the dbox format is ready for production environment, is sdbox or mdbox more stable/ready than the other or does it matter? What are people's experience/review with running this format (and if you reply can you tell me which one you're using).
sdbox is really simple. mdbox is more complex. A few people have used sdbox with v1.x, but it's more or less rewritten with v2.0. With v2.0 I'm not aware of any sdbox installations, but there is at least one somewhat large mdbox installation. They've had some problems, but I think those are mostly gone now.
Is there any docs/wiki page for migrating from maildir -> dbox?
dsync migrates between any two mailbox formats and does it as perfectly as it can. http://wiki2.dovecot.org/Tools/Dsync
I have ~450gigs / 14k pop accounts I would have to migrate. Oh thats another question, I have seen "hybrid" throw around, can I take this to mean dbox/mdbox/both support both formats, maybe only new mail goes into the dbox stuff? And if so (or even if not) is there any way to actually convert from maildir to dbox?
v1.x had some kind of a maildir/dbox hybrid.. until I realized it was just a bad idea. You can have per-user mail location defined in userdb though, so you can convert users a few at a time.
Another question, if there is such a conversion tool, does it break UIDLs and require pop3 clients to redownload messages?
dsync is awesome and preserves everything. But test it anyway just in case. I'd anyway recommend setting pop3_save_uidl=yes and sometimes after that (when you think everyone's logged in at least once) switching to pop3_uidl_format=%g while still running maildir. With Maildir the new UIDLs are then the filenames and with dbox the UIDLs will be message GUIDs. Those are much more permanent than anything based on IMAP UIDs. Not a requirement, but still :)
On Fri, Jul 16, 2010 at 2:31 PM, Timo Sirainen <tss@iki.fi> wrote:
On 16.7.2010, at 22.19, Brandon Lamb wrote:
My question is basically whether the dbox format is ready for production environment, is sdbox or mdbox more stable/ready than the other or does it matter? What are people's experience/review with running this format (and if you reply can you tell me which one you're using).
sdbox is really simple. mdbox is more complex. A few people have used sdbox with v1.x, but it's more or less rewritten with v2.0. With v2.0 I'm not aware of any sdbox installations, but there is at least one somewhat large mdbox installation. They've had some problems, but I think those are mostly gone now.
Any recommendation of one over the other? I came across a 2009 mailing list entry where you showed some benchmark tests that had mdbox as the fastest. Without any experience yet, my only possible hangup is being able to restore individual (lost, whatever other reason) mails for customers. Doesnt really happen but being able to restore individual versus their whole mailbox is nice, but it happens so rarely it may be a non issue.
Is there any docs/wiki page for migrating from maildir -> dbox?
dsync migrates between any two mailbox formats and does it as perfectly as it can. http://wiki2.dovecot.org/Tools/Dsync
Awesome, I'll go do some reading and mess with that
I have ~450gigs / 14k pop accounts I would have to migrate. Oh thats another question, I have seen "hybrid" throw around, can I take this to mean dbox/mdbox/both support both formats, maybe only new mail goes into the dbox stuff? And if so (or even if not) is there any way to actually convert from maildir to dbox?
v1.x had some kind of a maildir/dbox hybrid.. until I realized it was just a bad idea. You can have per-user mail location defined in userdb though, so you can convert users a few at a time.
Another question, if there is such a conversion tool, does it break UIDLs and require pop3 clients to redownload messages?
dsync is awesome and preserves everything. But test it anyway just in case. I'd anyway recommend setting pop3_save_uidl=yes and sometimes after that (when you think everyone's logged in at least once) switching to pop3_uidl_format=%g while still running maildir. With Maildir the new UIDLs are then the filenames and with dbox the UIDLs will be message GUIDs. Those are much more permanent than anything based on IMAP UIDs. Not a requirement, but still :)
Cool, if anything I could just have a script login to every users account and logout and that would do the same thing right? Or does it have to perform some kind of list or other action?
Any main/major/big reasons to try to go with mdbox over sdbox? sdbox being really simple = less prone to typos and other bad things happening?
Oh, what about if i migrate to sdbox, then decide to go with mdbox, will I want to know for sure ahead of time? Granted I guess thats kind of an obvious yes, but any big gotchas to know about if I did?
Oh yea and dovecot kicks ass btw =P
On Fri, Jul 16, 2010 at 4:07 PM, Brandon Lamb <brandonlamb@gmail.com> wrote:
On Fri, Jul 16, 2010 at 2:31 PM, Timo Sirainen <tss@iki.fi> wrote:
On 16.7.2010, at 22.19, Brandon Lamb wrote:
My question is basically whether the dbox format is ready for production environment, is sdbox or mdbox more stable/ready than the other or does it matter? What are people's experience/review with running this format (and if you reply can you tell me which one you're using).
sdbox is really simple. mdbox is more complex. A few people have used sdbox with v1.x, but it's more or less rewritten with v2.0. With v2.0 I'm not aware of any sdbox installations, but there is at least one somewhat large mdbox installation. They've had some problems, but I think those are mostly gone now.
Any recommendation of one over the other? I came across a 2009 mailing list entry where you showed some benchmark tests that had mdbox as the fastest. Without any experience yet, my only possible hangup is being able to restore individual (lost, whatever other reason) mails for customers. Doesnt really happen but being able to restore individual versus their whole mailbox is nice, but it happens so rarely it may be a non issue.
Is there any docs/wiki page for migrating from maildir -> dbox?
dsync migrates between any two mailbox formats and does it as perfectly as it can. http://wiki2.dovecot.org/Tools/Dsync
Awesome, I'll go do some reading and mess with that
I have ~450gigs / 14k pop accounts I would have to migrate. Oh thats another question, I have seen "hybrid" throw around, can I take this to mean dbox/mdbox/both support both formats, maybe only new mail goes into the dbox stuff? And if so (or even if not) is there any way to actually convert from maildir to dbox?
v1.x had some kind of a maildir/dbox hybrid.. until I realized it was just a bad idea. You can have per-user mail location defined in userdb though, so you can convert users a few at a time.
Another question, if there is such a conversion tool, does it break UIDLs and require pop3 clients to redownload messages?
dsync is awesome and preserves everything. But test it anyway just in case. I'd anyway recommend setting pop3_save_uidl=yes and sometimes after that (when you think everyone's logged in at least once) switching to pop3_uidl_format=%g while still running maildir. With Maildir the new UIDLs are then the filenames and with dbox the UIDLs will be message GUIDs. Those are much more permanent than anything based on IMAP UIDs. Not a requirement, but still :)
Cool, if anything I could just have a script login to every users account and logout and that would do the same thing right? Or does it have to perform some kind of list or other action?
Any main/major/big reasons to try to go with mdbox over sdbox? sdbox being really simple = less prone to typos and other bad things happening?
Oh, what about if i migrate to sdbox, then decide to go with mdbox, will I want to know for sure ahead of time? Granted I guess thats kind of an obvious yes, but any big gotchas to know about if I did?
Oh yea and dovecot kicks ass btw =P
Ok I got all migrated over to dovecot2 configuration, and started to dive into this converting business... one thing I immediately fell into is that the wiki doesnt say anything about namespace issues. I return maildir:~/maildir in my current setup via prefetch/sql, and I got errors trying to do
dsync -u brandon@olypen.com mirror mdbox:~/mdbox
Do I need some special namespace configuation/setup?
On Fri, Jul 16, 2010 at 8:34 PM, Brandon Lamb <brandonlamb@gmail.com> wrote:
On Fri, Jul 16, 2010 at 4:07 PM, Brandon Lamb <brandonlamb@gmail.com> wrote:
On Fri, Jul 16, 2010 at 2:31 PM, Timo Sirainen <tss@iki.fi> wrote:
On 16.7.2010, at 22.19, Brandon Lamb wrote:
My question is basically whether the dbox format is ready for production environment, is sdbox or mdbox more stable/ready than the other or does it matter? What are people's experience/review with running this format (and if you reply can you tell me which one you're using).
sdbox is really simple. mdbox is more complex. A few people have used sdbox with v1.x, but it's more or less rewritten with v2.0. With v2.0 I'm not aware of any sdbox installations, but there is at least one somewhat large mdbox installation. They've had some problems, but I think those are mostly gone now.
Any recommendation of one over the other? I came across a 2009 mailing list entry where you showed some benchmark tests that had mdbox as the fastest. Without any experience yet, my only possible hangup is being able to restore individual (lost, whatever other reason) mails for customers. Doesnt really happen but being able to restore individual versus their whole mailbox is nice, but it happens so rarely it may be a non issue.
Is there any docs/wiki page for migrating from maildir -> dbox?
dsync migrates between any two mailbox formats and does it as perfectly as it can. http://wiki2.dovecot.org/Tools/Dsync
Awesome, I'll go do some reading and mess with that
I have ~450gigs / 14k pop accounts I would have to migrate. Oh thats another question, I have seen "hybrid" throw around, can I take this to mean dbox/mdbox/both support both formats, maybe only new mail goes into the dbox stuff? And if so (or even if not) is there any way to actually convert from maildir to dbox?
v1.x had some kind of a maildir/dbox hybrid.. until I realized it was just a bad idea. You can have per-user mail location defined in userdb though, so you can convert users a few at a time.
Another question, if there is such a conversion tool, does it break UIDLs and require pop3 clients to redownload messages?
dsync is awesome and preserves everything. But test it anyway just in case. I'd anyway recommend setting pop3_save_uidl=yes and sometimes after that (when you think everyone's logged in at least once) switching to pop3_uidl_format=%g while still running maildir. With Maildir the new UIDLs are then the filenames and with dbox the UIDLs will be message GUIDs. Those are much more permanent than anything based on IMAP UIDs. Not a requirement, but still :)
Cool, if anything I could just have a script login to every users account and logout and that would do the same thing right? Or does it have to perform some kind of list or other action?
Any main/major/big reasons to try to go with mdbox over sdbox? sdbox being really simple = less prone to typos and other bad things happening?
Oh, what about if i migrate to sdbox, then decide to go with mdbox, will I want to know for sure ahead of time? Granted I guess thats kind of an obvious yes, but any big gotchas to know about if I did?
Oh yea and dovecot kicks ass btw =P
Ok I got all migrated over to dovecot2 configuration, and started to dive into this converting business... one thing I immediately fell into is that the wiki doesnt say anything about namespace issues. I return maildir:~/maildir in my current setup via prefetch/sql, and I got errors trying to do
dsync -u brandon@olypen.com mirror mdbox:~/mdbox
Do I need some special namespace configuation/setup?
I think I may have got it, I added the following to my 10-mail.conf
# maildir namespace { type = private separator = / prefix = # location = maildir:~/maildir inbox = yes hidden = no }
# mdbox namespace { type = private separator = / prefix = "#mdbox/" location = mdbox:~/mdbox inbox = no hidden = no }
On 17.7.2010, at 4.46, Brandon Lamb wrote:
Ok I got all migrated over to dovecot2 configuration, and started to dive into this converting business... one thing I immediately fell into is that the wiki doesnt say anything about namespace issues. I return maildir:~/maildir in my current setup via prefetch/sql, and I got errors trying to do
dsync -u brandon@olypen.com mirror mdbox:~/mdbox
What errors did you get?
Do I need some special namespace configuation/setup?
No..
I think I may have got it, I added the following to my 10-mail.conf
# maildir namespace { type = private separator = / prefix = # location = maildir:~/maildir inbox = yes hidden = no }
# mdbox namespace { type = private separator = / prefix = "#mdbox/" location = mdbox:~/mdbox inbox = no hidden = no }
This shouldn't be necessary.
On Sat, Jul 17, 2010 at 11:03 AM, Timo Sirainen <tss@iki.fi> wrote:
On 17.7.2010, at 4.46, Brandon Lamb wrote:
Ok I got all migrated over to dovecot2 configuration, and started to dive into this converting business... one thing I immediately fell into is that the wiki doesnt say anything about namespace issues. I return maildir:~/maildir in my current setup via prefetch/sql, and I got errors trying to do
dsync -u brandon@olypen.com mirror mdbox:~/mdbox
What errors did you get?
Do I need some special namespace configuation/setup?
No..
I think I may have got it, I added the following to my 10-mail.conf
# maildir namespace { type = private separator = / prefix = # location = maildir:~/maildir inbox = yes hidden = no }
# mdbox namespace { type = private separator = / prefix = "#mdbox/" location = mdbox:~/mdbox inbox = no hidden = no }
This shouldn't be necessary.
Without any namespaces it told me that I had to define the heirarchy separator. Then I added the namespaces initially with both of them specifying a location. But then when I tried to run dsync again, it told me the namespace of the one to be mirrored should have no location so I had to remove that. Then it worked.
I just wiped out and rebuilt my raid on my second box im working on so once thats back and i restore my base maildir i'll run it again to get the actual error output =P
On Sat, Jul 17, 2010 at 1:09 PM, Timo Sirainen <tss@iki.fi> wrote:
On 17.7.2010, at 20.39, Brandon Lamb wrote:
Without any namespaces it told me that I had to define the heirarchy separator.
Yes, that you need to set. But you need only one namespace.
Oh! I'll try that instead soon thanks!
On Sat, Jul 17, 2010 at 1:10 PM, Brandon Lamb <brandonlamb@gmail.com> wrote:
On Sat, Jul 17, 2010 at 1:09 PM, Timo Sirainen <tss@iki.fi> wrote:
On 17.7.2010, at 20.39, Brandon Lamb wrote:
Without any namespaces it told me that I had to define the heirarchy separator.
Yes, that you need to set. But you need only one namespace.
Oh! I'll try that instead soon thanks!
So I had another thought, if I have a user that has 500 megs of mail but its mostly large attachments, with the default mailfile size at 2 megs this means a 25 meg attachment is going to have to write to and read from 13 files every time? And if so I guess this is just a tradeoff of the usual case of mails being <100k so fitting lots into the 2m files?
On 07/18/2010 03:50 AM Brandon Lamb wrote:
So I had another thought, if I have a user that has 500 megs of mail but its mostly large attachments, with the default mailfile size at 2 megs this means a 25 meg attachment is going to have to write to and read from 13 files every time?
Interesting theory. ;-) But happily wrong. Messages with a size >2m will get their own storage file.
Regards, Pascal
The trapper recommends today: http://kopfkrebs.de/mitarbeiter/mitarbeiter_der_woche.html
On 17.7.2010, at 0.07, Brandon Lamb wrote:
sdbox is really simple. mdbox is more complex. A few people have used sdbox with v1.x, but it's more or less rewritten with v2.0. With v2.0 I'm not aware of any sdbox installations, but there is at least one somewhat large mdbox installation. They've had some problems, but I think those are mostly gone now.
Any recommendation of one over the other?
Depends on the filesystem and disks I guess. mdbox should reduce mailbox fragmentation because it uses larger files. But maybe with SSDs that doesn't matter and the extra metadata just slows down. Or maybe not. I don't have any real world benchmarks.
I came across a 2009 mailing list entry where you showed some benchmark tests that had mdbox as the fastest.
They were benchmarks of some unrealistic tests.
Without any experience yet, my only possible hangup is being able to restore individual (lost, whatever other reason) mails for customers.
This is more difficult with sdbox and mdbox than with maildir. I haven't really thought about this before. I guess with both you could get the one file (or the entire mailbox, either way) and then force a index rebuild and then copy/extract the mail in some way (e.g. doveadm fetch, imap client, etc).
dsync is awesome and preserves everything. But test it anyway just in case. I'd anyway recommend setting pop3_save_uidl=yes and sometimes after that (when you think everyone's logged in at least once) switching to pop3_uidl_format=%g while still running maildir. With Maildir the new UIDLs are then the filenames and with dbox the UIDLs will be message GUIDs. Those are much more permanent than anything based on IMAP UIDs. Not a requirement, but still :)
Cool, if anything I could just have a script login to every users account and logout and that would do the same thing right?
That's all.
Any main/major/big reasons to try to go with mdbox over sdbox? sdbox being really simple = less prone to typos and other bad things happening?
There is much less potential for mailbox corruption with sdbox, since it's so much simpler.
Oh, what about if i migrate to sdbox, then decide to go with mdbox, will I want to know for sure ahead of time? Granted I guess thats kind of an obvious yes, but any big gotchas to know about if I did?
You can use dsync to easily switch between them.
participants (3)
-
Brandon Lamb
-
Pascal Volk
-
Timo Sirainen