Re: Sis to deduplicate attachments does not work?
On April 23, 2019 10:54:38 PM luckydog xf luckydogxf@gmail.com wrote:
Is it worthwile to use dbox? seeing from http://www.linuxmail.info/mbox-maildir-mail-storage-formats/ it may cause file lock and easy to corrupt. As with everything - it depends. You're asking me so these are *my* opinions - and I do not claim to be anything more than a hobbyist/tinkerer when the comes to this.
mbox has potential use for long term read-only archives - I see no reason to use it for live mailboxes.
maildir is undoubtedly the least susceptible to corruption. It's also the slowest format for reading. How slow is "slow" depends on your hardware - it may be imperceptible with enough RAM and SSD's - or it may result in user complaints with large mailboxes.
dbox is Dovecot's preferred format. I know Timo has put a lot of effort into it. sdbox is similar to maildir in that each mail is a separate file. mdbox significantly reduces the number of files which can make file-based backups faster. Both dbox formats are dependent on their index files.
If you've got good hardware, including a proper UPS, I'd recommend dbox (my server is presently using sdbox). With large mailboxes and file-based backups you'll benefit from mdbox. When reliability is the #1 concern above anything else - use maildir. Depending on your use SIS can have significant impact on storage requirements - but storage these days is relatively cheap.
I haven't seen much feedback from users actively using SIS - I'd love to hear from high traffic sites with SIS experience to know if the corruption issues have been resolved. In my case there was at least a 30% reduction in space but I had too many errors - admittedly it's been a couple years since I last tried it.
-- Daniel
On Wed Apr 24 2019 04:12:30 GMT-0400 (Eastern Standard Time), Daniel Miller via dovecot dovecot@dovecot.org wrote:
If you've got good hardware, including a proper UPS, I'd recommend dbox (my server is presently using sdbox). With large mailboxes and file-based backups you'll benefit from mdbox. When reliability is the #1 concern above anything else - use maildir. Depending on your use SIS can have significant impact on storage requirements - but storage these days is relatively cheap.
My plan when I roll out my new server this year is to use mdbox, but put the indexes and other important meta data on a smallish volume using either ZFS or BTRFS, for the automatic self-healing capabilities (and the ability to expand it if necessary).
This pretty much eliminates the worry about data loss from index file corruption.
I haven't seen much feedback from users actively using SIS - I'd love to hear from high traffic sites with SIS experience to know if the corruption issues have been resolved. In my case there was at least a 30% reduction in space but I had too many errors - admittedly it's been a couple years since I last tried it.
I never tried it because of the problems with respect to backup/restore, and if I'm not mistaken, those problems have not been resolved.
Maybe its a design issue...
Or maybe it just isn't a high enough priority, like the missing x-original-to header in the LMTP code that will still prevent me from being able to use the otherwise much better LMTP delivery agent.
It seems Maildir is the safest mail format right now, as long as you could accept little performance and disk cost penalty.
On Wed, Apr 24, 2019 at 9:01 PM Tanstaafl via dovecot dovecot@dovecot.org wrote:
On Wed Apr 24 2019 04:12:30 GMT-0400 (Eastern Standard Time), Daniel Miller via dovecot dovecot@dovecot.org wrote:
If you've got good hardware, including a proper UPS, I'd recommend dbox (my server is presently using sdbox). With large mailboxes and file-based backups you'll benefit from mdbox. When reliability is the #1 concern above anything else - use maildir. Depending on your use SIS can have significant impact on storage requirements - but storage these days is relatively cheap.
My plan when I roll out my new server this year is to use mdbox, but put the indexes and other important meta data on a smallish volume using either ZFS or BTRFS, for the automatic self-healing capabilities (and the ability to expand it if necessary).
This pretty much eliminates the worry about data loss from index file corruption.
I haven't seen much feedback from users actively using SIS - I'd love to hear from high traffic sites with SIS experience to know if the corruption issues have been resolved. In my case there was at least a 30% reduction in space but I had too many errors - admittedly it's been a couple years since I last tried it.
I never tried it because of the problems with respect to backup/restore, and if I'm not mistaken, those problems have not been resolved.
Maybe its a design issue...
Or maybe it just isn't a high enough priority, like the missing x-original-to header in the LMTP code that will still prevent me from being able to use the otherwise much better LMTP delivery agent.
Thanks for your sharing, Daniel.
On Wed, Apr 24, 2019 at 4:12 PM Daniel Miller dmiller@amfes.com wrote:
On April 23, 2019 10:54:38 PM luckydog xf luckydogxf@gmail.com wrote:
Is it worthwile to use dbox? seeing from http://www.linuxmail.info/mbox-maildir-mail-storage-formats/ it may cause file lock and easy to corrupt.
As with everything - it depends. You're asking me so these are *my* opinions - and I do not claim to be anything more than a hobbyist/tinkerer when the comes to this.
mbox has potential use for long term read-only archives - I see no reason to use it for live mailboxes.
maildir is undoubtedly the least susceptible to corruption. It's also the slowest format for reading. How slow is "slow" depends on your hardware - it may be imperceptible with enough RAM and SSD's - or it may result in user complaints with large mailboxes.
dbox is Dovecot's preferred format. I know Timo has put a lot of effort into it. sdbox is similar to maildir in that each mail is a separate file. mdbox significantly reduces the number of files which can make file-based backups faster. Both dbox formats are dependent on their index files.
If you've got good hardware, including a proper UPS, I'd recommend dbox (my server is presently using sdbox). With large mailboxes and file-based backups you'll benefit from mdbox. When reliability is the #1 concern above anything else - use maildir. Depending on your use SIS can have significant impact on storage requirements - but storage these days is relatively cheap.
I haven't seen much feedback from users actively using SIS - I'd love to hear from high traffic sites with SIS experience to know if the corruption issues have been resolved. In my case there was at least a 30% reduction in space but I had too many errors - admittedly it's been a couple years since I last tried it.
-- Daniel
participants (3)
-
Daniel Miller
-
luckydog xf
-
Tanstaafl