ZLIB/ZSTD Compressed MBOX in Maildir++ layout
Hello everyone!
Our archive namespace is kept in MBOX format with Maildir++ layout due to:
- Need to have the email from all users/folders self contained (our base namespace uses MDBOX with single instance storage of attachments)
- MBOX/FS layout isn't supported because some folders have messages and subfolders.
I'm trying to compress that namespace using zstd/gzip following the informations from Dovecot documentation about the zlib plugin but when the (MBOX/Maildir++) compress and become with .zst/.zstd/.gz suffix (both tested) the system can't read the mailboxes and incorrectly reports the suffix (.zstd/.gz) as a subfolder name (following the Maildir++ file structure).
Is there any way to make the MBOX/Maildir++ compatible with the zlib compression?
Using Dovecot 2.3.14 on Debian10
Regards.
-- Best regards,
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pedro Ribeiro Politécnico de Lisboa, Serviços da Presidência Departamento de Sistemas de Informação e Comunicações Phone: +351 210 464 700 (general) / VoIP: 80100 Helpdesk: helpdesk@net.ipl.pt / https://www.net.ipl.pt
On Thu, Jun 24, 2021 at 10:35:24 +0100, Pedro Ribeiro wrote:
Hello everyone!
Our archive namespace is kept in MBOX format with Maildir++ layout due to:
- Need to have the email from all users/folders self contained (our base namespace uses MDBOX with single instance storage of attachments)
- MBOX/FS layout isn't supported because some folders have messages and subfolders.
You should be able to use the fs layout if you specify a dirname: https://doc.dovecot.org/configuration_manual/mail_location/mbox/mboxchildfol...
Then, the compression suffix being added shouldn't matter.
Jeff.
Hello!
Thank you for the suggestion.
That solved the problem of using layout FS with MBOX storage format.
After copying all the mail of the test user to the namespace in this format I've compressed all the storage files with ZSTD and that changed the name with the additional suffix .zst
Dovecot seems to need this to be .zstd so I've fixed the names.
But ... it doesn't work.
Jun 28 08:56:39 ribble dovecot: imap(pribeiro@net.ipl.pt)<82609><vi0h1M7FGMMKZP4/>: Error: Mailbox @HISTZLIB/IPLNet: stat(/mnt/mail-hist/historic-fs/net.ipl.pt/ro/pribeiro/mailboxes/IPLNet/mBoX-MeSsAgEs) failed: No such file or directory In that directory the filename is:
-rw------- 1 vmail vmail 853M Jun 25 17:25 mBoX-MeSsAgEs.zstd What am I doing wrong?
regards!
On 24/06/2021 14:53, Josef 'Jeff' Sipek wrote:
On Thu, Jun 24, 2021 at 10:35:24 +0100, Pedro Ribeiro wrote:
Hello everyone!
Our archive namespace is kept in MBOX format with Maildir++ layout due to:
- Need to have the email from all users/folders self contained (our base namespace uses MDBOX with single instance storage of attachments)
- MBOX/FS layout isn't supported because some folders have messages and subfolders. You should be able to use the fs layout if you specify a dirname: https://doc.dovecot.org/configuration_manual/mail_location/mbox/mboxchildfol...
Then, the compression suffix being added shouldn't matter.
Jeff.
-- Best regards,
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pedro Ribeiro Politécnico de Lisboa, Serviços da Presidência Departamento de Sistemas de Informação e Comunicações Phone: +351 210 464 700 (general) / VoIP: 80100 Helpdesk:helpdesk@net.ipl.pt /https://www.net.ipl.pt
On Mon, Jun 28, 2021 at 10:00:59 +0100, Pedro Ribeiro wrote:
Hello!
Thank you for the suggestion.
That solved the problem of using layout FS with MBOX storage format.
After copying all the mail of the test user to the namespace in this format I've compressed all the storage files with ZSTD and that changed the name with the additional suffix .zst
Dovecot seems to need this to be .zstd so I've fixed the names.
But ... it doesn't work.
Jun 28 08:56:39 ribble dovecot: imap(pribeiro@net.ipl.pt)<82609><vi0h1M7FGMMKZP4/>: Error: Mailbox @HISTZLIB/IPLNet: stat(/mnt/mail-hist/historic-fs/net.ipl.pt/ro/pribeiro/mailboxes/IPLNet/mBoX-MeSsAgEs) failed: No such file or directory In that directory the filename is:
-rw------- 1 vmail vmail 853M Jun 25 17:25 mBoX-MeSsAgEs.zstd What am I doing wrong?
Hm. It is possible that it just doesn't work. mbox doesn't get a lot of attention these days - its support is more of a migration tool.
Out of curiosity, why do you use mbox? In your earlier email you said that you want a self contained format. Why? What's the goal?
Jeff.
regards!
On 24/06/2021 14:53, Josef 'Jeff' Sipek wrote:
On Thu, Jun 24, 2021 at 10:35:24 +0100, Pedro Ribeiro wrote:
Hello everyone!
Our archive namespace is kept in MBOX format with Maildir++ layout due to:
- Need to have the email from all users/folders self contained (our base namespace uses MDBOX with single instance storage of attachments)
- MBOX/FS layout isn't supported because some folders have messages and subfolders. You should be able to use the fs layout if you specify a dirname: https://doc.dovecot.org/configuration_manual/mail_location/mbox/mboxchildfol...
Then, the compression suffix being added shouldn't matter.
Jeff.
-- Best regards,
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pedro Ribeiro Politécnico de Lisboa, Serviços da Presidência Departamento de Sistemas de Informação e Comunicações Phone: +351 210 464 700 (general) / VoIP: 80100 Helpdesk:helpdesk@net.ipl.pt /https://www.net.ipl.pt
Hello again and thanks for the quick answer!
Our base (inbox) namespace is based on MDBOX with SIS
SIS is fantastic optimizing the storage required to the active (daily accessed/moved) email.
But for archival and easy recover of some specific user mailbox/folder it isn't practical to rely on the shared attachment storage of SIS, also I think is more universal format in case some legal obligation require us to give the e-mail of some user to the authorities, including all the attachments.
I would opt for some tradeoff solution if I find a way to enable the altstorage or additional namespace (also with MDBOX) but without SIS so that I can store the old mails with the attachments included and compressed. (zstd does a great job!)
Is there any way to do it?
It seems to me SIS can't be enabled/disabled for each namespace.
zlib_save would benefit if it could be different for each namespace (eg. lz4 or none for the inbox namespace and zstd for the archival)
Regards.
On 6/28/21 13:42, Josef 'Jeff' Sipek wrote:
On Mon, Jun 28, 2021 at 10:00:59 +0100, Pedro Ribeiro wrote:
Hello!
Thank you for the suggestion.
That solved the problem of using layout FS with MBOX storage format.
After copying all the mail of the test user to the namespace in this format I've compressed all the storage files with ZSTD and that changed the name with the additional suffix .zst
Dovecot seems to need this to be .zstd so I've fixed the names.
But ... it doesn't work.
Jun 28 08:56:39 ribble dovecot: imap(pribeiro@net.ipl.pt)<82609><vi0h1M7FGMMKZP4/>: Error: Mailbox @HISTZLIB/IPLNet: stat(/mnt/mail-hist/historic-fs/net.ipl.pt/ro/pribeiro/mailboxes/IPLNet/mBoX-MeSsAgEs) failed: No such file or directory In that directory the filename is:
-rw------- 1 vmail vmail 853M Jun 25 17:25 mBoX-MeSsAgEs.zstd What am I doing wrong? Hm. It is possible that it just doesn't work. mbox doesn't get a lot of attention these days - its support is more of a migration tool.
Out of curiosity, why do you use mbox? In your earlier email you said that you want a self contained format. Why? What's the goal?
Jeff.
regards!
On 24/06/2021 14:53, Josef 'Jeff' Sipek wrote:
On Thu, Jun 24, 2021 at 10:35:24 +0100, Pedro Ribeiro wrote:
Hello everyone!
Our archive namespace is kept in MBOX format with Maildir++ layout due to:
* Need to have the email from all users/folders self contained (our base namespace uses MDBOX with single instance storage of attachments) * MBOX/FS layout isn't supported because some folders have messages and subfolders.You should be able to use the fs layout if you specify a dirname: https://doc.dovecot.org/configuration_manual/mail_location/mbox/mboxchildfol...
Then, the compression suffix being added shouldn't matter.
Jeff.
Best regards,
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pedro Ribeiro Politécnico de Lisboa, Serviços da Presidência Departamento de Sistemas de Informação e Comunicações Phone: +351 210 464 700 (general) / VoIP: 80100 Helpdesk:helpdesk@net.ipl.pt /https://www.net.ipl.pt
-- Best regards,
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pedro Ribeiro Politécnico de Lisboa, Serviços da Presidência Departamento de Sistemas de Informação e Comunicações Phone: +351 210 464 700 (general) / PBX Ext.: 80100 Helpdesk: helpdesk@net.ipl.pt / https://www.net.ipl.pt
On Mon, Jun 28, 2021 at 15:21:23 +0100, Pedro Ribeiro wrote:
Hello again and thanks for the quick answer!
Our base (inbox) namespace is based on MDBOX with SIS
SIS is fantastic optimizing the storage required to the active (daily accessed/moved) email.
But for archival and easy recover of some specific user mailbox/folder it isn't practical to rely on the shared attachment storage of SIS, also I think is more universal format in case some legal obligation require us to give the e-mail of some user to the authorities, including all the attachments.
I would opt for some tradeoff solution if I find a way to enable the altstorage or additional namespace (also with MDBOX) but without SIS so that I can store the old mails with the attachments included and compressed. (zstd does a great job!)
Is there any way to do it?
Hmmm, I see. How often do you sync the archive? (Keep in mind, that compressed mbox mailboxes are read-only.) Would maildir work for you for this archive? It is standard, and I've definitely seen compression work with it. And if someone needed the whole mailbox, it is easy enough to generate a tarball.
It seems to me SIS can't be enabled/disabled for each namespace.
Your configuration is using a bunch of features in dovecot (mbox, compression, sis) that I haven't ever used, so I'm not the most knowledgable. But, I think you are correct - enabling sis is global to all dbox namespaces on the server.
zlib_save would benefit if it could be different for each namespace (eg. lz4 or none for the inbox namespace and zstd for the archival)
Indeed.
Depending on what system you use, another option is to see if your storage system can do compression. For example, I have dovecot store all mail uncompressed but have zfs transparently compress everything with lz4. (Maybe one day I'll switch it to zstd.)
Jeff.
Regards.
On 6/28/21 13:42, Josef 'Jeff' Sipek wrote:
On Mon, Jun 28, 2021 at 10:00:59 +0100, Pedro Ribeiro wrote:
Hello!
Thank you for the suggestion.
That solved the problem of using layout FS with MBOX storage format.
After copying all the mail of the test user to the namespace in this format I've compressed all the storage files with ZSTD and that changed the name with the additional suffix .zst
Dovecot seems to need this to be .zstd so I've fixed the names.
But ... it doesn't work.
Jun 28 08:56:39 ribble dovecot: imap(pribeiro@net.ipl.pt)<82609><vi0h1M7FGMMKZP4/>: Error: Mailbox @HISTZLIB/IPLNet: stat(/mnt/mail-hist/historic-fs/net.ipl.pt/ro/pribeiro/mailboxes/IPLNet/mBoX-MeSsAgEs) failed: No such file or directory In that directory the filename is:
-rw------- 1 vmail vmail 853M Jun 25 17:25 mBoX-MeSsAgEs.zstd What am I doing wrong? Hm. It is possible that it just doesn't work. mbox doesn't get a lot of attention these days - its support is more of a migration tool.
Out of curiosity, why do you use mbox? In your earlier email you said that you want a self contained format. Why? What's the goal?
Jeff.
regards!
On 24/06/2021 14:53, Josef 'Jeff' Sipek wrote:
On Thu, Jun 24, 2021 at 10:35:24 +0100, Pedro Ribeiro wrote:
Hello everyone!
Our archive namespace is kept in MBOX format with Maildir++ layout due to:
* Need to have the email from all users/folders self contained (our base namespace uses MDBOX with single instance storage of attachments) * MBOX/FS layout isn't supported because some folders have messages and subfolders.You should be able to use the fs layout if you specify a dirname: https://doc.dovecot.org/configuration_manual/mail_location/mbox/mboxchildfol...
Then, the compression suffix being added shouldn't matter.
Jeff.
Best regards,
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pedro Ribeiro Politécnico de Lisboa, Serviços da Presidência Departamento de Sistemas de Informação e Comunicações Phone: +351 210 464 700 (general) / VoIP: 80100 Helpdesk:helpdesk@net.ipl.pt /https://www.net.ipl.pt
-- Best regards,
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pedro Ribeiro Politécnico de Lisboa, Serviços da Presidência Departamento de Sistemas de Informação e Comunicações Phone: +351 210 464 700 (general) / PBX Ext.: 80100 Helpdesk: helpdesk@net.ipl.pt / https://www.net.ipl.pt
participants (2)
-
Josef 'Jeff' Sipek
-
Pedro Ribeiro