[EXT] Re: ZLIB/ZSTD Compressed MBOX in Maildir++ layout

[Josef 'Jeff' Sipek]

On Mon, Jun 28, 2021 at 15:21:23 +0100, Pedro Ribeiro wrote:
> Hello again and thanks for the quick answer!
> 
> Our base (inbox) namespace is based on MDBOX with SIS
> 
> SIS is fantastic optimizing the storage required to the active (daily 
> accessed/moved) email.
> 
> But for archival and easy recover of some specific user mailbox/folder 
> it isn't practical to rely on the shared attachment storage of SIS, also 
> I think is more universal format in case some legal obligation require 
> us to give the e-mail of some user to the authorities, including all the 
> attachments.
> 
> I would opt for some tradeoff solution if I find a way to enable the 
> altstorage or additional namespace (also with MDBOX) but without SIS so 
> that I can store the old mails with the attachments included and 
> compressed. (zstd does a great job!)
>
> Is there any way to do it?

Hmmm, I see.  How often do you sync the archive?  (Keep in mind, that
compressed mbox mailboxes are read-only.)  Would maildir work for you for
this archive?  It is standard, and I've definitely seen compression work
with it.  And if someone needed the whole mailbox, it is easy enough to
generate a tarball.

> It seems to me SIS can't be enabled/disabled for each namespace.

Your configuration is using a bunch of features in dovecot (mbox,
compression, sis) that I haven't ever used, so I'm not the most
knowledgable.  But, I think you are correct - enabling sis is global to all
dbox namespaces on the server.

> zlib_save would benefit if it could be different for each namespace (eg. 
> lz4 or none for the inbox namespace and zstd for the archival)

Indeed.

Depending on what system you use, another option is to see if your storage
system can do compression.  For example, I have dovecot store all mail
uncompressed but have zfs transparently compress everything with lz4.
(Maybe one day I'll switch it to zstd.)

Jeff.

> 
> Regards.
> 
> On 6/28/21 13:42, Josef 'Jeff' Sipek wrote:
> > On Mon, Jun 28, 2021 at 10:00:59 +0100, Pedro Ribeiro wrote:
> >> Hello!
> >>
> >> Thank you for the suggestion.
> >>
> >> That solved the problem of using layout FS with MBOX storage format.
> >>
> >> After copying all the mail of the test user to the namespace in this
> >> format I've compressed all the storage files with ZSTD and that changed
> >> the name with the additional suffix .zst
> >>
> >> Dovecot seems to need this to be .zstd so I've fixed the names.
> >>
> >> But ... it doesn't work.
> >>
> >>> Jun 28 08:56:39 ribble dovecot:
> >>> imap(pribeiro at net.ipl.pt)<82609><vi0h1M7FGMMKZP4/>: Error: Mailbox
> >>> @HISTZLIB/IPLNet:
> >>> stat(/mnt/mail-hist/historic-fs/net.ipl.pt/ro/pribeiro/mailboxes/IPLNet/mBoX-MeSsAgEs)
> >>> failed: No such file or directory
> >> In that directory the filename is:
> >>
> >>> -rw------- 1 vmail vmail 853M Jun 25 17:25 mBoX-MeSsAgEs.zstd
> >> What am I doing wrong?
> > Hm.  It is possible that it just doesn't work.  mbox doesn't get a lot of
> > attention these days - its support is more of a migration tool.
> >
> > Out of curiosity, why do you use mbox?  In your earlier email you said that
> > you want a self contained format.  Why?  What's the goal?
> >
> > Jeff.
> >
> >> regards!
> >>
> >> On 24/06/2021 14:53, Josef 'Jeff' Sipek wrote:
> >>> On Thu, Jun 24, 2021 at 10:35:24 +0100, Pedro Ribeiro wrote:
> >>>> Hello everyone!
> >>>>
> >>>> Our archive namespace is kept in MBOX format with Maildir++ layout due to:
> >>>>
> >>>>     * Need to have the email from all users/folders self contained (our
> >>>>       base namespace uses MDBOX with single instance storage of attachments)
> >>>>     * MBOX/FS layout isn't supported because some folders have messages
> >>>>       and subfolders.
> >>> You should be able to use the fs layout if you specify a dirname:
> >>> https://doc.dovecot.org/configuration_manual/mail_location/mbox/mboxchildfolders/#messages-in-named-file
> >>>
> >>> Then, the compression suffix being added shouldn't matter.
> >>>
> >>> Jeff.
> >> -- 
> >> Best regards,
> >>
> >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> >> Pedro Ribeiro
> >> Politécnico de Lisboa, Serviços da Presidência
> >> Departamento de Sistemas de Informação e Comunicações
> >> Phone:   +351 210 464 700 (general) / VoIP: 80100
> >> Helpdesk:helpdesk at net.ipl.pt  /https://www.net.ipl.pt
> >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> >>
> >>
> >
> -- 
> Best regards,
> 
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Pedro Ribeiro
> Politécnico de Lisboa, Serviços da Presidência
> Departamento de Sistemas de Informação e Comunicações
> Phone:   +351 210 464 700 (general) / PBX Ext.: 80100
> Helpdesk: helpdesk at net.ipl.pt / https://www.net.ipl.pt
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> 
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://dovecot.org/pipermail/dovecot/attachments/20210628/e855b8d7/attachment-0001.sig>