I’ve been playing with zlib compressed mail archives. I can’t seem to get it to work. I followed the instructions here:
https://doc.dovecot.org/configuration_manual/zlib_plugin/
I’m not interested in being able to save, just read-only would be great.
I have some compressed mail archives, for example 2020.gz in my ~/mail/ directory. I have some uncompressed ones too like 2021. Both are mbox formatted. In my imap client, I see both 2020.gz and 2021 as imap folders. I can access mail in 2021 but not in 2020.gz. It just says it’s empty.
My local.conf has this in it:
mail_plugins = $mail_plugins zlib
I know for sure this is being read because if I change this to some non-existent plugin, I see an in the log complaining about a non-existent plugin. I’ve tried also adding in the lines for saving but no difference.
Would love to know if there’s something more I need to do. Also, should I expect to see a folder named 2020 instead of 2020.gz in my imap clients? My suspicion is that dovecot is treating these as uncompressed imap folders which it can’t read because they look like garbage. Is there some doveadm command to get it to re-scan folders that I need to run and then realize this is a folder using zlib?
Michael Grant
On 14/06/2023 14:28 EEST Michael Grant via dovecot dovecot@dovecot.org wrote:
I’ve been playing with zlib compressed mail archives. I can’t seem to get it to work. I followed the instructions here:
https://doc.dovecot.org/configuration_manual/zlib_plugin/
I’m not interested in being able to save, just read-only would be great.
I have some compressed mail archives, for example 2020.gz in my ~/mail/ directory. I have some uncompressed ones too like 2021. Both are mbox formatted. In my imap client, I see both 2020.gz and 2021 as imap folders. I can access mail in 2021 but not in 2020.gz. It just says it’s empty.
My local.conf has this in it:
mail_plugins = $mail_plugins zlib
I know for sure this is being read because if I change this to some non-existent plugin, I see an in the log complaining about a non-existent plugin. I’ve tried also adding in the lines for saving but no difference.
Would love to know if there’s something more I need to do. Also, should I expect to see a folder named 2020 instead of 2020.gz in my imap clients? My suspicion is that dovecot is treating these as uncompressed imap folders which it can’t read because they look like garbage. Is there some doveadm command to get it to re-scan folders that I need to run and then realize this is a folder using zlib?
Michael Grant
I checked with latest main, and it seems that:
- it requires the folder to be accessed as "folder.gz"
- if it has any broken emails, then it will fail to read the folder.
So this feature is not really working great. I would suggest to decompress your folders as workaround.
Aki
This morning, it looks like it’s working. It took a long time to decompress the compressed mail folder.
You’re right, it shows up in imap as “2021.gz”.
Up to now, I was decompressing it if I needed it, this is great, saves me a step. I don’t need to modify it, just search it sometimes.
From: Aki Tuomi via dovecot Sent: 15 June 2023 15:21 To: Michael Grant; Michael Grant via dovecot Subject: Re: zlib compressed folders
On 14/06/2023 14:28 EEST Michael Grant via dovecot dovecot@dovecot.org wrote:
I’ve been playing with zlib compressed mail archives. I can’t seem to get it to work. I followed the instructions here:
https://doc.dovecot.org/configuration_manual/zlib_plugin/
I’m not interested in being able to save, just read-only would be great.
I have some compressed mail archives, for example 2020.gz in my ~/mail/ directory. I have some uncompressed ones too like 2021. Both are mbox formatted. In my imap client, I see both 2020.gz and 2021 as imap folders. I can access mail in 2021 but not in 2020.gz. It just says it’s empty.
My local.conf has this in it:
mail_plugins = $mail_plugins zlib
I know for sure this is being read because if I change this to some non-existent plugin, I see an in the log complaining about a non-existent plugin. I’ve tried also adding in the lines for saving but no difference.
Would love to know if there’s something more I need to do. Also, should I expect to see a folder named 2020 instead of 2020.gz in my imap clients? My suspicion is that dovecot is treating these as uncompressed imap folders which it can’t read because they look like garbage. Is there some doveadm command to get it to re-scan folders that I need to run and then realize this is a folder using zlib?
Michael Grant
I checked with latest main, and it seems that:
- it requires the folder to be accessed as "folder.gz"
- if it has any broken emails, then it will fail to read the folder.
So this feature is not really working great. I would suggest to decompress your folders as workaround.
Aki
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
15.06.2023 19:30, Michael Grant via dovecot wrote:
This morning, it looks like it’s working. It took a long time to decompress the compressed mail folder.
You’re right, it shows up in imap as “2021.gz”.
Up to now, I was decompressing it if I needed it, this is great, saves me a step. I don’t need to modify it, just search it sometimes.
FWIW, I found modern filesystems (such as btrfs) does an excellent job at compressing files. It is more, modern compression algorithms (such as zstd) compresses better than gzip and does it faster. And when the compression support is part of the filesystem, you don't need to worry about decompressing it anymore or check if the tools you're using support (de)compression or not, - it just works.
btrfs itself has its own interesting.. properties, one has to be careful and know a few easy rules when using its advanced features.
For example, in context of compression, when compression is enabled but the data is written to a given file in small portions and especially with fsyncs in between, btrfs will mark this file as "not compressible" (m attribute), even if the data itself is actually well-compressible. This is an optimization by btrfs to avoid spending time compressing this stuff. If you know the data is compressible, you'll have to set +c attribute instead (like force-compress) and recompress it as a whole with btrfs filesystem defrag -czstd filename (this operation is safe wrt fsyncs, due to CoW features). So this is something to keep an eye on, but once you know how it works, and do some book-keeping, it works fine.
/mjt
participants (3)
-
Aki Tuomi
-
Michael Grant
-
Michael Tokarev