[Dovecot] zlib plugin
hi,
Could anyone give some info on this plugin? When i read the wiki (http://wiki.dovecot.org/Plugins/Zlib) i'm still left with some questions...
I use Maildir to store my mails, if i enable this zlib-plugin, should i 'manually' compress all messages? or wil the uncompressed messages still be accessible by dovecot? wil new messages be compressed automagically?
how should the compressed messages be named? a message now has the name '1223224520.2783_1.opensuse:2,S' how would a compressed message be named? ['1223224520.2783_1.opensuse:2,SZ' or '1223224520.2783_1.opensuse:2,S,Z' or ???]
Last but not least, what gain will i have by using this plugin in combination with Maildir, or will it have more impact when mails are stored in mbox.
-- Luuk $ dovecot --version dovecot version 1.1.1 $ uname -srpmio Linux 2.6.18.8-0.10-default x86_64 x86_64 x86_64 GNU/Linux
On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote:
hi,
Could anyone give some info on this plugin? When i read the wiki (http://wiki.dovecot.org/Plugins/Zlib) i'm still left with some questions...
I use Maildir to store my mails, if i enable this zlib-plugin, should i 'manually' compress all messages? or wil the uncompressed messages still be accessible by dovecot? wil new messages be compressed automagically?
Dovecot can read both compressed and uncompressed messages.
how should the compressed messages be named?
It doesn't matter from Dovecot's point of view since it detects the compression by reading the first few bytes of the file. But since you'll be compressing them manually you should mark them in some way so you don't compress anything twice (and that would break Dovecot too).
a message now has the name '1223224520.2783_1.opensuse:2,S' how would a compressed message be named? ['1223224520.2783_1.opensuse:2,SZ'
This is OK.
or '1223224520.2783_1.opensuse:2,S,Z'
This isn't a valid maildir file name.
Last but not least, what gain will i have by using this plugin in combination with Maildir, or will it have more impact when mails are stored in mbox.
I guess it depends on the size of your mails. With mboxes it can be used only with read-only archives, so it's not all that useful. With maildir it can be used transparently with read-write mailboxes. I've no idea what the performance difference or space improvements are. I guess that'll also depend on your system.
Timo Sirainen schreef:
On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote:
hi,
Could anyone give some info on this plugin? When i read the wiki (http://wiki.dovecot.org/Plugins/Zlib) i'm still left with some questions...
I use Maildir to store my mails, if i enable this zlib-plugin, should i 'manually' compress all messages? or wil the uncompressed messages still be accessible by dovecot? wil new messages be compressed automagically?
Dovecot can read both compressed and uncompressed messages.
how should the compressed messages be named?
It doesn't matter from Dovecot's point of view since it detects the compression by reading the first few bytes of the file. But since you'll be compressing them manually you should mark them in some way so you don't compress anything twice (and that would break Dovecot too).
a message now has the name '1223224520.2783_1.opensuse:2,S' how would a compressed message be named? ['1223224520.2783_1.opensuse:2,SZ'
This is OK.
or '1223224520.2783_1.opensuse:2,S,Z'
This isn't a valid maildir file name.
Last but not least, what gain will i have by using this plugin in combination with Maildir, or will it have more impact when mails are stored in mbox.
I guess it depends on the size of your mails. With mboxes it can be used only with read-only archives, so it's not all that useful. With maildir it can be used transparently with read-write mailboxes. I've no idea what the performance difference or space improvements are. I guess that'll also depend on your system.
ok, thank you, in the near furure i will try to install this zlib-plugin (on a test-system) and see what happens...
-- Luuk $ dovecot --version dovecot version 1.1.1 $ uname -srpmio Linux 2.6.18.8-0.10-default x86_64 x86_64 x86_64 GNU/Linux
On 279, 10 05, 2008 at 10:38:02PM +0300, Timo Sirainen wrote:
On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote:
Last but not least, what gain will i have by using this plugin in combination with Maildir, or will it have more impact when mails are stored in mbox.
I guess it depends on the size of your mails. With mboxes it can be used only with read-only archives, so it's not all that useful. With maildir it can be used transparently with read-write mailboxes. I've no idea what the performance difference or space improvements are. I guess that'll also depend on your system.
I use zlib compressed maildirs on my primary server (12000+ users) and it achieves average compression ratio of 1.8, so if your system is io-bound you can spend some cpu cycles to significantly decrease disk load.
-- Andrey Panin | Linux and UNIX system administrator pazke@donpac.ru | PGP key: wwwkeys.pgp.net
On Tue, 2008-10-07 at 11:38 +0400, Andrey Panin wrote:
On 279, 10 05, 2008 at 10:38:02PM +0300, Timo Sirainen wrote:
On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote:
Last but not least, what gain will i have by using this plugin in combination with Maildir, or will it have more impact when mails are stored in mbox.
I guess it depends on the size of your mails. With mboxes it can be used only with read-only archives, so it's not all that useful. With maildir it can be used transparently with read-write mailboxes. I've no idea what the performance difference or space improvements are. I guess that'll also depend on your system.
I use zlib compressed maildirs on my primary server (12000+ users) and it achieves average compression ratio of 1.8, so if your system is io-bound you can spend some cpu cycles to significantly decrease disk load.
But is that 1.8 in bytes or disk blocks? Or does it make much of a difference either way? Does 2x less space per message even matter with small messages since most disk I/O probably goes to seeking, so reading a single 4 kB message takes pretty much the same time as a 8 kB message?
Timo Sirainen wrote:
On Tue, 2008-10-07 at 11:38 +0400, Andrey Panin wrote:
On 279, 10 05, 2008 at 10:38:02PM +0300, Timo Sirainen wrote:
On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote:
Last but not least, what gain will i have by using this plugin in combination with Maildir, or will it have more impact when mails are stored in mbox. I guess it depends on the size of your mails. With mboxes it can be used only with read-only archives, so it's not all that useful. With maildir it can be used transparently with read-write mailboxes. I've no idea what the performance difference or space improvements are. I guess that'll also depend on your system. I use zlib compressed maildirs on my primary server (12000+ users) and it achieves average compression ratio of 1.8, so if your system is io-bound you can spend some cpu cycles to significantly decrease disk load.
But is that 1.8 in bytes or disk blocks? Or does it make much of a difference either way? Does 2x less space per message even matter with small messages since most disk I/O probably goes to seeking, so reading a single 4 kB message takes pretty much the same time as a 8 kB message?
Just throwing in my 0.02 currency units...
... obviously, which messages you choose to compress is up to you, and should be tuned depending on your system. Certainly, I wouldn't bother compressing anything smaller than 1 allocation block (whatever that is on your FS), and even 2, unless your CPU is really idle.
The larger the message, the greater the potential rewards.
PS: Timo... "2x smaller"? ... somehow, I expected better of you :)
-- Curtis Maloney cmaloney@cardgate.net
On 281, 10 07, 2008 at 03:10:47PM +0300, Timo Sirainen wrote:
On Tue, 2008-10-07 at 11:38 +0400, Andrey Panin wrote:
On 279, 10 05, 2008 at 10:38:02PM +0300, Timo Sirainen wrote:
On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote:
Last but not least, what gain will i have by using this plugin in combination with Maildir, or will it have more impact when mails are stored in mbox.
I guess it depends on the size of your mails. With mboxes it can be used only with read-only archives, so it's not all that useful. With maildir it can be used transparently with read-write mailboxes. I've no idea what the performance difference or space improvements are. I guess that'll also depend on your system.
I use zlib compressed maildirs on my primary server (12000+ users) and it achieves average compression ratio of 1.8, so if your system is io-bound you can spend some cpu cycles to significantly decrease disk load.
But is that 1.8 in bytes or disk blocks?
Bytes.
Or does it make much of a difference either way? Does 2x less space per message even matter with small messages since most disk I/O probably goes to seeking, so reading a single 4 kB message takes pretty much the same time as a 8 kB message?
Yes, for small messages it's not very usefull, but multimegabyte mails are quite common nowadays and for them compression is a clear win. Also I remember that load average decreased when I started to use message compression. I have no real numbers now, but it was noticeable.
-- Andrey Panin | Linux and UNIX system administrator pazke@donpac.ru | PGP key: wwwkeys.pgp.net
participants (4)
-
Andrey Panin
-
Curtis Maloney
-
Luuk
-
Timo Sirainen