Re: Questions about hardlinks, alternate storage and compression
Hi Javier, thanks for your reply.
I already checked SIS and, while interesting, is not what I want, because:
- it can be difficult to restore a single message/attachment from a backup
- only the attachments, and not the entire messages, are deduped.
Message-based hardlinks really exists for a reason. The good news is that I found _why_ they are not working: it depends from how dovecot and its sieve plugin (pigenhole) interact. Basically, if I define anything for the before_sieve and after_sieve variables, dovecot stops creating hardlinks for multiple copies of the same message.
On the other hand, private (per-user) sieve file works without interfering with hardlinks. In a similar manner, disabling sieve also permits dovecot to create multiple hardlinks for a single message.
Does someone know if newer dovecot versions change anything in this regard?
Thank you all.
On 13/07/15 11:10, Javier Miguel RodrÃguez wrote:
Search about "single instance storage dovecot". This is what you need.
Regards
Javier
On 27/06/2015 18:18, Gionatan Danti wrote:
Hi all, I have some questions about hardlinks, alternate storage and compression. I already scanned the list for related information and I have an idea of how things works, but I would like to have a definite answer.
System spec:
- CentOS 6.6 x64
- dovecot-2.0.9-8.el6_6.4.x86_64 RPM package/version
- sdbox mail store
About hardlinks: when sending the same message to two different recipients, I see that the two u.x files are created as two different files. Diffing them, I see that the only difference is a single char (see [1] for an example). My questions are: a) it is possible to tell dovecot to create a single file + a single hardlink (linkref=2)? As other IMAP servers support that features (eg: Cyrus, CommunigatePro, etc) I am wondering if I missed something in configuring dovecot... b) If it is not possible, can I run a script that compare the various files and substitute equal ones (minus the changing line) with hardlinks, or it will confuse dovecot? As a side note, why the changing line ever exists?
About alternate storage and compression: actually, I use a single mail_location without compression. I would like to have an alternate storage and to enable compression on it only, leaving the main location without compression. I if understand it correctly, it _should_ be done using a command similar to "doveadm -Dv -o "plugin/zlib_save=gz" altmove -uu testuser sentbefore 8d". I'm right thinking that it should work? I will really end with a primary uncompressed mail store and an alternate, zlib-compressed one?
Thank you all and sorry if I did some naive questions.
[1] 63c63 < G2fd0811c64be8e553d970000eaa8309f
G2ed0811c64be8e553d970000eaa8309f
-- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Mon, 13 Jul 2015, Gionatan Danti wrote:
On the other hand, private (per-user) sieve file works without interfering with hardlinks. In a similar manner, disabling sieve also permits dovecot to create multiple hardlinks for a single message.
Does someone know if newer dovecot versions change anything in this regard?
LMTP adds Delivered-To header, so all delivered messages are unique and you cannot hardlink messages regardless of Sieve.
If Dovecot LDA adds headers, too, I do not know.
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1
iQEVAwUBVaSpX3z1H7kL/d9rAQKuPwf/e4GddZvm/qj9sfAnVgV3H5iC62fnS6Ny /TPaXcuLcN5Tx9slhLTwIx8/GRROUVwLVqKYjaXwQciV2yytBu5vkC0+lowIZGq9 kJAAKPp4h3Ia6SDGhI8E5Im9VGGSpbXyLKR+V3rf1G/sOyvJTITliVe4ckf76xrI c1LGYumW0BGZeNZAAA0lYHZGrgy5meCrL20CMupmahoHsOFw5cA3HhJ/dEBRPlOJ y886BScRh7dWJXyS+PUzPFlbFOeULKvh6fVwCK7b4+aFkfjLedDLew5TThWiblK5 c5+rx0pAh8xVdXGZyQXzPjUl22KbQmGfzv78XWlN2WksCnMVaFXe2g== =3iPP -----END PGP SIGNATURE-----
On 14/07/15 08:17, Steffen Kaiser wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Mon, 13 Jul 2015, Gionatan Danti wrote:
On the other hand, private (per-user) sieve file works without interfering with hardlinks. In a similar manner, disabling sieve also permits dovecot to create multiple hardlinks for a single message.
Does someone know if newer dovecot versions change anything in this regard?
LMTP adds Delivered-To header, so all delivered messages are unique and you cannot hardlink messages regardless of Sieve.
If Dovecot LDA adds headers, too, I do not know.
Mmm... I'm using LMTP, but I can't find the "Delivered-To" header. Below you can see an example of successfully hard-linked email [1]
I am missing something?
[1] Return-Path: g.danti@assyoma.it Received: from mail.gruppocrimi.it by mail.gruppocrimi.it (Dovecot) with LMTP id VFA8Fj3OmlUStwAA6qgwnw ; Mon, 06 Jul 2015 20:51:41 +0200 Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.gruppocrimi.it (Postfix) with ESMTP id 22AB4A1A85; Mon, 6 Jul 2015 20:51:41 +0200 (CEST) X-Virus-Scanned: amavisd-new at gruppocrimi.it Received: from mail.gruppocrimi.it ([127.0.0.1]) by localhost (mail.gruppocrimi.it [127.0.0.1]) (amavisd-new, port 10024) with LMTP id NC3YcizeDFPO; Mon, 6 Jul 2015 20:51:40 +0200 (CEST) Received: from mr003msb.fastweb.it (mr003msb.fastweb.it [85.18.95.87]) by mail.gruppocrimi.it (Postfix) with ESMTP id 4380DA1A7C; Mon, 6 Jul 2015 20:51:40 +0200 (CEST) Received: from ceres.assyoma.it (93.63.55.57) by mr003msb.fastweb.it (8.5.140.03) id 55501C9F0432631D; Mon, 6 Jul 2015 20:51:40 +0200 Received: by ceres.assyoma.it (Postfix, from userid 48) id B7B912643B4; Mon, 6 Jul 2015 20:51:39 +0200 (CEST) To: gionatan.danti@gruppocrimi.it, g.danti@gruppocrimi.it Subject: test invio X-PHP-Originating-Script: 0:rcube.php MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Mon, 06 Jul 2015 20:51:39 +0200 From: Gionatan Danti g.danti@assyoma.it Organization: Assyoma s.r.l. Message-ID: ef3149d82ef4350c1c9b6da9c1f03fae@assyoma.it X-Sender: g.danti@assyoma.it User-Agent: Roundcube Webmail/1.0.5
-- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tue, 14 Jul 2015, Gionatan Danti wrote:
On 14/07/15 08:17, Steffen Kaiser wrote:
On Mon, 13 Jul 2015, Gionatan Danti wrote:
On the other hand, private (per-user) sieve file works without interfering with hardlinks. In a similar manner, disabling sieve also permits dovecot to create multiple hardlinks for a single message.
Does someone know if newer dovecot versions change anything in this regard?
LMTP adds Delivered-To header, so all delivered messages are unique and you cannot hardlink messages regardless of Sieve.
If Dovecot LDA adds headers, too, I do not know.
Mmm... I'm using LMTP, but I can't find the "Delivered-To" header. Below you can see an example of successfully hard-linked email [1]
You asked about "newer dovecot versions", v2.2 does so.
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1
iQEVAwUBVaTjzXz1H7kL/d9rAQK1oAf/fdUuBY8kseVEFa5kyXG01cyUjc3RfBNl o0EYm+e2hvoz5B4N96pkbmilYjaCtgUw/qlMnGkzFbmJDwrqOiAhxOG71Aewjvbx q42cXHtw7CsOCr6y+eshNUfU3T20f7wgvyJDqLAOwg/pSP3CjU9m93D2zCqUgDXO MHuDV1zEEljlrxXmtdG8GI5YlwkBqvWXQuPbXr7PhoQ4HTKhvKHWurGvVkfBlg6k cpuy40mSWY3ZXwNDcnHP0o82EezGAdgzDE/EoV4fV0JDvANbTjpwwqE4gMW+wOM+ lUJnMyawkVuvfbB85K/tkK+a0lIVnZOwdUy0RaUcJFeZHXdRsixvIg== =HzBs -----END PGP SIGNATURE-----
On 14/07/15 12:26, Steffen Kaiser wrote:
You asked about "newer dovecot versions", v2.2 does so.
Fair enough :)
So, with v2.2+ the hardlink approach is irremediably gone, at least with LMTP (and without relying to SiS)?
-- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
Il 14-07-2015 14:44 Gionatan Danti ha scritto:
On 14/07/15 12:26, Steffen Kaiser wrote:
You asked about "newer dovecot versions", v2.2 does so.
Fair enough :)
So, with v2.2+ the hardlink approach is irremediably gone, at least with LMTP (and without relying to SiS)?
Dear list, sorry if I resume this (relatively) old post, but I would like to know if someone has some good suggestions/ideas.
A quick recap... System spec:
- CentOS 6.6 x64
- dovecot-2.0.9-8.el6_6.4.x86_64 RPM package/version
- sdbox mail store
I have two problems: possible?
- hardlinks are not created when the same email is sent to multiple rcpt, nor when sending to the very same user (a non-linked copy is kept both in Inbox and Sent)
- I would like to have an alternate storage and to enable compression on it only, leaving the main location without compression. It is
Relative to the hard-link problem, I found that not using before_sieve and after_sieve solves my problem, so that hardlinks are created correctly. However, this is far from ideal because I lose global sieve rules (global/defaul rules are only applied when no customized rules exist).
Any idea to how to solve these two problem? Thanks.
-- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
Am Dienstag, den 14.07.2015, 12:26 +0200 schrieb Steffen Kaiser:
On Tue, 14 Jul 2015, Gionatan Danti wrote:
On 14/07/15 08:17, Steffen Kaiser wrote:
On Mon, 13 Jul 2015, Gionatan Danti wrote:
On the other hand, private (per-user) sieve file works without interfering with hardlinks. In a similar manner, disabling sieve also permits dovecot to create multiple hardlinks for a single message.
Does someone know if newer dovecot versions change anything in this regard?
LMTP adds Delivered-To header, so all delivered messages are unique and you cannot hardlink messages regardless of Sieve.
If Dovecot LDA adds headers, too, I do not know.
Mmm... I'm using LMTP, but I can't find the "Delivered-To" header. Below you can see an example of successfully hard-linked email [1]
You asked about "newer dovecot versions", v2.2 does so.
I just updated my Dovecot 2.2.13 to the current 2.2.18 This config option was added in the meanwhile:
# Which recipient address to use for Delivered-To: header and Received: # header. The default is "final", which is the same as the one given to # RCPT TO command. "original" uses the address given in RCPT TO's ORCPT # parameter, "none" uses nothing. Note that "none" is currently always used # when a mail has multiple recipients. #lmtp_hdr_delivery_address = final
Doestn't that mean if you set it to none that no Delived-To: header gets added then?
participants (3)
-
Felix Zielcke
-
Gionatan Danti
-
Steffen Kaiser