[Dovecot] Problem with hard links in lda - please help
Hello,
I'm trying to enable hardlinks for messages sent to mutiple users. (I need this because I have mailing lists with 5000 users used many times a day).
I've read that, to do this, I have to write a script that uses the /usr/local/libexec/dovecot/deliver command in this way: deliver -p <FILE> -d <USER1> -d <USER2>
Of course, I enabled the "socket listen" option in the configuration file to get the command line work, and set the maildir_copy_with_hardlinks = yes
The deliver command works, so I have the messages correctly in the "new" directory in the maildir folder.
But if I list the files, they have the "links" number = 1, so I see that the file was copied, not hardlinked!
I see in the page http://wiki.dovecot.org/LDA that is possible, in fact it says
Parameters [...] -p <path>: Path to the mail to be delivered instead of reading from stdin. If using maildir the file is hard linked to the destination if possible. This allows a single mail to be delivered to multiple users using hard links, but currently it also prevents deliver from updating cache file so it shouldn't be used unless really necessary. (v1.1+)
Of course, I use a version 1.1+ (1.2.12).
Please help, thanks in advance
Fabio Ferrari
On Fri, 2010-07-16 at 15:44 +0200, Fabio Ferrari wrote:
I'm trying to enable hardlinks for messages sent to mutiple users. (I need this because I have mailing lists with 5000 users used many times a day).
I've read that, to do this, I have to write a script that uses the /usr/local/libexec/dovecot/deliver command in this way: deliver -p <FILE> -d <USER1> -d <USER2>
You can't give more than -d parameter. deliver can deliver a mail only to a single user at a time.
The important part is that your MTA calls a script (that you need to create) that gets multiple users as parameters, and then you can call deliver -p for each one of them separately. I don't think MTA can call the delivery script with 5000 parameters, so there must be some duplication always.
Also mailing lists often send separate mails to users. If this is happening, hard linking can't be done because the mails are different.
On 2010-07-16 10:13 AM, Timo Sirainen wrote:
The important part is that your MTA calls a script (that you need to create) that gets multiple users as parameters, and then you can call deliver -p for each one of them separately. I don't think MTA can call the delivery script with 5000 parameters, so there must be some duplication always.
Also mailing lists often send separate mails to users. If this is happening, hard linking can't be done because the mails are different.
So... is this a bad time to ask about status of Single Instance Storage? Have you even had time to give it any more thought, or even better, actually started coding anything?
Sorry, its just that this is one of my biggest wishes...
Thanks
Charles
On Fri, 2010-07-16 at 14:16 -0400, Charles Marcus wrote:
So... is this a bad time to ask about status of Single Instance Storage?
I've been meaning to write a mail about this.. Probably will send it to list on Monday.
Have you even had time to give it any more thought, or even better, actually started coding anything?
Yes and yes. It's a very isolated piece of code (i.e. no chance of breaking anything if it's disabled), so maybe I'll even add it to v2.0 when it looks stable.
On 2010-07-16 3:01 PM, Timo Sirainen wrote:
On Fri, 2010-07-16 at 14:16 -0400, Charles Marcus wrote:
So... is this a bad time to ask about status of Single Instance Storage?
I've been meaning to write a mail about this.. Probably will send it to list on Monday.
Have you even had time to give it any more thought, or even better, actually started coding anything?
Yes and yes. It's a very isolated piece of code (i.e. no chance of breaking anything if it's disabled), so maybe I'll even add it to v2.0 when it looks stable.
And the crowd went wild! :)
Thanks Timo, I look forward to your email elaborating...
Have a great weekend!
Yes, I tried in many ways, the
deliver -p <FILE> -d <USER1> -d <USER2>
is only one of the ways I tried to hardlink a message. I tried even to do
deliver -p <FILE> -d <USER1> deliver -p <FILE> -d <USER2> ... deliver -p <FILE> -d <USERN>
but all commands have the same result, that is the message is copied and not hardlinked.
The question is, do I have a way to make hardlinks for the same message with dovecot lda or I have to write my own script that hardlinks the file in the correct directories bypassing lda?
thank in advance for the answer
Fabio Ferrari
On Fri, 2010-07-16 at 15:44 +0200, Fabio Ferrari wrote:
I'm trying to enable hardlinks for messages sent to mutiple users. (I need this because I have mailing lists with 5000 users used many times a day).
I've read that, to do this, I have to write a script that uses the /usr/local/libexec/dovecot/deliver command in this way: deliver -p <FILE> -d <USER1> -d <USER2>
You can't give more than -d parameter. deliver can deliver a mail only to a single user at a time.
The important part is that your MTA calls a script (that you need to create) that gets multiple users as parameters, and then you can call deliver -p for each one of them separately. I don't think MTA can call the delivery script with 5000 parameters, so there must be some duplication always.
Also mailing lists often send separate mails to users. If this is happening, hard linking can't be done because the mails are different.
On 20.7.2010, at 11.03, Fabio Ferrari wrote:
deliver -p <FILE> -d <USER1> deliver -p <FILE> -d <USER2> ... deliver -p <FILE> -d <USERN>
This is how it should be done. The file was exactly the same between all the calls? Did you try running this manually from command line instead of via MTA? Do you have maildir_copy_with_hardlinks=yes?
On 20.7.2010, at 11.03, Fabio Ferrari wrote:
deliver -p <FILE> -d <USER1> deliver -p <FILE> -d <USER2> ... deliver -p <FILE> -d <USERN>
This is how it should be done. The file was exactly the same between all the calls? Did you try running this manually from command line instead of via MTA? Do you have maildir_copy_with_hardlinks=yes?
The answer is: yes.
I called it via command line, the 'diff' command shows no differences between the two destination files (correctly delivered in the 'new' directory in the two maildirs) and I have the maildir_copy_with_hardlinks=yes line in the /usr/local/etc/dovecot.conf file.
Has anyone tested this function or it is a non-existing feature and the documentation about hardlinks means something else from what I mean?
Fabio Ferrari
On Tue, 2010-07-20 at 12:53 +0200, Fabio Ferrari wrote:
deliver -p <FILE> -d <USER1> deliver -p <FILE> -d <USER2> ... deliver -p <FILE> -d <USERN>
I noticed one thing: The created file's permissions must be the same as the destination maildir's. So typically the file needs to be 0600 mode. Also all of the users must have the same UID.
Ok, this was the problem, if I manually chown the maildirs it makes the hardlinks.
Then, I tried to enable virtual users in postfix and dovecot, to make it work in a more senseful environment. (I'm using passwd files)
But now the deliver -p FILE -d USER crashes, giving me a very bad segmentation fault.
Any ideas to debug the problem, of course no logs are written?
Fabio Ferrari
On Tue, 2010-07-20 at 12:53 +0200, Fabio Ferrari wrote:
deliver -p <FILE> -d <USER1> deliver -p <FILE> -d <USER2> ... deliver -p <FILE> -d <USERN>
I noticed one thing: The created file's permissions must be the same as the destination maildir's. So typically the file needs to be 0600 mode. Also all of the users must have the same UID.
On 23.7.2010, at 11.22, Fabio Ferrari wrote:
But now the deliver -p FILE -d USER crashes, giving me a very bad segmentation fault.
Often it happens if you have a something=no setting as the first setting in config file. This triggers a glibc bug and crashes it. Move the setting down.
Ok it works now! Thank you very much for your help!
Fabio Ferrari
On 23.7.2010, at 11.22, Fabio Ferrari wrote:
But now the deliver -p FILE -d USER crashes, giving me a very bad segmentation fault.
Often it happens if you have a something=no setting as the first setting in config file. This triggers a glibc bug and crashes it. Move the setting down.
participants (3)
-
Charles Marcus
-
Fabio Ferrari
-
Timo Sirainen