[Dovecot] LMTP revisited
During the early Dovecot LDA brainstorming discussions, there was some talk about the possibility of including an LMTP interface, and I've noticed that there's an "is it needed?" question about LMTP in Dovecot's TODO file.
I'd like to put in a vote for LMTP, and I'm wondering if anyone else has a need for it. Specifically, I'd love an LMTP-enabled LDA that uses hard links when when delivering the same message to multiple recipients (I'm thinking Maildir files), as it would be a nice way to facilitate so-called Single Instance Storage.
Glenn
Quoting Glenn Leavell:
I'd like to put in a vote for LMTP, and I'm wondering if anyone else has a
Seconded. I think it's The Right Thing. I'm thinking for a while now about the possibility of a plugin in dovecot itself, not some (semi-)external program, that listens to a unix or tcp socket for lmtp connections, so there's no overhead for fork/exec etc. Don't know if that's possible with the current plugin interface, though, and I myself will not be able to look into that in the near future, let alone casting it into working code. (additionally, right now I have no real need for that right now...) I'd not need the sieve stuff (my MTA is already doing that), but it should be able to put the message into a subfolder (e.g. with user+subfolder@domain).
need for it. Specifically, I'd love an LMTP-enabled LDA that uses hard links when when delivering the same message to multiple recipients (I'm thinking Maildir files), as it would be a nice way to facilitate so-called Single Instance Storage.
Don't know if dovecot's storage API allows that. It would be a nice thing, but there are implications (e.g. quota) and the overall advantage is not that big, so I don't think it'd be worth the hassle.
On 6/17/06, Jakob Hirsch jh@plonk.de wrote:
need for it. Specifically, I'd love an LMTP-enabled LDA that uses hard links when when delivering the same message to multiple recipients (I'm thinking Maildir files), as it would be a nice way to facilitate so-called Single Instance Storage.
Don't know if dovecot's storage API allows that. It would be a nice thing, but there are implications (e.g. quota) and the overall advantage is not that big, so I don't think it'd be worth the hassle.
It some instances, the advantage is great. Quota shouldn't be affected, depending on what quota methods you use. This is something I really want, as I know we will be mailing large numbers of our users, the same message, which goes to an alias, that expands to each user. Atm, there are plans for a 'runner' to handle these mailings, but if dovecot-lda will do this hardlinking magic, then I don't need to write a runner!
(P.s. I don't know much about LMTP, so can't put a vote in, until I understand it better)
Tim
Linux Counter user #273956
On Fri, Jun 16, 2006 at 09:26:52PM -0400, Glenn Leavell wrote:
During the early Dovecot LDA brainstorming discussions, there was some talk about the possibility of including an LMTP interface, and I've noticed that there's an "is it needed?" question about LMTP in Dovecot's TODO file.
I'd like to put in a vote for LMTP, and I'm wondering if anyone else has a need for it. Specifically, I'd love an LMTP-enabled LDA that uses hard links when when delivering the same message to multiple recipients (I'm thinking Maildir files), as it would be a nice way to facilitate so-called Single Instance Storage.
hardlinks only work if the mailboxes reside on the same filesystem, so the choice of hardlink or symlink would be useful, if this was being implemented. on one of my systems, each user's mailbox is a separate ZFS filesystem. :)
grant.
grant beattie wrote:
hardlinks only work if the mailboxes reside on the same filesystem, so the choice of hardlink or symlink would be useful, if this was being implemented. on one of my systems, each user's mailbox is a separate ZFS filesystem. :)
grant.
For those of us lucky enough to be able to live with Solaris's hardware support... personally I am frothing at the mouth for ZFS support in linux. We should have it via FUSE in a month or two thanks to Google's Summer of Code sponsorship, which is not quite optimal for speed but should still be quite nice.
On Sat, 2006-06-17 at 05:20, grant beattie wrote:
I'd like to put in a vote for LMTP, and I'm wondering if anyone else has a need for it. Specifically, I'd love an LMTP-enabled LDA that uses hard links when when delivering the same message to multiple recipients (I'm thinking Maildir files), as it would be a nice way to facilitate so-called Single Instance Storage.
hardlinks only work if the mailboxes reside on the same filesystem, so the choice of hardlink or symlink would be useful, if this was being implemented. on one of my systems, each user's mailbox is a separate ZFS filesystem. :)
Hardlinks do the right thing when the last one is deleted. With symlinks you wouldn't know when to remove the target file.
--
Les Mikesell lesmikesell@gmail.com
On Fri, 2006-06-16 at 21:26 -0400, Glenn Leavell wrote:
I'd like to put in a vote for LMTP, and I'm wondering if anyone else has a need for it. Specifically, I'd love an LMTP-enabled LDA that uses hard links when when delivering the same message to multiple recipients (I'm thinking Maildir files), as it would be a nice way to facilitate so-called Single Instance Storage.
This doesn't require LMTP. It could be implemented just as well with the current Dovecot LDA.
The one thing that disturbs me with this kind of hardlinking is that then there are no Delivered-To headers (and others?) which are normally added to mails.
I'm not planning to implement LMTP server anytime soon, but single instance storage is planned for dbox format. Maildir-hardlinking in LDA could probably implemented while doing it.
On Sat, Jun 17, 2006 at 02:14:27PM +0300, Timo Sirainen wrote:
On Fri, 2006-06-16 at 21:26 -0400, Glenn Leavell wrote:
I'd like to put in a vote for LMTP, and I'm wondering if anyone else has a need for it. Specifically, I'd love an LMTP-enabled LDA that uses hard links when when delivering the same message to multiple recipients (I'm thinking Maildir files), as it would be a nice way to facilitate so-called Single Instance Storage.
This doesn't require LMTP. It could be implemented just as well with the current Dovecot LDA.
I think I'm probably showing my ignorance here: My understanding is that Dovecot's LDA gets called once per recipient, so that it wouldn't know whether any given message is a duplicate. Or is the idea that Dovecot's LDA would be able to easily detect duplicates based on its own index information? If that's the case, would you start to include inode data in Dovecot's index/metadata in order to perform the Maildir linking?
The reason I was thinking along the lines of LMTP is that it inherently accepts multiple recipients for a single message, and I wasn't thinking about the possibility of Dovecot's LDA being able to pick up on them in some other way.
The one thing that disturbs me with this kind of hardlinking is that then there are no Delivered-To headers (and others?) which are normally added to mails.
Yes, I guess we'd have to take that into account when deciding whether to use single instance storage.
I'm not planning to implement LMTP server anytime soon, but single instance storage is planned for dbox format. Maildir-hardlinking in LDA could probably implemented while doing it.
Thank you!
Glenn
On Jun 18, 2006, at 5:21 AM, Glenn Leavell wrote:
This doesn't require LMTP. It could be implemented just as well
with the current Dovecot LDA.I think I'm probably showing my ignorance here: My understanding
is that Dovecot's LDA gets called once per recipient,
That's how it normally works, and it doesn't support anything else
currently. But at least Postfix supports giving it multiple
recipients by increasing transport_destination_recipient_limit setting.
On Sun, Jun 18, 2006 at 12:11:08PM +0300, Timo Sirainen wrote:
On Jun 18, 2006, at 5:21 AM, Glenn Leavell wrote:
This doesn't require LMTP. It could be implemented just as well
with the current Dovecot LDA.I think I'm probably showing my ignorance here: My understanding
is that Dovecot's LDA gets called once per recipient,That's how it normally works, and it doesn't support anything else
currently. But at least Postfix supports giving it multiple
recipients by increasing transport_destination_recipient_limit setting.
In this sort of configuration, does Dovecot's LDA provide individual status reports for each local recipient in a way that Postfix (or MTAs in general) can understand? I know that this is a key feature of LMTP, because it allows MTAs to be smarter about how to handle failed local deliveries. For example, without status reports for each individual recipient, an MTA will be forced to retry an entire batch of local deliveries if any one recipient fails with a transient error, causing duplicate mail for some of the recipients.
Thanks, Glenn
On Mon, 2006-06-19 at 15:22 -0400, Glenn Leavell wrote:
That's how it normally works, and it doesn't support anything else
currently. But at least Postfix supports giving it multiple
recipients by increasing transport_destination_recipient_limit setting.In this sort of configuration, does Dovecot's LDA provide individual status reports for each local recipient in a way that Postfix (or MTAs in general) can understand? I know that this is a key feature of LMTP, because it allows MTAs to be smarter about how to handle failed local deliveries. For example, without status reports for each individual recipient, an MTA will be forced to retry an entire batch of local deliveries if any one recipient fails with a transient error, causing duplicate mail for some of the recipients.
I don't know really. I'd guess that since such feature exists there's a reasonable way to return temporary errors for some of the recipients. But perhaps not.. Guess I'll have to check this before I start implementing anything.
Quoting Glenn Leavell:
I think I'm probably showing my ignorance here: My understanding is that Dovecot's LDA gets called once per recipient, so that it wouldn't know whether any given message is a duplicate. Or is the idea that Dovecot's
That's the common way to use LDAs. On many systems, mailboxes belong to seperate users, so for delivering into multiple mailboxes, a LDA would need to have root rights, either by being called from a root process (at least Postfix and Exim will not do that) or be SUID root. Procmail is said to be able to do multi-deliveries this way, but it's an ugly piece of software.
LDA would be able to easily detect duplicates based on its own index information?
That would be unnecessarily cumbersome.
The one thing that disturbs me with this kind of hardlinking is that then there are no Delivered-To headers (and others?) which are normally added to mails. Yes, I guess we'd have to take that into account when deciding whether to use single instance storage.
Are they necessary for anything special? Exim does not set this header (unless specially configured to do so), its default header is Envelope-to, which contains the list of envelope recipients.
Anyway, if you have so many large messages going to multiple recipients, couldn't these people be put into groups and use shared mailboxes?
On Sun, Jun 18, 2006 at 01:28:51PM +0200, Jakob Hirsch wrote:
Quoting Glenn Leavell:
I think I'm probably showing my ignorance here: My understanding is that Dovecot's LDA gets called once per recipient, so that it wouldn't know whether any given message is a duplicate. Or is the idea that Dovecot's
That's the common way to use LDAs. On many systems, mailboxes belong to seperate users, so for delivering into multiple mailboxes, a LDA would need to have root rights, either by being called from a root process (at least Postfix and Exim will not do that) or be SUID root. Procmail is said to be able to do multi-deliveries this way, but it's an ugly piece of software.
LDA would be able to easily detect duplicates based on its own index information?
That would be unnecessarily cumbersome.
Perhaps so. Timo mentioned that the Dovecot LDA could be modified to handle Single Instance Storage without LMTP. Because I'm accustomed to the LDA being called once per recipient (without LMTP, of course), I was just wondering how he planned for the LDA to keep up with duplicate messages (the indexes came to mind). But based on his response, it sounds like his thinking that the MTA (Postfix in his example) would call the Dovecot LDA once with multiple recipients as arguments.
The one thing that disturbs me with this kind of hardlinking is that then there are no Delivered-To headers (and others?) which are normally added to mails. Yes, I guess we'd have to take that into account when deciding whether to use single instance storage.
Are they necessary for anything special? Exim does not set this header (unless specially configured to do so), its default header is Envelope-to, which contains the list of envelope recipients.
I don't think Delivered-To: is critical, but apparently some systems use Delivered-To: to help with loop detection. This is from Postfix's local(8) man page:
In order to stop mail forwarding loops early, the
software adds an optional Delivered-To: header with
the final envelope recipient address. If mail arrives for
a recipient that is already listed in a Delivered-To:
header, the message is bounced.And D. J. Bernstein mentions it in his "Tools in the war on mail loops" document [ http://cr.yp.to/proto/mailloops.txt ]:
The basic weapon against post-delivery loops is a new header
field, Delivered-To, tracing all the forwarders and mailing
lists that a message has been through. This field has the
side benefit of making it much easier for a user (or for a
postmaster seeing a bounce) to figure out the path that the
message took. Delivered-To is similar to RFC 1327's
DL-Expansion-History, but (1) it omits the time stamp,
removing any need for parsing, and (2) it has a much better
name.Anyway, if you have so many large messages going to multiple recipients, couldn't these people be put into groups and use shared mailboxes?
A solution like that may well help in some of our situations. But that solution has at least some political implications (users experience change!), and I was focusing on Single Instance Storage as a transparent feature that might help to some degree.
Thanks, Glenn
participants (7)
-
Glenn Leavell
-
grant beattie
-
Jakob Hirsch
-
Les Mikesell
-
Peter Fern
-
Timo Sirainen
-
Timothy White