[Dovecot] Moving new email from the mail spool to the inbox
We are considering switching from the Washington UW IMAP server to Dovecot for performance reasons, but we make use of the feature in the UW server that automatically moves new email from the mail spool to the IMAP INBOX. Has anyone implemented this in Dovecot, or considered implementing it ? We have a large number of users, so cannot easily change the way that we deliver email.
--
Adrian Barker, Information Systems University College London, Gower Street, London WC1E 6BT External phone: +44 20 7679 5140, Fax (+44) 20 7388 5406 Email: A.Barker@ucl.ac.uk
Hi Adrian,
I'm thinking this is more of an issue with your MTA, as usually that's responsible for delivering into the mailbox's Inbox.
You might want to look at Dovecot's LDA, "deliver" (http://wiki.dovecot.org/LDA). Deliver takes an e-mail piped from your MTA, with appropriate options on the command-line, and delivers it into the relevant mailbox correctly.
Regards,
Andy.
Adrian Barker wrote:
We are considering switching from the Washington UW IMAP server to Dovecot for performance reasons, but we make use of the feature in the UW server that automatically moves new email from the mail spool to the IMAP INBOX. Has anyone implemented this in Dovecot, or considered implementing it ? We have a large number of users, so cannot easily change the way that we deliver email.
Thanks for replying. We cannot easily change the way we deliver email, as we have over 30,000 users, who use a mixture of imap, pop and Unix email clients, so we have to continue to deliver email to a central mail spool. The MTA that we run is Exim, which has the flexibility to deliver into the 'Inbox', but we need to remain compatible with non-IMAP mailers.
Adrian Barker, University College London.
Andy Shellam wrote:
Hi Adrian,
I'm thinking this is more of an issue with your MTA, as usually that's responsible for delivering into the mailbox's Inbox.
You might want to look at Dovecot's LDA, "deliver" (http://wiki.dovecot.org/LDA). Deliver takes an e-mail piped from your MTA, with appropriate options on the command-line, and delivers it into the relevant mailbox correctly.
Regards,
Andy.
Adrian Barker wrote:
We are considering switching from the Washington UW IMAP server to Dovecot for performance reasons, but we make use of the feature in the UW server that automatically moves new email from the mail spool to the IMAP INBOX. Has anyone implemented this in Dovecot, or considered implementing it ? We have a large number of users, so cannot easily change the way that we deliver email.
--
Adrian Barker, Information Systems University College London, Gower Street, London WC1E 6BT External phone: +44 20 7679 5140, Fax (+44) 20 7388 5406 Internal phone: x 25140 Email: A.Barker@ucl.ac.uk
Adrian Barker wrote:
Thanks for replying. We cannot easily change the way we deliver email, as we have over 30,000 users, who use a mixture of imap, pop and Unix email clients, so we have to continue to deliver email to a central mail spool. The MTA that we run is Exim, which has the flexibility to deliver into the 'Inbox', but we need to remain compatible with non-IMAP mailers.
Take a look at the convert_mail plugin option. The example is:
convert_mail = mbox:/home/%u:INBOX=/var/spool/mail/%u
but something similar might help. There's some discussion in the mail archives.
-- Jonathan
Adrian Barker wrote:
We are considering switching from the Washington UW IMAP server to Dovecot for performance reasons, but we make use of the feature in the UW server that automatically moves new email from the mail spool to the IMAP INBOX. Has anyone implemented this in Dovecot, or considered implementing it ? We have a large number of users, so cannot easily change the way that we deliver email.
Andy Shellam wrote I'm thinking this is more of an issue with your MTA, as usually that's responsible for delivering into the mailbox's Inbox.
You might want to look at Dovecot's LDA, "deliver" (http://wiki.dovecot.org/LDA). Deliver takes an e-mail piped from your MTA, with appropriate options on the command-line, and delivers it into the relevant mailbox correctly.
Adrian Barker wrote: Thanks for replying. We cannot easily change the way we deliver email, as we have over 30,000 users, who use a mixture of imap, pop and Unix email clients, so we have to continue to deliver email to a central mail spool. The MTA that we run is Exim, which has the flexibility to deliver into the 'Inbox', but we need to remain compatible with non-IMAP mailers.
This is not clear at all, why does the number of users effect the delivery method? What mailbox format are you using?
-- Kenny Dail kend@amigo.net
Kenny Dail wrote:
Adrian Barker wrote: Thanks for replying. We cannot easily change the way we deliver email, as we have over 30,000 users, who use a mixture of imap, pop and Unix email clients, so we have to continue to deliver email to a central mail spool. The MTA that we run is Exim, which has the flexibility to deliver into the 'Inbox', but we need to remain compatible with non-IMAP mailers.
This is not clear at all, why does the number of users effect the delivery method? What mailbox format are you using?
I believe Adrian is saying that only some of his 30,000 clients use IMAP and he does not want to break existing behaviour for those using Unix command line clients.
-- Jonathan
Jonathan wrote:
Kenny Dail wrote:
Adrian Barker wrote: Thanks for replying. We cannot easily change the way we deliver email, as we have over 30,000 users, who use a mixture of imap, pop and Unix email clients, so we have to continue to deliver email to a central mail spool. The MTA that we run is Exim, which has the flexibility to deliver into the 'Inbox', but we need to remain compatible with non-IMAP mailers.
This is not clear at all, why does the number of users effect the delivery method? What mailbox format are you using?
I believe Adrian is saying that only some of his 30,000 clients use IMAP and he does not want to break existing behaviour for those using Unix command line clients.
That's right: we cannot change the way that we deliver email, at least in the short to medium term, so need to maintain compatibility with the UW IMAP server. If we do decide to write a plugin that copies new email from the mail spool to the IMAP inbox, is there any documentation on how to write plugins ? I could not find any on the wiki.
Adrian Barker, University College London.
Adrian Barker wrote:
That's right: we cannot change the way that we deliver email, at least in the short to medium term, so need to maintain compatibility with the UW IMAP server. If we do decide to write a plugin that copies new email from the mail spool to the IMAP inbox, is there any documentation on how to write plugins ? I could not find any on the wiki.
Adrian Barker, University College London.
Another thought occurred to me. It needn't be a plugin as such, as it just needs to run before the imap process proper - see http://wiki.dovecot.org/PostLoginScripting. It could be something as simple as a shell script doing (though with rather more safeguards):-
exim_lock /var/spool/$USER
"cat /var/spool/$USER >> ~/mbox && rm /var/spool/$USER"
exec /path/to/dovecot/libexec/imap
though you could probably write a utility using c-client to do it properly.
Best Wishes, Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tue, 8 May 2007, Chris Wakelin wrote:
Another thought occurred to me. It needn't be a plugin as such, as it just needs to run before the imap process proper - see
Three reasons why this is not enough:
- Long-time IMAP connections,
- New mail arriving during logged in and
- out-of-quota condition.
In case 3: Ideally, as soon as a mail is expunged, another pending one can be slurped.
Bye,
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iQEVAwUBRkFrmi9SORjhbDpvAQLuYggAhOnzUMuOyuk5+NpPXip0+4HFJR7Z4R90 WKFdViOeqZyiKh/Yvbn7PKT3KJPOH6jLvT7WgreBmVHbMlWdbCe7QbbQYjYQ1ujT mtR7O/Cg5Gv5B2w2uP2nPkV//j2vj0BQWSGHq/TmJYgzLyYWcchdpEXo+SrmJVX1 9EZhI2MkZWD54ikw7PrnpkSKt5cZHc8GVf7jUyG27Wk8fkhMsRHy3kyHA8s5/QJB GwMNVyY9H18W5kezbsqCJ6AGtOTC0lUqgVnt1j5oCQ2yzwW/Q158rX3VDdO5FDB7 /zpTGKPhHtIwcC9zthTfcXdpj50PEvnT2cemsYeo6CJ5j9VO4QgU4g== =4GHo -----END PGP SIGNATURE-----
On Sat, 2007-05-05 at 14:48 +0100, Adrian Barker wrote:
That's right: we cannot change the way that we deliver email, at least in the short to medium term, so need to maintain compatibility with the UW IMAP server. If we do decide to write a plugin that copies new email from the mail spool to the IMAP inbox, is there any documentation on how to write plugins ? I could not find any on the wiki.
There's currently no documentation for them. You could look at mail-log plugin or quota plugin for an example. In any case what you'd want to do is hook into sync_init() and do the moving there before calling the parent sync_init().
There are two ways to do this:
a) Dummy copying to ~/mbox. This is slower because then Dovecot needs to reread the new messages from ~/mbox and write X-UID etc. headers.
b) Open /var/mail/user as read-only mailbox with in-memory indexes. You'll need to temporarily add MAIL_STORAGE_FLAG_FULL_FS_ACCESS to storage->flags so that you can open the mailbox. Then use mailbox_copy() to copy them to ~/mbox.
In both cases after copying is successful, truncate the /var/mail/user. If something went wrong... well, you should probably at least try to not leave duplicate mails lying there. With b) option it's easy with mailbox_transaction_rollback().
See lib-storage/mail-storage.h and lib-storage/mail-storage-private.h
I guess you're delivering mail to /var/mail/username, and UW-IMAP "snarfs" (to use the UW term :)) the messages to ~/mbox (if it exists) whenever the user opens INBOX in IMAP.
If so, I guess the problem is the Unix file-system clients which expect to see mail in /var/mail/username? Dovecot can do both IMAP and POP very well, (and you can specify the mail location for each user independently if you have to).
You could configure Exim to deliver directly to the user's ~/mbox if, say, /var/mail/username is empty and ~/mbox exists, and assume that file-system clients will never have a ~/mbox or let /var/mail/username be empty. A bit yucky though!
It might also be possible to write a Dovecot plugin to emulate the UW-IMAP "snarf" behaviour, but as far as I know, nobody has.
We never used the "snarf" behaviour ourselves (except accidentally, when the new version of UW-IMAP I'd compiled had it built in by default, unnoticed by me, and a few ancient users had ~/mbox files ...)
I have to say that our UW-IMAP -> Dovecot transition was painless (transparent!) and that the performance gains were huge (otherwise we'd still be "pruning" user's INBOXes into folders for them every few months).
Best Wishes, Chris
Adrian Barker wrote:
Thanks for replying. We cannot easily change the way we deliver email, as we have over 30,000 users, who use a mixture of imap, pop and Unix email clients, so we have to continue to deliver email to a central mail spool. The MTA that we run is Exim, which has the flexibility to deliver into the 'Inbox', but we need to remain compatible with non-IMAP mailers.
Adrian Barker, University College London.
Andy Shellam wrote:
Hi Adrian,
I'm thinking this is more of an issue with your MTA, as usually that's responsible for delivering into the mailbox's Inbox.
You might want to look at Dovecot's LDA, "deliver" (http://wiki.dovecot.org/LDA). Deliver takes an e-mail piped from your MTA, with appropriate options on the command-line, and delivers it into the relevant mailbox correctly.
Regards,
Andy.
Adrian Barker wrote:
We are considering switching from the Washington UW IMAP server to Dovecot for performance reasons, but we make use of the feature in the UW server that automatically moves new email from the mail spool to the IMAP INBOX. Has anyone implemented this in Dovecot, or considered implementing it ? We have a large number of users, so cannot easily change the way that we deliver email.
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
Quoting Chris Wakelin c.d.wakelin@reading.ac.uk:
I guess you're delivering mail to /var/mail/username, and UW-IMAP "snarfs" (to use the UW term :)) the messages to ~/mbox (if it exists) whenever the user opens INBOX in IMAP.
Yes, we used to do this also, but we stopped when we went to dovecot.
Since we're a small installation, we just merged the ~mbox files back into the /var/spool/mail/username files...
Since the person posting the question has 30K users, this may not be feasible. Depends on what percentage of them use the ~mbox feature.
It might also be possible to write a Dovecot plugin to emulate the UW-IMAP "snarf" behaviour, but as far as I know, nobody has.
AFAIK there is none also, but one would be "way cool" to have...
I have to say that our UW-IMAP -> Dovecot transition was painless (transparent!) and that the performance gains were huge (otherwise we'd still be "pruning" user's INBOXes into folders for them every few months).
I agree completely. No one noticed anything except for the blazing performance change, and I also was able to stop "pruning" user's inboxes. Was not only painless, but removed a lot of existing pain!
Best Wishes, Chris
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
participants (8)
-
Adrian Barker
-
Andy Shellam
-
Chris Wakelin
-
Eric Rostetter
-
Jonathan
-
Kenny Dail
-
Steffen Kaiser
-
Timo Sirainen