[Dovecot] Server-side sieve for client-side copies
I know I could test this - but I'd rather ask first. To my knowledge, we haven't come up with a good server-side implementation for savings copies of sent messages (by all means correct my ignorance in this regard). So the typical way is to enable it in clients like Thunderbird (this prompted a whole discussion of how to save these messages without sending them to the server multiple times).
My understanding is that sending a message from a client (use
Thunderbird for simplicity of this conversation) is performed via SMTP.
Saving a copy into a sent folder is performed via IMAP (hence the
multiple transfers to the server). Now that I've laid a background -
let's make it Dovecot specific. I don't know how "behind-the-scenes"
Dovecot performs the act of saving mail messages that it receives from
IMAP, instead of SMTP. Specifically, if I'm using sieve filters via
deliver - can I setup a filter that will place mail copies to specific
recipients into specific subfolders?
Daniel
Daniel L. Miller escreveu:
My understanding is that sending a message from a client (use Thunderbird for simplicity of this conversation) is performed via SMTP. Saving a copy into a sent folder is performed via IMAP (hence the multiple transfers to the server). Now that I've laid a background - let's make it Dovecot specific. I don't know how "behind-the-scenes" Dovecot performs the act of saving mail messages that it receives from IMAP, instead of SMTP. Specifically, if I'm using sieve filters via deliver - can I setup a filter that will place mail copies to specific recipients into specific subfolders?
Thunderbird and other mail clients store a copy of the sent message by using the IMAP APPEND command, which is a generic command to store a mail message (or anything else, if you are brave enough) in a mailbox.
Sieve filters are run for messages that are received via SMTP (usually from another host), and need to be saved in the user's mailbox. (Provided the mail server is configured to use dovecot's deliver to do that.)
So, in short, these are two very different processes, and no filtering is done for messages saved in a folder (such as Sent).
on 7-28-2008 9:38 AM Daniel L. Miller spake the following:
I know I could test this - but I'd rather ask first. To my knowledge, we haven't come up with a good server-side implementation for savings copies of sent messages (by all means correct my ignorance in this regard). So the typical way is to enable it in clients like Thunderbird (this prompted a whole discussion of how to save these messages without sending them to the server multiple times).
My understanding is that sending a message from a client (use Thunderbird for simplicity of this conversation) is performed via SMTP.
Saving a copy into a sent folder is performed via IMAP (hence the multiple transfers to the server). Now that I've laid a background - let's make it Dovecot specific. I don't know how "behind-the-scenes" Dovecot performs the act of saving mail messages that it receives from IMAP, instead of SMTP. Specifically, if I'm using sieve filters via deliver - can I setup a filter that will place mail copies to specific recipients into specific subfolders? Not easy, but set client to BCC yourself, and set filters for messages from you/to you. Until they modify the IMAP spec, and then every mail client is updated, it just isn't going to be an easy fix.
-- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't!!!!
Hi,
On Mon, 28 Jul 2008 09:38:19 -0700, "Daniel L. Miller" <dmiller@amfes.com> wrote:
My understanding is that sending a message from a client (use Thunderbird for simplicity of this conversation) is performed via SMTP.
Saving a copy into a sent folder is performed via IMAP (hence the multiple transfers to the server). Now that I've laid a background - let's make it Dovecot specific. I don't know how "behind-the-scenes" Dovecot performs the act of saving mail messages that it receives from IMAP, instead of SMTP. Specifically, if I'm using sieve filters via deliver - can I setup a filter that will place mail copies to specific recipients into specific subfolders?
It seems to me, that you are assuming that Dovecot also speaks SMTP and is able to actually handle sending of mails to other hosts. To clarify: Dovecot is an IMAP/POP3 mail server (with integrated Mail Delivery Agent (MDA) or Local Delivery Agent (LDA) (synonyms)). SMTP is being handled by a Mail Transfer Agent (MTA) such as Postfix.
So as you can see, the Mail User Agent (MUA) (for example Thunderbird) interacts with two different server programs, one for sending mail (via SMTP), one for accessing stored mail (via IMAP/POP3).
The only way I can see to make the double uploading go away, would be, to change the MTA's configuration so that it automatically sends a blind carbon copy of the outgoing message to the sender. Then set up a sieve rule in Dovecot to file this mail into the Sent folder. Of course don't forget to switch off the 'copy sent mails to Sent folder' feature in every mail client which is using this setup.
Patrick.
-- STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key: https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005
Hi,
On Mon, 28 Jul 2008 09:38:19 -0700, "Daniel L. Miller" <dmiller@amfes.com> wrote:
My understanding is that sending a message from a client (use Thunderbird for simplicity of this conversation) is performed via SMTP.
Saving a copy into a sent folder is performed via IMAP (hence the multiple transfers to the server). Now that I've laid a background - let's make it Dovecot specific. I don't know how "behind-the-scenes" Dovecot performs the act of saving mail messages that it receives from IMAP, instead of SMTP. Specifically, if I'm using sieve filters via deliver - can I setup a filter that will place mail copies to specific recipients into specific subfolders?It seems to me, that you are assuming that Dovecot also speaks SMTP and is able to actually handle sending of mails to other hosts. To clarify: Dovecot is an IMAP/POP3 mail server (with integrated Mail Delivery Agent (MDA) or Local Delivery Agent (LDA) (synonyms)). SMTP is being handled by a Mail Transfer Agent (MTA) such as Postfix. While it's usually appropriate to assume I'm ignorant, naive, or generally speaking out of my @ss, I'm afraid the responses I've received don't answer the question I asked. I do understand how the MUA talks to
Patrick Nagel wrote: the MTA and the whole SMTP/IMAP thing. I rambled a little bit just in case someone came up with a better solution for the double transmit scenario - but that wasn't the question.
The question: can Dovecot be configured to support sieve filters on IMAP (not just SMTP) operations (Eduardo helped me here - so I'll specifically ask about the APPEND command).
If not , I'll make that a feature-request - "Timo, can Dovecot be modified to support sieve-filters on APPEND operations?". Failing that, can some specific operation subsets of sieve be supported on a cron-like basis? My understanding is that operations involving indexes are extremely fast - and that things like sender/recipient are included in the indexes. So filters based on sender/recipient might be possible?
-- Daniel
Daniel L. Miller escreveu:
The question: can Dovecot be configured to support sieve filters on IMAP (not just SMTP) operations (Eduardo helped me here - so I'll specifically ask about the APPEND command).
If not , I'll make that a feature-request - "Timo, can Dovecot be modified to support sieve-filters on APPEND operations?". Failing that, can some specific operation subsets of sieve be supported on a cron-like basis? My understanding is that operations involving indexes are extremely fast - and that things like sender/recipient are included in the indexes. So filters based on sender/recipient might be possible?
At this moment it can't. And as a matter of fact, I doubt it ever will, because it doesn't really make sense.
I'm no expert in the IMAP protocol, but as far as I understand, the APPEND command is used to store a new mail in a given mailbox. It would not make sense to run a filter to store the message somewhere other than where the client asked when using that command (just to use the specific example).
Eduardo M KALINOWSKI wrote:
Daniel L. Miller escreveu:
The question: can Dovecot be configured to support sieve filters on IMAP (not just SMTP) operations (Eduardo helped me here - so I'll specifically ask about the APPEND command).
If not , I'll make that a feature-request - "Timo, can Dovecot be modified to support sieve-filters on APPEND operations?". Failing that, can some specific operation subsets of sieve be supported on a cron-like basis? My understanding is that operations involving indexes are extremely fast - and that things like sender/recipient are included in the indexes. So filters based on sender/recipient might be possible?
At this moment it can't. And as a matter of fact, I doubt it ever will, because it doesn't really make sense.
I'm no expert in the IMAP protocol, but as far as I understand, the APPEND command is used to store a new mail in a given mailbox. It would not make sense to run a filter to store the message somewhere other than where the client asked when using that command (just to use the specific example). I never said it was a GOOD idea....but, actually, what's wrong with it?
Once again, an opportunity for me to use my lower mouth.
IMAP is based on supporting relatively high-speed links (not INTENDED
for dial-up use...). So mail is (normally) not stored on the
client-side - and changes made on the server are quickly reflected by
the client (for LAN use, I normally set my update freq to 1 minute).
And this is the kind of thing that would be implemented with the user's
knowledge (presumably) since it's implemented using the user-configured
sieve filter. So, without changing the MUA/MTA/IMAP interaction, the
IMAP server will simply file new messages according to user-set rules.
Doesn't address the multiple-transfer issue at all, but does provide an
option for centralized control of message filing.
-- Daniel
Daniel L. Miller escreveu:
So, without changing the MUA/MTA/IMAP interaction, the IMAP server will simply file new messages according to user-set rules. Doesn't address the multiple-transfer issue at all, but does provide an option for centralized control of message filing.
With the APPEND command, storing the mail somewhere that is not the default location would be a violation of the protocol:
6.3.11. APPEND Command
Arguments: mailbox name OPTIONAL flag parenthesized list OPTIONAL date/time string message literal
Responses: no specific responses for this command
Result: OK - append completed NO - append error: can't append to that mailbox, error in flags or date/time or message text BAD - command unknown or arguments invalid
The APPEND command appends the literal argument as a new message
to the end of the specified destination mailbox. This argument
SHOULD be in the format of an [RFC-2822 <http://www.faqs.org/rfcs/rfc2822.html>] message.The simplest solution would be, as already mentioned, configure the client to BCC yourself, and filter that message. (And disable the 'Store copy of sent mails' option.) I do not think running filters for APPEND'ed messages is an option (even if one not active by default).
Daniel L. Miller escreveu:
So, without changing the MUA/MTA/IMAP interaction, the IMAP server will simply file new messages according to user-set rules. Doesn't address the multiple-transfer issue at all, but does provide an option for centralized control of message filing.
With the APPEND command, storing the mail somewhere that is not the default location would be a violation of the protocol:
6.3.11. APPEND Command
Arguments: mailbox name OPTIONAL flag parenthesized list OPTIONAL date/time string message literal
Responses: no specific responses for this command
Result: OK - append completed NO - append error: can't append to that mailbox, error in flags or date/time or message text BAD - command unknown or arguments invalid
The APPEND command appends the literal argument as a new message to the end of the specified destination mailbox. This argument SHOULD be in the format of an [RFC-2822<http://www.faqs.org/rfcs/rfc2822.html>] message.
The simplest solution would be, as already mentioned, configure the client to BCC yourself, and filter that message. (And disable the 'Store copy of sent mails' option.) I do not think running filters for APPEND'ed messages is an option (even if one not active by default). First - I can't argue the point about BCC'ing - you're certainly correct
Eduardo M KALINOWSKI wrote:
that that would be the "simplest" solution in terms of immediate
implementation - if we assume users are willing to change their habits.
Since I don't - I'm exploring server-side options
I disagree about "violating" the protocol. Nothing about the mail server/client communication would change - the client would still receive an "OK" at the end. The difference would be after the message appeared in the "Sent" folder (or never show at all), a moment later it would disappear and be placed in the sieve-directed folder. I don't see how this violates the protocol - though I do agree it would be quite confusing for someone who wasn't aware of this behavior.
Daniel
Daniel L. Miller wrote:
First - I can't argue the point about BCC'ing - you're certainly correct that that would be the "simplest" solution in terms of immediate implementation - if we assume users are willing to change their habits. Since I don't - I'm exploring server-side options
If you are able to identify the sender, you can use something like the always_bcc setting in Postfix, and have it bcc to a filter that stores the message back in the right Sent folder.
Anders.
Anders Melchiorsen wrote:
Daniel L. Miller wrote:
First - I can't argue the point about BCC'ing - you're certainly correct that that would be the "simplest" solution in terms of immediate implementation - if we assume users are willing to change their habits. Since I don't - I'm exploring server-side options
If you are able to identify the sender, you can use something like the always_bcc setting in Postfix, and have it bcc to a filter that stores the message back in the right Sent folder. Now that's not fair! Providing a reasonable alternative, instead of agreeing with my feature request, totally blows my argument! Especially since I'm using Postfix!
But...then I still need to get my users to not have their "Sent" enabled or I'll have a lot of duplication. Including the users who are using remote clients. Hah! I still want my APPEND-Sieve support!
-- Daniel
Daniel L. Miller wrote:
But...then I still need to get my users to not have their "Sent" enabled or I'll have a lot of duplication. Including the users who are using remote clients. Hah! I still want my APPEND-Sieve support!
(writing again, now with my subscribed sender address)
What is it that you want to do with this APPEND Sieve support?
You must always send the message via SMTP, so any IMAP solution on top of that means that you will have to transmit the message twice.
Anders.
Daniel L. Miller wrote:
The question: can Dovecot be configured to support sieve filters on IMAP (not just SMTP) operations (Eduardo helped me here - so I'll specifically ask about the APPEND command). It is quite funny that you bring this up right now, because as a matter of fact there is a new IMAP extension in the works that should support just that:
http://www.ietf.org/internet-drafts/draft-ietf-lemonade-imap-sieve-05.txt
However, as it is now, it is still cause for much debate in the Sieve community, so I don't think this will yield an RFC any time soon.
If not , I'll make that a feature-request - "Timo, can Dovecot be modified to support sieve-filters on APPEND operations?". I would call this a plan for the distant future. There are quite a few challenges to make something like this work efficiently without too much of a performance impact. And, as some people have argued, the Sieve language is (currently) not tailored to operate directly in an IMAP context and that can - and probably will - cause unforeseen problems.
Regards,
Stephan
Stephan Bosch wrote:
Daniel L. Miller wrote:
The question: can Dovecot be configured to support sieve filters on IMAP (not just SMTP) operations (Eduardo helped me here - so I'll specifically ask about the APPEND command).
It is quite funny that you bring this up right now, because as a matter of fact there is a new IMAP extension in the works that should support just that:
http://www.ietf.org/internet-drafts/draft-ietf-lemonade-imap-sieve-05.txt
However, as it is now, it is still cause for much debate in the Sieve community, so I don't think this will yield an RFC any time soon.
Drooling with anticipation...%$#! I almost shorted out my keyboard!
Daniel
While it's usually appropriate to assume I'm ignorant, naive, or generally speaking out of my @ss, I'm afraid the responses I've received don't answer the question I asked. I do understand how the MUA talks to the MTA and the whole SMTP/IMAP thing. I rambled a little bit just in case someone came up with a better solution for the double transmit scenario - but that wasn't the question.
The question: can Dovecot be configured to support sieve filters on IMAP (not just SMTP) operations (Eduardo helped me here - so I'll specifically ask about the APPEND command). no, you can't ..and even if it will be implemented on server side.. Do you new any IMAP client supporting such behavior? As far as I know
Daniel L. Miller wrote: there is no such IMAP clients...
Failing that, can some specific operation subsets of sieve be supported on a cron-like basis? My understanding is that operations involving indexes are extremely fast - and that things like sender/recipient are included in the indexes. So filters based on sender/recipient might be possible? hmm... in theory it is possible:
- you must use maildir
- configure con to run dovecot's deliver ; you my use alternate configuration file providing different sieve setup. To speed up things use -p option; As alternative to ordinary con you my use incron - inotify kernel api based "con" to catch message creation events. ...few lines of bash scripting and this my work.
Some time ago I used similar setup.. but old Outlook clients simply crush when message "disappear".
"I disagree about "violating" the protocol. [...] the client would still receive an "OK" at the end. [...] I don't see how this violates the protocol - though I do agree it would be quite confusing for someone who wasn't aware of this behavior."
First of all it will confuse IMAP client software. Many clients don't simply trust server side "OK", they verifies this, updates their indexes after "Scesseful APPEND" without FETCH etc. Every client expects to see message where it is saved. At last immediately after save.You may:
- got an error ...
- run into endless loop of attempts to save message into folder.
- some clients my even crush.
Besides this, such behavior will led to problems with CATENATE extension to APPEND.
The CATENATE extension to the Internet Message Access Protocol (IMAP) extends the APPEND command to allow clients to create messages on the IMAP server that may contain a combination of new data along with parts of (or entire) messages already on the server.
http://tools.ietf.org/html/rfc4469
Uldis
Hello
I have this messages in log: Jul 29 00:02:56 10.0.4.16 dovecot: imap-login: Can\'t connect to auth server at default: Resource temporarily unavailable Jul 29 00:02:56 10.0.4.16 last message repeated 6 times Jul 29 00:02:56 10.0.4.16 dovecot: pop3-login: Can\'t connect to auth server at default: Resource temporarily unavailable Jul 29 00:02:56 10.0.4.16 last message repeated 6 times Jul 29 00:02:56 10.0.4.16 dovecot: imap-login: Can\'t connect to auth server at default: Resource temporarily unavailable Jul 29 00:02:56 10.0.4.16 last message repeated 15 times Jul 29 00:02:56 10.0.4.16 dovecot: pop3-login: Can\'t connect to auth server at default: Resource temporarily unavailable
In this way users can't connect top pop3, imap services. Dovecot restart only help me to restore my services. I have 80k users and ~250k sessions per day. i thank you very much for any valid hint.
Debian Linux 2.6.18-4-686
dovecot -n output:
# 1.1.2: /etc/dovecot/dovecot.conf log_timestamp: %Y-%m-%d %H:%M:%S syslog_facility: local0 protocols: imap pop3 disable_plaintext_auth: no shutdown_clients: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_process_size: 512 login_processes_count: 33 login_max_processes_count: 1000 login_max_connections: 1000 max_mail_processes: 1024 verbose_proctitle: yes first_valid_uid: 99 mail_location: maildir:%h mail_debug: yes mail_executable(default): /usr/lib/dovecot/imap-gaa mail_executable(imap): /usr/lib/dovecot/imap-gaa mail_executable(pop3): /usr/lib/dovecot/pop3-gaa mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 mail_log_max_lines_per_sec: 0 auth default: cache_size: 16384 cache_ttl: 600 verbose: yes debug: yes debug_passwords: yes passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf passdb: driver: sql args: /etc/dovecot/dovecot-sql1.conf userdb: driver: sql args: /etc/dovecot/dovecot-sql.conf socket: type: listen master: path: /var/run/dovecot/auth-master mode: 511 user: mail
-- Best regards, Alexey Gorbov gaa@udmvt.ru
On Jul 29, 2008, at 12:20 AM, Alexey Gorbov wrote:
I have this messages in log: Jul 29 00:02:56 10.0.4.16 dovecot: imap-login: Can\'t connect to
auth server at default: Resource temporarily unavailable
Has this happened more than once?
In this way users can't connect top pop3, imap services. Dovecot
restart only help me to restore my services.
How long did it happen until you restarted?
It sounds like dovecot-auth process got stuck for some reason. If it
happens again, you could:
Run strace -p
pidof dovecot-authfor a few seconds and save the
output (if you have multiple dovecot-auth processes, then strace the
one which doesn't have -w parameter)Run gdb -p
pidof dovecot-auth(again the process without -w) and
"bt full" after that.
passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf
What SQL database are you using?
participants (9)
-
Alexey Gorbov
-
Anders Melchiorsen
-
Daniel L. Miller
-
Eduardo M KALINOWSKI
-
Patrick Nagel
-
Scott Silva
-
Stephan Bosch
-
Timo Sirainen
-
Uldis Pakuls