[Dovecot] SMTP Submission/Proxy server - WAS Re: Released Pigeonhole v0.4.0 for Dovecot v2.2.1.

[Charles Marcus]

On 2013-05-09 5:23 PM, Stephan Bosch <stephan at rename-it.nl> wrote:
> On 5/9/2013 6:05 PM, Charles Marcus wrote:
>> On 2013-05-09 10:35 AM, Stephan Bosch <stephan at rename-it.nl> wrote:
>>
>>> Currently, I'm building an SMTP submission proxy server.
>>
>> Can you elaborate on this?
>
> It basically acts as a front-end to your normal MTA. First of all, it 
> provides a convenient way to add SMTP AUTH support to any MTA.

Excellent, thanks Stephan.

Just to make sure I understand this correctly, basically, this means 
that if someone needs to provide SASL *client* capability on a 
postfix+dovecot system - ie, so that postfix can relay certain emails to 
certain destinations through an alternate relay server that requires 
SASL based SMTP AUTH - they would no longer need cyrus-sasl to 
accomplish this?

>> ... and auto-save-to-sent, avoiding the overhead of the 'Copy to 
>> Sent' behavior we are currently forced to use where a message is 
>> first uploaded when it is sent, then again when it is saved to the 
>> sent folder?

> Depends a bit on what you have in mind. The LEMONADE profile has a 
> forward-without-download scheme for this, using the SMTP BURL 
> extension (https://tools.ietf.org/html/rfc4468) and IMAP CATENATE 
> (https://tools.ietf.org/html/rfc4469) and URLAUTH 
> (https://tools.ietf.org/html/rfc4467). Using CATENATE, the client can 
> combine existing message parts with new text to compose a new message. 
> Using SMTP BURL and IMAP URLAUTH, the SMTP server can access that 
> message directly from the IMAP server without the need for the client 
> to download it first.
>
> Some more direct approach is also possible, e.g. let the submission 
> server store the message in the Sent folder implicitly (as Google 
> apparently does). This has a few problems though, mainly that the mail 
> client will have to be configured correctly not to store an additional 
> copy of its own. Unfortunately, there is no standardized method of 
> signalling this from server to client. Google probably filters out the 
> duplicates, we don't really know. Also, which folder does the user use 
> as Sent folder? We could use the IMAP SPECIAL-USE 
> (https://www.ietf.org/rfc/rfc6154.txt) extension to find out. Anyway, 
> adding support for implicitly storing sent messages in the \Sent 
> folder should be easy enough, but it is not a fool-proof solution. 
> Timo was discussing this a while back on the SMTP mailinglist, but 
> people there weren't too enthusiastic about standardizing a feature 
> like this so far.

Ok, I agree the main problem would be the possibility of duplicate 
messages, but I would think with the powerful filtering capabilities of 
sieve, it should be possible (not sure how easy though) to hard code a 
filter to watch for and filter/remove/delete any duplicate that the MUA 
uploads.

> The LEMONADE profile is rather elaborate and not many clients or 
> servers support it yet. I'm hoping that by providing a chicken, more 
> eggs will follow soon.

I like that dovecot is willing to take a chance on being first to 
support these kinds of enhanced services, but I will say, it is very 
important that any support for said enhancements be rock-solid.

> To provide some sort of solution for the short term, I guess I'll just 
> add an optional auto-save-to-sent feature.

Sounds great to me, but...

In my opinion, because of the ubiquitous nature of MUAs saving messages 
to a sent folder, having a reliable and low-impact method for 
automatically filtering/removing/deleting these duplicates out should be 
a requirement before this feature is considered ready. It will be a big 
and immediate problem for any installation that chooses to enable this 
feature, as virtually all MUAs will be configured to save sent messages 
to a/the sent folder. It will also be an ongoing problem for all 
installations (existing and new alike), as users add their accounts to 
new computers, phones, tablets and other devices/MUAs, totally ignoring 
the instructions from their providers that they no longer need to enable 
this feature.

In fact... after thinking about this some more, I wonder...

Would there be some reasonably reliable way to detect when an MUA is 
uploading/saving messages to the Sent folder, and if so, could the 
LEMONADE protocol be leveraged to create/send a 'notification' email to 
that user based on some kind of system template (hard coded? 
customizable?), informing them that there is no need to do this, and 
even including a link to a dovecot wiki page explaining how to disable 
the 'Save copy to Sent folder' feature in common MUAs?

Then it would be up to individual SysAdmins to keep the wiki updated 
with sections for any clients they become aware of that aren't already 
on the page.

Maybe future enhancements could even (try to) detect the MUA client (is 
this possible to do reliably?), and a direct link to the section of the 
wiki for that specific client could be provided?

Another thing that I know that google is really good at is automatically 
filtering (I guess they're deleting?) any and all duplicate emails. I 
have noticed this when copying a message store from one IMAP server to a 
gmail account. I had cases where the number of messages in certain 
folders wasn't the same, and upon investigation, noticed that the 
original/source in fact had some duplicate messages in certain folders.

So, maybe you could 'kill two birds with one stone' so to speak. and 
whatever is done to address the duplicate Sent messages could also be 
leveraged to address duplicate messages in general? Although I guess it 
is not the same problem, so maybe not...

> When the submission service has direct access to the user's mail 
> storage, that is trivial to implement. However, if the submission 
> service is unprivileged, that will be a little more difficult.

Are you talking about the difference between dovecot accessing mails 
with one system user, vs accessing mails with the individual users userID?

> Probably, in that case I'll make it use a special support service to 
> perform the actual delivery to the sent folder. Any suggestions are 
> welcome.

Thanks Stephan,

-- 

Best regards,

Charles