[Dovecot] should dovecot store maildir files with CRLF or LF?
Hi.
I've made a strange observation. When having Dovecot (at least) with maildir and moving (via IMAP) mail received by some client (Evolution 3.4) into it the following happens:
Regardless of whether the mail was originally(!) set with CRLF or LF (i.e. when I use netcat to submit the plain SMTP to the relaying MTA).
When the client (Evolution) had received the mail via POP3 before moving it via IMAP into Dovecot... then the maildir file within dovecot is all LF.
When the client however received it via IMAP, before in turn moving it on via IMAP into Dovecot, then the maildir file is mixed CRLF and LF, i.e. the body is CRLF, the headers are terminated...
Well the actually bug here is probably in Evolution (as so many others... o.O)... but I wondered... what is Dovecot expected to write files? Platform end-of-line markers (i.e. LF in case of UNIX) or always network end-of-line markers (CRLF)?
Cheers, Chris.
I'm not sure if this has any bearing on what you reported here, but I do this:
fetchmail->procmail->dovecot->Claws-Mail
When I open certain emails, the empty lines between paragraphs are missing. If this is part of what you're reporting, I can pay more attention as to which emails display this anomaly.
Thanks,
SteveT
Steve Litt * http://www.troubleshooters.com/ Troubleshooting Training * Human Performance
On Wed, 22 May 2013 00:45:49 +0200 Christoph Anton Mitterer <calestyo@scientia.net> wrote:
Hi.
I've made a strange observation. When having Dovecot (at least) with maildir and moving (via IMAP) mail received by some client (Evolution 3.4) into it the following happens:
Regardless of whether the mail was originally(!) set with CRLF or LF (i.e. when I use netcat to submit the plain SMTP to the relaying MTA).
When the client (Evolution) had received the mail via POP3 before moving it via IMAP into Dovecot... then the maildir file within dovecot is all LF.
When the client however received it via IMAP, before in turn moving it on via IMAP into Dovecot, then the maildir file is mixed CRLF and LF, i.e. the body is CRLF, the headers are terminated...
Well the actually bug here is probably in Evolution (as so many others... o.O)... but I wondered... what is Dovecot expected to write files? Platform end-of-line markers (i.e. LF in case of UNIX) or always network end-of-line markers (CRLF)?
Cheers, Chris.
At 12AM +0200 on 22/05/13 you (Christoph Anton Mitterer) wrote:
I've made a strange observation. When having Dovecot (at least) with maildir and moving (via IMAP) mail received by some client (Evolution 3.4) into it the following happens:
Regardless of whether the mail was originally(!) set with CRLF or LF (i.e. when I use netcat to submit the plain SMTP to the relaying MTA).
Mail sent by SMTP has to have CRLF line endings. (Unless you're using BINARYMIME, but I don't think that's at all widely implemented yet.) If your mailserver accepts LF-only line endings it ought to translate them into CRLF before sending the message on.
When the client (Evolution) had received the mail via POP3 before moving it via IMAP into Dovecot... then the maildir file within dovecot is all LF.
What line endings is the POP server sending? Can you verify this without involving Evolution? Is the POP server Dovecot or something else?
The original (djb) definition of Maildir assumed that messages would be written to the maildir with LF line endings, and both MTA and POP server would translate back to CRLF as needed. Dovecot (as a POP server) can deal with messages in either format, and should always return them to clients with CRLF.
When the client however received it via IMAP, before in turn moving it on via IMAP into Dovecot, then the maildir file is mixed CRLF and LF, i.e. the body is CRLF, the headers are terminated...
IMAP is similar, in that line endings on the wire are always supposed to be CRLF; it's a little more complicated in that clients can also upload messages. If I APPEND a message with mixed line endings to a Maildir Dovecot mailbox, the message is written to the Maildir with LF-only endings and comes back over IMAP with CRLF-only.
Can you confirm what is actually being sent over the wire?
Ben
On Wed, 2013-05-22 at 02:54 +0100, Ben Morrow wrote:
Mail sent by SMTP has to have CRLF line endings. (Unless you're using BINARYMIME, but I don't think that's at all widely implemented yet.) If your mailserver accepts LF-only line endings it ought to translate them into CRLF before sending the message on. Sure, and I assume postfix does the later... but what's mandatory for SMTP isn't a "standard/recommendation/best-practise/etc" on what IMAP servers should do internally.
What line endings is the POP server sending? Can you verify this without involving Evolution? Yeah... guess I should teach myself speaking POP3 as well ;)
Is the POP server Dovecot or something else? no idea... that's the one of my ISP...
The original (djb) definition of Maildir assumed that messages would be written to the maildir with LF line endings, and both MTA and POP server would translate back to CRLF as needed. Dovecot (as a POP server) can deal with messages in either format, and should always return them to clients with CRLF.
When the client however received it via IMAP, before in turn moving it on via IMAP into Dovecot, then the maildir file is mixed CRLF and LF, i.e. the body is CRLF, the headers are terminated...
IMAP is similar, in that line endings on the wire are always supposed to be CRLF; it's a little more complicated in that clients can also upload messages. If I APPEND a message with mixed line endings to a Maildir Dovecot mailbox, the message is written to the Maildir with LF-only endings and comes back over IMAP with CRLF-only. IIRC such mixing is forbidding by the most recent RFC defining the format of internet mail messages... neither CR nor LF is allowed to exist (which was allowed to in the old standards and then didn't mean a newline, but rather the character CR respectively LF for itself).
This is also the reason why I wonder a bit what Dovecot is doing, cause if it's compliant, then the outside world should always only see CRLF now, right?
=> But then it makes no sense to store mixed CRLF / LF / CR, if a buggy client presents it with that via IMAP.
=> It does IMHO however make sense to consider whether it may store mails in LF-only(! no mixing)... when this is the platform's native end-of-line marker.
=> On the other hand; I'd prefer to have this homogenous... so at least when dovecot get's new mail via IMAP, I would recommend it should use generally CRLF and - one would need to think about the following more carefully - convert and single LF / CR to it. Whether it can/should do such conversation with mail picked up from new/ respectively the LDA/MDA... is another topic... but if possible, I'd do so as well.
Timo, if you read along, what do you think?
Can you confirm what is actually being sent over the wire? You mean when Dovecot re-exports the stuff via IMAP? Haven't checked that yet..
Cheers, Chris.
On 22.5.2013, at 1.45, Christoph Anton Mitterer <calestyo@scientia.net> wrote:
Hi.
I've made a strange observation. When having Dovecot (at least) with maildir and moving (via IMAP) mail received by some client (Evolution 3.4) into it the following happens:
Regardless of whether the mail was originally(!) set with CRLF or LF (i.e. when I use netcat to submit the plain SMTP to the relaying MTA).
When the client (Evolution) had received the mail via POP3 before moving it via IMAP into Dovecot... then the maildir file within dovecot is all LF.
mail_save_crlf setting controls if mails are saved as CRLF or LF to storage.
When the client however received it via IMAP, before in turn moving it on via IMAP into Dovecot, then the maildir file is mixed CRLF and LF, i.e. the body is CRLF, the headers are terminated…
Sounds like the client was saving mails using CRCRLF, and Dovecot changed that to CRLF. Perhaps Dovecot should have changed the other CR to some other character.. You could verify by looking at the IMAP traffic, e.g. http://wiki2.dovecot.org/Debugging/Rawlog
On Wed, 2013-05-22 at 20:06 +0300, Timo Sirainen wrote:
mail_save_crlf setting controls if mails are saved as CRLF or LF to storage. I see... hmm that seems to be missing from the documentation?! Can it be added? What values can it have?
What exactly happens there? Any LF or CRLF converted to the value set? or does it also handle single CRs (and therefore also LFCRs)?
Are there other such options which control what of a mail / how they are stored into maildirs? I mean regarding the content of the mails? I'm aware of the one that controls how subfolders are made in maildirs (LAYOUT=fs)... but anything else?
Sounds like the client was saving mails using CRCRLF, and Dovecot changed that to CRLF. Perhaps Dovecot should have changed the other CR to some other character.. Perhaps you should add handling single CRs (to CRLF or LF depending on mail_save_crlf) as well.
You could verify by looking at the IMAP traffic, e.g. http://wiki2.dovecot.org/Debugging/Rawlog I'll try to debug that further.
Thx, Chris.
participants (4)
-
Ben Morrow
-
Christoph Anton Mitterer
-
Steve Litt
-
Timo Sirainen