Error: o_stream_send_istream and Disconnected in APPEND
Hi,
I'm running some Dovecot servers configured with LVS + Director + Backend + NFS and version 2.2.36.3 (a7d78f5a2).
In the last days I see an increased number of these error:
Error: o_stream_send_istream(/nfs/mail/company.com/info/Maildir/.Sent/tmp/1562771349.M255624P9151.pop01) failed: Broken pipe
always with action "Disconnected in APPEND", when users try to upload a message in Sent or Drafts.
Here the logs:
Director: Jul 10 14:53:06 imap-login: Info: proxy(info@company.com): started proxying to 10.0.0.20:143: user=<info@company.com>, method=PLAIN, rip=84.1.2.3, lip=195.1.2.3, lport=993, TLS, session=<eA6XKFONwP1f5Sh4> Jul 10 15:09:56 imap-login: Info: proxy(info@company.com): disconnecting 84.1.2.3 (Disconnected by client: EOF(0s idle, in=648413, out=6602)): user=<info@company.com>, method=PLAIN, rip=84.1.2.3, lip=195.1.2.3, lport=993, TLS, session=<eA6XKFONwP1f5Sh4>
Backend: Jul 10 14:53:06 pop01 dovecot: imap-login: ID sent: x-session-id=eA6XKFONwP1f5Sh4, x-originating-ip=84.1.2.3, x-originating-port=64960, x-connected-ip=195.1.2.3, x-connected-port=993, x-proxy-ttl=4: user=<>, rip=84.1.2.3, lip=195.1.2.3, secured, session=<eA6XKFONwP1f5Sh4> Jul 10 14:53:06 pop01 dovecot: imap-login: Login: user=<info@company.com>, method=PLAIN, rip=84.1.2.3, lip=195.1.2.3, mpid=9151, secured, session=<eA6XKFONwP1f5Sh4> Jul 10 14:53:07 pop01 dovecot: imap(info@company.com) session=<eA6XKFONwP1f5Sh4>: ID sent: name=Thunderbird, version=60.7.2 Jul 10 15:09:56 pop01 dovecot: imap(info@company.com) session=<eA6XKFONwP1f5Sh4>: Error: o_stream_send_istream(/nfs/mail/company.com/info/Maildir/.Sent/tmp/1562771349.M255624P9151.pop01) failed: Broken pipe Jul 10 15:09:57 pop01 dovecot: imap(info@company.com) session=<eA6XKFONwP1f5Sh4>: Disconnected in APPEND (1 msgs, 48 secs, 0/2657336 bytes) in=886661 out=26654 del=1 expu=0 trash=0
I can't understand if is a network problem (Firewall? Load Balancer? Switch?) or a users LAN problem. Or others.
When this happen user see an error message like "Unable to save email in Sent folder".
Any suggestions?
Thanks
-- Alessio Cecchi Postmaster @ http://www.qboxmail.it https://www.linkedin.com/in/alessice
Hi,
with a Python script I have found that all users that have this errors are using Thunderbird:
# ./check.py /var/log/dovecot/dovecot.log <p4YB0WKNUcNf8zAf> ==> name=Thunderbird, version=60.7.2 <kVE0RGKN/8Jf8zAf> ==> name=Thunderbird, version=60.7.2 <oFUykWONFctdPUnr> ==> name=Thunderbird, version=60.7.2 <dUyUlmONVMUCLHgk> ==> name=Thunderbird, version=60.7.2 <dfaphWONS8UCLHgk> ==> name=Thunderbird, version=60.7.2 <qV/6jWSN2s9PHpgH> ==> name=Thunderbird, version=60.8.0 <QLeWB0+NuvBTlali> ==> name=Thunderbird, version=60.7.2 <RQFSxE+NyXdQEudp> ==> name=Thunderbird, version=60.7.2 <MFXR2U+N17NQEudp> ==> name=Thunderbird, version=60.7.2 <8WsIY1CN5uoCLHgk> ==> name=Thunderbird, version=60.7.2 <x6uQxU+NgeMCLHgk> ==> name=Thunderbird, version=60.7.2 <4ezjaFCNfJVQEudp> ==> name=Thunderbird, version=60.7.2
So seem a problem with only Thunderbird, but why? And can be mitigated by us?
Thanks
Il 11/07/19 09:13, Alessio Cecchi via dovecot ha scritto:
Hi,
I'm running some Dovecot servers configured with LVS + Director + Backend + NFS and version 2.2.36.3 (a7d78f5a2).
In the last days I see an increased number of these error:
Error: o_stream_send_istream(/nfs/mail/company.com/info/Maildir/.Sent/tmp/1562771349.M255624P9151.pop01) failed: Broken pipe
always with action "Disconnected in APPEND", when users try to upload a message in Sent or Drafts.
Here the logs:
Director: Jul 10 14:53:06 imap-login: Info: proxy(info@company.com): started proxying to 10.0.0.20:143: user=<info@company.com>, method=PLAIN, rip=84.1.2.3, lip=195.1.2.3, lport=993, TLS, session=<eA6XKFONwP1f5Sh4> Jul 10 15:09:56 imap-login: Info: proxy(info@company.com): disconnecting 84.1.2.3 (Disconnected by client: EOF(0s idle, in=648413, out=6602)): user=<info@company.com>, method=PLAIN, rip=84.1.2.3, lip=195.1.2.3, lport=993, TLS, session=<eA6XKFONwP1f5Sh4>
Backend: Jul 10 14:53:06 pop01 dovecot: imap-login: ID sent: x-session-id=eA6XKFONwP1f5Sh4, x-originating-ip=84.1.2.3, x-originating-port=64960, x-connected-ip=195.1.2.3, x-connected-port=993, x-proxy-ttl=4: user=<>, rip=84.1.2.3, lip=195.1.2.3, secured, session=<eA6XKFONwP1f5Sh4> Jul 10 14:53:06 pop01 dovecot: imap-login: Login: user=<info@company.com>, method=PLAIN, rip=84.1.2.3, lip=195.1.2.3, mpid=9151, secured, session=<eA6XKFONwP1f5Sh4> Jul 10 14:53:07 pop01 dovecot: imap(info@company.com) session=<eA6XKFONwP1f5Sh4>: ID sent: name=Thunderbird, version=60.7.2 Jul 10 15:09:56 pop01 dovecot: imap(info@company.com) session=<eA6XKFONwP1f5Sh4>: Error: o_stream_send_istream(/nfs/mail/company.com/info/Maildir/.Sent/tmp/1562771349.M255624P9151.pop01) failed: Broken pipe Jul 10 15:09:57 pop01 dovecot: imap(info@company.com) session=<eA6XKFONwP1f5Sh4>: Disconnected in APPEND (1 msgs, 48 secs, 0/2657336 bytes) in=886661 out=26654 del=1 expu=0 trash=0
I can't understand if is a network problem (Firewall? Load Balancer? Switch?) or a users LAN problem. Or others.
When this happen user see an error message like "Unable to save email in Sent folder".
Any suggestions?
Thanks
-- Alessio Cecchi Postmaster @http://www.qboxmail.it https://www.linkedin.com/in/alessice
-- Alessio Cecchi Postmaster @ http://www.qboxmail.it https://www.linkedin.com/in/alessice
On 11 Jul 2019, at 10.13, Alessio Cecchi via dovecot <dovecot@dovecot.org> wrote:
Hi,
I'm running some Dovecot servers configured with LVS + Director + Backend + NFS and version 2.2.36.3 (a7d78f5a2).
In the last days I see an increased number of these error:
Error: o_stream_send_istream(/nfs/mail/company.com/info/Maildir/.Sent/tmp/1562771349.M255624P9151.pop01) failed: Broken pipe
always with action "Disconnected in APPEND", when users try to upload a message in Sent or Drafts.
I think it simply means that Dovecot sees that the client disconnected while it was APPENDing the mail. Although I don't know why they would suddenly start now. And I especially don't understand why the error is "Broken pipe". Dovecot uses it internally when it closes input streams, so it's possibly that, but why would isn't that happening elsewhere then.. Did you upgrade your kernel recently? I guess it's also possible that there is some bug in Dovecot, but I don't remember any changes related to this for a long time.
I guess it could actually be writing as well, because "Broken pipe" is set also for closed output streams, so maybe some failed NFS write could cause it (although it really should have logged a different error in that case, so if that was the reason this is a bug).
Dovecot v2.3.x would log a different error depending on if the problem was reading or writing, which would make this clearer.
Il 11/07/19 23:31, Timo Sirainen ha scritto:
On 11 Jul 2019, at 10.13, Alessio Cecchi via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> wrote:
Hi,
I'm running some Dovecot servers configured with LVS + Director + Backend + NFS and version 2.2.36.3 (a7d78f5a2).
In the last days I see an increased number of these error:
Error: o_stream_send_istream(/nfs/mail/company.com/info/Maildir/.Sent/tmp/1562771349.M255624P9151.pop01 <http://company.com/info/Maildir/.Sent/tmp/1562771349.M255624P9151.pop01>) failed: Broken pipe
always with action "Disconnected in APPEND", when users try to upload a message in Sent or Drafts.
I think it simply means that Dovecot sees that the client disconnected while it was APPENDing the mail. Although I don't know why they would suddenly start now. And I especially don't understand why the error is "Broken pipe". Dovecot uses it internally when it closes input streams, so it's possibly that, but why would isn't that happening elsewhere then.. Did you upgrade your kernel recently? I guess it's also possible that there is some bug in Dovecot, but I don't remember any changes related to this for a long time.
I guess it could actually be writing as well, because "Broken pipe" is set also for closed output streams, so maybe some failed NFS write could cause it (although it really should have logged a different error in that case, so if that was the reason this is a bug).
Dovecot v2.3.x would log a different error depending on if the problem was reading or writing, which would make this clearer.
Thanks Timo,
the operating system is CentOS 6 from years, and we doing regular update every time available. And also the NFS storage is the same from years.
The error is not starting to show now, I see it occasionally since 2017 but after the last network upgrade (Firewall and Switch) are coming more frequently. And only for Thunderbird and sometimes Apple iOS Mail.
The error never occurred on old servers when we didn't have a physical firewall, but we used iptables on individual servers, and MTU on network interfaces ha MTU 9000, but many others components was updated in the meantime.
We will upgrade to Dovecot 2.3 in the next months.
-- Alessio Cecchi Postmaster @ http://www.qboxmail.it https://www.linkedin.com/in/alessice
participants (2)
-
Alessio Cecchi
-
Timo Sirainen