[Dovecot] emails getting mangled when dragging from Exchange account to IMAP shared folders
I'm having the most frustrating issue, and I'm at a loss for what is happening. I'm not even sure it is with Dovecot, but that's why I'm posting this here... if it isn't a dovecot issue, maybe someone can get me headed in the right direction? I've posted relevant dovecot and config info at the bottom of this post.
Here's the scenario: We recently migrated our email from Postfix+Dovecot+Amavis+SA to Exchange (not my choice, unfortunately). Now, users have to use Outlook (as opposed to before, when they could choose between Outlook and Thunderbird), and have the new Exchange account set up, as well as their old IMAP account, which they still use for shared project folders. So, emails come in to Exchange, then they drag the emails from the Exchange mailbox to the shared branch under their IMAP account, and into the appropriate job folder.
Emails to one specific user in my organization (let's call him Roy@Ocean.org) is having his emails mangled when dragging certain items from his Exchange mailbox to the IMAP account. I can't reproduce the issue with any other user. Also, this only seems to happens with emails from one other specific domain (let's call them tornado.org). So, in summary, user1@Tornado.orgsend an email to Roy@Ocean.organd Sue@Ocean.org. Both drag the email from their Exchange box into any folder (shared or otherwise) in the IMAP tree. Sue's copy makes it in just fine, while Roy's copy gets mangled. Here's the headers of the resultant email after Roy has dragged it into the shared folder:
======================================== From: "User1" <User1@tornado.org> To: "Roy", "Sue" References: <B3457C1920D7A2438CCC19EF2A527293372B11@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A527293372D9A@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272933A2C3D@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272933A310E@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272933A3683@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272933A3B97@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272933D103B@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272933D192F@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272933D1B12@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272933FD341@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272933FD8F8@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272933FD9A0@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272933FDFDD@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272934C8927@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272934C8B46@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272934C8D76@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272934C8D7A@abc027.abc.local> <B3457C1920
And here's the email body:
======================================== D7A2438CCC19EF2A5272934C8D95@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272934C8DB6@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272934C8F0D@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272935008B5@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A527293500C27@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272935011BC@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A527293525B1D@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A527293525D81@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A527293526139@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A527293526416@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272935524A6@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272935524C9@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A527293589E1C@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272935BF6A9@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272935E7414@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272935E75E2@abc027.abc.local> <B3457C1920D7A2438CCC19EF2A5272935E75E8@abc027.abc.local> <B3457C1920
D7A2438CCC19EF2A5272935E75F4@abc027.abc.local><B3457C1920D7A2438CCC19EF2A5272935E800E@abc027.abc.local>
Subject: RE: Some Subject
Date: Tue, 26 Oct 2010 15:11:01 -0700
Message-ID: <B3457C1920D7A2438CCC19EF2A52729364D0A0@abc027.abc.local>
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0118_01CB7755.52354760"X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AcrhmOPG84jptRwERxazrnj4Ne2YAQAACMtAAAAjcPAAAAu+gAAAE1hAAAAZWCAAABMaEAAADGzwACO3fdAACN0HsAEpVUrgAPt2jvAAAcrbcABrFxCQAAAXwjAAABGv8AAAI35QAYbDMHAAAPjDYAADNREgAA5XR4AAHoOFcAFtPWTAAIubCCAALwf/MAADBptQACzN6nAAAC61kAAAEN+AAACfkjAAO7MG4AAHLrMQALmKX6AAAUlCcAAAq9SgAAuoo+AAABqqMABXq+FwAD5jAAABYqZYoAOISoLgAAAd//AAAKVPYAApG8DwAAt6dsAAKxpd8AEy9g6gAADrZVAAAE4WgAAlXdkQAPY4tNAAAWY+kAFsS/RgAFtG7qAAAGJT8AAAPLUwAAA2xsAAACamEAAC4ufAADaZyfAAKizZ0AAAZ9EAAAWRHAAAAOgkgAAAUUOgAAQWBBAAij+IkAAE2k1wAAPgAgAANGrwsAACt+wgAC0EW9AAAEDpoAA12EJgAAAyAtAAAyATYAAgyfBQAKS/7/AAAEPWEAABNsTAAAAZujAALLkhkAAwwG5AAAMVt8AAAE3iwAEhzNqgAA8qbEAAIqDaMAA0UTmgAEE0hxAAlsz9IAABAdrAAFIwRcAAZxd5oAAOziPQAJEu45AAOaFlIAAeeCygADwtnbAD9zXHwAApeNOAACk/pfAAABxl0AAAWgGAAAB7d4AACGbyQACNOVmwABF68KAAKuKysAAsiR5gADDcY1AAOMAK8ACO561QABCk+7AAI5IqsAAJGVEgAFjaIkAANWSCQAACuaAwAAAKUQAABugScAAAbmTQAADspwAAzNO+MAAiFH8wAAEA4mAAMv6E0AEtz2wQADWsQHAAO4pvkACIRgpAAAApFcAAABO4UABmP4/QAAEJ/lABNOByMAAF28awAACwfDA=
Content-Language: en-us
X-OlkEid: C664F3259FA9393D4124CD409B5CAE330696EE82
content-class: urn:content-classes:message
This is a multi-part message in MIME format.
------=_NextPart_000_0118_01CB7755.52354760
Content-Type: text/plain;
charset="iso-8859-1"Content-Transfer-Encoding: 7bit
<SNIP>
Obviously, it's breaking the message in the References header, and I've tracked down the following byte sequence at the breaking point: 0D:09:0A:09 ...which is CR,TAB,CR,LF. What puzzles me is why Roy's gets broken here, when the others' do not; they're the exact same email, so you'd think they would all get broken. I can't figure out A) where that byte sequence is coming from (our Exchange server, Tornado.net's exchange server, or one of the two of our external spam filtering service... they use BigFish.com, and we use MxLogic), or B) if that's even the issue.
I've rebuilt Roy's entire machine, reconfigured Outlook from scratch, still happening. When he logs onto another machine and sets up his Exchange account on another outlook, it still happens. Our Exchange administrator (I don't have access to Exchange) says there is nothing wrong with his Exchange account, for what it's worth.
I don't expect anyone to have the solution here, since there are many variables, and there is other info that would probably be needed. What I DO want is to be pointed in the right direction on how I might track this down. For example, what is the path that this mangled email would take in this case? Even with full logging in Dovecot and postfix, I can't see any info on the path that this email follows in our system, and where it might be getting mangled. Does it actually go through the same path as an email that was actually sent to the IMAP account, or is it different when it is dragged in from another account in the same mail client? Is this more likely a problem with Postfix, or Sieve, or....? (I've disabled Amavis and SA to get them out of the picture, and same problem). Here's the dovecot config info; please let me know if you want any other config info, and thanks ahead of time!
dovecot --version
1.1.19
dovecot -n
1.1.19: /etc/dovecot/dovecot.conf
OS: Linux 2.6.28-hardened-r9 x86_64 Gentoo Base System release 1.12.13
protocols: imaps imap managesieve listen(default): 127.0.0.1:143 listen(imap): 127.0.0.1:143 listen(managesieve): 127.0.0.1:2000 ssl_listen(default): *:993 ssl_listen(imap): *:993 ssl_listen(managesieve): ssl_cert_file: /etc/ssl/dovecot/imapd.crt ssl_key_file: /etc/ssl/dovecot/imapd.key login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(managesieve): /usr/libexec/dovecot/managesieve-login login_greeting_capability(default): yes login_greeting_capability(imap): yes login_greeting_capability(managesieve): no valid_chroot_dirs: /var/mail first_valid_uid: 206 last_valid_uid: 206 first_valid_gid: 206 last_valid_gid: 206 maildir_copy_preserve_filename: yes mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(managesieve): /usr/libexec/dovecot/managesieve mail_plugins(default): acl mail_plugins(imap): acl mail_plugins(managesieve): mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(managesieve): /usr/lib64/dovecot/managesieve imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(managesieve): sieve_storage(default): sieve_storage(imap): sieve_storage(managesieve): ~ sieve(default): sieve(imap): sieve(managesieve): ~/.dovecot.sieve namespace: type: private separator: . location: maildir:~/Maildir inbox: yes list: yes subscriptions: yes namespace: type: public separator: . prefix: shared. location: maildir:/var/mail/shared list: yes lda: postmaster_address: postmaster@jensenmaritime.com hostname: mail.jensenmaritime.com mail_plugins: cmusieve mail_plugin_dir: /usr/lib/dovecot/lda sieve_global_path: /var/mail/.dovecot.sieve sieve: ~/.dovecot.sieve sendmail_path: /usr/lib/sendmail auth_socket_path: /var/run/dovecot/deliver-auth auth lda: default_realm: jensenmaritime.com user: postmaster passdb: driver: ldap args: /etc/dovecot/lda-ldap.conf userdb: driver: static args: allow_all_users=yes user=%Lu uid=206 gid=206 home=/var/mail/%Lu socket: type: listen master: path: /var/run/dovecot/deliver-auth mode: 384 user: vmail group: vmail auth default: mechanisms: PLAIN LOGIN default_realm: jensenmaritime.com user: postmaster passdb: driver: ldap args: /etc/dovecot/default-ldap.conf userdb: driver: static args: user=%Lu uid=206 gid=206 home=/var/mail/%Lu socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix plugin: acl: vfile
grep -v '^ *\(#.*\)\?$' /etc/dovecot/dovecot-ldap.conf
uris = ldap://127.0.0.1:389/ ldap_version = 3 base = ou=users,ou=accounts,dc=jensenmaritime,dc=com scope = onelevel auth_bind = yes pass_filter = (&(objectClass=CourierMailAccount)(!(disableimap=*))(mail=%Lu)) pass_attrs = auth_username_format=%Lu
uname -a
Linux milne 2.6.28-hardened-r9 #4 SMP Mon Jun 7 10:50:28 PDT 2010 x86_64 Intel(R) Xeon(R) CPU E5405 @ 2.00GHz GenuineIntel GNU/Linux (Using Gentoo 64-bit Linux)
On 3.11.2010, at 19.53, Scott Goodwin wrote:
Is the client using SSL? If not, looking at the actual IMAP traffic would be helpful. Best would be to look it at the Outlook machine, but I don't know what Windows tools exist for that. You could also enable Dovecot's rawlog, which works also with SSL enabled. If in the output you see the same corruption then it's caused by Outlook (or some antivirus/firewall in the middle). http://wiki.dovecot.org/Debugging/Rawlog
Thanks for the reply, Timo. Yes, clients use SSL, which is required, but I could disable it, and may do so. (Since our Exchange migration, none of the IMAP traffic happens outside of our firewall). I missed the rawlog on the debugging page earlier -- I'll report back my findings. --scott
On Wed, Nov 3, 2010 at 1:32 PM, Timo Sirainen <tss@iki.fi> wrote:
FYI, I got rawlog working and it shows the same break in the raw logs as in the broken headers. Below is a snippet from the rawlog (names and other identifiers redacted). The offending sequence is always in the References headers section, and you can see the line breaks there that show this. So it sounds like this can't be an issue with Dovecot, am I right? I wish I could get this figured out. Such a headache.... *sigh*
Thanks!
======================================================================== poht IDLE DONE slwf LIST "" "INBOX" 36oi LSUB "" "*" 2by6 IDLE DONE p5dx SELECT "shared.IT" uz85 FETCH 6 (UID) f925 UID FETCH 1:51 (UID FLAGS) kd5i IDLE DONE y9u7 UID FETCH 51 (UID FLAGS BODY.PEEK[]) qoyj IDLE DONE rfxa UID STORE 51 +FLAGS (\Deleted \Seen) h8ly IDLE DONE fe4g EXPUNGE fauq IDLE DONE 1wa2 UID FETCH 49 (UID FLAGS BODY.PEEK[]) 6403 IDLE DONE luq8 UID FETCH 50 (UID FLAGS BODY.PEEK[]) 700u IDLE DONE jl6b APPEND "shared.IT" (\Seen \Answered) " 2-Nov-2010 11:46:19 -0700" {12898938} From: "User" <User@xxx.com> To: "zzz, Roy", "zzz, Mandred", "zzz, Jerry" Cc: "zzz, Johan", "Bob zzz" <bobw@xxx.com>, "Brandon zzz" <brandonh@xxx.com>, "Mark zzz" <MarkBe@xxx.com>, "Cassie zzz" <CassieE@xxx.com>, "Erin zzz" <erinb@xxx.com> References: <B3457C1920D7A2438CCC19EF2A527293372B11@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A527293372D9A@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272933A2C3D@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272933A310E@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272933A3683@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272933A3B97@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272933D103B@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272933D192F@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272933D1B12@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272933FD341@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272933FD8F8@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272933FD9A0@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272933FDFDD@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272934C8927@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272934C8B46@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272934C8D76@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272934C8D7A@yyy027.yyy.local> <B3457C1920
D7A2438CCC19EF2A5272934C8D95@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272934C8DB6@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272934C8F0D@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272935008B5@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A527293500C27@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272935011BC@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A527293525B1D@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A527293525D81@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A527293526139@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A527293526416@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272935524A6@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272935524C9@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A527293589E1C@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272935BF6A9@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272935E7414@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272935E75E2@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272935E75E8@yyy027.yyy.local> <B3457C1920 D7A2438CCC19EF2A5272935E75F4@yyy027.yyy.local> <B3457C1920D7A2438CCC19EF2A5272935E800E@yyy027.yyy.local> Subject: RE: Handrails Rev A0 Date: Mon, 1 Nov 2010 08:48:46 -0700 Message-ID: <B3457C1920D7A2438CCC19EF2A52729364DDF9@yyy027.yyy.local> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0008_01CB7B6A.94700430" X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: AcrhmOPG84jptRwERxazrnj4Ne2YAQAACMtAAAAjcPAAAAu+gAAAE1hAAAAZWCAAABMaEAAADGzwACO3fdAACN0HsAEpVUrgAPt2jvAAAcrbcABrFxCQAAAXwjAAABGv8AAAI35QAYbDMHAAAPjDYAADNREgAA5XR4AAHoOFcAFtPWTAAIubCCAALwf/MAADBptQACzN6nAAAC61kAAAEN+AAACfkjAAO7MG4AAHLrMQALmKX6AAAUlCcAAAq9SgAAuoo+AAABqqMABXq+FwAD5jAAABYqZYoAOISoLgAAAd//AAAKVPYAApG8DwAAt6dsAAKxpd8AEy9g6gAADrZVAAAE4WgAAlXdkQAPY4tNAAAWY+kAFsS/RgAFtG7qAAAGJT8AAAPLUwAAA2xsAAACamEAAC4ufAADaZyfAAKizZ0AAAZ9EAAAWRHAAAAOgkgAAAUUOgAAQWBBAAij+IkAAE2k1wAAPgAgAANGrwsAACt+wgAC0EW9AAAEDpoAA12EJgAAAyAtAAAyATYAAgyfBQAKS/7/AAAEPWEAABNsTAAAAZujAALLkhkAAwwG5AAAMVt8AAAE3iwAEhzNqgAA8qbEAAIqDaMAA0UTmgAEE0hxAAlsz9IAABAdrAAFIwRcAAZxd5oAAOziPQAJEu45AAOaFlIAAeeCygADwtnbAD9zXHwAApeNOAACk/pfAAABxl0AAAWgGAAAB7d4AACGbyQACNOVmwABF68KAAKuKysAAsiR5gADDcY1AAOMAK8ACO561QABCk+7AAI5IqsAAJGVEgAFjaIkAANWSCQAACuaAwAAAKUQAABugScAAAbmTQAADspwAAzNO+MAAiFH8wAAEA4mAAMv6E0AEtz2wQADWsQHAAO4pvkACIRgpAAAApFcAAABO4UABmP4/QAAEJ/lABNOByMAAF28awACN9DjAAA8KVIAD4yrBA Content-Language: en-us X-OlkEid: 3AC4FD25109011935C76A940AD2D228DF22261EA content-class: urn:content-classes:message
On Wed, Nov 3, 2010 at 1:32 PM, Timo Sirainen <tss@iki.fi> wrote:
Looks like there was some more discussion of this via the digest, but to keep the original thread intact, I'll just reply here. I'm still a little confused over whether or not my issue is an Outlook/Exchange problem or something else, since I'm not certain how I should be interpreting this raw log. I've attached the rawlog to this email (domain names and a few surnames redacted, and about 40K crap between the <HEAD></HEAD> tags removed, but everything else intact). Can anyone take a peek at this and see if it indicates a problem upstream or downstream from the rawlog? Thanks much. Also note the other oddity that stands out is that some of the envelope recipients are broken (actually, all the recipients in my domain). For example, lines 19 - 21:
To: "Roy",
"Mandred",
"Jerry"
...don't show any email addresses -- just the names. I know that this is a problem, but I'm just not sure if it is a separate issue, or related to the email mangling problem. At any rate, line 28 shows the ~1134 character References header, with the breakage at the end there.
Note that this is the rawlog.out, not the rawlog.in. My confusion here lies in the fact that the breakage seems to be in the out-log only. I just don't know how to read these logs. The in-log is basically exactly the same as the out-log, except that it doesn't contain the first 38 lines that the out-log contains... so does that mean the email came in just fine, but didn't go "out" ok? Or does this still confirm that the email was mangled from outside of Dovecot? Thanks ahead of time. I can send the in-log if you want.
On Thu, Nov 4, 2010 at 8:08 AM, Timo Sirainen <tss@iki.fi> wrote:
Le 9 nov. 2010 à 01:57:17, Scott Goodwin a écrit :
Hello Scott,
If I understood you correctly, the raw log shows fetches against a problematic message already stored in the IMAP mailbox; I would be tempted to say that Dovecot just returns the garbage it has been asked to store...
Perhaps would it be nice to trace the whole history of such a problematic message; quoting your initial post:
Le 3 nov. 2010 à 20:53:01, Scott Goodwin a écrit :
it might prove useful to have, for example:
- the raw source of the message posted by user1@Tornado.org as stored in user1's MUA 2a. the raw source of the message received by Roy@Ocean.org as displayed by Roy's Outlook when received in the Exchange account 2b. the raw source of the message received by Sue@Ocean.org as displayed by Sues's Outlook when received in the Exchange account 3a. the raw log describing what happens when Roy drags the message into his IMAP mailbox 3b. the raw log describing what happens when Sue drags the message into her IMAP mailbox
HTH, Axel
in-log attached. I don't see any truncation in there, though you can see that there is the byte sequence 0D090A09 in there, which I think is what is breaking it (whatever "it" is). Still not sure if this indicates the mangling happens somewhere on my linux server, or somewhere upstream in Outlook/Exchange. God I hate having to deal with Outlook.
Also, to Axel: I don't have access to any of the raw email source on the Exchange/Outlook end, which is frustrating, but unavoidable. If I find the mangling happening on the Postfix/Dovecot end of things, this is actually good news to me, since I have full access to that portion of the email trail.
On Tue, Nov 9, 2010 at 9:54 AM, Timo Sirainen <tss@iki.fi> wrote:
On 9.11.2010, at 22.16, Scott Goodwin wrote:
Yeah. Outlook adds the extra empty line there in the middle.
You'd have to look at the outgoing IMAP traffic on the Outlook machine. (I don't know how.)
participants (3)
-
Axel Luttgens
-
Scott Goodwin
-
Timo Sirainen