[Dovecot] Re: corrupt attachments
Hello all, any suggestions on this? It seems like a pretty serious issue and it hasn't reoccurred since we switched back to UW imapd.
As I mentioned, it seems to be related to dovecot compiled on Solaris 10/x86... The same code compiled on Solaris 8/sparc seemed to work well.
-Brian
Brian Thompson wrote:
Not sure if this has already been covered, but any pointers or suggestions would be greatly appreciated.
We just realized last night that dovecot-stable seems to be corrupting the attachments within users' inboxes.
In an attempt to resolve the problem, we upgraded the mail server here to Dovecot 1.0alpha4 but it didn't help.
To the best of our knowledge we didn't start experiencing this phenomena until we migrated from Solaris 8/Sparc to Solaris 10/x86 about 3 weeks ago.
To be more specific: A non-corrupt attachment can be sitting within a user's mail spool (mbox format) and live there without a problem until an imap client goes to download the attachment via dovecot.
At that time, dovecot seems to be stripping the last four characters from each base64 encoded ascii line within the attachment and rewriting the message within the mail spool without those characters. From then on, the attachment is permanently corrupt, even via local unix direct file access clients such as pine or elm.
We've temporarily switched back to UW imapd, but users have noticed the decrease in speed - not having the indexing available.
Again, any suggestions would be greatly appreciated.
Thanks, Brian
I'm seeing the same thing. I'm running Fedora 4 and have seen a rise in complaints about corrupted attachments since we started using Dovecot. I am not able to fix by direct text editing on the server either.
---------- Original Message ----------- From: Brian Thompson <brian@eng.wayne.edu> To: dovecot@dovecot.org Sent: Tue, 06 Dec 2005 21:18:59 -0500 Subject: [Dovecot] Re: corrupt attachments
Hello all, any suggestions on this? It seems like a pretty serious issue and it hasn't reoccurred since we switched back to UW imapd.
As I mentioned, it seems to be related to dovecot compiled on Solaris 10/x86... The same code compiled on Solaris 8/sparc seemed to work well.
-Brian
Brian Thompson wrote:
Not sure if this has already been covered, but any pointers or suggestions would be greatly appreciated.
We just realized last night that dovecot-stable seems to be corrupting the attachments within users' inboxes.
In an attempt to resolve the problem, we upgraded the mail server here to Dovecot 1.0alpha4 but it didn't help.
To the best of our knowledge we didn't start experiencing this phenomena until we migrated from Solaris 8/Sparc to Solaris 10/x86 about 3 weeks ago.
To be more specific: A non-corrupt attachment can be sitting within a user's mail spool (mbox format) and live there without a problem until an imap client goes to download the attachment via dovecot.
At that time, dovecot seems to be stripping the last four characters from each base64 encoded ascii line within the attachment and rewriting the message within the mail spool without those characters. From then on, the attachment is permanently corrupt, even via local unix direct file access clients such as pine or elm.
We've temporarily switched back to UW imapd, but users have noticed the decrease in speed - not having the indexing available.
Again, any suggestions would be greatly appreciated.
Thanks, Brian
------- End of Original Message -------
Help protect your family and your marriage! Take a look at PROTECTED internet service from American Family Online - http://www.afo.net
Just to check:
it sounds like you're both using mbox format... have you verified that Dovecot, your LDA, and anything else accessing the mailboxes is using exactly the same locking, _in the same order_ ?
-- Curtis Maloney cmaloney@cardgate.net
Curtis,
We don't use LDAP, only POP3. The lock-related stuff in dovecot.conf looks like this:
mbox_locks = dotlock mbox_read_dotlock = yes mbox_lock_timeout = 300 mbox_dotlock_change_timeout = 30
Cliff
---------- Original Message ----------- From: Curtis Maloney <cmaloney@cardgate.net> To: chayes@afo.net Cc: Brian Thompson <brian@eng.wayne.edu>, dovecot@dovecot.org Sent: Wed, 07 Dec 2005 15:19:30 +1100 Subject: Re: [Dovecot] Re: corrupt attachments
Just to check:
it sounds like you're both using mbox format... have you verified that Dovecot, your LDA, and anything else accessing the mailboxes is using exactly the same locking, _in the same order_ ?
-- Curtis Maloney cmaloney@cardgate.net ------- End of Original Message -------
Help protect your family and your marriage! Take a look at PROTECTED internet service from American Family Online - http://www.afo.net
chayes@afo.net wrote:
Curtis,
We don't use LDAP, only POP3. The lock-related stuff in dovecot.conf looks like this:
I assume you meant "IMAP", not "LDAP".
mbox_locks = dotlock mbox_read_dotlock = yes mbox_lock_timeout = 300 mbox_dotlock_change_timeout = 30
And what are the lock settings on your delivery agent?
Does anything other than Dovecot use the mail spool?
-- Curtis Maloney cmaloney@cardgate.net
Curtis,
You're right, I meant IMAP.
We have a simple setup here. Only three things access mail: sendmail puts it in, and dovecot & openwebmail take it out.
I'm not sure how to check lock methods for sendmail and openwebmail.
Cliff
---------- Original Message ----------- From: Curtis Maloney <cmaloney@cardgate.net> To: chayes@afo.net Cc: Brian Thompson <brian@eng.wayne.edu>, dovecot@dovecot.org Sent: Wed, 07 Dec 2005 15:30:47 +1100 Subject: Re: [Dovecot] Re: corrupt attachments
chayes@afo.net wrote:
Curtis,
We don't use LDAP, only POP3. The lock-related stuff in dovecot.conf looks like this:
I assume you meant "IMAP", not "LDAP".
mbox_locks = dotlock mbox_read_dotlock = yes mbox_lock_timeout = 300 mbox_dotlock_change_timeout = 30
And what are the lock settings on your delivery agent?
Does anything other than Dovecot use the mail spool?
-- Curtis Maloney cmaloney@cardgate.net ------- End of Original Message -------
Help protect your family and your marriage! Take a look at PROTECTED internet service from American Family Online - http://www.afo.net
Thanks Curtis, I didn't think about what would happen if Sendmail tries to attach a new message to the bottom of the mbox at the exact same time that Dovecot goes to rewrite an attachment to the middle of the mbox. We are using fcntl() over NFS for both Sendmail and Dovecot. The mail server is the only NFS client that accesses the mail spools on the fileserver, so there's no chance of another NFS client causing issues. The only real reason we're using NFS is because the fileserver gets backed up on a regular basis but the mail server doesn't. The backups happen at night and this phenomena was happening in the middle of the day, so it's not a conflict with the backup software.
At first I thought the last four characters of every line were getting cut off, but it turns out that the characters I thought were missing were pushed to the next line. It appears that Dovecot is shifting the position of the CRLF by four characters
- not necessary a problem or related to the corruption issue.
I have a "before" and "after" document in base64 format that I wish I could post to the list, but the content of the decoded document is somewhat confidential (it's a vendor quote).
The total number of characters in the good document is 7,006 and the total number of characters in the corrupt document is 6,308. If it's a locking issue, I'd imagine that as Dovecot is writing the attachment, it would get interrupted and cut off, leaving only the first portion of the attachment written and the corrupted (missing) portion would be the tail end of the attachment.
In this case, comparing the two files visually character by character, the missing characters are a solid chunk right out of the middle. The beginning and end of the attachment are correct. Sort of odd...
-Brian
Curtis Maloney wrote:
Just to check:
it sounds like you're both using mbox format... have you verified that Dovecot, your LDA, and anything else accessing the mailboxes is using exactly the same locking, _in the same order_ ?
-- Curtis Maloney cmaloney@cardgate.net
On Wed, 2005-12-07 at 06:36 -0500, Brian Thompson wrote:
At first I thought the last four characters of every line were getting cut off, but it turns out that the characters I thought were missing were pushed to the next line. It appears that Dovecot is shifting the position of the CRLF by four characters
- not necessary a problem or related to the corruption issue.
Do you really have CR+LF linefeeds, or did you mean just LFs? Dovecot isn't tested too well with CR+LFs.
Can you easily reproduce this corruption?
After my coworker and I realized what was happening, it was pretty easy to reproduce. Both he and I were able to verify by killing a few more attachments opening them via our imap clients.
The reason I say 'was' is because I don't have another system to play with at the moment that has both Solaris 10/x86 and Dovecot installed.
You're correct, I should have said LFs. They didn't get turned into CRLFs until I cut/pasted the base64 data out of my imap client into Windows Notepad.
-Brian
Timo Sirainen wrote:
On Wed, 2005-12-07 at 06:36 -0500, Brian Thompson wrote:
At first I thought the last four characters of every line were getting cut off, but it turns out that the characters I thought were missing were pushed to the next line. It appears that Dovecot is shifting the position of the CRLF by four characters
- not necessary a problem or related to the corruption issue.
Do you really have CR+LF linefeeds, or did you mean just LFs? Dovecot isn't tested too well with CR+LFs.
Can you easily reproduce this corruption?
The order appears to be very important, as we were having mailbox corruption problems. When I switched our dovecot.conf file locking style and order to match "procmail -v" things got a lot better. Unfortunately, Netscape and Thunderbird still get confused from time to time and we have to remove the user's INBOX.msf and have the mailer make a new one.
One fix was to move Netscape Junk mail folder to the local disk, but that probably doesn't apply in your case.
While I love how easy Dovecot is to set up and manage, I find that it's much picker and touchier than UW imap. (Not that I miss the "Do Not Delete This Message" hack).
Ellen Clary Senior System Administrator Dynamic Graphics
Curtis Maloney wrote:
Just to check:
it sounds like you're both using mbox format... have you verified that Dovecot, your LDA, and anything else accessing the mailboxes is using exactly the same locking, _in the same order_ ?
-- Curtis Maloney cmaloney@cardgate.net
participants (5)
-
Brian Thompson
-
chayes＠afo.net
-
Curtis Maloney
-
Ellen Clary
-
Timo Sirainen