[Dovecot] SIS errors - was Error from mobile client (Palm Pre)
Continuing to see errors on this with some new messages. Is it possible this has anything to do with zlib plugin, or possibly having a mix of zlib compressed & uncompressed messages with mdbox? I recently added zlib to deliver plugin list.
Daniel L. Miller, VP - Engineering, SET AM Fire & Electronic Services, Inc. [AMFES] dmiller@amfes.com 702-312-5276
On 11/30/2010 8:09 PM, Daniel L. Miller wrote:
Continuing to see errors on this with some new messages. Is it possible this has anything to do with zlib plugin, or possibly having a mix of zlib compressed & uncompressed messages with mdbox? I recently added zlib to deliver plugin list. I don't *think* I'm losing mail with this - but some clients are handling this worse than others. I think it's a result of running the nightly purge, and having new mails being zlib compressed via deliver while having old mails uncompressed - and then doveadm combines them into a given mdbox file during the purge. Something's being messed up in the indexes as a result.
-- Daniel L. Miller, VP - Engineering, SET AM Fire & Electronic Services, Inc. [AMFES] dmiller@amfes.com 702-312-5276
On 12/1/2010 10:49 AM, Daniel L. Miller wrote:
On 11/30/2010 8:09 PM, Daniel L. Miller wrote:
Continuing to see errors on this with some new messages. Is it possible this has anything to do with zlib plugin, or possibly having a mix of zlib compressed & uncompressed messages with mdbox? I recently added zlib to deliver plugin list. I don't *think* I'm losing mail with this - but some clients are handling this worse than others. I think it's a result of running the nightly purge, and having new mails being zlib compressed via deliver while having old mails uncompressed - and then doveadm combines them into a given mdbox file during the purge. Something's being messed up in the indexes as a result.
Tried something new. Seeing index error messages from this new message: Dec 1 10:53:13 bubba dovecot: imap(dmiller@amfes.com): Error: FETCH [] for mailbox Trash UID 17487 got too little data: 266965 vs 273669 Dec 1 10:53:13 bubba dovecot: imap(dmiller@amfes.com): Error: Corrupted index cache file /var/mail/amfes.com/dmiller/mdbox/mailboxes/Trash/dbox-Mails/dovecot.index.cache: Broken virtual size for mail UID 17487
Without obliterating the message, just shifting it between mailboxes (deleting from Inbox and moving to Trash) did nothing. However, I then tried moving the message to a different account - and then brought it back. This eliminated the index errors, and now yields: Dec 1 10:54:52 bubba dovecot: imap(dmiller@amfes.com): Error: Attachment file /var/mail/attachments/77/92/77925241140facf5fd6540aea9fe57bc0a47f35b-db0a1a24e699f64cce3800003bc41c5f larger than expected (247862)
Does this help you at all?
Daniel
On 12/1/2010 11:02 AM, Daniel L. Miller wrote:
On 12/1/2010 10:49 AM, Daniel L. Miller wrote:
On 11/30/2010 8:09 PM, Daniel L. Miller wrote:
Continuing to see errors on this with some new messages. Is it possible this has anything to do with zlib plugin, or possibly having a mix of zlib compressed & uncompressed messages with mdbox?
I recently added zlib to deliver plugin list. I don't *think* I'm losing mail with this - but some clients are handling this worse than others. I think it's a result of running the nightly purge, and having new mails being zlib compressed via deliver while having old mails uncompressed - and then doveadm combines them into a given mdbox file during the purge. Something's being messed up in the indexes as a result.
Timo - any suggestions? Are you working on this or is there something I should be doing?
-- Daniel
On Thu, 2010-12-02 at 10:40 -0800, Daniel L. Miller wrote:
On 12/1/2010 11:02 AM, Daniel L. Miller wrote:
On 12/1/2010 10:49 AM, Daniel L. Miller wrote:
On 11/30/2010 8:09 PM, Daniel L. Miller wrote:
Continuing to see errors on this with some new messages. Is it possible this has anything to do with zlib plugin, or possibly having a mix of zlib compressed & uncompressed messages with mdbox?
I recently added zlib to deliver plugin list. I don't *think* I'm losing mail with this - but some clients are handling this worse than others. I think it's a result of running the nightly purge, and having new mails being zlib compressed via deliver while having old mails uncompressed - and then doveadm combines them into a given mdbox file during the purge. Something's being messed up in the indexes as a result.Timo - any suggestions? Are you working on this or is there something I should be doing?
I'll try to look into this soon.. You could of course see if disabling zlib plugin makes it work again? Or the best would be if you could get a copy of such mdbox storage where this is reproducible with a doveadm fetch command. I could then send some debug patches if I can't reproduce this myself.
On 12/2/2010 11:17 AM, Timo Sirainen wrote:
I'll try to look into this soon.. You could of course see if disabling zlib plugin makes it work again? Or the best would be if you could get a copy of such mdbox storage where this is reproducible with a doveadm fetch command. I could then send some debug patches if I can't reproduce this myself.
When I do a "doveadm fetch -u user hdr uid 12557" I receive more than one message. Is this normal?
At the moment, when I have a message with attachments that generate an error, I have a single error log entry for a corrupted index cache file. Re-loading the message in the client gives the message with attachments - no further errors. If I then forward the message to myself - the cycle repeats.
-- Daniel
On 12/2/2010 3:39 PM, Daniel L. Miller wrote:
On 12/2/2010 11:17 AM, Timo Sirainen wrote:
I'll try to look into this soon.. You could of course see if disabling zlib plugin makes it work again? Or the best would be if you could get a copy of such mdbox storage where this is reproducible with a doveadm fetch command. I could then send some debug patches if I can't reproduce this myself.
First evidence of lost mail. Recent message that previously produced the log errors, but was then readable with attachments in the client - after a doveadm purge, mail is empty with no error messages shown in log.
I have removed the global zlib plugin, added it to imap, and kept it off lda.
Daniel
On 2.12.2010, at 23.39, Daniel L. Miller wrote:
When I do a "doveadm fetch -u user hdr uid 12557" I receive more than one message. Is this normal?
Yes, because you didn't specify mailbox. Also don't just fetch hdr, that won't trigger the error. So something like:
doveadm fetch -u user text uid 12557 mailbox INBOX
Are you using zlib compression? (You have plugin { zlib_save = gz } ?)
I fixed a couple of bugs:
http://hg.dovecot.org/dovecot-2.0/rev/e5dcc12f8dba http://hg.dovecot.org/dovecot-2.0/rev/4455f79f964d
But yeah, looks like there are some more bugs when mail compression is enabled. It somehow truncates attachments sometimes. I'll continue looking into it..
On 3.12.2010, at 9.26, Timo Sirainen wrote:
I fixed a couple of bugs:
http://hg.dovecot.org/dovecot-2.0/rev/e5dcc12f8dba http://hg.dovecot.org/dovecot-2.0/rev/4455f79f964d
But yeah, looks like there are some more bugs when mail compression is enabled. It somehow truncates attachments sometimes. I'll continue looking into it..
Fixed:
http://hg.dovecot.org/dovecot-2.0/rev/251612ef4186 http://hg.dovecot.org/dovecot-2.0/rev/235383a75fc7
On 12/3/2010 2:17 AM, Timo Sirainen wrote:
On 3.12.2010, at 9.26, Timo Sirainen wrote:
I fixed a couple of bugs:
http://hg.dovecot.org/dovecot-2.0/rev/e5dcc12f8dba http://hg.dovecot.org/dovecot-2.0/rev/4455f79f964d
But yeah, looks like there are some more bugs when mail compression is enabled. It somehow truncates attachments sometimes. I'll continue looking into it.. Fixed:
http://hg.dovecot.org/dovecot-2.0/rev/251612ef4186 http://hg.dovecot.org/dovecot-2.0/rev/235383a75fc7
Applied these four patches. Re-enabled zlib on lda. Watching & waiting...
-- Daniel
On 12/3/2010 9:15 AM, Daniel L. Miller wrote:
On 12/3/2010 2:17 AM, Timo Sirainen wrote:
On 3.12.2010, at 9.26, Timo Sirainen wrote:
I fixed a couple of bugs:
http://hg.dovecot.org/dovecot-2.0/rev/e5dcc12f8dba http://hg.dovecot.org/dovecot-2.0/rev/4455f79f964d
But yeah, looks like there are some more bugs when mail compression is enabled. It somehow truncates attachments sometimes. I'll continue looking into it.. Fixed:
http://hg.dovecot.org/dovecot-2.0/rev/251612ef4186 http://hg.dovecot.org/dovecot-2.0/rev/235383a75fc7
Applied these four patches. Re-enabled zlib on lda. Watching & waiting...
Still breaking on new mails.
-- Daniel
On 12/3/2010 3:32 PM, Timo Sirainen wrote:
On 3.12.2010, at 23.28, Daniel L. Miller wrote:
Still breaking on new mails. Show some error logs.
Dec 3 15:24:45 bubba dovecot: imap(dmiller@amfes.com): Error: FETCH [1.2.MIME] for mailbox INBOX UID 10921 got too little data: 0 vs 94 Dec 3 15:24:45 bubba dovecot: imap(dmiller@amfes.com): Error: Corrupted index cache file /var/mail/amfes.com/dmiller/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache: Broken MIME parts for mail UID 10921
-- Daniel
On 4.12.2010, at 0.46, Daniel L. Miller wrote:
On 12/3/2010 3:32 PM, Timo Sirainen wrote:
On 3.12.2010, at 23.28, Daniel L. Miller wrote:
Still breaking on new mails. Show some error logs.
Dec 3 15:24:45 bubba dovecot: imap(dmiller@amfes.com): Error: FETCH [1.2.MIME] for mailbox INBOX UID 10921 got too little data: 0 vs 94
1.2.MIME is a MIME header.. Do you have other errors? Are they also about *.MIME?
On 12/2/2010 11:52 PM, Timo Sirainen wrote:
On 2.12.2010, at 23.39, Daniel L. Miller wrote:
When I do a "doveadm fetch -u user hdr uid 12557" I receive more than one message. Is this normal? Yes, because you didn't specify mailbox. Also don't just fetch hdr, that won't trigger the error. So something like:
doveadm fetch -u user text uid 12557 mailbox INBOX Ok. Are you using zlib compression? (You have plugin { zlib_save = gz } ?)
Yes.
-- Daniel
participants (2)
-
Daniel L. Miller
-
Timo Sirainen