failed: Cached message size smaller than expected
Every few days, my mailbox seizes up. No mail come in to my imap clients.
I'm getting these errors over and over with my mailbox:
Error: Mailbox INBOX: Deleting corrupted cache record uid=371208: UID 371208: Broken physical size in mailbox INBOX: read(/var/mail/mgrant) failed: Cached message size smaller than expected (17212 < 17222, box=INBOX, UID=371208) Error: Mailbox INBOX: UID=371208: read(/var/mail/mgrant) failed: Cached message size smaller than expected (17212 < 17222, box=INBOX, UID=371208) (FETCH BODY[]) Error: Mailbox INBOX: Deleting corrupted cache record uid=371203: UID 371203: Broken physical size in mailbox INBOX: read(/var/mail/mgrant) failed: Cached message size smaller than expected (3904 < 3914, box=INBOX, UID=371203) Error: Mailbox INBOX: UID=371203: read(/var/mail/mgrant) failed: Cached message size smaller than expected (3904 < 3914, box=INBOX, UID=371203) (FETCH BODY[])
My inbox is an mbox file. I'm running dovecot installed on Debian Bullseye, the dovecot packages are all: 1:2.3.13+dfsg1-1
I am running sendmail and using procmail for local delivery.
I suspect, but am not certain, that this may be some locking issue between procmail and dovecot but I have never been able to prove that. The final procmail rule which appends messages to my mailbox looks like this, the trailing ':' causes procmail to use a lockfile:
:0: /var/mail/mgrant
The locking config lines in 10-mail.conf are commented, but I have also tried uncommenting them, did not help:
#mbox_read_locks = fcntl #mbox_write_locks = fcntl dotlock
Though sometimes it seems to fix itself after a few hours, the only way I have found to fix this quickly is to manually remove the cache files and restart dovecot:
rm ~/mail/.imap/INBOX/* systemctl restart dovecot
I am not even sure this is a locking issue. Something definitely gets corrupted though. I do have several IMAP clients hitting the same mailbox (phone, laptop, desktop). On the phone, I run K9 and also the gmail client which talks imap. Also using thunderbird, outlook, and w10 mail, though typically not all at the same time. You could definitely say I am stress testing this setup a bit!
Any ideas on how to resolve this?
Michael Grant
On Fri, Apr 02, 2021 at 04:45:36PM -0400, Michael Grant wrote:
Every few days, my mailbox seizes up. No mail come in to my imap clients.
I'm getting these errors over and over with my mailbox:
Error: Mailbox INBOX: Deleting corrupted cache record uid=371208: UID 371208: Broken physical size in mailbox INBOX: read(/var/mail/mgrant) failed: Cached message size smaller than expected (17212 < 17222, box=INBOX, UID=371208) Error: Mailbox INBOX: UID=371208: read(/var/mail/mgrant) failed: Cached message size smaller than expected (17212 < 17222, box=INBOX, UID=371208) (FETCH BODY[]) Error: Mailbox INBOX: Deleting corrupted cache record uid=371203: UID 371203: Broken physical size in mailbox INBOX: read(/var/mail/mgrant) failed: Cached message size smaller than expected (3904 < 3914, box=INBOX, UID=371203) Error: Mailbox INBOX: UID=371203: read(/var/mail/mgrant) failed: Cached message size smaller than expected (3904 < 3914, box=INBOX, UID=371203) (FETCH BODY[])
My inbox is an mbox file. I'm running dovecot installed on Debian Bullseye, the dovecot packages are all: 1:2.3.13+dfsg1-1
I am running sendmail and using procmail for local delivery.
I suspect, but am not certain, that this may be some locking issue between procmail and dovecot but I have never been able to prove that. The final procmail rule which appends messages to my mailbox looks like this, the trailing ':' causes procmail to use a lockfile:
:0: /var/mail/mgrant
The locking config lines in 10-mail.conf are commented, but I have also tried uncommenting them, did not help:
#mbox_read_locks = fcntl #mbox_write_locks = fcntl dotlock
Though sometimes it seems to fix itself after a few hours, the only way I have found to fix this quickly is to manually remove the cache files and restart dovecot:
rm ~/mail/.imap/INBOX/* systemctl restart dovecot
I am not even sure this is a locking issue. Something definitely gets corrupted though. I do have several IMAP clients hitting the same mailbox (phone, laptop, desktop). On the phone, I run K9 and also the gmail client which talks imap. Also using thunderbird, outlook, and w10 mail, though typically not all at the same time. You could definitely say I am stress testing this setup a bit!
Any ideas on how to resolve this?
I still see this corruption every day or so. Anyone have any ideas how to debug this or resolve it?
Michael Grant
On 17/04/2021 23:07 Michael Grant mgrant@grant.org wrote:
On Fri, Apr 02, 2021 at 04:45:36PM -0400, Michael Grant wrote:
Every few days, my mailbox seizes up. No mail come in to my imap clients.
I'm getting these errors over and over with my mailbox:
Error: Mailbox INBOX: Deleting corrupted cache record uid=371208: UID 371208: Broken physical size in mailbox INBOX: read(/var/mail/mgrant) failed: Cached message size smaller than expected (17212 < 17222, box=INBOX, UID=371208) Error: Mailbox INBOX: UID=371208: read(/var/mail/mgrant) failed: Cached message size smaller than expected (17212 < 17222, box=INBOX, UID=371208) (FETCH BODY[]) Error: Mailbox INBOX: Deleting corrupted cache record uid=371203: UID 371203: Broken physical size in mailbox INBOX: read(/var/mail/mgrant) failed: Cached message size smaller than expected (3904 < 3914, box=INBOX, UID=371203) Error: Mailbox INBOX: UID=371203: read(/var/mail/mgrant) failed: Cached message size smaller than expected (3904 < 3914, box=INBOX, UID=371203) (FETCH BODY[])
My inbox is an mbox file. I'm running dovecot installed on Debian Bullseye, the dovecot packages are all: 1:2.3.13+dfsg1-1
I am running sendmail and using procmail for local delivery.
I suspect, but am not certain, that this may be some locking issue between procmail and dovecot but I have never been able to prove that. The final procmail rule which appends messages to my mailbox looks like this, the trailing ':' causes procmail to use a lockfile:
:0: /var/mail/mgrant
The locking config lines in 10-mail.conf are commented, but I have also tried uncommenting them, did not help:
#mbox_read_locks = fcntl #mbox_write_locks = fcntl dotlock
Though sometimes it seems to fix itself after a few hours, the only way I have found to fix this quickly is to manually remove the cache files and restart dovecot:
rm ~/mail/.imap/INBOX/* systemctl restart dovecot
I am not even sure this is a locking issue. Something definitely gets corrupted though. I do have several IMAP clients hitting the same mailbox (phone, laptop, desktop). On the phone, I run K9 and also the gmail client which talks imap. Also using thunderbird, outlook, and w10 mail, though typically not all at the same time. You could definitely say I am stress testing this setup a bit!
Any ideas on how to resolve this?
I still see this corruption every day or so. Anyone have any ideas how to debug this or resolve it?
Michael Grant
Hi!
We don't really fix issues with mbox files anymore, other than read issues.. Our focus is enabling people to move to other formats, such as maildir. I would strongly recommend you to consider using maildir instead of mbox.
I would also recommend you use dovecot-lda in procmail to deliver mail, if you are not already doing so.
Aki
Le 19/04/2021 à 09:01, Aki Tuomi a écrit :
On 17/04/2021 23:07 Michael Grant mgrant@grant.org wrote:
On Fri, Apr 02, 2021 at 04:45:36PM -0400, Michael Grant wrote:
Every few days, my mailbox seizes up. No mail come in to my imap clients.
I'm getting these errors over and over with my mailbox:
Error: Mailbox INBOX: Deleting corrupted cache record uid=371208: UID 371208: Broken physical size in mailbox INBOX: read(/var/mail/mgrant) failed: Cached message size smaller than expected (17212 < 17222, box=INBOX, UID=371208) Error: Mailbox INBOX: UID=371208: read(/var/mail/mgrant) failed: Cached message size smaller than expected (17212 < 17222, box=INBOX, UID=371208) (FETCH BODY[]) Error: Mailbox INBOX: Deleting corrupted cache record uid=371203: UID 371203: Broken physical size in mailbox INBOX: read(/var/mail/mgrant) failed: Cached message size smaller than expected (3904 < 3914, box=INBOX, UID=371203) Error: Mailbox INBOX: UID=371203: read(/var/mail/mgrant) failed: Cached message size smaller than expected (3904 < 3914, box=INBOX, UID=371203) (FETCH BODY[])
My inbox is an mbox file. I'm running dovecot installed on Debian Bullseye, the dovecot packages are all: 1:2.3.13+dfsg1-1
I am running sendmail and using procmail for local delivery.
I suspect, but am not certain, that this may be some locking issue between procmail and dovecot but I have never been able to prove that. The final procmail rule which appends messages to my mailbox looks like this, the trailing ':' causes procmail to use a lockfile:
:0: /var/mail/mgrant
The locking config lines in 10-mail.conf are commented, but I have also tried uncommenting them, did not help:
#mbox_read_locks = fcntl #mbox_write_locks = fcntl dotlock
Though sometimes it seems to fix itself after a few hours, the only way I have found to fix this quickly is to manually remove the cache files and restart dovecot:
rm ~/mail/.imap/INBOX/* systemctl restart dovecot
I am not even sure this is a locking issue. Something definitely gets corrupted though. I do have several IMAP clients hitting the same mailbox (phone, laptop, desktop). On the phone, I run K9 and also the gmail client which talks imap. Also using thunderbird, outlook, and w10 mail, though typically not all at the same time. You could definitely say I am stress testing this setup a bit!
Any ideas on how to resolve this? I still see this corruption every day or so. Anyone have any ideas how to debug this or resolve it?
Michael Grant Hi!
We don't really fix issues with mbox files anymore, other than read issues.. Our focus is enabling people to move to other formats, such as maildir. I would strongly recommend you to consider using maildir instead of mbox.
I would also recommend you use dovecot-lda in procmail to deliver mail, if you are not already doing so.
Aki So please put mbox code read only, or kill it. Corruption is not acceptable. At it is not at the expected level or quality dovecot used to be or claim to be.
Mbox code is slow and you will do nothing to get it faster. Ok we could buy it. Optimisation and feature on the other formats could make the Mbox code slower and slower because no investements in the Mbox code. Ok too, make sense.
But no, corruption is not acceptable. It is a bug. You're each time mbox corruption reports pops-up ask to switch to another format as the only answer. It make me nervous as beeing in my opinion an unfair and incomplete answer. This time I allow myself to react : Please put this code read only or disable it. If what you need is funding for proper basic maintenance of R/W (or even RO) Mbox code, it will make it more obvious to your users / customers.
Regards, Emmanuel.
On 19/04/2021 13:21 FUSTE Emmanuel emmanuel.fuste@thalesgroup.com wrote:
Le 19/04/2021 à 09:01, Aki Tuomi a écrit :
On 17/04/2021 23:07 Michael Grant mgrant@grant.org wrote:
On Fri, Apr 02, 2021 at 04:45:36PM -0400, Michael Grant wrote:
Every few days, my mailbox seizes up. No mail come in to my imap clients.
I'm getting these errors over and over with my mailbox:
Error: Mailbox INBOX: Deleting corrupted cache record uid=371208: UID 371208: Broken physical size in mailbox INBOX: read(/var/mail/mgrant) failed: Cached message size smaller than expected (17212 < 17222, box=INBOX, UID=371208) Error: Mailbox INBOX: UID=371208: read(/var/mail/mgrant) failed: Cached message size smaller than expected (17212 < 17222, box=INBOX, UID=371208) (FETCH BODY[]) Error: Mailbox INBOX: Deleting corrupted cache record uid=371203: UID 371203: Broken physical size in mailbox INBOX: read(/var/mail/mgrant) failed: Cached message size smaller than expected (3904 < 3914, box=INBOX, UID=371203) Error: Mailbox INBOX: UID=371203: read(/var/mail/mgrant) failed: Cached message size smaller than expected (3904 < 3914, box=INBOX, UID=371203) (FETCH BODY[])
My inbox is an mbox file. I'm running dovecot installed on Debian Bullseye, the dovecot packages are all: 1:2.3.13+dfsg1-1
I am running sendmail and using procmail for local delivery.
I suspect, but am not certain, that this may be some locking issue between procmail and dovecot but I have never been able to prove that. The final procmail rule which appends messages to my mailbox looks like this, the trailing ':' causes procmail to use a lockfile:
:0: /var/mail/mgrant
The locking config lines in 10-mail.conf are commented, but I have also tried uncommenting them, did not help:
#mbox_read_locks = fcntl #mbox_write_locks = fcntl dotlock
Though sometimes it seems to fix itself after a few hours, the only way I have found to fix this quickly is to manually remove the cache files and restart dovecot:
rm ~/mail/.imap/INBOX/* systemctl restart dovecot
I am not even sure this is a locking issue. Something definitely gets corrupted though. I do have several IMAP clients hitting the same mailbox (phone, laptop, desktop). On the phone, I run K9 and also the gmail client which talks imap. Also using thunderbird, outlook, and w10 mail, though typically not all at the same time. You could definitely say I am stress testing this setup a bit!
Any ideas on how to resolve this? I still see this corruption every day or so. Anyone have any ideas how to debug this or resolve it?
Michael Grant Hi!
We don't really fix issues with mbox files anymore, other than read issues.. Our focus is enabling people to move to other formats, such as maildir. I would strongly recommend you to consider using maildir instead of mbox.
I would also recommend you use dovecot-lda in procmail to deliver mail, if you are not already doing so.
Aki So please put mbox code read only, or kill it. Corruption is not acceptable. At it is not at the expected level or quality dovecot used to be or claim to be.
Mbox code is slow and you will do nothing to get it faster. Ok we could buy it. Optimisation and feature on the other formats could make the Mbox code slower and slower because no investements in the Mbox code. Ok too, make sense.
But no, corruption is not acceptable. It is a bug. You're each time mbox corruption reports pops-up ask to switch to another format as the only answer. It make me nervous as beeing in my opinion an unfair and incomplete answer. This time I allow myself to react : Please put this code read only or disable it. If what you need is funding for proper basic maintenance of R/W (or even RO) Mbox code, it will make it more obvious to your users / customers.
Regards, Emmanuel.
None of our customers use mbox and there is no mail corruption in mbox. Just dovecot cache is disagreeing with the mail contents, which is automatically healed by removing the cache entry from cache.
I am not sure what's going on with that, hard to say without knowing more about how the system is set up and configured.
Aki
We don't really fix issues with mbox files anymore, other than read issues.. Our focus is enabling people to move to other formats, such as maildir. I would strongly recommend you to consider using maildir instead of mbox.
Ugh, so many people still have their mail in mbox that I find it hard to believe this is "deprecated".
I would also recommend you use dovecot-lda in procmail to deliver mail, if you are not already doing so.
I wasn't, so it took me a couple days to try this and wait for it to happen again. It's still happening. I therefore think this probably doesn't have anything todo with procmail.
So please put mbox code read only, or kill it. Corruption is not acceptable. At it is not at the expected level or quality dovecot used to be or claim to be.
You'll get a lot of pushback if you do this! I'm not the only one using it. mailx and the gnu mail tools use it and I don't know if I can so easily migrate things to maildir format. I think it could be a nightmare migrating to maildir.
I am not sure what's going on with that, hard to say without knowing more about how the system is set up and configured.
First, yes I agree that it's only the cache files, not the mbox. Just removing the cache files causes dovecot to recreates them. There's no lost mail, it's just annoying and makes mail freeze.
My setup does not seem complex on the server side. It's dovecot using mbox files and local users, no virtual users. All the users are in the password file. It seems to be using Debian's default locking scheme which is set by the package.
I'm now using dovecot's 'deliver' to append messages to my mbox from procmail. I do sometimes use mutt to access the mbox but I was pretty careful not to actually write out or change the mbox today. Mutt locks the mbox when it writes so it shouldn't be a problem in theory.
I seem to be or may be the only one on this server seeing the issue. I am also a heavy imap user. At any given time, I have 2 to 4 IMAP connections happening to my mailbox. One from my laptop, one from my desktop, 2 from my phone: one from K9, a second backup one from the gmail app. I'm using Windows 10 mail on the computers and sometimes Outlook.
I'm definitely the only one using the k9 app on this server.
Is there any way 2 separate IMAP clients talking to dovecot could cause this? Is there something in the configuration that I should be setting when I have multiple IMAP clients to be more stringent on the locking so one doesn't stomp on the other inside of Dovecot?
I'm really hoping that this is just some config issue.
On 21 Apr 2021, at 11:31, Michael Grant mgrant@grant.org wrote:
We don't really fix issues with mbox files anymore, other than read issues.. Our focus is enabling people to move to other formats, such as maildir. I would strongly recommend you to consider using maildir instead of mbox.
Ugh, so many people still have their mail in mbox that I find it hard to believe this is "deprecated".
It's a terrible format and was developed when most people had mail that maybe took up several kilobytes of space.
I would also recommend you use dovecot-lda in procmail to deliver mail, if you are not already doing so.
I wasn't, so it took me a couple days to try this and wait for it to happen again. It's still happening. I therefore think this probably doesn't have anything todo with procmail.
Procmail is old, creaky, has several known bugs, and is unmaintained. But also works perfectly well with Maildir, a vast improvement over mbox.
You'll get a lot of pushback if you do this! I'm not the only one using it.
Yeah, stirring gigabytes of mail in a single flat text file is a fantastic idea, I can see why some people are so hesitant to give it up.
Honestly, it's bad. Don't use it.
mailx and the gnu mail tools use it and I don't know if I can so easily migrate things to maildir format.
GNU Mail certainly can. You will be hard pressed to find a mail client that doesn't. (I've never used mail, but I would be surprised if it was mbox only). From the command line I only use mutt anymore.
I think it could be a nightmare migrating to maildir.
It is not. In fact, it is a facility that is basically built in to formail (that's procmail's formail) which I think is in the formail man page (But it's been a few years since I stopped using procmail). There is also a mb2md.py utility.
https://wiki.dovecot.org/Migration/MailFormat
I seem to be or may be the only one on this server seeing the issue. I am also a heavy imap user.
mbox and IMAP together are a big bag of hurt. I switched from mbox back around 2000 specifically because of moving too IMAP.
Is there any way 2 separate IMAP clients talking to dovecot could cause this?
With mbox files? I would sure think so. You have two processes potentially accessing the same file. That is never good.
I'm really hoping that this is just some config issue.
It is, just not the config issue you think it is.
-- WORDS IN THE HEART CANNOT BE TAKEN --Feet of Clay
Am 21.04.2021 um 19:31 schrieb Michael Grant mgrant@grant.org:
We don't really fix issues with mbox files anymore, other than read issues.. Our focus is enabling people to move to other formats, such as maildir. I would strongly recommend you to consider using maildir instead of mbox.
Ugh, so many people still have their mail in mbox that I find it hard to believe this is "deprecated".
I don’t think that „so many people still have their mail in mbox“, at least not in conjucntion with IMAP. mbox may be good for some server generated messages from cronjobs, etc., but for nothing more.
You'll get a lot of pushback if you do this! I'm not the only one using it. mailx and the gnu mail tools use it and I don't know if I can so easily migrate things to maildir format. I think it could be a nightmare migrating to maildir.
mailx is able to use other mail backends like Maildir to. Just point the evironment variable MAIL to your Maildir instead of mbox in ~/.bash_profile (or system globally in /etc/profile) and you’re done: MAIL=/srv/mail/localhost/<USERNAME>/Maildir/
I’m doing it this way to ready mails directly on the server, works perfect.
For the rest I agree with the other posters, that mbox should really no be used in conjucntion with IMAP. You would also not use sqlite for hosting a dynamic website.
Steven
On Mon, 19 Apr 2021 14:36:07 +0300 (EEST), Aki Tuomi wrote:
[mbox index corruption]
None of our customers use mbox and there is no mail corruption in mbox. Just dovecot cache is disagreeing with the mail contents, which is automatically healed by removing the cache entry from cache.
We see that a lot here,
read [...] failed: Cached message size smaller than expected (5074 < 5075, box=INBOX, UID=94)
and it makes for a bad user experience: Clients are hanging while attempting to open the mail, and users have to close the window, or click on another mail then come back to the first one in order to successfully open it. I get a lot of complaints.
I am not sure what's going on with that, hard to say without knowing more about how the system is set up and configured.
There are longstanding problems with Dovecot mbox indexing. I have a mail here from 2004 reporting issues. ISTR they went away in late 1.x releases (maybe should check with a test install?), then came back with 2.x.
I've seen index problems reported against maildir and Dovecot's custom format. My personal take: If a project cannot fix support of a standard format, I am surely not going to follow them into a vendor lock-in.
Would be great if this could be sorted out.
Cheerio, Hauke
-- The ASCII Ribbon Campaign Hauke Fath () No HTML/RTF in email Institut für Nachrichtentechnik /\ No Word docs in email TU Darmstadt Respect for open standards Ruf +49-6151-16-21344
On 23 Apr 2021, at 08:52, Hauke Fath hf@spg.tu-darmstadt.de wrote:
There are longstanding problems with Dovecot mbox indexing.
…
Would be great if this could be sorted out.
It can be, simply stop using nbox.
-- Help me, Obi-wan Kenobi. You're my only hope.
On 23.4.2021 17.52, Hauke Fath wrote:
On Mon, 19 Apr 2021 14:36:07 +0300 (EEST), Aki Tuomi wrote:
[mbox index corruption]
None of our customers use mbox and there is no mail corruption in mbox. Just dovecot cache is disagreeing with the mail contents, which is automatically healed by removing the cache entry from cache. We see that a lot here,
read [...] failed: Cached message size smaller than expected (5074 < 5075, box=INBOX, UID=94)
and it makes for a bad user experience: Clients are hanging while attempting to open the mail, and users have to close the window, or click on another mail then come back to the first one in order to successfully open it. I get a lot of complaints.
I am not sure what's going on with that, hard to say without knowing more about how the system is set up and configured. There are longstanding problems with Dovecot mbox indexing. I have a mail here from 2004 reporting issues. ISTR they went away in late 1.x releases (maybe should check with a test install?), then came back with 2.x.
I've seen index problems reported against maildir and Dovecot's custom format. My personal take: If a project cannot fix support of a standard format, I am surely not going to follow them into a vendor lock-in.
Would be great if this could be sorted out.
Cheerio, Hauke
We are not removing maildir support, which is a non-proprietary format. If you are experiencing same issues with maildir using latest dovecot release, we are happy to look at that.
Aki
participants (6)
-
＠lbutlr
-
Aki Tuomi
-
FUSTE Emmanuel
-
Hauke Fath
-
Michael Grant
-
Steven Varco