Corrupted index cache after migrating from maildir to mdbox
Hi guys,
I'm running dovecot/postfix on my Debian Jessie maschine:
# dovecot --version 2.2.13 doveconf -n output: http://p.bastelfreak.de/HsVwS/
today I migrated from maildir(located at /var/vmail) to mdbox with these commands: rsync -av -f"+ */" -f"- *" /var/vmail/ /home/vmail/ rm /home/vmail/*/*/Maildir -rf for i in $(doveadm user \*); do u=$(echo $i | cut -d@ -f1); d=$(echo $i |cut -d@ -f2); dsync -v -u $i mirror mdbox:/home/vmail/$d/$u/mdbox; done
Then I changed the mail_location to mdbox:/home/vmail/%d/%n/mdbox and did the following commands:
service dovecot restart for i in $(doveadm user \*); do u=$(echo $i | cut -d@ -f1); d=$(echo $i |cut -d@ -f2); dsync -v -u $i mirror maildir:/var/vmail/$d/$u/Mail; done doveadm index -A all INBOX for i in $(doveadm user \*); do doveadm -Dv fts rescan -u $i;done
unfortunately I'm getting the following two kinds of error messages for all mail boxes in my mail.log:
Error: Corrupted index cache file /home/vmail/%domain/%user/mdbox/mailboxes/%maildir/dbox-Mails/dovecot.index.cache: Broken virtual size for mail UID 2564
and:
Error: read(/home/vmail/%domain/%user/mdbox/storage/m.108 ): FETCH BODY[] for mailbox %maildir UID 2564 got too little data: 2417 vs 6319
please let me know if you have any suggestions for fixing this issue or if you need more information.
kind regards, Tim
Hi,
You have broken maildir files with wrong S= in them. Usually produced by maildrop or courier imap's tools. You can fix them with http://www.dovecot.org/tools/maildir-size-fix.pl
If you compare ls -la and S= in the files you will see the difference.
Cheers, Toni
On 30.9.2014 10:05, Tim wrote:
I think you could try to simply delete dovecot.index.cache file and have dovecot automatically recreate it on the next delivery or login. And then see if the error messages go away for this mailbox.
With mdbox the dovecot.index file must not be deleted however, you would otherwise lose "seen" flags.
Anyway, better have a backup prepared, before deleting any files ;)
Regards Daniel
Hi,
I mv'ed all dovecot.index.cache files for one mailbox away but I still get error messages:
Oct 12 22:47:12 mail01 dovecot: imap(foobar@example.com): Error: read(/home/vmail/example.com/foobar/mdbox/storage/m.1): FETCH BODY[] for mailbox INBOX UID 3 got too little data: 738 vs 1543 Oct 12 22:47:12 mail01 dovecot: imap(foobar@example.com): Error: Corrupted index cache file /home/vmail/example.com/foobar/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache: Broken virtual size for mail UID 3
So dovecot recreates the cache file but they are instant broken again? Any thoughts here?
On 12.10.2014 14:54, Daniel Parthey wrote:
Maybe the index is corrupted too and the index.cache is a cached version of the corrupted index?
Are you using NFS with mdbox and accessing one nfs mailbox directory concurrently from several dovecot hosts?
On 13.10.2014 02:37, Daniel Parthey wrote:
Maybe the index is corrupted too and the index.cache is a cached version of the corrupted index?
Are you using NFS with mdbox and accessing one nfs mailbox directory concurrently from several dovecot hosts?
all mails are stored on an xfs filesystem and only the local dovecot accesses it. Possible to fix the index.cache files? I havn't found anything suitable in the docs. The doveadm-index commands are only for the fts plugin I think.
all mails are stored on an xfs filesystem and only the local dovecot accesses it. Possible to fix the index.cache files? I havn't found anything suitable in the docs. The doveadm-index commands are only for the fts plugin I think.
On 13.10.2014 02:37, Daniel Parthey wrote:
Maybe the index is corrupted too and the index.cache is a cached version of the corrupted index?
Are you using NFS with mdbox and accessing one nfs mailbox directory concurrently from several dovecot hosts?
Hey guys,
I'm still getting theses errors while fetching mails via imap: Error: read(/home/vmail/.../mdbox/storage/m.7): FETCH BODY[] for mailbox INBOX UID 1823 got too little data: 284572 vs 47173 Error: Corrupted index cache file /home/vmail/.../mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache: Broken virtual size for mail UID 1823
I tried to export them with doveadm fetch. this works fine and doesnt create any errors at all at the loge file. Any further toughts here? Timo maybe?
On 13.10.2014 10:20, Tim wrote:
I was poking around in the code and I think there's
only 1 last_indexed_uid in the data structure.
I have no clue if this value is used outside of fts
but you could poke it with a stick and
try the doveadm fts rescan command to reset this to 0.
(Sorry if you tried this already. I recently signed up on the list.)
I'm seeing last_indexed_uid set to the wrong value (v2.2.9, Subject: Bug in fts (solr?, Maildir?) incorrect last_indexed_uid) in some cases. Your error feels like the same kind of thing.
On 10/31/2014 05:47:51 AM, Tim wrote:
Karl <kop@meme.com> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein
On 27 Sep 2014, at 17:48, Tim <tim@bastelfreak.de> wrote:
If that happens to all the mails, something went wrong. Did you maybe use zlib plugin while writing the messages (compressed), but now you're trying to read the mails without zlib plugin?
The error messages may be somewhat confusing. The problem isn't dovecot.index.cache file, but that the message sizes in the m.* files don't match the contents. There's no easy way to fix those, because they should just about never be wrong.
Hi Timo, sorry for the late reply, missed this mail. On 15.10.2014 05:05, Timo Sirainen wrote:
the migration from maildir to mdbox is a few weeks in the past so I don't remember correctly. IMHO I migrated from maildir with disabled zlib to mdbox. with mdbox, zlib was active the hole time and it is still active. Here is my doveconf -n output: http://p.bastelfreak.de/l2wq/ . Do you need anything else for debugging?
The error messages may be somewhat confusing. The problem isn't dovecot.index.cache file, but that the message sizes in the m.* files don't match the contents. There's no easy way to fix those, because they should just about never be wrong.
Hi,
I think your issue is due to your corrupted maildir sizes in filenames so you got corrupted index that way to your mdbox. That is not something you can easily fix afterwords other than fixing the maildir first and then migrate to mdbox.
Best Regards, Toni
On 21.11.2014 11:59, Tim wrote:
participants (6)
-
Daniel Parthey
-
Karl O. Pinc
-
Tim
-
Tim Meusel
-
Timo Sirainen
-
Toni Mattila