On 5/8/2011 5:21 AM, Kamil Jońca wrote:
Stan Hoeppnerstan@hardwarefreak.com writes:
On 5/6/2011 3:07 PM, Kamil Jońca wrote:
I have some archive mails in gzipped mboxes. I could use them with dovecot 1.x without problems. But recently I have installed dovecot 2.0.12, and they are slow. very slow.
Creating index files takes about 10 minutes for ~20M file with 560 messages for bzipped mbox, for gzipped is little better but still unusable :(
What other software, if any, was also upgraded/changed when you upgraded to Dovecot 2.0.12? Libraries? Filesystem? Daemons? What
IIRC only dovecot - I simply upgrade debian package via aptitude.
The latest Debian stable dovecot package is 1.2.15-4. If 'aptitude upgrade' pulled 2.0.12 then you are running either testing or unstable, or you're using non-official mirrors. Either way, you can expect to have some problems. Also, you probably need to be asking on debian-user or asking the maintainers directly. And you need to be able to give them an actual bug report. I'm guessing the problem is Debian specific and not vanilla Dovecot 2.0.12 specific. Timo hasn't responded to you yet, which may be a good indication of this.
they have been mildly corrupted along the way? Did this bad behavior start directly after the upgrade or did 2.0.12 run the zipped mbox Yes, immediately after upgrade.
Look at your aptitude and/or dkpg logs to see what other packages, if any, got upgraded/replaced when you installed dovecot.
files at acceptable speed for a while? Did you add/enable any new Dovecot plugins that you weren't running in 1.2.x?
No, the only thing was converting old config to dovecot-2 config.
Stracing dovecot process shows that every ~ 20 messages it rereads complete mbox file.
Can you be a bit more specific here? What do you mean by "rereads complete mbox file"? I'm not a dev, but that sounds suspiciously like
Sorry, my fault, more correctly is to say: "regularly".
an error handling mechanism. I.e. an error occurred while processing, or the file may have changed while processing, so we start over.
I'm almost sure that file is not changed.
It probably didn't, given the fact that Dovecot won't write to zipped mbox files, period. But if you have a broken inotify/dnotify setup it may appear to Dovecot that the file has changed. Such things are common with testing/unstable distros. Changes to the kernel, APIs, and apps occur rapidly. Such distros are meant for developers and end users with the knowledge and ability to file concise bug reports after identifying problems.
Inotify may not be the problem at all, but it seems a possibility given that Dovecot is apparently stopping decompression and rereading the file multiple times until finished. I've not looked at the Dovecot source, but this seems a likely cause of the reread.
Could you have a buggy inotify/dnotify or something along those lines? How to check it?
If you're running testing/unstable you should already know how to check this. Inotify is a kernel API. For Debian Dovecot to use inotify it must be compiled with the build option 'notify=inotify'. You'll need to see the package maintainer's build script. You'll also need to look at the kernel .config used to build your kernel as inotify must be built into your kernel.
Do you now have something else running say, at the filesystem level, that that is making Dovecot think the file has changed even though it hasn't? Are you zipping these mbox files via a cron job that is running every few seconds instead of every few hours or days? No. These files were compressed once by mutt, and then only read as archive via dovecot.
Was mutt upgraded along with dovecot when you ran 'aptitude --safe upgrade'?
Have you tested any other IMAP client such as Thunderbird to eliminate mutt as the cause of the problem?
Something is apparently causing Dovecot to reread these files regularly, and I'd guess it's probably not a Dovecot bug. Did you run strace when accessing a non-compressed mbox file for comparison?
http://strony.aster.pl/kjonca/dovecot.log.gz<- uncompressed mbox http://strony.aster.pl/kjonca/dovecot.gz.log.gz<- gzipped mbox
I didn't ask for the files but the results of your analysis. This is your system and it's your job to troubleshoot it. I'm simply trying to assist you in your efforts.
If this is a production system I suggest you downgrade to your previous Dovecot version that was working properly, then build a test rig to troubleshoot this. If that's not in your cards, I suggest sticking with Debian Stable and newer Dovecot backports as they become available.
-- Stan