[Dovecot] compressed mboxes very slow

[Stan Hoeppner]

On 5/8/2011 5:21 AM, Kamil Jońca wrote:
> Stan Hoeppner<stan at hardwarefreak.com>  writes:
>
>> On 5/6/2011 3:07 PM, Kamil Jońca wrote:
>>>
>>> I have some archive mails in gzipped mboxes. I could use them with
>>> dovecot 1.x without problems.
>>> But recently I have installed dovecot 2.0.12, and they are slow. very
>>> slow.
>>>
>>> Creating index files takes about 10 minutes for ~20M file with 560
>>> messages for bzipped mbox, for gzipped is little better but still
>>> unusable :(
>>
>> What other software, if any, was also upgraded/changed when you
>> upgraded to Dovecot 2.0.12?  Libraries?  Filesystem?  Daemons?  What
>
> IIRC only dovecot - I simply upgrade debian package via aptitude.

The latest Debian stable dovecot package is 1.2.15-4.  If 'aptitude 
upgrade' pulled 2.0.12 then you are running either testing or unstable, 
or you're using non-official mirrors.  Either way, you can expect to 
have some problems.  Also, you probably need to be asking on debian-user 
or asking the maintainers directly.  And you need to be able to give 
them an actual bug report.  I'm guessing the problem is Debian specific 
and not vanilla Dovecot 2.0.12 specific.  Timo hasn't responded to you 
yet, which may be a good indication of this.

>> they have been mildly corrupted along the way?  Did this bad behavior
>> start directly after the upgrade or did 2.0.12 run the zipped mbox
> Yes, immediately after upgrade.

Look at your aptitude and/or dkpg logs to see what other packages, if 
any, got upgraded/replaced when you installed dovecot.

>> files at acceptable speed for a while?  Did you add/enable any new
>> Dovecot plugins that you weren't running in 1.2.x?
>>
> No, the only thing was converting old config to dovecot-2 config.
>
>>> Stracing dovecot process shows that every ~ 20 messages it rereads
>>> complete mbox file.
>>
>> Can you be a bit more specific here?  What do you mean by "rereads
>> complete mbox file"?  I'm not a dev, but that sounds suspiciously like
>
> Sorry, my fault, more correctly is to say: "regularly".
>
>> an error handling mechanism.  I.e. an error occurred while processing,
>> or the file may have changed while processing, so we start over.
>
> I'm almost sure that file is not changed.

It probably didn't, given the fact that Dovecot won't write to zipped 
mbox files, period.  But if you have a broken inotify/dnotify setup it 
may appear to Dovecot that the file has changed.  Such things are common 
with testing/unstable distros.  Changes to the kernel, APIs, and apps 
occur rapidly.  Such distros are meant for developers and end users with 
the knowledge and ability to file concise bug reports after identifying 
problems.

Inotify may not be the problem at all, but it seems a possibility given 
that Dovecot is apparently stopping decompression and rereading the file 
multiple times until finished.  I've not looked at the Dovecot source, 
but this seems a likely cause of the reread.

>> Could you have a buggy inotify/dnotify or something along those lines?
> How to check it?

If you're running testing/unstable you should already know how to check 
this.  Inotify is a kernel API.  For Debian Dovecot to use inotify it 
must be compiled with the build option 'notify=inotify'.  You'll need to 
see the package maintainer's  build script.  You'll also need to look at 
the kernel .config used to build your kernel as inotify must be built 
into your kernel.

>> Do you now have something else running say, at the filesystem level,
>> that that is making Dovecot think the file has changed even though it
>> hasn't?  Are you zipping these mbox files via a cron job that is
>> running every few seconds instead of every few hours or days?
> No.  These files were compressed once by mutt, and then only read as
> archive via dovecot.

Was mutt upgraded along with dovecot when you ran 'aptitude --safe upgrade'?

Have you tested any other IMAP client such as Thunderbird to eliminate 
mutt as the cause of the problem?

>> Something is apparently causing Dovecot to reread these files
>> regularly, and I'd guess it's probably not a Dovecot bug.  Did you run
>> strace when accessing a non-compressed mbox file for comparison?
>
> http://strony.aster.pl/kjonca/dovecot.log.gz<- uncompressed mbox
> http://strony.aster.pl/kjonca/dovecot.gz.log.gz<- gzipped mbox

I didn't ask for the files but the results of your analysis.  This is 
your system and it's your job to troubleshoot it.  I'm simply trying to 
assist you in your efforts.

If this is a production system I suggest you downgrade to your previous 
Dovecot version that was working properly, then build a test rig to 
troubleshoot this.  If that's not in your cards, I suggest sticking with 
Debian Stable and newer Dovecot backports as they become available.

-- 
Stan