Slow reading of large dovecot-uidlist files

[Steffen Kaiser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, 12 Apr 2016, Bostjan Skufca wrote:
> On 12 April 2016 at 10:23, A.L.E.C <alec at alec.pl> wrote:
>
>> I don't know dovecot's code, but I suppose it uses uidlist file to get
>> mailbox statistics that it returns as EXISTS, RECENT, UNSEEN, UIDNEXT,
>> UIDVALIDITY, etc, which are required by IMAP standard. I don't know,
>> maybe they could be stored in more optimized way, but I think in most
>> cases this data is also needed for SORT/THREAD/FETCH which is sent after
>> SELECT in many cases - so it will be needed anyway. There are cases
>> (e.g. mailbox synchronization) when you indeed do only SELECT.
>>
>
> (Oh, Alec, hi :)
>
> Can someone (who is more intimate with Dovecot's internals) comment on what
> dovecot is actually doing, where this time is being spent? That would be
> awesome:)
>
> My uidfile is 4MB "large" and reading it takes about 3ms on my system, so
> there is still ~160ms (98%) of dovecot's delay that is unaccounted for.

You could try to trace Dovecot and see if there are a lot of syscalls when 
a new connection starts up, e.g. with strace or dtruss.

> BTW: I would have thought that the whole process goes like this:
> - user logs in, dovecot imap process starts
> - "SELECT MyMailbox" is issued for the first time
> - dovecot reads uidlist file for that mailbox and caches it along with
> timestamp metadata

does your mailboxes change without Dovecot? Do you have some script or 
something like that that causes the mtime of the directory change? If 
Dovecot thinks an external program changed the mailbox, it validates all 
messages in cache and from store.

> - on subsequent requests, uidlist file is stat-ed to detect any changes
> (changed by i.e. another process that handles same user's connection, from
> another device) and trigger rereading when needed, otherwise cache is used
> - I would imagine having certain (fairly low) TTL on the cache to prevent
> too much memory consumption would be nice, too.
>
> Would that make sense?
>
> (I am just trying to figure out the way how to optimize response time to
> SELECT and thus make rendering mailbox listing in Roundcube faster, or,
> better, acceptable.)
>
>
> Best,
> b.
>
>
>
>>
>> --
>> Aleksander 'A.L.E.C' Machniak
>> Kolab Groupware Developer        [http://kolab.org]
>> Roundcube Webmail Developer  [http://roundcube.net]
>> ---------------------------------------------------
>> PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl
>>
>

- -- 
Steffen Kaiser
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEVAwUBVwzkWnz1H7kL/d9rAQLxzgf/VYvyXO2AhhpMO96EWHekgUZAFx8f+Wn+
UKJcVcbRVUvabo+8CDHyqClLbBugGDJfYDVMTOLplIJ3ZrExRqbh3sENmoRYnk8q
nlWoKuwoskUYYM30Ax9yYuVxDzhGOdrI7TDrojsnJgyV1w8u8jMW6iCi1ziYvcIc
u34DvOBLL16biP/rqziCouTbOZcsy9wL/T7RgBE/27ph651I7A8kQ3udfKWMW0gn
+EIaDWquVnHmgozP93Ln0TpN5tMi9GW50zNmT8tWUnr07aYc3E5vaS3uYLvwxVx9
P5WIVgEObF4+BrZVCFfVRpkp51hxngk2k70RdmjgzD1dOSGGLELZkw==
=j6AD
-----END PGP SIGNATURE-----