Re: Slow reading of large dovecot-uidlist files

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, 12 Apr 2016, Bostjan Skufca wrote:

On 12 April 2016 at 10:23, A.L.E.C wrote:

...

I don't know dovecot's code, but I suppose it uses uidlist file to get mailbox statistics that it returns as EXISTS, RECENT, UNSEEN, UIDNEXT, UIDVALIDITY, etc, which are required by IMAP standard. I don't know, maybe they could be stored in more optimized way, but I think in most cases this data is also needed for SORT/THREAD/FETCH which is sent after SELECT in many cases - so it will be needed anyway. There are cases (e.g. mailbox synchronization) when you indeed do only SELECT.

(Oh, Alec, hi :)

Can someone (who is more intimate with Dovecot's internals) comment on what dovecot is actually doing, where this time is being spent? That would be awesome:)

My uidfile is 4MB "large" and reading it takes about 3ms on my system, so there is still ~160ms (98%) of dovecot's delay that is unaccounted for.

You could try to trace Dovecot and see if there are a lot of syscalls when a new connection starts up, e.g. with strace or dtruss.

BTW: I would have thought that the whole process goes like this: - user logs in, dovecot imap process starts - "SELECT MyMailbox" is issued for the first time - dovecot reads uidlist file for that mailbox and caches it along with timestamp metadata

does your mailboxes change without Dovecot? Do you have some script or something like that that causes the mtime of the directory change? If Dovecot thinks an external program changed the mailbox, it validates all messages in cache and from store.

- on subsequent requests, uidlist file is stat-ed to detect any changes (changed by i.e. another process that handles same user's connection, from another device) and trigger rereading when needed, otherwise cache is used - I would imagine having certain (fairly low) TTL on the cache to prevent too much memory consumption would be nice, too.

Would that make sense?

(I am just trying to figure out the way how to optimize response time to SELECT and thus make rendering mailbox listing in Roundcube faster, or, better, acceptable.)

Best, b.

...

-- Aleksander 'A.L.E.C' Machniak Kolab Groupware Developer [http://kolab.org] Roundcube Webmail Developer [http://roundcube.net] --------------------------------------------------- PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl

- -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEVAwUBVwzkWnz1H7kL/d9rAQLxzgf/VYvyXO2AhhpMO96EWHekgUZAFx8f+Wn+ UKJcVcbRVUvabo+8CDHyqClLbBugGDJfYDVMTOLplIJ3ZrExRqbh3sENmoRYnk8q nlWoKuwoskUYYM30Ax9yYuVxDzhGOdrI7TDrojsnJgyV1w8u8jMW6iCi1ziYvcIc u34DvOBLL16biP/rqziCouTbOZcsy9wL/T7RgBE/27ph651I7A8kQ3udfKWMW0gn +EIaDWquVnHmgozP93Ln0TpN5tMi9GW50zNmT8tWUnr07aYc3E5vaS3uYLvwxVx9 P5WIVgEObF4+BrZVCFfVRpkp51hxngk2k70RdmjgzD1dOSGGLELZkw== =j6AD -----END PGP SIGNATURE-----