imapc fetch optimization

[Nagy, Attila]

On 05/04/15 17:45, Timo Sirainen wrote:
> On 04 May 2015, at 16:59, Nagy, Attila <bra at fsn.hu> wrote:
>> Hi,
>>
>> On 05/04/15 10:59, Timo Sirainen wrote:
>>> On 28 Apr 2015, at 23:49, Nagy, Attila <bra at fsn.hu> wrote:
>>>> Hi,
>>>>
>>>> imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it works even with the dumbest IMAP server, altough it really kills performance, especially on high latency lines.
>>>>
>>>> I wonder: if IMAP servers can effectively handle boundless fetches (like a list with all wanted UIDs, or simply 1:* if all are needed), do you see this as a good addition to develop?
>>> Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. The higher the value, the more memory/disk space is used for storing the received mails.
>>>
>> I'm aware of that, but it doesn't, or at least not always.
>> For example Thunderbird issues this:
>> 13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)])
>> Dovecot does this to the IMAP backend
>> 18 UID FETCH 333574 (BODY.PEEK[])
> Oh, there were several bugs related to that. Fixed:
>
> http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc
> http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2
> http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b
I've just tried 2.2.18, which has the above modifications, but the 
effect is the same.
I have these in the config:
# doveconf -n | egrep 'prefe|imapc'
imapc_features = rfc822.size fetch-headers
imapc_host = 10.3.34.12
mail_location = imapc:~/imapc
mail_prefetch_count = 100

With an empty imapc directory a POP3 login yields these on the backend 
IMAP server:
2 LOGIN "user" "pass"
1 LIST "" ""
3 SELECT "INBOX"
4 NOOP
5 UID FETCH 1:* (FLAGS)
6 UID FETCH 2 (RFC822.SIZE)
7 UID FETCH 3 (RFC822.SIZE)
8 UID FETCH 4 (RFC822.SIZE)
9 UID FETCH 5 (RFC822.SIZE)
10 UID FETCH 6 (RFC822.SIZE)
11 UID FETCH 7 (RFC822.SIZE)
12 UID FETCH 8 (RFC822.SIZE)
[...]
so on, for every message...

The response times for each of the FETCHes are around 13-16 ms, so 
logging into dovecot for the first time takes more than a minute with 
5000 mails in the INBOX and a half day with around a million...

It would be good to have a:
6 UID FETCH 1:* (RFC822.SIZE)
or even a:
5 UID FETCH 1:* (FLAGS RFC822.SIZE)
if needed (dovecot first fills up its cache, or other cases when the 
full list is needed)

If dovecot would issue the latter for this 5000-mail inbox, the first 
fetch (and POP login) would take only 500 ms instead of the 75 seconds 
that it takes now...
Could you please tweak this some more? :)

Thanks,