imapc fetch optimization

[Nagy, Attila]

On 07/02/2015 04:22 PM, Nagy, Attila wrote:
> On 05/04/15 17:45, Timo Sirainen wrote:
>> On 04 May 2015, at 16:59, Nagy, Attila <bra at fsn.hu> wrote:
>>> Hi,
>>>
>>> On 05/04/15 10:59, Timo Sirainen wrote:
>>>> On 28 Apr 2015, at 23:49, Nagy, Attila <bra at fsn.hu> wrote:
>>>>> Hi,
>>>>>
>>>>> imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, 
>>>>> because it works even with the dumbest IMAP server, altough it 
>>>>> really kills performance, especially on high latency lines.
>>>>>
>>>>> I wonder: if IMAP servers can effectively handle boundless fetches 
>>>>> (like a list with all wanted UIDs, or simply 1:* if all are 
>>>>> needed), do you see this as a good addition to develop?
>>>> Set mail_prefetch_count = 10 or 100 or something and it'll do 
>>>> larger FETCHes. The higher the value, the more memory/disk space is 
>>>> used for storing the received mails.
>>>>
>>> I'm aware of that, but it doesn't, or at least not always.
>>> For example Thunderbird issues this:
>>> 13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS 
>>> BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID 
>>> Priority X-Priority References Newsgroups In-Reply-To Content-Type 
>>> Reply-To)])
>>> Dovecot does this to the IMAP backend
>>> 18 UID FETCH 333574 (BODY.PEEK[])
>> Oh, there were several bugs related to that. Fixed:
>>
>> http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc
>> http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2
>> http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b
> I've just tried 2.2.18, which has the above modifications, but the 
> effect is the same.
> I have these in the config:
> # doveconf -n | egrep 'prefe|imapc'
> imapc_features = rfc822.size fetch-headers
> imapc_host = 10.3.34.12
> mail_location = imapc:~/imapc
> mail_prefetch_count = 100
>
> With an empty imapc directory a POP3 login yields these on the backend 
> IMAP server:
> 2 LOGIN "user" "pass"
> 1 LIST "" ""
> 3 SELECT "INBOX"
> 4 NOOP
> 5 UID FETCH 1:* (FLAGS)
> 6 UID FETCH 2 (RFC822.SIZE)
> 7 UID FETCH 3 (RFC822.SIZE)
> 8 UID FETCH 4 (RFC822.SIZE)
> 9 UID FETCH 5 (RFC822.SIZE)
> 10 UID FETCH 6 (RFC822.SIZE)
> 11 UID FETCH 7 (RFC822.SIZE)
> 12 UID FETCH 8 (RFC822.SIZE)
> [...]
> so on, for every message...
>
> The response times for each of the FETCHes are around 13-16 ms, so 
> logging into dovecot for the first time takes more than a minute with 
> 5000 mails in the INBOX and a half day with around a million...
>
> It would be good to have a:
> 6 UID FETCH 1:* (RFC822.SIZE)
> or even a:
> 5 UID FETCH 1:* (FLAGS RFC822.SIZE)
> if needed (dovecot first fills up its cache, or other cases when the 
> full list is needed)
>
> If dovecot would issue the latter for this 5000-mail inbox, the first 
> fetch (and POP login) would take only 500 ms instead of the 75 seconds 
> that it takes now...
> Could you please tweak this some more? :)
Running 2.2.25 I can observe the same behaviour:
Dovecot fetches each message's RFC822.SIZE independently, instead of 
doing a much more efficient fetch 1:* for all messages.
This delays the first login way too long, most clients just time out 
which Dovecot is busy fetching the sizes and try again and again and 
again, eating up all resources for some time.
Any chance that this will be fixed?

Thanks,