Hi,
imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it works even with the dumbest IMAP server, altough it really kills performance, especially on high latency lines.
I wonder: if IMAP servers can effectively handle boundless fetches (like a list with all wanted UIDs, or simply 1:* if all are needed), do you see this as a good addition to develop?
This could be a new imapc_features setting afterall...
Thanks,
On 28 Apr 2015, at 23:49, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it works even with the dumbest IMAP server, altough it really kills performance, especially on high latency lines.
I wonder: if IMAP servers can effectively handle boundless fetches (like a list with all wanted UIDs, or simply 1:* if all are needed), do you see this as a good addition to develop?
Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. The higher the value, the more memory/disk space is used for storing the received mails.
Hi,
On 05/04/15 10:59, Timo Sirainen wrote:
On 28 Apr 2015, at 23:49, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it works even with the dumbest IMAP server, altough it really kills performance, especially on high latency lines.
I wonder: if IMAP servers can effectively handle boundless fetches (like a list with all wanted UIDs, or simply 1:* if all are needed), do you see this as a good addition to develop? Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. The higher the value, the more memory/disk space is used for storing the received mails.
I'm aware of that, but it doesn't, or at least not always. For example Thunderbird issues this: 13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)]) Dovecot does this to the IMAP backend 18 UID FETCH 333574 (BODY.PEEK[]) 19 UID FETCH 333575 (BODY.PEEK[]) 20 UID FETCH 333576 (BODY.PEEK[]) 21 UID FETCH 333577 (BODY.PEEK[]) And so on.
I have mail_prefetch_count set to 100:
doveconf -n | grep prefet
mail_prefetch_count = 100
On 04 May 2015, at 16:59, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
On 05/04/15 10:59, Timo Sirainen wrote:
On 28 Apr 2015, at 23:49, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it works even with the dumbest IMAP server, altough it really kills performance, especially on high latency lines.
I wonder: if IMAP servers can effectively handle boundless fetches (like a list with all wanted UIDs, or simply 1:* if all are needed), do you see this as a good addition to develop? Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. The higher the value, the more memory/disk space is used for storing the received mails.
I'm aware of that, but it doesn't, or at least not always. For example Thunderbird issues this: 13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)]) Dovecot does this to the IMAP backend 18 UID FETCH 333574 (BODY.PEEK[])
Oh, there were several bugs related to that. Fixed:
http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2 http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b
On 04 May 2015, at 18:45, Timo Sirainen <tss@iki.fi> wrote:
On 04 May 2015, at 16:59, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
On 05/04/15 10:59, Timo Sirainen wrote:
On 28 Apr 2015, at 23:49, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it works even with the dumbest IMAP server, altough it really kills performance, especially on high latency lines.
I wonder: if IMAP servers can effectively handle boundless fetches (like a list with all wanted UIDs, or simply 1:* if all are needed), do you see this as a good addition to develop? Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. The higher the value, the more memory/disk space is used for storing the received mails.
I'm aware of that, but it doesn't, or at least not always. For example Thunderbird issues this: 13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)]) Dovecot does this to the IMAP backend 18 UID FETCH 333574 (BODY.PEEK[])
Oh, there were several bugs related to that. Fixed:
http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2 http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b
Also if you want imapc to use the full set of features with latest Dovecot, use:
imapc_features=rfc822.size fetch-headers search
After that it shouldn't be fetching BODY.PEEK[] anymore unless the body was actually wanted to be fetched.
On 05/04/15 17:45, Timo Sirainen wrote
I'm aware of that, but it doesn't, or at least not always. For example Thunderbird issues this: 13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)]) Dovecot does this to the IMAP backend 18 UID FETCH 333574 (BODY.PEEK[]) Oh, there were several bugs related to that. Fixed:
http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2 http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b Thank you very much for the quick fix(es), I will try them.
On 05/04/15 17:47, Timo Sirainen wrote:
Also if you want imapc to use the full set of features with latest Dovecot, use: imapc_features=rfc822.size fetch-headers search After that it shouldn't be fetching BODY.PEEK[] anymore unless the body was actually wanted to be fetched. Very nice, and also the path you follow is great: allow the backend IMAP server to be dumb and yet -with these feature flags- we can exploit it more if needed.
And having the ability to deliver mails through lmtpd to the IMAP backend is priceless. :)
On 05/04/15 17:45, Timo Sirainen wrote:
On 04 May 2015, at 16:59, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
On 05/04/15 10:59, Timo Sirainen wrote:
On 28 Apr 2015, at 23:49, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it works even with the dumbest IMAP server, altough it really kills performance, especially on high latency lines.
I wonder: if IMAP servers can effectively handle boundless fetches (like a list with all wanted UIDs, or simply 1:* if all are needed), do you see this as a good addition to develop? Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. The higher the value, the more memory/disk space is used for storing the received mails.
I'm aware of that, but it doesn't, or at least not always. For example Thunderbird issues this: 13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)]) Dovecot does this to the IMAP backend 18 UID FETCH 333574 (BODY.PEEK[]) Oh, there were several bugs related to that. Fixed:
http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2 http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b I've just tried 2.2.18, which has the above modifications, but the effect is the same. I have these in the config:
doveconf -n | egrep 'prefe|imapc'
imapc_features = rfc822.size fetch-headers imapc_host = 10.3.34.12 mail_location = imapc:~/imapc mail_prefetch_count = 100
With an empty imapc directory a POP3 login yields these on the backend IMAP server: 2 LOGIN "user" "pass" 1 LIST "" "" 3 SELECT "INBOX" 4 NOOP 5 UID FETCH 1:* (FLAGS) 6 UID FETCH 2 (RFC822.SIZE) 7 UID FETCH 3 (RFC822.SIZE) 8 UID FETCH 4 (RFC822.SIZE) 9 UID FETCH 5 (RFC822.SIZE) 10 UID FETCH 6 (RFC822.SIZE) 11 UID FETCH 7 (RFC822.SIZE) 12 UID FETCH 8 (RFC822.SIZE) [...] so on, for every message...
The response times for each of the FETCHes are around 13-16 ms, so logging into dovecot for the first time takes more than a minute with 5000 mails in the INBOX and a half day with around a million...
It would be good to have a: 6 UID FETCH 1:* (RFC822.SIZE) or even a: 5 UID FETCH 1:* (FLAGS RFC822.SIZE) if needed (dovecot first fills up its cache, or other cases when the full list is needed)
If dovecot would issue the latter for this 5000-mail inbox, the first fetch (and POP login) would take only 500 ms instead of the 75 seconds that it takes now... Could you please tweak this some more? :)
Thanks,
On 07/02/2015 04:22 PM, Nagy, Attila wrote:
On 05/04/15 17:45, Timo Sirainen wrote:
On 04 May 2015, at 16:59, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
On 05/04/15 10:59, Timo Sirainen wrote:
On 28 Apr 2015, at 23:49, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it works even with the dumbest IMAP server, altough it really kills performance, especially on high latency lines.
I wonder: if IMAP servers can effectively handle boundless fetches (like a list with all wanted UIDs, or simply 1:* if all are needed), do you see this as a good addition to develop? Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. The higher the value, the more memory/disk space is used for storing the received mails.
I'm aware of that, but it doesn't, or at least not always. For example Thunderbird issues this: 13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)]) Dovecot does this to the IMAP backend 18 UID FETCH 333574 (BODY.PEEK[]) Oh, there were several bugs related to that. Fixed:
http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2 http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b I've just tried 2.2.18, which has the above modifications, but the effect is the same. I have these in the config:
doveconf -n | egrep 'prefe|imapc'
imapc_features = rfc822.size fetch-headers imapc_host = 10.3.34.12 mail_location = imapc:~/imapc mail_prefetch_count = 100
With an empty imapc directory a POP3 login yields these on the backend IMAP server: 2 LOGIN "user" "pass" 1 LIST "" "" 3 SELECT "INBOX" 4 NOOP 5 UID FETCH 1:* (FLAGS) 6 UID FETCH 2 (RFC822.SIZE) 7 UID FETCH 3 (RFC822.SIZE) 8 UID FETCH 4 (RFC822.SIZE) 9 UID FETCH 5 (RFC822.SIZE) 10 UID FETCH 6 (RFC822.SIZE) 11 UID FETCH 7 (RFC822.SIZE) 12 UID FETCH 8 (RFC822.SIZE) [...] so on, for every message...
The response times for each of the FETCHes are around 13-16 ms, so logging into dovecot for the first time takes more than a minute with 5000 mails in the INBOX and a half day with around a million...
It would be good to have a: 6 UID FETCH 1:* (RFC822.SIZE) or even a: 5 UID FETCH 1:* (FLAGS RFC822.SIZE) if needed (dovecot first fills up its cache, or other cases when the full list is needed)
If dovecot would issue the latter for this 5000-mail inbox, the first fetch (and POP login) would take only 500 ms instead of the 75 seconds that it takes now... Could you please tweak this some more? :) Running 2.2.25 I can observe the same behaviour: Dovecot fetches each message's RFC822.SIZE independently, instead of doing a much more efficient fetch 1:* for all messages. This delays the first login way too long, most clients just time out which Dovecot is busy fetching the sizes and try again and again and again, eating up all resources for some time. Any chance that this will be fixed?
Thanks,
participants (2)
-
Nagy, Attila
-
Timo Sirainen