[Dovecot] Sorting a folder (by THREAD) takes a long time

[Asheesh Laroia]

On Fri, 7 Mar 2008, Timo Sirainen wrote:

> How large is your INBOX? Unless you've disabled index files the threading 
> should go pretty fast even for thousands of messages.

(now it's "archive" and "Lists" that are so big)

ca. 100K messages

> But if it really is because of THREAD, you're in luck. I've almost 
> finished thread index implementation and I'm also considering making it 
> available for v1.1 as a plugin.

That would rule.  (Unless I can already get it as part of the 1.2 hg tree, 
in which case I'm not averse to upgrading)

On Fri, 7 Mar 2008, Benjamin R. Haskell wrote:

> I'd do the traces if I were you. Maybe it's something to do with "Thread 
> Sorts By Arrival" or similar Alpine threading quirks. There was also traffic 
> a couple months ago about threading scaling non-linearly in Alpine depending 
> on that setting. I assume you're running a recent snapshot, but I don't 
> recall whether that issue was resolved.

Good question about "Thread sorts by arrival".  I disabled that for now.

Attaching to the imap process in gdb and interrupting to get a stack trace 
every once in a while, I find it's mostly (80%) in this:

#2  0xb7e28ff3 in readdir64 () from /lib/tls/i686/cmov/libc.so.6
#3  0x0806fbe1 in maildir_scan_dir (ctx=0x80f6170, new_dir=false) at 
maildir-sync.c:421
#4  0x0807060d in maildir_sync_context (ctx=0x80f6170, forced=<value 
optimized out>, lost_files_r=0xbf9a7aab) at maildir-sync.c:778
#5  0x080706f6 in maildir_storage_sync_init (box=0x81076a0, flags=<value 
optimized out>) at maildir-sync.c:837
#6  0x080ba49f in mailbox_sync (box=0x81076a0, 
flags=MAILBOX_SYNC_FLAG_FULL_READ, status_items=111, status_r=0xbf9a7b18) 
at mail-storage.c:536
#7  0x0805fb75 in cmd_select_full (cmd=0x8100c78, readonly=false) at 
cmd-select.c:39
#8  0x0805fd19 in cmd_select (cmd=0x8100c78) at cmd-select.c:87
#9  0x08061389 in client_command_input (cmd=0x8100c78) at client.c:505

and maybe (maybe less) 20% in this:

(gdb) bt
#0  mail_cache_lookup_iter_next (ctx=0xbf9a78b0, field_r=0xbf9a78d0) at 
mail-cache-lookup.c:226
#1  0x0809e208 in mail_cache_field_exists (view=0x8109820, seq=106065, 
field=1) at mail-cache-lookup.c:260
#2  0x080929fe in index_mail_set_seq (_mail=0x8116390, seq=106065) at 
index-mail.c:1054
#3  0x0809656a in index_storage_search_next_nonblock (_ctx=0x8108358, 
mail=0x8116390, tryagain_r=0xbf9a7a7b) at index-search.c:1039
#4  0xb7f0d441 in fts_mailbox_search_next_nonblock (ctx=0x8108358, 
mail=0x8116390, tryagain_r=0xbf9a7a7b) at fts-storage.c:617
#5  0x080ba642 in mailbox_search_next (ctx=0x8108358, mail=0x8116390) at 
mail-storage.c:633
#6  0x080680e9 in imap_thread (cmd=0x8100c78, charset=0x8104f50 
"US-ASCII", args=0x8109bf8, type=MAIL_THREAD_REFERENCES) at 
imap-thread.c:142

Opening the folder takes some 10 seconds.  FWIW, I know that only one 
message has changed in the folder since I last SELECTed it - that was an 
APPEND, which Dovecot knew about - maybe it could have resynchronized the 
folder in the background during that APPEND so the next open would be 
instantaneous.

Can maybe Dovecot show me the last cached version of the folder so I can 
see the index immediately, and in the background re-sync that?  That way 
opening folders could be instantaneous, and each next IMAP action could 
give me a chance to see the synchronized state once it's ready.

-- Asheesh.

-- 
we:
 	The single most important word in the world.