[Dovecot] slow squat fts index creation

[Timo Sirainen]

On Tue, 2011-05-24 at 17:01 +0200, Cor Bosman wrote:
> Hi all, ive been playing with squat indexes.  Up to about 300.000 emails in a single mailbox this was working flawlessly. The search index file is about 500MB at that time. Ive now added some more emails, and at 450.000 or so emails im seeing a serious problem with squat index creation. It takes...f o r e v e r .  The .tmp file is being so slowly, it will probably take 2-3 hours to create. Upto this point it took maybe a minute. 
> 
> Im doing this in an openvz container, so theoretically i may be hitting some openvz resource limit. But ive upped all the limits and dont see any improvements. I dont see any resources starvation either. 
> 
> Could there be some dovecot issue when the search index reaches say 1GB? (im estimating that it's now trying to save about 1GB search index). 

Initially squat just builds a large unorganized index. The last step is
the organization. This is the main problem with Squat's indexing speed.
The file is mmaped() and the accessed in pretty random order. As long as
you have enough memory to keep all of this mmaped data in physical
memory this works pretty fast, but otherwise the kernel starts page
faulting like crazy and it takes forever. That's why the Squat has this
code:

	/* Tell the kernel we're going to use the uidlist data, so it loads
	   it into memory and keeps it there. */
	(void)madvise(uidlist->mmap_base, uidlist->mmap_size, MADV_WILLNEED);
	/* It also speeds up a bit for us to sequentially load everything
	   into memory, although at least Linux catches up quite fast even
	   without this code. Compiler can quite easily optimize away this
	   entire for loop, but volatile seems to help with gcc 4.2. */
	for (i = 0; i < uidlist->mmap_size; i += page_size)
		((const volatile char *)uidlist->data)[i];