[Dovecot] what to expect from changing index location

[William Blunn]

I concede that this is most likely a WIBNI (Wouldn't It Be Nice If...) 
and most likely will end up on the list of WIBNIs, never to be implemented.

But I would like to take the brainstorm forward another step, just to see.

On 30/06/2011 05:35, Timo Sirainen wrote:
>> To allow for migration of existing installations, it might be an idea to
>> make Dovecot look for both "ddb" and "index" when opening, but use 
>> "ddb" when creating new files.
>
> This makes it annoying. It wastes disk I/O.. 

OK fair enough.

(Though not actually *disk* I/O /per se/. It is not like we would create 
any further sync-to-disk requirement (i.e. requiring to wait for another 
revolution), but rather that it would require more system calls.)

Presumably it's important that it works correctly for existing users 
with minimal risk of problems if people take the path of least 
resistance (and people don't read the release notes). I imagine many 
people will not be bothered about some extra failed "open" calls. But we 
should still have a way to tune for optimal I/O usage so that systems 
which are "up against it" for performance can be tuned. OK, how about this:

A configuration directive like this:

filename_word_ddb = ddb index

This specifies a list of words which will be tried in the place where we 
mean to say "ddb" in a filename.

If the directive is not present, then the default value would be as per 
the example above. This should allow existing installations to work 
correctly using old configuration files.

If a new file needs to be created, then it will use the first entry in 
the list.

So new installs will use "ddb" for all such files, and will be optimal 
where the file exists already, but mildly sub-optimal where the file 
doesn't exist (because Dovecot would have to try opening each possible 
variation before being able to know that the file was not openable). In 
order to tune for I/O, the administrator can reconfigure the list to be 
just "ddb".

Old installs will have existing files with "index" with new files being 
created with "ddb". This will work correctly, but with some degree of 
sub-optimality. In order to tune for I/O, the administrator would need to:

1. Configure filename_word_ddb to "ddb index ddb" (to mitigate the race 
condition where a file is renamed after "ddb" is tried but before 
"index" is tried)
2. Re-name existing files (from "...index..." to "...ddb...")
3. Check that no files with old names exist
4. Change the list to "ddb"

This means that things should work correctly by default, and only get 
messed-up when people actively go and try to optimise things without 
paying attention to what they're doing.

> BTW. Cyrus also has "cyrus.index" file, which is the only storage for 
> message flags. So Dovecot isn't alone with this.

Though two is still a small sample compared to the weight of existing 
terminology usage.

Besides, Cyrus is somewhat "in-bred", and we would expect it to be 
quirky :-)

Bill