[Dovecot] More efficient Deleting of a whole folder (or Purging Trash)

[Bill Cole]

At 6:32 PM -0800 1/29/08, Asheesh Laroia  imposed structure on a 
stream of electrons, yielding:
>On Tue, 29 Jan 2008, Bill Cole wrote:
>
>>For Dovecot, the mailbox index also has to be updated for each 
>>deletion, and while that in itself is not terribly expensive per 
>>message, it has to be done one message at a time to maintain 
>>consistency between the index and the real mail.
>
>In general, this isn't really true - because Dovecot can deal with 
>other programs accessing the Maildir, it can't treat the index as 
>canonical.

I was speaking specifically of what Dovecot might be able to do to 
speed up user experience, which would be to update (i.e. clear) the 
index, then spawn off something to actually do the slow task of 
deleting the files. To keep that behavior from cascading into 
pointless index rebuilds during the cleanup time, there would need to 
be some big changes in how Dovecot keeps index files in synch.

Or put another way: the consistency problem is not that Dovecot can't 
regenerate its index files from the mailbox, but rather that it can 
and will do so whenever it sees a directory change, and it always 
trusts the data over the index (that's GOOD.) If the index were to be 
updated ahead of a slow change of the data in order to provide the 
user the appearance of speed, it violates the design of the  index 
system.

>Therefore if Dovecot knows it is deleting a whole folder, Dovecot 
>could just as easily skip over the index updates or equivalently 
>"decide to do them all at once at the end", at which point the index 
>is empty.

Except that it wouldn't really help. The index updates are not the 
expensive part, the changes to the filesystem are. In this case it 
looks like an issue with XFS, a filesystem that is not normally 
recommended for usage that involves the creation and deletion of a 
lot of small files. Changing a little bit of data in an index file is 
very fast compared to the deletion of a file from a directory with 
35k entries.


-- 
Bill Cole                                  
bill at scconsult.com