[Dovecot] More efficient Deleting of a whole folder (or Purging Trash)

[Asheesh Laroia]

On Tue, 29 Jan 2008, Bill Cole wrote:

> At 6:32 PM -0800 1/29/08, Asheesh Laroia imposed structure on a stream 
> of electrons, yielding:
>> On Tue, 29 Jan 2008, Bill Cole wrote:
>> 
>>> For Dovecot, the mailbox index also has to be updated for each 
>>> deletion, and while that in itself is not terribly expensive per 
>>> message, it has to be done one message at a time to maintain 
>>> consistency between the index and the real mail.
>> 
>> In general, this isn't really true - because Dovecot can deal with 
>> other programs accessing the Maildir, it can't treat the index as 
>> canonical.
>
> I was speaking specifically of what Dovecot might be able to do to speed 
> up user experience, which would be to update (i.e. clear) the index, 
> then spawn off something to actually do the slow task of deleting the 
> files. To keep that behavior from cascading into pointless index 
> rebuilds during the cleanup time, there would need to be some big 
> changes in how Dovecot keeps index files in synch.

Ah, I see.

> Or put another way: the consistency problem is not that Dovecot can't 
> regenerate its index files from the mailbox, but rather that it can and will 
> do so whenever it sees a directory change, and it always trusts the data over 
> the index (that's GOOD.) If the index were to be updated ahead of a slow 
> change of the data in order to provide the user the appearance of speed, it 
> violates the design of the  index system.
>
>> Therefore if Dovecot knows it is deleting a whole folder, Dovecot could 
>> just as easily skip over the index updates or equivalently "decide to do 
>> them all at once at the end", at which point the index is empty.
>
> Except that it wouldn't really help. The index updates are not the expensive 
> part, the changes to the filesystem are. In this case it looks like an issue 
> with XFS, a filesystem that is not normally recommended for usage that 
> involves the creation and deletion of a lot of small files. Changing a little 
> bit of data in an index file is very fast compared to the deletion of a file 
> from a directory with 35k entries.

I misunderstood the cause of the slowness then, if it is really true that 
updating the index is not an important factor.  I don't remember seeing 
profiling, but since I don't have any myself I'll take your word for it.

-- Asheesh.

-- 
"Atomic batteries to power, turbines to speed."
 		-- Robin, The Boy Wonder