At 6:32 PM -0800 1/29/08, Asheesh Laroia imposed structure on a stream of electrons, yielding:
On Tue, 29 Jan 2008, Bill Cole wrote:
For Dovecot, the mailbox index also has to be updated for each deletion, and while that in itself is not terribly expensive per message, it has to be done one message at a time to maintain consistency between the index and the real mail.
In general, this isn't really true - because Dovecot can deal with other programs accessing the Maildir, it can't treat the index as canonical.
I was speaking specifically of what Dovecot might be able to do to speed up user experience, which would be to update (i.e. clear) the index, then spawn off something to actually do the slow task of deleting the files. To keep that behavior from cascading into pointless index rebuilds during the cleanup time, there would need to be some big changes in how Dovecot keeps index files in synch.
Or put another way: the consistency problem is not that Dovecot can't regenerate its index files from the mailbox, but rather that it can and will do so whenever it sees a directory change, and it always trusts the data over the index (that's GOOD.) If the index were to be updated ahead of a slow change of the data in order to provide the user the appearance of speed, it violates the design of the index system.
Therefore if Dovecot knows it is deleting a whole folder, Dovecot could just as easily skip over the index updates or equivalently "decide to do them all at once at the end", at which point the index is empty.
Except that it wouldn't really help. The index updates are not the expensive part, the changes to the filesystem are. In this case it looks like an issue with XFS, a filesystem that is not normally recommended for usage that involves the creation and deletion of a lot of small files. Changing a little bit of data in an index file is very fast compared to the deletion of a file from a directory with 35k entries.
--
Bill Cole
bill@scconsult.com