[Dovecot] dovecot-uidlist housekeeping (was Re: Need a quick, safe method to empty /home/user/Maildir/{.Junk, .Trash})

[Bill Cole]

At 5:08 AM +0300 6/13/08, Timo Sirainen wrote:
>On Thu, 2008-06-12 at 19:28 -0400, Bill Cole wrote:
>>  At 1:17 AM +0300 6/13/08, Timo Sirainen wrote:
>>  >On Thu, 2008-06-12 at 11:10 -0400, Bill Cole wrote:
>>
>>  [...]
>>  >>  However, I DO clobber the dovecot-uidlist in .Trash as part of my
>>  >>  monthly housekeeping because it tends to get very large
>>  >
>>  >All the expunged messages get removed from it the next time a new
>>  >message is added to the mailbox. So there should be no need to delete
>>  >it. (v1.1 doesn't always recreate the file, but it gets rewritten if the
>>  >resulting file would be 70% of the current size or less.)
>>
>>  I confess to running 1.0.0, which apparently lacks this bit of housekeeping.
>>
>>  At the moment I have no messages in my Trash folder (so there are no
>>  files in ~/Maildir/.Trash/cur) and ~Maildir/.Trash/dovecot-uidlist
>>  has 10273 lines, and the first messages listed are nowhere to be
>>  found anywhere under Maildir.
>
>As I said, they're removed *the next time a new message is added to the
>mailbox*. So basically it works like:
>
>1. 1000 messages added to mailbox -> dovecot-uidlist has 1001 lines.
>2. 1000 messages expunged from mailbox -> dovecot-uidlist has 1001
>lines.
>3. 1 message added to the mailbox -> dovecot-uidlist has 2 lines.

As I said, this isn't happening with 1.0.0. I guess it might be 
another motivator to update to a more current version, if this is a 
more recent feature.
And I really am sure of this behavior. I add a steady stream of 
messages to the Trash mailbox, due to how my primary desktop client 
(MacOS Eudora 6.2.4) handles logical message deletion and movement of 
messages from the IMAP server to client-side archives. I also move 
messages from the inbox to other IMAP mailboxes based on client-side 
filters. It appears that those mailboxes which (like Trash,) only 
ever get new messages by way of Eudora using UID COPY commands are 
never getting any cleanup of their dovecot-uidlist file. I haven't 
tried to track that path in the code, but just off the top of my 
head, I would think that this could be explained if your cleanup is 
triggered by finding fresh deliveries by other software in new, but 
the UID COPY implementation avoids that. A little experimentation 
shows that moving messages on the client side results in new message 
files in the new directory, new lines in dovecot-uidlist, and changes 
to dovecot.index and dovecot.index.log. Since I'm doing initial 
delivery with procmail, obviously Dovecot is doing a different set of 
steps when it moves files from new to cur.

>Neither v1.0 nor v1.1 updates dovecot-uidlist when messages are
>expunged. I don't know if it should - it's just unnecessary extra disk
>I/O, but I guess if the space savings are large maybe it should..

My concern was not really disk space but rather the inefficiency of 
having the UID->file  map containing scores of thousands of entries 
for a mailbox that usually is empty and rarely holds more than a 
handful of messages.

-- 
Bill Cole
bill at scconsult.com