[Dovecot] dovecot-uidlist cleanup
From http://www.dovecot.org/list/dovecot/2008-June/031235.html:
I have noticed that the entries in the dovecot-uidlist files don't seem to get updated when an email is deleted or moved to .Trash. It appears that this list list would just keep growing over time, being full of no longer used filenames, and must tend to increase its processing overhead.
The deleted entries are removed when a new message arrives to the mailbox. So it's just an optimization not to waste disk I/O on rewriting the file when it's not necessary. If this isn't happening, there's something wrong or you're misunderstanding something.
From http://www.dovecot.org/list/dovecot/2008-June/031403.html:
However, I DO clobber the dovecot-uidlist in .Trash as part of my monthly housekeeping because it tends to get very large
All the expunged messages get removed from it the next time a new message is added to the mailbox. So there should be no need to delete it. (v1.1 doesn't always recreate the file, but it gets rewritten if the resulting file would be 70% of the current size or less.)
If I understand this correctly, dovecot is supposed to clean out all deleted/expunged messages from dovecot-uidlist whenever a new message is added to the respective mailbox - right? This doesn't actually seem to be the case at least with dovecot-1.0:
[root@airframe]:~# dovecot --version 1.0.14 [root@airframe]:~# [root@airframe]:~# du -h ~user/Maildir/.Trash/dovecot-uidlist 13M /home/user/Maildir/.Trash/dovecot-uidlist [root@airframe]:~# wc -l < ~user/Maildir/.Trash/dovecot-uidlist 289769 [root@airframe]:~# ls -1 ~user/Maildir/.Trash/cur/ | wc -l 14 [root@airframe]:~# [root@airframe]:~# du -h ~user/Maildir/.Logs.System\ Checks/dovecot-uidlist 3,5M /home/user/Maildir/.Logs.System Checks/dovecot-uidlist [root@airframe]:~# wc -l < ~user/Maildir/.Logs.System\ Checks/dovecot-uidlist 85188 [root@airframe]:~# ls -1 ~user/Maildir/.Logs.System\ Checks/cur/ | wc -l 16069
The only mailbox where this seems to work properly is INBOX:
[root@airframe]:~# du -h ~user/Maildir/.INBOX/dovecot-uidlist 4,0K /home/user/Maildir/.INBOX/dovecot-uidlist [root@airframe]:~# wc -l < ~user/Maildir/.INBOX/dovecot-uidlist 18 [root@airframe]:~# ls -1 ~user/Maildir/.INBOX/cur/ | wc -l 17 [root@airframe]:~#
The server runs sendmail, the LDA is procmail, not dovecot's deliver. The .procmailrc is empty except for the bare minimum, though, all other sorting is done either via Thunderbird message filters or manually:
[root@airframe]:~# cat ~user/.procmailrc MAILDIR=$HOME/Maildir DEFAULT=.INBOX/ [root@airframe]:~#
Any hints?
Thanks, Thomas
At 9:00 PM +0200 6/17/08, Thomas Zajic wrote:
From http://www.dovecot.org/list/dovecot/2008-June/031235.html:
I have noticed that the entries in the dovecot-uidlist files don't seem to get updated when an email is deleted or moved to .Trash. It appears that this list list would just keep growing over time, being full of no longer used filenames, and must tend to increase its processing overhead.
The deleted entries are removed when a new message arrives to the mailbox. So it's just an optimization not to waste disk I/O on rewriting the file when it's not necessary. If this isn't happening, there's something wrong or you're misunderstanding something.
From http://www.dovecot.org/list/dovecot/2008-June/031403.html:
However, I DO clobber the dovecot-uidlist in .Trash as part of my > monthly housekeeping because it tends to get very large
All the expunged messages get removed from it the next time a new message is added to the mailbox. So there should be no need to delete it. (v1.1 doesn't always recreate the file, but it gets rewritten if the resulting file would be 70% of the current size or less.)
If I understand this correctly, dovecot is supposed to clean out all deleted/expunged messages from dovecot-uidlist whenever a new message is added to the respective mailbox - right?
That's what Timo was saying, but as I noted in http://www.dovecot.org/list/dovecot/2008-June/031435.html it isn't happening for me on 1.0.0.
This doesn't actually seem to be the case at least with dovecot-1.0:
[root@airframe]:~# dovecot --version 1.0.14
That eliminates my hypothesis that it might have been fixed in later 1.0.x versions.
[root@airframe]:~# [root@airframe]:~# du -h ~user/Maildir/.Trash/dovecot-uidlist 13M /home/user/Maildir/.Trash/dovecot-uidlist [root@airframe]:~# wc -l < ~user/Maildir/.Trash/dovecot-uidlist 289769 [root@airframe]:~# ls -1 ~user/Maildir/.Trash/cur/ | wc -l 14 [root@airframe]:~# [root@airframe]:~# du -h ~user/Maildir/.Logs.System\ Checks/dovecot-uidlist 3,5M /home/user/Maildir/.Logs.System Checks/dovecot-uidlist [root@airframe]:~# wc -l < ~user/Maildir/.Logs.System\ Checks/dovecot-uidlist 85188 [root@airframe]:~# ls -1 ~user/Maildir/.Logs.System\ Checks/cur/ | wc -l 16069
The only mailbox where this seems to work properly is INBOX:
[root@airframe]:~# du -h ~user/Maildir/.INBOX/dovecot-uidlist 4,0K /home/user/Maildir/.INBOX/dovecot-uidlist [root@airframe]:~# wc -l < ~user/Maildir/.INBOX/dovecot-uidlist 18 [root@airframe]:~# ls -1 ~user/Maildir/.INBOX/cur/ | wc -l 17 [root@airframe]:~# The server runs sendmail, the LDA is procmail, not dovecot's deliver. The .procmailrc is empty except for the bare minimum, though, all other sorting is done either via Thunderbird message filters or manually:
[root@airframe]:~# cat ~user/.procmailrc MAILDIR=$HOME/Maildir DEFAULT=.INBOX/ [root@airframe]:~# Any hints?
Your observations are consistent with mine: normal delivery with procmail (i.e. leaving a new message in the 'new' directory) of any mailbox results in Dovecot cleaning up the dovecot-uidlist file, but adding a message to a mailbox via the IMAP "UID COPY" command does not. That's the natural command to use for client filtering to move a message from one mailbox to another under the same account, so it is probably what T-Bird uses.
--
Bill Cole
bill@scconsult.com
On Tue, 2008-06-17 at 16:52 -0400, Bill Cole wrote:
Your observations are consistent with mine: normal delivery with procmail (i.e. leaving a new message in the 'new' directory) of any mailbox results in Dovecot cleaning up the dovecot-uidlist file, but adding a message to a mailbox via the IMAP "UID COPY" command does not.
Right. I wish someone would have told me about this earlier rather than kludging around it by deleting the file in a cronjob. ;)
This should help: http://hg.dovecot.org/dovecot-1.0/rev/92c3d6e64f2a and for v1.1: http://hg.dovecot.org/dovecot-1.1/rev/dbdb33bf8893
The recreate-on-append would have been nicer I think, but it doesn't work because saving a mail doesn't sync the maildir so it doesn't know which messages have been expunged.
- Timo Sirainen, 2008-06-18 02:30
Right. I wish someone would have told me about this earlier rather than kludging around it by deleting the file in a cronjob. ;)
This should help: http://hg.dovecot.org/dovecot-1.0/rev/92c3d6e64f2a and for v1.1: http://hg.dovecot.org/dovecot-1.1/rev/dbdb33bf8893 [...]
Wow ... yes, this did in fact help a lot! :-) Deleting a message in Thunderbird (= moving it to the Trash folder) is now an instant action again, rather than the 2-3+ seconds (and sometimes even more) it used to take before with the huge dovecot-uidlist:
[root@airframe]:~# du -h ~user/Maildir/.Trash/dovecot-uidlist 8,0K /home/user/Maildir/.Trash/dovecot-uidlist [root@airframe]:~# wc -l < ~user/Maildir/.Trash/dovecot-uidlist 104 [root@airframe]:~# ls -1 ~user/Maildir/.Trash/cur/ | wc -l 103 [root@airframe]:~# [root@airframe]:~# du -h ~user/Maildir/.Logs.System\ Checks/dovecot-uidlist 664K /home/user/Maildir/.Logs.System Checks/dovecot-uidlist [root@airframe]:~# wc -l < ~user/Maildir/.Logs.System\ Checks/dovecot-uidlist 16112 [root@airframe]:~# ls -1 ~user/Maildir/.Logs.System\ Checks/cur/ | wc -l 16110 [root@airframe]:~#
Thanks a lot Timo, you rock! :-)
Bye, Thomas
PS: I would have told you earlier, but I didn't even know that dovecot was supposed to shrink its dovecot-uidlist files every now and then. ;-) I only checked these files when the discussion came up recently, and only then I found out that I actually suffered from the same problem.
On Wed, 2008-06-18 at 07:33 +0200, Thomas Zajic wrote:
Wow ... yes, this did in fact help a lot! :-) Deleting a message in Thunderbird (= moving it to the Trash folder) is now an instant action again, rather than the 2-3+ seconds (and sometimes even more) it used to take before with the huge dovecot-uidlist:
BTW. If anything except SEARCH takes more than an instant in a mailbox smaller than 10k messages on a non-loaded server, it's most likely a bug. And I just guessed that 10k value because that's how large many of my mailboxes are on a 10 years old server, probably 30-50k messages should still be ok.
- Timo Sirainen, 2008-06-18 07:43
BTW. If anything except SEARCH takes more than an instant in a mailbox smaller than 10k messages on a non-loaded server, it's most likely a bug. And I just guessed that 10k value because that's how large many of my mailboxes are on a 10 years old server, probably 30-50k messages should still be ok.
Thanks for the hint, I'll keep that in mind. :-)
Bye, Thomas
participants (3)
-
Bill Cole
-
Thomas Zajic
-
Timo Sirainen