[Dovecot] Need a quick, safe method to empty /home/user/Maildir/{.Junk, .Trash}
I have some (Thunderbird client, dovecot-1.0.13, Maildir) users who get an appalling amount of spam-with-attachment, and it's causing backups to take an inordinate amount of time.
I'll implement some quota and server-side spam management when I go to dovecot-1.1, but in the meantime:
What is the safest way to empty all messages within, but not delete, the following folders from the server command line:
/home/user/Maildir/.Junk /home/user/Maildir/.Trash
I don't want the Thunderbird-2.0.14 client to report corrupt indexing, or worse not show the folder at next use.
Thanks.
At 9:22 AM -0400 6/12/08, Jeff Kowalczyk wrote:
I have some (Thunderbird client, dovecot-1.0.13, Maildir) users who get an appalling amount of spam-with-attachment, and it's causing backups to take an inordinate amount of time.
I'll implement some quota and server-side spam management when I go to dovecot-1.1, but in the meantime:
What is the safest way to empty all messages within, but not delete, the following folders from the server command line:
/home/user/Maildir/.Junk /home/user/Maildir/.Trash
I don't want the Thunderbird-2.0.14 client to report corrupt indexing, or worse not show the folder at next use.
The way to do that in a shell (rather than via Dovecot) would be to remove all of the current message files and all of Dovecot's indexing and metadata caching:
cd /home/user/Maildir/.Junk rm dovecot-index* cur/* cd /home/user/Maildir/.Trash rm dovecot-index* cur/*
(restructure that as you like, I've written it that way to make the operations clear)
There is no need normally to delete the dovecot-uidlist or dovecot-keywords files, although in principle you could do so and a well-designed client would be untroubled. The former maps IMAP UID's to specific messages, and the later maps custom keywords to numeric ID's. Dovecot will rebuild those as needed, but if you happen to whack them while a message is being delivered or moved into that folder you could end up with a confused client and lost keywords. However, I DO clobber the dovecot-uidlist in .Trash as part of my monthly housekeeping because it tends to get very large and I'm not so worried about clients getting confused about the contents of the Trash. If you are forcibly removing the messages and index files at a time when there isn't likely to be client activity in progress, there is very little risk in removing the dovecot-uidlist. There's unlikely to be any point in ever removing the dovecot-keywords file, since it tends to only have a handful of entries anyway and there's no harm in keeping the ones not actually in use by current messages.
-- Bill Cole bill@scconsult.com
On Thursday, June 12 at 11:10 AM, quoth Bill Cole:
What is the safest way to empty all messages within, but not delete, the following folders from the server command line:
/home/user/Maildir/.Junk /home/user/Maildir/.Trash
I don't want the Thunderbird-2.0.14 client to report corrupt indexing, or worse not show the folder at next use.
The way to do that in a shell (rather than via Dovecot) would be to remove all of the current message files and all of Dovecot's indexing and metadata caching:
cd /home/user/Maildir/.Junk rm dovecot-index* cur/* cd /home/user/Maildir/.Trash rm dovecot-index* cur/*
(restructure that as you like, I've written it that way to make the operations clear)
If you have a *TON* of messages in those folders (i.e. several thousand), your shell may complain that there are too many arguments to the rm command. If that happens, these may be better commands to delete them:
find /home/user/Maildir/.Junk/cur -type f -delete find /home/user/Maildir/.Trash/cur -type f -delete
... and then get rid of the index files as indicated above, of course.
~Kyle
Men, as an organization, are getting more women than any other group working anywhere in the world. Wherever women are, we have men looking into it. -- Jerry Seinfeld
On Thu, 12 Jun 2008 10:19:48 -0500, Kyle Wheeler wrote:
If you have a *TON* of messages in those folders (i.e. several thousand), your shell may complain that there are too many arguments to the rm command. If that happens, these may be better commands to delete them:
find /home/user/Maildir/.Junk/cur -type f -delete find /home/user/Maildir/.Trash/cur -type f -delete
Thanks, I'm going to do that. And per dmeissler, I'm going to consider xargs.
(http://dmiessler.com/study/find/) :
"... With xargs, you build up the input into bundles and run them through the called command as few times as possible, which is usually just once. When dealing with hundreds or thousands of elements this is a big win for xargs.
Any thoughts on a variant using find -name that could safely iterate over /home/*/Maildir for all users? Otherwise I would script it in python.
Thanks.
Jeff Kowalczyk wrote:
Any thoughts on a variant using find -name that could safely iterate over /home/*/Maildir for all users? Otherwise I would script it in python.
What about:
find /home/*/Maildir/.Junk/cur -type f -delete
Alexander
On Thu, 12 Jun 2008 19:07:36 +0200, Alexander Prinsier wrote:
Jeff Kowalczyk wrote:
Any thoughts on a variant using find -name that could safely iterate over /home/*/Maildir for all users? Otherwise I would script it in python.
What about:
find /home/*/Maildir/.Junk/cur -type f -delete
find /home/*/Maildir/{.Junk,.Trash}/cur -type d
Seems to work well as a sanity check. It even showed me that a few users don't have .Junk folders, and I should get in touch with them to configure their Thunderbird clients to process Junk mail.
But about the safety of that wildcard, are there any edge cases where that wildcard could find the whole user directory (and delete it)? All usernames a flname form (jsmith), so no punctuation involved in users created on my watch.
Thanks.
On Thursday, June 12 at 01:02 PM, quoth Jeff Kowalczyk:
On Thu, 12 Jun 2008 10:19:48 -0500, Kyle Wheeler wrote:
If you have a *TON* of messages in those folders (i.e. several thousand), your shell may complain that there are too many arguments to the rm command. If that happens, these may be better commands to delete them:
find /home/user/Maildir/.Junk/cur -type f -delete find /home/user/Maildir/.Trash/cur -type f -delete
Thanks, I'm going to do that. And per dmeissler, I'm going to consider xargs.
<shrug> xargs is good for other things, but if all you're doing is deleting files, find can do it itself without involving other applications. But whatever makes you comfortable.
Any thoughts on a variant using find -name that could safely iterate over /home/*/Maildir for all users? Otherwise I would script it in python.
find /home/ -path '*/Maildir/.Junk/cur/*' -type f -delete
~Kyle
Nothing makes a woman more beautiful than the belief she is beautiful. -- Sophia Loren
On Thu, 12 Jun 2008, Kyle Wheeler wrote:
On Thursday, June 12 at 01:02 PM, quoth Jeff Kowalczyk:
On Thu, 12 Jun 2008 10:19:48 -0500, Kyle Wheeler wrote:
If you have a *TON* of messages in those folders (i.e. several thousand), your shell may complain that there are too many arguments to the rm command. If that happens, these may be better commands to delete them:
find /home/user/Maildir/.Junk/cur -type f -delete find /home/user/Maildir/.Trash/cur -type f -delete
Thanks, I'm going to do that. And per dmeissler, I'm going to consider xargs.
<shrug> xargs is good for other things, but if all you're doing is deleting files, find can do it itself without involving other applications. But whatever makes you comfortable.
Not all 'find's are created equal. I think '-delete' is a GNU extension (or a relatively new option). It's not in Solaris 8 (Yeah, I know, it's old.), for example.
Any thoughts on a variant using find -name that could safely iterate over /home/*/Maildir for all users? Otherwise I would script it in python.
find /home/ -path '*/Maildir/.Junk/cur/*' -type f -delete
~Kyle
On Thu, 2008-06-12 at 11:10 -0400, Bill Cole wrote:
At 9:22 AM -0400 6/12/08, Jeff Kowalczyk wrote:
I have some (Thunderbird client, dovecot-1.0.13, Maildir) users who get an appalling amount of spam-with-attachment, and it's causing backups to take an inordinate amount of time.
I'll implement some quota and server-side spam management when I go to dovecot-1.1, but in the meantime:
What is the safest way to empty all messages within, but not delete, the following folders from the server command line:
/home/user/Maildir/.Junk /home/user/Maildir/.Trash
I don't want the Thunderbird-2.0.14 client to report corrupt indexing, or worse not show the folder at next use.
The way to do that in a shell (rather than via Dovecot) would be to remove all of the current message files and all of Dovecot's indexing and metadata caching:
cd /home/user/Maildir/.Junk rm dovecot-index* cur/* cd /home/user/Maildir/.Trash rm dovecot-index* cur/*
There's no real need to delete dovecot.index*. They get updated automatically. Probably better if you don't delete them, because if client has the mailbox opened at that time Dovecot will probably log errors about losing the files.
However, I DO clobber the dovecot-uidlist in .Trash as part of my monthly housekeeping because it tends to get very large
All the expunged messages get removed from it the next time a new message is added to the mailbox. So there should be no need to delete it. (v1.1 doesn't always recreate the file, but it gets rewritten if the resulting file would be 70% of the current size or less.)
At 1:17 AM +0300 6/13/08, Timo Sirainen wrote:
On Thu, 2008-06-12 at 11:10 -0400, Bill Cole wrote:
[...]
However, I DO clobber the dovecot-uidlist in .Trash as part of my monthly housekeeping because it tends to get very large
All the expunged messages get removed from it the next time a new message is added to the mailbox. So there should be no need to delete it. (v1.1 doesn't always recreate the file, but it gets rewritten if the resulting file would be 70% of the current size or less.)
I confess to running 1.0.0, which apparently lacks this bit of housekeeping.
At the moment I have no messages in my Trash folder (so there are no files in ~/Maildir/.Trash/cur) and ~Maildir/.Trash/dovecot-uidlist has 10273 lines, and the first messages listed are nowhere to be found anywhere under Maildir. A quick scan of Maildir shows that of 14 logical folders including the inbox, 6 have dovecot-uidlist files whose linecount is greater than the number of files in cur plus one.
Looking at which directories have oversized dovecot-uidlist files, it seems like they are the ones that do not get direct deliveries of new mail but only receive messages by an IMAP client shuffling messages around.
Bill Cole bill@scconsult.com
On Thu, 2008-06-12 at 19:28 -0400, Bill Cole wrote:
At 1:17 AM +0300 6/13/08, Timo Sirainen wrote:
On Thu, 2008-06-12 at 11:10 -0400, Bill Cole wrote:
[...]
However, I DO clobber the dovecot-uidlist in .Trash as part of my monthly housekeeping because it tends to get very large
All the expunged messages get removed from it the next time a new message is added to the mailbox. So there should be no need to delete it. (v1.1 doesn't always recreate the file, but it gets rewritten if the resulting file would be 70% of the current size or less.)
I confess to running 1.0.0, which apparently lacks this bit of housekeeping.
At the moment I have no messages in my Trash folder (so there are no files in ~/Maildir/.Trash/cur) and ~Maildir/.Trash/dovecot-uidlist has 10273 lines, and the first messages listed are nowhere to be found anywhere under Maildir.
As I said, they're removed *the next time a new message is added to the mailbox*. So basically it works like:
- 1000 messages added to mailbox -> dovecot-uidlist has 1001 lines.
- 1000 messages expunged from mailbox -> dovecot-uidlist has 1001 lines.
- 1 message added to the mailbox -> dovecot-uidlist has 2 lines.
Neither v1.0 nor v1.1 updates dovecot-uidlist when messages are expunged. I don't know if it should - it's just unnecessary extra disk I/O, but I guess if the space savings are large maybe it should..
At 5:08 AM +0300 6/13/08, Timo Sirainen wrote:
On Thu, 2008-06-12 at 19:28 -0400, Bill Cole wrote:
At 1:17 AM +0300 6/13/08, Timo Sirainen wrote:
On Thu, 2008-06-12 at 11:10 -0400, Bill Cole wrote:
[...]
However, I DO clobber the dovecot-uidlist in .Trash as part of my monthly housekeeping because it tends to get very large
All the expunged messages get removed from it the next time a new message is added to the mailbox. So there should be no need to delete it. (v1.1 doesn't always recreate the file, but it gets rewritten if the resulting file would be 70% of the current size or less.)
I confess to running 1.0.0, which apparently lacks this bit of housekeeping.
At the moment I have no messages in my Trash folder (so there are no files in ~/Maildir/.Trash/cur) and ~Maildir/.Trash/dovecot-uidlist has 10273 lines, and the first messages listed are nowhere to be found anywhere under Maildir.
As I said, they're removed *the next time a new message is added to the mailbox*. So basically it works like:
- 1000 messages added to mailbox -> dovecot-uidlist has 1001 lines.
- 1000 messages expunged from mailbox -> dovecot-uidlist has 1001 lines.
- 1 message added to the mailbox -> dovecot-uidlist has 2 lines.
As I said, this isn't happening with 1.0.0. I guess it might be another motivator to update to a more current version, if this is a more recent feature. And I really am sure of this behavior. I add a steady stream of messages to the Trash mailbox, due to how my primary desktop client (MacOS Eudora 6.2.4) handles logical message deletion and movement of messages from the IMAP server to client-side archives. I also move messages from the inbox to other IMAP mailboxes based on client-side filters. It appears that those mailboxes which (like Trash,) only ever get new messages by way of Eudora using UID COPY commands are never getting any cleanup of their dovecot-uidlist file. I haven't tried to track that path in the code, but just off the top of my head, I would think that this could be explained if your cleanup is triggered by finding fresh deliveries by other software in new, but the UID COPY implementation avoids that. A little experimentation shows that moving messages on the client side results in new message files in the new directory, new lines in dovecot-uidlist, and changes to dovecot.index and dovecot.index.log. Since I'm doing initial delivery with procmail, obviously Dovecot is doing a different set of steps when it moves files from new to cur.
Neither v1.0 nor v1.1 updates dovecot-uidlist when messages are expunged. I don't know if it should - it's just unnecessary extra disk I/O, but I guess if the space savings are large maybe it should..
My concern was not really disk space but rather the inefficiency of having the UID->file map containing scores of thousands of entries for a mailbox that usually is empty and rarely holds more than a handful of messages.
-- Bill Cole bill@scconsult.com
On Jun 12, 2008, at 9:22 AM, Jeff Kowalczyk wrote:
I have some (Thunderbird client, dovecot-1.0.13, Maildir) users who
get an appalling amount of spam-with-attachment, and it's causing backups
to take an inordinate amount of time./home/user/Maildir/.Junk /home/user/Maildir/.Trash
What backup software are you using? Could you exclude backing up
those folders? I have users that would be very sad if I emptied their
Trash for them (crazy- I know). Plus, there's many other benefits to
figuring out how to exclude certain paths and filetypes from backups
(~/Library/Caches or *.mp3 anyone?)
Cheers, JL
participants (7)
-
Alexander Prinsier
-
Benjamin R. Haskell
-
Bill Cole
-
Jeff Kowalczyk
-
Jurvis LaSalle
-
Kyle Wheeler
-
Timo Sirainen