[Dovecot] deleting maildir files
Greetings all...
Can we delete maildir files directly from the file system?
Basically, we use dovecot to train spam, and we want to delete messages that are older than 30 days, using a simple "find /maildirs -cname +30 -name *imap-server* -exec rm {} \;"
I have seen a post a while ago saying that dovecot can rebuild the indexes, but I don't remember if it's when the index is deleted or when the maildir files are deleted.
And if we cannot delete files with the 'rm' command, whats the best/proper way to delete these older files.
Thanks
Can we delete maildir files directly from the file system?
Yes.
Does this also go for special folders inside a maildir? Like, for example, the .Trash folder? I have a user who was trying to delete about 30,000 messages from a folder, and when he did this, TBird went into never-never land, and now every time he clicks on the Trash folder in TBird, it goes into never-never land again. I have already deleted the local cache files from his TBird profile, so there is something wrong on the server side.
I don't have much experience manually manipulating maildirs, so would appreciate comments...
What I was going to do was first delete the folder:
rm -ri /var/virtual/domain.com/maildir/.Trash
then once it is all gone, do:
maildirmake -f Trash /var/virtual/domain.com/maildir
By the way, if it matters, this server is still running Courier I'm sorry to say, but I think they're ready to let me migrate them to dovecot now that it has reached 'stable' status.
--
Best regards,
Charles
On 7/17/2007, Charles Marcus (CMarcus@Media-Brokers.com) wrote:
What I was going to do was first delete the folder:
rm -ri /var/virtual/domain.com/maildir/.Trash
then once it is all gone, do:
maildirmake -f Trash /var/virtual/domain.com/maildir
Anyone see any problems with fixing this this way? It is really getting annoying not being able to delete anything...
Thanks,
--
Best regards,
Charles
On 17.7.2007, at 19.56, Charles Marcus wrote:
On 7/17/2007, Charles Marcus (CMarcus@Media-Brokers.com) wrote:
What I was going to do was first delete the folder: rm -ri /var/virtual/domain.com/maildir/.Trash then once it is all gone, do: maildirmake -f Trash /var/virtual/domain.com/maildir
Anyone see any problems with fixing this this way? It is really
getting annoying not being able to delete anything...
Or just delete .Trash/cur and mkdir it back. That way UIDVALIDITY
doesn't change (although I think TB is able to handle it, but at
least Mail.app broke really badly last time I did that).
Timo Sirainen, on 7/17/2007 1:05 PM, said the following:
On 17.7.2007, at 19.56, Charles Marcus wrote:
On 7/17/2007, Charles Marcus (CMarcus@Media-Brokers.com) wrote:
What I was going to do was first delete the folder: rm -ri /var/virtual/domain.com/maildir/.Trash then once it is all gone, do: maildirmake -f Trash /var/virtual/domain.com/maildir
Anyone see any problems with fixing this this way? It is really getting annoying not being able to delete anything...
Or just delete .Trash/cur and mkdir it back. That way UIDVALIDITY doesn't change (although I think TB is able to handle it, but at least Mail.app broke really badly last time I did that).
Hmmm... but will that take care of /tmp as well? Don't forget, this is a different system that is still served by courier - but happily I've been given a gree light to start planning their migration to dovecot...
Anyway - here is the directory listing as it is now:
moria .Trash # ls -al total 47960 drwx------ 6 postfix postfix 256 Jul 17 07:08 . drwx------ 116 postfix postfix 4192 Jul 17 14:27 .. -rw-r--r-- 1 postfix postfix 17 Oct 31 2005 courierimapacl drwx------ 2 postfix postfix 72 Jul 17 07:08 courierimapkeywords -rw-r--r-- 1 postfix postfix 7603094 Jul 17 07:08 courierimapuiddb drwx------ 2 postfix postfix 9100472 Jul 17 12:40 cur -rw------- 1 postfix postfix 0 Oct 31 2005 maildirfolder drwx------ 2 postfix postfix 48 Oct 31 2005 new drwx------ 2 postfix postfix 32388568 Jul 17 12:04 tmp
And when I click on the Trash folder in Thunderbird, it just goes into never-never land...
--
Best regards,
Charles
On Tue, 2007-07-17 at 14:34 -0400, Charles Marcus wrote:
Or just delete .Trash/cur and mkdir it back. That way UIDVALIDITY doesn't change (although I think TB is able to handle it, but at least Mail.app broke really badly last time I did that).
Hmmm... but will that take care of /tmp as well?
tmp should be empty usually.
drwx------ 2 postfix postfix 32388568 Jul 17 12:04 tmp
Looks like Courier has crashed in the middle of copying/saving if it has this many files. :)
Timo Sirainen, on 7/17/2007 2:40 PM, said the following:
On Tue, 2007-07-17 at 14:34 -0400, Charles Marcus wrote:
Or just delete .Trash/cur and mkdir it back. That way UIDVALIDITY doesn't change (although I think TB is able to handle it, but at least Mail.app broke really badly last time I did that).
Hmmm... but will that take care of /tmp as well?
tmp should be empty usually.
drwx------ 2 postfix postfix 32388568 Jul 17 12:04 tmp
Looks like Courier has crashed in the middle of copying/saving if it has this many files. :)
I know... so - I should delete it as well, then recreate it, same as /cur? What about the courierimapuiddb file? (Also, sorry for asking something about courier, but I've already asked on the courier list and no one has responded yet)...
Sorry, guess I'm displaying my ignorance... but I'd rather be safe than sorry...
Thanks,
--
Best regards,
Charles
On Tue, 2007-07-17 at 15:11 -0400, Charles Marcus wrote:
Timo Sirainen, on 7/17/2007 2:40 PM, said the following:
On Tue, 2007-07-17 at 14:34 -0400, Charles Marcus wrote:
Or just delete .Trash/cur and mkdir it back. That way UIDVALIDITY doesn't change (although I think TB is able to handle it, but at least Mail.app broke really badly last time I did that).
Hmmm... but will that take care of /tmp as well?
tmp should be empty usually.
drwx------ 2 postfix postfix 32388568 Jul 17 12:04 tmp
Looks like Courier has crashed in the middle of copying/saving if it has this many files. :)
I know... so - I should delete it as well, then recreate it, same as /cur?
Yea. Or I mean in both cases you can just delete the files inside the directory, but I'm pretty sure rm tmp/* will give you an error. :)
What about the courierimapuiddb file? (Also, sorry for asking something about courier, but I've already asked on the courier list and no one has responded yet)...
Courier's courierimapuiddb file is pretty much the same as dovecot-uidlist file. You don't have to worry about it.
Timo Sirainen wrote:
Yea. Or I mean in both cases you can just delete the files inside the directory, but I'm pretty sure rm tmp/* will give you an error. :)
This:
$ ls -f ./tmp ./cur | xargs rmis really quite efficient, and doesn't fall prey to commandline globbing limits...
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748
On Tuesday, July 17 at 03:45 PM, quoth John Peacock:
This:
$ ls -f ./tmp ./cur | xargs rm
is really quite efficient, and doesn't fall prey to commandline globbing limits...
Another one I use often is:
find ./tmp ./cur -type f -exec rm {} \;~Kyle
The real problem is not whether machines think, but whether men do. -- B. F. Skinner
Am Mittwoch, 18. Juli 2007 schrieb Kyle Wheeler:
Another one I use often is: find ./tmp ./cur -type f -exec rm {} \;
Current GNU finds understand the -delete action:
find ./tmp ./cur -type f -delete
That should be as fast as the "ls" solution without getting you in trouble if a \r manages to get into one of the filenames somehow.
Another safe and fast solution would be
find ./tmp ./cur -type f -print0 | xargs -0 rm
which should also work with older GNU finds. I do not know which one is better supported (if at all) in finds from other vendors.
In any case both should be much faster than spawning an "rm" for every file, although this solution should also be safe. I would not trust the piped "ls-output-solution" (given in the grandparent) on a production system, however.
Greetings,
Gunter
I know... so - I should delete it as well, then recreate it, same as /cur?
Yea. Or I mean in both cases you can just delete the files inside the directory, but I'm pretty sure rm tmp/* will give you an error. :)
Yeah, me too - since it won't even list the files in there (ls -al goes into never-never land too)...
What about the courierimapuiddb file? (Also, sorry for asking something about courier, but I've already asked on the courier list and no one has responded yet)...
Courier's courierimapuiddb file is pretty much the same as dovecot-uidlist file. You don't have to worry about it.
Cool - many thanks...
--
Best regards,
Charles
Charles Marcus wrote:
Hmmm... but will that take care of /tmp as well? Don't forget, this is a different system that is still served by courier - but happily I've been given a gree light to start planning their migration to dovecot...
Don't forget that files exist in tmp only while they are being written to disk (and then they are mv'd directly to new). It is _never_ a problem to delete everything under cur. You don't even need to blow away the directory itself, just the files contained within ./cur...
If you want to be really paranoid, just make sure that the client is not accessing the mailbox at the time.
And when I click on the Trash folder in Thunderbird, it just goes into never-never land...
Just upgrading to Dovecot might fix that; also make sure you are using the latest TBird 2.x release (there were some known timeout issues fixed after the 1.5.x series).
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748
Tom Bombadil wrote:
Greetings all...
Can we delete maildir files directly from the file system?
Basically, we use dovecot to train spam, and we want to delete messages that are older than 30 days, using a simple "find /maildirs -cname +30 -name *imap-server* -exec rm {} \;"
I have seen a post a while ago saying that dovecot can rebuild the indexes, but I don't remember if it's when the index is deleted or when the maildir files are deleted.
And if we cannot delete files with the 'rm' command, whats the best/proper way to delete these older files.
Thanks
My opinion has always been that the data structure should not be replied upon.... if you want to do things with your mail items, then use the APIs/imap commands to do it. That protects you from any internal stuff you didn't know about, or changed from one server to another, or one version to another.
For example, I have the beginnings of a script to handle my "mail retention policies". It connects to Dovecot/imap to get the list of mail for specific folders, then uses the imap delete (or copy) API to delete mail older than n days, or to keep only the most recent n messages. That sort of thing.
The benefit is my script then doesn't care how Dovecot (or whichever server) stores things.... and if a mailbox changes from mbox to maildir format, or similar change.. no worries... my script just doesn't care... it always uses the APIs to manipulate mail.
Safe, but admittedly not as fast. I favor reliability over speed in these sorts of cases. :-)
- Don Russell [2007-07-16 10:23]:
Can we delete maildir files directly from the file system? [...] And if we cannot delete files with the 'rm' command, whats the best/proper way to delete these older files.
My opinion has always been that the data structure should not be replied upon.... if you want to do things with your mail items, then use the APIs/imap commands to do it. That protects you from any internal stuff you didn't know about, or changed from one server to another, or one version to another.
Maildir is a sort of a standard with some sort of an API, isn't it? The "standard" (<http://cr.yp.to/proto/maildir.html>) says following:
An MUA can read and delete messages while new mail is being
delivered: each message is stored in a separate file with a
unique name, so it isn't affected by operations on other messages.For example, I have the beginnings of a script to handle my "mail retention policies". It connects to Dovecot/imap to get the list of mail for specific folders, then uses the imap delete (or copy) API to delete mail older than n days, or to keep only the most recent n messages. That sort of thing.
With Maildir it's trivial to do this sort of things with a shell script.
The benefit is my script then doesn't care how Dovecot (or whichever server) stores things.... and if a mailbox changes from mbox to maildir format, or similar change.. no worries... my script just doesn't care... it always uses the APIs to manipulate mail.
Safe, but admittedly not as fast. I favor reliability over speed in these sorts of cases. :-)
That's a valid point. It's much easier to shoot one's own leg pretty ugly, when deleting/renaming/whatevert the files in the Maildir directly. IMAP SEARCH is IMHO a bit easier to understand than find(1).
Best wishes, Kirill
-- #!/usr/bin/perl -w print(&{sub{eval(qq(q(@_)))}}((join(''=>map{ord=~m(^106)?uc:lc}($[=> map{chr}(97..122))[map{int}grep{length}split(/(\d\d)/,'10211920011'. qq(41520080518190907140120211805))]))=~m(\A(\w{4})(\S+)(s\D+)$)),$/)
Kirill Miazine wrote:
- Don Russell [2007-07-16 10:23]:
Can we delete maildir files directly from the file system? [...] And if we cannot delete files with the 'rm' command, whats the best/proper way to delete these older files.
My opinion has always been that the data structure should not be replied upon.... if you want to do things with your mail items, then use the APIs/imap commands to do it. That protects you from any internal stuff you didn't know about, or changed from one server to another, or one version to another.
Maildir is a sort of a standard with some sort of an API, isn't it? The "standard" (<http://cr.yp.to/proto/maildir.html>) says following:
An MUA can read and delete messages while new mail is being delivered: each message is stored in a separate file with a unique name, so it isn't affected by operations on other messages.
True, but why tie my script to one format, when it can work equally well with *any* format supported by the IMAP server? :-) If new and improved formats develop, I don't have to rewrite my scripts. :-)
For example, I have the beginnings of a script to handle my "mail retention policies". It connects to Dovecot/imap to get the list of mail for specific folders, then uses the imap delete (or copy) API to delete mail older than n days, or to keep only the most recent n messages. That sort of thing.
With Maildir it's trivial to do this sort of things with a shell script.
Yes, I suppose it is. :-) However, in my case (and I wrote the script for my own use) my mail is in mbox format. Eventually I want to convert it to maildir or something else (dbox?) and since my script uses the IMAP interface, I won't have to change anything, the IMAP server does all the heavy lifting. :-)
The benefit is my script then doesn't care how Dovecot (or whichever server) stores things.... and if a mailbox changes from mbox to maildir format, or similar change.. no worries... my script just doesn't care... it always uses the APIs to manipulate mail.
Safe, but admittedly not as fast. I favor reliability over speed in these sorts of cases. :-)
That's a valid point. It's much easier to shoot one's own leg pretty ugly, when deleting/renaming/whatevert the files in the Maildir directly. IMAP SEARCH is IMHO a bit easier to understand than find(1).
Best wishes, Kirill
Cheers :-)
participants (8)
-
Charles Marcus
-
Don Russell
-
Gunter Ohrner
-
John Peacock
-
Kirill Miazine
-
Kyle Wheeler
-
Timo Sirainen
-
Tom Bombadil