excellent :)

thats good, I have used the same method to remove mail after my mail filter learns it as spam now.

Everythings comming together nicley with my new dovecot server, althought the server itself is still mostly in bits on the floor :)

Thanks,

Matt.

On Tue, Feb 12, 2008 at 10:45:08PM +0100, Rody wrote:

Funny, I asked a few days ago a similar question with the subject: [Dovecot] replacement for IMAP_EMPTYTRASH=Trash:7 You may want to look for the responses to that question as it's essentially the same. In short: you can do this with a find command to remove older mails. There should be no issues as maildir files don't need to be locked in order to do it properly. The opinions vary slightly when it comes to using mtime or ctime. I've chosen ctime because i believe using mtime will not garantee that there aren't any mails left which are actually older than 30 days. I believe there are cases where mtime may get changed, where ctime will not. Also, ctime starts counting from the moment the mail gets dropped in a certain mailbox. Anyway for 30 day's i'd say you need ctime +29 or mtime +29 as the man pages explain that +0 means actually up until 1 day old.

Regards, Rody

Op dinsdag 12 februari 2008 19:15, schreef Matt Richards:

...

Hello,

I have dovecot running with MailDir as a backend to store email and I would like to remove mail that is older that 30 days.

I can do this by running 'find' on the MailDir but will this cause any issues with dovecot?

Thanks,

Matt.

-- My software never has bugs. It just develops random features. Matt Richards

From what i have seen, you're absolutely correct. It looks like the use of ctime or mtime depends on wether you want the message removed x days after it was moved to say the trash folder (ctime) or will be removed x days after it originally arrived in the inbox (mtime). My personal opinion is currently that i would like it removed x days after it was placed in a certain folder, hence i use ctime.

Rody

Op woensdag 13 februari 2008 00:43, schreef Bill Cole:

At 10:45 PM +0100 2/12/08, Rody imposed structure on a stream of

electrons, yielding:

...

The opinions vary slightly when it comes to using mtime or ctime. I've chosen ctime because i believe using mtime will not garantee that there aren't any mails left which are actually older than 30 days. I believe there are cases where mtime may get changed, where ctime will not. Also, ctime starts counting from the moment the mail gets dropped in a certain mailbox.

Yes, but you may also care that ctime is reset when a client has Dovecot move a message from one subfolder to another within a Maildir. I'm not sure why Dovecot does it, but a look at the messages in the non-INBOX parts of my Maildir reveals that the ctime is always later than the mtime, and the contents (Received headers) makes it clear that Dovecot sets the mtime of messages to the original mtime (i.e. original delivery time) when copying them.

Hopefully Timo will speak up on this, but I have a vague recollection of him saying that Dovecot never modifies message contents as a matter of principle, and it seems to me that the design of Maildir assumes that the mailstore server follows that principle rigorously. That should make mtime quite static for an individual file, and it looks to me like Dovecot even makes an effort to preserve the delivery time of a message by replicating the mtime from the original file to the new one when copying a message between subfolders.

On Feb 13, 2008, at 7:21 AM, Benjamin R. Haskell wrote:

On Wed, 13 Feb 2008, Rody wrote:

...

Op woensdag 13 februari 2008 00:43, schreef Bill Cole:

...

Yes, but you may also care that ctime is reset when a client has Dovecot move a message from one subfolder to another within a Maildir. I'm not sure why Dovecot does it, but a look at the
messages in the non-INBOX parts of my Maildir reveals that the ctime is
always later than the mtime, and the contents (Received headers) makes it clear that Dovecot sets the mtime of messages to the original mtime (i.e. original delivery time) when copying them.

I think the answer to "why Dovecot does it" is actually that Dovecot
doesn't do anything with ctime. Under most *nix filesystems, ctime
is the last time the inode underlying the file/dir was changed ('c'
for "changed", not "created" -- [usually]). The inode gets changed
when the file's moved from one directory to another.

Right. Also there's no way to change ctime even if I wanted to.

The delivery time can also be determined by the first part of the
Maildir filename of the message:

e.g. ls -l 1202878863.24522_1.myhost:2, --rw------- 1 user group 551 2008-02-13 00:00
1202878863.24522_1.myhost:2,

The 1202878863 part is the Unix timestamp corresponding to midnight,
Feb 13, 2008 EST. So, it's not really a requirement of Maildir's
design to keep the mtime updated. It's just easier+quicker than
parsing the number out of the filename (where it *is* a requirement).

It's required to write it, but it's also required that you shouldn't
assume anything about the filename when reading it. So for example
some mbox -> maildir convertors use "mbox.#" as filenames and they
work fine.

...

[...] It looks like the use of ctime or mtime depends on wether you
want the message removed x days after it was moved to say the trash
folder (ctime) or will be removed x days after it originally
arrived in the inbox (mtime). My personal opinion is currently that
i would like it removed x days after it was placed in a certain
folder, hence i use ctime.

Depends on the context for me. If it's in a spam folder, I'd rather
not keep it around any longer than it'd take for someone to say
"Hey, didn't you get that?". So -mtime +13 (2 weeks from receipt).
(Thanks for the heads-up on +13 vs. +14, btw).

Also note that if messages are uploaded to server by IMAP client the
mtime may be the message's original date. Dovecot uses mtime as the
message's IMAP INTERNALDATE.

On 2/13/2008 Benjamin R. Haskell wrote:

Depends on the context for me. If it's in a spam folder, I'd rather not keep it around any longer than it'd take for someone to say "Hey, didn't you get that?". So -mtime +13 (2 weeks from receipt). (Thanks for the heads-up on +13 vs. +14, btw).

Trash on the other hand, seems like you'd want X to be "about a month" in: "Oh, no! I accidentally deleted that thing X ago!", since messages tend to stick around for a while in the INBOX. (Note the difference: "Oh, no! I accidentally deleted that thing that *arrived* X ago!".) So, -ctime +30 (trashed more than a month ago).

Re-scanning this thread, no one ever mentions what version, so, since no one's mentioned it yet - the 'Expire' plugin is made for this, and would be a good reason to move to 1.1 tree, if you're comfortable running non-release/RC versions...

http://wiki.dovecot.org/Plugins/Expire

Bill Cole ha scritto:

(one can argue that HFS+ is Bad and Wrong in many ways, but it is in widespread use...)

Widespread? On servers?

Well maybe would be nice to add a sort of warning to all apple sysadmin out there in a evenctual future faq.

cheers

-- Andrea Perotti

Bill Cole wrote:

At 12:21 AM -0500 2/13/08, Benjamin R. Haskell imposed structure on a stream of electrons, yielding:

...

On Wed, 13 Feb 2008, Rody wrote:

...

Op woensdag 13 februari 2008 00:43, schreef Bill Cole:

...

Yes, but you may also care that ctime is reset when a client has Dovecot move a message from one subfolder to another within a Maildir. I'm not sure why Dovecot does it, but a look at the messages in the non-INBOX parts of my Maildir reveals that the ctime is always later than the mtime, and the contents (Received headers) makes it clear that Dovecot sets the mtime of messages to the original mtime (i.e. original delivery time) when copying them.

I think the answer to "why Dovecot does it" is actually that Dovecot doesn't do anything with ctime. Under most *nix filesystems, ctime is the last time the inode underlying the file/dir was changed ('c' for "changed", not "created" -- [usually]). The inode gets changed when the file's moved from one directory to another.

Not on all filesystems. Note what HFS+ (MacOS) does:

~ $ ls -lc foo -rwxr-xr-x 1 wkc wkc 332 Jan 29 03:32 foo ~ $ mkdir foodir ~ $ mv foo foodir ~ $ ls -lc foodir/foo -rwxr-xr-x 1 wkc wkc 332 Jan 29 03:32 foodir/foo ~ $ date Wed Feb 13 08:39:24 EST 2008

The question is whether this is because of an fs limitation or is it for compatibility with some old tools.

Posix says:

Upon successful completion, /rename/() shall mark for update the /st_ctime/ and /st_mtime/ fields of the parent directory of each file.

and ctime is the last status change time. AFAICT, an mv is certainly a status change.

but maybe I disgress:)

At 11:53 PM +0100 2/13/08, mouss imposed structure on a stream of electrons, yielding:

Bill Cole wrote:

[...]

...

Not on all filesystems. Note what HFS+ (MacOS) does:

~ $ ls -lc foo -rwxr-xr-x 1 wkc wkc 332 Jan 29 03:32 foo ~ $ mkdir foodir ~ $ mv foo foodir ~ $ ls -lc foodir/foo -rwxr-xr-x 1 wkc wkc 332 Jan 29 03:32 foodir/foo ~ $ date Wed Feb 13 08:39:24 EST 2008

The question is whether this is because of an fs limitation or is it for compatibility with some old tools.

Posix says:

Upon successful completion, /rename/() shall mark for update the /st_ctime/ and /st_mtime/ fields of the parent directory of each file.

and ctime is the last status change time. AFAICT, an mv is certainly a status change.

but maybe I disgress:)

Since nothing but your POSIX quote refers to the ctime of the parent directory, maybe so. :)

I think that when you rename() (i.e. 'mv') a file, its ctime should change, if only because that is what traditional (e.g. UFS) filesystems do. I know better than to argue technical issues like that with Apple, just as I know better than to use my head to dismantle a brick wall, with the main difference being that I've never actually made the brick wall attempt.

But the relevant point is that Dovecot itself seems untroubled by this oddity.

-- Bill Cole
bill@scconsult.com

On Wed, 13 Feb 2008, Bill Cole wrote:

At 12:21 AM -0500 2/13/08, Benjamin R. Haskell imposed structure on a stream of electrons, yielding:

...

On Wed, 13 Feb 2008, Rody wrote:

...

Op woensdag 13 februari 2008 00:43, schreef Bill Cole:

...

Yes, but you may also care that ctime is reset when a client has Dovecot move a message from one subfolder to another within a Maildir. I'm not sure why Dovecot does it, but a look at the messages in the non-INBOX parts of my Maildir reveals that the ctime is always later than the mtime, and the contents (Received headers) makes it clear that Dovecot sets the mtime of messages to the original mtime (i.e. original delivery time) when copying them.

I think the answer to "why Dovecot does it" is actually that Dovecot doesn't do anything with ctime. Under most *nix filesystems, ctime is the last time the inode underlying the file/dir was changed ('c' for "changed", not "created" -- [usually]). The inode gets changed when the file's moved from one directory to another.

Not on all filesystems. Note what HFS+ (MacOS) does:

[example of ctime not being updated]

Hence my qualifying it as "Under most *nix filesystems". But, good to know there's a commonly-used exception.

...

...

...

Hopefully Timo will speak up on this, but I have a vague recollection of him saying that Dovecot never modifies message contents as a matter of principle, and it seems to me that the design of Maildir assumes that the mailstore server follows that principle rigorously. That should make mtime quite static for an individual file, and it looks to me like Dovecot even makes an effort to preserve the delivery time of a message by replicating the mtime from the original file to the new one when copying a message between subfolders.

The delivery time can also be determined by the first part of the Maildir filename of the message:

e.g. ls -l 1202878863.24522_1.myhost:2, --rw------- 1 user group 551 2008-02-13 00:00 1202878863.24522_1.myhost:2,

The 1202878863 part is the Unix timestamp corresponding to midnight, Feb 13, 2008 EST. So, it's not really a requirement of Maildir's design to keep the mtime updated. It's just easier+quicker than parsing the number out of the filename (where it *is* a requirement).

Maildir DOES NOT require a timestamp in the filename, it's just common.

DJB's Maildir spec isn't RFC-esque (so it's not a MUST, in that sense). As far as I've seen, everything does what's described below. But, like you say, not a requirement. I think I skipped over the second and third sentences from this excerpt from http://cr.yp.to/proto/maildir.html (because of the first sentence) :

["" Unless you're writing messages to a maildir, the format of a unique name is none of your business. A unique name can be anything that doesn't contain a colon (or slash) and doesn't start with a dot. Do not try to extract information from unique names.

Okay, so you're writing messages. A unique name has three pieces, separated by dots. On the left is the result of time() or the second counter from gettimeofday(). ""]

Also note that that snippet expressly warns against doing what I said could be done (parsing the timestamp out of the filename).

I'm curious: do you have examples of mail software that doesn't use the timestamp? (I could see some run-once script not doing it, but I'd be surprised if widely-used software didn't.)

On my system, the timestamps in the filenames match the ctime, NOT the mtime:

~/Maildir/.Tagged/cur $ ls -lcrt |tail -1 -rw------- 1 wkc staff 13293 Feb 12 23:45 1202877918.P26840Q0M183345.toaster.scconsult.com:2,S ~/Maildir/.Tagged/cur $ ls -l 1202877918.P26840Q0M183345.toaster.scconsult.com\:2\,S -rw------- 1 wkc staff 13293 Feb 12 23:43 1202877918.P26840Q0M183345.toaster.scconsult.com:2,S ~/Maildir/.Tagged/cur wkc$ uepoch 1202877918 1202877918 is Tue Feb 12 23:45:18 2008

The earlier mtime matches the delivery time.

I think it'd generally be the case that the filename timestamp will match the *initial* ctime (before any possible ctime changes), since that should be synonymous with creation time. And, mtime could vary, because Dovecot syncs the mtime to the delivery time.

But, I still don't quite follow your example. Call the mailfile X (so I don't have to type it):

ls -lc X shows 23:45 - so, ctime = 23:45 ls -l X shows 23:43 - so, mtime = 23:43 And, the filename timestamp == ctime.

But, you say the mtime matches the delivery time. So, the filename timestamp is two minutes *after* the delivery time? Did it take your LDA two minutes (or at least 1 min 19sec) to get the message from the MTA to the Maildir?

Also, I think (but, maybe not on HFS+) there'd be a ctime change on renaming the file from X to X,S (adding the 'Seen' flag). Do you know whether that's the case?

(e.g. try:) touch foo ls -lc foo mv foo foo,S ls -lc foo

...

...

[...] It looks like the use of ctime or mtime depends on wether you want the message removed x days after it was moved to say the trash folder (ctime) or will be removed x days after it originally arrived in the inbox (mtime). My personal opinion is currently that i would like it removed x days after it was placed in a certain folder, hence i use ctime.

Depends on the context for me. [... blah blah. Me blabbering...]

Again, that is FS-specific, and HFS+ is again a counter-example.

(one can argue that HFS+ is Bad and Wrong in many ways, but it is in widespread use...)

I guess the take-home message is that ctime isn't quite as consistent cross-filesystem as mtime. But, I still think ctime has its place in non-"Bad and Wrong" filesystems.

And, as Charles pointed out, in Dovecot 1.1 there's a plugin to do this. Does the expunge plugin work without the user logging in, though? (not that that's an enormous problem -- more curiosity)

Best, Ben

At 12:25 AM -0500 2/14/08, Benjamin R. Haskell imposed structure on a stream of electrons, yielding:

On Wed, 13 Feb 2008, Bill Cole wrote:

[...]

...

Maildir DOES NOT require a timestamp in the filename, it's just common.

DJB's Maildir spec isn't RFC-esque (so it's not a MUST, in that sense). As far as I've seen, everything does what's described below. But, like you say, not a requirement. I think I skipped over the second and third sentences from this excerpt from http://cr.yp.to/proto/maildir.html (because of the first sentence) :

["" Unless you're writing messages to a maildir, the format of a unique name is none of your business. A unique name can be anything that doesn't contain a colon (or slash) and doesn't start with a dot. Do not try to extract information from unique names.

Okay, so you're writing messages. A unique name has three pieces, separated by dots. On the left is the result of time() or the second counter from gettimeofday(). ""]

Also note that that snippet expressly warns against doing what I said could be done (parsing the timestamp out of the filename).

Yeah, and I did a little more research and am now less sure that it is *safe* to not use the timestamp. I would not be shocked if something out there working with Maildir/Maildir++ requires it, given the fuzzy spec(s)

Incidentally, some other illuminating docs are:

http://www.qmail.org/man/man5/maildir.html http://www.inter7.com/courierimap/README.maildirquota.html

I'm curious: do you have examples of mail software that doesn't use the timestamp? (I could see some run-once script not doing it, but I'd be surprised if widely-used software didn't.)

The procmailrc man page says that MSGPREFIX defaults to 'msg.' but it is not clear to me whether MSGPREFIX is used for Maildir delivery or just for MH delivery. I've played it safe and use "MSGPREFIX=date +%s" but maybe that's not needed...

...

On my system, the timestamps in the filenames match the ctime, NOT the mtime:

~/Maildir/.Tagged/cur $ ls -lcrt |tail -1 -rw------- 1 wkc staff 13293 Feb 12 23:45 1202877918.P26840Q0M183345.toaster.scconsult.com:2,S ~/Maildir/.Tagged/cur $ ls -l 1202877918.P26840Q0M183345.toaster.scconsult.com\:2\,S -rw------- 1 wkc staff 13293 Feb 12 23:43 1202877918.P26840Q0M183345.toaster.scconsult.com:2,S ~/Maildir/.Tagged/cur wkc$ uepoch 1202877918 1202877918 is Tue Feb 12 23:45:18 2008

The earlier mtime matches the delivery time.

I think it'd generally be the case that the filename timestamp will match the *initial* ctime (before any possible ctime changes), since that should be synonymous with creation time. And, mtime could vary, because Dovecot syncs the mtime to the delivery time.

It seems that Dovecot maintains the mtime as the original delivery time of the message, i.e. delivery to the INBOX. It seems that Dovecot names files based on when the specific file (i.e. in that maildir folder/subdirectory) was created.

But, I still don't quite follow your example. Call the mailfile X (so I don't have to type it):

ls -lc X shows 23:45 - so, ctime = 23:45 ls -l X shows 23:43 - so, mtime = 23:43 And, the filename timestamp == ctime.

But, you say the mtime matches the delivery time.

Yes, it does.

So, the filename timestamp is two minutes *after* the delivery time?

Yes.

Did it take your LDA two minutes (or at least 1 min 19sec) to get the message from the MTA to the Maildir?

No.

It took 2 minutes for a MUA (Eudora) to see the message and tell Dovecot to move it to the "Tagged" IMAP folder (i.e. the .Tagged subdirectory)

Also, I think (but, maybe not on HFS+) there'd be a ctime change on renaming the file from X to X,S (adding the 'Seen' flag). Do you know whether that's the case?

(e.g. try:) touch foo ls -lc foo mv foo foo,S ls -lc foo

No need to touch, as I had a foo hanging around with a usefully aged ctime that would be visible if it changed:

~ $ ls -l foo -rwxr-xr-x 1 wkc wkc 332 Jan 29 03:32 foo ~ $ ls -lc foo -rwxr-xr-x 1 wkc wkc 332 Jan 29 03:32 foo ~ $ mv foo foo,S ~ $ ls -lc foo,S -rwxr-xr-x 1 wkc wkc 332 Jan 29 03:32 foo,S

Again, this is *HFS+* so it's goofy.

On Solaris UFS and VxFS 'mv' changes ctime. I'd bet on ext{2,3} behaving similarly.

...

...

...

[...] It looks like the use of ctime or mtime depends on wether you want the message removed x days after it was moved to say the trash folder (ctime) or will be removed x days after it originally arrived in the inbox (mtime). My personal opinion is currently that i would like it removed x days after it was placed in a certain folder, hence i use ctime.

Depends on the context for me. [... blah blah. Me blabbering...]

Again, that is FS-specific, and HFS+ is again a counter-example.

(one can argue that HFS+ is Bad and Wrong in many ways, but it is in widespread use...)

I guess the take-home message is that ctime isn't quite as consistent cross-filesystem as mtime. But, I still think ctime has its place in non-"Bad and Wrong" filesystems.

I think a case could be made to support the HFS+ behavior IFF you ignore 20 years of Unix FS behavior. #include applebugsme.h

The diversity of timestamps is useful to support a diversity of reasonable policies. It seems that Dovecot works to preserve the initial delivery time (as mtime) and that even on a filesystem where rename() does not update ctime, the way Dovecot handles MUA-initiated message moves assures that the ctime and the (new) filename both reflect the move time.

And, as Charles pointed out, in Dovecot 1.1 there's a plugin to do this. Does the expunge plugin work without the user logging in, though? (not that that's an enormous problem -- more curiosity)

A lot of this is mostly about curiousity...

-- Bill Cole
bill@scconsult.com

On Thu, 2008-02-14 at 00:25 -0500, Benjamin R. Haskell wrote:

And, as Charles pointed out, in Dovecot 1.1 there's a plugin to do this. Does the expunge plugin work without the user logging in, though?

Yes. That's exactly what it does. It keeps user/mailbox <-> oldest_msg mapping in a Berkeley DB. In a nightly (or whatever) run it then uses the database to open only those mailboxes that have messages to be expunged.

At 12:27 PM +0100 2/14/08, Edgar Fuß wrote:

Am 13.02.2008 um 14:56 schrieb Bill Cole:

...

Not on all filesystems. Note what HFS+ (MacOS) does:

~ $ ls -lc foo -rwxr-xr-x 1 wkc wkc 332 Jan 29 03:32 foo ~ $ mkdir foodir ~ $ mv foo foodir ~ $ ls -lc foodir/foo -rwxr-xr-x 1 wkc wkc 332 Jan 29 03:32 foodir/foo ~ $ date Wed Feb 13 08:39:24 EST 2008 It's what I'd expect. In fact, it's what UFS does. Renaming a file doesn't change it's inode.

That's not true for the Solaris 9 implementation of UFS:

$ uname -a SunOS sysadm05 5.9 Generic_118558-21 sun4u sparc SUNW,Ultra-80 $ df -k . Filesystem kbytes used avail capacity Mounted on /dev/md/dsk/d2 6050021 5050411 939110 85% / $ grep dsk/d2 /etc/mnttab /dev/md/dsk/d2 / ufs rw,intr,largefiles,logging,xattr,onerror=panic,suid,dev=1540002 1202637838 $ ls -lc jar_cache30536.tmp -rw-r--r-- 1 colew adm 64258 Jan 13 22:28 jar_cache30536.tmp $ mv $_ foo $ ls -lc foo
-rw-r--r-- 1 colew adm 64258 Feb 14 09:53 foo

-- Bill Cole
bill@scconsult.com

...

It's what I'd expect. In fact, it's what UFS does. Renaming a file
doesn't change it's inode.

That's not true for the Solaris 9 implementation of UFS: Sorry, I meant UFS on NetBSD. I didn't know Solaris would behave
differently.