Helping doveadm sync detect moves between Maildir mailboxes? (reproducible example of bad behaviour)
Hi,
Moving mail files between Maildir mailboxes often confuses doveadm sync: I end up with copies in both the original and new maildir. Exact details and script to reproduce follow.
I can work around this by running doveadm move, but it does not fit well with my workflow, which is based on paths to mail files (using mblaze as my mail client).
Two questions:
Is there anything I can do to help dovecot notice the moves, other than using doveadm move instead of just mv? I have tried liberal use of force-resync but the problem is still there.
Is this a bug?
Details:
OS: OpenBSD current
$ dovecot --version 2.3.21.1 (d492236fa0)
config: My repro script below uses a one-line config, only setting mail_location.
(An OpenBSD 7.5 server running Dovecot 2.3.21 is involved in my real setup, but I was able to reproduce it locally without the server involved.)
To reproduce: (shell script follows)
- Make two maildirs "a" and "b", each with a subfolder ".Archive".
- Manually write a mail file to a/cur.
- doveadm sync
- force-resync everything
- manually move the file to a/.Archive/cur
- force-resync everything again
- doveadm sync
- Now the mail file appears four times: in a, a/.Archive, b, and b/.Archive.
Script to do the above follows my signature.
-- James
#!/bin/sh set -e
# Run in a new directory with two maildirs named a and b and a simple config. mkdir bug; cd bug mkdir -p {a,b}/{,.Archive/}{cur,new,tmp} cat > dovecot.conf <<EOF mail_location=maildir:$PWD/a EOF
# Deliver an email and sync. EXAMPLE_ID=12345.R67890.example.com cat > a/cur/$EXAMPLE_ID:2, <<EOF some email EOF doveadm -c dovecot.conf sync maildir:b
# Move the email and sync again. Run force-resync between steps in case it helps. doveadm -c dovecot.conf force-resync INBOX doveadm -c dovecot.conf force-resync Archive doveadm -c dovecot.conf -o mail_location=maildir:$PWD/b force-resync INBOX doveadm -c dovecot.conf -o mail_location=maildir:$PWD/b force-resync Archive mv a/cur/$EXAMPLE_ID* a/.Archive/cur doveadm -c dovecot.conf force-resync INBOX doveadm -c dovecot.conf force-resync Archive doveadm -c dovecot.conf -o mail_location=maildir:$PWD/b force-resync INBOX doveadm -c dovecot.conf -o mail_location=maildir:$PWD/b force-resync Archive doveadm -c dovecot.conf sync maildir:b
echo in a: doveadm -c dovecot.conf -fflow fetch 'mailbox guid' all echo echo in b: doveadm -c dovecot.conf -fflow -o mail_location=maildir:$PWD/b fetch 'mailbox guid' all
"James" == James Cook via dovecot <dovecot@dovecot.org> writes:
Moving mail files between Maildir mailboxes often confuses doveadm sync: I end up with copies in both the original and new maildir. Exact details and script to reproduce follow.
Are you accessing your Maildir from both a mail client, as well as using IMAP/POP through doveadm? I would say that this is your problem. You should (probably) be only using one method or another.
I can work around this by running doveadm move, but it does not fit well with my workflow, which is based on paths to mail files (using mblaze as my mail client).
Yeah, so mblaze is a bunch of utilities for interacting with Maildir files. So since you're screwing around with them behind dovecot's back, of course you're going to see problems. So don't do that!
Two questions:
- Is there anything I can do to help dovecot notice the moves, other than using doveadm move instead of just mv? I have tried liberal use of force-resync but the problem is still there.
- Is this a bug?
No, you're just using it wrong.
Details:
OS: OpenBSD current
$ dovecot --version 2.3.21.1 (d492236fa0)
config: My repro script below uses a one-line config, only setting mail_location.
(An OpenBSD 7.5 server running Dovecot 2.3.21 is involved in my real setup, but I was able to reproduce it locally without the server involved.)
To reproduce: (shell script follows)
- Make two maildirs "a" and "b", each with a subfolder ".Archive".
- Manually write a mail file to a/cur.
- doveadm sync
- force-resync everything
- manually move the file to a/.Archive/cur
- force-resync everything again
- doveadm sync
- Now the mail file appears four times: in a, a/.Archive, b, and b/.Archive.
Script to do the above follows my signature.
-- James
#!/bin/sh set -e
# Run in a new directory with two maildirs named a and b and a simple config. mkdir bug; cd bug mkdir -p {a,b}/{,.Archive/}{cur,new,tmp} cat > dovecot.conf <<EOF mail_location=maildir:$PWD/a EOF
# Deliver an email and sync. EXAMPLE_ID=12345.R67890.example.com cat > a/cur/$EXAMPLE_ID:2, <<EOF some email EOF doveadm -c dovecot.conf sync maildir:b
# Move the email and sync again. Run force-resync between steps in case it helps. doveadm -c dovecot.conf force-resync INBOX doveadm -c dovecot.conf force-resync Archive doveadm -c dovecot.conf -o mail_location=maildir:$PWD/b force-resync INBOX doveadm -c dovecot.conf -o mail_location=maildir:$PWD/b force-resync Archive mv a/cur/$EXAMPLE_ID* a/.Archive/cur doveadm -c dovecot.conf force-resync INBOX doveadm -c dovecot.conf force-resync Archive doveadm -c dovecot.conf -o mail_location=maildir:$PWD/b force-resync INBOX doveadm -c dovecot.conf -o mail_location=maildir:$PWD/b force-resync Archive doveadm -c dovecot.conf sync maildir:b
echo in a: doveadm -c dovecot.conf -fflow fetch 'mailbox guid' all echo echo in b: doveadm -c dovecot.conf -fflow -o mail_location=maildir:$PWD/b fetch 'mailbox guid' all
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
"John Stoffel" <john@stoffel.org> wrote:
"James" == James Cook via dovecot <dovecot@dovecot.org> writes:
Moving mail files between Maildir mailboxes often confuses doveadm sync: I end up with copies in both the original and new maildir. Exact details and script to reproduce follow.
Are you accessing your Maildir from both a mail client, as well as using IMAP/POP through doveadm? I would say that this is your problem. You should (probably) be only using one method or another.
I can work around this by running doveadm move, but it does not fit well with my workflow, which is based on paths to mail files (using mblaze as my mail client).
Yeah, so mblaze is a bunch of utilities for interacting with Maildir files. So since you're screwing around with them behind dovecot's back, of course you're going to see problems. So don't do that!
Two questions:
- Is there anything I can do to help dovecot notice the moves, other than using doveadm move instead of just mv? I have tried liberal use of force-resync but the problem is still there.
- Is this a bug?
No, you're just using it wrong.
Thanks for your reply, John.
My repro steps don't involve any IMAP, but do combine doveadm sync with manual mail operations. In practice I am using IMAP too.
I note that Dovecot's maildir documentation talks about other MUAs accessing the maildir; see [0]. My understanding (based on that and trying to understand the source in lib-storage/index/maildir) is that Dovecot's maildir backend goes to some effort to accommodate other MUAs.
But maybe I've been too optimistic about how far that support is intended to extend.
I had been hoping all that I'm missing is some way to get dovecot to notice the mail was expunged from a, the same way it would have been logged if I'd used doveadm move. I've been poking around the source trying to understand how moving and expunging work but haven't quite understood it.
[0] https://doc.dovecot.org/2.3/admin_manual/mailbox_formats/maildir/
-- James
"falsifian" == falsifian <falsifian@falsifian.org> writes:
"John Stoffel" <john@stoffel.org> wrote:
> "James" == James Cook via dovecot <dovecot@dovecot.org> writes:
Moving mail files between Maildir mailboxes often confuses doveadm sync: I end up with copies in both the original and new maildir. Exact details and script to reproduce follow.
Are you accessing your Maildir from both a mail client, as well as using IMAP/POP through doveadm? I would say that this is your problem. You should (probably) be only using one method or another.
I can work around this by running doveadm move, but it does not fit well with my workflow, which is based on paths to mail files (using mblaze as my mail client).
Yeah, so mblaze is a bunch of utilities for interacting with Maildir files. So since you're screwing around with them behind dovecot's back, of course you're going to see problems. So don't do that!
Two questions:
- Is there anything I can do to help dovecot notice the moves, other than using doveadm move instead of just mv? I have tried liberal use of force-resync but the problem is still there.
- Is this a bug?
No, you're just using it wrong.
Thanks for your reply, John.
You're welcome!
My repro steps don't involve any IMAP, but do combine doveadm sync with manual mail operations. In practice I am using IMAP too.
I've found that mixing IMAP and regular Maildir access at the same time lead to problems. But I admit I've never gotten replication setup either, which I do want to do at some point myself.
I note that Dovecot's maildir documentation talks about other MUAs accessing the maildir; see [0]. My understanding (based on that and trying to understand the source in lib-storage/index/maildir) is that Dovecot's maildir backend goes to some effort to accommodate other MUAs.
I think they key line is this:
Note that messages must not be modified once they’ve been
delivered. IMAP (and Dovecot) requires that messages are immutable. If
you wish to modify them in any way, create a new message instead and
expunge the old one.And from what you wrote before, I think you're moving messages between folders, which breaks Dovecot's knowledge of where emails are.
But maybe I've been too optimistic about how far that support is intended to extend.
I think it's a terrible idea to try and mix regular MUA access to mail handled by dovecot, it just leads to problems.
I had been hoping all that I'm missing is some way to get dovecot to notice the mail was expunged from a, the same way it would have been logged if I'd used doveadm move. I've been poking around the source trying to understand how moving and expunging work but haven't quite understood it.
[0] https://doc.dovecot.org/2.3/admin_manual/mailbox_formats/maildir/
-- James
I think they key line is this:
Note that messages must not be modified once they’ve been delivered. IMAP (and Dovecot) requires that messages are immutable. If you wish to modify them in any way, create a new message instead and expunge the old one.And from what you wrote before, I think you're moving messages between folders, which breaks Dovecot's knowledge of where emails are.
The problem still happens if I replace mv a/cur/$EXAMPLE_ID* a/.Archive/cur with cp a/cur/$EXAMPLE_ID_1:2, a/.Archive/cur/$EXAMPLE_ID_2:2, rm a/cur/$EXAMPLE_ID_1:2, in my repro script (and define $EXAMPLE_ID_1 and $EXAMPLE_ID_2 to be different).
Anyway, I thought that prohibition was about changing the content of a message file, not moving messages.
But maybe I've been too optimistic about how far that support is intended to extend.
I think it's a terrible idea to try and mix regular MUA access to mail handled by dovecot, it just leads to problems.
Only this problem so far. But maybe you are right. One reason I started this thread was to understand whether Dovecot's developers intend for this to work or not. If they do, I am happy to try to help fix whatever is happening.
It might be worth noting: sometimes doveadm sync does the right thing. In particular, I haven't had any trouble when moving emails that have been around for a while. (I tried making sure I don't move directly from new, but that's not enough.)
-- James
participants (2)
-
falsifian＠falsifian.org
-
John Stoffel