dsync unstable? (other strange detail)
Hi,
I'm trying to migrate from Cyrus (remote side) to Dovecot 2.2.24 (local). On the local side the destinations folders, and indexes are empty.
The command I'm using is
doveadm
-o mail_plugins=
-o imapc_master_user=<cyrus-user>
-o imapc_password=<cyrus-password>
-o imapc_host=<cyrus-ip>
-o imapc_ssl_verify=no
-o imapc_ssl=imaps
-o imapc_port=993
backup -f -u "heiko" -R imapc:
|| {
rc=$?
echo "EXIT: $rc" >&2
exit $rc
}
On successive runs of the above command I get:
dsync(heiko): Warning: Deleting mailbox 'Serververwaltung.Mailinglisten Anforderung': UID=16 GUID= is missing locally
EXIT: 75
dsync(heiko): Warning: Deleting mailbox 'Serververwaltung.Mailman': UID=2 GUID= is missing locally
EXIT: 75
dsync(heiko): Warning: Deleting mailbox 'Serververwaltung.Servermeldungen': UID=514 GUID= is missing locally
EXIT: 75
dsync(heiko): Warning: Deleting mailbox 'Serververwaltung.Servermeldungen.AIDE': UID=188292 GUID= is missing locally
EXIT: 75
dsync(heiko): Warning: Deleting mailbox 'Serververwaltung.Servermeldungen.AIDE.AIDE - 2012': UID=9343 GUID= is missing locally
EXIT: 75Any idea where to look next? Is 'doveadm backup' the wrong tool for such migration? (I'd say with about 2.2.9 I had similar problems, but at least it didn't stop at every subfolder.)
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann-- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
On 27 Jun 2016, at 08:28, Heiko Schlittermann <hs@schlittermann.de> wrote:
Hi,
I'm trying to migrate from Cyrus (remote side) to Dovecot 2.2.24 (local). On the local side the destinations folders, and indexes are empty.
The command I'm using is
doveadm
-o mail_plugins=
-o imapc_master_user=<cyrus-user>
-o imapc_password=<cyrus-password>
-o imapc_host=<cyrus-ip>
-o imapc_ssl_verify=no
-o imapc_ssl=imaps
-o imapc_port=993
backup -f -u "heiko" -R imapc:
|| { rc=$? echo "EXIT: $rc" >&2 exit $rc }On successive runs of the above command I get:
dsync(heiko): Warning: Deleting mailbox 'Serververwaltung.Mailinglisten Anforderung': UID=16 GUID= is missing locally
This means that on Dovecot side there are messages after UID=16, but either:
a) UID=16 was expunged from Dovecot side or
b) UID=16 suddenly appeared on Cyrus side even though it wasn't there earlier. This isn't allowed by IMAP standard.
Dovecot can't insert UIDs, so it'll delete the folder and re-sync everything on the next run.
Any idea where to look next? Is 'doveadm backup' the wrong tool for such migration? (I'd say with about 2.2.9 I had similar problems, but at least it didn't stop at every subfolder.)
If you allow local access already that can do modification, use doveadm sync -1 after that.
Hi, Timo Sirainen <tss@iki.fi> (Di 28 Jun 2016 23:30:38 CEST):
On successive runs of the above command I get:
dsync(heiko): Warning: Deleting mailbox 'Serververwaltung.Mailinglisten Anforderung': UID=16 GUID= is missing locally
This means that on Dovecot side there are messages after UID=16, but either: a) UID=16 was expunged from Dovecot side or
On the dovecot side nobody is accessing the mail system.
b) UID=16 suddenly appeared on Cyrus side even though it wasn't there earlier. This isn't allowed by IMAP standard.
Hm, this seems to be a possible reason. So, successive numbers?
It seems to happen mostly on huuge mailboxes.
Heiko
On 29 Jun 2016, at 00:53, Heiko Schlittermann <hs@schlittermann.de> wrote:
Hi, Timo Sirainen <tss@iki.fi> (Di 28 Jun 2016 23:30:38 CEST):
On successive runs of the above command I get:
dsync(heiko): Warning: Deleting mailbox 'Serververwaltung.Mailinglisten Anforderung': UID=16 GUID= is missing locally
This means that on Dovecot side there are messages after UID=16, but either: a) UID=16 was expunged from Dovecot side or
On the dovecot side nobody is accessing the mail system.
b) UID=16 suddenly appeared on Cyrus side even though it wasn't there earlier. This isn't allowed by IMAP standard.
Hm, this seems to be a possible reason. So, successive numbers?
It seems to happen mostly on huuge mailboxes.
It's still strange if Cyrus is doing that. It's generally a pretty well behaving IMAP server. What version is it?
Timo Sirainen <tss@iki.fi> (Mi 29 Jun 2016 00:00:11 CEST): …
b) UID=16 suddenly appeared on Cyrus side even though it wasn't there earlier. This isn't allowed by IMAP standard. It's still strange if Cyrus is doing that. It's generally a pretty well behaving IMAP server. What version is it?
- OK srvlx Cyrus IMAP4 v2.2.12 server ready
Maybe, did you read my previous post with a similar subject? There I had an empty local destination and some nasty effects too.
In case it helps:
mail_location = maildir:~:INBOX=/volumes/dovecot/inbox/%2.256Nn/%n:INDEX=/volumes/dovecot/cache/%2.256Nn/%nwhich leads to
/volumes/dovecot/{cache,home,inbox}/<hash>/<user>is used for the maildir storage. As I'm writing this, I'm not sure, if I really purged the /var/vmail/cache/ hierarchy. But home/ and inbox/ where clean as a baby.
The storage is imported via NFS. But the other backends (we're using a director/backend setup) are switched off, to really be sure the we don't have concurrent access.
-- Heiko
On 29 Jun 2016, at 01:13, Heiko Schlittermann <hs@schlittermann.de> wrote:
Timo Sirainen <tss@iki.fi> (Mi 29 Jun 2016 00:00:11 CEST): …
b) UID=16 suddenly appeared on Cyrus side even though it wasn't there earlier. This isn't allowed by IMAP standard. It's still strange if Cyrus is doing that. It's generally a pretty well behaving IMAP server. What version is it?
- OK srvlx Cyrus IMAP4 v2.2.12 server ready
Maybe, did you read my previous post with a similar subject? There I had an empty local destination and some nasty effects too.
There was another mail with "highest than remote's UIDs" error. Do you mean that one? I don't see others. That's also kind of strange. Dovecot had seen mails that suddenly no longer existed on Cyrus side. It's as if you're syncing to two different Cyrus servers that are somewhat out of sync themselves. Is that possible?
In case it helps:
mail_location = maildir:~:INBOX=/volumes/dovecot/inbox/%2.256Nn/%n:INDEX=/volumes/dovecot/cache/%2.256Nn/%n
which leads to
/volumes/dovecot/{cache,home,inbox}/<hash>/<user>
is used for the maildir storage. As I'm writing this, I'm not sure, if I really purged the /var/vmail/cache/ hierarchy. But home/ and inbox/ where clean as a baby.
The storage is imported via NFS. But the other backends (we're using a director/backend setup) are switched off, to really be sure the we don't have concurrent access.
An out-of-date index with Maildir shouldn't really matter since it should get automatically updated.
Timo Sirainen <tss@iki.fi> (Mi 29 Jun 2016 00:20:05 CEST): …
Maybe, did you read my previous post with a similar subject? There I had an empty local destination and some nasty effects too.
There was another mail with "highest than remote's UIDs" error. Do you mean that one? I don't see others. That's also kind of strange. Dovecot had seen mails that suddenly no longer existed on Cyrus side. It's as if you're syncing to two different Cyrus servers that are somewhat out of sync themselves. Is that possible?
Yes, dsync(heiko): Warning: Deleting mailbox 'Trash': UID=18290 already exists locally for a different mail: highest than remote's UIDs (remote UIDNEXT=19588) This happend during a sync to an empty local destination
The source (cyrus) is an active/passive cluster, the IP I'm connecting to should be on the same machine for the time the syncronisation runs. But I'll check this.
Thank you for responding… It give me the hope that it *should* work. (Meanwhile I'm writing 'yet-another-imap2imap' sync tool, but using dsync would be the better choice, definitivly)
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann-- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
participants (2)
-
Heiko Schlittermann
-
Timo Sirainen