On Wed, 2012-02-01 at 13:29 +0100, Jan-Frode Myklebust wrote:
I'm surprised that the destination server is so large, was expecting zlib and mdbox and SIS would compress it down to much less than what we're seeing (12TB -> 5TB):
Note that with SIS the attachments aren't compressed.
Lots and lots of the attachement storage is duplicated into identical files, instead of hard linked.
Something's wrong then.
When running "doveadm purge -u $user", we're seeing lots of
Error: unlink(/srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-057274283bb51f4f917e0000bf34f6ab) failed: No such file or directory
Something's wrong.
"/srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-057274283bb51f4f917e0000bf34f6ab" is missing, but there are 205 other copies of this file named /srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-* with identical sha1sum.
All of them have a link count of 2, with the other link being in hashes/ directory?
Also on the source side, during dsync, we see too many problems.
That is most likely related to your troubles. If the dsync runs crash, the result could leave extra files lying around etc..
Some samples:
Error: Mailboxes don't have unique GUIDs: 08b46439069d3d4db0490000e671bf84 is shared by INBOX and INBOX
This is a little bit strange. What is the doveconf -n output of the source server?
Error: Failed to sync mailbox INBOX.ferie 2006.: Invalid mailbox name
Is this a namespace prefix? It shouldn't be trying to sync a mailbox named this (there's an extra "." suffix).
Error: read() from proxy client failed: EOF
I guess the remote dsync crashes or otherwise aborted.
Error: Failed to sync mailbox INBOX.INBOX.Gerda: Mailbox doesn't exist: INBOX/Gerda
I guess some kind of mismatch related to namespace configuration.
Error: read() failed: Broken pipe Panic: file dsync-worker-local.c: line 1678 (local_worker_save_msg_continue): assertion failed: (ret == -1)
Probably can't handle properly when remote dsync dies. Of course it still shouldn't crash. There seems to be some bugs left when dsyncing to a remote host (instead of locally).
It would help if I could reproduce the errors that you're seeing. Can you easily reproduce them with some accounts? If so, if you can give enough details for me to reproduce the problems I can fix them. (Except for the "file not found" issues, since that problems occurred earlier already. I should probably somehow make Dovecot fix those missing files though..)