[Dovecot] Doubts about dsync, mdbox, SIS

[Timo Sirainen]

On Wed, 2012-02-01 at 13:29 +0100, Jan-Frode Myklebust wrote:

> I'm surprised that the destination server is so large, was expecting zlib and
> mdbox and SIS would compress it down to much less than what we're seeing 
> (12TB -> 5TB):

Note that with SIS the attachments aren't compressed.

> Lots and lots of the attachement storage is duplicated into identical files,
> instead of hard linked.

Something's wrong then.

> When running "doveadm purge -u $user", we're seeing lots of 
> 
> 	Error: unlink(/srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-057274283bb51f4f917e0000bf34f6ab) failed: No such file or directory

Something's wrong.

> "/srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-057274283bb51f4f917e0000bf34f6ab" is
> missing, but there are 205 other copies of this file named
> /srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-* with
> identical sha1sum.

All of them have a link count of 2, with the other link being in hashes/
directory?

> Also on the source side, during dsync, we see too many problems. 

That is most likely related to your troubles. If the dsync runs crash,
the result could leave extra files lying around etc..

> Some samples:
> 
> 	Error: Mailboxes don't have unique GUIDs: 08b46439069d3d4db0490000e671bf84 is shared by INBOX and INBOX

This is a little bit strange. What is the doveconf -n output of the
source server?

> 	Error: Failed to sync mailbox INBOX.ferie 2006.: Invalid mailbox name

Is this a namespace prefix? It shouldn't be trying to sync a mailbox
named this (there's an extra "." suffix).

> 	Error: read() from proxy client failed: EOF

I guess the remote dsync crashes or otherwise aborted.

> 	Error: Failed to sync mailbox INBOX.INBOX.Gerda: Mailbox doesn't exist: INBOX/Gerda

I guess some kind of mismatch related to namespace configuration.

> 	Error: read() failed: Broken pipe
> 	Panic: file dsync-worker-local.c: line 1678 (local_worker_save_msg_continue): assertion failed: (ret == -1)

Probably can't handle properly when remote dsync dies. Of course it
still shouldn't crash. There seems to be some bugs left when dsyncing to
a remote host (instead of locally).

It would help if I could reproduce the errors that you're seeing. Can
you easily reproduce them with some accounts? If so, if you can give
enough details for me to reproduce the problems I can fix them. (Except
for the "file not found" issues, since that problems occurred earlier
already. I should probably somehow make Dovecot fix those missing files
though..)