[Dovecot] Doubts about dsync, mdbox, SIS

Jan-Frode Myklebust janfrode at tanso.net
Thu Feb 2 13:31:20 EET 2012


On Thu, Feb 02, 2012 at 12:23:01PM +0200, Timo Sirainen wrote:
> 
> Note that with SIS the attachments aren't compressed.

Yes, I know. 

> 
> > "/srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-057274283bb51f4f917e0000bf34f6ab" is
> > missing, but there are 205 other copies of this file named
> > /srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-* with
> > identical sha1sum.
> 
> All of them have a link count of 2, with the other link being in hashes/
> directory?

No, these has link count=207. I don't know what you mean by link being
in hashes directory.

# ls -l /srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-*|head
-rw------- 207 mailbackup mailbackup 149265 Jan  9 23:31 /srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-0069222e0c080f4f754a0000bf34f6ab
-rw------- 207 mailbackup mailbackup 149265 Jan  9 23:31 /srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-00ffb9312a370e4f6b610000bf34f6ab
-rw------- 207 mailbackup mailbackup 149265 Jan  9 23:31 /srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-0442c5163ad3114fb4780000bf34f6ab
-rw------- 207 mailbackup mailbackup 149265 Jan  9 23:31 /srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-04f288390052144f012d0000bf34f6ab
-rw------- 207 mailbackup mailbackup 149265 Jan  9 23:31 /srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-053b6c0f185a0d4fc4210000bf34f6ab
-rw------- 207 mailbackup mailbackup 149265 Jan  9 23:31 /srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-06c98213c3b30e4fac3c0000bf34f6ab
-rw------- 207 mailbackup mailbackup 149265 Jan  9 23:31 /srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-076573234fbd0b4fa8620000bf34f6ab

This is just one example, I can provide tons of other examples.. Hmm,
I see now that there are 206 files of that first example with the 207
links, and here's another other example with numlinks=7:

	# ls -l  /srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-*|wc -l
	206

and numlinks=4:

	# ls -al /srv/mailbackup/attachments/c3/1b/c31beb42ef78810f7fb81a7086144034fb0fd794*|wc -l
	3

is dovecot somehow creating numlinks+1 copies of every file it
hardlinks?? Would explain my diskusage :-)

	

> That is most likely related to your troubles. If the dsync runs crash,
> the result could leave extra files lying around etc..

If dsync backup is supposed to be a viable backup solution, I think it
should fail much better. If it see errors on the target side it should
clear the target and do a full sync. Manually cleaning up after it's
problems is too much work.

> 
> > Some samples:
> > 
> > 	Error: Mailboxes don't have unique GUIDs: 08b46439069d3d4db0490000e671bf84 is shared by INBOX and INBOX
> 
> This is a little bit strange. What is the doveconf -n output of the
> source server?

----------------------------------------------------------------------------
# 2.0.14: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.18-194.26.1.el5 x86_64 Red Hat Enterprise Linux Server
# release 5.5 (Tikanga) 
auth_cache_size = 100 M
auth_verbose = yes
auth_verbose_passwords = sha1
disable_plaintext_auth = no
login_trusted_networks = 192.168.0.0/16
mail_gid = 3000
mail_home = /srv/mailstore/%256RHu/%d/%n
mail_location = maildir:~/:INDEX=/indexes/%1u/%1.1u/%u
mail_max_userip_connections = 20
mail_plugins = quota zlib
mail_uid = 3000
maildir_stat_dirs = yes
maildir_very_dirty_syncs = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date
mmap_disable = yes
namespace {
  inbox = yes
  location = 
  prefix = INBOX.
  separator = .
  type = private
}
passdb {
  args = /etc/dovecot/dovecot-ldap.conf.ext
  driver = ldap
}
plugin {
  quota = dict:UserQuota::file:%h/dovecot-quota
  sieve = /sieve/%1u/%1.1u/%u/.dovecot.sieve
  sieve_dir = /sieve/%1u/%1.1u/%u
  sieve_max_script_size = 1M
  zlib_save = gz
  zlib_save_level = 6
}
postmaster_address = postmaster at example.net
protocols = imap pop3 lmtp sieve
service auth-worker {
  user = $default_internal_user
}
service auth {
  client_limit = 4521
  unix_listener auth-userdb {
    group = 
    mode = 0600
    user = atmail
  }
}
service imap-login {
  inet_listener imap {
    address = *
    port = 143
  }
  process_min_avail = 4
  service_count = 0
  vsz_limit = 1 G
}
service imap-postlogin {
  executable = script-login /usr/local/sbin/imap-postlogin.sh
}
service imap {
  executable = imap imap-postlogin
  process_limit = 2048
}
service lmtp {
  client_limit = 1
  inet_listener lmtp {
    address = *
    port = 24
  }
  process_limit = 25
}
service managesieve-login {
  inet_listener sieve {
    address = *
    port = 4190
  }
  service_count = 1
}
service pop3-login {
  inet_listener pop3 {
    address = *
    port = 110
  }
  process_min_avail = 4
  service_count = 0
  vsz_limit = 1 G
}
service pop3-postlogin {
  executable = script-login /usr/local/sbin/pop3-postlogin.sh
}
service pop3 {
  executable = pop3 pop3-postlogin
  process_limit = 2048
}
ssl = no
userdb {
  args = /etc/dovecot/dovecot-ldap.conf.ext
  driver = ldap
}
protocol lmtp {
  mail_plugins = quota zlib sieve
}
protocol imap {
  imap_client_workarounds = delay-newmail
  mail_plugins = quota zlib imap_quota
}
protocol pop3 {
  mail_plugins = quota zlib
  pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
  pop3_uidl_format = UID%u-%v
}
protocol sieve {
  managesieve_logout_format = bytes=%i/%o
}
----------------------------------------------------------------------------

> 
> > 	Error: Failed to sync mailbox INBOX.ferie 2006.: Invalid mailbox name
> 
> Is this a namespace prefix? It shouldn't be trying to sync a mailbox
> named this (there's an extra "." suffix).

I believe it's a folder named "INBOX.ferie 2006.", with the user using
the namespace separator in the folder name..  I believe dovecot allows
this, so it should also handle backing it up.

> 
> > 	Error: Failed to sync mailbox INBOX.INBOX.Gerda: Mailbox doesn't exist: INBOX/Gerda
> 
> I guess some kind of mismatch related to namespace configuration.

They both have same namespace config I think, but Maildir vs. mdbox
maybe changes something..

> 
> It would help if I could reproduce the errors that you're seeing. Can
> you easily reproduce them with some accounts? If so, if you can give
> enough details for me to reproduce the problems I can fix them. (Except
> for the "file not found" issues, since that problems occurred earlier
> already. I should probably somehow make Dovecot fix those missing files
> though..)

I'll look trough the logs to see if there are any errors that keep
repeating for the same accounts.


  -jf



More information about the dovecot mailing list