[Dovecot] reproducible dsync backup panic (dovecot 2.2.7)
Hi all,
I'm receiving a reproducible dsync backup panic for some of my users. The error message is as follows:
dsync-remote(user@<removed>): Panic: file dsync-brain-mailbox-tree-sync.c: line 111 (dsync_brain_mailbox_tree_sync_change): assertion failed: (brain->no_backup_overwrite) dsync-remote(user@<removed>): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x69b70) [0x7f922dad9b70] -> /usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7f922dad9c4a] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f922da94725] -> /usr/bin/dsync(dsync_brain_mailbox_tree_sync_change+0x382) [0x42f912] -> /usr/bin/dsync(dsync_brain_recv_mailbox_tree_deletes+0x10b) [0x42f4db] -> /usr/bin/dsync(dsync_brain_run+0x305) [0x42c6f5] -> /usr/bin/dsync() [0x42cbe8] -> /usr/bin/dsync() [0x43e040] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7f922daea066] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f922daeaf07] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f922dae9b78] -> /usr/bin/dsync() [0x4297f9] -> /usr/bin/dsync() [0x4133cd] -> /usr/bin/dsync(doveadm_mail_try_run+0x251) [0x414091] -> /usr/bin/dsync(main+0x3c8) [0x412f78] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7f922d704ead] -> /usr/bin/dsync() [0x413191] dsync-local(user@<removed>): Error: read(<remote-mail-server>) failed: EOF dsync-local(user@<removed>): Error: Remote command returned error 134
The command I'm using to back up individual user's mail is something like:
sudo -u vmail dsync -v -u user@server.com -R backup ssh -i sshid-file login@server.com sudo -u vmail dsync -v -u user@server.com
I use the above command at the backup server. The error started to appear a few weeks after we started using the new mail system. First it appeared for a single user, now it appears for some more users as well. It is deterministic and reproducible.
Any ideas?
I included the relevant part of my config below.
Best regards,
Peter
# 2.2.7 (5c876ed0c3e0): /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.2 auth_master_user_separator = * auth_mechanisms = plain login hostname = <removed> listen = * login_greeting = $hostname IMAP server mail_attachment_dir = /mail/mailboxes/%d/_attachments_ mail_attachment_fs = sis-queue $mail_attachment_dir/queue:posix mail_attachment_min_size = 16 k mail_location = mdbox:~ mail_plugins = " zlib acl" mdbox_rotate_interval = 5 days mdbox_rotate_size = 5 M namespace { inbox = yes location = prefix = separator = / type = private } namespace { list = children location = mdbox:/mail/mailboxes/%%d/%%n:INDEX=/mail/mailboxes/%%d/%%n prefix = Accounts/%%n/ separator = / subscriptions = no type = shared } passdb { args = /mail/config/dovecot/passwd driver = passwd-file } passdb { args = /mail/config/dovecot/passwd-master driver = passwd-file master = yes pass = yes } plugin { acl = vfile acl_defaults_from_inbox = yes acl_shared_dict = file:/mail/mailboxes/%d/shared-mailboxes.db zlib_save = gz zlib_save_level = 6 } postmaster_address = postmaster@<removed> protocols = imap service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } service imap-login { inet_listener imap { port = 0 } inet_listener imaps { port = 993 ssl = yes } } syslog_facility = local6 userdb { args = uid=vmail gid=vmail master_user=%u home=/mail/mailboxes/%d/%n driver = static } protocol imap { mail_plugins = " zlib acl imap_acl" }
On 17.12.2013, at 11.54, Peter Benko pbopbo@freemail.hu wrote:
I'm receiving a reproducible dsync backup panic for some of my users. The error message is as follows:
dsync-remote(user@<removed>): Panic: file dsync-brain-mailbox-tree-sync.c: line 111 (dsync_brain_mailbox_tree_sync_change): assertion failed: (brain->no_backup_overwrite)
Can you get it to dump core and use
gdb /usr/bin/doveadm core bt full fr 5 p *change
The command I'm using to back up individual user's mail is something like:
sudo -u vmail dsync -v -u user@server.com -R backup ssh -i sshid-file login@server.com sudo -u vmail dsync -v -u user@server.com
Probably an easier way would be if you could copy the local and remote mdboxes to e.g. /tmp/mdbox-local and /tmp/mdbox-remote. Then you should be able to do:
gdb --args doveadm -o mail=mdbox:/tmp/mdbox-local backup -R mdbox:/tmp/mdbox-remote run bt full fr 5 p *change
Oh, and in any case this would be helpful in figuring out the problem. Run these on both local and remote servers and send the output to me:
doveadm mailbox status -u user@domain 'guid uidvalidity uidnext' '*' doveadm dump /mail/mailboxes/domain/user/dovecot.mailbox.log
participants (2)
-
Peter Benko
-
Timo Sirainen