[Dovecot] Sync errors trying to sync old mbox inboxes to dovecot

[Michael Smith]

Timo,

I tried importing the anonymized file, and there were no errors.  So, it's something with the actual data.  UTF-8 issues?


While I can't send you a raw customer's mailbox, one of the failures is a 'role' account mailbox, with no critical/useful data in it.

Here is a link (expires in 30 days) to download the mailbox, it's 107M.
https://truck.it/p/HNlzl9hWbO

The error I'm getting is 
dsync(webmail): Error: Sync failed for mbox file /var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally needed 10 bytes, now needs 12 bytes

The seq and uid are different, depending on the mailbox being migrated.  Some of the problem mailboxes only report a single failure, and some report multiple failures (where the seq/uid are different for each failure).  For a given mailbox, the seq and uid reported seem to always be consistent.

I create a directory called migrate, copy the file in there and name it inbox

The command I'm using is
Dsync -u webmail mirror mbox:migrate/

Here's my 2.2.rc3 doveconf -n output

# 2.2.rc3: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-279.19.1.el6.x86_64 x86_64 CentOS release 6.3 (Final)
auth_debug = yes
auth_debug_passwords = yes
auth_default_realm = 
auth_mechanisms = plain login
auth_verbose = yes
auth_verbose_passwords = plain
debug_log_path = /var/log/dovecot-debug.log
disable_plaintext_auth = no
dotlock_use_excl = no
lda_mailbox_autocreate = yes
lock_method = dotlock
log_path = /var/log/dovecot.log
mail_access_groups = mail
mail_debug = yes
mail_fsync = always
mail_location = mdbox:~/mail:INDEX=~/index
mail_plugins = quota zlib
mail_privileged_group = mail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date
mbox_read_locks = dotlock
mbox_write_locks = dotlock
mdbox_rotate_interval = 1 weeks
mdbox_rotate_size = 20 M
mmap_disable = yes
passdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  autocreate = Trash
  autocreate2 = Junk
  autocreate3 = Drafts
  autocreate4 = Sent
  autocreate5 = INBOX
  autosubscribe = Trash
  autosubscribe2 = Junk
  autosubscribe3 = Drafts
  autosubscribe4 = Sent
  autosubscribe5 = INBOX
  mail_max_userip_connections = 10
  managesieve_max_line_length = 65536
  quota = dict:User quota::file:%h/dovecot.quota
  quota2_rule = Trash:storage=+10%%
  quota3_rule = Junk:storage=+20%%
  quota_rule = *:storage=100M:messages=100000
  recipient_delimiter = +
  sieve_before = /var/spool/mail/global.sieve/
  zlib_save = gz
  zlib_save_level = 9
}
pop3_reuse_xuidl = yes
protocols = imap pop3 sieve lmtp
service auth {
  inet_listener auth {
    port = 113
  }
  unix_listener auth-userdb {
    user = nobody
  }
}
service lmtp {
  inet_listener lmtp {
    port = 24
  }
  process_min_avail = 10
  service_count = 1
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
  process_min_avail = 0
  service_count = 1
  vsz_limit = 64 M
}
ssl = no
ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
ssl_key = </etc/pki/dovecot/private/dovecot.pem
userdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
protocol lda {
  mail_plugins = quota zlib sieve
}
protocol imap {
  mail_plugins = quota zlib imap_quota autocreate
}
protocol sieve {
  mail_max_userip_connections = 10
  mail_plugins = quota zlib autocreate
  managesieve_max_line_length = 65536
  managesieve_notify_capability = mailto
  managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date
}
protocol lmtp {
  mail_plugins = quota zlib sieve
}


--
Michael Smith

-----Original Message-----
From: Timo Sirainen [mailto:tss at iki.fi] 
Sent: Sunday, April 07, 2013 4:36 PM
To: Michael Smith
Cc: dovecot at dovecot.org
Subject: Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot

On 6.4.2013, at 0.10, Michael Smith <msmith at giganews.com> wrote:

> I am trying to 'dsync mirror' old Qpopper/mbox inboxes to our new Dovecot/mdbox servers.  For most of the several thousand accounts, it works without issue.  But there are a handful of accounts that are generating the following errors.
> 
> dsync(joe): Error: Sync failed for mbox file /var/opt/migrate/users/joe/inbox: seq=981 uid=981 uid_broken=0 originally needed 9 bytes, now needs 11 bytes
> 
> dsync(john): Error: Sync failed for mbox file /var/opt/migrate/users/john/inbox: seq=115028 uid=115028 uid_broken=0 originally needed 34 bytes, now needs 35 bytes

These are some generic mbox bugs. I think the error goes away if you just repeat the run? 

> dsync(john): Error: Sync failed for mbox file /var/opt/migrate/users/john/inbox: seq=114779 uid=114779 uid_broken=0 originally needed 34 bytes, now needs 35 bytes
> dsync(john): Error: Sync failed for mbox file /var/opt/migrate/users/john/inbox: seq=114778 uid=114778 uid_broken=0 originally needed 34 bytes, now needs 35 bytes

.. or are these all separate dsync runs giving different errors?

Anyway, if you can send me an mbox file where I can reproduce this, I could fix it. You can use http://dovecot.org/tools/mbox-anonymize.pl script to replace all the actual data in the mbox file with 'x' chars.