Timo,

I tried importing the anonymized file, and there were no errors. So, it's something with the actual data. UTF-8 issues?

While I can't send you a raw customer's mailbox, one of the failures is a 'role' account mailbox, with no critical/useful data in it.

Here is a link (expires in 30 days) to download the mailbox, it's 107M. https://truck.it/p/HNlzl9hWbO

The error I'm getting is dsync(webmail): Error: Sync failed for mbox file /var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally needed 10 bytes, now needs 12 bytes

The seq and uid are different, depending on the mailbox being migrated. Some of the problem mailboxes only report a single failure, and some report multiple failures (where the seq/uid are different for each failure). For a given mailbox, the seq and uid reported seem to always be consistent.

I create a directory called migrate, copy the file in there and name it inbox

The command I'm using is Dsync -u webmail mirror mbox:migrate/

Here's my 2.2.rc3 doveconf -n output

# 2.2.rc3: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-279.19.1.el6.x86_64 x86_64 CentOS release 6.3 (Final) auth_debug = yes auth_debug_passwords = yes auth_default_realm = auth_mechanisms = plain login auth_verbose = yes auth_verbose_passwords = plain debug_log_path = /var/log/dovecot-debug.log disable_plaintext_auth = no dotlock_use_excl = no lda_mailbox_autocreate = yes lock_method = dotlock log_path = /var/log/dovecot.log mail_access_groups = mail mail_debug = yes mail_fsync = always mail_location = mdbox:~/mail:INDEX=~/index mail_plugins = quota zlib mail_privileged_group = mail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date mbox_read_locks = dotlock mbox_write_locks = dotlock mdbox_rotate_interval = 1 weeks mdbox_rotate_size = 20 M mmap_disable = yes passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { autocreate = Trash autocreate2 = Junk autocreate3 = Drafts autocreate4 = Sent autocreate5 = INBOX autosubscribe = Trash autosubscribe2 = Junk autosubscribe3 = Drafts autosubscribe4 = Sent autosubscribe5 = INBOX mail_max_userip_connections = 10 managesieve_max_line_length = 65536 quota = dict:User quota::file:%h/dovecot.quota quota2_rule = Trash:storage=+10%% quota3_rule = Junk:storage=+20%% quota_rule = *:storage=100M:messages=100000 recipient_delimiter = + sieve_before = /var/spool/mail/global.sieve/ zlib_save = gz zlib_save_level = 9 } pop3_reuse_xuidl = yes protocols = imap pop3 sieve lmtp service auth { inet_listener auth { port = 113 } unix_listener auth-userdb { user = nobody } } service lmtp { inet_listener lmtp { port = 24 } process_min_avail = 10 service_count = 1 } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 0 service_count = 1 vsz_limit = 64 M } ssl = no ssl_cert =

-- Michael Smith

-----Original Message----- From: Timo Sirainen [mailto:tss@iki.fi] Sent: Sunday, April 07, 2013 4:36 PM To: Michael Smith Cc: dovecot@dovecot.org Subject: Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot

On 6.4.2013, at 0.10, Michael Smith msmith@giganews.com wrote:

I am trying to 'dsync mirror' old Qpopper/mbox inboxes to our new Dovecot/mdbox servers. For most of the several thousand accounts, it works without issue. But there are a handful of accounts that are generating the following errors.

dsync(joe): Error: Sync failed for mbox file /var/opt/migrate/users/joe/inbox: seq=981 uid=981 uid_broken=0 originally needed 9 bytes, now needs 11 bytes

dsync(john): Error: Sync failed for mbox file /var/opt/migrate/users/john/inbox: seq=115028 uid=115028 uid_broken=0 originally needed 34 bytes, now needs 35 bytes

These are some generic mbox bugs. I think the error goes away if you just repeat the run?

dsync(john): Error: Sync failed for mbox file /var/opt/migrate/users/john/inbox: seq=114779 uid=114779 uid_broken=0 originally needed 34 bytes, now needs 35 bytes dsync(john): Error: Sync failed for mbox file /var/opt/migrate/users/john/inbox: seq=114778 uid=114778 uid_broken=0 originally needed 34 bytes, now needs 35 bytes

.. or are these all separate dsync runs giving different errors?

Anyway, if you can send me an mbox file where I can reproduce this, I could fix it. You can use http://dovecot.org/tools/mbox-anonymize.pl script to replace all the actual data in the mbox file with 'x' chars.

It shouldn't. Part of the migration is to remove any existing dovecot mailbox for the user. It does this by deleting the entire user's directory.

I just downloaded the file and checked. Oops, looks like I grabbed the post dsync mirror file.

I've uploaded the correct file (pre dsync mirror). Downloaded it and confirmed I'm getting the following error on dsync mirror.

dsync(webmail): Error: Sync failed for mbox file /var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally needed 10 bytes, now needs 12 bytes

-- Michael Smith

-----Original Message----- From: Timo Sirainen [mailto:tss@iki.fi] Sent: Tuesday, April 09, 2013 4:05 PM To: Michael Smith Cc: 'dovecot@dovecot.org' Subject: Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot

On 9.4.2013, at 1.41, Michael Smith msmith@giganews.com wrote:

I tried importing the anonymized file, and there were no errors. So, it's something with the actual data. UTF-8 issues?

While I can't send you a raw customer's mailbox, one of the failures is a 'role' account mailbox, with no critical/useful data in it.

Here is a link (expires in 30 days) to download the mailbox, it's 107M. https://truck.it/p/HNlzl9hWbO

The error I'm getting is dsync(webmail): Error: Sync failed for mbox file /var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally needed 10 bytes, now needs 12 bytes

I can't reproduce with that file, dsync succeeds without errors. Maybe the problem requires something specific in the index files also?

Yes, I can reproduce it now. I'll try to get it fixed on Monday.

On 12.4.2013, at 21.13, Michael Smith msmith@giganews.com wrote:

Timo,

Any luck reproducing the sync error with the updated/correct mbox file?

-- Michael Smith

-----Original Message----- From: dovecot-bounces@dovecot.org [mailto:dovecot-bounces@dovecot.org] On Behalf Of Michael Smith Sent: Wednesday, April 10, 2013 3:52 PM To: 'Timo Sirainen' Cc: 'dovecot@dovecot.org' Subject: Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot

It shouldn't. Part of the migration is to remove any existing dovecot mailbox for the user. It does this by deleting the entire user's directory.

I just downloaded the file and checked. Oops, looks like I grabbed the post dsync mirror file.

I've uploaded the correct file (pre dsync mirror). Downloaded it and confirmed I'm getting the following error on dsync mirror.

dsync(webmail): Error: Sync failed for mbox file /var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally needed 10 bytes, now needs 12 bytes

-- Michael Smith

-----Original Message----- From: Timo Sirainen [mailto:tss@iki.fi] Sent: Tuesday, April 09, 2013 4:05 PM To: Michael Smith Cc: 'dovecot@dovecot.org' Subject: Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot

On 9.4.2013, at 1.41, Michael Smith msmith@giganews.com wrote:

...

I tried importing the anonymized file, and there were no errors. So, it's something with the actual data. UTF-8 issues?

While I can't send you a raw customer's mailbox, one of the failures is a 'role' account mailbox, with no critical/useful data in it.

Here is a link (expires in 30 days) to download the mailbox, it's 107M. https://truck.it/p/HNlzl9hWbO

The error I'm getting is dsync(webmail): Error: Sync failed for mbox file /var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally needed 10 bytes, now needs 12 bytes

I can't reproduce with that file, dsync succeeds without errors. Maybe the problem requires something specific in the index files also?

On Tue, 2013-04-16 at 16:01 +0000, Michael Smith wrote:

Just checking if you've had a chance to identify the problem. Yesterday I saw a patch for multipart messages. Was this related, or a separate issue?

Here: http://hg.dovecot.org/dovecot-2.2/rev/c473f8d2540e