Corruption & migration

[Daniel Miller]

Now I'm seeing a bunch of these:

02:28 bubba dovecot: dsync-local(user at mydomain.com): Error: 
bzlib.read(/var/mail/mydomain.com/user/mdbox/storage/m.20): corrupted 
data at 24123649
Mar 21 11:02:28 bubba dovecot: dsync-local(user at mydomain.com): Error: 
dsync(oldserver.host): 
read(zlib(/var/mail/mydomain.com/user/mdbox/storage/m.20)) failed: 
read(/var/mail/mydomain.com/user/mdbox/storage/m.20) failed: 
bzlib.read(/var/mail/mydomain.com/user/mdbox/storage/m.20): corrupted 
data at 24123649 (last sent=mail, last recv=mail_request (EOL))
Mar 21 11:02:28 bubba dovecot: doveadm: Error: 
dsync-remote(user at mydomain.com): Error: dsync(localhost): read() failed: 
read((fd)) failed: dot-input stream ends without '.' line (last 
sent=mail_request (EOL), last recv=mail)

Daniel

On 3/20/2017 11:30 PM, Daniel Miller wrote:
> I'm getting close - I believe I have one-way replication processing 
> now.  It's amazing watching the mails magically appear in the new 
> server!  I'll tender some updates to the wiki in the hopes it will 
> help others.
>
> At this time, I have two errors I haven't been able to eliminate with 
> config changes:
>
> dovecot: doveadm: Error: dsync-remote(user at mydomain.com): Warning: 
> Transaction log file 
> /var/mail/mydomain.com/user/sdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.log 
> was locked for 109 seconds (rotating while syncing)
> Mar 20 23:22:04 bubba dovecot: dsync-local(user at mydomain.com): Error: 
> Remote command returned error 75: ssh -lvmail 192.168.0.4 doveadm 
> dsync-server -uuser at mydomain.com
>
> I don't know if the transaction lock issue will fix itself via 
> repeated replication runs.  But I have no idea what the "error 75" is.
>
> Daniel
>
>
> On 3/20/2017 10:05 PM, Daniel Miller wrote:
>> I have an existing server using mdbox & sis. There is a quantity of 
>> corruption - I have no idea where it came in.  I'd had a previous 
>> rebuild some years ago and I thought I had a clean datastore.  By 
>> "corruption" I mean missing attachments and either missing or 
>> mixed-up emails.
>>
>> I've setup a new server on a virtual machine.  Instead of copying the 
>> old configs I hand-adjusted individual settings using the current 
>> distribution example.  The new server is running...but empty.
>>
>> So now my intent is to migrate the mail to the new machine.  A few 
>> items to note:
>> 1.  The existing mail server is the virtual host.  The mails are 
>> stored locally, in mdbox format, with sis enabled.  And possibly zlib.
>> 2.  The new server, via VirtualBox, is presently configured for 
>> Maildir.  I may change to sdbox - I'm now a little reserved about mdbox.
>> 3.  Instead of storing the mails within the virtual machine, I'm 
>> using the host's drive.  This is via an exported NFS4 folder from the 
>> host, mounted on the guest, using the options: 
>> "auto,noatime,hard,intr,async,nordirplus".
>> 4.  My intent is to have the new virtual server become the primary 
>> mail server.  The mail store will continue to be accessed by NFS4. No 
>> other servers or processes (besides backup) should access the mail 
>> store.
>>
>> So...my question is what is the best method for the migration?
>>
>> Reading the docs on Dovecot replication I'm still very unsure of how 
>> to implement it - particularly the "master-master" or "master-client" 
>> relationships.  The docs are not 100% clear on which server gets 
>> which options.  I can guess...which I'd really rather not do.  But it 
>> seems to me getting one-way replication working would be the most 
>> "elegant" solution here.
>>
>> Manual dsync is also an option - but again I'm not on which machine I 
>> should execute which options.
>>
>> Imapsync is a possibility and the syntax is clear - but very slow in 
>> execution for 100k mails.
>>
>> Any suggestions or pointers would be welcome.
>>