[Dovecot] repeating dsync - questions

[Ken A]

Maybe take a look at "vmstat 2" and see if i/o is blocking a lot or you
are hitting swap space? Someone else may be more informed about the
inner workings of dsync and how it handles maildir as opposed to mbox.
Best of luck,
Ken


On 4/6/2013 1:09 AM, Bob Gustafson wrote:
> Something must be wrong with my setup. It took 14+ hours for the first
> dsync pass and it hasn't finished yet on the rerun to pick up the latest
> mail (around 12+ hours). I have about 9 GB of mail! The destination
> system is not fast, but..
> 
> Maybe I will wait for the 2.2 release..
> 
> Thanks for your response.
> 
> Bob G
> 
> On Fri, 2013-04-05 at 22:45 -0500, Ken A wrote:
>> It's about 300GB of mboxes (I don't know how many messages) and dsync
>> took about 3 hours from scratch in the destination. But, when I sync
>> more frequently, it's much quicker, and depends on the number of
>> changes. With no changes it runs in about 15 min.
>> Ken A.
>>
>>
>> On 4/5/2013 1:54 PM, Bob Gustafson wrote:
>>> I tried that a week or so ago, with a 'dsync -R backup', but got the
>>> funny named directories, so I read more and am trying the 'dsync
>>> mirror' which doesn't require the -R.
>>>
>>> How long does the sync step take for you? (Normalize to # of
>>> messages..)
>>>
>>> Bob G
>>>
>>> On Apr 5, 2013, at 13:48, Ken A <ka at pacific.net> wrote:
>>>
>>>> I've found that when going in one direction, using "backup -R"
>>>> rather than mirror, works better. I'm going from mbox to sdbox, but
>>>> doing roughly the same thing you are, rsync and then dsync.
>>>>
>>>> Ken A.
>>>>
>>>> On 4/5/2013 10:40 AM, Bob Gustafson wrote:
>>>>>
>>>>> I'm moving/converting email from one system to another
>>>>>
>>>>> The source system is: Dual core x86_64 6GB memory 180 GB raid1
>>>>> disks ext4 Fedora 9 Dovecot 1.0.15 Maildir format
>>>>>
>>>>> The destination system is: Dual core x86_64 2GB memory 1TB raid1
>>>>> disks ext4 Fedora 18 Dovecot 2.1.15 sdbox format
>>>>>
>>>>> I am moving mail in a series of steps:
>>>>>
>>>>> 0) cd ( to mail user home directory ) mkdir Maildir
>>>>>
>>>>> 1) # rsync mail from source system (hoho4) to current (hoho0) 
>>>>> time rsync -arv --times hoho4:/home/bobgus/Maildir/
>>>>> /home/bobgus/Maildir/
>>>>>
>>>>> This step takes about 37 minutes the first time
>>>>>
>>>>> 2) # dsync mirror/convert to sdbox format on current system time
>>>>> dsync mirror maildir:~/Maildir
>>>>>
>>>>> This step takes about 858 minutes (!!)
>>>>>
>>>>> Looking at -D messages indicates that dsync is deciding between 
>>>>> duplicates much of the time.
>>>>>
>>>>> Looking at the results in a mail browser (Evolution), it seems
>>>>> fine, although the latest mails are not there.
>>>>>
>>>>> 3) # pick up more current mail - use same script as in 1) time
>>>>> rsync -arv --times hoho4:/home/bobgus/Maildir/
>>>>> /home/bobgus/Maildir/
>>>>>
>>>>> This step takes about 5 minutes although this varies depending on
>>>>> the amount of new mail.
>>>>>
>>>>> 4) # repeat the 2) dsync step time dsync mirror
>>>>> maildir:~/Maildir
>>>>>
>>>>> This is where things look peculiar. When I look at the directory
>>>>> of sdbox/mailboxes, I see duplicated directories
>>>>>
>>>>> [bobgus at hoho0 sdbox]$ cd mailboxes [bobgus at hoho0 mailboxes]$ ls 
>>>>> Apple Mail To Do Apple Mail To
>>>>> Do_2a47983780615e5179600000ba55d82c Deleted Messages Deleted
>>>>> Messages_2847983780615e5179600000ba55d82c Drafts 
>>>>> Drafts_2447983780615e5179600000ba55d82c Important 
>>>>> Important_2947983780615e5179600000ba55d82c INBOX 
>>>>> INBOX_1547983780615e5179600000ba55d82c Sent 
>>>>> Sent_2547983780615e5179600000ba55d82c Trash 
>>>>> Trash_2747983780615e5179600000ba55d82c
>>>>>
>>>>> The 2nd dsync step has not completed yet. I'm wondering whether
>>>>> the extra directory will be magically moved into the older
>>>>> directory.
>>>>>
>>>>> (This is unlikely because the file names are duplicated in the
>>>>> new directory)
>>>>>
>>>>> -------
>>>>>
>>>>> Is this expected behavior? Are there command changes I can make
>>>>> to speed up the process? Eliminate the duplicate directories?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> -- Ken Anderson Pacific Internet - http://www.pacific.net
>>>
>>
> 
> 

-- 
Ken Anderson
Pacific Internet - http://www.pacific.net