[Dovecot] repeating dsync - questions

[Bob Gustafson]

I whacked Maildir and sdbox and started over.

Tweek of rsync script - removed trailing / on destination.

Did yum update, restart

Avoided running Firefox on that machine..

It is now running the 1st pass of dsync - estimated finish is about 4.3 hours on 14G of mails

vmstat 2 shows no swapping (now..)

Thanks for your suggestion.

Bob G

On Apr 6, 2013, at 10:19, Ken A <ka at pacific.net> wrote:

> Maybe take a look at "vmstat 2" and see if i/o is blocking a lot or you
> are hitting swap space? Someone else may be more informed about the
> inner workings of dsync and how it handles maildir as opposed to mbox.
> Best of luck,
> Ken
> 
> 
> On 4/6/2013 1:09 AM, Bob Gustafson wrote:
>> Something must be wrong with my setup. It took 14+ hours for the first
>> dsync pass and it hasn't finished yet on the rerun to pick up the latest
>> mail (around 12+ hours). I have about 9 GB of mail! The destination
>> system is not fast, but..
>> 
>> Maybe I will wait for the 2.2 release..
>> 
>> Thanks for your response.
>> 
>> Bob G
>> 
>> On Fri, 2013-04-05 at 22:45 -0500, Ken A wrote:
>>> It's about 300GB of mboxes (I don't know how many messages) and dsync
>>> took about 3 hours from scratch in the destination. But, when I sync
>>> more frequently, it's much quicker, and depends on the number of
>>> changes. With no changes it runs in about 15 min.
>>> Ken A.
>>> 
>>> 
>>> On 4/5/2013 1:54 PM, Bob Gustafson wrote:
>>>> I tried that a week or so ago, with a 'dsync -R backup', but got the
>>>> funny named directories, so I read more and am trying the 'dsync
>>>> mirror' which doesn't require the -R.
>>>> 
>>>> How long does the sync step take for you? (Normalize to # of
>>>> messages..)
>>>> 
>>>> Bob G
>>>> 
>>>> On Apr 5, 2013, at 13:48, Ken A <ka at pacific.net> wrote:
>>>> 
>>>>> I've found that when going in one direction, using "backup -R"
>>>>> rather than mirror, works better. I'm going from mbox to sdbox, but
>>>>> doing roughly the same thing you are, rsync and then dsync.
>>>>> 
>>>>> Ken A.
>>>>> 
>>>>> On 4/5/2013 10:40 AM, Bob Gustafson wrote:
>>>>>> 
>>>>>> I'm moving/converting email from one system to another
>>>>>> 
>>>>>> The source system is: Dual core x86_64 6GB memory 180 GB raid1
>>>>>> disks ext4 Fedora 9 Dovecot 1.0.15 Maildir format
>>>>>> 
>>>>>> The destination system is: Dual core x86_64 2GB memory 1TB raid1
>>>>>> disks ext4 Fedora 18 Dovecot 2.1.15 sdbox format
>>>>>> 
>>>>>> I am moving mail in a series of steps:
>>>>>> 
>>>>>> 0) cd ( to mail user home directory ) mkdir Maildir
>>>>>> 
>>>>>> 1) # rsync mail from source system (hoho4) to current (hoho0) 
>>>>>> time rsync -arv --times hoho4:/home/bobgus/Maildir/
>>>>>> /home/bobgus/Maildir/
>>>>>> 
>>>>>> This step takes about 37 minutes the first time
>>>>>> 
>>>>>> 2) # dsync mirror/convert to sdbox format on current system time
>>>>>> dsync mirror maildir:~/Maildir
>>>>>> 
>>>>>> This step takes about 858 minutes (!!)
>>>>>> 
>>>>>> Looking at -D messages indicates that dsync is deciding between 
>>>>>> duplicates much of the time.
>>>>>> 
>>>>>> Looking at the results in a mail browser (Evolution), it seems
>>>>>> fine, although the latest mails are not there.
>>>>>> 
>>>>>> 3) # pick up more current mail - use same script as in 1) time
>>>>>> rsync -arv --times hoho4:/home/bobgus/Maildir/
>>>>>> /home/bobgus/Maildir/
>>>>>> 
>>>>>> This step takes about 5 minutes although this varies depending on
>>>>>> the amount of new mail.
>>>>>> 
>>>>>> 4) # repeat the 2) dsync step time dsync mirror
>>>>>> maildir:~/Maildir
>>>>>> 
>>>>>> This is where things look peculiar. When I look at the directory
>>>>>> of sdbox/mailboxes, I see duplicated directories
>>>>>> 
>>>>>> [bobgus at hoho0 sdbox]$ cd mailboxes [bobgus at hoho0 mailboxes]$ ls 
>>>>>> Apple Mail To Do Apple Mail To
>>>>>> Do_2a47983780615e5179600000ba55d82c Deleted Messages Deleted
>>>>>> Messages_2847983780615e5179600000ba55d82c Drafts 
>>>>>> Drafts_2447983780615e5179600000ba55d82c Important 
>>>>>> Important_2947983780615e5179600000ba55d82c INBOX 
>>>>>> INBOX_1547983780615e5179600000ba55d82c Sent 
>>>>>> Sent_2547983780615e5179600000ba55d82c Trash 
>>>>>> Trash_2747983780615e5179600000ba55d82c
>>>>>> 
>>>>>> The 2nd dsync step has not completed yet. I'm wondering whether
>>>>>> the extra directory will be magically moved into the older
>>>>>> directory.
>>>>>> 
>>>>>> (This is unlikely because the file names are duplicated in the
>>>>>> new directory)
>>>>>> 
>>>>>> -------
>>>>>> 
>>>>>> Is this expected behavior? Are there command changes I can make
>>>>>> to speed up the process? Eliminate the duplicate directories?
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> -- Ken Anderson Pacific Internet - http://www.pacific.net
>>>> 
>>> 
>> 
>> 
> 
> -- 
> Ken Anderson
> Pacific Internet - http://www.pacific.net