[Dovecot] dsync on large mailbox "fails"

[Jeff Kletsky]

I've been trying to convert an existing set of mbox mail to maildir 
using dsync, but it seems to fail when it gets to the large boxes.

$ dsync -Dv mirror maildir:~/Maildir

Everything seems to work reasonably well until the first of the large 
boxes hits:

dsync(jeff): Warning: Maildir /home/jeff/Maildir/.cron.2010-07-10: 
Synchronization took 1210 seconds (102289 new msgs, 0 flag change 
attempts, 0 expunge attempts)
dsync(jeff): Warning: Transaction log file 
/home/jeff/Maildir/.cron.2010-07-10/dovecot.index.log was locked for 
1210 seconds

After this, the doveadm process goes from ~75% of the CPU core 
utilization to 100% and stays there, even overnight.

I've also tried

$ dsync -Dv backup maildir:~/Maildir

with similar results.

While I did see some threads on dsync speed, they seemed to be minutes 
for a few GB of data, not hours.

There is plenty of space on the drive, and, from what I understand, 
there is not a file-count limit on FreeBSD's ufs (at least not one that 
I'm anywhere near)

Any suggestions?

Thanks!

Jeff



[jeff at mail ~]$ du -h -d 0 ~/mail
3.9G    /home/jeff/mail

[jeff at mail ~]$ ls -lh /var/mail/jeff
-rw-------  1 jeff  jeff   554M Apr 23 11:27 /var/mail/jeff

[jeff at mail ~]$ df -h
Filesystem      Size    Used   Avail Capacity  Mounted on
/dev/ada0s2g     38G     18G     16G    53%    /




(Note that the hostname and the location of the ssl certificate has been 
manually edited for this message in the following output)

 From dmesg.boot:

Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-STABLE #0: Sat Apr 21 17:13:07 PDT 2012
     root at my.host.name:/usr/obj/usr/src/sys/GENERIC i386
CPU: Intel(R) Atom(TM) CPU  330   @ 1.60GHz (1600.03-MHz 686-class CPU)
   Origin = "GenuineIntel"  Id = 0x106c2  Family = 6  Model = 1c  
Stepping = 2
   
Features=0xbfe9fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
   Features2=0x40e31d<SSE3,DTES64,MON,DS_CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE>
   AMD Features=0x20100000<NX,LM>
   AMD Features2=0x1<LAHF>
   TSC: P-state invariant, performance statistics
real memory  = 2147483648 (2048 MB)
avail memory = 2077757440 (1981 MB)




[jeff at mail ~]$ doveconf -n
# 2.1.3: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 9.0-STABLE i386
auth_mechanisms = plain login
disable_plaintext_auth = no
first_valid_gid = 1000
first_valid_uid = 1000
listen = *
mail_location = mbox:~/mail/:INBOX=/var/mail/%u
mail_privileged_group = mail
namespace {
   inbox = yes
   location =
   prefix =
   separator = /
}
passdb {
   args = session=yes dovecot
   driver = pam
}
protocols = imap
service auth {
   unix_listener /var/spool/postfix/private/auth {
     group = postfix
     mode = 0660
     user = postfix
   }
   unix_listener auth-master {
     mode = 0600
   }
   user = root
}
service imap-login {
   inet_listener imap {
     port = 0
   }
   process_limit = 200
}
ssl = required
ssl_cert = </path/to/certificate.pem
ssl_key = </path/to/certificate.pem
userdb {
   args = blocking=yes
   driver = passwd
}
verbose_proctitle = yes
protocol imap {
   imap_client_workarounds = tb-extra-mailbox-sep
}