On 4/23/12 12:56 PM, Jeff Kletsky wrote:

I've been trying to convert an existing set of mbox mail to maildir using dsync, but it seems to fail when it gets to the large boxes.

$ dsync -Dv mirror maildir:~/Maildir

Everything seems to work reasonably well until the first of the large boxes hits:

dsync(jeff): Warning: Maildir /home/jeff/Maildir/.cron.2010-07-10: Synchronization took 1210 seconds (102289 new msgs, 0 flag change attempts, 0 expunge attempts) dsync(jeff): Warning: Transaction log file /home/jeff/Maildir/.cron.2010-07-10/dovecot.index.log was locked for 1210 seconds

After this, the doveadm process goes from ~75% of the CPU core utilization to 100% and stays there, even overnight.

I believe that some of problem may be that there were a handful of boxes named with "." in them. These were not "flagged" in the conversion process in any way.

http://wiki2.dovecot.org/Migration/MailFormat indicates that you can't have a "." in the name of a Maildir box

(I dutifully stopped reading that page at "If you can successfully use dsync, you can skip the rest of this page.")

In my case, it may be a contributing factor that there were boxes that ended up "stacking" inside each other:

• cron
• cron.2010-07-10
• cron.2010-07-10.remaining

Still not blisteringly fast, but

dsync(jeff): Warning: Maildir /home/jeff/Maildir/.cron_2010-07-10: Synchronization took 686 seconds (102289 new msgs, 0 flag change attempts, 0 expunge attempts) dsync(jeff): Warning: Transaction log file /home/jeff/Maildir/.cron_2010-07-10/dovecot.index.log was locked for 686 seconds

is certainly better.

I'm still seeing 100% CPU utilization, with the tail of ls -Altr in ~/Maildir showing

drwx------ 5 jeff jeff 512 Apr 23 18:19 .XDA drwx------ 5 jeff jeff 512 Apr 23 18:19 .Spam drwx------ 5 jeff jeff 512 Apr 23 18:42 .Health drwx------ 5 jeff jeff 512 Apr 23 18:42 .cron_2010-07-10_remaining drwx------ 5 jeff jeff 512 Apr 23 18:58 .cron_2010-07-10

It appears from looking in the .cron_2010-07-10 directory that indexing is the action underway

[jeff@mail ~/Maildir]$ ls -Altr .cron_2010-07-10 total 27 drwx------ 2 jeff jeff 512 Apr 23 18:15 new -rw------- 1 jeff jeff 0 Apr 23 18:15 maildirfolder drwx------ 2 jeff jeff 5819392 Apr 23 18:42 tmp drwx------ 2 jeff jeff 5237248 Apr 23 18:42 cur -rw------- 1 jeff jeff 4122971 Apr 23 18:42 dovecot-uidlist -rw------- 1 jeff jeff 36 Apr 23 18:42 dovecot-keywords -rw------- 1 jeff jeff 1650764 Apr 23 18:42 dovecot.index.log.2 -rw------- 1 jeff jeff 1637008 Apr 23 18:42 dovecot.index -rw------- 1 jeff jeff 56124 Apr 23 19:04 dovecot.index.log -rw------- 1 jeff jeff 9442304 Apr 23 19:04 dovecot.index.cache -rw------- 1 jeff jeff 24 Apr 23 19:04 dovecot-uidlist.lock

(time passes)

It is now 20:35 and it has not advanced. I now no longer believe the problem is the mailbox naming, but the indexing.

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 25934 jeff 1 103 0 135M 102M CPU2 2 128:47 100.00% doveadm

[jeff@mail ~/Maildir]$ ls -Altr .cron_2010-07-10 total 28 drwx------ 2 jeff jeff 512 Apr 23 18:15 new -rw------- 1 jeff jeff 0 Apr 23 18:15 maildirfolder drwx------ 2 jeff jeff 5819392 Apr 23 18:42 tmp drwx------ 2 jeff jeff 5237248 Apr 23 18:42 cur -rw------- 1 jeff jeff 4122971 Apr 23 18:42 dovecot-uidlist -rw------- 1 jeff jeff 36 Apr 23 18:42 dovecot-keywords -rw------- 1 jeff jeff 1650764 Apr 23 18:42 dovecot.index.log.2 -rw------- 1 jeff jeff 1637008 Apr 23 20:26 dovecot.index -rw------- 1 jeff jeff 289120 Apr 23 20:36 dovecot.index.log -rw------- 1 jeff jeff 9442304 Apr 23 20:36 dovecot.index.cache -rw------- 1 jeff jeff 24 Apr 23 20:36 dovecot-uidlist.lock

Any suggestions as to how to proceed?

Jeff