[Dovecot] Quick and dirty server optimized for IMAP upload speed?

[Adam McDougall]

Timo Sirainen wrote:
> On Aug 28, 2009, at 8:38 PM, Adam McDougall wrote:
>
>> Early next week I need to upload over 100,000 emails to an IMAP 
>> server as quickly as possible from an Outlook client. I am looking 
>> for any methods I can use to (temporarily?) speed up the rate at 
>> which dovecot can accept and store IMAP uploads, whether it be 
>> storing on local disk, ram disk, etc.  I can setup a temporary server 
>> on a laptop for example and once the upload has finished I can use 
>> standard file copying methods to transfer the mail to stable, 
>> permanent storage.  I haven't been able to see over about 7 msgs/sec 
>> upload speed from a local folder in any mail client to dovecot (only 
>> NFS or ZFS backend tested so far with Maildir).  Is there something 
>> horribly wrong with the speed I am seeing or are there just tricks I 
>> can try?  Any tips?  I'll be working on it all weekend until I find 
>> something satisfactory.  It seems like I can upload mails to an 
>> Exchange server quicker.  I'll setup just about anything that my 
>> experience allows me to, I can be very resourceful with adhoc 
>> hardware and software.
>
> From Dovecot's side the only thing you can do is fsync_disable=yes. 
> The main problem is probably network latency, because Outlook doesn't 
> support MULTIAPPEND extension (and perhaps not even LITERAL+ 
> extension?) Did you already try running Dovecot on the same computer 
> as Outlook (some virtual thingy or maybe it works in cygwin)?
>
I just tried fsync_disable=yes but with NFS and had to turn off 
mail_nfs_index = yes as well but the speed was the same.  Do you think 
it would be different with a UFS or ZFS backend with fsync_disable?  I 
have not tried running dovecot on the same computer. When you mention 
dovecot+cygwin I think of the reported issues in the past on the mailing 
list and don't know if they were resolved.  I could try dovecot in 
virtualbox I suppose (I put it on my list to try). 
>> Alternatively I'll take a fast way of converting Exchange email to a 
>> tree of local mbox files which I can then run mb2md on.
>
> If the mails are in Exchange, can't you connect to it using IMAP?
>
In theory yes, but I don't have access to the actual Exchange server 
until Monday at the earliest, and the user is using "cached exchange 
mode" which in past experience leaves the possibility of local mail 
which is not actually on the server due to a desync.  Unless I am sure 
it is perfectly in sync, I've seen a second Outlook connect to Exchange 
using the native protocols and it initiated a massive deletion of mail 
which we had to toil to recover from obscure cache files on the original 
client.  I don't know if an IMAP connection might trigger the same 
issue.  For performance testing's sake, I'll see if I can upload some 
mail to our own Exchange server and see how fast an mbox capable mail 
client can download it.  I can do some limited testing in the real 
environment on Monday but I'm expected to do the real migration on 
Tuesday unless I have to cancel.  Thanks for the ideas.