[Dovecot] dsync and pop3 migration plugin fails for large inbox due to idle timeout

[Nikolai Schupbach]

Hi Timo,

Thanks for your fix! Just want to confirm that the re-order of connections has fixed the issue. We still have occasional issues with the POP3 server (locking issues and busy timeouts etc). Because dsync reports sync ok, we have parse to output looking for 'Error: pop3_migration' to determine if it was in-fact ok or if we need to do a re-sync.

We have seen some IMAP 'Server dsconnected unexpectedly' errors also (on large mailboxes again), but it doesn't appear to cause any issues. Does dsync reconnect if the IMAP connection is lost?

Thank you.

Regards, Nikolai.

On 4/08/2012, at 2:40 AM, Timo Sirainen wrote:

> On 3.8.2012, at 10.04, Nikolai Schupbach wrote:
> 
>> There is a problem with the dsync POP3 migration plugin when syncing a large INBOX. What happens is dsync establishes a connection to the POP3 server at the beginning of the sync, but then it sits there idle until it's used to sync the UIDLs. On a very large inbox with many thousands of messages it will be idle for longer than a POP3 server's idle timeout setting. 
> 
> What should happen is:
> 
> 1. pop3c connects to server
> 2. issues LIST and UIDL commands
> 3. IMAP server issues fetch 1:* RFC822.SIZE (or I guess it does this with a ton of separate commands)
> 4. pop3c attempts to do some TOP commands.
> 
>> When the UIDL sync occurs dsync produces the following errors (because the connection has been lost due to the POP3 server's inactivity timeout):
>> 
>> dsync(first.last at test.co.nz): Error: pop3c(pop3.test.com): Server disconnected unexpectedly
>> dsync(first.last at test.co.nz): Error: pop3_migration: Failed to get header for msg 13: Disconnected
> 
> It looks like the 3 step takes a long time. Do you have imapc_features=rfc822.size? Also setting mail_prefetch_count to something between 10..100 should help a lot. And this should also fix it by moving 3 step to 1: http://hg.dovecot.org/dovecot-2.1/rev/ab6a4455b27d
> 
>> Also at the end of the dsync it actually has an exit status of 0, but it shouldn't because the POP3 sync failed, and none of the UIDLs are retained.
> 
> Yes, looks like dsync simply continues without assigning POP3 UIDL. Hmm. Perhaps it shouldn't do that on temporary failures..
> 
>> Dsync should try to keep the POP3 connection alive, by sending a regular 'NOOP' command to the POP3 server,
> 
> This isn't easily possible.
> 
>> or it should simply reconnect if sees the connection is lost.
> 
> It should probably do that, yes.