[Dovecot] Poor pop3 over nfs performance

[Mark Zealey]

In case anyone else is experiencing this issue, using the following in a
postlogin script is a reasonable workaround:

if [[ ! -e "$HOME/.lastlogin_imap" &&
"$HOME/Maildir/dovecot.index.cache" -ot "$HOME/Maildir/new" &&
"$HOME/Maildir/dovecot-uidlist" -ot "$HOME/Maildir/new" ]]; then
  rm -f "$HOME/Maildir/dovecot.index.cache"
fi

(whenever a user logs in via imap we touch .lastlogin_imap, so we only
kill the cache file if the user is pop3-only and both the cache and
uidlist are older than the new directory).

Mark

--
Mark Zealey -- Shared Hosting Team Leader
Product Development * Webfusion
123-reg.co.uk, webfusion.co.uk, donhost.co.uk, supanames.co.uk

This mail is subject to http://www.gxn.net/disclaimer 

> -----Original Message-----
> From: dovecot-bounces+mark.zealey=webfusion.com at dovecot.org 
> [mailto:dovecot-bounces+mark.zealey=webfusion.com at dovecot.org]
>  On Behalf Of Mark Zealey
> Sent: 02 July 2008 11:01
> To: Dovecot Mailing List
> Subject: Re: [Dovecot] Poor pop3 over nfs performance
> 
> > rc11 had this fix which may be relevant:
> > 
> > 	- dovecot-uidlist is now recreated if it results in 
> > file shrinking
> > 	  over 25%.
> 
> Not sure this fix is relevent; but I have upgraded to 1.1.1
> 
> > With v1.1 INDEX=MEMORY shouldn't matter that much anymore, since it
> > should cache the virtual message size to dovecot-uidlist.
> > 
> > 1) Do you have ,W=<size> in maildir filenames? Apparently not?
> 
> No; yesterday though I did switch exim to delivering through lda -
> surprisingly though there was not much effect on the graphs.
> 
> > 2) Do you have W=<size> added to dovecot-uidlist?
> 
> I have found something here which I believe is a bug; will 
> detail below
> 
> > 3) Are your POP3 users download + delete or do they leave 
> mail on the
> > server?
> 
> A mixture; I believe at least 80% will download+delete; 
> probably closer
> to 95% but we do have some very large mail stores - most of 
> these would
> be accessed by a combination of pop and imap.
> 
> > You should be able to use dotlock_use_excl=yes.
> ...
> > dotlock is the slowest locking method. Did you try how well 
> > fcntl would
> > have worked?
> 
> Not tried fcntl; I don't really trust nfs locking. I have enabled the
> excl in the config; again not much change.
> 
> Now what I discovered in testing a particularly problematic 
> user with an
> 800mb cur directory (under v1.1.1; I have never used v1.0 on this mbox
> but it was converted with your courier convert script); initially they
> had the following:
> 
> drwx------ 2 exim exim   2048 Apr 11  2007 courierimapkeywords
> -rw-r--r-- 1 exim exim     47 Apr 22 15:56 courierimapsubscribed
> -rw-r--r-- 1 exim exim 142493 Jun 23 19:57 courierimapuiddb
> -rw-r--r-- 1 exim exim 195309 Jun 24 18:52 courierpop3dsizelist
> drwx------ 2 exim exim 499712 Jul  2 10:35 cur
> -rw------- 1 exim exim  20912 Jul  2 10:51 dovecot.index
> -rw------- 1 exim exim  33056 Jun 17 17:34 dovecot.index.cache
> -rw------- 1 exim exim  31336 Jul  2 10:37 dovecot.index.log
> -rw------- 1 exim exim 158465 Jul  2 10:13 dovecot-uidlist
> drwx------ 2 exim exim   4096 Jul  2 10:29 new
> -rw------- 1 exim exim     23 Jun 24 22:49 subscriptions
> drwx------ 2 exim exim   2048 Jul  2 10:42 tmp
> 
> As you can see, the cache file is very old, and the uidlist file is
> behind the last time that the logged in (which was 10:35). 
> When I log in
> over pop and do a stat/list, it re-reads everything in cur/new but
> didn't update the dovecot-uidlist file or the index.cache 
> file (the log
> file is updated though). Dovecot-uidlist file does not have 
> the extra W=
> tags in it. I removed the dovecot.index.cache file and performed the
> same login/stat/list operation and logged out, and suddenly the
> dovecot-uidlist file was updated to the new v1.1 format, and the
> directory listing looks like:
> 
> total 1068
> drwx------ 2 exim exim   2048 Apr 11  2007 courierimapkeywords
> -rw-r--r-- 1 exim exim     47 Apr 22 15:56 courierimapsubscribed
> -rw-r--r-- 1 exim exim 142493 Jun 23 19:57 courierimapuiddb
> -rw-r--r-- 1 exim exim 195309 Jun 24 18:52 courierpop3dsizelist
> drwx------ 2 exim exim 499712 Jul  2 10:35 cur
> -rw------- 1 exim exim  20912 Jul  2 10:53 dovecot.index
> -rw------- 1 exim exim  31428 Jul  2 10:53 dovecot.index.log
> -rw------- 1 exim exim 175508 Jul  2 10:53 dovecot-uidlist
> drwx------ 2 exim exim   4096 Jul  2 10:29 new
> -rw------- 1 exim exim     23 Jun 24 22:49 subscriptions
> drwx------ 2 exim exim   2048 Jul  2 10:42 tmp
> 
> This looks like a bug to me; I will have a look in the source 
> and see if
> I can find anything, but I thought I'd send this email as 
> you'll be able
> to find the bug faster than I can.
> 
> Thanks,
> 
> Mark
>