[Dovecot] Performance, dovecot vs tpop3d

[Nate]

At 04:07 PM 1/30/2007, Jens Knoell wrote:
>On 01/30/2007 04:31 PM, Nate wrote:
>>At 02:52 PM 1/30/2007, Curtis Maloney wrote:
>>>Nate wrote:
>>>>files.  The process takes many minutes to complete.  Is it 
>>>>rebuilding the index on every login?
>>>
>>>The only case I can see where Dovecot would rebuild indexes on 
>>>_every_ connect would be if you had INDEX=MEMORY in your mail_location.
>>
>>mail_location = maildir:/var/spool/mail/%d/%n
>>That is my whole config line, so it stores the indexes within the 
>>mail directory.  here's a directory listing of the user previously mentioned.
>>[root at s3 brock]# ls -alh
>>total 7.4M
>>drwx------     5 postfix postdrop  304 Jan 30 15:21 .
>>drwx------  3108 postfix postdrop  79K Jan 30 13:06 ..
>>drwx------     2 postfix postdrop 939K Jan 30 15:21 cur
>>-rw-------     1 postfix postdrop 202K Jan 30 15:21 dovecot.index
>>-rw-------     1 postfix postdrop 5.3M Jan 30 15:21 dovecot.index.cache
>>-rw-------     1 postfix postdrop  17K Jan 30 15:21 dovecot.index.log
>>-rw-------     1 postfix postdrop 133K Jan 28 11:18 dovecot.index.log.2
>>-rw-------     1 postfix postdrop 743K Jan 30 15:19 dovecot-uidlist
>>drwx------     2 postfix postdrop   48 Jan 30 15:21 new
>>drwx------     2 postfix postdrop   48 Jan 30 15:19 tmp
>>
>>For kicks I copied their entire mailspool (minus the index files) 
>>to a test account and connected to it via telnet.  It took roughly 
>>90 seconds to build the indexes, then it output the list 
>>command.  If I disconnect and immediately reconnect it returns a 
>>list immediately and does not appear to rebuild the index; however, 
>>if I wait 5 minutes and reconnect, it appears to at least somewhat 
>>rebuild the index (guessing at this).  I see it accessing the 
>>individual mail files inside the cur/ directory with lsof and it 
>>takes about 30 seconds to retrieve a list.
>>
>>- Nate
>Just a wild guess... you're not using dovecot's own LDA, do you? If 
>not, that would at least somewhat explain the re-indexing.

That is correct, postfix is delivering mail directly for now.

Here's what get's me.  I just setup a test, I copied the user 
originally referenced to a test account, one that receives no 
mail.  Here's the steps I performed.

1.  login to test account.  the account has no indexes, just the 
17158 emails in the cur/ folder.  It takes dovecot ~30 seconds to 
index the email and generate a LIST.  lsof -p <pid> shows that pop3 
is accessing email files inside the cur/ directory
2.  logout, log back in immediatly, there is no re-indexing and the 
list is available in < 1 second.
3.  logout, log back in 12 minutes later.  The list is returned in 
apx 15 seconds and again in a lsof I see pop3 accessing files inside 
the cur/ directory.

Bear in mind, this test account had 17158 emails to begin with, and 
in step 3 it still outputted 17158 emails in the LIST.  So even with 
no new email, it is accessing multiple files inside the cur/ 
directory, presumably re-indexing?

Perhaps it's checking to ensure email message status's are up to date 
in the index by just looking at the headers of the messages (total guess)?

- Nate

ps.  Appreciating the replies and conversation!