[Dovecot] Performance, dovecot vs tpop3d

[Nate]

At 04:43 PM 1/30/2007, Nate wrote:
>At 04:07 PM 1/30/2007, Jens Knoell wrote:
>>On 01/30/2007 04:31 PM, Nate wrote:
>>>At 02:52 PM 1/30/2007, Curtis Maloney wrote:
>>>>Nate wrote:
>>>>>files.  The process takes many minutes to complete.  Is it 
>>>>>rebuilding the index on every login?
>>>>
>>>>The only case I can see where Dovecot would rebuild indexes on 
>>>>_every_ connect would be if you had INDEX=MEMORY in your mail_location.
>>>
>>>mail_location = maildir:/var/spool/mail/%d/%n
>>>That is my whole config line, so it stores the indexes within the 
>>>mail directory.  here's a directory listing of the user previously mentioned.
>>>[root at s3 brock]# ls -alh
>>>total 7.4M
>>>drwx------     5 postfix postdrop  304 Jan 30 15:21 .
>>>drwx------  3108 postfix postdrop  79K Jan 30 13:06 ..
>>>drwx------     2 postfix postdrop 939K Jan 30 15:21 cur
>>>-rw-------     1 postfix postdrop 202K Jan 30 15:21 dovecot.index
>>>-rw-------     1 postfix postdrop 5.3M Jan 30 15:21 dovecot.index.cache
>>>-rw-------     1 postfix postdrop  17K Jan 30 15:21 dovecot.index.log
>>>-rw-------     1 postfix postdrop 133K Jan 28 11:18 dovecot.index.log.2
>>>-rw-------     1 postfix postdrop 743K Jan 30 15:19 dovecot-uidlist
>>>drwx------     2 postfix postdrop   48 Jan 30 15:21 new
>>>drwx------     2 postfix postdrop   48 Jan 30 15:19 tmp
>>>
>>>For kicks I copied their entire mailspool (minus the index files) 
>>>to a test account and connected to it via telnet.  It took roughly 
>>>90 seconds to build the indexes, then it output the list 
>>>command.  If I disconnect and immediately reconnect it returns a 
>>>list immediately and does not appear to rebuild the index; 
>>>however, if I wait 5 minutes and reconnect, it appears to at least 
>>>somewhat rebuild the index (guessing at this).  I see it accessing 
>>>the individual mail files inside the cur/ directory with lsof and 
>>>it takes about 30 seconds to retrieve a list.
>>>
>>>- Nate
>>Just a wild guess... you're not using dovecot's own LDA, do you? If 
>>not, that would at least somewhat explain the re-indexing.
>
>That is correct, postfix is delivering mail directly for now.
>
>Here's what get's me.  I just setup a test, I copied the user 
>originally referenced to a test account, one that receives no 
>mail.  Here's the steps I performed.
>
>1.  login to test account.  the account has no indexes, just the 
>17158 emails in the cur/ folder.  It takes dovecot ~30 seconds to 
>index the email and generate a LIST.  lsof -p <pid> shows that pop3 
>is accessing email files inside the cur/ directory
>2.  logout, log back in immediatly, there is no re-indexing and the 
>list is available in < 1 second.
>3.  logout, log back in 12 minutes later.  The list is returned in 
>apx 15 seconds and again in a lsof I see pop3 accessing files inside 
>the cur/ directory.
>
>Bear in mind, this test account had 17158 emails to begin with, and 
>in step 3 it still outputted 17158 emails in the LIST.  So even with 
>no new email, it is accessing multiple files inside the cur/ 
>directory, presumably re-indexing?
>
>Perhaps it's checking to ensure email message status's are up to 
>date in the index by just looking at the headers of the messages (total guess)?
>
>- Nate
>
>ps.  Appreciating the replies and conversation!

Of added note, when running a stat on the index files (dovecot.index, 
dovecot.index.cache, dovecot.index.log), the Modify and Change 
parameters remain static (no changes).  So if it's rebuilding 
anything it's doing it in memory and not modifying the index files.

If I deliver a new message to the account, it does rebuild the files 
for sure and increments the Modify and Change params.

- Nate