[Dovecot] Performance, dovecot vs tpop3d
Hello, I am new to the list and have spent extensive time researching performance suggestions through the list and the documentation. Particularly interesting is the recent performance thread related to the upcoming release.
I just upgraded one of our email servers from tpop3d to dovecot 1.0rc19. tpop3d has served us well for many years; however, it basically has lost all it's support and developers. My main liking of tpop3d was it's speed. After the upgrade to dovecot, the server began to crawl. This server runs postfix for the delivery agent and now dovecot for pop3 and imap. The box serves roughly 9000 virtual maildir mailboxes with roughly 20 pop3 processes active at any given time.
I ran some tests this morning to determine an exact difference between running tpop3d and dovecot. dovecot: I/O load averages 81%, System load averages 12.00 tpop3d: I/O load averages 20%, System load averages 0.50
To ensure it wasn't just a fluke of logins, I let each app run for several minutes, and did the test multiple times. The results were not any abnormal user usage and definitely related to application performance.
My current settings in dovecot related to performance are: mmap_diable = yes mmap_no_write = yes Modifying these to no seems to decrease performance slightly but only a few percent.
I have also played with various other settings, which didn't give me any significant performance boost.
I'm eager for the next release which will have the fsync_disable option to see if that helps, it certainly looks promising.
One observation where the majority of the load seems to be generated from is when a pop3 connection is loading up a very large maildir. The pop3 process seems to remain around using I/O for a considerable time, even when the user has no new email. Perhaps it is rebuilding or verifying it's index files? Here is an example of a lsof of the pop3 process with a user who has roughly 17000 messages. They check their email every 10 minutes which seems to add significant load. Note the final line, it's reading an individual message file, yet the user has no new email since the last login. If i keep lsofing the process, the final file seems to change and keep going through many of their mailbox files. The process takes many minutes to complete. Is it rebuilding the index on every login?
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME pop3 15148 postfix cwd DIR 9,2 152 850160 /home/domain.net/brock pop3 15148 postfix rtd DIR 9,2 816 2 / pop3 15148 postfix txt REG 9,2 1888690 7096106 /usr/local/libexec/dovecot/pop3 pop3 15148 postfix mem REG 9,2 107724 182934 /lib/ld-2.3.2.so pop3 15148 postfix mem REG 9,2 1578228 182936 /lib/tls/libc-2.3.2.so pop3 15148 postfix mem REG 9,2 16312 88977 /lib/libdl-2.3.2.so pop3 15148 postfix 0u IPv4 873280915 TCP domain.net:pop3->dsl.dyn.06-99.uslogin.net:40950 (CLOSE_WAIT) pop3 15148 postfix 1u IPv4 873280915 TCP domain.net:pop3->dsl.dyn.06-99.uslogin.net:40950 (CLOSE_WAIT) pop3 15148 postfix 2w FIFO 0,5 873280916 pipe pop3 15148 postfix 3r CHR 1,9 27963 /dev/urandom pop3 15148 postfix 4r FIFO 0,5 873281456 pipe pop3 15148 postfix 5w FIFO 0,5 873281456 pipe pop3 15148 postfix 6r FIFO 0,5 873281457 pipe pop3 15148 postfix 7w FIFO 0,5 873281457 pipe pop3 15148 postfix 8u REG 9,2 205740 7272682 /var/spool/mail/domain.net/brock/dovecot.index pop3 15148 postfix 9u REG 9,2 15020 7225137 /var/spool/mail/domain.net/brock/dovecot.index.log pop3 15148 postfix 10u REG 9,2 1585152 7290906 /var/spool/mail/domain.net/brock/dovecot.index.cache pop3 15148 postfix 11r REG 9,2 1992 5040734 /var/spool/mail/domain.net/brock/cur/1165164651.V902I4cea5e.s3.visp.net:2,S
I'm eager to solve this new I/O issue which has cropped up after upgrading to dovecot. Despite the severe performance degradation, I feel dovecot has awesome potential and is feature rich, and most importantly it's being maintained, so I want to stick with it!
Thank you for any help offered! I'm hopeful for any suggestions to solve this performance issue or diagnose it further to determine the cause.
- Nate
Hi..
wow.. this is the first time I can recall someone complaining Dovecot is so much SLOWER than their old install. :P
Nate wrote:
files. The process takes many minutes to complete. Is it rebuilding the index on every login?
The only case I can see where Dovecot would rebuild indexes on _every_ connect would be if you had INDEX=MEMORY in your mail_location.
-- Curtis Maloney cmaloney@cardgate.net
At 02:52 PM 1/30/2007, Curtis Maloney wrote:
Hi..
wow.. this is the first time I can recall someone complaining Dovecot is so much SLOWER than their old install. :P
Sorry. =( hehe, well we stuck with tpop3d for so long because it was so blazing fast. It was faster than any pop3 client we had ever used in the past 10 years. However, since it's no longer maintained, it seemed the right time to move. Dovecot has such a great userbase that it seemed the natural solution!
Nate wrote:
files. The process takes many minutes to complete. Is it rebuilding the index on every login?
The only case I can see where Dovecot would rebuild indexes on _every_ connect would be if you had INDEX=MEMORY in your mail_location.
mail_location = maildir:/var/spool/mail/%d/%n That is my whole config line, so it stores the indexes within the mail directory. here's a directory listing of the user previously mentioned. [root@s3 brock]# ls -alh total 7.4M drwx------ 5 postfix postdrop 304 Jan 30 15:21 . drwx------ 3108 postfix postdrop 79K Jan 30 13:06 .. drwx------ 2 postfix postdrop 939K Jan 30 15:21 cur -rw------- 1 postfix postdrop 202K Jan 30 15:21 dovecot.index -rw------- 1 postfix postdrop 5.3M Jan 30 15:21 dovecot.index.cache -rw------- 1 postfix postdrop 17K Jan 30 15:21 dovecot.index.log -rw------- 1 postfix postdrop 133K Jan 28 11:18 dovecot.index.log.2 -rw------- 1 postfix postdrop 743K Jan 30 15:19 dovecot-uidlist drwx------ 2 postfix postdrop 48 Jan 30 15:21 new drwx------ 2 postfix postdrop 48 Jan 30 15:19 tmp
For kicks I copied their entire mailspool (minus the index files) to a test account and connected to it via telnet. It took roughly 90 seconds to build the indexes, then it output the list command. If I disconnect and immediately reconnect it returns a list immediately and does not appear to rebuild the index; however, if I wait 5 minutes and reconnect, it appears to at least somewhat rebuild the index (guessing at this). I see it accessing the individual mail files inside the cur/ directory with lsof and it takes about 30 seconds to retrieve a list.
- Nate
At 02:52 PM 1/30/2007, Curtis Maloney wrote:
Nate wrote:
files. The process takes many minutes to complete. Is it rebuilding the index on every login?
The only case I can see where Dovecot would rebuild indexes on _every_ connect would be if you had INDEX=MEMORY in your mail_location.
mail_location = maildir:/var/spool/mail/%d/%n That is my whole config line, so it stores the indexes within the mail directory. here's a directory listing of the user previously mentioned. [root@s3 brock]# ls -alh total 7.4M drwx------ 5 postfix postdrop 304 Jan 30 15:21 . drwx------ 3108 postfix postdrop 79K Jan 30 13:06 .. drwx------ 2 postfix postdrop 939K Jan 30 15:21 cur -rw------- 1 postfix postdrop 202K Jan 30 15:21 dovecot.index -rw------- 1 postfix postdrop 5.3M Jan 30 15:21 dovecot.index.cache -rw------- 1 postfix postdrop 17K Jan 30 15:21 dovecot.index.log -rw------- 1 postfix postdrop 133K Jan 28 11:18 dovecot.index.log.2 -rw------- 1 postfix postdrop 743K Jan 30 15:19 dovecot-uidlist drwx------ 2 postfix postdrop 48 Jan 30 15:21 new drwx------ 2 postfix postdrop 48 Jan 30 15:19 tmp
For kicks I copied their entire mailspool (minus the index files) to a test account and connected to it via telnet. It took roughly 90 seconds to build the indexes, then it output the list command. If I disconnect and immediately reconnect it returns a list immediately and does not appear to rebuild the index; however, if I wait 5 minutes and reconnect, it appears to at least somewhat rebuild the index (guessing at this). I see it accessing the individual mail files inside the cur/ directory with lsof and it takes about 30 seconds to retrieve a list.
- Nate Just a wild guess... you're not using dovecot's own LDA, do you? If not,
On 01/30/2007 04:31 PM, Nate wrote: that would at least somewhat explain the re-indexing.
Jen
At 04:07 PM 1/30/2007, Jens Knoell wrote:
On 01/30/2007 04:31 PM, Nate wrote:
At 02:52 PM 1/30/2007, Curtis Maloney wrote:
Nate wrote:
files. The process takes many minutes to complete. Is it rebuilding the index on every login?
The only case I can see where Dovecot would rebuild indexes on _every_ connect would be if you had INDEX=MEMORY in your mail_location.
mail_location = maildir:/var/spool/mail/%d/%n That is my whole config line, so it stores the indexes within the mail directory. here's a directory listing of the user previously mentioned. [root@s3 brock]# ls -alh total 7.4M drwx------ 5 postfix postdrop 304 Jan 30 15:21 . drwx------ 3108 postfix postdrop 79K Jan 30 13:06 .. drwx------ 2 postfix postdrop 939K Jan 30 15:21 cur -rw------- 1 postfix postdrop 202K Jan 30 15:21 dovecot.index -rw------- 1 postfix postdrop 5.3M Jan 30 15:21 dovecot.index.cache -rw------- 1 postfix postdrop 17K Jan 30 15:21 dovecot.index.log -rw------- 1 postfix postdrop 133K Jan 28 11:18 dovecot.index.log.2 -rw------- 1 postfix postdrop 743K Jan 30 15:19 dovecot-uidlist drwx------ 2 postfix postdrop 48 Jan 30 15:21 new drwx------ 2 postfix postdrop 48 Jan 30 15:19 tmp
For kicks I copied their entire mailspool (minus the index files) to a test account and connected to it via telnet. It took roughly 90 seconds to build the indexes, then it output the list command. If I disconnect and immediately reconnect it returns a list immediately and does not appear to rebuild the index; however, if I wait 5 minutes and reconnect, it appears to at least somewhat rebuild the index (guessing at this). I see it accessing the individual mail files inside the cur/ directory with lsof and it takes about 30 seconds to retrieve a list.
- Nate Just a wild guess... you're not using dovecot's own LDA, do you? If not, that would at least somewhat explain the re-indexing.
That is correct, postfix is delivering mail directly for now.
Here's what get's me. I just setup a test, I copied the user originally referenced to a test account, one that receives no mail. Here's the steps I performed.
- login to test account. the account has no indexes, just the 17158 emails in the cur/ folder. It takes dovecot ~30 seconds to index the email and generate a LIST. lsof -p <pid> shows that pop3 is accessing email files inside the cur/ directory
- logout, log back in immediatly, there is no re-indexing and the list is available in < 1 second.
- logout, log back in 12 minutes later. The list is returned in apx 15 seconds and again in a lsof I see pop3 accessing files inside the cur/ directory.
Bear in mind, this test account had 17158 emails to begin with, and in step 3 it still outputted 17158 emails in the LIST. So even with no new email, it is accessing multiple files inside the cur/ directory, presumably re-indexing?
Perhaps it's checking to ensure email message status's are up to date in the index by just looking at the headers of the messages (total guess)?
- Nate
ps. Appreciating the replies and conversation!
At 04:43 PM 1/30/2007, Nate wrote:
At 04:07 PM 1/30/2007, Jens Knoell wrote:
On 01/30/2007 04:31 PM, Nate wrote:
At 02:52 PM 1/30/2007, Curtis Maloney wrote:
Nate wrote:
files. The process takes many minutes to complete. Is it rebuilding the index on every login?
The only case I can see where Dovecot would rebuild indexes on _every_ connect would be if you had INDEX=MEMORY in your mail_location.
mail_location = maildir:/var/spool/mail/%d/%n That is my whole config line, so it stores the indexes within the mail directory. here's a directory listing of the user previously mentioned. [root@s3 brock]# ls -alh total 7.4M drwx------ 5 postfix postdrop 304 Jan 30 15:21 . drwx------ 3108 postfix postdrop 79K Jan 30 13:06 .. drwx------ 2 postfix postdrop 939K Jan 30 15:21 cur -rw------- 1 postfix postdrop 202K Jan 30 15:21 dovecot.index -rw------- 1 postfix postdrop 5.3M Jan 30 15:21 dovecot.index.cache -rw------- 1 postfix postdrop 17K Jan 30 15:21 dovecot.index.log -rw------- 1 postfix postdrop 133K Jan 28 11:18 dovecot.index.log.2 -rw------- 1 postfix postdrop 743K Jan 30 15:19 dovecot-uidlist drwx------ 2 postfix postdrop 48 Jan 30 15:21 new drwx------ 2 postfix postdrop 48 Jan 30 15:19 tmp
For kicks I copied their entire mailspool (minus the index files) to a test account and connected to it via telnet. It took roughly 90 seconds to build the indexes, then it output the list command. If I disconnect and immediately reconnect it returns a list immediately and does not appear to rebuild the index; however, if I wait 5 minutes and reconnect, it appears to at least somewhat rebuild the index (guessing at this). I see it accessing the individual mail files inside the cur/ directory with lsof and it takes about 30 seconds to retrieve a list.
- Nate Just a wild guess... you're not using dovecot's own LDA, do you? If not, that would at least somewhat explain the re-indexing.
That is correct, postfix is delivering mail directly for now.
Here's what get's me. I just setup a test, I copied the user originally referenced to a test account, one that receives no mail. Here's the steps I performed.
- login to test account. the account has no indexes, just the 17158 emails in the cur/ folder. It takes dovecot ~30 seconds to index the email and generate a LIST. lsof -p <pid> shows that pop3 is accessing email files inside the cur/ directory
- logout, log back in immediatly, there is no re-indexing and the list is available in < 1 second.
- logout, log back in 12 minutes later. The list is returned in apx 15 seconds and again in a lsof I see pop3 accessing files inside the cur/ directory.
Bear in mind, this test account had 17158 emails to begin with, and in step 3 it still outputted 17158 emails in the LIST. So even with no new email, it is accessing multiple files inside the cur/ directory, presumably re-indexing?
Perhaps it's checking to ensure email message status's are up to date in the index by just looking at the headers of the messages (total guess)?
- Nate
ps. Appreciating the replies and conversation!
Of added note, when running a stat on the index files (dovecot.index, dovecot.index.cache, dovecot.index.log), the Modify and Change parameters remain static (no changes). So if it's rebuilding anything it's doing it in memory and not modifying the index files.
If I deliver a new message to the account, it does rebuild the files for sure and increments the Modify and Change params.
- Nate
That is correct, postfix is delivering mail directly for now.
Here's what get's me. I just setup a test, I copied the user originally referenced to a test account, one that receives no mail. Here's the steps I performed.
- login to test account. the account has no indexes, just the 17158 emails in the cur/ folder. It takes dovecot ~30 seconds to index the email and generate a LIST. lsof -p <pid> shows that pop3 is accessing email files inside the cur/ directory
- logout, log back in immediatly, there is no re-indexing and the list is available in < 1 second.
- logout, log back in 12 minutes later. The list is returned in apx 15 seconds and again in a lsof I see pop3 accessing files inside the cur/ directory.
Bear in mind, this test account had 17158 emails to begin with, and in step 3 it still outputted 17158 emails in the LIST. So even with no new email, it is accessing multiple files inside the cur/ directory, presumably re-indexing?
Perhaps it's checking to ensure email message status's are up to date in the index by just looking at the headers of the messages (total guess)?
I'm curious about your setup, what are the times when you do that with tpop3d? 17K messages makes for a huge maildir, what OS, filesystem, and how is it attached?
Kenny Dail kend@amigo.net
At 06:07 PM 1/30/2007, Kenny Dail wrote:
I'm curious about your setup, what are the times when you do that with tpop3d? 17K messages makes for a huge maildir, what OS, filesystem, and how is it attached?
I just ran the test on tpop3d, it's a production server, so naturally being early evening the server load is much higher right now, so the #'s are a bit off from the earlier testing times.
tpop3d takes ~60 seconds to build the index (right now, dovecot takes about the same). each connection afterwards returns instant results. I'll do more thorough testing tomorrow when load is down.
The machine is a 1.6ghz Athlon 512mb ram (swap is never dipped into) Local reiserfs filesystem linux software raid-1 using 2 disks disks are sata 7200 rpm seagate enterprise
tpop3d is not configured to store the index on disk, so apparently it does it all in memory; however, oddly enough, a restart doesn't slow down the next login so it must have a cache file somewhere i'm overlooking. it's late and i'll check it out tomorrow.
One significant difference I just noticed between tpop3d and dovecot, is tpop3d scans the directory and builds it's index directly after the password is verified and before any commands can be issues. Dovecot begins building at the same time; however, allows commands. A LIST command in dovecot will not output results until the index rebuild is complete. I don't think this has any impact on performance server-side, but might prompt something different to happen client side. Unsure.
Perhaps another test i'll run, just to compare apples to apples tomorrow would be to use a memory index with dovecot as well, and see if that drops the I/O load to the same as when tpop3d is running.
participants (5)
-
Curtis Maloney
-
Iain Sims
-
Jens Knoell
-
Kenny Dail
-
Nate