Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 23 Jun 2010, Rajesh M wrote:
Roundcube 0.3.1 seems to have a hard coded limit of 200 mails per page.
it again took around 4-5 mins
I do not know nothing about Roundcube, therefore I wonder where the 5min delay come from:
a) Do you keep index files on Dovecot side?
b) Do you deliver / manipulate the mailbox via Dovecot only? E.g. do you use Dovecot deliver?
c) If you rawlog or sniff the connection and issue the same commands as roundcube to Dovecot via, say, telnet next morning, is the delay the same? http://wiki.dovecot.org/Debugging/Rawlog http://wiki.dovecot.org/TestInstallation?highlight=%28telnet%29
a) and b) should avoid rereading of the mailbox. c) is to ensure that Roundcube does not add the delay itself.
the extra 1000 emails were to be added to the cache .. does dovecot go thru all the 170000 emails again to form the meta data ? or is there some cache expiry period ?
Usually no.
Regards,
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iQEVAwUBTCG2V7+Vh58GPL/cAQJyHAf+MayOiuayYay2mCM5i0DJH296pbuDMAXc kbXKMRXFOP/KGeAYmHirL/9/63h0IjeRtSuBe0HJxmbg87aZVWOejVaXSPOPTAUR n8d++siuBnQ8xiDFw0LKgep2HetYFdxepDRjdPTOM/Drpv+THFRGOVnem6JYb7yw 0XqkQG7M4pKBaxZA+0HFH4c4ADz3XykkOluTHhmU0zDXcdkK0TybfEiwGRY3R7i8 IozQ62pKv/v2t2z2clYOXsqLe6pM14B8/2SXUX4Anrkz8AoKyocT1Ks9hOCkizEn bG99r7U+WObrMmbTpy6jsq//amh496ot37S4DO7DBKa3cKYqOJwwXg== =gqnn -----END PGP SIGNATURE-----
hi
did the tests once again on this large email box
i set maildir_very_dirty_syncs = yes this was done in the morning and dovecot was restarted
i logged after about 4 hours after the previous login again it took around 5 mins to login
i was monitoring my server load which around 1.5 - 2 on my dual core dual xeon machine
this increased to around 8-9 during the login process
the no of emails in the box had increase by around 7500 emails
setting maildir_very_dirty_syncs = yes
does not seem to help
i have pasted below my dovecot.conf file
################
base_dir = /var/run/dovecot/ protocols = imap imaps log_path = /backup1/qmaillog/dovecot.log #ssl_disable = no ssl_cert_file = /var/qmail/control/servercert.pem ssl_key_file = /var/qmail/control/servercert.pem ssl_cipher_list = djdjjd verbose_ssl = yes
protocol imap { listen = *:143 ssl_listen = *:993 } ## Login processes #login_dir = /usr/local/var/run/dovecot/login login_user = dovecot login_process_per_connection = no login_processes_count = 3 login_process_size = 128 login_max_processes_count = 512 login_greeting = Ready #login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c
## Mailbox locations and namespaces mail_location = maildir:~/Maildir namespace private {
separator = .
prefix = INBOX.
inbox = yes} # Mail processes verbose_proctitle = yes first_valid_uid = 89 last_valid_uid = 89
# Maximum number of running mail processes. When this limit is reached, max_mail_processes = 200
# Set max. process size in megabytes. Most of the memory goes to mmap()ing # files, so it shouldn't harm much even if this limit is set pretty high. mail_process_size = 256
## Maildir-specific settings maildir_very_dirty_syncs = yes
## Authentication processes disable_plaintext_auth = yes
auth default { mechanisms = plain login digest-md5 cram-md5 passdb vpopmail { args = webmail=127.0.0.1 } userdb vpopmail { } user = vpopmail count = 1 ssl_require_client_cert = no } ################
thanks rajesh
On 23.06.2010 13:22, Rajesh M wrote:
did the tests once again on this large email box
First of all, are you using one client app or many? Make sure it's not the client issue. You should also enable some debug to see what commands are sent to IMAP server and what is the response and what is response time. For example, there's imap_debug option in Roundcube.
-- Aleksander 'A.L.E.C' Machniak http://alec.pl gg:2275252 LAN Management System Developer http://lms.org.pl Roundcube Webmail Developer http://roundcube.net
A.L.E.C put forth on 6/23/2010 6:51 AM:
On 23.06.2010 13:22, Rajesh M wrote:
did the tests once again on this large email box
First of all, are you using one client app or many? Make sure it's not the client issue. You should also enable some debug to see what commands are sent to IMAP server and what is the response and what is response time. For example, there's imap_debug option in Roundcube.
I think he's actually using Squirrelmail guys. I previously mentioned performance I get with Roundcube+Dovecot+mbox strictly as a comparison. That's how RC sneaked into the thread.
-- Stan
On 6/23/2010 12:27, Stan Hoeppner wrote:
A.L.E.C put forth on 6/23/2010 6:51 AM:
On 23.06.2010 13:22, Rajesh M wrote:
did the tests once again on this large email box
First of all, are you using one client app or many? Make sure it's not the client issue. You should also enable some debug to see what commands are sent to IMAP server and what is the response and what is response time. For example, there's imap_debug option in Roundcube.
I think he's actually using Squirrelmail guys. I previously mentioned performance I get with Roundcube+Dovecot+mbox strictly as a comparison. That's how RC sneaked into the thread.
When using SquirrelMail, make sure these settings are 'true':
Allow server thread sort : true Allow server-side sorting : true
Otherwise performance will suffer because it will do these things itself rather than letting the server do them.
~Seth
Seth Mattinen put forth on 6/23/2010 4:30 PM:
When using SquirrelMail, make sure these settings are 'true':
Allow server thread sort : true Allow server-side sorting : true
Otherwise performance will suffer because it will do these things itself rather than letting the server do them.
Is this still true if both SM and Dovecot are on the same box? I.e. is this a CPU bandwidth issue or a network bandwidth issue? If these settings solve a network b/w issue then they're irrelevant when both SM and Dovecot are on the same host.
-- Stan
On 6/23/2010 15:41, Stan Hoeppner wrote:
Seth Mattinen put forth on 6/23/2010 4:30 PM:
When using SquirrelMail, make sure these settings are 'true':
Allow server thread sort : true Allow server-side sorting : true
Otherwise performance will suffer because it will do these things itself rather than letting the server do them.
Is this still true if both SM and Dovecot are on the same box? I.e. is this a CPU bandwidth issue or a network bandwidth issue? If these settings solve a network b/w issue then they're irrelevant when both SM and Dovecot are on the same host.
It's more along the lines of SM will not use commands like SORT and attempt to do things itself because your server does not implement it. For example, I believe SurgeMail does not implement IMAP SORT so you'd have to set it to 'false'. Dovecot does, and is much better at executing SORT than having squirrel simulate it.
~Seth
On 6/23/2010 15:41, Stan Hoeppner wrote:
Seth Mattinen put forth on 6/23/2010 4:30 PM:
When using SquirrelMail, make sure these settings are 'true':
Allow server thread sort : true Allow server-side sorting : true
Otherwise performance will suffer because it will do these things itself rather than letting the server do them.
Is this still true if both SM and Dovecot are on the same box? I.e. is this a CPU bandwidth issue or a network bandwidth issue? If these settings solve a network b/w issue then they're irrelevant when both SM and Dovecot are on the same host.
Found it in the docs:
http://squirrelmail.org/docs/admin/admin-6.html#ss6.3
I would very strongly suggest that IMAP SORT be enabled if the server supports it, which of course Dovecot does. IMAP THREAD is the other one.
~Seth
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 23 Jun 2010, Rajesh M wrote:
a) Do you keep index files on Dovecot side?
b) Do you deliver / manipulate the mailbox via Dovecot only? E.g. do you use Dovecot deliver?
c) If you rawlog or sniff the connection and issue the same commands as roundcube to Dovecot via, say, telnet next morning, is the delay the same? http://wiki.dovecot.org/Debugging/Rawlog http://wiki.dovecot.org/TestInstallation?highlight=%28telnet%29
a) and b) should avoid rereading of the mailbox. c) is to ensure that Roundcube does not add the delay itself.
did the tests once again on this large email box
Your reply seems to answer a) as your conf does not contain INDEX=MEMORY, so I guess there are current index files. Can you verify there are?
What about b) and c)?
Do you have log entries like "UID inserted in the middle", "Invalid cache" or something? Is the mail storage located locally or remote / NFS / ... ?
Regards,
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iQEVAwUBTCH297+Vh58GPL/cAQIFdQf/fsARIYwP4zSAwLliGmklLU2D0QqRpH+W beiWgPcxC6Bp3xxkMOKUvtD6fzK6mSEFDZ3q0tlw4edhV2fYKCA2CvhgujTPKGWD 9muH2FGD8WJICqrFbqXCRkiuj7HKWFu7wuaEX7Wxz+YYZLiyELecAHVtOQ4Hx65T MqyRhYhE7J4Ec9LFwF+H2uwmMIL8aw/NpcEEJ8EQ9Iria0NfINZV/bshXAa11GHi A8NMLIrXi5kW2kKoKzbBJKTsXDuIqeIZFQZcVeGg0Y2Jhs5fYnv944MozpE2VikQ E3TsQEGtaIOCKznFurZCoCEMY3XMh/fkUv5b6Brdni5HOlRbAcirSw== =r/IY -----END PGP SIGNATURE-----
On 06/23/2010 01:22 PM Rajesh M wrote:
i set maildir_very_dirty_syncs = yes this was done in the morning and dovecot was restarted
i logged after about 4 hours after the previous login again it took around 5 mins to login
i was monitoring my server load which around 1.5 - 2 on my dual core dual xeon machine
this increased to around 8-9 during the login process
the no of emails in the box had increase by around 7500 emails
setting maildir_very_dirty_syncs = yes
does not seem to help
i have pasted below my dovecot.conf file
################
base_dir = /var/run/dovecot/ protocols = imap imaps log_path = /backup1/qmaillog/dovecot.log #ssl_disable = no ssl_cert_file = /var/qmail/control/servercert.pem ssl_key_file = /var/qmail/control/servercert.pem ssl_cipher_list = djdjjd verbose_ssl = yes
protocol imap { listen = *:143 ssl_listen = *:993 } ## Login processes #login_dir = /usr/local/var/run/dovecot/login login_user = dovecot login_process_per_connection = no login_processes_count = 3 login_process_size = 128 login_max_processes_count = 512 login_greeting = Ready #login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c
## Mailbox locations and namespaces mail_location = maildir:~/Maildir namespace private {
separator = . prefix = INBOX. inbox = yes} # Mail processes verbose_proctitle = yes first_valid_uid = 89 last_valid_uid = 89
# Maximum number of running mail processes. When this limit is reached, max_mail_processes = 200
# Set max. process size in megabytes. Most of the memory goes to mmap()ing # files, so it shouldn't harm much even if this limit is set pretty high. mail_process_size = 256
## Maildir-specific settings maildir_very_dirty_syncs = yes
## Authentication processes disable_plaintext_auth = yes
auth default { mechanisms = plain login digest-md5 cram-md5 passdb vpopmail { args = webmail=127.0.0.1 } userdb vpopmail { } user = vpopmail count = 1 ssl_require_client_cert = no } ################
thanks rajesh
Hm, I can't see a auth master socket in your pasted configuration. (BTW:
send only dovecot -n output.) So I guess, you let Qmail store the
messages into the Maildir.
If you would use deliver, Dovecot's LDA <http://wiki.dovecot.org/LDA>,
deliver would keep your index files up to date. This should reduce the
delay at login time.
Regards, Pascal
The trapper recommends today: deadbeef.1017413@localdomain.org
Pascal Volk wrote:
On 06/23/2010 01:22 PM Rajesh M wrote:
i set maildir_very_dirty_syncs = yes this was done in the morning and dovecot was restarted
i logged after about 4 hours after the previous login again it took around 5 mins to login
i was monitoring my server load which around 1.5 - 2 on my dual core dual xeon machine
this increased to around 8-9 during the login process
the no of emails in the box had increase by around 7500 emails
setting maildir_very_dirty_syncs = yes
does not seem to help
i have pasted below my dovecot.conf file
################
base_dir = /var/run/dovecot/ protocols = imap imaps log_path = /backup1/qmaillog/dovecot.log #ssl_disable = no ssl_cert_file = /var/qmail/control/servercert.pem ssl_key_file = /var/qmail/control/servercert.pem ssl_cipher_list = djdjjd verbose_ssl = yes
protocol imap { listen = *:143 ssl_listen = *:993 } ## Login processes #login_dir = /usr/local/var/run/dovecot/login login_user = dovecot login_process_per_connection = no login_processes_count = 3 login_process_size = 128 login_max_processes_count = 512 login_greeting = Ready #login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c
## Mailbox locations and namespaces mail_location = maildir:~/Maildir namespace private {
separator = . prefix = INBOX. inbox = yes} # Mail processes verbose_proctitle = yes first_valid_uid = 89 last_valid_uid = 89
# Maximum number of running mail processes. When this limit is reached, max_mail_processes = 200
# Set max. process size in megabytes. Most of the memory goes to mmap()ing # files, so it shouldn't harm much even if this limit is set pretty high. mail_process_size = 256
## Maildir-specific settings maildir_very_dirty_syncs = yes
## Authentication processes disable_plaintext_auth = yes
auth default { mechanisms = plain login digest-md5 cram-md5 passdb vpopmail { args = webmail=127.0.0.1 } userdb vpopmail { } user = vpopmail count = 1 ssl_require_client_cert = no } ################
thanks rajesh
Hm, I can't see a auth master socket in your pasted configuration. (BTW: send only
dovecot -noutput.) So I guess, you let Qmail store the messages into the Maildir. If you would use deliver, Dovecot's LDA <http://wiki.dovecot.org/LDA>, deliver would keep your index files up to date. This should reduce the delay at login time.Regards, Pascal
Not that it matters, but I think Pascal's correct about this.
Rajesh, can you give dovecot's 'deliver' a go? Let us know if you need a hand with that. Oh, and try to take good notes. ;)
-- -Eric 'shubes'
On 23/06/2010 12:22, Rajesh M wrote:
did the tests once again on this large email box i set maildir_very_dirty_syncs = yes this was done in the morning and dovecot was restarted
i logged after about 4 hours after the previous login again it took around 5 mins to login
i was monitoring my server load which around 1.5 - 2 on my dual core dual xeon machine
this increased to around 8-9 during the login process
the no of emails in the box had increase by around 7500 emails
When you ask Dovecot to open a mailbox I believe it builds an index of the key fields in all messages in the inbox. This means it needs to scan the list of all files in the mailbox to see if any new files are in the directory (so login to the server and time "time ls -al" just before using dovecot - this is a measure of the time the OS needs just to list all the files in the mailbox (I presume maildir, not mbox?)
Secondly 7,500 mails over 5 mins means an indexing rate of 25 mails/sec. This would not be out of order for a heavily fragmented drive which is IO bound? Each file needs to be opened to scan the headers so likely you need one disk seek and I guess it's easy to be IO bound? What does iotop show you during dovecot's thrashing?
Dovecot2 has an mdbox option which sounds like it could be beneficial for your performance requirements (but it's not "stable" yet)
Otherwise I guess you need to investigate dovecot's delivery agent which does incremental index updates at deliver time (pay the cost one email at a time rather than every 7,500 emails). Or consider alternative filesystems which are more performant for this requirement?
Good luck!
Ed W
Ed W put forth on 6/23/2010 4:18 PM:
Secondly 7,500 mails over 5 mins means an indexing rate of 25 mails/sec. This would not be out of order for a heavily fragmented drive which is IO bound? Each file needs to be opened to scan the headers so likely you need one disk seek and I guess it's easy to be IO bound? What does iotop show you during dovecot's thrashing?
It seems he is I/O bound a degree. If I read his answer to my disk subsystem question correctly he's storing user maildirs on a single local 1TB SATA drive. However, given that his 2nd successive login is 4-5 seconds instead of 5 minutes, it would seem index and cache being current are the problem, not I/O saturation. Faster disk would always help, but it's not close to a total solution to his problem.
Putting his maildir on a 16 disk RAID 0 stripe of the same model 1TB disk he already has would yield a 16x improvement in seek throughput, cutting his 'stale' login time to ~20 seconds, if my math is correct. 20 seconds is still unacceptably high IMHO, though it's much better than 300 seconds. Ok so lets assume the filesystem underlying his maildir is heavily fragmented and that defragging it would yield a 100% improvement for argument sake (50% improvement is almost unheard of, normal is about 20%). He'd still be looking at a 10 second login time after spending anywhere from $3k to $8k USD on a 16 disk array depending on what vendor he chooses. Throwing money and hardware at this problem isn't the proper or optimal solution.
Dovecot2 has an mdbox option which sounds like it could be beneficial for your performance requirements (but it's not "stable" yet)
I don't think migrating to a new mailbox format is what he really needs, or wants. IIRC he's been using qmail for years, which means he's been using maildir for years. He's likely very comfortable with maildir and probably wants to stick with it.
Otherwise I guess you need to investigate dovecot's delivery agent which does incremental index updates at deliver time (pay the cost one email at a time rather than every 7,500 emails). Or consider alternative filesystems which are more performant for this requirement?
On this I completely agree with you, and I suggested it previously in the same thread I suggested dirty_syncs. This should be his next step. If LDA alone doesn't fix the problem, then he should try dirty_syncs again in conjunction with LDA.
-- Stan
stan
you are right ...
i am running qmail toaster for years now and this is production mail server with over 5000 email boxes .. i dont want to migrate at all.
i recently upgraded to dovecot only for the imap part because courier was trashing my server .. i was getting load levels of over 30-40 during peak times
once courier was implemented my load levels were down to around 2-3
i am using squirrel mail and it has server side sort enabled
most users of mine have at the most around 100 to 200 mb of emails and just a few users having 3-4 gb boxes
i am absolutely happy with the performance of dovecot
a 3.8 gb mail box opens in 8-10 seconds the first time and next time in around 3 seconds
this 30 gb email box test is just to see how best dovecot can perform
my system is on centos with ext3 and is using 60 percent of drive space so i believe it shud not be fragmented
i also feel that this is related to "index and cache" which is getting expired or something like that.
i will try out what you suggested and revert
rajesh
Ed W put forth on 6/23/2010 4:18 PM:
Secondly 7,500 mails over 5 mins means an indexing rate of 25 mails/sec. This would not be out of order for a heavily fragmented drive which is IO bound? Each file needs to be opened to scan the headers so likely you need one disk seek and I guess it's easy to be IO bound? What does iotop show you during dovecot's thrashing?
It seems he is I/O bound a degree. If I read his answer to my disk subsystem question correctly he's storing user maildirs on a single local 1TB SATA drive. However, given that his 2nd successive login is 4-5 seconds instead of 5 minutes, it would seem index and cache being current are the problem, not I/O saturation. Faster disk would always help, but it's not close to a total solution to his problem.
Putting his maildir on a 16 disk RAID 0 stripe of the same model 1TB disk he already has would yield a 16x improvement in seek throughput, cutting his 'stale' login time to ~20 seconds, if my math is correct. 20 seconds is still unacceptably high IMHO, though it's much better than 300 seconds. Ok so lets assume the filesystem underlying his maildir is heavily fragmented and that defragging it would yield a 100% improvement for argument sake (50% improvement is almost unheard of, normal is about 20%). He'd still be looking at a 10 second login time after spending anywhere from $3k to $8k USD on a 16 disk array depending on what vendor he chooses. Throwing money and hardware at this problem isn't the proper or optimal solution.
Dovecot2 has an mdbox option which sounds like it could be beneficial for your performance requirements (but it's not "stable" yet)
I don't think migrating to a new mailbox format is what he really needs, or wants. IIRC he's been using qmail for years, which means he's been using maildir for years. He's likely very comfortable with maildir and probably wants to stick with it.
Otherwise I guess you need to investigate dovecot's delivery agent which does incremental index updates at deliver time (pay the cost one email at a time rather than every 7,500 emails). Or consider alternative filesystems which are more performant for this requirement?
On this I completely agree with you, and I suggested it previously in the same thread I suggested dirty_syncs. This should be his next step. If LDA alone doesn't fix the problem, then he should try dirty_syncs again in conjunction with LDA.
-- Stan
participants (8)
-
A.L.E.C
-
Ed W
-
Eric Shubert
-
Pascal Volk
-
Rajesh M
-
Seth Mattinen
-
Stan Hoeppner
-
Steffen Kaiser