[Dovecot] Is there a way to limit multiple POP3 connections?
Hi all,
Please correct me if I'm going about this the wrong way.
We currently have a 3 system Mail server set up - two machines are the load balanced frontends running Dovecot and Postfix, and the third machine is our NFS. Over the past 4 months or so, I've tweaked and optimized our mail servers, and we receive roughly 3 million emails a day combined through this setup.
Sometimes, the IO Wait is horrible on the load balanced servers - it can get up into the 80-90% range sometimes - it's usually when a lot of email is being passed, or there are a lot of POP3 and IMAP connections.
One item which I believe to be hurting I/O performance is the fact I used a RAID 10 setup using SATA II drives on an LSI Logic SATA I controller, in 32 bit mode. Until we rebuild our mail setup (which may be another year down the road), I want to make sure these servers are optimized to the fullest just in case we get heavy I/O usage, it doesn't slow down if at all possible.
One thing I noticed is that when issuing a 'ps aux | grep -i pop' or 'ps aux | grep -i imap' I see the same username 2,3,4,5 times, sometimes more. When issuing a 'tail -f /var/log/dovecot.log' I notice that some users log into POP3 every 3 seconds, sometimes sooner.
Without upsetting the customer, is there anything I can do on my end to disallow people logging into POP3 multiple times at once?
Any tips on this is very much appreciated. Thanks in advance.
Patrick
On Jan 9, 2008 5:58 PM, patrickm patrickm@garlic.com wrote:
Any tips on this is very much appreciated. Thanks in advance.
This is pretty easy to achieve with imap, some clients fork off a separate session. You can see this when you use a client, enter a folder then enter another folder before the first one is finished with its headers and so on.
If from your workstation you "while true; do fetchmail --silent --keep; done" (or similar) and just grind a single mailbox with a single process does it sooner than later "stall" if you run two or more does it?
I have "crash-tested" a few times maillooping the above over a few servers and it takes a long while to get impact from dovecot, especially if dovecot's lda is delivering the mail (and thus updating the indexes to be used by the pop/imap requests)
Other people might be interested in auth mechanism, mail storage type and index locations.
-- Gabriel Millerd
On Wed, 2008-01-09 at 15:58 -0800, patrickm wrote:
We currently have a 3 system Mail server set up - two machines are the load balanced frontends running Dovecot and Postfix, and the third machine is our NFS. Over the past 4 months or so, I've tweaked and optimized our mail servers, and we receive roughly 3 million emails a day combined through this setup.
Sometimes, the IO Wait is horrible on the load balanced servers - it can get up into the 80-90% range sometimes - it's usually when a lot of email is being passed, or there are a lot of POP3 and IMAP connections.
I updated http://wiki.dovecot.org/POP3Server. See maildir performance and session locking.
Timo Sirainen a écrit :
I updated http://wiki.dovecot.org/POP3Server. See maildir performance and session locking.
Why with mbox the performance is getting worse for users that don't keep mails on server ? Is it because dovecot does not delete the index entry when it delete a mail ? So index file is becoming huge ? Anyway if the index file is correctly indexed access time should still be very fast. And what about maildir format ?
-- Laurent PELLISSIER @ Ecole des Mines d'Alès
On Thu, 2008-01-10 at 11:17 +0100, Laurent PELLISSIER wrote:
Timo Sirainen a écrit :
I updated http://wiki.dovecot.org/POP3Server. See maildir performance and session locking.
Why with mbox the performance is getting worse for users that don't keep mails on server ?
That wasn't exactly the point. Updated the text, maybe this is clearer:
Index files are quite useless if your users don't keep mails in the server. They get first updated when the POP3 session starts to include all the messages, and after the user has deleted all the mails, they again get updated to contain zero mails. With this kind of a session the index reads and writes could have been avoided if the index files had just been completely disabled.
You may want to try how performance changes if you disable indexes for POP3 users. You can also try preserving indexes but try different values for mbox_min_index_size setting.
Is it because dovecot does not delete the index entry when it delete a mail ? So index file is becoming huge ?
Mails are deleted from indexes.
On Thursday 10 January 2008 05:59:27 Timo Sirainen wrote:
I updated http://wiki.dovecot.org/POP3Server. See maildir performance and session locking.
so according to this for POP3-only setups and v1.1 it is better to disable index files completely? Are there any drawbacks of disabling index files for POP3-only setups?
Regards, Arvids
On 1/10/2008, arvids (arvids@mpe.lv) wrote:
I updated http://wiki.dovecot.org/POP3Server. See maildir performance and session locking.
so according to this for POP3-only setups and v1.1 it is better to disable index files completely? Are there any drawbacks of disabling index files for POP3-only setups?
Thats not exactly what it says...
It says IF your users don't keep messages on the server (this has always been fairly unreliable with POP as far as my experience goes anyway) then you may want to disable them completely or use the mbox_min_index_size setting.
Maybe you should read it again... I know I often end up having to read these things many times before I can get my head around them - not becasue they are badly written, but because my head has a very strange shape and doesn't mold itself well to other shapes... ;).
--
Best regards,
Charles
On Thursday 10 January 2008 18:37:51 Charles Marcus wrote:
On 1/10/2008, arvids (arvids@mpe.lv) wrote:
I updated http://wiki.dovecot.org/POP3Server. See maildir performance and session locking.
so according to this for POP3-only setups and v1.1 it is better to disable index files completely? Are there any drawbacks of disabling index files for POP3-only setups?
Thats not exactly what it says...
It says IF your users don't keep messages on the server (this has always been fairly unreliable with POP as far as my experience goes anyway) then you may want to disable them completely or use the mbox_min_index_size setting.
Maybe you should read it again... I know I often end up having to read these things many times before I can get my head around them - not becasue they are badly written, but because my head has a very strange shape and doesn't mold itself well to other shapes... ;).
I mean 'Maildir Performance' section, not 'mbox Performance' - according to it all that POP3 needs is in dovecot-uidlist file(with v.1.1), so I assume that index files are not needed at all for POP3-only and v1.1(no matter if users keep their mails on server or not), and I just asked if I am right, because I am not completely sure about it. So maybe you also should read it again :)
Regards, Arvids
On 1/10/2008, arvids (arvids@mpe.lv) wrote:
So maybe you also should read it again :)
Ok... ... done. ;)
I mean 'Maildir Performance' section, not 'mbox Performance' - according to it all that POP3 needs is in dovecot-uidlist file(with v.1.1), so I assume that index files are not needed at all for POP3-only and v1.1(no matter if users keep their mails on server or not), and I just asked if I am right, because I am not completely sure about it.
Ok, I guess we both may need to read it a few more time - not surprising in my case... :)
It doesn't say you don't need the index files - it says you don't need the dovecot.index.CACHE file...
According to this they are not the same:
http://wiki.dovecot.org/IndexFiles
--
Best regards,
Charles
On Thursday 10 January 2008 20:08:43 Charles Marcus wrote:
Ok, I guess we both may need to read it a few more time - not surprising in my case... :) OK, done :)
It doesn't say you don't need the index files - it says you don't need the dovecot.index.CACHE file...
According to this they are not the same:
If we disable index files (INDEX=MEMORY), indexing is not switched off, just all structures are kept only in memory and index.cache file is not created. And http://wiki.dovecot.org/POP3Server says that: 'for POP3-only users there's no need to create dovecot.index.cache file at all'. So from this I conclude that if we do not want index.cache file created we have to disable index files with INDEX=MEMORY. We are disabling index files on disk, not indexing.
Regards, Arvids
On Thu, 2008-01-10 at 20:30 +0200, arvids wrote:
On Thursday 10 January 2008 20:08:43 Charles Marcus wrote:
Ok, I guess we both may need to read it a few more time - not surprising in my case... :) OK, done :)
It doesn't say you don't need the index files - it says you don't need the dovecot.index.CACHE file...
According to this they are not the same:
If we disable index files (INDEX=MEMORY), indexing is not switched off, just all structures are kept only in memory and index.cache file is not created. And http://wiki.dovecot.org/POP3Server says that: 'for POP3-only users there's no need to create dovecot.index.cache file at all'. So from this I conclude that if we do not want index.cache file created we have to disable index files with INDEX=MEMORY. We are disabling index files on disk, not indexing.
The point of that was that v1.1 just doesn't create/update dovecot.index.cache files for POP3-only users. There's no need to configure it that way. I'll try to update the wiki to say that. :)
dovecot.index and dovecot.index.log files still get created. I don't know how much of a difference it makes if you disable them instead of just letting them be updated.
On Friday 11 January 2008 06:02:12 Timo Sirainen wrote:
The point of that was that v1.1 just doesn't create/update dovecot.index.cache files for POP3-only users. There's no need to configure it that way. I'll try to update the wiki to say that. :)
then it seems that there is some problem - dovecot.index.cache files are created/updated for all users, even with configuration where IMAP is completely disabled(I am using v1.1b10 and v1.1b12).
Regards, Arvids
On 11.1.2008, at 15.30, arvids wrote:
On Friday 11 January 2008 06:02:12 Timo Sirainen wrote:
The point of that was that v1.1 just doesn't create/update dovecot.index.cache files for POP3-only users. There's no need to configure it that way. I'll try to update the wiki to say that. :)
then it seems that there is some problem - dovecot.index.cache
files are created/updated for all users, even with configuration where IMAP is completely disabled(I am using v1.1b10 and v1.1b12).
Hmm. Not here, at least with latest hg..
export MAIL=~/Maildir rm -f ~/Maildir/dovecot* printf "list\ntop 1\nretr 1\ndele 1\nquit\n" | ./pop3 ls ~/Maildir/dovecot* /home/tss/Maildir/dovecot.index.log /home/tss/Maildir/dovecot-uidlist
If you do "idxview ~/Maildir" (idxview should be in libexec/dovecot/
directory) what cached fields does it show for messages?
On Friday 11 January 2008 15:37:51 Timo Sirainen wrote:
Hmm. Not here, at least with latest hg..
export MAIL=~/Maildir rm -f ~/Maildir/dovecot* printf "list\ntop 1\nretr 1\ndele 1\nquit\n" | ./pop3 ls ~/Maildir/dovecot* /home/tss/Maildir/dovecot.index.log /home/tss/Maildir/dovecot-uidlist
If you do "idxview ~/Maildir" (idxview should be in libexec/dovecot/ directory) what cached fields does it show for messages?
just tested with the latest hg(740a17139b67) - removed the whole mail directory, then sent two mails, then retrieved them with POP3, then repeated sending and retrieving. After each step cache file was updated(both after deliver and POP3 session). Here are cached fields from idxview output after the test: -- Cache fields -- # Name Type Size Dec Last used 0: flags bit 4 no 1970-01-01 03:00 1: date.sent fix 8 no 1970-01-01 03:00 2: date.received fix 4 no 1970-01-01 03:00 3: date.save fix 4 no 1970-01-01 03:00 4: size.virtual fix 8 tmp 2008-01-11 16:11 5: size.physical fix 8 tmp 2008-01-11 16:11 6: imap.body str - no 1970-01-01 03:00 7: imap.bodystructure str - no 1970-01-01 03:00 8: imap.envelope str - no! 1970-01-01 03:00 9: mime.parts var - no 1970-01-01 03:00
here is dovecot -n output: # 1.1.beta13: /usr/local/dovecot/etc/dovecot.conf base_dir: /var/run/dovecot/ syslog_facility: local0 protocols: pop3 ssl_disable: yes disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable: /usr/local/dovecot/libexec/dovecot/pop3-login login_greeting: Server. login_process_per_connection: no login_processes_count: 5 login_max_processes_count: 32 verbose_proctitle: yes first_valid_uid: 95 first_valid_gid: 95 mail_uid: 95 mail_gid: 95 mail_location: maildir:~/Maildir fsync_disable: yes mail_executable: /usr/local/dovecot/libexec/dovecot/pop3 mail_plugins: quota mail_plugin_dir: /usr/local/dovecot/lib/dovecot/pop3 pop3_enable_last: yes pop3_client_workarounds: outlook-no-nuls oe-ns-eoh auth default: mechanisms: plain login digest-md5 cram-md5 apop cache_size: 1024 cache_ttl: 600 cache_negative_ttl: 60 user: doveauth verbose: yes passdb: driver: sql args: /usr/local/dovecot/etc/dovecot-sql.conf userdb: driver: prefetch userdb: driver: sql args: /usr/local/dovecot/etc/dovecot-sql.conf socket: type: listen master: path: /var/run/dovecot/auth-master mode: 384 user: vmail group: vmail plugin: quota: dict:::proxy:/var/run/dovecot/dict-server:quotadict dict: quotadict: mysql:/usr/local/dovecot/etc/dovecot-sql-dict-quota.conf
Regards, Arvids
On 11.1.2008, at 16.23, arvids wrote:
just tested with the latest hg(740a17139b67) - removed the whole
mail directory, then sent two mails, then retrieved them with POP3, then repeated
sending and retrieving. After each step cache file was updated(both after
deliver and POP3 session). Here are cached fields from idxview output after the test: . 5: size.physical fix 8 tmp
2008-01-11 16:11 .. mail_plugins: quota
OK, quota is causing it. I'll see what to do about this..
On Fri, 2008-01-11 at 16:28 +0200, Timo Sirainen wrote:
On 11.1.2008, at 16.23, arvids wrote:
just tested with the latest hg(740a17139b67) - removed the whole
mail directory, then sent two mails, then retrieved them with POP3, then repeated
sending and retrieving. After each step cache file was updated(both after
deliver and POP3 session). Here are cached fields from idxview output after the test: . 5: size.physical fix 8 tmp
2008-01-11 16:11 .. mail_plugins: quotaOK, quota is causing it. I'll see what to do about this..
on 1/10/2008 8:37 AM Charles Marcus spake the following:
On 1/10/2008, arvids (arvids@mpe.lv) wrote:
I updated http://wiki.dovecot.org/POP3Server. See maildir performance and session locking.
so according to this for POP3-only setups and v1.1 it is better to disable index files completely? Are there any drawbacks of disabling index files for POP3-only setups?
Thats not exactly what it says...
It says IF your users don't keep messages on the server (this has always been fairly unreliable with POP as far as my experience goes anyway) then you may want to disable them completely or use the mbox_min_index_size setting.
Maybe you should read it again... I know I often end up having to read these things many times before I can get my head around them - not because they are badly written, but because my head has a very strange shape and doesn't mold itself well to other shapes... ;).
Then you are not banging it hard enough on the other shape! ;-P
-- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't!!!!
participants (7)
-
arvids
-
Charles Marcus
-
Gabriel Millerd
-
Laurent PELLISSIER
-
patrickm
-
Scott Silva
-
Timo Sirainen