- dovecot - dovecot.org

How to dsync mdbox compressed to maildir uncompressed
by Daniel Betz 21 Feb '17

21 Feb '17

Hello, we are using doveadm sync to export mdbox to maildir format, so we can use an external tool to convert into an pst file. Since we have enabled zlib compression doveadm sync always exports the maildir gzip compressed. Are there any ways to prevent the doveadm sync to export the maildir compressed ? Have tried this: doveadm -o "maildir_copy_with_hardlinks=no" sync -u imap(a)test.de maildir:~/Maildir >From Wiki: If you want to use dsync to convert to a compressed Maildir you may need -o maildir_copy_with_hardlinks=no (this is set to yes by default and will prevent compression). Regards, Daniel # 2.2.27 (c0f36b0): /usr/local/dovecot2/etc/dovecot/dovecot.conf doveconf: Warning: service auth { client_limit=60000 } is lower than required under max. load (500500) # OS: Linux 3.10.0-327.36.3.el7.x86_64 x86_64 CentOS Linux release 7.2.1511 (Core) auth_cache_negative_ttl = 1 mins auth_cache_size = 64 M auth_cache_ttl = 2 hours auth_mechanisms = plain login auth_username_chars = base_dir = /var/run/dovecot/ debug_log_path = /dev/null default_login_user = dovecot default_vsz_limit = 750 M disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it doveadm_port = 12345 first_valid_gid = 1001 first_valid_uid = 1001 info_log_path = /var/log/dovecot/messages lda_mailbox_autocreate = yes lda_original_recipient_header = X-Envelope-To log_path = /dev/stderr login_log_format_elements = user=[%u] method=%m rip=%r lip=%l %c mail_gid = 1001 mail_location = mdbox:~:INDEX=%h/INDEX mail_plugins = quota notify mail_log zlib mail_uid = 1001 mbox_write_locks = fcntl namespace { inbox = yes location = mailbox Drafts { auto = no special_use = \Drafts } mailbox "Gesendete Elemente" { auto = no special_use = \Sent } mailbox "Infizierte Objekte" { auto = no special_use = \Junk } mailbox Sent { auto = no special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Spam { auto = no special_use = \Junk } mailbox Trash { auto = no special_use = \Trash } prefix = separator = . type = private } namespace inbox { hidden = yes inbox = no list = no location = prefix = INBOX. separator = . } passdb { args = /usr/local/dovecot2/etc/dovecot/dovecot-ldap.conf driver = ldap } passdb { args = /usr/local/dovecot2/etc/dovecot/dovecot-ldap2.conf driver = ldap } plugin { quota = dict:User quota::file:%h/mdbox/dovecot-quota quota_rule1 = Trash:storage=+100M quota_rule2 = INBOX.Trash:storage=+100M quota_warning = storage=85%% quota-warning 85 %u quota_warning1 = storage=95%% quota-warning 95 %u quota_warning2 = storage=99%% quota-warning 99 %u zlib_save = gz zlib_save_level = 6 } replication_max_conns = 30 sendmail_path = /usr/local/exim/bin/exim service aggregator { fifo_listener replication-notify-fifo { mode = 0666 user = popuser } unix_listener replication-notify { mode = 0666 user = popuser } } service anvil { client_limit = 60000 } service auth { client_limit = 60000 unix_listener auth-userdb { mode = 0666 user = popuser } unix_listener auth { mode = 0666 user = popuser } } service config { unix_listener config { user = popuser } } service dict { unix_listener dict { mode = 0666 user = popuser } } service dns_client { process_limit = 6000 process_min_avail = 12 unix_listener dns-client { mode = 0666 user = popuser } } service doveadm { inet_listener { port = 12345 } user = popuser } service imap-login { chroot = login client_limit = 6000 process_limit = 100 process_min_avail = 16 service_count = 0 } service imap { executable = /usr/local/dovecot2/libexec/dovecot/imap process_limit = 250000 process_min_avail = 50 service_count = 250 } service ipc { client_limit = 60000 unix_listener ipc { mode = 0650 user = dovecot } unix_listener login/ipc-proxy { mode = 0650 user = dovecot } } service lmtp { unix_listener lmtp { mode = 0666 user = popuser } } service pop3-login { chroot = login client_limit = 6000 process_limit = 100 process_min_avail = 16 service_count = 0 } service pop3 { executable = /usr/local/dovecot2/libexec/dovecot/pop3 process_limit = 250000 process_min_avail = 50 service_count = 250 } service quota-warning { executable = script /usr/local/dovecot2/bin/quota-warning.sh unix_listener quota-warning { mode = 0600 user = popuser } user = popuser } service replicator { unix_listener replicator-doveadm { mode = 0600 user = popuser } } ssl_cert = </usr/local/dovecot2/etc/dovecot/certs/cert1.crt ssl_dh_parameters_length = 2048 ssl_key = # hidden, use -P to show it ssl_protocols = TLSv1.2 TLSv1.1 TLSv1 !SSLv3 !SSLv2 userdb { driver = prefetch } userdb { args = /usr/local/dovecot2/etc/dovecot/dovecot-ldap.conf driver = ldap } verbose_proctitle = yes protocol doveadm { mail_plugin_dir = /usr/local/dovecot2/lib/dovecot/ passdb { args = /usr/local/dovecot2/etc/dovecot/dovecot-doveadm.conf driver = ldap name = } userdb { args = /usr/local/dovecot2/etc/dovecot/dovecot-doveadm.conf driver = ldap name = } } protocol lda { info_log_path = /var/log/dovecot-lda log_path = /var/log/dovecot-lda log_timestamp = "%b %d %H:%M:%S " mail_plugin_dir = /usr/local/dovecot2/lib/dovecot/ postmaster_address = %u } protocol imap { imap_client_workarounds = mail_max_userip_connections = 100 mail_plugin_dir = /usr/local/dovecot2/lib/dovecot/ mail_plugins = quota notify mail_log zlib imap_quota imap_zlib } protocol pop3 { mail_plugin_dir = /usr/local/dovecot2/lib/dovecot/ pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_enable_last = yes pop3_fast_size_lookups = yes pop3_reuse_xuidl = no pop3_uidl_format = %g }

3 2

IMAP directory structure.
by C5ace 21 Feb '17

21 Feb '17

Hello, I use Dovecot 2.1.7. and like to know how to force the IMAP directory structure to be: The IMAP server's desired directory structure is: root@server-2:/var/vmail/c5ace.com/test/Maildir# /.INBOX /.INBOX.Archives /.INBOX.Drafts /.INBOX.Junk /.INBOX.Sent /.INBOX.Templates /.INBOX.Trash and to prevent the mail clients like Thunderbird, Claws Mail, etc. to add additional out of tree directories. Thanks in advance. Elmar

2 1

v2.2.28 release candidate released
by Timo Sirainen 21 Feb '17

21 Feb '17

http://dovecot.org/releases/2.2/rc/dovecot-2.2.28.rc1.tar.gz http://dovecot.org/releases/2.2/rc/dovecot-2.2.28.rc1.tar.gz.sig Pretty large release. Please test before the final v2.2.28, which should be out in a few days. BTW. Our plan is to start making new releases approximately every month from now on. * director: "doveadm director move" to same host now refreshes user's timeout. This allows keeping user constantly in the same backend by just periodically moving the user there. * When new mailbox is created, use initially INBOX's dovecot.index.cache caching decisions. * Expunging mails writes GUID to dovecot.index.log now only if the GUID is quickly available from index/cache. * imap: When BINARY FETCH sees invalid content, return NO [PARSE] reply instead of [UNKNOWNCTE] (which is now used only for actually unknown Content-Transfer-Encoding headers). * pop3c: Increase timeout for PASS command to 5 minutes. * Mail access errors are no longer ignored when searching or sorting. With IMAP the untagged SEARCH/SORT reply is still sent the same as before, but NO reply is returned instead of OK. + Make dovecot.list.index's filename configurable. This is needed when there are multiple namespaces pointing to the same mail root (e.g. lazy_expunge namespace for mdbox). + Add size.virtual to dovecot.index when folder vsizes are accessed (e.g. quota=count). This is mainly a workaround to avoid slow quota recalculation performance when message sizes get lost from dovecot.index.cache due to corruption or some other reason. + auth: Support OAUTHBEARER and XOAUTH2 mechanisms. Also support them in lib-dsasl for client side. + auth: Support filtering by SASL mechanism: passdb { mechanisms } + Shrink the mail processes' memory usage by not storing settings duplicated unnecessarily many times. + imap: Add imap_fetch_failure setting to control what happens when FETCH fails for some mails (see example-config). + imap: Include info about last command in disconnection log line. + imap: Created new SEARCH=X-MIMEPART extension. It's currently not advertised by default, since it's not fully implemented. + fts-solr: Add support for basic authentication. + Cassandra: Support automatically retrying failed queries if execution_retry_interval and execution_retry_times are set. + doveadm: Added "mailbox path" command. + mail_log plugin: If plugin { mail_log_cached_only=yes }, log the wanted fields only if it doesn't require opening the email. + mail_vsize_bg_after_count setting added (see example-config). + mail_sort_max_read_count setting added (see example-config). - Index files: day_first_uid wasn't updated correctly since v2.2.26. This caused dovecot.index.cache to be non-optimal. - imap: SEARCH/SORT may have assert-crashed in client_check_command_hangs - imap: FETCH X-MAILBOX may have assert-crashed in virtual mailboxes. - imap: Running time in tagged command reply was often wrongly 0. - search: Using NOT n:* or NOT UID n:* wasn't handled correctly - director: doveadm director kick was broken - director: Fix crash when using director_flush_socket - director: Fix some bugs when moving users between backends - imapc: Various error handling fixes and improvements - master: doveadm process status output had a lot of duplicates. - autoexpunge: If mailbox's rename timestamp is newer than mail's save-timestamp, use it instead. This is useful when autoexpunging e.g. Trash/* and an entire mailbox is deleted by renaming it under Trash to prevent it from being autoexpunged too early. - autoexpunge: Multiple processes may have been trying to expunge the same mails simultaneously. This was problematic especially with lazy_expunge plugin. - auth: %{passdb:*} was empty in auth-worker processes - auth-policy: hashed_password was always sent empty. - dict-sql: Merge multiple UPDATEs to a single statement if possible. - fts-solr: Escape {} chars when sending queries - fts: fts_autoexpunge_exclude = \Special-use caused crashes - doveadm-server: Fix leaks and other problems when process is reused for multiple requests (service_count != 1) - sdbox: Fix assert-crash on mailbox create race - lda/lmtp: deliver_log_format values weren't entirely correct if Sieve was used. especially %{storage_id} was broken.

8 10

Released Pigeonhole v0.4.17.rc1 for Dovecot v2.2.28.rc1.
by Stephan Bosch 21 Feb '17

21 Feb '17

Hello Dovecot users, Several bugs were found in the course of the last few months, but no new features were created. So, this will be strictly a bug-fixing release. The previous Pigeonhole release should still work with the new Dovecot release, which means you are not required to upgrade. However, my advice would be to upgrade once this release candidate matures into a final release, since at least the first bug listed below is pretty severe. Changelog v0.4.17: - LDA Sieve plugin: Fixed handling of an early explicit keep during multiscript execution. Action side-effects and the message snapshot would be lost at the final stage where the implicit keep is evaluated. This could result in the IMAP flags assigned to the message to be forgotten or that headers modified by the "editheader" extension would revert to their original state. - file script storage: Amended the up-to-date time stamp comparison for on-disk binaries to include nanoseconds. This will fix problems occurring when both binary and script are saved within the same second. This fix is ineffective on older systems that have no support for nanoseconds in stat() timestamps, which should be pretty rare nowadays. - file script storage: Improve saving and listing permission error to include more details. - imapsieve plugin: Make sure "INBOX" is upper case in static mailbox rules. Otherwise, the mailbox name would never match, since matching is performed case-sensitively and Dovecot only returns the upper-cased "INBOX". - imapsieve plugin: Fixed assert failure occurring when used with virtual mailboxes. - doveadm sieve plugin: Fixed crash when setting Sieve script via attribute's string value. The release is available as follows: http://pigeonhole.dovecot.org/releases/2.2/rc/dovecot-2.2-pigeonhole-0.4.17… http://pigeonhole.dovecot.org/releases/2.2/rc/dovecot-2.2-pigeonhole-0.4.17… Refer to http://pigeonhole.dovecot.org and the Dovecot v2.x wiki for more information. Have fun testing this release candidate and don't hesitate to notify me when there are any problems. Regards, -- Stephan Bosch stephan(a)rename-it.nl

3 2

Sieve not filtering
by Ben 20 Feb '17

20 Feb '17

Hi, I have copied accross a known-good sieve file from a working server and its not filtering. Everything just gets chucked into INBOX. doveconf-n at the bottom of this mail Feb 17 16:05:20 server postfix/smtpd[51562]: 7FA5E12CBBC: client=unknown[192.168.167.57] Feb 17 16:05:23 server postfix/cleanup[51565]: 7FA5E12CBBC: message-id=<> Feb 17 16:05:23 server postfix/qmgr[45471]: 7FA5E12CBBC: from=<sender(a)example.com>, size=182, nrcpt=1 (queue active) Feb 17 16:05:23 server dovecot: lmtp(51467): Connect from local Feb 17 16:05:23 server dovecot: auth-worker(51568): passwd(recipient(a)example.com): unknown user Feb 17 16:05:23 server dovecot: lmtp(51467, recipient(a)example.com) 1JK2B0Mfp1gLyQAAHLpRfg: sieve: msgid=unspecified: stored mail into mailbox 'INBOX' Feb 17 16:05:23 server dovecot: lmtp(51467): Disconnect from local: Successful quit Feb 17 16:05:23 server postfix/lmtp[51566]: 7FA5E12CBBC: to=<recipient+something(a)example.com>, orig_to=<recipient+something(a)example.com>, relay=server.example.com[private/dovecot-lmtp], delay=4.9, delays=4.9/0.01/0/0.05, dsn=2.0.0, status=sent (250 2.0.0 <bp+nanog(a)example.com> 1JK2B0Mfp1gLyQAAHLpRfg Saved) Feb 17 16:05:23 server postfix/qmgr[45471]: 7FA5E12CBBC: removed This is the syntax I'm using in my sieve file: require ["fileinto","envelope"]; if anyof (address :is :all "to" ["recipient+something@example.com”,”mailing_list@example.com”], header :contains ["Cc", "Delivered-To"] ["recipient+something@example.com”,”mailing_list@example.com”]) { fileinto “THIS_FOLDER”; stop; } # 2.2.10: /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-514.6.1.el7.x86_64 x86_64 CentOS Linux release 7.3.1611 (Core) auth_mechanisms = plain login auth_verbose = yes auth_verbose_passwords = sha1 first_valid_uid = 1000 mail_location = maildir:~/Maildir managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body environment mailbox date ihave enotify mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } passdb { driver = pam } passdb { args = scheme=CRYPT username_format=%u /etc/dovecot/users driver = passwd-file } plugin { sieve = ~/.dovecot.sieve } protocols = imap lmtp service auth { unix_listener /var/spool/postfix/private/dovecot-auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = its_virtmail mode = 0660 user = its_virtmail } } service imap-login { process_min_avail = 3 } service lmtp { process_min_avail = 5 unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } user = its_virtmail } service managesieve-login { inet_listener sieve { port = 4190 } inet_listener sieves { address = port = 5190 ssl = yes } } ssl = required ssl_cert = <REMOVED ssl_cipher_list = REMOVED ssl_dh_parameters_length = 2048 ssl_key = </REMOVED ssl_prefer_server_ciphers = yes ssl_protocols = !SSlv2 !SSLv3 userdb { driver = passwd } userdb { args = username_format=%u /etc/dovecot/users driver = passwd-file } protocol lmtp { mail_plugins = sieve postmaster_address = me(a)example.com } protocol lda { deliver_log_format = msgid=%m: %$ mail_plugins = sieve postmaster_address = postmaster quota_full_tempfail = yes rejection_reason = Your message to <%t> was automatically rejected:%n%r } protocol imap { imap_client_workarounds = delay-newmail mail_max_userip_connections = 20 }

5 5

Issue connecting to dovecot from remote machine
by Antoine Sirven 20 Feb '17

20 Feb '17

Hi, I've set up a postfix +dovecot configuration on my debian jessie. But I have a connection issue. When I try to connect from thunderbird it doesn not work. When I check out my debug logs I get : auth-worker(22252): Info: pam(myuser,hostIP): pam_authenticate() failed: Authentication failure (password mismatch?) (given password: correctPassword) Running doveadm auth test tells me I can authenticate with the same password. So I tried connecting via openssl. When I connect from my local host, everything goes fine : * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN AUTH=CRAM-MD5] Dovecot ready. a login myUser myPassword a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE] Logged in But when I run the same thing from my remote laptop, I never get any reply from the imap server. It's as if the request wasn't reaching the server. Although I do get a first reply from the server, I just can't log in. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN AUTH=CRAM-MD5] Dovecot ready. a login myUser myPassword (... and then nothing...) The port is properly open when I run nmap. I really don't get what's wrong. Any of you has any idea? Here are my versions & conf : # dovecot -n # 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-042stab120.16 x86_64 Debian 8.7 auth_debug = yes auth_debug_passwords = yes auth_mechanisms = plain login cram-md5 auth_verbose = yes auth_verbose_passwords = yes disable_plaintext_auth = no info_log_path = /var/log/maildebug.log mail_debug = yes mail_location = mbox:~/mail:INBOX=/var/mail/%u mail_privileged_group = mail namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = failure_show_msg=yes session=yes dovecot driver = pam } passdb { args = username_format=%u /etc/dovecot/users driver = passwd-file } protocols = " imap" service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { group = postfix mode = 0666 user = postfix } } service imap-login { inet_listener imap { port = 0 } inet_listener imaps { port = 993 ssl = yes } } service pop3-login { inet_listener pop3 { port = 0 } inet_listener pop3s { port = 0 } } ssl = required ssl_cert = </etc/ssl/certs/mailcert.pem ssl_key = </etc/ssl/private/mail.key userdb { driver = passwd } userdb { args = username_format=%u /etc/dovecot/users driver = passwd-file } verbose_ssl = yes Thanks a lot

2 2

Re: Upgrade from 1.2 to 2.2
by Joseph Tam 16 Feb '17

16 Feb '17

Eduardo M KALINOWSKI writes in reesponse to Gandalf Corvotempesta post: >> I've already read that, and as wrote previously, everything broke down. >> dovecont -n wasn't able to convert the configuration file and dovecot >> wasn't started properly. >> The only way to fix was to downgrade. >> >> As this is a production server, I would like to avoid this kind of issue..... > > Set up a server replicating exactly the configuration you have at > present. Doesn't need to be anything fancy, a virtual machine is > enough. Actually, a virtual machine is a great tool if you can make > snapshots and then rollback to known states if any attempt at a change > fails. If you absolutely can't test on a VM or another test host, you can run the new version along side your current version in another installation directory. Configure your new installation to run on alternate ports. You can work off copies of mail data as well if you really want to isolate your test. Then you can gradually phase in your new version by swapping alternate port/data with production port/data. Joseph Tam <jtam.home(a)gmail.com>

1 0

doveadm backup and quota
by florent＠coppint.com 15 Feb '17

15 Feb '17

Hi, I'm looking to convert all mailboxes from Maildir format to mdbox. When I try my procedure on a user having ~45GB of messages, its quota is growing very fast and is now up to 66GB. I run this command : doveadm -v backup -u user(a)domain.com mdbox:/mnt/maildata1/domain.com/user/mdbox I expected this command to make a copy of user mailbox, it does not make sense to count the backup in its quota. Maybe quota is simply recalculated ? But running a "du -sh" on its home directory is showing a 46G disk usage ! (backup is not stored in home dir) Can someone tell me what happens on my setup ? I use Dovecot 2.2.24. Thank you. Florent

1 1

Upgrade from 1.2 to 2.2
by Gandalf Corvotempesta 15 Feb '17

15 Feb '17

Hi, I have a production server running Debian Squeeze with Dovecot 1.2 I would like to upgrade everything to Jessie, running 2.2 Last time I did something similiar, but from Lenny to Squeeze, the whole dovecot installation broke. Any suggestion on how to upgrade everything ? Can I test our current configuration with a newer dovecot version to be sure that everything would be converted properly with "doveconf -n -c /etc/dovecot/dovecot.conf > dovecot-2.conf" ? Thank you

4 6

Re: dovecot config for 1500 simultaneous connection
by Rajesh M 15 Feb '17

15 Feb '17

christian the servers i currently own are dell servers. The servers i plan to buy are Dell R530, 2U rack servers with 8 x 3.5 inch drives, with 64 gb ram each, Hardware raid. I am thinking of 2 X 300 gb ssds raid1 and 6 x 2 tb drives in raid10 for data. I do not have any experience in setting up drdb (that would be my next step) ... primarily using standalone servers with hardware level redundancy. For backup-ups i rsync data to backup servers, also use qmail taps for backing up sent/recd emails on a per user basis. i am using mysql authentication backend my mail server is qmailtoaster, ie with vpopmail, spamdyke, simscan. you had mentioned as follows ############## With 4000 users and 5 logins/s I can't imagine anything that wouldn't be able to: a) put all of the relevant data into RAM and b) thus handle auth requests in very timely manner. The example I gave earlier about my servers with 140 logins/s authenticates against LDAP and isn't very busy doing so while also fitting all 2.7GB of data for about 1 million users into RAM. ############## >>>>>>>>>>>>>>>>>>how do i put all relevant data into the RAM ? thanks rajesh ----- Original Message ----- From: Christian Balzer [mailto:chibi@gol.com] To: dovecot(a)dovecot.org Sent: Tue, 14 Feb 2017 16:40:01 +0900 Subject: Hello, On Mon, 13 Feb 2017 22:53:23 +0530 Rajesh M wrote: > thanks for your help > > happy to say that the performance dramatically improved after i use the high performance settings from here > http://wiki.dovecot.org/LoginProcess > That's why that page is there. > grep Login: /var/log/mail.log.1 |wc -l > with the mail.log being of a typical, busy day. > 412992 > So about 5 login/s, not that busy, but that of course also depends on your HW and authentication backend. > i also picked up the imap and pop3 connections during peak hours > [root@ns1 domains]# doveadm who | awk '/imap/{m+=$2}/pop3/{n+=$2}END{print m,n}' > username # proto (pids) (ips) > > max figures i got was 535 for imap and 97 for pop > > > Question 1 > i wish to improve the performance further by caching the logins. current the same is kept disable because when user's change passwords then they are not able to immediately login with the new password for some time. How to solve this issue. > There is no solution, caching auth data is by default fraught with that sort of peril. And how did you determine that caching auth data would actually improve things? What is your authentication backend? With 4000 users and 5 logins/s I can't imagine anything that wouldn't be able to: a) put all of the relevant data into RAM and b) thus handle auth requests in very timely manner. The example I gave earlier about my servers with 140 logins/s authenticates against LDAP and isn't very busy doing so while also fitting all 2.7GB of data for about 1 million users into RAM. > > > further to the above. > i am planning to put up a large mail server with one of the following options > Don't plan 1, always plan a pair and then use whatever you're comfortable with to turn them into a HA cluster, like Pacemaker with DRBD. > option 1 > OS and mailqueue : 2 X 600 gb 15k rpm raid 1 What kind of RAID, HW or MD? > data : 4 X 2000 gb raid 10 for data > backup : 2 X 2000 gb raid 10 for backup While with Linux MD a 2 disk RAID10 would be possible, it wouldn't be particular helpful (slower than a RAID 1 with writes). > 32 gb ram > > > option 2 > os and mail queue : 4 X 600 gb 15k rpm raid 10 Those 15k HDDs tend to be expensive. Estimate your expected writes per day (iostat on an existing server can give you some hints) and get 2 SSDs instead, DC S3520s if 1 DWPD is sufficient, DC S3610s if you need 3 DWPD (doubtful), S3710 if you need 10 DWPD (very doubtful). Similar for Samsung DC level SSDs, check their specs. For example, my busiest mailbox servers only write about 250KB/s averaged, which mean about 50GB/day and thus would be a fit even for a small S3520. I'm still using 200GB S3710s because I can afford them, like the speed and never ever want to worry about wearout. > data : 4 X 2000 gb raid 10 for data > 32 gb ram > > i am having the OS and mail queue on primary drives > > i understand that Raid10 give a faster write and read whereas Raid 1 will have slow writes and fast reads > While that is true, you're missing the real difference here, the number of HDDs. With 4 of them in the RAID10 they will be twice as fast as the 2 disk RAID1. This can be tweaked even further with the various RAID 10 layout options, "man md". > the question is will there be huge performance difference between Raid1 and Raid 10 when it comes small files like mail queue ? > Again as explained above, double the IOPS, so yes. Christian > > thanks > rajesh > ----- Original Message ----- > From: Christian Balzer [mailto:chibi@gol.com] > To: dovecot(a)dovecot.org > Cc: kevin(a)my.walr.us > Sent: Mon, 13 Feb 2017 10:46:13 +0900 > Subject: > > > Hello, > > On Sun, 12 Feb 2017 08:27:21 -0500 KT Walrus wrote: > > > Thanks for the info. I do have one further question for you. On your servers that are currently handling 50k IMAP sessions, how many users does that correspond to? Since many users will have multiple IMAP sessions on multiple devices, I’d like to hear about some real-world numbers that could be used for budgeting a new project like mine. > > > > Those servers would actually be the wrong ones to look at, as they are > primarily accessed by the aforementioned broken client, so the numbers are > somewhat skewed. > However looking at other servers with a more "normal" user spread things > aren't actually too different. > > The average number of sessions per user tends to be 2. > The clear majority (over 50%) only has one session open (people with well > behaved and configured clients watching the INBOX mostly). > Another 30% has 2 sessions open, again the typical state of this would be > clients watching another mailbox besides INBOX, typically SENT. > The rest has 3 and more sessions open. > > The number of sessions could of course be drastically reduced if any > client would support IMAP NOTIFY, alas none that I'm aware of do. > > Lastly no more than 60% of the session seem to be in IDLE at any given > time, so my comments about RAM and IMAP hibernation effectiveness stand. > > > Also, do you use Dovecot IMAP proxies in front of your backend servers? If so, how many IMAP sessions can one proxy server handle (assuming the proxy does authorization using MySQL running on a separate server)? And, could the proxy server be tuned to help in optimizing mostly IDLE backend sessions? > > > > Yes to Dovecot Proxying, of course. > > No idea about MySQL, with (Open)LDAP nothing is breaking a sweat at an > average of 140 logins per second (IMAP and POP) on the 2 proxy servers. > If you can fit your entire dataset into RAM it should be fine, my LDAP > servers fall into that category and take about 10% of a slow (1.2GHz, 34%, > power-save mode) core only to handle the that load (also 2 servers). > And the rate of logins/s is what you need to worry about most and optimize > for. > > The proxies will of course have to do the shuffling of data and SSL > en/de-coding, but again they're not particular busy with that. > > The number of sessions comes into play when looking at the number of login > processes on the proxies and their memory footprint. > An IMAP login process on the proxies in performance mode with a client > limit of 1000 will consume about 55MB at most. > So assume at least 55KB RAM per session. > > Read Timo's mail I linked to about IMAP hibernation, AFAIK nothing has > happened to make proxies more supportive for this though. > > Christian > > > On Feb 12, 2017, at 1:58 AM, Christian Balzer <chibi(a)gol.com> wrote: > > > > > > > > > Hello, > > > > > > On Fri, 10 Feb 2017 14:50:03 -0500 KT Walrus wrote: > > > > > >>> 1. 256GB of real RAM, swap is for chums. > > >> > > >> Are you sure that 100,000 IMAP sessions wouldn’t work well with SWAP, especially with fast SSD storage (which is a lot cheaper than RAM)? > > >> > > > > > > I'm sure about tax and death, not much else. > > > > > > But as a rule of thumb I'd avoid swapping out stuff on production servers, > > > even if it were to SSDs. > > > Incidentally the servers I'm talking about here have their OS and swap on > > > Intel DC S3710s (200GB) and the actual storage on plenty of 1.6TB DC > > > S3610s. > > > > > > Relying on the kernel to make swap decisions is likely to result in much > > > reduced performance even with fast SWAP when you're overcommitting things > > > on that scale. > > > > > > > > > But read on. > > > > > >> Seems that these IMAP processes are long lived processes (idling most of the time) that don’t need that much of the contents of real memory available for much of the life of the process. I use a database proxy in front of MySQL (for my web apps) so that there can be a large number of TCP connections to the proxy where the frontend requests are queued for execution using a small number of backend connections. > > >> > > >> Could Dovecot IMAP be re-written to be more efficient so it works more like MySQL (or other scalable data servers) that could handle a million or more IMAP sessions on a server with 32GBs or less of RAM? Those IMAP sessions aren’t doing much most of the time and shouldn’t really average 2MB of active data per session that needs to be resident in main memory at all times. > > >> > > > See IMAP hibernation: > > > https://www.mail-archive.com/dovecot@dovecot.org/msg63429.html <https://www.mail-archive.com/dovecot@dovecot.org/msg63429.html> > > > > > > I'm going to deploy/test this in production in about 2 months from now, > > > but if you look at the link and the consequent changelog entries you'll see > > > that it has certain shortcomings and bug fixes in pretty much each release > > > after it was introduced. > > > > > > But this is the correct way to tackle things, not SWAP. > > > > > > Alas I'm not expecting miracles and if more than 20% of the IMAP sessions > > > here will be hibernated at any given time I'd be pleasantly surprised. > > > > > > Because between: > > > > > > 1. Finding a sensible imap_hibernate_timeout. > > > > > > 2. Having well behaved clients that keep idling instead of restarting the > > > sequence (https://joshdata.wordpress.com/2014/08/09/how-bad-is-imap-idle/ <https://joshdata.wordpress.com/2014/08/09/how-bad-is-imap-idle/>) > > > > > > 3. Having lots of mobile clients who either get disconnected (invisible to > > > Dovecot) or have aggressive IDLE timers to overcome carrier NAT timeouts > > > (a large mobile carrier here times out idle TCP sessions after 2 minutes, > > > forcing people to use 1 minute IDLE renewals, making 1. up there a > > > nightmare). > > > > > > 4. Having really broken clients (don't ask, I can't tell) which open IMAP > > > sessions, don't put them into IDLE and thus having them expire after 30 > > > minutes. > > > > > > the pool of eligible IDLE sessions isn't as big as it could be, in my case > > > at least. > > > > > >> My mail server isn’t that large yet as I haven’t fully deployed Dovecot outside my own small group yet, but it would be nice if scaling Dovecot IMAP to millions of users wasn’t limited to 50,000 IMAP sessions on a server... > > >> > > > > > > Scaling up is nice and desirable from a cost (rack space, HW) perspective, > > > but the scalability of things OTHER than Dovecot as I pointed out plus > > > that little detail of failure domains (do you really want half of your > > > eggs in one basket?) argue for scaling out after a certain density. > > > > > > I'm feeling my way there at this time, but expect more than 100k sessions > > > per server to be tricky. > > > > > > Lastly, when I asked about 500k sessions per server here not so long ago, > > > ( http://www.dovecot.org/list/dovecot/2016-November/106284.html <http://www.dovecot.org/list/dovecot/2016-November/106284.html> ) > > > Timo mentioned that he's not aware of anybody doing more than 50k per > > > server, something I got licked already and definitely will go to 100k > > > eventually. > > > > > > Regards, > > > > > > Christian > > >>> On Feb 10, 2017, at 11:07 AM, Christian Balzer <chibi(a)gol.com> wrote: > > >>> > > >>> On Fri, 10 Feb 2017 07:59:52 -0500 KT Walrus wrote: > > >>> > > >>>>> 1500 IMAP sessions will eat up about 3GB alone. > > >>>> > > >>>> Are you saying that Dovecot needs 2MB of physical memory per IMAP session? > > >>>> > > >>> That depends on the IMAP session, read the mailbox size and index size, > > >>> etc. > > >>> Some are significantly larger: > > >>> --- > > >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > >>> 1033864 mail 20 0 97600 67m 54m S 0 0.1 0:01.15 imap > > >>> --- > > >>> > > >>> But yes, as somebody who has mailbox servers with 55k+ session the average > > >>> is around 1.6MB. > > >>> > > >>>> If I want to support a max 100,000 IMAP sessions per server, I should configure the server to have at least 200GBs of SWAP? > > >>>> > > >>> You will want: > > >> > > >>> 2. Understanding how to tune Dovecot and more importantly the overall > > >>> system to such a task (see that PID up there?). > > >>> 3. Be willing to deal with stuff like top and ps taking ages to start/run > > >>> and others like atop actually killing dovecot (performance wise, not > > >>> literally) when doing their obviously flawed cleanup on exit. Some things > > >>> clearly do NOT scale well. > > >>> > > >>> My current goal is to have 100k capable servers that work well, 200k in a > > >>> failover scenario, but that won't be particular enjoyable. > > >>> > > >>> Christian > > >>> > > >>>>> On Feb 10, 2017, at 3:58 AM, Christian Balzer <chibi(a)gol.com> wrote: > > >>>>> > > >>>>> On Fri, 10 Feb 2017 01:13:20 +0530 Rajesh M wrote: > > >>>>> > > >>>>>> hello > > >>>>>> > > >>>>>> could somebody with experience let me know the dovecot config file settings to handle around 1500 simultaneous connections over pop3 and 1500 connection over imap simultaneously. > > >>>>>> > > >>>>> > > >>>>> Be very precise here, you expect to see 1500 as the result of > > >>>>> "doveadm who |grep pop3 |wc -l"? > > >>>>> > > >>>>> Because that implies an ungodly number of POP3 connects per second, given > > >>>>> the typically short duration of these. > > >>>>> > > >>>>> 1500 IMAP connections (note that frequently a client will have more than > > >>>>> the INBOX open and thus have more than one session and thus process on the > > >>>>> server) are a much easier proposition, provided they are of the typical > > >>>>> long lasting type. > > >>>>> > > >>>>> So can you put a number to your expected logins per second (both protocols)? > > >>>>> > > >>>>>> my server > > >>>>>> > > >>>>>> server configuration > > >>>>>> hex core processor, 16 gb ram 1 X 600 gb 15 k rpm for main drive and 2 X 2000 > > >>>>>> gb hdd for data (No raid) > > >>>>>> > > >>>>> No RAID and no other replication like DRBD? > > >>>>> Why would you even bother? > > >>>>> > > >>>>> How many users/mailboxes in total with what quota? > > >>>>> > > >>>>> 1500 IMAP sessions will eat up about 3GB alone. > > >>>>> You will want more memory, simply to keep all relevant SLAB bits (inodes, > > >>>>> dentries) in RAM. > > >>>>> > > >>>>> If you really have several hundreds logins/s, you're facing several > > >>>>> bottlenecks: > > >>>>> 1. Login processes themselves (easily fixed by high performance mode) > > >>>>> 2. Auth processes (that will depend on your backends, method mostly) > > >>>>> 3. Dovecot master process (spawning mail processes) > > >>>>> > > >>>>> The later is a single-threaded process, so it will benefit from a faster > > >>>>> CPU core. > > >>>>> It can be dramatically improved by enabling process re-usage, see: > > >>>>> http://wiki.dovecot.org/PerformanceTuning > > >>>>> > > >>>>> However that also means more memory usage. > > >>>>> > > >>>>> > > >>>>> > > >>>>> Christian > > >>>>> > > >>>>>> > > >>>>>> thanks > > >>>>>> rajesh > > >>>>>> > > >>>>> > > >>>>> [snip] > > >>>>> -- > > >>>>> Christian Balzer Network/Systems Engineer > > >>>>> chibi(a)gol.com Global OnLine Japan/Rakuten Communications > > >>>>> http://www.gol.com/ > > >>>> > > >>> > > >>> > > >>> -- > > >>> Christian Balzer Network/Systems Engineer > > >>> chibi(a)gol.com <mailto:chibi@gol.com> <mailto:chibi@gol.com <mailto:chibi@gol.com>> Global OnLine Japan/Rakuten Communications > > >>> http://www.gol.com/ <http://www.gol.com/> <http://www.gol.com/ <http://www.gol.com/>> > > > > > > > > > -- > > > Christian Balzer Network/Systems Engineer > > > chibi(a)gol.com <mailto:chibi@gol.com> Global OnLine Japan/Rakuten Communications > > > http://www.gol.com/ <http://www.gol.com/> > > > > -- Christian Balzer Network/Systems Engineer chibi(a)gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/

2 1