November 2014 - dovecot

High I/O wait with Dovecot POP3
by Frido Otten 20 Nov '14

20 Nov '14

Hello, In the past we've had 4 loadbalanced servers configured with courier pop3 and dovecot handling respectively POP3 and IMAP connections. These 4 servers have a shared NFS storage on a Netapp FAS3220 controller with SAS disks. This setup is handling 50 POP3 sessions per second without any problems. Now we're installing 6 new loadbalanced servers to do the same job on the same storage, but using Dovecot to handle the POP3 sessions instead of Courier. The IMAP sessions are already handled by those 6 new servers, but we're encountering high iowait issues when using POP3 on those servers. I've applied the hints from http://wiki2.dovecot.org/NFS but this had no improvement. The 6 new servers are quad core Intel(R) Xeon(R) CPU X3323 @ 2.50GHz, with 4 GB RAM connected with gigabit ethernet connection to the storage. The 4 old servers are single core (w/o HT) P4 2,8 GHz with 1 GB RAM. # dovecot --version 2.1.7 # /usr/sbin/dovecot -n # 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-5-686-bigmem i686 Debian 6.0.10 ext4 auth_cache_negative_ttl = 0 auth_cache_size = 8 k auth_mechanisms = plain digest-md5 cram-md5 login base_dir = /var/run/imap/ dict { acl = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no login_greeting = Dovecot MX. login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c mail_fsync = always mail_location = maildir:/var/spool/mail/%1u/%1.1u/%u mail_nfs_index = yes mail_nfs_storage = yes maildir_stat_dirs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave mmap_disable = yes namespace { inbox = yes location = prefix = separator = . type = private } namespace { list = children location = maildir:/var/spool/mail/%%1u/%%1.1u/%%u/:INDEX=/var/spool/mail/%1u/%1.1u/%u/shared/%%u prefix = shared.%%u. separator = . subscriptions = no type = shared } passdb { args = /etc/dovecot/dovecot-sql-auth.conf driver = sql } plugin { acl = vfile acl_shared_dict = proxy::acl quota = maildir quota_rule2 = Trash:storage=+25M } protocols = imap service anvil { client_limit = 4614 } service auth { client_limit = 2560 executable = /usr/lib/dovecot/auth unix_listener auth-master { mode = 0600 user = exim } user = dovecot } service dict { unix_listener dict { group = exim mode = 0660 user = exim } } service imap-login { chroot = login client_limit = 2048 executable = /usr/lib/dovecot/imap-login inet_listener imap { address = *, :: port = 143 } inet_listener imaps { address = *, :: port = 993 } process_limit = 2048 process_min_avail = 3 service_count = 0 user = dovecot vsz_limit = 128 M } service imap { executable = /usr/lib/dovecot/imap process_limit = 2048 } service managesieve-login { chroot = login client_limit = 2048 process_limit = 2048 process_min_avail = 3 service_count = 0 user = dovecot vsz_limit = 128 M } service managesieve { process_limit = 2048 } service pop3-login { chroot = login client_limit = 512 inet_listener pop3 { address = 1.2.3.4 port = 110 } inet_listener pop3s { address = 1.2.3.4 port = 0 } process_limit = 512 process_min_avail = 3 service_count = 0 user = dovecot vsz_limit = 128 M } service pop3 { process_limit = 2560 } ssl_cert = </etc/ssl/certs/wildcardcertificate.crt ssl_key = </etc/ssl/private/wildcardcertificate.key syslog_facility = local7 userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql-auth.conf driver = sql } protocol imap { mail_plugins = quota imap_quota acl imap_acl } protocol pop3 { pop3_uidl_format = %v.%u } protocol lda { auth_socket_path = /var/run/imap/auth-master info_log_path = /var/log/dovecot-deliver.log log_path = /var/log/dovecot-deliver.log mail_plugins = sieve quota postmaster_address = Mailer-Daemon(a)mx.domain.nl rejection_reason = Your message with subject "%s" was automatically rejected:%n%r sendmail_path = /usr/sbin/exim } Best regards, Frido

3 3

redis dict auth and default_fields
by brd 20 Nov '14

20 Nov '14

Hi all, i'm trying to configure default_fields with redis dict (allow_nets, proxy, ...) but it seems that the properties defined are not used even if they appear on output of doveconf. It is a known behaviour? Does default_fields works only for ldap db? I have a director+NFS environment and with dict passdb i'have to put proxy=y for every entry in pass/userdb; to limit the redis memory footprint it should be cool if the proxy=y can be configured as a default_field. It is a limit of using dict for pass/userdb cause using SQL you can manage it as a SELECT fixed field, using LDAP you can manage with default_fields but with dict you are forced to put all inside the db entries. doveconf output: [...] passdb { args = /etc/dovecot/dovecot-dict-auth.conf.ext default_fields = proxy=y driver = dict } [...] Regards -brd

1 0

Dovecot auth process repeatedly crashes on OS X Server 10.9
by James Reynolds 19 Nov '14

19 Nov '14

My dovecot/auth process crashes several times a minute, sometimes at the same second, sometimes randomly. I’ve turned on auth_debug and been pouring over everything trying to see if I can see what is going on but I can’t find anything. I’m running OS X Server 10.9 and dovecot 2.2.5. I can’t say for sure, but it looks to me like the crashed process doesn’t even save anything to mail-debug.log before it crashes. It takes about 2 seconds for it to come up. Once it crashed about 10 times in 2 seconds. Can anyone give me any ideas where I should look? Here are relevant portions from logs: mail-info.log Nov 06 17:19:10 pop3-login: Info: Aborted login (auth process communication failure): user=<>, method=DIGEST-MD5, rip=155.101.106.229, lip=10.0.0.128, TLS mail-err.log Nov 06 17:19:10 pop3-login: Warning: Auth connection closed with 1 pending requests (max 0 secs, pid=66114, EOF) Nov 06 17:19:10 auth: Fatal: master: service(auth): child 66100 killed with signal 11 (core dumps disabled) It looks exactly like what is discussed here: https://discussions.apple.com/thread/6482632?start=0&tstart=0 I do have clients using Outlook. I haven’t run wireshark or tried to verify it is Outlook by testing different clients. postconf -n # 2.2.5: /Library/Server/Mail/Config/dovecot/dovecot.conf # OS: Darwin 13.3.0 x86_64 hfs auth_mechanisms = cram-md5 digest-md5 plain login auth_realms = biomail.biology.utah.edu auth_socket_path = /var/run/dovecot/auth-userdb auth_username_format = %n debug_log_path = /Library/Logs/Mail/mail-debug.log default_internal_user = _dovecot default_login_user = _dovenull first_valid_gid = 6 first_valid_uid = 6 imap_id_log = * imap_id_send = "name" * "version" * imap_urlauth_submit_user = submit info_log_path = /Library/Logs/Mail/mail-info.log log_path = /Library/Logs/Mail/mail-err.log login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c mail_access_groups = mail mail_attribute_dict = file:/Library/Server/Mail/Data/attributes/attributes.dict mail_location = maildir:/Library/Server/Mail/Data/mail/%u mail_log_prefix = "%s(pid %p user %u): " mail_plugins = quota zlib acl fts fts_sk managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave mdbox_rotate_size = 200 M namespace acl-mailboxes { list = children location = maildir:/Library/Server/Mail/Data/mail/users/%%u:INDEX=/Library/Server/Mail/Data/mail/shared/%%u prefix = shared.%%u. separator = . subscriptions = no type = shared } namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } namespace list-archives { list = children location = maildir:/Library/Server/Mail/Data/listserver/messages/archive/lists/%%u:INDEX=/Library/Server/Mail/Data/listserver/messages/archive/shared/%%u prefix = archives.%%u. separator = . subscriptions = no type = shared } passdb { driver = od } passdb { args = /Library/Server/Mail/Config/dovecot/submit.passdb driver = passwd-file } plugin { acl = vfile:/Library/Server/Mail/Config/dovecot/global-acls:cache_secs=300 acl_shared_dict = file:/Library/Server/Mail/Data/shared/shared-mailboxes fts = sk quota = maildir:User quota quota_warning = storage=100%% quota-exceeded %u sieve = /Library/Server/Mail/Data/rules/%u/dovecot.sieve sieve_dir = /Library/Server/Mail/Data/rules/%u stats_refresh = 30 secs stats_track_cmds = yes } postmaster_address = postmaster(a)biology.utah.edu protocols = imap pop3 lmtp sieve quota_full_tempfail = yes service auth { idle_kill = 15 mins unix_listener auth-userdb { user = _dovecot } } service dns_client { unix_listener dns-client { mode = 0600 } } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } service_count = 0 } service imap { client_limit = 5 process_limit = 200 service_count = 0 } service indexer-worker { user = _dovecot } service lmtp { unix_listener lmtp { mode = 0600 } } service managesieve-login { inet_listener sieve { port = 4190 } } service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 ssl = yes } } service pop3 { client_limit = 5 process_limit = 200 service_count = 0 } service quota-exceeded { executable = script /Applications/Server.app/Contents/ServerRoot/usr/libexec/dovecot/quota-exceeded.sh unix_listener quota-exceeded { group = mail mode = 0660 user = _dovecot } user = _dovecot } service quota-warning { executable = script /Applications/Server.app/Contents/ServerRoot/usr/libexec/dovecot/quota-warning.sh unix_listener quota-warning { group = mail mode = 0660 user = _dovecot } user = _dovecot } service stats { fifo_listener stats-mail { mode = 0600 user = _dovecot } } ssl_ca = </etc/certificates/biomail.biology.utah.edu.CE3811751089890AEEC60B91DF2C083EE1BAD5EB.chain.pem ssl_cert = </etc/certificates/biomail.biology.utah.edu.CE3811751089890AEEC60B91DF2C083EE1BAD5EB.cert.pem ssl_cipher_list = ALL:!LOW:!SSLv2:!EXP:!aNULL:!ADH:!eNULL ssl_key = </etc/certificates/biomail.biology.utah.edu.CE3811751089890AEEC60B91DF2C083EE1BAD5EB.key.pem ssl_key_path = /etc/certificates/biomail.biology.utah.edu.CE3811751089890AEEC60B91DF2C083EE1BAD5EB.key.pem userdb { args = partition=/Library/Server/Mail/Config/dovecot/partition_map.conf enforce_quotas=no driver = od } userdb { args = /Library/Server/Mail/Config/dovecot/submit.passdb driver = passwd-file } verbose_proctitle = yes protocol lmtp { mail_plugins = quota zlib acl fts fts_sk sieve } protocol lda { mail_plugins = quota zlib acl fts fts_sk sieve } protocol imap { mail_max_userip_connections = 20 mail_plugins = quota zlib acl fts fts_sk imap_acl imap_quota imap_zlib } protocol pop3 { mail_max_userip_connections = 6 } Here’s part of a crash report: Process: auth [60325] Path: /Applications/Server.app/Contents/ServerRoot/usr/libexec/dovecot/auth Identifier: auth Version: 294 Code Type: X86-64 (Native) Parent Process: dovecotd [149] Responsible: dovecotd [149] User ID: 214 Date/Time: 2014-11-06 16:34:20.777 -0700 OS Version: Mac OS X 10.9.4 (13E28) Report Version: 11 Anonymous UUID: 9EB07E95-1536-96FA-A8C7-3784045DCDDE Crashed Thread: 0 Dispatch queue: com.apple.main-thread Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000 VM Regions Near 0: --> __TEXT 0000000103b39000-0000000103b6a000 [ 196K] r-x/rwx SM=COW /Applications/Server.app/Contents/ServerRoot/usr/libexec/dovecot/auth Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 libsystem_c.dylib 0x00007fff8a181732 strlen + 18 1 libdovecot.0.dylib 0x0000000103bdc2b8 vstrconcat + 72 Thread 1: 0 libsystem_kernel.dylib 0x00007fff858f9e6a __workq_kernreturn + 10 1 libsystem_pthread.dylib 0x00007fff89901f08 _pthread_wqthread + 330 2 libsystem_pthread.dylib 0x00007fff89904fb9 start_wqthread + 13 Thread 2:: Dispatch queue: com.apple.libdispatch-manager 0 libsystem_kernel.dylib 0x00007fff858fa662 kevent64 + 10 1 libdispatch.dylib 0x00007fff8808f421 _dispatch_mgr_invoke + 239 2 libdispatch.dylib 0x00007fff8808f136 _dispatch_mgr_thread + 52 Thread 3: 0 libsystem_kernel.dylib 0x00007fff858f9e6a __workq_kernreturn + 10 1 libsystem_pthread.dylib 0x00007fff89901f08 _pthread_wqthread + 330 2 libsystem_pthread.dylib 0x00007fff89904fb9 start_wqthread + 13 Thread 0 crashed with X86 Thread State (64-bit): rax: 0x00007fff5c0c6878 rbx: 0x0000000000000001 rcx: 0x0000000000000001 rdx: 0x0000000000000001 rdi: 0x0000000000000000 rsi: 0x0000000103b6354d rbp: 0x00007fff5c0c6800 rsp: 0x00007fff5c0c6800 r8: 0x0000000103b6354e r9: 0x0000000000000000 r10: 0x00007fff8a1c92a8 r11: 0x00007fb60dca54c3 r12: 0x0000000000000028 r13: 0x00007fff5c0c6920 r14: 0x0000000000000028 r15: 0x0000000000000009 rip: 0x00007fff8a181732 rfl: 0x0000000000010246 cr2: 0x0000000000000000 Logical CPU: 0 Error Code: 0x00000004 Trap Number: 14

1 1

Handle of "subscription" file/folder for IMAP
by Jorge Bastos 19 Nov '14

19 Nov '14

Hi, For IMAP users, there's the subscription file in the user's home, and if I try to subscribe more imap folders that are invisible, there's that one, but as it is a file, nothing is supposed to happen. Is there an way to, allow users to create the "subscriptions" folder without any collision, or just deny or don't show this file as an possible folder (which Is not) to subscribe. What does people normally do? Thanks in advanced,

1 0

"subscription" file/folder for IMAP
by Jorge Bastos 19 Nov '14

19 Nov '14

Hi, For IMAP users, there's the subscription file in the user's home, and if I try to subscribe more imap folders that are invisible, there's that one, but as it is an file, nothing is supposed to happen. Is there an way to, allow users to create the "subscriptions" folder without any collision, or just deny or don't show this file as an possible folder (which Is not) to subscribe. What does people normally do? Thanks in advanced,

1 0

Storage IOPs Calculation for Qmail Server
by SasiKumar.K 19 Nov '14

19 Nov '14

Dear DovecotORG, In my organization, we are about to implement Qmail Server. * The number of current users will be 800, in future it may increase upto 1200. * The number of concurrent users will be 300. I am the engineer to deploy the Qmail in Linux server. I need to tell the storage team on the IOPs requirement. I requested 8TB usable space for the mail storage (can scale up in future). Kindly provide your suggestion on this. Thanks & Regards, K.SasiKumar Systems Engineer, Locuz Enterprise Solutions Limited, Bangalore

2 1

userdb-nss crash
by Yuriy Tabolin 18 Nov '14

18 Nov '14

Hi All. I have a some problem with dovecot, when I try to use nss in auth userdb. nss was setup to resolve users and group by winbind. nss and winbind works well, command "id user" shows remote user's info. I would like to use nss userdb, but dovecot crashes on start. Is this a bug or incorrect settings? Thank you for any help. dovecot log Nov 18 14:58:36 mail dovecot: auth: Panic: file userdb-nss.c: line 110 (userdb_nss_preinit): assertion failed: (bufsize > 0) Nov 18 14:58:36 mail dovecot: auth: Fatal: master: service(auth): child 19387 killed with signal 6 (core dumped) Nov 18 14:58:36 mail dovecot: master: Error: service(auth): command startup failed, throttling for 8 secs # doveconf -n # 2.2.15: /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 10.0-RELEASE-p12 amd64 ufs auth_cache_size = 256 M auth_failure_delay = 3 secs auth_username_format = %n auth_verbose = yes auth_winbind_helper_path = /usr/local/bin/ntlm_auth base_dir = /var/run/dovecot/ disable_plaintext_auth = no listen = * mail_location = maildir:/home/%u/Maildir passdb { args = dovecot driver = pam } protocols = imap service anvil { client_limit = 2051 } service auth { client_limit = 3000 } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } process_limit = 1000 process_min_avail = 5 service_count = 1 vsz_limit = 64 M } service imap { process_limit = 2048 vsz_limit = 256 M } ssl = no userdb { args = service=winbind driver = nss } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 20 } # gdb /usr/local/libexec/dovecot/auth /home/tabolin/auth.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)... Core was generated by `auth'. Program terminated with signal 6, Aborted. Reading symbols from /usr/local/lib/dovecot/libdovecot.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/libdovecot.so.0 Reading symbols from /lib/libcrypt.so.5...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.5 Reading symbols from /usr/lib/libpam.so.5...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libpam.so.5 Reading symbols from /usr/local/lib/libldap-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libldap-2.4.so.2 Reading symbols from /usr/local/lib/liblber-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/liblber-2.4.so.2 Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/lib/libssl.so.7...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libssl.so.7 Reading symbols from /lib/libcrypto.so.7...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypto.so.7 Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x00000000114ece1a in kill () from /lib/libc.so.7 (gdb) # cat /etc/nsswitch.conf group: files winbind group_compat: nis hosts: files dns networks: files passwd: files winbind passwd_compat: nis shells: files services: compat services_compat: nis protocols: files rpc: files # cat /etc/pam.d/dovecot auth required /usr/local/lib/pam_winbind.so account required /usr/local/lib/pam_winbind.so -- <http://speechpro.com> С уважением, *Таболин Юрий Олегович* Системный администратор Центр Речевых Технологий Best regards, Tabolin Yuriy System administrator Speech Technology Center Тел/Tel: +7 (812) 325-8848, доб/ext 6225 Факс/Fax: +7 (812) 327-92-97 Моб/Mob: +7 (981) 130-23-03 <http://www.linkedin.com/company/speech-technology-center> <https://twitter.com/speechpro_ru> <http://vk.com/speechpro> <https://www.facebook.com/speechpro.ru>

1 0

custom ssh port doveadm sync
by dvm＠rdyn.net 18 Nov '14

18 Nov '14

Hi all, First sorry i really couldn't find it back been searching everywhere even the docs and its bit confusing. I am using # doveadm -D sync -u *(a)mydomain.net remote:myserver.net # which works perfect however for security reasons we now need to change the sshd port on the remote server. How do i tell doveadm to use a custom SSH port (would love to continue to use just the ssh connection), if anyone could explain that would be great, because all i found is custom tcp or tcp ssl which is not what we want to use, as i wish to stick with custom ssh port. Regards, r. -- This email was Virus checked by UTM 9. http://www.astaro.org

2 3

Different SSL certificates per IP and protocol
by tree 18 Nov '14

18 Nov '14

Hi Dovecot Users and Developers, I am hosting the server with multiple domains. I have setup the dovecot with the instructions from http://wiki2.dovecot.org/SSL/DovecotConfiguration Each domain has it's own ip address. However, when I connect via Thunderbird or any other mail clients, the client is ONLY picking up the top-level "default" ssl_key and ssl_cert and the certificat in /etc/ssl/dovecot.pem Is this a bug? This is on Debian Wheezy. Any help pointing in the right direction would be appreciated thanks. Here is the ssl section: ----------- ssl_cert = </etc/ssl/dovecot.pem ssl_key = </etc/ssl/dovecot.pem local 192.0.2.10 { # instead of IP you can also use hostname, which will be resolved protocol imap { ssl_cert = </etc/ssl/dovecot/imap-01.example.com.cert.pem ssl_key = </etc/ssl/dovecot/imap-01.example.com.key.pem } protocol pop3 { ssl_cert = </etc/ssl/dovecot/pop-01.example.com.cert.pem ssl_key = </etc/ssl/dovecot/pop-01.example.com.key.pem } } local 192.0.2.20 { protocol imap { ssl_cert = </etc/ssl/dovecot/imap-02.example.com.cert.pem ssl_key = </etc/ssl/dovecot/imap-02.example.com.key.pem } protocol pop3 { ssl_cert = </etc/ssl/dovecot/pop-02.example.com.cert.pem ssl_key = </etc/ssl/dovecot/pop-02.example.com.key.pem } } -----------------------

2 1

Mail sync at filesystem level
by Cédric Jeanneret 18 Nov '14

18 Nov '14

Hello, I'm currently building a mail infra using some decentralized filesystem for the storage. As I'm wanting to get multiple "entry-points" (servers running dovecot + postfix + other mail-related stuff), all of them sharing the same storage (most probably xtreemfs), how will dovecot cope in this case? As I see things: server1 gets a mail for User1; it pushes it to the xtreemfs directory. server2 shares the same xtreemfs directory, thus it can access the same mail listing User1 connects to server2: will he see the new mail? Will dovecot on server2 know about it? Thanks! Cheers, C.

1 0