- dovecot - dovecot.org

Dict quota calculation errors "remote disconnected"/"broken pipe" on 2.22.
by ygrishin-lists＠mail2.ca 13 Feb '17

13 Feb '17

Keywords: dovecot, dict, quota, postgre sql, broken pipe, remote disconnected Having Dovecot 2.2.22 (fe789d2) with Postgre SQL 9.5 (9.5.5-0ubuntu0.16.04) as the backend. I do not understand why quota service is not working, not seeing it as a configuration error at least. My quotas are DICT/SQL based. OS: Ubuntu 16.0.4.1 32-bit (Linux XXX 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:36:54 UTC 2017 i686 i686 i686 GNU/Linux) dovecot --build-options: *********************** Build options: ioloop=epoll notify=inotify ipv6 openssl io_block_size=8192 Mail storages: shared mdbox sdbox maildir mbox cydir imapc pop3c raw fail SQL driver plugins: mysql postgresql sqlite Passdb: checkpassword ldap pam passwd passwd-file shadow sql Userdb: checkpassword ldap(plugin) nss passwd prefetch passwd-file sql /etc/dovecot/conf.d/10-master.conf: *********************************** service quota-warning { executable = script /etc/dovecot/some-script.sh unix_listener quota-warning { user = Debian-exim mode = 0660 } } service dict { unix_listener dict { mode = 0660 user = Debian-exim group = Debian-exim } } /etc/dovecot/conf.d/90-quota.conf: ********************************** plugin { quota = dict:user_quota::proxy::sqlquota quota_rule2 = Trash:storage=+10%% quota_rule3 = Junk:storage=+10%% quota_grace = 10%% quota_warning = storage=100%% quota-exceeded 100 %u quota_warning2 = storage=95%% quota-warning 95 %u quota_warning3 = storage=90%% quota-warning 90 %u quota_warning4 = storage=75%% quota-warning 75 %u } dict { sqlquota = pgsql:/etc/dovecot/dovecot-dict-sql-user.conf } /etc/dovecot/dovecot-dict-sql-user.conf: **************************************** connect = host=A.B.C.D dbname=db user=DDD password=YYY map { pattern = priv/quota/storage table = quota2 username_field = username value_field = bytes } map { pattern = priv/quota/messages table = quota2 username_field = username value_field = messages } I will not be able to provide full "doveconf -n" output unfortunately. Logging ALL incoming DB queries: ******************************** ... 2017-02-04 12:03:12 MST [29500-10] DDD@db LOG: statement: SELECT password FROM mailbox WHERE local_part = 'YYY' AND domain = 'XXX' AND active ='t' LIMIT 1; 2017-02-04 12:03:12 MST [29501-10] DDD@db LOG: statement: SELECT 111 AS uid, 222 AS gid, '/var/mail/AAA/' || 'BBB' || '/' || 'YYY' AS home, '*:bytes=' || mailbox.quota AS quota_rule FROM mailbox WHERE local_part = 'YYY' AND active ='t' LIMIT 1; <END OF FILE HERE> And seeing that BOTH dict statements are missing: SELECT and UPDATE. dovecot-lda-erros.log: ********************** Feb 04 14:23:33 lda(testuser@XXX): Error: read(/var/run/dovecot/dict) failed: Remote disconnected Feb 04 14:23:33 lda(testuser@XXX): Error: Internal quota calculation error Feb 04 14:23:33 lda(testuser@XXX): Error: Internal quota calculation error dovecot.log: ************ Feb 04 13:57:06 imap(YYY@XXX): Error: write(/var/run/dovecot/dict) failed: Broken pipe Feb 04 13:57:06 imap(YYY@XXX): Error: write(/var/run/dovecot/dict) failed: Broken pipe ... Feb 04 13:57:07 imap(YYY@XXX): Error: write(/var/run/dovecot/dict) failed: Broken pipe ... Feb 04 13:57:10 imap(YYY@XXX): Error: write(/var/run/dovecot/dict) failed: Broken pipe I have tried stopping dovecot, removing /var/run/dovecot/dict manually and starting dovecot. This has not changed the behavior. dovecot-debug.log: ****************** Feb 04 13:18:12 lda(YYY@XXX): Error: read(/var/run/dovecot/dict) failed: Remote disconnected Feb 04 13:18:12 lda(YYY@XXX): Error: dict quota: Quota update failed, it's now desynced Feb 04 13:57:07 lda(testuser@XXX): Error: write(/var/run/dovecot/dict) failed: Broken pipe Feb 04 13:57:07 lda(testuser@XXX): Error: Internal quota calculation error Feb 04 13:57:07 lda(testuser@XXX): Error: Internal quota calculation error Running 'user_query' manually on the server: ******************************************** db=> SELECT 111 AS uid, 222 AS gid, '/var/mail/AAA/' || 'XXX' || '/' || 'testuser' AS home, '*:bytes=' || mailbox.quota AS quota_rule FROM mailbox WHERE local_part = 'testuser' AND active ='t' LIMIT 1; uid | gid | home | quota_rule -----+-----+-----------------------------------+------------------ 111 | 222 | /var/mail/AAA/BBB/testuser | *:bytes=10485760 (1 row) The identity had been granted 'all' privilege for 'quota2' table: ***************************************************************** db=> SELECT table_catalog, table_schema, table_name, privilege_type db-> FROM information_schema.table_privileges db-> WHERE grantee='DDD'; table_catalog | table_schema | table_name | privilege_type ---------------+--------------+------------+---------------- mail | public | quota2 | INSERT mail | public | quota2 | SELECT mail | public | quota2 | UPDATE mail | public | quota2 | DELETE mail | public | quota2 | TRUNCATE mail | public | quota2 | REFERENCES mail | public | quota2 | TRIGGER mail | public | mailbox | SELECT mail | public | quota | INSERT mail | public | quota | SELECT mail | public | quota | UPDATE mail | public | quota | DELETE mail | public | quota | TRUNCATE mail | public | quota | REFERENCES mail | public | quota | TRIGGER (15 rows) Rows are obviously not getting inserted: *************************************** db=> select * from quota2; username | bytes | messages ----------+-------+---------- (0 rows) I have checked logs for any crash-related messages, there's none. Also, I have tried to deploy 2.27-release version on the server to see whether the behavior is gone. I set-up a testing environment which consists of ubuntu-16.0.4-i386 server, installed build-essential package and unpacked the dovecot-2.2.27.tar.gz there. Getting exactly same issue as Ricardo Machini was having in his post to "v2.2.27 released" thread on Sat Dec 3 21:20:05 UTC 2016: **************************************** checking that generated files are newer than configure... done configure: error: conditional "SSL_VERSION_GE_102" was never defined. Usually this means the macro was only invoked conditionally. All I ran was ./configure 'openssl' package is installed as well.

2 2

dovecot config for 1500 simultaneous connection
by Rajesh M 13 Feb '17

13 Feb '17

hello could somebody with experience let me know the dovecot config file settings to handle around 1500 simultaneous connections over pop3 and 1500 connection over imap simultaneously. my server server configuration hex core processor, 16 gb ram 1 X 600 gb 15 k rpm for main drive and 2 X 2000 gb hdd for data (No raid) thanks rajesh my current config file settings as such # 2.2.7: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-431.29.2.el6.x86_64 x86_64 CentOS release 6.5 (Final) # NOTE: Send doveconf -n output instead when asking for help. auth_anonymous_username = anonymous auth_cache_negative_ttl = 0 auth_cache_size = 0 auth_cache_ttl = 0 auth_debug = no auth_debug_passwords = yes auth_default_realm = auth_failure_delay = 2 secs auth_gssapi_hostname = auth_krb5_keytab = auth_master_user_separator = auth_mechanisms = plain login digest-md5 cram-md5 auth_proxy_self = auth_realms = auth_socket_path = auth-userdb auth_ssl_require_client_cert = no auth_ssl_username_from_cert = no auth_use_winbind = no auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@ auth_username_format = %Lu auth_username_translation = auth_verbose = no auth_verbose_passwords = no auth_winbind_helper_path = /usr/bin/ntlm_auth auth_worker_max_count = 30 base_dir = /var/run/dovecot config_cache_size = 1 M debug_log_path = default_client_limit = 1000 default_idle_kill = 1 mins default_internal_user = dovecot default_login_user = vpopmail default_process_limit = 100 default_vsz_limit = 256 M deliver_log_format = msgid=%m: %$ dict_db_config = director_doveadm_port = 0 director_mail_servers = director_servers = director_user_expire = 15 mins director_username_hash = %u disable_plaintext_auth = no dotlock_use_excl = yes doveadm_allowed_commands = doveadm_password = doveadm_port = 0 doveadm_socket_path = doveadm-server doveadm_worker_count = 0 dsync_alt_char = _ dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -U first_valid_gid = 89 first_valid_uid = 89 hostname = imap_capability = imap_client_workarounds = imap_id_log = imap_id_send = name * imap_idle_notify_interval = 2 mins imap_logout_format = in=%i out=%o imap_max_line_length = 64 k imap_metadata = no imap_urlauth_host = imap_urlauth_logout_format = in=%i out=%o imap_urlauth_port = 143 imapc_features = imapc_host = imapc_list_prefix = imapc_master_user = imapc_max_idle_time = 29 mins imapc_password = imapc_port = 143 imapc_rawlog_dir = imapc_ssl = no imapc_ssl_verify = yes imapc_user = import_environment = TZ DEBUG_OUTOFMEM info_log_path = instance_name = dovecot last_valid_gid = 0 last_valid_uid = 0 lda_mailbox_autocreate = no lda_mailbox_autosubscribe = no lda_original_recipient_header = libexec_dir = /usr/libexec/dovecot listen = *, :: lmtp_address_translate = lmtp_proxy = no lmtp_rcpt_check_quota = no lmtp_save_to_detail_mailbox = no lock_method = fcntl log_path = /var/log/dovecot.log log_timestamp = "%b %d %H:%M:%S " login_access_sockets = login_greeting = ready. login_log_format = %$: %s login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c session=<%{session}> login_trusted_networks = mail_access_groups = mail_always_cache_fields = mail_attachment_dir = mail_attachment_fs = sis posix mail_attachment_hash = %{sha1} mail_attachment_min_size = 128 k mail_attribute_dict = mail_cache_fields = flags mail_cache_min_mail_count = 0 mail_chroot = mail_debug = no mail_fsync = optimized mail_full_filesystem_access = no mail_gid = mail_home = mail_location = mail_log_prefix = "%s(%u): " mail_max_keyword_length = 50 mail_max_lock_timeout = 0 mail_max_userip_connections = 10 mail_never_cache_fields = imap.envelope mail_nfs_index = no mail_nfs_storage = no mail_plugin_dir = /usr/lib64/dovecot mail_plugins = " quota" mail_prefetch_count = 0 mail_privileged_group = mail_save_crlf = no mail_shared_explicit_inbox = no mail_temp_dir = /tmp mail_temp_scan_interval = 1 weeks mail_uid = mailbox_idle_check_interval = 30 secs mailbox_list_index = no maildir_broken_filename_sizes = no maildir_copy_with_hardlinks = yes maildir_stat_dirs = no maildir_very_dirty_syncs = no managesieve_client_workarounds = managesieve_implementation_string = Dovecot Pigeonhole managesieve_logout_format = bytes=%i/%o managesieve_max_compile_errors = 5 managesieve_max_line_length = 65536 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave master_user_separator = mbox_dirty_syncs = yes mbox_dotlock_change_timeout = 2 mins mbox_lazy_writes = yes mbox_lock_timeout = 5 mins mbox_md5 = apop3d mbox_min_index_size = 0 mbox_read_locks = fcntl mbox_very_dirty_syncs = no mbox_write_locks = dotlock fcntl mdbox_preallocate_space = no mdbox_rotate_interval = 0 mdbox_rotate_size = 2 M mmap_disable = no namespace { disabled = no hidden = no ignore_on_failure = no inbox = yes list = yes location = prefix = separator = . subscriptions = yes type = private } passdb { args = cache_key=%u webmail=127.0.0.1 default_fields = deny = no driver = vpopmail master = no override_fields = pass = no result_failure = continue result_internalfail = continue result_success = return-ok skip = never } plugin { quota = maildir:ignore=Trash quota_rule = ?:storage=0 } pop3_client_workarounds = pop3_deleted_flag = pop3_enable_last = no pop3_fast_size_lookups = no pop3_lock_session = no pop3_logout_format = top=%t/%p, retr=%r/%b, del=%d/%m, size=%s pop3_no_flag_updates = no pop3_reuse_xuidl = no pop3_save_uidl = no pop3_uidl_duplicates = allow pop3_uidl_format = %08Xu%08Xv pop3c_host = pop3c_master_user = pop3c_password = pop3c_port = 110 pop3c_rawlog_dir = pop3c_ssl = no pop3c_ssl_verify = yes pop3c_user = %u postmaster_address = protocols = imap pop3 quota_full_tempfail = no recipient_delimiter = + rejection_reason = Your message to <%t> was automatically rejected:%n%r rejection_subject = Rejected: %s replication_full_sync_interval = 1 days replication_max_conns = 10 replicator_host = replicator replicator_port = 0 sendmail_path = /usr/sbin/sendmail service aggregator { chroot = . client_limit = 0 drop_priv_before_exec = no executable = aggregator extra_groups = fifo_listener replication-notify-fifo { group = mode = 0600 user = } group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = unix_listener replication-notify { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service anvil { chroot = empty client_limit = 0 drop_priv_before_exec = no executable = anvil extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 1 protocol = service_count = 0 type = anvil unix_listener anvil-auth-penalty { group = mode = 0600 user = } unix_listener anvil { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service auth-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = auth -w extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 1 type = unix_listener auth-worker { group = mode = 0600 user = $default_internal_user } user = vsz_limit = 18446744073709551615 B } service auth { chroot = client_limit = 0 drop_priv_before_exec = no executable = auth extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener auth-client { group = mode = 0600 user = $default_internal_user } unix_listener auth-login { group = mode = 0600 user = $default_internal_user } unix_listener auth-master { group = mode = 0600 user = } unix_listener auth-userdb { group = mode = 0666 user = $default_internal_user } unix_listener login/login { group = mode = 0666 user = } unix_listener token-login/tokenlogin { group = mode = 0666 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service config { chroot = client_limit = 0 drop_priv_before_exec = no executable = config extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = config unix_listener config { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service dict { chroot = client_limit = 1 drop_priv_before_exec = no executable = dict extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = unix_listener dict { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service director { chroot = . client_limit = 0 drop_priv_before_exec = no executable = director extra_groups = fifo_listener login/proxy-notify { group = mode = 00 user = } group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener director-admin { group = mode = 0600 user = } unix_listener login/director { group = mode = 00 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service dns_client { chroot = client_limit = 1 drop_priv_before_exec = no executable = dns-client extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = unix_listener dns-client { group = mode = 0666 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service doveadm { chroot = client_limit = 1 drop_priv_before_exec = no executable = doveadm-server extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 1 type = unix_listener doveadm-server { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service imap-login { chroot = login client_limit = 0 drop_priv_before_exec = no executable = imap-login extra_groups = group = idle_kill = 0 inet_listener imap { address = port = 143 reuse_port = no ssl = no } inet_listener imaps { address = port = 993 reuse_port = no ssl = yes } privileged_group = process_limit = 256 process_min_avail = 50 protocol = imap service_count = 1 type = login user = $default_login_user vsz_limit = 18446744073709551615 B } service imap-urlauth-login { chroot = token-login client_limit = 0 drop_priv_before_exec = no executable = imap-urlauth-login extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = imap service_count = 1 type = login unix_listener imap-urlauth { group = mode = 0666 user = } user = $default_login_user vsz_limit = 18446744073709551615 B } service imap-urlauth-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = imap-urlauth-worker extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1024 process_min_avail = 0 protocol = imap service_count = 1 type = unix_listener imap-urlauth-worker { group = mode = 0600 user = $default_internal_user } user = vsz_limit = 18446744073709551615 B } service imap-urlauth { chroot = client_limit = 1 drop_priv_before_exec = no executable = imap-urlauth extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1024 process_min_avail = 0 protocol = imap service_count = 1 type = unix_listener token-login/imap-urlauth { group = mode = 0666 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service imap { chroot = client_limit = 1 drop_priv_before_exec = no executable = imap extra_groups = group = idle_kill = 0 privileged_group = process_limit = 2048 process_min_avail = 50 protocol = imap service_count = 1 type = unix_listener login/imap { group = mode = 0666 user = } user = vsz_limit = 512 M } service indexer-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = indexer-worker extra_groups = group = idle_kill = 0 privileged_group = process_limit = 10 process_min_avail = 0 protocol = service_count = 0 type = unix_listener indexer-worker { group = mode = 0600 user = $default_internal_user } user = vsz_limit = 18446744073709551615 B } service indexer { chroot = client_limit = 0 drop_priv_before_exec = no executable = indexer extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener indexer { group = mode = 0666 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service ipc { chroot = empty client_limit = 0 drop_priv_before_exec = no executable = ipc extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener ipc { group = mode = 0600 user = } unix_listener login/ipc-proxy { group = mode = 0600 user = $default_login_user } user = $default_internal_user vsz_limit = 18446744073709551615 B } service lmtp { chroot = client_limit = 1 drop_priv_before_exec = no executable = lmtp extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = lmtp service_count = 0 type = unix_listener lmtp { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service log { chroot = client_limit = 0 drop_priv_before_exec = no executable = log extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = log unix_listener log-errors { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service managesieve-login { chroot = login client_limit = 0 drop_priv_before_exec = no executable = managesieve-login extra_groups = group = idle_kill = 0 inet_listener sieve { address = port = 4190 reuse_port = no ssl = no } privileged_group = process_limit = 0 process_min_avail = 0 protocol = sieve service_count = 1 type = login user = $default_login_user vsz_limit = 18446744073709551615 B } service managesieve { chroot = client_limit = 1 drop_priv_before_exec = no executable = managesieve extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = sieve service_count = 1 type = unix_listener login/sieve { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service pop3-login { chroot = login client_limit = 0 drop_priv_before_exec = no executable = pop3-login extra_groups = group = idle_kill = 0 inet_listener pop3 { address = port = 110 reuse_port = no ssl = no } inet_listener pop3s { address = port = 995 reuse_port = no ssl = yes } privileged_group = process_limit = 256 process_min_avail = 25 protocol = pop3 service_count = 1 type = login user = $default_login_user vsz_limit = 18446744073709551615 B } service pop3 { chroot = client_limit = 1 drop_priv_before_exec = no executable = pop3 extra_groups = group = idle_kill = 0 privileged_group = process_limit = 256 process_min_avail = 25 protocol = pop3 service_count = 1 type = unix_listener login/pop3 { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service replicator { chroot = client_limit = 0 drop_priv_before_exec = no executable = replicator extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener replicator-doveadm { group = mode = 00 user = $default_internal_user } unix_listener replicator { group = mode = 0600 user = $default_internal_user } user = vsz_limit = 18446744073709551615 B } service ssl-params { chroot = client_limit = 0 drop_priv_before_exec = no executable = ssl-params extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = startup unix_listener login/ssl-params { group = mode = 0666 user = } unix_listener ssl-params { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service stats { chroot = empty client_limit = 0 drop_priv_before_exec = no executable = stats extra_groups = fifo_listener stats-mail { group = mode = 0600 user = } group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener stats { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } shutdown_clients = yes ssl = yes ssl_ca = ssl_cert = </var/qmail/control/servercert.pem ssl_cert_username_field = commonName ssl_cipher_list = ALL:!LOW:!SSLv2:!EXP:!aNULL ssl_client_ca_dir = ssl_client_ca_file = ssl_client_cert = ssl_client_key = ssl_crypto_device = ssl_dh_parameters_length = 2048 ssl_key = </var/qmail/control/servercert.pem ssl_key_password = ssl_parameters_regenerate = 0 ssl_prefer_server_ciphers = no ssl_protocols = !SSLv2 ssl_require_crl = yes ssl_verify_client_cert = no state_dir = /var/lib/dovecot stats_command_min_time = 1 mins stats_domain_min_time = 12 hours stats_ip_min_time = 12 hours stats_memory_limit = 16 M stats_session_min_time = 15 mins stats_user_min_time = 1 hours submission_host = syslog_facility = mail userdb { args = cache_key=%u quota_template=quota_rule=*:backend=%q default_fields = driver = vpopmail override_fields = } valid_chroot_dirs = verbose_proctitle = no verbose_ssl = no version_ignore = no protocol imap { imap_client_workarounds = delay-newmail mail_max_userip_connections = 200 mail_plugins = " quota imap_quota" } protocol pop3 { mail_max_userip_connections = 40 pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_fast_size_lookups = yes pop3_lock_session = no pop3_no_flag_updates = yes }

3 7

Plugin sieve vs. imap_sieve
by Matthias Fechner 12 Feb '17

12 Feb '17

Dear all, I just saw that a "new" plugin imap_sieve is available: https://wiki2.dovecot.org/Pigeonhole/Sieve/Plugins/IMAPSieve I configured everything using the plugin sieve. Was the plugin sieve just renamed to imap_sieve or is it completely different? Thanks a lot for explaining what had been changed here, I cannot find anything in the wiki. Gruß Matthias -- "Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to produce bigger and better idiots. So far, the universe is winning." -- Rich Cook

2 1

antispam plugin pipe backend error when moving multiple emails
by Stéphane Cottin 12 Feb '17

12 Feb '17

Hi, I've got some trouble with the dovecot antispam plugin and the pipe backend. I'm using dovecot 2.2.18 with maildirs and zlib compression enabled. When moving 2 or more emails at once from the Junk folder to another one, I always have the following error : "Failed to copy to temporary file" In the server logs : imap(vvv(a)vvv.vvv): Error: read(zlib(/data/Maildir/.test/tmp/1446974366.M123890P936.vvv)) failed: Cached message size smaller than expected (13553 < 13562, box=test, UID=0) The same operation with one email at a time, on the same emails, works as expected. Stéphane

2 2

no Authnticationmethode from outlook to Imap
by Maurizio Caloro 11 Feb '17

11 Feb '17

Hello Debian, with postfix and Dovecot Imap "2.2.13" running. If I want connect from Outlook or Thunderbird I will become this error: Plaintext disalowd on non secure SSL/TLS Connection. but I will that this run with secure connection, I don't have any intention to use Plaintext auth. Also I don't know why, dovecot will switch from IMAP --> POP3-Login. Or it's this Outlook that will ask for it. 10-ssl.conf SSL = yes 10-auth.conf disable_plaintext_auth = yes Mail.log pop3-login: Info: Disconnected (tried to use disallowed plaintext auth): user=<>, rip=IP, lip=ip, session=<bnmZNB1AUZ+Blw> Regards

2 3

Re: dovecot config for 1500 simultaneous connection
by Joseph Tam 11 Feb '17

11 Feb '17

"Rajesh M" <24x7server(a)24x7server.net> writes: > during peak times here are the results for connections > > [root@ns1 domains]# doveadm who |grep imap |wc -l > username # proto (pids) (ips) > 631 > [root@ns1 domains]# doveadm who |grep pop3 |wc -l > username # proto (pids) (ips) > 233 Thare are user counts, not connections. Ae a user can launch multiple IMAP connections, and if they have some MaxOSX reader, that can peak to several dozens during searches. This counts connections doveadm who | awk '/imap/{m+=$2}/pop3/{n+=$2}END{print m,n}' Or you can parse the output of netstat. I'm suprised you have so many POP3 connections though -- they tend to be connect/process/disconnect. n=0 most of the time on my modest server. Joseph Tam <jtam.home(a)gmail.com>

2 1

Re: dovecot config for 1500 simultaneous connection
by Rajesh M 11 Feb '17

11 Feb '17

----- Original Message ----- From: Christian Balzer [mailto:chibi@gol.com] To: dovecot(a)dovecot.org Cc: 24x7server(a)24x7server.net Sent: Fri, 10 Feb 2017 17:58:58 +0900 Subject: On Fri, 10 Feb 2017 01:13:20 +0530 Rajesh M wrote: > hello > > could somebody with experience let me know the dovecot config file settings to handle around 1500 simultaneous connections over pop3 and 1500 connection over imap simultaneously. > Be very precise here, you expect to see 1500 as the result of "doveadm who |grep pop3 |wc -l"? Because that implies an ungodly number of POP3 connects per second, given the typically short duration of these. 1500 IMAP connections (note that frequently a client will have more than the INBOX open and thus have more than one session and thus process on the server) are a much easier proposition, provided they are of the typical long lasting type. So can you put a number to your expected logins per second (both protocols)? > my server > > server configuration > hex core processor, 16 gb ram 1 X 600 gb 15 k rpm for main drive and 2 X 2000 > gb hdd for data (No raid) > No RAID and no other replication like DRBD? Why would you even bother? How many users/mailboxes in total with what quota? 1500 IMAP sessions will eat up about 3GB alone. You will want more memory, simply to keep all relevant SLAB bits (inodes, dentries) in RAM. If you really have several hundreds logins/s, you're facing several bottlenecks: 1. Login processes themselves (easily fixed by high performance mode) 2. Auth processes (that will depend on your backends, method mostly) 3. Dovecot master process (spawning mail processes) The later is a single-threaded process, so it will benefit from a faster CPU core. It can be dramatically improved by enabling process re-usage, see: http://wiki.dovecot.org/PerformanceTuning However that also means more memory usage. Christian > > thanks > rajesh > [snip] -- Christian Balzer Network/Systems Engineer chibi(a)gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/ thanks christian during peak times here are the results for connections [root@ns1 domains]# doveadm who |grep imap |wc -l username # proto (pids) (ips) 631 [root@ns1 domains]# doveadm who |grep pop3 |wc -l username # proto (pids) (ips) 233 could you please guide me concerning the dovecot config files settings to handle the above 631 imap and 233 pop connections. number of mailboxes is around 4000 -- some users would consume 25 GB while others would be just around 10 MB this is a hex core machine with hyperthreading -- so 12 cores [root@ns1 domains]# iostat Linux 2.6.32-431.29.2.el6.x86_64 (ns1.bizmailserver.net) 02/10/2017 _x86_64_ (12 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 2.67 0.00 0.65 3.43 0.00 93.25 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sdd 44.95 1094.25 765.10 720884842 504041712 sdc 1.92 32.15 0.03 21178186 21248 sdb 34.71 1377.37 625.54 907398402 412102224 sda 49.88 124.29 2587.32 81879548 1704506408 thanks rajesh

2 1

Solr 6.4.1
by Daniel Miller 11 Feb '17

11 Feb '17

I've been running Solr for a while (4.10.3) - wanted to make the jump to the latest & greatest. I installed 6.4.1, copied over my schema.xml - and after a couple false starts where I needed to tweak it work with the new version...it works! I did not copy the database, started from scratch, and executed a "doveadm fts rescan -A". But... Judging solely from at least one client - it's fine. But looking in the logs I see: 1. The first scan of a mailbox dovecot's error log gives: dovecot: imap(dmiller(a)amfes.com): Error: fts_solr: Lookup failed: Bad Request 2. Subsequent scans do not appear to generate any dovecot error logs - but I'm not certain. Each new mailbox/subfolder scanned will each have one error on the initial scan. 3. Solr's log gives me the following - on every search. 2017-02-09 05:50:12.412 ERROR (qtp205125520-15) [ x:dovecot] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Bad contentType for search handler :text/xml request={q=from:"test"+OR+to:"test"+OR+cc:"test"+OR+subject:"test"+OR+body:"test"&fl=uid,score&sort=uid+asc&fq=%2Bbox:c1af150abfc9df4d7f7a00003bc41c5f+%2Buser:"dmiller(a)amfes.com"&rows=67135} at org.apache.solr.request.json.RequestUtil.processParams(RequestUtil.java:72) at org.apache.solr.util.SolrPluginUtils.setDefaults(SolrPluginUtils.java:180) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:163) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2306) at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:745) 2017-02-09 05:50:12.412 INFO (qtp205125520-15) [ x:dovecot] o.a.s.c.S.Request [dovecot] webapp=/solr path=/select params={q=from:"test"+OR+to:"test"+OR+cc:"test"+OR+subject:"test"+OR+body:"test"&fl=uid,score&sort=uid+asc&fq=%2Bbox:c1af150abfc9df4d7f7a00003bc41c5f+%2Buser:"dmiller(a)amfes.com"&rows=67135} status=400 QTime=1 My managed-schema file is: <?xml version="1.0" encoding="UTF-8"?>  <schema name="dovecot" version="1.5"> <uniqueKey>id</uniqueKey> <fieldType name="long" class="solr.TrieLongField" positionIncrementGap="0" precisionStep="0"/> <fieldType name="phonetic" class="solr.TextField" indexed="true" stored="false"> <analyzer> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.DoubleMetaphoneFilterFactory" inject="true"/> </analyzer> </fieldType> <fieldType name="string" class="solr.StrField" omitNorms="true" sortMissingLast="true"/> <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true"/> <fieldType name="booleans" class="solr.BoolField" sortMissingLast="true" multiValued="true"/> <fieldType name="tlongs" class="solr.TrieLongField" docValues="true" precisionStep="8" positionIncrementGap="0" multiValued="true"/> <fieldType name="tdoubles" class="solr.TrieDoubleField" docValues="true" precisionStep="8" positionIncrementGap="0" multiValued="true"/> <fieldType name="tdate" class="solr.TrieDateField" docValues="true" precisionStep="6" positionIncrementGap="0"/> <fieldType name="tdates" class="solr.TrieDateField" docValues="true" precisionStep="6" positionIncrementGap="0" multiValued="true"/> <fieldType name="text_en_splitting" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-FoldToASCII.txt"/> <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/> <charFilter class="solr.HTMLStripCharFilterFactory"/> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" catenateAll="0" catenateWords="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterFilterFactory" catenateNumbers="0" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" catenateAll="0" catenateWords="0"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> <field name="_version_" type="long" indexed="true" stored="true"/> <field name="bcc" type="text_en_splitting" indexed="true" stored="false"/> <field name="body" type="text_en_splitting" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="text_en_splitting" indexed="true" stored="false"/> <field name="from" type="text_en_splitting" indexed="true" stored="false"/> <field name="hdr" type="text_en_splitting" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="text_en_splitting" indexed="true" stored="false"/> <field name="text" type="text_en_splitting" multiValued="true" indexed="true" stored="false"/> <field name="to" type="text_en_splitting" indexed="true" stored="false"/> <field name="uid" type="long" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema> -- Daniel

2 5

Safely piping to a shell script
by Adam Shostack 10 Feb '17

10 Feb '17

Thanks for the previous answer on :execute. In thinking about malicious input, I am worried about the possibility that mail will be sent with a clever from line. (Section 7 of http://www.ietf.org/rfc/rfc5229.txt is great, btw) To address this, I'm considering the following, and would appreciate feedback. I'm aware that this doesn't capture all emails, those with non-alphanum are legit, and badly handled. I think that this restricts the input of the grepfrom script to be a single string, matching "a-zA-Z09@." if not address :regex "from" "^[:alnum]*@[:alnum]*([:alnum].)*$" { fileinto :create "wierd"; stop ; } elsif address :regex "from" "*" { set "sender" ${1}; if execute "grepfrom" "${sender}" { keep ; stop; } else {fileinto :create "neversent"} As an aside, https://www.joachim-breitner.de/blog/441-Goodbye_procmail,_Hello_Sieve contains a nice pattern, creating an include file to test, and that addresses many, but not all of my use cases. Should I worry if the match there is 5000+ strings? Adam

1 0

Prohibit dots in folder names
by Marek Grondecki 10 Feb '17

10 Feb '17

Hello Dovecot Community, I am using Dovecot 2.2.13-12~deb8u1 (Debian Jessie). separator = / listescape plugin is NOT active LAYOUT=fs is NOT active I would like to prohibit a creation of folders with dot "." in the names - so creation of a folder "foo.bar" should NOT be possible. Currently, when I create "foo.bar" in Outlook 2013 it will be created but won't be synchronised with the mail server - Outlook shows it as "foo.bar (Only this computer)". Ideally the user should receive an error message informing that "the folder foo.bar could not be created." Thank you for your help. Regards, Marek Grondecki

3 4