- dovecot - dovecot.org

bug in indexer/indexer-worker
by Infoomatic 18 Jun '15

18 Jun '15

Hi, I have already mentioned this in http://www.dovecot.org/pipermail/dovecot/2014-November/098592.html I could reproduce the errors above in a self-compiled v2.2.18 and the prebuilt packages from xi.rename-it.nl (in addition to version 2.2.15 mentioned in the link) The problem occurs when enabling fts_tika in the plugins (tried tika 1.6, 1.7 and 1.8). I tried to move a folder of my mailbox with about 2000 mails to my server (no users, modern hardware). At some point I get an error and from this time on dovecot keeps repeating the last lines with every mail that comes in ... see [1] I then get kernel syslog messages like in [2]. when I try to index all users (had another testuser) mailboxes with "doveadm index" I get the errors of [3]. The mailfolder does contain mails with attachments of all sorts, videos, office documents etc. When disabling fts_tika it works. dovecot -n output is [4] Could someone of the devs please have a look? best regards, infoomatic [1] Then comes the panic which produces loads of errormessages below the "Raw backtrace" part: Jun 16 10:11:08 testhost dovecot: indexer-worker(user1): Warning: I/O leak: 0x7f40b55c39d0 (line 411, fd 20) Jun 16 10:11:08 testhost dovecot: indexer-worker(user1): Warning: Timeout leak: 0x7f40b55c3780 (line 325) Jun 16 10:11:08 testhost dovecot: indexer-worker(user1): Panic: file ioloop.c: line 39 (io_add_file): assertion failed: (callback != NULL) Jun 16 10:11:08 testhost dovecot: indexer-worker(user1): Error: Raw backtrace: /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x78320) [0x7f40b55f4320] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x783fe) [0x7f40b55f43fe] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f40b55a3b25] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x897d0) [0x7f40b56057d0] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(io_loop_move_io+0x3e) [0x7f40b560643e] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(connection_switch_ioloop+0x14) [0x7f40b55f21d4] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(http_client_switch_ioloop+0x28) [0x7f40b55c8148] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x44536) [0x7f40b55c0536] -> /opt/dovecot/lib/dovecot/lib21_fts_solr_plugin.so(solr_connection_post_more+0x49) [0x7f40b108d469] -> /opt/dovecot/lib/dovecot/lib21_fts_solr_plugin.so(+0x3ac9) [0x7f40b1089ac9] -> /opt/dovecot/lib/dovecot/lib21_fts_solr_plugin.so(+0x3b6d) [0x7f40b1089b6d] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0x86ec) [0x7f40b495c6ec] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_deinit+0x23) [0x7f40b495c723] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0xe5cc) [0x7f40b49625cc] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0xf451) [0x7f40b4963451] -> /opt/dovecot/lib/dovecot/lib10_quota_plugin.so(+0xb454) [0x7f40b4b97454] -> /opt/dovecot/lib/dovecot/lib01_acl_plugin.so(+0xcfea) [0x7f40b4daafea] -> /opt/dovecot/lib/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit_get_changes+0x41) [0x7f40b58a64f1] -> /opt/dovecot/lib/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit+0xe) [0x7f40b58a658e] -> dovecot/indexer-worker() [0x40250c] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x4c) [0x7f40b560615c] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xbb) [0x7f40b56070ab] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9) [0x7f40b56061c9] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f40b5606248] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f40b55a9093] -> dovecot/indexer-worker(main+0xe3) [0x401f03] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f40b51d8ec5] -> dovecot/indexer-worker() [0x401fc2] Jun 16 10:11:08 testhost dovecot: indexer: Error: Indexer worker disconnected, discarding 2 requests for user1 Jun 16 10:11:08 testhost dovecot: indexer-worker(user1): Fatal: master: service(indexer-worker): child 2066 killed with signal 6 (core dumped) Jun 16 10:11:09 testhost dovecot: indexer-worker(user1): Warning: I/O leak: 0x7f25bc1689d0 (line 120, fd 23) Jun 16 10:11:09 testhost dovecot: indexer-worker(user1): Warning: Timeout leak: 0x7f25bc168780 (line 325) Jun 16 10:11:09 testhost dovecot: indexer: Error: Indexer worker disconnected, discarding 2 requests for user1 Jun 16 10:11:09 testhost dovecot: indexer-worker(user1): Fatal: master: service(indexer-worker): child 2124 killed with signal 11 (core dumped) Jun 16 10:11:09 testhost dovecot: indexer-worker(user1): Warning: I/O leak: 0x7f6cabb149d0 (line 120, fd 23) Jun 16 10:11:09 testhost dovecot: indexer-worker(user1): Warning: Timeout leak: 0x7f6cabb14780 (line 325) Jun 16 10:11:09 testhost dovecot: indexer: Error: Indexer worker disconnected, discarding 2 requests for user1 Jun 16 10:11:09 testhost dovecot: indexer-worker(user1): Fatal: master: service(indexer-worker): child 2125 killed with signal 11 (core dumped) [2] kernel syslog: [Tue Jun 16 09:51:15 2015] indexer-worker[7592]: segfault at 8 ip 00007f066e8c8c27 sp 00007fff36a038b0 error 4 in libdovecot.so.0.0.0[7f066e832000+e7000] [Tue Jun 16 09:51:16 2015] indexer-worker[7594]: segfault at 8 ip 00007f702671ac27 sp 00007ffc6e134840 error 4 in libdovecot.so.0.0.0[7f7026684000+e7000] [Tue Jun 16 09:51:16 2015] indexer-worker[7596]: segfault at 20 ip 00007fb8a5af2c47 sp 00007fff0a344780 error 4 in libdovecot.so.0.0.0[7fb8a5a69000+e7000] [Tue Jun 16 09:51:17 2015] indexer-worker[7598]: segfault at 20 ip 00007f5834b6ac47 sp 00007ffd54f35c00 error 4 in libdovecot.so.0.0.0[7f5834ae1000+e7000] [Tue Jun 16 09:51:18 2015] indexer-worker[7600]: segfault at 20 ip 00007f5b2f6fac47 sp 00007fff1a697690 error 4 in libdovecot.so.0.0.0[7f5b2f671000+e7000] [Tue Jun 16 09:51:19 2015] indexer-worker[7602]: segfault at 20 ip 00007fd6084ddc47 sp 00007ffdb4495af0 error 4 in libdovecot.so.0.0.0[7fd608454000+e7000] [Tue Jun 16 09:51:20 2015] indexer-worker[7604]: segfault at 20 ip 00007f7049789c47 sp 00007ffe71e4bdd0 error 4 in libdovecot.so.0.0.0[7f7049700000+e7000] [Tue Jun 16 09:51:20 2015] indexer-worker[7606]: segfault at 20 ip 00007f771d54dc47 sp 00007ffe01c6ff80 error 4 in libdovecot.so.0.0.0[7f771d4c4000+e7000] [Tue Jun 16 09:51:21 2015] indexer-worker[7608]: segfault at 20 ip 00007fc19d482c47 sp 00007ffc9a0b0280 error 4 in libdovecot.so.0.0.0[7fc19d3f9000+e7000] [Tue Jun 16 09:51:22 2015] indexer-worker[7610]: segfault at 29 ip 00007fa14df25c2b sp 00007ffeb246e5f0 error 4 in libdovecot.so.0.0.0[7fa14de8f000+e7000] [Tue Jun 16 09:51:23 2015] indexer-worker[7612]: segfault at 8 ip 00007fe8e41f2c27 sp 00007ffc700e2170 error 4 in libdovecot.so.0.0.0[7fe8e415c000+e7000] [Tue Jun 16 09:51:24 2015] indexer-worker[7614]: segfault at 8 ip 00007ff9a0c76c27 sp 00007fffecd44c90 error 4 in libdovecot.so.0.0.0[7ff9a0be0000+e7000] [Tue Jun 16 09:51:25 2015] indexer-worker[7616]: segfault at 8 ip 00007fd27867dc27 sp 00007ffedd8407f0 error 4 in libdovecot.so.0.0.0[7fd2785e7000+e7000] [Tue Jun 16 09:51:25 2015] indexer-worker[7618]: segfault at 8 ip 00007f94f8753c27 sp 00007fff56cc5ea0 error 4 in libdovecot.so.0.0.0[7f94f86bd000+e7000] [3] errors from "doveadm index": /opt/dovecot/bin/doveadm index -A '*' doveadm(user1): Warning: I/O leak: 0x7f0e072519d0 (line 120, fd 27) doveadm(user1): Warning: Timeout leak: 0x7f0e07251780 (line 325) Segmentation fault (core dumped) /opt/dovecot/bin/doveadm index -u user1 '*' doveadm(user1): Warning: I/O leak: 0x7f99b4fba9d0 (line 411, fd 14) doveadm(user1): Warning: Timeout leak: 0x7f99b4fba780 (line 325) doveadm(user1): Fatal: pool_system_realloc(17179869184): Out of memory doveadm(user1): Error: Raw backtrace: /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x78320) [0x7f99b4feb320] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7f99b4feb38a] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f99b4f9ac7f] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x8f3a3) [0x7f99b50023a3] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x7472d) [0x7f99b4fe772d] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(buffer_get_space_unsafe+0x5f) [0x7f99b4fe7f5f] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(io_loop_handle_add+0x24) [0x7f99b4ffdd94] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x89761) [0x7f99b4ffc761] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(io_loop_move_io+0x3e) [0x7f99b4ffd43e] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(connection_switch_ioloop+0x14) [0x7f99b4fe91d4] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(http_client_switch_ioloop+0x28) [0x7f99b4fbf148] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x44536) [0x7f99b4fb7536] -> /opt/dovecot/lib/dovecot/lib21_fts_solr_plugin.so(solr_connection_post_more+0x49) [0x7f99b0c9d469] -> /opt/dovecot/lib/dovecot/lib21_fts_solr_plugin.so(+0x373f) [0x7f99b0c9973f] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0x91e8) [0x7f99b43541e8] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0x95f0) [0x7f99b43545f0] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x714) [0x7f99b4354d34] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0xf17e) [0x7f99b435a17e] -> /opt/dovecot/lib/dovecot/libdovecot-storage.so.0(mail_precache+0x19) [0x7f99b5294e89] -> /opt/dovecot/bin/doveadm() [0x423211] -> /opt/dovecot/bin/doveadm() [0x41ecc6] -> /opt/dovecot/bin/doveadm(doveadm_mail_try_run+0x27f) [0x41fb8f] -> /opt/dovecot/bin/doveadm(main+0x38f) [0x412a9f] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f99b4bcfec5] -> /opt/dovecot/bin/doveadm() [0x412b99] [4] dovecot -n # 2.2.18: /opt/dovecot/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.8 (0c4ae064f307+) # OS: Linux 3.13.0-52-generic x86_64 Ubuntu 14.04.2 LTS ext4 auth_username_format = %n auth_verbose = yes disable_plaintext_auth = no imap_idle_notify_interval = 5 mins mail_gid = vmail mail_home = /var/lib/vmail/home/%d/%u mail_location = mdbox:/var/lib/vmail/mail/%d/%u mail_plugins = fts fts_solr zlib acl quota zlib fts fts_solr acl mail_uid = vmail mailbox_idle_check_interval = 1 mins managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate imapflags notify mdbox_rotate_size = 32 M namespace { list = children location = mdbox:/var/lib/vmail/mail/%d/%%u prefix = shared/%%u/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = / type = private } passdb { args = /opt/dovecot/etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile acl_defaults_from_inbox = yes acl_shared_dict = file:/var/lib/vmail/shared-mailboxes fts = solr fts_autoindex = yes fts_solr = url=http://localhost:8983/solr/ break-imap-search fts_tika = http://localhost:8081/tika quota = dict:Quota::file:%h/dovecot-quota quota_rule = *:storage=1G quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_extensions = +notify +imapflags zlib_save = gz zlib_save_level = 9 } protocols = imap sieve service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { group = vmail user = vmail } } service imap-login { inet_listener imaps { port = 0 } } service managesieve-login { inet_listener sieve_deprecated { port = 2000 } } service pop3-login { inet_listener pop3 { port = 0 } inet_listener pop3s { port = 0 } } ssl = required ssl_cert = </etc/ssl/testhost/testhost.crt ssl_cipher_list = EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:+CAMELLIA256:+AES256:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!ECDSA:CAMELLIA256-SHA:AES256-SHA:CAMELLIA128-SHA:AES128-SHA ssl_key = </etc/ssl/testhost/testhost.key ssl_prefer_server_ciphers = yes ssl_protocols = !SSLv2 !SSLv3 userdb { args = /opt/dovecot/etc/dovecot/dovecot-ldap.conf.ext driver = ldap } protocol lda { mail_plugins = fts fts_solr zlib sieve acl } protocol imap { mail_plugins = fts fts_solr zlib acl quota imap_acl imap_quota } protocol sieve { mail_plugins = fts fts_solr }

3 3

FTS lucene and english + german mails
by Felix Zielcke 17 Jun '15

17 Jun '15

Hi, I'm currently looking over the FTS pages to enable it in my dovecot. But I'm unsure what the best settings of the lucene plugin are, if you receive german and english mails. Wiki says: textcat_conf=<path> textcat_dir=<path>: If specified, enable guessing the stemming language for emails and search keywords. This is a little bit problematic in practice, since indexing and searching languages may differ and may not find even exact words because they stem differently. On Debian libstemmer is included in the debian-lucene package. So what settings are the best to have not the problem that exact words can't be found?

2 2

Sorry, wrong paste buffer
by Steve Matzura 17 Jun '15

17 Jun '15

Please ignore last message.

1 0

Re: Mail to nowhere
by Steve Matzura 17 Jun '15

17 Jun '15

On Wed, 17 Jun 2015 09:10:26 -0400, you wrote: >Please send the output of postconf -n to the mailing list. Paul: I purposefully obfuscated my FQDN and domain, but they both have real and good data in them. alias_database = hash:/etc/aliases alias_maps = hash:/etc/aliases, hash:/var/lib/mailman/data/aliases command_directory = /usr/sbin config_directory = /etc/postfix daemon_directory = /usr/libexec/postfix data_directory = /var/lib/postfix debug_peer_level = 2 debugger_command = PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin ddd $daemon_directory/$process_name $process_id & sleep 5 html_directory = no inet_interfaces = localhost inet_protocols = all mail_owner = postfix mailq_path = /usr/bin/mailq.postfix manpage_directory = /usr/share/man mydestination = $myhostname, localhost.$mydomain, localhost myhostname = <my-FQDN> newaliases_path = /usr/bin/newaliases.postfix queue_directory = /var/spool/postfix readme_directory = /usr/share/doc/postfix/README_FILES sample_directory = /usr/share/doc/postfix/samples sendmail_path = /usr/sbin/sendmail.postfix setgid_group = postdrop smtp_tls_session_cache_database = btree:${queue_directory}/smtp_scache smtpd_sasl_auth_enable = yes smtpd_tls_auth_only = yes smtpd_tls_key_file = /etc/pki/dovecot/certs/tgv2015.crt smtpd_tls_received_header = yes smtpd_tls_security_level = may smtpd_tls_session_cache_database = btree:${queue_directory}/smtpd_scache smtpd_use_tls = yes unknown_local_recipient_reject_code = 550 virtual_mailbox_base = /var/mail/vhosts virtual_mailbox_domains = <my-domain> virtual_transport = lmtp:unix:private/dovecot-lmtp

1 0

SOLR 5
by Robert Gierzinger 17 Jun '15

17 Jun '15

Hello, I just wanted to give SOLR 5 a try, however there probably have changed quite some bits in the config files, did not even manage to create a core with various solrconfig.xml and schema.xml files, but I am absolutely no expert in solr. Has anybody given it a try or are there some tips on how to get it running? regards, Robert

2 1

dsync userdb and required fields?
by lejeczek 17 Jun '15

17 Jun '15

hi, I'm fiddling with dsync and get these in the logs: auth: Error: password(ps655@): Password data is not valid for scheme SSHA: Input isn't valid base64 encoded data .. auth: Error: ldap(ns580(a),172.25.12.215): ldap_search(base=ou=,ou=mail,dc=virtual,dc=hosting filter=(&(objectclass=person) .... This is on a dovecot that I'd call second, to which I'd like to replicate, first dovecot which is already populated with users, etc. does not show these errors. and I also have seen complains about possixAccunt I think. Do these mean that there is a base minimum required from dovecot/dsync in order to deem/find a user as a valid user? Also I can tell that those users do belong to different domain (not even this RDN) than the one in which the second dovecot is trying to look. These fields/attributes are nowhere defined/mentioned in my configs thus I wonder. many thanks

1 0

Imap process crashes when search in virtual mailbox
by mihaiush 17 Jun '15

17 Jun '15

Hi, When searching in a virtual mailbox imap process crashes if the virtual mailbox has a lot of physical mailboxes behind (more than 80 - 100 mailboxes). Dovecot 2.2.18 build from sources ./configure --prefix=/opt/dovecot2 --with-mysql --with-sqlite --with-solr --with-ssl --disable-rpath --disable-static. Debian Wheezy 3.2.63-2 x86_64. Filesystem is ZFS. docecot -n : # 2.2.18: /services/dovecot1/etc/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.7 auth_master_user_separator = * auth_mechanisms = plain login base_dir = /services/dovecot1/var/run default_client_limit = 10240 default_login_user = dovecot dict { quotadict = mysql:/services/dovecot1/etc/quota.conf } disable_plaintext_auth = no log_path = login_log_format_elements = user=%u session=%{session} mpid=%e rip=%r rrip=%{real_rip} mail_location = mdbox:~/mails mail_log_prefix = "store107: %s(%u): %{session}: " mail_max_userip_connections = 0 mail_plugins = quota virtual fts fts_solr notify mail_log mailbox_list_index = yes mdbox_rotate_size = 256 k mmap_disable = yes namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / } namespace virtual { hidden = yes list = no location = virtual:/services/dovecot1/etc/virtual:INDEX=~/mails/virtual mailbox all { auto = create special_use = \All } mailbox unseen { auto = create } prefix = virtual/ separator = / } passdb { args = /services/dovecot1/etc/passwd.masterusers driver = passwd-file master = yes pass = yes } passdb { args = /services/dovecot1/etc/mysql.conf driver = sql } plugin { fts = solr fts_solr = break-imap-search url=http://10.78.18.127:8080/solr/ mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename flag_change save mailbox_create mail_log_fields = uid box msgid flags quota = dict:user::proxy::quotadict quota_rule = *:storage=1GB quota_rule2 = Trash:storage=+152M quota_rule3 = Trash:messages=+5 } protocols = imap service auth { unix_listener auth-client { mode = 0666 } unix_listener auth-master { group = nogroup mode = 0660 } } service dict { process_limit = 400 unix_listener dict { mode = 0600 user = nobody } } service imap-login { chroot = login client_limit = 2048 inet_listener imap { address = 10.78.18.127 port = 143 } process_limit = 1024 service_count = 0 } service imap { process_limit = 10240 vsz_limit = 320 M } service indexer-worker { process_limit = 4 process_min_avail = 2 service_count = 1 vsz_limit = 1 G } shutdown_clients = no ssl = no syslog_facility = local3 userdb { args = /services/dovecot1/etc/mysql.conf driver = sql } protocol imap { mail_plugins = quota virtual fts fts_solr notify mail_log imap_quota } protocol lda { auth_socket_path = /services/dovecot1/var/run/auth-master mail_plugins = quota virtual fts fts_solr notify mail_log sieve event_broker submission_host = osmtp:8090 } /services/dovecot1/etc/virtual/all/dovecot-virtual : * -Trash -Junk all Error log: 2015-06-16T12:49:25.502864+02:00 sr-be-store07 dovecot: store107: imap( mihaiush(a)sunrise.ch) hCdxU6AY/wAKThJr: Panic: file mail-index-map.c: line 547 (mail_index_map_lookup_seq_range): assertion failed: (first_uid > 0) 2015-06-16T12:49:25.503429+02:00 sr-be-store07 dovecot: store107: imap( mihaiush(a)sunrise.ch) hCdxU6AY/wAKThJr: Error: Raw backtrace: /opt/dovecot2/lib/dovecot/libdovecot.so.0(+0x7c13f) [0x7fe2e433713f] -> /opt/dovecot2/lib/dovecot/libdovecot.so.0(+0x7c19e) [0x7fe2e433719e] -> /opt/dovecot2/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fe2e42e5555] -> /opt/dovecot2/lib/dovecot/libdovecot-storage.so.0(+0xc4d32) [0x7fe2e466fd32] -> /opt/dovecot2/lib/dovecot/libdovecot-storage.so.0(mail_index_lookup_seq_range+0xa) [0x7fe2e46817fa] -> /opt/dovecot2/lib/dovecot/lib20_fts_plugin.so(+0xd0b1) [0x7fe2e30c40b1] -> /opt/dovecot2/lib/dovecot/lib20_fts_plugin.so(+0xd963) [0x7fe2e30c4963] -> /opt/dovecot2/lib/dovecot/lib20_fts_plugin.so(fts_search_lookup+0xa6) [0x7fe2e30c4c36] -> /opt/dovecot2/lib/dovecot/lib20_fts_plugin.so(+0xfdef) [0x7fe2e30c6def] -> dovecot/imap(imap_search_start+0x70) [0x41dec0] -> dovecot/imap(cmd_sort+0x205) [0x414565] -> dovecot/imap(command_exec+0x4c) [0x418aec] -> dovecot/imap() [0x417ad0] -> dovecot/imap() [0x417b64] -> dovecot/imap(client_handle_input+0x115) [0x417e45] -> dovecot/imap(client_input+0x72) [0x418212] -> /opt/dovecot2/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x5b) [0x7fe2e43494fb] -> /opt/dovecot2/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xbb) [0x7fe2e434a54b] -> /opt/dovecot2/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9) [0x7fe2e4349589] -> /opt/dovecot2/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7fe2e4349608] -> /opt/dovecot2/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7fe2e42eabc3] -> dovecot/imap(main+0x27b) [0x40c60b] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7fe2e3f4eead] -> dovecot/imap() [0x40c769] 2015-06-16T12:49:25.552150+02:00 sr-be-store07 dovecot: store107: imap( mihaiush(a)sunrise.ch) hCdxU6AY/wAKThJr: Fatal: master: service(imap): child 3923 killed with signal 6 (core dumped) IMAP session: # telnet store107 143 Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN] Dovecot ready. x login mihaiush(a)sunrise.ch XXX x OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SEARCH=FUZZY NOTIFY SPECIAL-USE QUOTA] Logged in x select virtual/all * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Label3 $Label2 $Label5 $Label1 $Label4 parsed attachment) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Label3 $Label2 $Label5 $Label1 $Label4 parsed attachment \*)] Flags permitted. * 1963 EXISTS * 1256 RECENT * OK [UNSEEN 2] First unseen. * OK [UIDVALIDITY 1434024437] UIDs valid * OK [UIDNEXT 2677] Predicted next UID * OK [HIGHESTMODSEQ 76] Highest x OK [READ-WRITE] Select completed (0.087 secs). x sort (reverse arrival) utf-8 text vpopmail Connection closed by foreign host. GDB: # gdb /opt/dovecot2/libexec/dovecot/imap /services/dovecot1/var/data/ sunrise.ch/m/mihaiush/core GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /opt/dovecot2/libexec/dovecot/imap...done. [New LWP 3923] warning: Can't read pathname for load map: Input/output error. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `dovecot/imap'. Program terminated with signal 6, Aborted. #0 0x00007fe2e3f62165 in raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) (gdb) bt full #0 0x00007fe2e3f62165 in raise () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00007fe2e3f653e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #2 0x00007fe2e4337135 in default_fatal_finish (type=<optimized out>, status=status@entry=0) at failures.c:202 backtrace = 0x2166c38 "/opt/dovecot2/lib/dovecot/libdovecot.so.0(+0x7c13f) [0x7fe2e433713f] -> /opt/dovecot2/lib/dovecot/libdovecot.so.0(+0x7c19e) [0x7fe2e433719e] -> /opt/dovecot2/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x"... #3 0x00007fe2e433719e in i_internal_fatal_handler (ctx=0x7fffa5660ea0, format=<optimized out>, args=<optimized out>) at failures.c:671 status = 0 #4 0x00007fe2e42e5555 in i_panic (format=format@entry=0x7fe2e468d068 "file %s: line %d (%s): assertion failed: (%s)") at failures.c:276 ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0, timestamp_usecs = 0} args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 0x7fffa5660f90, reg_save_area = 0x7fffa5660ed0}} #5 0x00007fe2e466fd32 in mail_index_map_lookup_seq_range (map=<optimized out>, first_uid=0, last_uid=<optimized out>, first_seq_r=<optimized out>, last_seq_r=<optimized out>) at mail-index-map.c:547 __FUNCTION__ = "mail_index_map_lookup_seq_range" #6 0x00007fe2e46817fa in mail_index_lookup_seq_range (view=<optimized out>, first_uid=<optimized out>, last_uid=<optimized out>, first_seq_r=first_seq_r@entry=0x7fffa5660fe8, last_seq_r=last_seq_r@entry =0x7fffa5660fec) at mail-index-view.c:513 No locals. #7 0x00007fe2e45f819c in mailbox_get_seq_range (box=<optimized out>, uid1=<optimized out>, uid2=<optimized out>, seq1_r=seq1_r@entry=0x7fffa5660fe8, seq2_r=seq2_r@entry=0x7fffa5660fec) at mailbox-get.c:11 No locals. #8 0x00007fe2e30c40b1 in uid_range_to_seqs (uid_range=uid_range@entry=0x7fffa56610b0, seq_range=seq_range@entry=0x25e4840, fctx=<error reading variable: Unhandled dwarf expression opcode 0xfa>, fctx=<error reading variable: Unhandled dwarf expression opcode 0xfa>) at fts-search.c:28 range = 0x2166738 i = <optimized out> count = <optimized out> seq1 = 41023456 seq2 = 0 #9 0x00007fe2e30c4963 in multi_add_lookup_result (args=0x2363d50, level=0x25e4840, fctx=0x23637e0, result=<optimized out>) at fts-search.c:138 br = 0x271f7e0 vuids = {arr = {buffer = 0x2166700, element_size = 8}, v = 0x2166700, v_modifiable = 0x2166700} orig_size = <optimized out> i = <optimized out> vbox = 0x2197c70 #10 fts_search_lookup_level_multi (and_args=<optimized out>, args=0x2363d50, fctx=0x23637e0) at fts-search.c:198 flags = FTS_LOOKUP_FLAG_AND_ARGS mailboxes_arr = {arr = {buffer = 0x2717670, element_size = 8}, v = 0x2717670, v_modifiable = 0x2717670} backend = <optimized out> j = 69 vbox = <optimized out> mailboxes = 0x27176a8 level = 0x25e4840 result = {pool = 0x2717650, box_results = 0x271f7a0} i = <optimized out> mailbox_count = 69 #11 fts_search_lookup_level (fctx=fctx@entry=0x23637e0, args=0x2363d50, and_args=and_args@entry=true) at fts-search.c:212 _data_stack_cur_id = 4 ret = <optimized out> #12 0x00007fe2e30c4c36 in fts_search_lookup (fctx=fctx@entry=0x23637e0) at fts-search.c:358 last_uid = 2676 seq1 = 0 seq2 = 0 __FUNCTION__ = "fts_search_lookup" #13 0x00007fe2e30c6def in fts_mailbox_search_init (t=<optimized out>, args=0x2363c90, sort_program=0x7fffa5661250, wanted_fields=<optimized out>, wanted_headers=<optimized out>) at fts-storage.c:240 ft = 0x23908a0 fbox = <optimized out> flist = <optimized out> ctx = 0x2695dc0 fctx = 0x23637e0 #14 0x000000000041dec0 in imap_search_start (ctx=ctx@entry=0x2195590, sargs=0x2363c90, sort_program=sort_program@entry=0x7fffa5661250) at imap-search.c:587 cmd = 0x2195450 __FUNCTION__ = "imap_search_start" #15 0x0000000000414565 in cmd_sort (cmd=0x2195450) at cmd-sort.c:141 ctx = 0x2195590 sargs = 0x2363c90 sort_program = {4097, MAIL_SORT_END, 56, MAIL_SORT_END, 3831136736, 32738, 35045104, MAIL_SORT_END, MAIL_SORT_ARRIVAL, MAIL_SORT_END, MAIL_SORT_ARRIVAL, MAIL_SORT_END} args = 0x216cb98 list_args = 0x216ccd0 charset = 0x216ce30 "utf-8" ret = <optimized out> #16 0x0000000000418aec in command_exec (cmd=cmd@entry=0x2195450) at imap-commands.c:167 hook = 0x216a120 diff = <optimized out> ---Type <return> to continue, or q <return> to quit--- finished = <optimized out> #17 0x0000000000417ad0 in client_command_input (cmd=cmd@entry=0x2195450) at imap-client.c:814 client = 0x2194850 command = <optimized out> __FUNCTION__ = "client_command_input" #18 0x0000000000417b64 in client_command_input (cmd=0x2195450) at imap-client.c:874 client = 0x2194850 command = <optimized out> __FUNCTION__ = "client_command_input" #19 0x0000000000417e45 in client_handle_next_command (remove_io_r=<synthetic pointer>, client=0x2194850) at imap-client.c:912 No locals. #20 client_handle_input (client=client@entry=0x2194850) at imap-client.c:924 _data_stack_cur_id = 3 ret = 2 remove_io = false handled_commands = false __FUNCTION__ = "client_handle_input" #21 0x0000000000418212 in client_input (client=0x2194850) at imap-client.c:966 cmd = <optimized out> output = 0x2195270 bytes = 46 __FUNCTION__ = "client_input" #22 0x00007fe2e43494fb in io_loop_call_io (io=0x2195360) at ioloop.c:501 ioloop = 0x2169760 t_id = 2 __FUNCTION__ = "io_loop_call_io" #23 0x00007fe2e434a54b in io_loop_handler_run_internal (ioloop=ioloop@entry=0x2169760) at ioloop-epoll.c:220 ctx = 0x216a490 events = 0xf53 event = 0x216b300 list = 0x216bef0 io = <optimized out> tv = {tv_sec = 1799, tv_usec = 999663} events_count = <optimized out> msecs = <optimized out> ret = 1 i = <optimized out> call = <optimized out> __FUNCTION__ = "io_loop_handler_run_internal" #24 0x00007fe2e4349589 in io_loop_handler_run (ioloop=ioloop@entry=0x2169760) at ioloop.c:548 No locals. #25 0x00007fe2e4349608 in io_loop_run (ioloop=0x2169760) at ioloop.c:525 __FUNCTION__ = "io_loop_run" #26 0x00007fe2e42eabc3 in master_service_run (service=0x21695f0, callback=callback@entry=0x421140 <client_connected>) at master-service.c:581 No locals. #27 0x000000000040c60b in main (argc=1, argv=0x2169390) at main.c:431 set_roots = {0x428c20, 0x0} login_set = {auth_socket_path = 0x2161060 "\210\020\026\002", postlogin_socket_path = 0x0, postlogin_timeout_secs = 60, callback = 0x420f80 <login_client_connected>, failure_callback = 0x4210f0 <login_client_failed>, request_auth_token = 1} service_flags = <optimized out> storage_service_flags = <optimized out> username = 0x0 c = <optimized out> (gdb) If necessary I can provide the mailbox, it has 7MB.

2 1

Mail to nowhere
by Steve Matzura 17 Jun '15

17 Jun '15

The next phase of my testing procedure involves the simple act of delivering mail to my test box. When I send a message to either a valid or relayed user at my remote machine's address, it never gets there. I know this virtually for sure because I'm tailing /var/log/maillog and nothing new has been added for the past hour. I'm thinking eventually my ISP will send the message back to me as undeliverable, and in that bounced message there may be some information about why it was undeliverable, but maybe someone has a thought as to why I'm not even seeing anything attempt to connect?

5 10

Re: Mail to nowhere
by Richard 17 Jun '15

17 Jun '15

> Date: Tuesday, June 16, 2015 07:05:43 PM -0400 > From: Steve Matzura <number6(a)noisynotes.com> > > On Tue, 16 Jun 2015 17:31:07 +0000, you wrote: > >> By default, most MTAs only listen to localhost, so don't accept >> externally derived mail. Try telnetting to port 25 on the machine >> from some place off your machine's network. If you get a "hang", >> it's likely a firewall issue. If you get a "connection refused" >> message it's probably that the MTA isn't listening on the external >> interface. >> >> If the MTA responds, do a manual delivery exchange and see what >> you get. >> >> If you're not seeing anything in your maillog, this is very likely >> an issue with your MTA's configuration, not dovecot. > > Now that I think about and concentrate on what you wrote, make > perfect sense. Add to that the fact that I received bounces from > my ISP that all said the same thing: Unable to establish an SMTP > connection. The return code was 4.5.1. A 4xx indicates a transient error (i.e., within configured limits, delivery retries will be attempted), 5xx is permanent (immediate reject, no retries), otherwise the specific number isn't all that telling. >From everything you've indicated, this is an MTA issue.

2 1

replication of a big mailbox
by Tim Kuijsten 16 Jun '15

16 Jun '15

I'm running dovecots replicator successfully on 33 mailboxes. There is only one mailbox (the biggest one, 167000 mails) that is not replicating successfully. After 10 minutes the "source" starts logging these messages: dovecot: dsync-local(foo(a)example.com): Error: dsync(imap.netsend.nl): I/O has stalled, no activity for 600 seconds dovecot: dsync-local(foo(a)example.com): Error: Timeout during state=sync_mails (send=mail_requests recv=changes) dovecot: dsync-local(foo(a)example.com): Error: Remote command process isn't dying, killing it dovecot: dsync-local(foo(a)example.com): Error: kill(17459, SIGKILL) failed: Operation not permitted dovecot: doveadm: Error: Pseudo-terminal will not be allocated because stdin is not a terminal. dovecot: doveadm: Error: dsync-remote(foo(a)example.com): Error: Couldn't lock //.dovecot-sync.lock: Timed out after 30 seconds dovecot: doveadm: Error: Pseudo-terminal will not be allocated because stdin is not a terminal. dovecot: doveadm: Error: dsync-remote(foo(a)example.com): Error: Couldn't lock //.dovecot-sync.lock: Timed out after 30 seconds The 30 seconds timeout keeps repeating, while the doveadm process on the destination keeps on going for hours and hours. Destination imap.netsend.nl is creating a lot of hardlinks (at least one hardlink per message) in .temp.... files. This takes about 1,5 hour, after that the number of hardlinks decline again. But it looks like the connection with source is already lost. I've patched the source to overcome the timeout by changing it from 10 minutes to 3 hours[1] but now I'm running into a memory error: dovecot: dsync-local(foo(a)example.com): Error: dsync(imap.netsend.nl): I/O has stalled, no activity for 10800 seconds dovecot: dsync-local(foo(a)example.com): Error: Timeout during state=sync_mails (send=done recv=mails) dovecot: dsync-local(foo(a)example.com): Error: mremap_anon(211677184) failed: Cannot allocate memory The allocate memory error shows up every 10 seconds. Next I'll try to look into the memory allocation code, but maybe someone already has a better idea? Thanks. -Tim Config on source: # 2.2.15: /etc/dovecot/dovecot.conf # OS: OpenBSD 5.7 amd64 auth_debug = yes auth_mechanisms = plain login auth_verbose = yes dsync_remote_cmd = /usr/bin/ssh imap.netsend.nl first_valid_gid = 30000 first_valid_uid = 30000 last_valid_gid = 30100 last_valid_uid = 30100 mail_debug = yes mail_home = /srv/vmail/%d/%n mail_location = maildir:~/Maildir mail_plugins = " notify replication" passdb { args = /passwd driver = passwd-file } passdb { args = /passwd_smtponly driver = passwd-file } plugin { mail_replica = remoteprefix:imap.netsend.nl } protocols = imap lmtp service aggregator { fifo_listener replication-notify-fifo { mode = 0666 } unix_listener replication-notify { mode = 0666 } } service auth-worker { chroot = /etc/mail unix_listener /etc/mail/auth-worker { mode = 0600 user = $default_internal_user } user = $default_internal_user } service auth { chroot = /etc/mail unix_listener /var/spool/postfix/private/dovecot-auth { group = _postfix mode = 0600 user = _postfix } } service imap-login { inet_listener imap { port = 0 } process_min_avail = 1 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = _postfix mode = 0600 user = _postfix } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } ssl = required ssl_cert = </etc/ssl/private/2015/chain.crt ssl_cipher_list = ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA; ssl_dh_parameters_length = 2048 ssl_key = </etc/ssl/private/the.key ssl_prefer_server_ciphers = yes ssl_protocols = !SSLv2 !SSLv3 userdb { args = /passwd default_fields = uid=30000 gid=30000 home=/srv/vmail/%d/%n/./ driver = passwd-file } valid_chroot_dirs = /srv/vmail verbose_ssl = yes protocol imap { mail_max_userip_connections = 20 } config on destination imap.netsend.nl: # 2.2.15: /etc/dovecot/dovecot.conf # OS: OpenBSD 5.7 amd64 auth_debug = yes auth_mechanisms = plain login auth_verbose = yes first_valid_gid = 30000 first_valid_uid = 30000 last_valid_gid = 30100 last_valid_uid = 30100 mail_debug = yes mail_home = /srv/vmail/%d/%n mail_location = sdbox:~/dbox passdb { args = /passwd driver = passwd-file } passdb { args = /passwd_smtponly driver = passwd-file } protocols = imap lmtp service auth-worker { chroot = /etc/mail unix_listener /etc/mail/auth-worker { mode = 0600 user = $default_internal_user } user = $default_internal_user } service auth { chroot = /etc/mail unix_listener /var/spool/postfix/private/dovecot-auth { group = _postfix mode = 0600 user = _postfix } } service imap-login { inet_listener imap { port = 0 } process_min_avail = 1 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = _postfix mode = 0600 user = _postfix } } ssl = required ssl_cert = </etc/ssl/private/2015/chain.crt ssl_cipher_list = ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA; ssl_dh_parameters_length = 2048 ssl_key = </etc/ssl/private/the.key ssl_prefer_server_ciphers = yes ssl_protocols = !SSLv2 !SSLv3 userdb { args = /passwd default_fields = uid=30000 gid=30000 home=/srv/vmail/%d/%n/./ driver = passwd-file } valid_chroot_dirs = /srv/vmail verbose_ssl = yes protocol imap { mail_max_userip_connections = 20 } [1] --- src/doveadm/dsync/dsync-ibc-stream.c 2014-09-29 13:07:07.000000000 +0200 +++ src/doveadm/dsync/dsync-ibc-stream.c2 2015-06-15 12:28:56.000000000 +0200 @@ -24,7 +24,7 @@ #include <stdlib.h> -#define DSYNC_IBC_STREAM_TIMEOUT_MSECS (60*10*1000) +#define DSYNC_IBC_STREAM_TIMEOUT_MSECS (60*180*1000) #define DSYNC_IBC_STREAM_OUTBUF_THROTTLE_SIZE (1024*128) #define DSYNC_PROTOCOL_VERSION_MAJOR 3

1 0