December 2019 - dovecot

dovecot full text search
by Wojciech Puchar 16 Dec '19

16 Dec '19

what FTP module should i use instead of squat that is probably no longer supported or no longer at all? i want to upgrade my dovecot installation. it currently uses squat but i found it often crashes on FTS on large mailboxes. i found "xapian" addon for dovecot but while it works excellent AFTER database is created, i found it need like 20 or so minutes to index less than 10GB of mails and while doing this - generate many tens of megabytes/s constant write traffic on it's database files. Excellent way of killing SSD. something must be broken. my config is plugin { plugin = fts fts_xapian fts = xapian fts_xapian = partial=2 full=20 verbose=0 fts_autoindex = yes fts_enforced = yes # fts_autoindex_exclude = \Junk # fts_autoindex_exclude2 = \Trash } any ideas?

3 3

Bug: indexer-worker segfaults with fts_xapian 1.2.5
by Martynas Bendorius 16 Dec '19

16 Dec '19

Core was generated by `dovecot/indexer-worker'. Program terminated with signal 11, Segmentation fault. #0 0x00007f30f7ad056d in __exchange_and_add (__val=-1, __mem=0xfffffffffffffff8) at /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/ext/atomicity.h:49 49 { return __atomic_fetch_add(__mem, __val, __ATOMIC_ACQ_REL); } (gdb) bt full #0 0x00007f30f7ad056d in __exchange_and_add (__val=-1, __mem=0xfffffffffffffff8) at /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/ext/atomicity.h:49 No locals. #1 __exchange_and_add_dispatch (__val=-1, __mem=0xfffffffffffffff8) at /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/ext/atomicity.h:82 No locals. #2 std::string::_Rep::_M_dispose (this=0xffffffffffffffe8, __a=...) at /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.h:246 No locals. #3 0x00007f30f7b3407e in _M_dispose (__a=..., this=<optimized out>) at /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.tcc:254 No locals. #4 std::string::assign (this=this@entry=0x55c7b93e0168, __str="return-path") at /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.tcc:250 __a = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>} #5 0x00007f30fa8776ce in operator= (__str="return-path", this=0x55c7b93e0168) at /usr/include/c++/4.8.2/bits/basic_string.h:547 No locals. #6 fts_backend_xapian_update_set_build_key (_ctx=0x55c7b93e0140, key=0x7ffc1ba2c380) at fts-backend-xapian.cpp:303 ctx = 0x55c7b93e0140 i = <optimized out> f2 = "return-path" backend = <optimized out> field = 0x55c7b943fb5f "RETURN-PATH" j = 11 #7 0x00007f30fb0b8cda in fts_backend_update_set_build_key (ctx=0x55c7b93e0140, key=key@entry=0x7ffc1ba2c380) at fts-api.c:174 __func__ = "fts_backend_update_set_build_key" #8 0x00007f30fb0ba243 in fts_build_mail_header (block=0x7ffc1ba2c340, block=0x7ffc1ba2c340, ctx=0x7ffc1ba2c3b0) at fts-build-mail.c:173 hdr = <optimized out> key = {uid = 6006200, type = FTS_BACKEND_BUILD_KEY_HDR, part = 0x55c7b935eed0, hdr_name = 0x55c7b943fb5f "RETURN-PATH", body_content_type = 0x0, body_content_disposition = 0x0} ret = <optimized out> #9 fts_build_mail_real (may_need_retry_r=0x7ffc1ba2c2f3, retriable_err_msg_r=0x7ffc1ba2c300, mail=0x55c7b93dcff8, update_ctx=0x55c7b93e0140) at fts-build-mail.c:568 block = {part = 0x55c7b935eed0, hdr = 0x55c7b93d7f18, data = 0x55c7b938b8a8 "", size = 0} ret = <optimized out> input = 0x55c7b93d78c0 raw_block = {part = 0x55c7b935eed0, hdr = 0x55c7b93d80b0, data = 0x0, size = 0} skip_body = false ctx = {mail = 0x55c7b93dcff8, update_ctx = 0x55c7b93e0140, content_type = 0x0, content_disposition = 0x0, body_parser = 0x0, word_buf = 0x0, ---Type <return> to continue, or q <return> to quit--- pending_input = 0x0, cur_user_lang = 0x0} prev_part = 0x55c7b935eed0 parser = 0x55c7b93d7b38 decoder = 0x55c7b93d7f00 parts = 0x7ffc1ba2c414 body_part = false body_added = false binary_body = <optimized out> error = 0x5ba5b8 <Address 0x5ba5b8 out of bounds> #10 fts_build_mail (update_ctx=0x55c7b93e0140, mail=mail@entry=0x55c7b93dcff8) at fts-build-mail.c:617 _data_stack_cur_id = 6 attempts = 2 retriable_err_msg = 0x11e900729 <Address 0x11e900729 out of bounds> may_need_retry = false #11 0x00007f30fb0c1102 in fts_mail_index (_mail=0x55c7b93dcff8) at fts-storage.c:550 ft = 0x55c7b9396880 flist = 0x55c7b938b8a8 pmail = 0x55c7b93dcff8 #12 fts_mail_precache (_mail=0x55c7b93dcff8) at fts-storage.c:571 _data_stack_cur_id = 5 mail = 0x55c7b93dcff8 fmail = <optimized out> ft = 0x55c7b9396880 __func__ = "fts_mail_precache" #13 0x00007f30fc1b7d64 in mail_precache (mail=0x55c7b93dcff8) at mail.c:432 _data_stack_cur_id = 4 p = 0x55c7b93dcff8 #14 0x000055c7b8b4c844 in index_mailbox_precache (conn=<optimized out>, box=0x55c7b938ef18) at master-connection.c:102 counter = 0 max = 21569 percentage_sent = 0 storage = <optimized out> status = {messages = 21569, recent = 0, unseen = 0, uidvalidity = 1462447525, uidnext = 6027769, first_unseen_seq = 0, first_recent_uid = 6027758, last_cached_seq = 0, highest_modseq = 0, highest_pvt_modseq = 0, keywords = 0x0, permanent_flags = 0, flags = 0, permanent_keywords = false, allow_new_keywords = false, nonpermanent_modseqs = false, no_modseq_tracking = false, have_guids = true, have_save_guids = true, have_only_guid128 = false} uids = <optimized out> username = 0x55c7b9385988 "msg(a)domain.com" first_uid = 6006200 ---Type <return> to continue, or q <return> to quit--- percentage_str = "\003\000\000" percentage = <optimized out> error = MAIL_ERROR_NONE trans = 0x55c7b9390480 ctx = 0x55c7b93d15f0 last_uid = 6006200 ret = 0 box_vname = 0x55c7b938f280 "INBOX.lfd.SSH login alerts" errstr = <optimized out> search_args = 0x0 mail = 0x55c7b93dcff8 metadata = {guid = '\000' <repeats 15 times>, virtual_size = 0, physical_size = 0, first_save_date = 0, cache_fields = 0x0, precache_fields = (MAIL_FETCH_STREAM_HEADER | MAIL_FETCH_STREAM_BODY | MAIL_FETCH_RECEIVED_DATE | MAIL_FETCH_SAVE_DATE | MAIL_FETCH_PHYSICAL_SIZE | MAIL_FETCH_VIRTUAL_SIZE | MAIL_FETCH_UIDL_BACKEND | MAIL_FETCH_GUID | MAIL_FETCH_POP3_ORDER), backend_ns_prefix = 0x0, backend_ns_type = (unknown: 0)} seq = <optimized out> #15 index_mailbox (user=<optimized out>, user=<optimized out>, what=<optimized out>, max_recent_msgs=<optimized out>, mailbox=<optimized out>, conn=0x55c7b9370df0) at master-connection.c:205 box = 0x55c7b938ef18 status = {messages = 463652304, recent = 32764, unseen = 1, uidvalidity = 0, uidnext = 4225857431, first_unseen_seq = 32560, first_recent_uid = 4226053086, last_cached_seq = 32560, highest_modseq = 94316285707297, highest_pvt_modseq = 94316294200320, keywords = 0x7f30fbe17797 <__x86_return_thunk+5>, permanent_flags = 4226053495, flags = 32560, permanent_keywords = true, allow_new_keywords = true, nonpermanent_modseqs = true, no_modseq_tracking = false, have_guids = true, have_save_guids = false, have_only_guid128 = false} errstr = <optimized out> ret = 0 ns = <optimized out> path = 0x55c7b935e7f8 "/home/username/imap/domain.com/msg/Maildir/.INBOX.lfd.SSH login alerts" error = 21569 sync_flags = <optimized out> #16 master_connection_input_line (line=<optimized out>, conn=0x55c7b9370df0) at master-connection.c:247 str = <optimized out> max_recent_msgs = 0 ret = <optimized out> args = <optimized out> input = {parent_event = 0x0, module = 0x55c7b8b4d05f "mail", service = 0x55c7b8b4d050 "indexer-worker", username = 0x55c7b935d038 "msg(a)domain.com", session_id = 0x0, session_id_prefix = 0x55c7b935d063 "MAYmHNGX9l1rviwA+Lg9Sw", session_create_time = 0, local_ip = {family = 0, u = {ip6 = {__in6_u = { __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ip4 = {s_addr = 0}}}, remote_ip = { family = 0, u = {ip6 = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ip4 = { s_addr = 0}}}, local_port = 0, remote_port = 0, userdb_fields = 0x0, flags_override_add = (unknown: 0), flags_override_remove = (unknown: 0), ---Type <return> to continue, or q <return> to quit--- no_userdb_lookup = false, debug = false, conn_secured = false, conn_ssl_secured = false} service_user = 0x55c7b93732e8 user = 0x55c7b9385868 error = 0x0 #17 master_connection_input (conn=0x55c7b9370df0) at master-connection.c:287 _data_stack_cur_id = 3 line = <optimized out> #18 0x00007f30fbed69af in io_loop_call_io (io=0x55c7b9370e30) at ioloop.c:718 ioloop = 0x55c7b9365d70 t_id = 2 __func__ = "io_loop_call_io" #19 0x00007f30fbed8413 in io_loop_handler_run_internal (ioloop=ioloop@entry=0x55c7b9365d70) at ioloop-epoll.c:222 ctx = 0x55c7b9366ea0 list = 0x55c7b9370e90 io = <optimized out> tv = {tv_sec = 2147483, tv_usec = 0} events_count = <optimized out> msecs = <optimized out> ret = 1 i = 0 call = <optimized out> __func__ = "io_loop_handler_run_internal" #20 0x00007f30fbed6ab9 in io_loop_handler_run (ioloop=ioloop@entry=0x55c7b9365d70) at ioloop.c:770 __func__ = "io_loop_handler_run" #21 0x00007f30fbed6cf8 in io_loop_run (ioloop=0x55c7b9365d70) at ioloop.c:743 __func__ = "io_loop_run" #22 0x00007f30fbe46d83 in master_service_run (service=0x55c7b9365c00, callback=callback@entry=0x55c7b8b4c3e0 <client_connected>) at master-service.c:809 No locals. #23 0x000055c7b8b4c21e in main (argc=1, argv=0x55c7b9365960) at indexer-worker.c:77 storage_service_flags = (MAIL_STORAGE_SERVICE_FLAG_USERDB_LOOKUP | MAIL_STORAGE_SERVICE_FLAG_TEMP_PRIV_DROP | MAIL_STORAGE_SERVICE_FLAG_NO_IDLE_TIMEOUT) c = <optimized out> -- Best regards, Martynas Bendorius

2 1

Dovecot v2.3.9.2 released
by aki.tuomi＠dovecot.fi 15 Dec '19

15 Dec '19

We are pleased to release v2.3.9.2 of Dovecot. Please find it from locations below https://dovecot.org/releases/2.3/dovecot-2.3.9.2.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.9.2.tar.gz.sig Binary packages in https://repo.dovecot.org/ Docker images in https://hub.docker.com/r/dovecot/dovecot --- - Mails with empty From/To headers can also cause crash in push notification drivers. --- Aki Tuomi Open-Xchange oy

2 1

Bug: icu linking issue in fts plugin
by Martynas Bendorius 15 Dec '19

15 Dec '19

FTS plugin seems not to be linking dynamic libraries correctly: [root@server ~]# ldd /usr/lib/dovecot/lib20_fts_plugin.so linux-vdso.so.1 => (0x00007ffd2eb95000) libicui18n.so.64 => not found libicuuc.so.64 => not found libicudata.so.64 => not found libc.so.6 => /lib64/libc.so.6 (0x00007f1cb5566000) /lib64/ld-linux-x86-64.so.2 (0x00007f1cb5b73000) ICU is in custom directory /usr/local/icu, it's not set in ld.so.conf, so, the linker would need to get this added automatically (that's how it's handled in PHP, for example): -Wl,-rpath=/usr/local/icu/lib Thank you! -- Best regards, Martynas Bendorius

1 0

Solr commit and optimize: which user?
by John Gateley 15 Dec '19

15 Dec '19

Hi, I have Solr FTR working with dovecot, and need to do the commit and optimize recommended here: https://wiki.dovecot.org/Plugins/FTS/Solr Should this run as root, or as each mail user. The documentation above implies, but doesn't state, that it is root. The docs here are similar: http://grimore.org/networking/dovecot/full_text_search But the docs here say it has to be done for each user: http://www.unixsamurai.com/dovecot-full-text-search-jetty-solr/ I don't know Solr well enough to answer. Which is correct? Thanks John

2 1

Parsing variables in config files
by Eudald Valcàrcel Lacasa 15 Dec '19

15 Dec '19

Hello, I'm trying to set up a mailbox for a bunch of domains. To do so I'm running some docker containers (I know I can use multidomain set up and I'm doing so, but I need to have some domains on different containers for specific reasons). In order to keep it all clean, I want to use different PostgreSQL databases for each container, and I'm running the container with an environment file containing database parameters, such as: DB_USER DB_HOST DB_NAME I've been trying to pass these parameters to dovecot's configuration, but they don't get parsed and I end up with messages like: dovecot: auth: Error: pgsql(%{env:DB_HOST}): Connect failed to database %{env:DB_NAME} I've tried to pass variables alone, using import_environment = DB_HOST DB_NAME DB_USER, but I'm stuck at the same errors. Is there anything I could do to fix this? Thank you! Eudald

2 2

Doveadm HTTP API and CORS
by Peter Chiochetti 13 Dec '19

13 Dec '19

While toying around with Doveadmin HTTP api I noticed, there is not much hype around it. Let's change that. Doveadmin REST API so to speak provides a web server. When doing /ajax/ requests, modern browsers block that, unless the response comes with a specific CORS header. Here how to work around with a proxy: > # /etc/dovecot/local.conf > doveadm_allowed_commands = fetch > doveadm_api_key = secret > service doveadm { > inet_listener http { > port = 8084 > } > } > # secret64 = echo -n secret | base64 > # /etc/apache/.../some-site.conf > ProxyVia Block > <Location /doveadm/v1> > RequestHeader set Authorization "X-Dovecot-API secret64" > ProxyPass http://localhost:8084/doveadm/v1 retry=0 timeout=5 > ProxyPassReverse http://localhost:8084/doveadm/v1 > </Location> Intranet only this might be good enough, TLS with username/password Basic-Authentication passthrough might be used for better security. Shameless plug: taken from https://gist.github.com/hungerburg/00d582bf1a6bf3c622797bf5e759f75b -- peter

2 1

BUG: panic when using fs:posix as dict for acl_shared_dict
by Marco Giunta 13 Dec '19

13 Dec '19

Hi, I'm using Dovecot 2.2.33.2 on a RHEL 7, new installation. When I use fs:posix as dict for acl_shared_dict, like in Dovecot wiki (https://wiki.dovecot.org/SharedMailboxes/ClusterSetup) doveadm-server crash with error: # doveadm acl set -u USERNAME FOLDER user=DEST_USERNAME lookup read write-seen doveadm(USERNAME): Panic: file dict-fs.c: line 127 (fs_dict_iterate_init): assertion failed: ((flags & DICT_ITERATE_FLAG_RECURSE) == 0) doveadm(USERNAME): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x9f3de) [0x7f0e4a4b23de] -> /usr/lib64/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7f0e4a4b244a] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f0e4a44377c] -> /usr/lib64/dovecot/libdovecot.so.0(+0x31688) [0x7f0e4a444688] -> /usr/lib64/dovecot/libdovecot.so.0(dict_iterate_init_multiple+0x4d) [0x7f0e4a47cadd] -> /usr/lib64/dovecot/libdovecot.so.0(dict_iterate_init+0x29) [0x7f0e4a47cb89] -> /usr/lib64/dovecot/lib01_acl_plugin.so(acl_lookup_dict_rebuild+0x3e1) [0x7f0e49a40371] -> /usr/lib64/dovecot/lib01_acl_plugin.so(acl_backend_vfile_acllist_rebuild+0x488) [0x7f0e49a3dd18] -> /usr/lib64/dovecot/lib01_acl_plugin.so(acl_backend_vfile_object_update+0x3c7) [0x7f0e49a3e867] -> /usr/lib64/dovecot/lib01_acl_plugin.so(acl_mailbox_update_acl+0x68) [0x7f0e49a41e28] -> /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so(+0x2c11) [0x7f0e48da1c11] -> /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so(+0x3060) [0x7f0e48da2060] -> doveadm(+0x2b41c) [0x556f1280b41c] -> doveadm(+0x2c01a) [0x556f1280c01a] -> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x23b) [0x556f1280ce7b] -> doveadm(doveadm_cmd_run_ver2+0x50c) [0x556f1281c73c] -> doveadm(doveadm_cmd_try_run_ver2+0x37) [0x556f1281c7d7] -> doveadm(main+0x1e4) [0x556f127fb944] -> /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f0e4a071c05] -> doveadm(+0x1bd35) [0x556f127fbd35] Aborted Attached coredump and configuration. I'm not a programmer, but seems that assert is raised by 'fs_dict_iterate_init' function inside 'src/lib-dict-extra/dict-fs.c': static struct dict_iterate_context * fs_dict_iterate_init(struct dict *_dict, const char *const *paths, enum dict_iterate_flags flags) { ... /* these flags are not supported for now */ i_assert((flags & DICT_ITERATE_FLAG_RECURSE) == 0); ... because it is called by 'acl_lookup_dict_iterate_read' function in file 'src/plugins/acl/acl-lookup-dict.c' static void acl_lookup_dict_iterate_read(struct acl_lookup_dict_iter *iter) { ... dict_iter = dict_iterate_init(iter->dict->dict, prefix, DICT_ITERATE_FLAG_RECURSE); ... with DICT_ITERATE_FLAG_RECURSE set. Same problem also with Dovecot 2.3.0. Thanks, Marco -- Marco Giunta - ITCS SysAdmin Via Bonomea, 265 34136 - Trieste, Italy Tel: +39-040-3787-503 Fax: +39-040-3787-244

4 6

Is is possible to log TLS client versions
by Götz Reinicke 13 Dec '19

13 Dec '19

Hi, is it possible to log the version of TLS a clients is using to connect to dovecot? I’v not found any switch to do that. Thanks and regards . Götz

2 2

CVE-2019-19722: Critical vulnerability in Dovecot
by Aki Tuomi 13 Dec '19

13 Dec '19

Open-Xchange Security Advisory 2019-12-13 Product: Dovecot IMAP/POP3 Server Vendor: OX Software GmbH Internal reference: DOV-3719 Vulnerability type: NULL Pointer Dereference (CWE-476) Vulnerable version: 2.3.9 Vulnerable component: push notification driver Report confidence: Confirmed Solution status: Fixed by Vendor Fixed version: 2.3.9.1 Researcher credits: Frederik Schwan, Michael Stilkerich Vendor notification: 2019-12-10 Solution date: 2019-12-12 Public disclosure: 2019-12-13 CVE reference: CVE-2019-19722 CVSS: 5.3 (CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L/E:F/RL:O/RC:C) Vulnerability Details: Mail with group address as sender will cause a signal 11 crash in push notification drivers. Group address as recipient can cause crash in some drivers. Risk: Repeated delivery attempts are made for the problematic mail, causing queueing in MTA. Steps to reproduce: 1. Configure dovecot with push notifications enabled, such as OX push notification driver. This can also be observed with 3rd party plugin XAPS. 2. Send mail a group address as sender Solution: Operators should update to the latest Patch Release.

2 1