January 2019 - dovecot

Re: [FTS Xapian] Beta release
by Tuomi, Aki 14 Jan '19

14 Jan '19

You copied your Makefile.am there. Stephan made you a working version, can you try that? (sorry for dup) Aki -------- Original message --------From: Joan Moreau <jom(a)grosjo.net> Date: 13/01/2019 21:39 (GMT+02:00) To: Stephan Bosch <stephan(a)rename-it.nl> Cc: Aki Tuomi <aki.tuomi(a)open-xchange.com> Subject: Re: [FTS Xapian] Beta release I used the skeleton from Aki : https://github.com/grosjo/fts-xapian However, when I try to act as a visitor, I reach teh follwoing error: # autoreconf -vi autoreconf: Entering directory `.' autoreconf: configure.ac: not using Gettext autoreconf: running: aclocal -I m4 autoreconf: configure.ac: tracing autoreconf: running: libtoolize --copy libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. libtoolize: copying file './ltmain.sh' libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. libtoolize: copying file 'm4/libtool.m4' libtoolize: copying file 'm4/ltoptions.m4' libtoolize: copying file 'm4/ltsugar.m4' libtoolize: copying file 'm4/ltversion.m4' libtoolize: copying file 'm4/lt~obsolete.m4' autoreconf: running: /usr/bin/autoconf autoreconf: running: /usr/bin/autoheader autoreconf: running: automake --add-missing --copy --no-force configure.ac:9: installing './compile' configure.ac:11: installing './config.guess' configure.ac:11: installing './config.sub' configure.ac:7: installing './install-sh' configure.ac:7: installing './missing' src/Makefile.am: installing './depcomp' /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL /usr/share/automake-1.16/am/depend2.am: The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX' /usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 'autoconf' again src/Makefile.am: error: C++ source seen but 'CXX' is undefined src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX' src/Makefile.am: to 'configure.ac' and run 'autoconf' again. src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no program or src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo) autoreconf: automake failed with exit status: 1 On 2019-01-13 20:24, Stephan Bosch wrote: Oh, right, a distribution tarball doesn't include some of the necessary files for your repository like autogen.sh and .gitignore. The attached tarball includes all those and is ready for `git init`. The previous tarball was made with `make distcheck` from this one. Regards, Stephan. Op 13/01/2019 om 20:14 schreef Stephan Bosch: Hi Joan, Op 13/01/2019 om 19:03 schreef Aki Tuomi: Yes, from compiling point of view it is done. Unfortunately what is not done is all the other work involved, such as fixing all the inevitable bugs it has and maintaining it. We do not want, at this moment, take up maintaining and developing yet another FTS plugin as we have plenty of things to do already. I invite you to setup your own repository and provide this plugin from there, being the maintainer of this plugin. We can add a link to your plugin on our FTS page so people can also find it. There are other plugins like this, e.g. https://github.com/st3fan/dovecot-xaps-plugin I turned the code you provided into a separate plugin package. The distribution tarball is attached. Notable changes: - Added example copyright headers and COPYING and AUTHORS files. You should modify those to your preference. - Added README and INSTALL files (in markdown using Pandoc). Those need to be amended with details. - Amended the plugin code to display a debug message with the plugin name and version upon plugin load. I advise you to turn this into a git repository and continue from there. I do not recommend releasing this plugin with the -fpermissive flag and the resulting warning as it is now. But I'm assuming this is still a work in progress, so that is OK. Regards, Stephan. On 13 January 2019 at 19:52 Joan Moreau <jom(a)grosjo.net> wrote: The only point here of this fts-xapian is to get rid of solr (because it is just a nightmare to setup) and squat (because it is considere obsolete). I already sent the changed in configure.ac, makefile.am, etc.. in order to include it in the dovecot, and it compiles properly The only remaining point is to push it in hte git (yes, everything is already done) On 2019-01-13 18:45, Aki Tuomi wrote: On 13 January 2019 at 17:05 Joan Moreau via dovecot <dovecot(a)dovecot.org> wrote: Hi Please find attached the beta release of FTS Xapian, with the objective to replace fts_squat that is being deprecated. Configuration is exactly the same as for fts_squat: plugin { plugin = fts fts_xapian (...) fts = xapian fts_autoindex = yes fts_enforced = yes fts_xapian = partial=2 full=20 This is installed on my production server (>120Gb of mailboxes), and I will observe it during the coming days. I will definitely appreciate that this is added in the core git of docevot, in order to have a versionning of it, to remove squat and let basic users able to avoid Solr alternative as much as possible. Thanks JM Hi! I still recommend you setup a, say, github repository for your plugin. We are not able to currently include your work in dovecot core as it is more work than just pushing the code into the repo. Maybe it can be included in the future. If you want, I can help you in setting up the required configuration scripts and such to make it possible to compile it as plugin. Then anyone can download it and install it for their dovecot, even if dovecot itself has been installed from packages, and also makes it possible for package maintainers to consider including it in distributions. Aki

6 22

What does the mdbox_rotate_size influence?
by Marc Roos 14 Jan '19

14 Jan '19

I wondered why mdbox_rotate_size is 2MB by default. I thought if I increased it to 16MB, maybe there would be less disk io, but I dont see any difference. Furthermore I read in some thread that increasing it from 2MB, could give problems when deleting messages, can someone explain this? Does anyone have this on a cephfs mount, does it make sense to set this to 4MB? [@test2 ~]# pr -m -t mail04-mdbox-vdb-append-64kb-4MB-6.log mail04-mdbox-vdb-append-64kb-16MB-7.log |less Logi Sele Appe Logi Sele Appe 100% 100% 100% 100% 100% 100% 1 1 6 1/ 1 1 1 8 1/ 1 0 0 9 1/ 1 0 0 9 1/ 1 0 0 7 1/ 1 0 0 7 1/ 1 0 0 10 1/ 1 0 0 8 1/ 1 0 0 7 1/ 1 0 0 10 1/ 1 0 0 8 1/ 1 0 0 9 1/ 1 0 0 9 1/ 1 0 0 10 1/ 1 0 0 10 1/ 1 0 0 9 1/ 1 0 0 9 1/ 1 0 0 7 1/ 1 0 0 11 1/ 1 0 0 8 1/ 1 29 3 1093 ms/cmd avg 37 1 1110 ms/cmd avg Logi Sele Appe Logi Sele Appe 100% 100% 100% 100% 100% 100% 0 0 10 1/ 1 0 0 8 1/ 1 0 0 9 1/ 1 0 0 8 1/ 1 0 0 10 1/ 1 0 0 10 1/ 1 0 0 9 1/ 1 0 0 10 1/ 1 0 0 11 1/ 1 0 0 8 1/ 1 0 0 9 1/ 1 0 0 9 1/ 1 0 0 8 1/ 1 0 0 6 1/ 1 0 0 11 1/ 1 0 0 9 1/ 1 0 0 9 1/ 1 0 0 7 1/ 1 0 0 9 1/ 1 0 0 9 1/ 1 0 0 1063 ms/cmd avg 0 0 1202 ms/cmd avg Logi Sele Appe Logi Sele Appe 100% 100% 100% 100% 100% 100% 0 0 8 1/ 1 0 0 8 1/ 1 0 0 10 1/ 1 0 0 10 1/ 1 0 0 7 1/ 1 0 0 11 1/ 1 0 0 10 1/ 1 0 0 10 1/ 1 0 0 10 1/ 1 0 0 10 1/ 1 0 0 11 1/ 1 0 0 9 1/ 1 0 0 7 1/ 1 0 0 8 1/ 1 0 0 11 1/ 1 0 0 9 1/ 1 0 0 8 1/ 1 0 0 10 1/ 1 0 0 10 1/ 1 0 0 7 1/ 1 0 0 1077 ms/cmd avg 0 0 1060 ms/cmd avg Logi Sele Appe Logi Sele Appe 100% 100% 100% 100% 100% 100% 0 0 7 1/ 1 0 0 9 1/ 1 0 0 9 1/ 1 0 0 10 1/ 1 0 0 6 1/ 1 0 0 6 1/ 1 0 0 9 1/ 1 0 0 7 1/ 1 0 0 8 1/ 1 0 0 9 1/ 1 0 0 10 1/ 1 0 0 10 1/ 1 0 0 9 1/ 1 0 0 9 1/ 1 0 0 7 1/ 1 0 0 10 1/ 1 0 0 9 1/ 1 0 0 9 1/ 1 0 0 7 1/ 1 0 0 9 1/ 1 0 0 1246 ms/cmd avg 0 0 1138 ms/cmd avg ... ... ... ... 0 0 8 1/ 1 0 0 9 1/ 1 0 0 6 1/ 1 0 0 8 1/ 1 0 0 6 1/ 1 0 0 7 1/ 1 0 0 7 1/ 1 0 0 8 1/ 1 0 0 5 1/ 1 0 0 9 1/ 1 0 0 5 1/ 1 0 0 10 1/ 1 0 0 6 1/ 1 0 0 11 1/ 1 0 0 7 1/ 1 0 0 9 1/ 1 0 0 6 1/ 1 0 0 9 1/ 1 0 0 7 1/ 1 0 0 10 1/ 1 0 0 1596 ms/cmd avg 0 0 1126 ms/cmd avg Logi Sele Appe Logi Sele Appe 100% 100% 100% 100% 100% 100% 0 0 7 1/ 1 0 0 8 1/ 1 0 0 1583 ms/cmd avg 0 0 1140 ms/cmd avg Totals: Totals: Logi Sele Appe Logi Sele Appe 100% 100% 100% 100% 100% 100% 1 1 1978 1 1 2021

1 0

mdbox import error from read-only filesystem
by hby 14 Jan '19

14 Jan '19

Dovecot version: 2.2.34 doveadm import tries to call open() on the source indexes/logs of mdbox data, but even if it should work as it is just a read-related call, it fails on read-only filesystems. The main use case for read-only filesystem is restoring from backup: the massive deduplicated backup data is exposed using FUSE. For deduplication to work efficiently, we have to backup the whole mail storage and thus the only way to handle the restore to mount the whole data. I would like to ask for some proper solution for this problem. Suggestions: first, the doveadm import should never modify the indexes, and second: the open() calls should respect the read-only filesystem and should fall back to "r" flag when using open. The problem goes further when using SIS (which goes beyond multiple terrabytes). The only workaround currently is to export the whole mdbox data to a writeable storage, and parametrize the import command to use that and the mounted SIS data separately for import...which is just problematic, ugly and error prune. Thank you for the continous work on dovecot, I hope you get this feedback in a good way.

2 2

Freebsd: Fatal error - Support not compiled in for passdb driver 'sql'
by Graham S. Jarvis 14 Jan '19

14 Jan '19

Hello, I have the following fatal error when I uncomment #!include auth-sql.conf.ext in .../conf.d/10-auth.conf dovecot: auth: Fatal: Support not compiled in for passdb driver 'sql' I have installed via 'pkg'. Do I have to install the ports tree to get sql support ? Thanks, -Graham-

4 9

managesieve configuration
by Dominik Menke 14 Jan '19

14 Jan '19

Hello list, I'm trying to get pidgeonhole/managesieve running, and I'm stuck connecting clients to the server (Dovecot 2.2.33.2-1ubuntu4.1 on an Ubnutu 18.04 machine). So far, my config looks like this: protocols = imap lmtp sieve disable_plaintext_auth = yes auth_mechanisms = plain login scram-sha-1 service managesieve-login { inet_listener sieve { port = 4190 ssl = yes } service_count = 1 } service managesieve { process_limit = 256 } protocol sieve { managesieve_max_line_length = 65536 } (please let me know if you need more details). When I connect via openssl s_client -connect $myserver:4190 I get the following prompt (after the usual certificate prompt): "IMPLEMENTATION" "Dovecot (Ubuntu) Pigeonhole" "SIEVE" "fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext vacation-seconds imapsieve vnd.dovecot.imapsieve" "NOTIFY" "mailto" "SASL" "PLAIN LOGIN SCRAM-SHA-1" "VERSION" "1.0" OK "Dovecot (Ubuntu) ready." and I can login successfully: AUTHENTICATE "PLAIN" "base64(0x00 $user 0x00 $password)" OK "Logged In." Note how this is different from the troubleshooting guide [1], which suggests gnutls-bin and waiting for the STARTTLS capability before hitting Ctrl-D. This is what I get with gnutls-bin: $ gnutls-cli --starttls --insecure -p 4190 $myserver Processed 0 CA certificate(s). Resolving '$myserver:4190'... Connecting to '$myserverip:4190'... - Simple Client Mode: _ where "_" denotes the waiting prompt. When I hit Ctrl-D here, I get an output similar that of s_client. Now, my problem are the clients: neither Thunderbird's sieve extenstion [2], nor the Ruby ManageSieve class [3], nor Roundcube's managesieve plugin [4] (via Net_Sieve module [5]) are able to communicate with my server. They all wait for a "STARTTLS" line, before they attempt to perform a TLS handshake. This leads me to my question: How do I force Dovecot to print at least a STARTTLS line after a client connects to port 4190? Looking Kind regards, Dominik [1]: https://wiki.dovecot.org/Pigeonhole/ManageSieve/Troubleshooting#Manual_TLS_… [2]: https://github.com/thsmi/sieve [3]: https://www.rubydoc.info/gems/ruby-managesieve/0.4.3/ManageSieve [4]: https://github.com/roundcube/roundcubemail/tree/1.3.8/plugins/managesieve [5]: http://pear.php.net/package/Net_Sieve/docs/1.3.4/Net_Sieve/Net_Sieve.html

4 11

degfault imaptest
by Marc Roos 14 Jan '19

14 Jan '19

imaptest host=mail04.local port=143 user=xxx pass=xxx mbox=test.mbox [Fri Jan 11 15:09:14 2019] imaptest[7634]: segfault at 1354ff0 ip 0000000001354ff0 sp 00007ffed5d8f558 error 15 [Fri Jan 11 15:09:22 2019] imaptest[7635]: segfault at 2267ff0 ip 0000000002267ff0 sp 00007ffee5890308 error 15 [Fri Jan 11 15:10:47 2019] imaptest[7648]: segfault at 10e6ff0 ip 00000000010e6ff0 sp 00007ffedbb19f98 error 15 CentOS Linux release 7.6.1810 (Core) imaptest-20151005-1.el7.x86_64 Linux test2 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

3 4

Indexing paralelism
by Joan Moreau 14 Jan '19

14 Jan '19

Hi Observing the processes of FTS, I observe the following: 1 - For one user, indexer-wroker does not start several threads for each request. On teh contrary, it waits for the first request to finish before starting the second. How to make sure all requests (or a limited number of it, for instance linked to the CPU number on the machine) are started asap ? 2 - If a IMAP query is received, the dovecot checks the last UID from FTS and launch a request of indexing to finish the index *before* running the search query . THis creates timeouts (and can take a while if many request are pending - see point 1) How to prevent that (i.e. the serach request is launched (read only) no matter what ? THe completeion of missing UIDs is launched in a separate thread ? Thank you

2 1

[FTS Xapian] Beta release
by Joan Moreau 13 Jan '19

13 Jan '19

Hi Please find attached the beta release of FTS Xapian, with the objective to replace fts_squat that is being deprecated. Configuration is exactly the same as for fts_squat: plugin { plugin = fts fts_xapian (...) fts = xapian fts_autoindex = yes fts_enforced = yes fts_xapian = partial=2 full=20 This is installed on my production server (>120Gb of mailboxes), and I will observe it during the coming days. I will definitely appreciate that this is added in the core git of docevot, in order to have a versionning of it, to remove squat and let basic users able to avoid Solr alternative as much as possible. Thanks JM

2 9

[FTS Xapian] Beta release
by Joan Moreau 13 Jan '19

13 Jan '19

Hi Please find attached the beta release of FTS Xapian, with the objective to replace fts_squat that is being deprecated. Configuration is exactly the same as for fts_squat: plugin { plugin = fts fts_xapian (...) fts = xapian fts_autoindex = yes fts_enforced = yes fts_xapian = partial=2 full=20 This is installed on my production server, and I will observe it during the coming days. I will definitely appreciate that this is added in the core git of docevot, in order to have a versionning of it, to remove squat and let basic users able to avoid Solr alternative as much as possible. THanks JM

1 0

Issue with LMTP proxying and port number
by Laz C. Peterson 13 Jan '19

13 Jan '19

Hello there, Everything was working fine on Dovecot 2.2.10 (on CentOS 7), but after updating to version 2.2.36, our director servers are not able to proxy LMTP. We are sending mail from exim to the local Dovecot LMTP socket, which then used to send it to our internal mail servers on port 24. Besides the update, nothing has changed in the config files. The configuration has been working great for months … Then we had a great idea of doing holiday maintenance. (Currently, we are working around this issue by having exim send mail to the IP and port, instead of the socket.) Doing a tcpdump capture shows no TCP traffic either. The logs seem to say that it is trying to go to port 0, so my guess is that it’s not getting the port directive from either static or sql — since the docs state that if these settings aren't specified, it will always use the same connection type for the proxy that it received the connection on. Any thoughts? I can share config if necessary. ~ Laz Peterson Paravis, LLC

3 5