June 2021 - dovecot - dovecot.org

Panic on indexer-worker and Dovecot stop to search on Solr
by Alessio Cecchi 03 Jun '21

03 Jun '21

Hi, I have setup on a (little busy) Dovecot server FTS with Solr and Virtual folder to enable "search in all folders" for users. All works fine until for some users the indexer-worker process crash. After this crash Dovecot stop to query Solr for new search in BODY, returning SERVERBUG, for all users on the server and only with a dovecot restart users can search in BODY again. Before the dovecot restart I can see that on Solr server no query from dovecot arriving anymore (for SEARCH BODY because SEARCH in FROM/TO/SUBJECT still works fine) Dovecot version is 2.3.14 (from RPM) and Apache Solr si 7.7.3. Here is the error found in the log: May 31 14:58:13 Panic: indexer-worker(user(a)email.com) session=<VA0oxZ/DXtG5Ydlx:0KzOIkLdtGDSawAAxbYlJw>: file http-client-request.c: line 1240 (http_client_request_send_more): assertion failed: (req->payload_input != NULL) May 31 14:58:13 Error: indexer-worker(user(a)email.com) session=<VA0oxZ/DXtG5Ydlx:0KzOIkLdtGDSawAAxbYlJw>: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x42) [0x7f3d91b2cac2] -> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f3d91b2cbce] -> /usr/lib64/dovecot/libdovecot.so.0(+0xf3cde) [0x7f3d91b39cde] -> /usr/lib64/dovecot/libdovecot.so.0(+0xf3d81) [0x7f3d91b39d81] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f3d91a8b25a] -> /usr/lib64/dovecot/libdovecot.so.0(http_client_request_send_more+0x3dd) [0x7f3d91ad39ad] -> /usr/lib64/dovecot/libdovecot.so.0(http_client_connection_output+0xf1) [0x7f3d91ad8101] -> /usr/lib64/dovecot/libdovecot.so.0(+0x11d2b0) [0x7f3d91b632b0] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x65) [0x7f3d91b51f45] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12b) [0x7f3d91b538fb] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x59) [0x7f3d91b52049] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f3d91b52288] -> /usr/lib64/dovecot/libdovecot.so.0(http_client_wait+0xcd) [0x7f3d91ae120d] -> /usr/lib64/dovecot/lib21_fts_solr_plugin.so(solr_connection_select+0xe4) [0x7f3d8fca1174] -> /usr/lib64/dovecot/lib21_fts_solr_plugin.so(+0x45d4) [0x7f3d8fc9d5d4] -> /usr/lib64/dovecot/lib20_fts_plugin.so(fts_backend_get_last_uid+0x6e) [0x7f3d90e103fe] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0xf952) [0x7f3d90e16952] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0x10ef6) [0x7f3d90e17ef6] -> /usr/lib64/dovecot/lib20_virtual_plugin.so(+0x966a) [0x7f3d90bfc66a] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0x10ba6) [0x7f3d90e17ba6] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_precache+0x2e) [0x7f3d91e3cb0e] -> dovecot/indexer-worker [user(a)email.com Virtual/All - 5300/20832](+0x2924) [0x5648464be924] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x65) [0x7f3d91b51f45] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12b) [0x7f3d91b538fb] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x59) [0x7f3d91b52049] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f3d91b52288] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f3d91abcbb3] -> dovecot/indexer-worker [user(a)email.com Virtual/All - 5300/20832](main+0xd7) [0x5648464be1f7] -> /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f3d9169a555] -> dovecot/indexer-worker [user(a)email.com Virtual/All - 5300/20832](+0x22ba) [0x5648464be2ba] May 31 14:58:13 Fatal: indexer-worker(user(a)email.com) session=<VA0oxZ/DXtG5Ydlx:0KzOIkLdtGDSawAAxbYlJw>: master: service(indexer-worker): child 27602 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps) on Solr side I found this error log: org.apache.solr.common.SolrException: Early EOF at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:190) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551) at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:733) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:531) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395) [...] and [com.ctc.wstx.exc.WstxLazyException] com.ctc.wstx.exc.WstxIOException: Early EOF at com.ctc.wstx.exc.WstxLazyException.throwLazily(WstxLazyException.java:45) at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:728) at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3678) at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:860) at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:419) at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:260) at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:188) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97) This is the relevant part of dovecot configuration: plugin { fts = solr fts_autoindex = yes fts_decoder = decode2text fts_enforced = body fts_index_timeout = 5s fts_solr = url=http://10.0.1.3:8983/solr/dovecot/ [...] } Is this an already know bug? Any workaround? Thanks -- Alessio Cecchi Postmaster @ http://www.qboxmail.it https://www.linkedin.com/in/alessice

2 4

Dovecot Stats for Logwatch
by Plutocrat 03 Jun '21

03 Jun '21

Hi all, I run logwatch on a few servers and the script on there which parses the dovecot logs provides waaaaaay too much detail, and contains a bunch of stuff I'm not interested in. There's not much point in having a log summary if its 100Kb long! The one for exim is also terrible, but I've replaced that with eximstats, and a bit of bash scripting and have something I can work with. I might at some point look at replacing that too. Now to my question. I haven't found anything in the dovecot family which compares to eximstats. I was wondering if there possibly is a report tool, or a way of extracting stats from doveadm just for key metrics such as 'number of POP logins in the last 24 hours', number of failed passwords, etc. Things that might be interesting on a logwatch report. Or maybe someone has written a script already that can be used in logwatch in perl, python, bash, whatever. Any pointers, suggestions, or ideas for useful summary metrics to pull out of the logfiles are appreciated. I suspect I might end up writing one myself anyway, but I'm just keen to see what already exists, get the benefit of anyone else's experiences in this matter before I start. P.

3 8

fts_autoindex_exclude for everything in a namespace?
by Daniel J. Luke 03 Jun '21

03 Jun '21

Hi, I've recently moved a system from using procmail to delivering via dovecot-lmtp w/ sieve - and it's great! I've got a copy of incoming mail going into mbox archives (that I compress after months end), and my mail log is full of "Error: fts: Failed to sync mailbox #Archive: Mailbox isn't selectable", though. I /think/ fts_autoindex_exclude = #Archive/* is what I want, but it doesn't seem to silence the error. I don't need the #Archive mailboxes visible to clients, but I don't mind if they are. I tried some setting the archive prefix = AutoArchive and setting list = yes, but that didn't silence the error. I feel like I'm probably missing something obvious :) dovecot 2.3.14 / pigenohole 0.5.14 relevant config from dovecot -n: mail_location = mbox:~/Mail/:INBOX=~/.mbox:INDEX=~/.dovecot-indexes mail_plugins = fts fts_lucene zlib namespace { inbox = yes location = mailbox Drafts { auto = no special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = no special_use = \Trash } mailbox spam { auto = create special_use = \Junk } prefix = separator = / } namespace archive { inbox = no list = no location = mbox:~/MailArchive/ prefix = "#Archive/" separator = / } plugin { fts = lucene fts_autoindex = yes fts_lucene = whitespace_chars=@. recipient_delimiter = + sieve = file:~/.sieve;active=~/.dovecot.sieve sieve_default = /usr/local/etc/dovecot/sieve/default.sieve sieve_default_name = file_spam } and the archive sieve script (which works) looks like this: if currentdate :matches "month" "*" { set "month" "${1}"; } if currentdate :matches "year" "*" { set "year" "${1}"; } if true { fileinto :copy :create "#Archive/${year}-${month}"; } -- Daniel J. Luke

1 0

userdb volatiledir setting not working
by Александер Скобельцын 02 Jun '21

02 Jun '21

Hi all! I just can't deal with VOLATILEDIR userdb setting on my mailservers. For example dovecot-uidlist.lock files are still being created over our network mail store. So, several days ago I changed my user and passwd query setting to include this option and restarted all my dovecot servers. Now it looks like (we use mysql driver): user_query = \ SELECT \ `u`.`pop` AS `home`, \ `u`.`uid`, \ `u`.`gid`, \ CONCAT('maildir:', `u`.`smtp`, ':INDEX=/var/dovecot_indexes/%d/%u:VOLATILEDIR=/var/dovecot_indexes/%d/%u') AS `mail`, \ CONCAT('*:bytes=', `u`.`quota`, 'M') AS `quota_rule` ,\ CONCAT('url=http://', fts_solr, ':8983/solr/dovecot/') AS fts_solr \ FROM `users` AS `u` \ JOIN `domains` AS `d` ON `d`.`domain_id` = `u`.`domain_id` \ WHERE \ `u`.`username` = '%u' \ AND `u`.`enabled` = 1 \ AND `d`.`enabled` = 1 \ AND `u`.`type` IN ('local', 'piped') password_query = \ SELECT \ `u`.`username` AS `user`, \ `u`.`crypt` AS `password`, \ `u`.`pop` AS `userdb_home`, \ `u`.`uid` AS `userdb_uid`, \ `u`.`gid` AS `userdb_gid`, \ CONCAT('maildir:', `u`.`smtp`, ':INDEX=/var/dovecot_indexes/%d/%u:VOLATILEDIR=/var/dovecot_indexes/%d/%u') AS `userdb_mail`, \ CONCAT('*:bytes=', `u`.`quota`, 'M') AS `userdb_quota_rule` \ FROM `users` AS `u` \ JOIN `domains` AS `d` ON `d`.`domain_id` = `u`.`domain_id` \ WHERE \ `u`.`username` = '%u' \ AND `u`.`enabled` = 1 \ AND `d`.`enabled` = 1 \ AND `u`.`type` IN ('local', 'piped') The aim was to move .lock and another temp files out of network storage our mails are stored. We use CephFS for storing user emails (maildir format). In addition we use 3 imap servers, each mount exact same path in cephfs and 2 dovecot directors to handle user=imapserver mapping, so user shouldn't exist on more than one imapserver at a time. Am I must use CONTROL directive in addition to force move control files from network store for VOLATILEDIR setting to work properly or I just misconfigured something? Extra info: dovecot --version 2.3.14 (cee3cbc0d)

1 0

debugging file permissions wrong
by Laura Steynes 02 Jun '21

02 Jun '21

Hi, In trying to debug a strange error where client can't login, I enabled all the usual debug settings, this is all good, it works for imap and pop3 fine, but the problem is when used with dovecot's LDA there is a nasty issue. the file created by debug_log_path in this case /var/log/dovecot/debug.log , this file created as root, again this is fine for nice logging of imap and pop3, but this causes postfix not to deliver mail, for write permission denied, LDA runs as vmail, all my sub sections like *_listener also = vmail and all my uid/gid settings are also to user/group vmail obviously you wont main log files to not be owned by vmail for case of security, so is there a way to set the ownership of the debug file - apart from the obvious of remembering 40 minutes later when you get alert of high mailq level to chown the file :) If there is no way, may the developers take this as a feature request please. Thanks Loz

2 3