- dovecot - dovecot.org

Getting panic in http-client-request.c: line 1240 during indexing on Ubuntu 20.04
by Dean Carpenter 18 Feb '21

18 Feb '21

Getting panic in http-client-request.c: line 1240 during indexing I'm testing the install/setup of dovecot on an Ubuntu Focal 20.04 system, using the dovecot repo. The packages that are installed are harryb@dove1:~$ dpkg -l dovecot* solr* jetty* tesser* | fgrep ii ii dovecot-core 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP server - core files ii dovecot-imapd 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP server - IMAP daemon ii dovecot-lmtpd 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP server - LMTP server ii dovecot-managesieved 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP server - ManageSieve server ii dovecot-mysql 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP server - MySQL support ii dovecot-sieve 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP server - Sieve filters support ii dovecot-solr 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP server - Solr support ii jetty9 9.4.26-1 all Java servlet engine and webserver ii solr-common 3.6.2+dfsg-22 all Enterprise search server based on Lucene3 - common files ii solr-jetty 3.6.2+dfsg-22 all Enterprise search server based on Lucene3 - Jetty integration ii tesseract-ocr 4.1.1-2build2 amd64 Tesseract command line OCR tool ii tesseract-ocr-eng 1:4.00~git30-7274cfa-1 all tesseract-ocr language files for English ii tesseract-ocr-osd 1:4.00~git30-7274cfa-1 all tesseract-ocr language files for script and orientation Apache Tika v1.25 is also installed harryb@dove1:~$ l /usr/local/bin/tika* -rw-r--r-- 1 root root 79337717 Feb 3 12:40 /usr/local/bin/tika-server-1.25.jar and running pretty standard-ish /usr/bin/java -jar /usr/local/bin/tika-server-1.25.jar --port 9998 --host localhost -enableUnsecureFeatures -enableFileUrl --log info I use mbsync to pull in sample mailboxes for testing The solr conf in /etc/dovecot/conf.d/ is as follows plugin { fts = solr fts_solr = url=http://localhost:8693/solr/ fts_autoindex = yes fts_filters = lowercase snowball stopwords fts_tokenizers = generic email-address fts_tokenizer_generic = algorithm=simple fts_autoindex_exclude = Junk fts_autoindex_exclude2 = Trash fts_autoindex_exclude3 = Spam } plugin { fts_tika = http://localhost:9998/tika/ } Unfortunately indexing is panic'ing. Whether I index a test folder manually, or just looking in /var/log/dovecot.log I'll see the following very consistently : harryb@dove1:~$ sudo doveadm -D index -u harryb(a)example.com Receipts Debug: Loading modules from directory: /usr/lib/dovecot/modules Debug: Module loaded: /usr/lib/dovecot/modules/lib10_mail_crypt_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_fts_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_replication_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so doveadm(harryb(a)example.com): Debug: auth-master: passdb lookup(harryb(a)example.com): Started passdb lookup doveadm(harryb(a)example.com): Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb: Connecting doveadm(harryb(a)example.com): Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb (pid=165148,uid=0): Client connected (fd=9) doveadm(harryb(a)example.com): Debug: auth-master: passdb lookup(harryb(a)example.com): auth PASS input: user=harryb(a)example.com doveadm(harryb(a)example.com): Debug: auth-master: passdb lookup(harryb(a)example.com): Finished passdb lookup (user=harryb(a)example.com ) doveadm(harryb(a)example.com)<180749><>: Debug: auth-master: userdb lookup(harryb(a)example.com): Started userdb lookup doveadm(harryb(a)example.com)<180749><>: Debug: auth-master: userdb lookup(harryb(a)example.com): auth USER input: harryb(a)example.com home=/var/mail/example.com/harryb uid=1001 gid=1001 doveadm(harryb(a)example.com)<180749><>: Debug: auth-master: userdb lookup(harryb(a)example.com): Finished userdb lookup (username=harryb(a)example.com home=/var/mail/example.com/harryb uid=1001 gid=1001) doveadm(harryb(a)example.com): Debug: Effective uid=1001, gid=1001, home=/var/mail/example.com/harryb doveadm(harryb(a)example.com): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:~/Maildir:LAYOUT=fs:INBOX=~/Maildir/INBOX doveadm(harryb(a)example.com): Debug: fs: root=/var/mail/example.com/harryb/Maildir, index=, indexpvt=, control=, inbox=/var/mail/example.com/harryb/Maildir/INBOX, alt= doveadm(harryb(a)example.com): Debug: Mailbox Receipts: Mailbox opened because: index doveadm(harryb(a)example.com): Info: Receipts: Caching mails seq=1..72 doveadm(harryb(a)example.com): Debug: Mailbox Receipts: UID 1: Opened mail because: prefetch doveadm(harryb(a)example.com): Debug: Mailbox Receipts: UID 1: Opened mail because: fts indexing doveadm(harryb(a)example.com): Debug: http-client: peer [::1]:9998 (shared): Peer created doveadm(harryb(a)example.com): Debug: http-client: peer [::1]:9998: Peer pool created doveadm(harryb(a)example.com): Debug: http-client[2]: peer [::1]:9998: Peer created doveadm(harryb(a)example.com): Debug: http-client[2]: queue http://localhost:9998: Setting up connection to [::1]:9998 (1 requests pending) doveadm(harryb(a)example.com): Debug: http-client[2]: peer [::1]:9998: Linked queue http://localhost:9998 (1 queues linked) doveadm(harryb(a)example.com): Debug: http-client[2]: queue http://localhost:9998: Started new connection to [::1]:9998 doveadm(harryb(a)example.com): Debug: http-client[2]: request [Req2: PUT http://localhost/tika/] Submitted (requests left=1) doveadm(harryb(a)example.com): Debug: http-client[2]: request [Req2: PUT http://localhost/tika/] Waiting for request to finish doveadm(harryb(a)example.com): Debug: http-client[2]: peer [::1]:9998: Creating 1 new connections to handle requests (already 0 usable, connecting to 0, closing 0) doveadm(harryb(a)example.com): Debug: http-client[2]: peer [::1]:9998: Making new connection 1 of 1 (0 connections exist, 0 pending) doveadm(harryb(a)example.com): Debug: http-client: conn [::1]:9998 [2]: Connecting doveadm(harryb(a)example.com): Debug: http-client: conn [::1]:9998 [2]: Waiting for connect (fd=20) to finish for max 0 msecs doveadm(harryb(a)example.com): Debug: http-client: conn [::1]:9998 [2]: HTTP connection created (1 parallel connections exist) doveadm(harryb(a)example.com): Panic: file http-client-request.c: line 1240 (http_client_request_send_more): assertion failed: (req->payload_input != NULL) doveadm(harryb(a)example.com): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x41) [0x7ffb60d697a1] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x7ffb60d698c2] -> /usr/lib/dovecot/libdovecot.so.0(+0x104b3b) [0x7ffb60d75b3b] -> /usr/lib/dovecot/libdovecot.so.0(+0x104b77) [0x7ffb60d75b77] -> /usr/lib/dovecot/libdovecot.so.0(+0x5b4d5) [0x7ffb60ccc4d5] -> /usr/lib/dovecot/libdovecot.so.0(+0x5291e) [0x7ffb60cc391e] -> /usr/lib/dovecot/libdovecot.so.0(http_client_connection_output+0xf2) [0x7ffb60d19f82] -> /usr/lib/dovecot/libdovecot.so.0(+0x12b1c5) [0x7ffb60d9c1c5] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x6d) [0x7ffb60d8bd4d] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x145) [0x7ffb60d8d395] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x54) [0x7ffb60d8bdf4] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x48) [0x7ffb60d8bf68] -> /usr/lib/dovecot/libdovecot.so.0(+0xa482d) [0x7ffb60d1582d] -> /usr/lib/dovecot/libdovecot.so.0(http_client_request_send_payload+0x34) [0x7ffb60d159c4] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x1068d) [0x7ffb604ef68d] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_parser_more+0x2b) [0x7ffb604ee4ab] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xd77f) [0x7ffb604ec77f] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_build_mail+0x52) [0x7ffb604ecec2] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x13573) [0x7ffb604f2573] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_precache+0x32) [0x7ffb60e92ed2] -> doveadm(+0x3982f) [0x55d3519ed82f] -> doveadm(+0x33e35) [0x55d3519e7e35] -> doveadm(+0x34a16) [0x55d3519e8a16] -> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x225) [0x55d3519e97b5] -> doveadm(doveadm_cmd_run_ver2+0x500) [0x55d3519f9e50] -> doveadm(doveadm_cmd_try_run_ver2+0x3e) [0x55d3519f9eae] -> doveadm(main+0x1d4) [0x55d3519d8dc4] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7ffb608fd0b3] -> doveadm(_start+0x2e) [0x55d3519d92ae] Aborted The key line above is doveadm(harryb(a)example.com): Panic: file http-client-request.c: line 1240 (http_client_request_send_more): assertion failed: (req->payload_input != NULL) It's the same each time. In /var/log/syslog there are usually a couple of extra lines Feb 03 13:25:17 indexer: Error: Indexer worker disconnected, discarding 1 requests for harryb(a)example.com Feb 03 13:25:17 indexer-worker(harryb(a)example.com)<202106><p+xgBonqGmAIFQMATj4HXg:IvE7Eo3qGmB6FQMATj4HXg>: Fatal: master: service(indexer-worker): child 202106 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps) -- Dean Carpenter deano is at areyes dot com 203 six oh four 6644

7 17

doveadm import uninterruptable?
by Ralf Hildebrandt 17 Feb '21

17 Feb '21

I'm using % doveadm import -u restore(a)backup.invalid mdbox:/home/archive/mdbox "" mailbox INBOX header From $PATTERN sometimes, I mistype the pattern and immediately try to stop the doveadm command with CTRL-C - and this never succeeds. Is it inentional that I cannot interrupt the "doveadm import" command? Using 2.3.13 here. -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | https://www.charite.de

2 1

Pigeonhole v0.5.14.rc1 released
by Aki Tuomi 17 Feb '21

17 Feb '21

We are pleased to release first release candidate for v0.5.14. We have done changes to packaging so please give us any feedback on how it works. https://pigeonhole.dovecot.org/releases/2.3/rc/dovecot-2.3-pigeonhole-0.5.1… https://pigeonhole.dovecot.org/releases/2.3/rc/dovecot-2.3-pigeonhole-0.5.1… Binary packages in https://repo.dovecot.org/ Docker images are not available for this release candidate. Kind regards, Aki Tuomi Open-Xchange oy --- * IMAP FILTER command: cmd-filter-sieve - Do not allow NIL as script name argument.

1 0

imapc settings for specific mailbox / namespaces
by Tobia Grosselle [ Terralink ] 17 Feb '21

17 Feb '21

Hello, Is it possible to use imapc for specific mailbox or namespace? Something like this: namespace Test { list = no type = private inbox = no hidden=no prefix = TEST/ separator = / location = imapc: imapc_host = imap.example.com imapc_password = secret imapc_port = 143 imapc_user = user(a)example.com } or is only intended for general use as general proxy in the global configuration?

1 0

Logfile flooding with lookup failure for root user using pam
by Michael Ruiz 16 Feb '21

16 Feb '21

Hi all, I am getting the error where root is constantly being lookedup over imap I am guessing. I am pretty sure services like fail2ban and logwatch are setup to send from another user instead of root @ localhost so I think this may be an imap issue. I tried adding root to /etc/dovecot/deny-users , but this error still persists. I know imap as root is a big security issue and is disabled, but I cannot pinpoint how or why this is happening so often. I am using PAM and passwd for authentication and authorization. How can I prevent root user from being looked up in the first place ? Regards, Michael dovecot[606167]: lmtp(606199): Error: lmtp-server: conn unix:pid=606196,uid=73 [2]: rcpt root@<domain>: Failed to lookup user root@<domain>: Invalid settings in userdb: userdb returned 0 as uid replaced my domain name with <domain> in this example.

2 1

quota_max_mail_size and dsync
by Edgaras Lukoševičius 15 Feb '21

15 Feb '21

Hello, as we have enabled quota_max_mail_size, dsync is now having problems moving oversized emails between servers. Is there a way to configure dovecot (dsync) to allow syncing of emails larger than quota_max_mail_size but keep limiting clients over imap/pop/lmtp? The error logged is: dsync(local(a)domain.tld): Info: imapc(remote.imap.host.tld): Connected to II.III.III.II:PPP (local II.III.III.II:PPPPP) dsync(local(a)domain.tld): Warning: Failed to copy source UID=24 mail: Mail size is larger than the maximum size allowed by server configuration - falling back to regular saving dsync(local(a)domain.tld): Error: Mailbox INBOX.Sent: Saving failed: Mail size is larger than the maximum size allowed by server configuration

2 1

Mail.app namespaces
by Dan Mahoney (Gushi) 15 Feb '21

15 Feb '21

All, Dovecot 2.3.13 under Freebsd 11.3, installed via pkg. I had a question related to dovecot and OSX mail.app. My primary mailer is "alpine". Sometimes I need to see a message in a graphical app so I occasionally turn to mail.app. I keep a lot of old mail. Mail.app on recent macos versions insist on downloading *everything* for easier searching and indexing. Most of which I don't need. (20G of mail over the last 20+ years, plus a bunch of mailing lists I keep archived, but only read when I need to search them). ios mail doesn't do this. Subscriptions don't work (in mail.app, subscriptions are only for public/shared folders). Ergo, is there a server-side way to make just one client not able to see/list/download ALL folders? Perhaps by presenting them as subscription folders, or perhaps by hiding them in the namespace. I'm not unwilling to configure an alternate namespace for myself. (It's helps that I'm the sysadmin here). Ideally, this would be a thing I could only set for my account -- match on a user-agent/username string of some sort? (My other option has been "create a different username and forward to myself there", but this feels like it should be solvable). -Dan My doveconf follows: # 2.3.13 (89f716dc2): /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 11.4-RELEASE-p3 amd64 # Hostname: prime.gushi.org disable_plaintext_auth = no mail_full_filesystem_access = yes mail_location = mbox:%h/mail:INBOX=%h/.mail:INDEX=/var/db/indexes/%u namespace { alias_for = hidden = yes list = no location = prefix = mail/ separator = / } namespace { alias_for = hidden = no list = yes location = prefix = ~/mail/ separator = / subscriptions = yes } namespace { alias_for = hidden = yes list = no location = prefix = ~%u/mail/ separator = / } namespace { hidden = no list = yes location = mbox:~%u/.incoming/ prefix = Inbounds/ separator = / } namespace inbox { inbox = yes location = prefix = separator = / } passdb { driver = pam } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size } service imap-login { inet_listener imap { port = 143 } inet_listener imap_other { port = 1143 } inet_listener imaps { port = 993 ssl = yes } inet_listener imaps_other { port = 1993 ssl = yes } } service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3_other { port = 1110 } inet_listener pop3s { port = 995 ssl = yes } inet_listener pop3s_other { port = 1995 ssl = yes } } ssl_cert = </etc/mail/certs/prime.gushi.org.certkey ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = passwd } -- --------Dan Mahoney-------- Techie, Sysadmin, WebGeek Gushi on efnet/undernet IRC FB: fb.com/DanielMahoneyIV LI: linkedin.com/in/gushi Site: http://www.gushi.org ---------------------------

2 3

fts_encoder
by Joan Moreau 11 Feb '21

11 Feb '21

Hello, I am trying to deal properly with email attachements in fts-xapian plugins. I tried the default script with a PDF file. The data I receive in the fts plugin part ("xxx_build_more") is the original document, no the output of the pdftotext Is there anything I am missing ? Here my config: plugin { plugin = fts_xapian managesieve sieve fts = xapian fts_xapian = partial=2 full=20 verbose=1 attachments=1 fts_autoindex = yes fts_enforced = yes fts_autoindex_exclude = \Trash fts_autoindex_exclude2 = \Drafts fts_decoder = decode2text sieve = /data/mail/%d/%n/local.sieve sieve_after = /data/mail/after.sieve sieve_before = /data/mail/before.sieve sieve_dir = /data/mail/%d/%n/sieve sieve_global_dir = /data/mail sieve_global_path = /data/mail/global.sieve } ... service decode2text { executable = script /usr/libexec/dovecot/decode2text.sh user = dovecot unix_listener decode2text { mode = 0666 } } Thank you

3 18

doveadm expunge logging
by Arkadiusz Miśkiewicz 11 Feb '21

11 Feb '21

Hello. dovecot 2.3.13 here, using doveadm -c /etc/my.conf expunge -A mailbox SomeFolder savedbefore 31d and my.conf includes plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename save } unfortunately expunged messages are not logged anywhere. Tried -v and -D to see if these will get logged to output (so I could pipe these to syslog) but verbose/debug also don't show which messages were expunged. Is there a way to get this logged? Log like regular dovecot logs would be nice: doveadm(arekm): pid=<14783>, expunge: box=INBOX, uid=0, msgid=<E1l2ZJp-0002NM-Vv(a)example.com>, size=15077 -- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )

3 4

Re: Quota-status service on Director
by Richard Pablo Mérida González 11 Feb '21

11 Feb '21

Did you solve this problem of quota status? I am in the same situation right now, i would appreciate you let me know any helpfull information Sent with Aqua Mail for Android https://www.mobisystems.com/aqua-mail

1 0