Is there anything I can do here ? This makes Tika unusable :( That really sucks because we have a *lot* of attachments.

Thanks -

On 2021-02-03 2:07 pm, Dean Carpenter wrote:

Just noticed this in the logs

...

doveadm(harryb@example.com): Debug: http-client[2]: request [Req2: PUT http://localhost/tika/]: Submitted (requests left=1) doveadm(harryb@example.com): Debug: http-client[2]: request [Req2: PUT http://localhost/tika/]: Waiting for request to finish

It doesn't have the port on the URL ...

On 2021-02-03 1:59 pm, Dean Carpenter wrote:

...

Getting panic in http-client-request.c: line 1240 during indexing

I'm testing the install/setup of dovecot on an Ubuntu Focal 20.04 system, using the dovecot repo. The packages that are installed are

harryb@dove1:~$ dpkg -l dovecot* solr* jetty* tesser* | fgrep ii ii dovecot-core 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP server - core files ii dovecot-imapd 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP server - IMAP daemon ii dovecot-lmtpd 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP server - LMTP server ii dovecot-managesieved 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP server - ManageSieve server ii dovecot-mysql 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP server - MySQL support ii dovecot-sieve 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP server - Sieve filters support ii dovecot-solr 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP server - Solr support ii jetty9 9.4.26-1 all Java servlet engine and webserver ii solr-common 3.6.2+dfsg-22 all Enterprise search server based on Lucene3 - common files ii solr-jetty 3.6.2+dfsg-22 all Enterprise search server based on Lucene3 - Jetty integration ii tesseract-ocr 4.1.1-2build2 amd64 Tesseract command line OCR tool ii tesseract-ocr-eng 1:4.00~git30-7274cfa-1 all tesseract-ocr language files for English ii tesseract-ocr-osd 1:4.00~git30-7274cfa-1 all tesseract-ocr language files for script and orientation

Apache Tika v1.25 is also installed

harryb@dove1:~$ l /usr/local/bin/tika* -rw-r--r-- 1 root root 79337717 Feb 3 12:40 /usr/local/bin/tika-server-1.25.jar

and running pretty standard-ish

/usr/bin/java -jar /usr/local/bin/tika-server-1.25.jar --port 9998 --host localhost -enableUnsecureFeatures -enableFileUrl --log info

I use mbsync to pull in sample mailboxes for testing

The solr conf in /etc/dovecot/conf.d/ is as follows plugin { fts = solr fts_solr = url=http://localhost:8693/solr/ fts_autoindex = yes fts_filters = lowercase snowball stopwords fts_tokenizers = generic email-address fts_tokenizer_generic = algorithm=simple fts_autoindex_exclude = Junk fts_autoindex_exclude2 = Trash fts_autoindex_exclude3 = Spam } plugin { fts_tika = http://localhost:9998/tika/ }

Unfortunately indexing is panic'ing. Whether I index a test folder manually, or just looking in /var/log/dovecot.log

I'll see the following very consistently :

harryb@dove1:~$ sudo doveadm -D index -u harryb@example.com Receipts

Debug: Loading modules from directory: /usr/lib/dovecot/modules Debug: Module loaded: /usr/lib/dovecot/modules/lib10_mail_crypt_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_fts_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_replication_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so doveadm(harryb@example.com): Debug: auth-master: passdb lookup(harryb@example.com): Started passdb lookup doveadm(harryb@example.com): Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb: Connecting doveadm(harryb@example.com): Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb (pid=165148,uid=0): Client connected (fd=9) doveadm(harryb@example.com): Debug: auth-master: passdb lookup(harryb@example.com): auth PASS input: user=harryb@example.com doveadm(harryb@example.com): Debug: auth-master: passdb lookup(harryb@example.com): Finished passdb lookup (user=harryb@example.com ) doveadm(harryb@example.com)<180749><>: Debug: auth-master: userdb lookup(harryb@example.com): Started userdb lookup doveadm(harryb@example.com)<180749><>: Debug: auth-master: userdb lookup(harryb@example.com): auth USER input: harryb@example.com home=/var/mail/example.com/harryb uid=1001 gid=1001 doveadm(harryb@example.com)<180749><>: Debug: auth-master: userdb lookup(harryb@example.com): Finished userdb lookup (username=harryb@example.com home=/var/mail/example.com/harryb uid=1001 gid=1001) doveadm(harryb@example.com): Debug: Effective uid=1001, gid=1001, home=/var/mail/example.com/harryb doveadm(harryb@example.com): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:~/Maildir:LAYOUT=fs:INBOX=~/Maildir/INBOX doveadm(harryb@example.com): Debug: fs: root=/var/mail/example.com/harryb/Maildir, index=, indexpvt=, control=, inbox=/var/mail/example.com/harryb/Maildir/INBOX, alt= doveadm(harryb@example.com): Debug: Mailbox Receipts: Mailbox opened because: index doveadm(harryb@example.com): Info: Receipts: Caching mails seq=1..72 doveadm(harryb@example.com): Debug: Mailbox Receipts: UID 1: Opened mail because: prefetch doveadm(harryb@example.com): Debug: Mailbox Receipts: UID 1: Opened mail because: fts indexing doveadm(harryb@example.com): Debug: http-client: peer [::1]:9998 (shared): Peer created doveadm(harryb@example.com): Debug: http-client: peer [::1]:9998: Peer pool created doveadm(harryb@example.com): Debug: http-client[2]: peer [::1]:9998: Peer created doveadm(harryb@example.com): Debug: http-client[2]: queue http://localhost:9998: Setting up connection to [::1]:9998 (1 requests pending) doveadm(harryb@example.com): Debug: http-client[2]: peer [::1]:9998: Linked queue http://localhost:9998 (1 queues linked) doveadm(harryb@example.com): Debug: http-client[2]: queue http://localhost:9998: Started new connection to [::1]:9998 doveadm(harryb@example.com): Debug: http-client[2]: request [Req2: PUT http://localhost/tika/]: Submitted (requests left=1) doveadm(harryb@example.com): Debug: http-client[2]: request [Req2: PUT http://localhost/tika/]: Waiting for request to finish doveadm(harryb@example.com): Debug: http-client[2]: peer [::1]:9998: Creating 1 new connections to handle requests (already 0 usable, connecting to 0, closing 0) doveadm(harryb@example.com): Debug: http-client[2]: peer [::1]:9998: Making new connection 1 of 1 (0 connections exist, 0 pending) doveadm(harryb@example.com): Debug: http-client: conn [::1]:9998 [2]: Connecting doveadm(harryb@example.com): Debug: http-client: conn [::1]:9998 [2]: Waiting for connect (fd=20) to finish for max 0 msecs doveadm(harryb@example.com): Debug: http-client: conn [::1]:9998 [2]: HTTP connection created (1 parallel connections exist) doveadm(harryb@example.com): Panic: file http-client-request.c: line 1240 (http_client_request_send_more): assertion failed: (req->payload_input != NULL) doveadm(harryb@example.com): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x41) [0x7ffb60d697a1] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x7ffb60d698c2] -> /usr/lib/dovecot/libdovecot.so.0(+0x104b3b) [0x7ffb60d75b3b] -> /usr/lib/dovecot/libdovecot.so.0(+0x104b77) [0x7ffb60d75b77] -> /usr/lib/dovecot/libdovecot.so.0(+0x5b4d5) [0x7ffb60ccc4d5] -> /usr/lib/dovecot/libdovecot.so.0(+0x5291e) [0x7ffb60cc391e] -> /usr/lib/dovecot/libdovecot.so.0(http_client_connection_output+0xf2) [0x7ffb60d19f82] -> /usr/lib/dovecot/libdovecot.so.0(+0x12b1c5) [0x7ffb60d9c1c5] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x6d) [0x7ffb60d8bd4d] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x145) [0x7ffb60d8d395] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x54) [0x7ffb60d8bdf4] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x48) [0x7ffb60d8bf68] -> /usr/lib/dovecot/libdovecot.so.0(+0xa482d) [0x7ffb60d1582d] -> /usr/lib/dovecot/libdovecot.so.0(http_client_request_send_payload+0x34) [0x7ffb60d159c4] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x1068d) [0x7ffb604ef68d] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_parser_more+0x2b) [0x7ffb604ee4ab] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xd77f) [0x7ffb604ec77f] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_build_mail+0x52) [0x7ffb604ecec2] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x13573) [0x7ffb604f2573] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_precache+0x32) [0x7ffb60e92ed2] -> doveadm(+0x3982f) [0x55d3519ed82f] -> doveadm(+0x33e35) [0x55d3519e7e35] -> doveadm(+0x34a16) [0x55d3519e8a16] -> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x225) [0x55d3519e97b5] -> doveadm(doveadm_cmd_run_ver2+0x500) [0x55d3519f9e50] -> doveadm(doveadm_cmd_try_run_ver2+0x3e) [0x55d3519f9eae] -> doveadm(main+0x1d4) [0x55d3519d8dc4] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7ffb608fd0b3] -> doveadm(_start+0x2e) [0x55d3519d92ae] Aborted

The key line above is

doveadm(harryb@example.com): Panic: file http-client-request.c: line 1240 (http_client_request_send_more): assertion failed: (req->payload_input != NULL)

It's the same each time. In /var/log/syslog there are usually a couple of extra lines

Feb 03 13:25:17 indexer: Error: Indexer worker disconnected, discarding 1 requests for harryb@example.com Feb 03 13:25:17 indexer-worker(harryb@example.com)<202106>p+xgBonqGmAIFQMATj4HXg:IvE7Eo3qGmB6FQMATj4HXg: Fatal: master: service(indexer-worker): child 202106 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps)

-- Dean Carpenter deano is at areyes dot com 203 six oh four 6644

On 06 Feb 2021, at 11:06, John Fawcett john@voipsupport.it wrote:

19.08.20

Is that a malformed ISO date 2019-08-20 or a truncated europeans style 19-08-2020?

-- 'Yes, but humans are more important than animals,' said Brutha. 'This is a point of view often expressed by humans,' said Om. (Small Gods)

On 07/02/2021 20:15, @lbutlr wrote:

On 07 Feb 2021, at 02:07, @lbutlr kremels@kreme.com wrote:

...

On 06 Feb 2021, at 11:06, John Fawcett john@voipsupport.it wrote:

...

19.08.20 Is that a malformed ISO date 2019-08-20 or a truncated europeans style 19-08-2020? Either way, I cannot find the message in my dovecot folder.

Closest I can find is Message-ID: 20200820141341.GA1751@meili.valhalla.31bits.net from the 2020-08-20 at 16:13:52 but there is no patch in that mail.

here's the post in the list archives

https://dovecot.org/pipermail/dovecot/2020-August/119703.html

On 08/02/2021 18:40, deano-dovecot@areyes.com wrote:

On 2021-02-07 7:32 pm, John Fawcett wrote:

...

On 07/02/2021 20:15, @lbutlr wrote:

...

On 07 Feb 2021, at 02:07, @lbutlr mailto:kremels@kreme.com> wrote:

...

...

19.08.20 Is that a malformed ISO date 2019-08-20 or a truncated europeans

On 06 Feb 2021, at 11:06, John Fawcett mailto:john@voipsupport.it> wrote: style 19-08-2020? Either way, I cannot find the message in my dovecot folder. Closest I can find is Message-ID: <20200820141341.GA1751@meili.valhalla.31bits.net mailto:20200820141341.GA1751@meili.valhalla.31bits.net> from the 2020-08-20 at 16:13:52 but there is no patch in that mail. here's the post in the list archives

https://dovecot.org/pipermail/dovecot/2020-August/119703.html https://dovecot.org/pipermail/dovecot/2020-August/119703.html

Do we have when (or even if) that patch will make it into the main ?  I would really rather prefer pulling from repo ...

+1 from me.

I'd like to see this patch (or something equivalent go in). Without this Tika is unusable for me.

Tika is also unusable in my opinion without the (basic) limits on processing unbounded amounts of text. There is also another patch introducing basic auth which I find useful since my Tika server is password protected.

If all three could make it into the repo then I could avoid manual patching that I do at each release.

John

On 2/9/21 4:49 AM, deano-dovecot@areyes.com wrote:

Unfortunately they don't make the source repos (deb-src http://repo.dovecot.org/.....) available,

They do however provide the .dsc file, so you can use dget (from the devscripts package)

e.g.

mkdir dovecot-source; cd dovecot-source dget -u https://repo.dovecot.org/ce-2.3-latest/debian/buster/pool/main/2.3.13-2_ce/d...

That will do the same thing as apt-get source ... (you need -u because the .dsc file does not contain a signature)

The directories are browsable at https://repo.dovecot.org/ce-2.3-latest/ so you can look for the right file per distro/version.

On 08-02-2021 18:40, deano-dovecot@areyes.com wrote:

Do we have when (or even if) that patch will make it into the main ?  I would really rather prefer pulling from repo ...

Please! I struggle with this for at least two years now!

Arjen

On 10-02-2021 01:08, Scott Q. wrote:

Is this the same bug that John created a patch for but then Dovecot fixed differently in the latest version ? 

The one with the assertion ? If so, I thought it was fixed, no ?

No, this is still active, running dovecot-2.3.13-2.x86_64, updated this morning:

doveadm(dexter): Panic: file http-client-request.c: line 1240 (http_client_request_send_more): assertion failed: (req->payload_input != NULL) doveadm(dexter): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x42) [0x7f2faa8b64c2] -> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f2faa8b65ce] -> /usr/lib64/dovecot/libdovecot.so.0(+0xf262e) [0x7f2faa8c362e] -> /usr/lib64/dovecot/libdovecot.so.0(+0xf2671) [0x7f2faa8c3671] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f2faa815d1a] -> /usr/lib64/dovecot/libdovecot.so.0(http_client_request_send_more+0x3dd) [0x7f2faa85dffd] -> /usr/lib64/dovecot/libdovecot.so.0(http_client_connection_output+0xf1) [0x7f2faa862741] -> /usr/lib64/dovecot/libdovecot.so.0(+0x11ba30) [0x7f2faa8eca30] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x65) [0x7f2faa8db775] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12b) [0x7f2faa8dd10b] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x59) [0x7f2faa8db879] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f2faa8dbab8] -> /usr/lib64/dovecot/libdovecot.so.0(+0x8c526) [0x7f2faa85d526] -> /usr/lib64/dovecot/libdovecot.so.0(http_client_request_send_payload+0x2c) [0x7f2faa85d7ac] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0xd52d) [0x7f2fa7ba752d] -> /usr/lib64/dovecot/lib20_fts_plugin.so(fts_parser_more+0x27) [0x7f2fa7ba64b7] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0xa2a1) [0x7f2fa7ba42a1] -> /usr/lib64/dovecot/lib20_fts_plugin.so(fts_build_mail+0x54) [0x7f2fa7ba4c04] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0x10e32) [0x7f2fa7baae32] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_precache+0x2e) [0x7f2faabc639e] -> /bin/doveadm(+0x3288f) [0x56466593d88f] -> /bin/doveadm(+0x32e28) [0x56466593de28] -> /bin/doveadm(+0x2cbd7) [0x564665937bd7] -> /bin/doveadm(+0x2d9e7) [0x5646659389e7] -> /bin/doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x21d) [0x5646659396cd] -> /bin/doveadm(doveadm_cmd_run_ver2+0x472) [0x56466594a3c2] -> /bin/doveadm(doveadm_cmd_try_run_ver2+0x37) [0x56466594a4e7] -> /bin/doveadm(main+0x1d4) [0x564665927984] -> /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f2faa123555] -> /bin/doveadm(+0x1ce1f) [0x564665927e1f] Aborted

Regards,

Arjen

On 2021-02-07 7:32 pm, John Fawcett wrote:

On 07/02/2021 20:15, @lbutlr wrote: On 07 Feb 2021, at 02:07, @lbutlr kremels@kreme.com wrote: On 06 Feb 2021, at 11:06, John Fawcett john@voipsupport.it wrote: 19.08.20 Is that a malformed ISO date 2019-08-20 or a truncated europeans style 19-08-2020? Either way, I cannot find the message in my dovecot folder. Closest I can find is Message-ID: 20200820141341.GA1751@meili.valhalla.31bits.net from the 2020-08-20 at 16:13:52 but there is no patch in that mail.

here's the post in the list archives

https://dovecot.org/pipermail/dovecot/2020-August/119703.html [1]

Thanks so much for the help - I've verified the patch and it works nicely. Here's a simple script for 20.04 - it's very manual, you have to choose the version to download.

It's just two lines of patch - is there any chance this will be fixed in the next point release ?

Links:

[1] https://dovecot.org/pipermail/dovecot/2020-August/119703.html