<html>
<head>
<style type="text/css">
body,p,td,div,span{
        font-size:14px;font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol";
};
body p{
        margin:0px;
}
</style>
</head>
<body>I have implemented this change and the core dumps went away.<div><br></div><div>It appears so far that it's indeed the right fix.</div><div><br></div><div>Has this change been introduced in 2.3.11.3 ? </div><div><br></div><div>Thanks!<br><br>On Tuesday, 27/10/2020 at 08:20 John Fawcett wrote:<br><blockquote style="border:0;border-left: 2px solid #22437f; padding:0px; margin:0px; padding-left:5px; margin-left: 5px; ">
  
  
    <div class="moz-cite-prefix">On 22/10/2020 10:23, John Fawcett
      wrote:<br>
    </div>
    <blockquote type="cite" cite="mid:b2bf3d32-0711-9ff4-5b82-c8b6ed19a230@voipsupport.it">
      
      <div class="moz-cite-prefix">On 21/10/2020 19:00, John Fawcett
        wrote:<br>
      </div>
      <blockquote type="cite" cite="mid:7a159f93-02c2-fbd4-e5ae-cdd0d54db46a@voipsupport.it">
        
        <div class="moz-cite-prefix">On 21/10/2020 16:44, Patrik Peng
          wrote:<br>
        </div>
        <blockquote type="cite" cite="mid:48c5a4e9-c0a6-0b69-8aec-02d68633e1e3@hostpoint.ch">
          
          <div class="moz-cite-prefix">On 16.10.20 18:34, Patrik Peng
            wrote:<br>
          </div>
          <blockquote type="cite" cite="mid:f65f2083-4498-77a7-3025-0f5c2ec3f89e@hostpoint.ch">
            
            On 16.10.20 18:00, Scott Q. wrote:<br>
            <blockquote type="cite" cite="mid:f1cb783bd48ab765b73707c30d8848e1@polarismail--com.w.emailarray.com">
              
               This reminds me, the way I was able to reproduce this
              consistently was by having large headers ( 100+ lines ).
              <div><br>
              </div>
              <div><br>
                On Friday, 16/10/2020 at 11:49 Patrik Peng wrote:<br>
                <blockquote style="border:0;border-left: 2px solid                    #22437f; padding:0px; margin:0px; padding-left:5px;                    margin-left: 5px; ">
                  <p>On 19.08.20 17:37, Josef 'Jeff' Sipek wrote:<br>
                  </p>
                  <blockquote type="cite" cite="mid:20200819153716.GB1583@meili.valhalla.31bits.net">
                    <pre class="moz-quote-pre" wrap="">On Wed, Aug 19, 2020 at 17:03:57 +0200, Alessio Cecchi wrote:
</pre>
                    <blockquote type="cite">
                      <pre class="moz-quote-pre" wrap="">Hi,
after the upgrade to Dovecot 2.3.11.3, from 2.3.10.1, I see frequently 
these errors from different users:
</pre>
                    </blockquote>
                    <pre class="moz-quote-pre" wrap="">It looks like this has been around for a while and you just got unlucky and
started seeing this now.  Here's a quick & dirty patch that should fix this.
If you can try it, let us know how it went.
Jeff.
diff --git a/src/plugins/fts-solr/solr-connection.c b/src/plugins/fts-solr/solr-connection.c
index ae720b5e2870a852c1b6c440939e3c7c0fa72b5c..9d364f93e2cd1b716b9ab61bd39656a6c5b1ea04 100644
--- a/src/plugins/fts-solr/solr-connection.c
+++ b/src/plugins/fts-solr/solr-connection.c
@@ -103,7 +103,7 @@ int solr_connection_init(const struct fts_solr_settings *solr_set,
                http_set.ssl = ssl_client_set;
                http_set.debug = solr_set->debug;
                http_set.rawlog_dir = solr_set->rawlog_dir;
-               solr_http_client = http_client_init(&http_set);
+               solr_http_client = http_client_init_private(&http_set);
        }
        *conn_r = conn;
diff --git a/src/plugins/fts/fts-parser-tika.c b/src/plugins/fts/fts-parser-tika.c
index a4b8b5c3034f57e22e77caa759c090da6b62f8ba..b8b57a350b9a710d101ac7ccbcc14560d415d905 100644
--- a/src/plugins/fts/fts-parser-tika.c
+++ b/src/plugins/fts/fts-parser-tika.c
@@ -77,7 +77,7 @@ tika_get_http_client_url(struct mail_user *user, struct http_url **http_url_r)
                http_set.request_timeout_msecs = 60*1000;
                http_set.ssl = &ssl_set;
                http_set.debug = user->mail_debug;
-               tika_http_client = http_client_init(&http_set);
+               tika_http_client = http_client_init_private(&http_set);
        }
        *http_url_r = tuser->http_url;
        return 0;
</pre>
                  </blockquote>
                  <p> Greetings</p>
                  <p>I'm also experiencing these issues while running
                    Dovecot 2.3.11.3 with Solr 8.6.3 on FreeBSD 11.4. As
                    mentioned in a previous mail, the above patch is
                    already applied to Dovecot's FreeBSD Port, confirmed
                    by the patches being present in the portstree (<a target="_blank" class="moz-txt-link-freetext" href="https://svnweb.freebsd.org/ports/branches/2020Q3/mail/dovecot/files/" moz-do-not-send="true">https://svnweb.freebsd.org/ports/branches/2020Q3/mail/dovecot/files/</a>).</p>
                  <p>In a FreeBSD VM with the official image (<a target="_blank" class="moz-txt-link-freetext" href="https://download.freebsd.org/ftp/releases/VM-IMAGES/12.1-RELEASE/amd64/Latest/" moz-do-not-send="true">https://download.freebsd.org/ftp/releases/VM-IMAGES/12.1-RELEASE/amd64/Latest/</a>)
                    I compiled dovecot from git and was able to
                    reproduce the error with the patch mentioned above
                    applied and also without any patches at all. From
                    these results i conclude, that neither the patches
                    applied in FreeBSDs portstree or the patch above
                    have any influence.</p>
                  <p>I also managed to reproduce the same results on a
                    Debian 10 machine (also with and without the patch):</p>
                  <pre>doveadm(<a target="_blank" class="moz-txt-link-abbreviated" href="mailto:some.user2@example.com" moz-do-not-send="true">some.user@example.com</a>): Panic: file http-client-request.c: line 1232 (http_client_request_send_more): assertion failed: (req->payload_input != NULL)
doveadm(<a target="_blank" class="moz-txt-link-abbreviated" href="mailto:some.user2@example.com" moz-do-not-send="true">some.user@example.com</a>): Error: Raw backtrace: /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot.so</a>.0(backtrace_append+0x42) [0x7f093f7fc3c2]
-> /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot.so</a>.0(backtrace_get+0x1e) [0x7f093f7fc4ce] -> /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot.so</a>.0(+0xea341) [0x7f093f807341]
-> /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot.so</a>.0(+0xea381) [0x7f093f807381] -> /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot.so</a>.0(i_fatal+0) [0x7f093f75c074]
-> /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot.so</a>.0(http_client_request_send_more+0x378) [0x7f093f7a47a8]
-> /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot.so</a>.0(http_client_connection_output+0xe4) [0x7f093f7a90f4]
-> /usr/local/lib/dovecot/libssl_iostream_<a target="_blank" href="http://openssl.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">openssl.so</a>(+0x8bff) [0x7f093ec71bff]
-> /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot.so</a>.0(+0x1148b0) [0x7f093f8318b0]
-> /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot.so</a>.0(io_loop_call_io+0x69) [0x7f093f820259]
-> /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot.so</a>.0(io_loop_handler_run_internal+0x11b) [0x7f093f821b6b]
-> /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot.so</a>.0(io_loop_handler_run+0x59) [0x7f093f820369]
-> /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot.so</a>.0(io_loop_run+0x38) [0x7f093f820598]
-> /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot.so</a>.0(+0x86d1e) [0x7f093f7a3d1e]
-> /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot.so</a>.0(http_client_request_finish_payload+0x2e) [0x7f093f7a407e]
-> /usr/local/lib/dovecot/lib21_fts_solr_<a target="_blank" href="http://plugin.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">plugin.so</a>(solr_connection_post_end+0x32) [0x7f093b8492c2]
-> /usr/local/lib/dovecot/lib21_fts_solr_<a target="_blank" href="http://plugin.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">plugin.so</a>(+0x3a45) [0x7f093b844a45]
-> /usr/local/lib/dovecot/lib20_fts_<a target="_blank" href="http://plugin.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">plugin.so</a>(+0x94cc) [0x7f093e1104cc]
-> /usr/local/lib/dovecot/lib20_fts_<a target="_blank" href="http://plugin.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">plugin.so</a>(fts_backend_update_deinit+0x23) [0x7f093e110503]
-> /usr/local/lib/dovecot/lib20_fts_<a target="_blank" href="http://plugin.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">plugin.so</a>(+0x10a9b) [0x7f093e117a9b]
-> /usr/local/lib/dovecot/lib20_fts_<a target="_blank" href="http://plugin.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">plugin.so</a>(+0x119ca) [0x7f093e1189ca]
-> /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot-storage.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot-storage.so</a>.0(mailbox_transaction_commit_get_changes+0x56) [0x7f093fb16076]
-> /usr/local/lib/dovecot/<a target="_blank" href="http://libdovecot-storage.so" class="normal-link normal-link-url" rel="noopener noreferrer" moz-do-not-send="true">libdovecot-storage.so</a>.0(mailbox_transaction_commit+0x1e) [0x7f093fb1615e]
-> doveadm(+0x31370) [0x5607cfa1f370] -> doveadm(+0x2b2a8) [0x5607cfa192a8]
-> doveadm(+0x2bfb2) [0x5607cfa19fb2] -> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x215) [0x5607cfa1ae05]
-> doveadm(doveadm_cmd_run_ver2+0x57c) [0x5607cfa2bbec] -> doveadm(doveadm_cmd_try_run_ver2+0x37) [0x5607cfa2bc37]
-> doveadm(main+0x1d2) [0x5607cfa09492]
Aborted
</pre>
                  <p>During my tests I also did notice, that the error
                    appears more often depending of mail size and amount
                    of mails in a folder:<br>
                  </p>
                  <pre>Tested with: doveadm -v fts rescan -u <a target="_blank" class="moz-txt-link-abbreviated" href="mailto:some.user@example.com" moz-do-not-send="true">some.user@example.com</a> && doveadm -v index -u <a target="_blank" class="moz-txt-link-abbreviated" href="mailto:some.user@example.com" moz-do-not-send="true">some.user@example.com</a> '*'
1 Mail in INBOX with 9KB -> Error appeared 0 out of 20 times
1 Mail in INBOX with 136KB -> Error appeared 17 out of 20 times
3 Mails in INBOX with 408KB -> Error appeared 12 out of 20 times
20 Mails in INBOX with ~2MB -> Error appeared 0 out of 20 times
</pre>
                  <p>Maybe this info helps anyone.<br>
                    <br>
                    Patrik<br>
                  </p>
                </blockquote>
              </div>
            </blockquote>
            <p>Yeah, I read your mail and that's why I tested with
              different mail sizes.<br>
              I did some more tests, one with large headers (around 700
              lines of a long header line) but small body:<br>
            </p>
            <pre>1 Mail in INBOX with 583KB -> Error appeared 5 out of 20 times
</pre>
            <p> and another with a large mail body but normal headers:</p>
            <pre>1 Mail in INBOX with 585KB -> Error appeared 0 out of 20 times
</pre>
            I guess this kinda confirms your guess, but interestingly
            shows less errors than my previous test with a large header
            136KB Mail.<br>
            <p><br>
              Patrik<br>
            </p>
          </blockquote>
          <p>I did notice that changing the batch_size in<br>
          </p>
          <pre>fts_solr = url=<a target="_blank" class="moz-txt-link-freetext" href="http://solr.example.org:8983/solr/" moz-do-not-send="true">http://solr.example.org:8983/solr/</a> soft_commit=yes batch_size=1000
</pre>
          <p>does have an influence in how often the error occurs.
            Setting it to 1 or some huge number like 10000 reduces the
            chances quite a bit but not completely and also causes lots
            of small , or few but quite large requests to Solr, so its
            not a practical workaround.<br>
            <br>
            Whats the current state of this bug? A fix for it would be
            very welcome, as it causes some trouble in our setup.<br>
            <br>
            <br>
            Regards<br>
          </p>
        </blockquote>
        <p>Hi</p>
        <p>would someone care to post (preferably a link) to a test
          email with which the problem can be reproduced?</p>
        <p>thanks</p>
        <p>Joh<br>
        </p>
      </blockquote>
      <p>I can confirm that this is an error that was introduced in
        2.3.11.3. It did not happen on 2.3.10. I see it occurring on a
        Centos 7 installation compiled from source. Between these two
        releases I didn't see changes on the fts-solr code, but I do see
        some changes on the lib-http code. I suspect that the changes
        done on lib-http have introduced this issue. <br>
      </p>
      <p>One temporary solution might be to roll back the relevant
        commits that impacted lib-http and see if the problem resolves.
        Given the info in this thread about the records arriving on
        solr, I'm wondering if this is just a problem where the data has
        been split into more than one request, but the last one has no
        data left in it.<br>
      </p>
      <p>John<br>
      </p>
    </blockquote>
    <p>Hi</p>
    <p>I did a bit of tracing. From the albeit limited examples that I
      saw the panic and assert segfault always happens when 
      req->payload_finished is true. If that observation is correct
      in general, then the patch is simple, just put the test for
      payload_finished before the assert for payload_input and
      payload_output. Would anyone with the problem care to try it and
      give feedback?<br>
    </p>
    <p>diff -ur dovecot-2.3.11.3-orig/src/lib-http/http-client-request.c
      dovecot-2.3.11.3/src/lib-http/http-client-request.c<br>
      --- dovecot-2.3.11.3-orig/src/lib-http/http-client-request.c   
      2020-08-12 14:20:41.000000000 +0200<br>
      +++ dovecot-2.3.11.3/src/lib-http/http-client-request.c 2020-10-27
      13:06:09.352973130 +0100<br>
      @@ -1229,12 +1229,12 @@<br>
              const char *error;<br>
              uoff_t offset;<br>
      <br>
      -       i_assert(req->payload_input != NULL);<br>
      -       i_assert(req->payload_output != NULL);<br>
      -<br>
              if (req->payload_finished)<br>
                      return
      http_client_request_finish_payload_out(req);<br>
      <br>
      +       i_assert(req->payload_input != NULL);<br>
      +       i_assert(req->payload_output != NULL);<br>
      +<br>
              io_remove(&conn->io_req_payload);<br>
      <br>
              /* chunked ostream needs to write to the parent stream's
      buffer */<br>
      <br>
    </p>
  
</blockquote></div></body></html>