On 08/10/2019 02:52, Stephan Bosch via dovecot wrote:

On 07/10/2019 15:13, Fabian Kuran via dovecot wrote:

...

Hi,

...

Am 23.09.2019 um 09:36 schrieb Filip Hanes mailto:filip.hanes@ringier.sk>:

št 19. 9. 2019 o 15:30 Fabian via dovecot mailto:dovecot@dovecot.org> napísal(a):

    Are there any other hints or tips regarding this „invalid uid ‚0‘"-message?

from fts-solr source: case SOLR_XML_CONTENT_STATE_UID: if (uint32_parse(str, len, &ctx->uid) < 0 || ctx->uid == 0) { i_error("fts_solr: received invalid uid '%s'", t_strndup(str, len));

It means that plugin tries to parse '0' when in state of parsing uid in xml response from solr or there is a bug in xml response parser. I would enable rawlog_dir in config and look there for unusual response or extra tags near usual uid in xml response. For example: |plugin { fts = solr fts_solr = url=... rawlog_dir=/var/log/fts-solr/ }|

Thanks for this tip! The rawlog_dir option is not yet supported in our version 2.3.4. What we could provide now would be the XML-Response from Solr. The corresponding search query sometimes causes this error. But Solr's answer seems to always remain the same

• in my opinion this already indicates a bug of Dovecot?

The XML response can be downloaded here: https://owncloud.eideo.de/index.php/s/8oyJaf5HBK6zdbr

Based on the XML response above, I investigated this problem thoroughly and determined that this is a pretty severe bug in the Solr XML response parsing code. This occurs only when the response is rather large and the boundary between two read chunks falls in the middle of a numeric value (that happens to end in '0').

Tracking internally as DOP-1470.

I think this may also explain some of the weird reports about invalid UIDs that we've seen in recent history on this mailing list.

Preliminary fix is here: https://github.com/stephanbosch/dovecot-core/commits/fix-solr-xml-parser

It's big due to some code restructuring for making a proper unit test. I am not sure when this will be in a release.

Regards,

Stephan.

On 10/10/2019 04:05, Larry Rosenman via dovecot wrote:

Is there a way to get this as a patchset against 2.3.8?

https://github.com/stephanbosch/dovecot-core/tree/fix-solr-xml-parser-2.3.8

This hasn't seen any QA yet, so handle with care.

Regards,

Stephan.

On Tue, Oct 8, 2019 at 12:03 PM Stephan Bosch via dovecot mailto:dovecot@dovecot.org> wrote:

On 08/10/2019 02:52, Stephan Bosch via dovecot wrote:
>
>
> On 07/10/2019 15:13, Fabian Kuran via dovecot wrote:
>> Hi,
>>
>>> Am 23.09.2019 um 09:36 schrieb Filip Hanes
<filip.hanes@ringier.sk <mailto:filip.hanes@ringier.sk>
>>> <mailto:filip.hanes@ringier.sk <mailto:filip.hanes@ringier.sk>>>:
>>>
>>>
>>> št 19. 9. 2019 o 15:30 Fabian via dovecot <dovecot@dovecot.org
<mailto:dovecot@dovecot.org>
>>> <mailto:dovecot@dovecot.org <mailto:dovecot@dovecot.org>>>
napísal(a):
>>>
>>>     Are there any other hints or tips regarding this „invalid uid
>>>     ‚0‘"-message?
>>>
>>> from fts-solr source:
>>>     case SOLR_XML_CONTENT_STATE_UID:
>>>
        if (uint32_parse(str, len, &ctx->uid) < 0 || ctx->uid == 0) {
>>>
            i_error("fts_solr: received invalid uid '%s'", t_strndup(str, len));

>>>
>>> It means that plugin tries to parse '0' when in state of
parsing uid
>>> in xml response from solr or there is a bug in xml response
parser.
>>> I would enable rawlog_dir in config and look there for unusual
>>> response or extra tags near usual uid in xml response. For
example:
>>> |plugin { fts = solr fts_solr = url=...
>>> rawlog_dir=/var/log/fts-solr/ }|
>>>
>>
>> Thanks for this tip! The rawlog_dir option is not yet supported in
>> our version 2.3.4. What we could provide now would be the
>> XML-Response from Solr. The corresponding search query sometimes
>> causes this error. But Solr's answer seems to always remain the
same
>> - in my opinion this already indicates a bug of Dovecot?
>>
>> The XML response can be downloaded here:
>> https://owncloud.eideo.de/index.php/s/8oyJaf5HBK6zdbr
>
> Based on the XML response above, I investigated this problem
> thoroughly and determined that this is a pretty severe bug in
the Solr
> XML response parsing code. This occurs only when the response is
> rather large and the boundary between two read chunks falls in the
> middle of a numeric value (that happens to end in '0').
>
> Tracking internally as DOP-1470.
>
> I think this may also explain some of the weird reports about
invalid
> UIDs that we've seen in recent history on this mailing list.

Preliminary fix is here:
https://github.com/stephanbosch/dovecot-core/commits/fix-solr-xml-parser

It's big due to some code restructuring for making a proper unit
test. I
am not sure when this will be in a release.

Regards,

Stephan.

-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com mailto:larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Lets hope 2.3.10 ...

Aki

On 10/12/2019 20:57 Larry Rosenman via dovecot dovecot@dovecot.org wrote:

Any news on this getting into a release?

On Sun, Oct 13, 2019 at 2:24 AM Stephan Bosch stephan@rename-it.nl wrote:

...

On 10/10/2019 04:05, Larry Rosenman via dovecot wrote:

...

Is there a way to get this as a patchset against 2.3.8?

https://github.com/stephanbosch/dovecot-core/tree/fix-solr-xml-parser-2.3.8

This hasn't seen any QA yet, so handle with care.

Regards,

Stephan.

...

On Tue, Oct 8, 2019 at 12:03 PM Stephan Bosch via dovecot mailto:dovecot@dovecot.org> wrote:

On 08/10/2019 02:52, Stephan Bosch via dovecot wrote:

...

On 07/10/2019 15:13, Fabian Kuran via dovecot wrote:

...

Hi,

...

Am 23.09.2019 um 09:36 schrieb Filip Hanes

mailto:filip.hanes@ringier.sk

...

...

...

mailto:filip.hanes@ringier.sk>>:

št 19. 9. 2019 o 15:30 Fabian via dovecot mailto:dovecot@dovecot.org mailto:dovecot@dovecot.org>> napísal(a):

Are there any other hints or tips regarding this „invalid uid ‚0‘"-message?

from fts-solr source: case SOLR_XML_CONTENT_STATE_UID:

if (uint32_parse(str, len, &ctx->uid) < 0 || ctx->uid == 0) {

...

i_error("fts_solr: received invalid uid '%s'", t_strndup(str, len));

...

...

...

It means that plugin tries to parse '0' when in state of

...

in xml response from solr or there is a bug in xml response

parsing uid parser.

...

I would enable rawlog_dir in config and look there for unusual response or extra tags near usual uid in xml response. For example: |plugin { fts = solr fts_solr = url=... rawlog_dir=/var/log/fts-solr/ }|

Thanks for this tip! The rawlog_dir option is not yet supported in our version 2.3.4. What we could provide now would be the XML-Response from Solr. The corresponding search query sometimes causes this error. But Solr's answer seems to always remain the same

• in my opinion this already indicates a bug of Dovecot?

The XML response can be downloaded here: https://owncloud.eideo.de/index.php/s/8oyJaf5HBK6zdbr

Based on the XML response above, I investigated this problem thoroughly and determined that this is a pretty severe bug in the Solr XML response parsing code. This occurs only when the response is rather large and the boundary between two read chunks falls in the middle of a numeric value (that happens to end in '0').

Tracking internally as DOP-1470.

I think this may also explain some of the weird reports about invalid UIDs that we've seen in recent history on this mailing list.

Preliminary fix is here: https://github.com/stephanbosch/dovecot-core/commits/fix-solr-xml-parser

It's big due to some code restructuring for making a proper unit test. I am not sure when this will be in a release.

Regards,

Stephan.

-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com mailto:larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

--

Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail:larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106