Re: fts_solr: Error: fts_solr: received invalid uid '0'
Hi,
Am 23.09.2019 um 09:36 schrieb Filip Hanes <filip.hanes@ringier.sk>:
št 19. 9. 2019 o 15:30 Fabian via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> napísal(a): Are there any other hints or tips regarding this „invalid uid ‚0‘"-message? from fts-solr source: case SOLR_XML_CONTENT_STATE_UID: if (uint32_parse(str, len, &ctx->uid) < 0 || ctx->uid == 0) { i_error("fts_solr: received invalid uid '%s'", t_strndup(str, len)); It means that plugin tries to parse '0' when in state of parsing uid in xml response from solr or there is a bug in xml response parser. I would enable rawlog_dir in config and look there for unusual response or extra tags near usual uid in xml response. For example: plugin { fts = solr fts_solr = url=... rawlog_dir=/var/log/fts-solr/ }
Thanks for this tip! The rawlog_dir option is not yet supported in our version 2.3.4. What we could provide now would be the XML-Response from Solr. The corresponding search query sometimes causes this error. But Solr's answer seems to always remain the same - in my opinion this already indicates a bug of Dovecot?
The XML response can be downloaded here: https://owncloud.eideo.de/index.php/s/8oyJaf5HBK6zdbr
Kind regards, Fabian
On 07/10/2019 15:13, Fabian Kuran via dovecot wrote:
Hi,
Am 23.09.2019 um 09:36 schrieb Filip Hanes <filip.hanes@ringier.sk <mailto:filip.hanes@ringier.sk>>:
št 19. 9. 2019 o 15:30 Fabian via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> napísal(a):
Are there any other hints or tips regarding this „invalid uid ‚0‘"-message?from fts-solr source: case SOLR_XML_CONTENT_STATE_UID: if (uint32_parse(str, len, &ctx->uid) < 0 || ctx->uid == 0) { i_error("fts_solr: received invalid uid '%s'", t_strndup(str, len)); It means that plugin tries to parse '0' when in state of parsing uid in xml response from solr or there is a bug in xml response parser. I would enable rawlog_dir in config and look there for unusual response or extra tags near usual uid in xml response. For example: |plugin { fts = solr fts_solr = url=... rawlog_dir=/var/log/fts-solr/ }|
Thanks for this tip! The rawlog_dir option is not yet supported in our version 2.3.4. What we could provide now would be the XML-Response from Solr. The corresponding search query sometimes causes this error. But Solr's answer seems to always remain the same - in my opinion this already indicates a bug of Dovecot?
The XML response can be downloaded here: https://owncloud.eideo.de/index.php/s/8oyJaf5HBK6zdbr
Based on the XML response above, I investigated this problem thoroughly and determined that this is a pretty severe bug in the Solr XML response parsing code. This occurs only when the response is rather large and the boundary between two read chunks falls in the middle of a numeric value (that happens to end in '0').
Tracking internally as DOP-1470.
I think this may also explain some of the weird reports about invalid UIDs that we've seen in recent history on this mailing list.
Regards,
Stephan.
On 08/10/2019 02:52, Stephan Bosch via dovecot wrote:
On 07/10/2019 15:13, Fabian Kuran via dovecot wrote:
Hi,
Am 23.09.2019 um 09:36 schrieb Filip Hanes <filip.hanes@ringier.sk <mailto:filip.hanes@ringier.sk>>:
št 19. 9. 2019 o 15:30 Fabian via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> napísal(a):
Are there any other hints or tips regarding this „invalid uid ‚0‘"-message?
from fts-solr source: case SOLR_XML_CONTENT_STATE_UID: if (uint32_parse(str, len, &ctx->uid) < 0 || ctx->uid == 0) { i_error("fts_solr: received invalid uid '%s'", t_strndup(str, len));
It means that plugin tries to parse '0' when in state of parsing uid in xml response from solr or there is a bug in xml response parser. I would enable rawlog_dir in config and look there for unusual response or extra tags near usual uid in xml response. For example: |plugin { fts = solr fts_solr = url=... rawlog_dir=/var/log/fts-solr/ }|
Thanks for this tip! The rawlog_dir option is not yet supported in our version 2.3.4. What we could provide now would be the XML-Response from Solr. The corresponding search query sometimes causes this error. But Solr's answer seems to always remain the same
- in my opinion this already indicates a bug of Dovecot?
The XML response can be downloaded here: https://owncloud.eideo.de/index.php/s/8oyJaf5HBK6zdbr
Based on the XML response above, I investigated this problem thoroughly and determined that this is a pretty severe bug in the Solr XML response parsing code. This occurs only when the response is rather large and the boundary between two read chunks falls in the middle of a numeric value (that happens to end in '0').
Tracking internally as DOP-1470.
I think this may also explain some of the weird reports about invalid UIDs that we've seen in recent history on this mailing list.
Preliminary fix is here: https://github.com/stephanbosch/dovecot-core/commits/fix-solr-xml-parser
It's big due to some code restructuring for making a proper unit test. I am not sure when this will be in a release.
Regards,
Stephan.
Is there a way to get this as a patchset against 2.3.8?
On Tue, Oct 8, 2019 at 12:03 PM Stephan Bosch via dovecot < dovecot@dovecot.org> wrote:
On 08/10/2019 02:52, Stephan Bosch via dovecot wrote:
On 07/10/2019 15:13, Fabian Kuran via dovecot wrote:
Hi,
Am 23.09.2019 um 09:36 schrieb Filip Hanes <filip.hanes@ringier.sk <mailto:filip.hanes@ringier.sk>>:
št 19. 9. 2019 o 15:30 Fabian via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> napísal(a):
Are there any other hints or tips regarding this „invalid uid ‚0‘"-message?from fts-solr source: case SOLR_XML_CONTENT_STATE_UID: if (uint32_parse(str, len, &ctx->uid) < 0 || ctx->uid == 0) {
i_error("fts_solr: received invalid uid '%s'", t_strndup(str, len));It means that plugin tries to parse '0' when in state of parsing uid in xml response from solr or there is a bug in xml response parser. I would enable rawlog_dir in config and look there for unusual response or extra tags near usual uid in xml response. For example: |plugin { fts = solr fts_solr = url=... rawlog_dir=/var/log/fts-solr/ }|
Thanks for this tip! The rawlog_dir option is not yet supported in our version 2.3.4. What we could provide now would be the XML-Response from Solr. The corresponding search query sometimes causes this error. But Solr's answer seems to always remain the same
- in my opinion this already indicates a bug of Dovecot?
The XML response can be downloaded here: https://owncloud.eideo.de/index.php/s/8oyJaf5HBK6zdbr
Based on the XML response above, I investigated this problem thoroughly and determined that this is a pretty severe bug in the Solr XML response parsing code. This occurs only when the response is rather large and the boundary between two read chunks falls in the middle of a numeric value (that happens to end in '0').
Tracking internally as DOP-1470.
I think this may also explain some of the weird reports about invalid UIDs that we've seen in recent history on this mailing list.
Preliminary fix is here: https://github.com/stephanbosch/dovecot-core/commits/fix-solr-xml-parser
It's big due to some code restructuring for making a proper unit test. I am not sure when this will be in a release.
Regards,
Stephan.
-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
On 10/10/2019 04:05, Larry Rosenman via dovecot wrote:
Is there a way to get this as a patchset against 2.3.8?
https://github.com/stephanbosch/dovecot-core/tree/fix-solr-xml-parser-2.3.8
This hasn't seen any QA yet, so handle with care.
Regards,
Stephan.
On Tue, Oct 8, 2019 at 12:03 PM Stephan Bosch via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> wrote:
On 08/10/2019 02:52, Stephan Bosch via dovecot wrote: > > > On 07/10/2019 15:13, Fabian Kuran via dovecot wrote: >> Hi, >> >>> Am 23.09.2019 um 09:36 schrieb Filip Hanes <filip.hanes@ringier.sk <mailto:filip.hanes@ringier.sk> >>> <mailto:filip.hanes@ringier.sk <mailto:filip.hanes@ringier.sk>>>: >>> >>> >>> št 19. 9. 2019 o 15:30 Fabian via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org> >>> <mailto:dovecot@dovecot.org <mailto:dovecot@dovecot.org>>> napísal(a): >>> >>> Are there any other hints or tips regarding this „invalid uid >>> ‚0‘"-message? >>> >>> from fts-solr source: >>> case SOLR_XML_CONTENT_STATE_UID: >>> if (uint32_parse(str, len, &ctx->uid) < 0 || ctx->uid == 0) { >>> i_error("fts_solr: received invalid uid '%s'", t_strndup(str, len)); >>> >>> It means that plugin tries to parse '0' when in state of parsing uid >>> in xml response from solr or there is a bug in xml response parser. >>> I would enable rawlog_dir in config and look there for unusual >>> response or extra tags near usual uid in xml response. For example: >>> |plugin { fts = solr fts_solr = url=... >>> rawlog_dir=/var/log/fts-solr/ }| >>> >> >> Thanks for this tip! The rawlog_dir option is not yet supported in >> our version 2.3.4. What we could provide now would be the >> XML-Response from Solr. The corresponding search query sometimes >> causes this error. But Solr's answer seems to always remain the same >> - in my opinion this already indicates a bug of Dovecot? >> >> The XML response can be downloaded here: >> https://owncloud.eideo.de/index.php/s/8oyJaf5HBK6zdbr > > Based on the XML response above, I investigated this problem > thoroughly and determined that this is a pretty severe bug in the Solr > XML response parsing code. This occurs only when the response is > rather large and the boundary between two read chunks falls in the > middle of a numeric value (that happens to end in '0'). > > Tracking internally as DOP-1470. > > I think this may also explain some of the weird reports about invalid > UIDs that we've seen in recent history on this mailing list. Preliminary fix is here: https://github.com/stephanbosch/dovecot-core/commits/fix-solr-xml-parser It's big due to some code restructuring for making a proper unit test. I am not sure when this will be in a release. Regards, Stephan.-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com <mailto:larryrtx@gmail.com> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
Any news on this getting into a release?
On Sun, Oct 13, 2019 at 2:24 AM Stephan Bosch <stephan@rename-it.nl> wrote:
On 10/10/2019 04:05, Larry Rosenman via dovecot wrote:
Is there a way to get this as a patchset against 2.3.8?
https://github.com/stephanbosch/dovecot-core/tree/fix-solr-xml-parser-2.3.8
This hasn't seen any QA yet, so handle with care.
Regards,
Stephan.
On Tue, Oct 8, 2019 at 12:03 PM Stephan Bosch via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> wrote:
On 08/10/2019 02:52, Stephan Bosch via dovecot wrote: > > > On 07/10/2019 15:13, Fabian Kuran via dovecot wrote: >> Hi, >> >>> Am 23.09.2019 um 09:36 schrieb Filip Hanes <filip.hanes@ringier.sk <mailto:filip.hanes@ringier.sk> >>> <mailto:filip.hanes@ringier.sk <mailto:filip.hanes@ringier.sk: >>> >>> >>> št 19. 9. 2019 o 15:30 Fabian via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org> >>> <mailto:dovecot@dovecot.org <mailto:dovecot@dovecot.org>>> napísal(a): >>> >>> Are there any other hints or tips regarding this „invalid uid >>> ‚0‘"-message? >>> >>> from fts-solr source: >>> case SOLR_XML_CONTENT_STATE_UID: >>> if (uint32_parse(str, len, &ctx->uid) < 0 || ctx->uid == 0) { >>>
i_error("fts_solr: received invalid uid '%s'", t_strndup(str, len));>>> >>> It means that plugin tries to parse '0' when in state of parsing uid >>> in xml response from solr or there is a bug in xml response parser. >>> I would enable rawlog_dir in config and look there for unusual >>> response or extra tags near usual uid in xml response. For example: >>> |plugin { fts = solr fts_solr = url=... >>> rawlog_dir=/var/log/fts-solr/ }| >>> >> >> Thanks for this tip! The rawlog_dir option is not yet supported in >> our version 2.3.4. What we could provide now would be the >> XML-Response from Solr. The corresponding search query sometimes >> causes this error. But Solr's answer seems to always remain the same >> - in my opinion this already indicates a bug of Dovecot? >> >> The XML response can be downloaded here: >> https://owncloud.eideo.de/index.php/s/8oyJaf5HBK6zdbr > > Based on the XML response above, I investigated this problem > thoroughly and determined that this is a pretty severe bug in the Solr > XML response parsing code. This occurs only when the response is > rather large and the boundary between two read chunks falls in the > middle of a numeric value (that happens to end in '0'). > > Tracking internally as DOP-1470. > > I think this may also explain some of the weird reports about invalid > UIDs that we've seen in recent history on this mailing list. Preliminary fix is here:https://github.com/stephanbosch/dovecot-core/commits/fix-solr-xml-parser
It's big due to some code restructuring for making a proper unit test. I am not sure when this will be in a release. Regards, Stephan.-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com <mailto:larryrtx@gmail.com> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
Lets hope 2.3.10 ...
Aki
On 10/12/2019 20:57 Larry Rosenman via dovecot <dovecot@dovecot.org> wrote:
Any news on this getting into a release?
On Sun, Oct 13, 2019 at 2:24 AM Stephan Bosch <stephan@rename-it.nl> wrote:
On 10/10/2019 04:05, Larry Rosenman via dovecot wrote:
Is there a way to get this as a patchset against 2.3.8?
https://github.com/stephanbosch/dovecot-core/tree/fix-solr-xml-parser-2.3.8
This hasn't seen any QA yet, so handle with care.
Regards,
Stephan.
On Tue, Oct 8, 2019 at 12:03 PM Stephan Bosch via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> wrote:
On 08/10/2019 02:52, Stephan Bosch via dovecot wrote:
On 07/10/2019 15:13, Fabian Kuran via dovecot wrote:
Hi,
Am 23.09.2019 um 09:36 schrieb Filip Hanes
<filip.hanes@ringier.sk <mailto:filip.hanes@ringier.sk>
<mailto:filip.hanes@ringier.sk <mailto:filip.hanes@ringier.sk>>>:
št 19. 9. 2019 o 15:30 Fabian via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org> <mailto:dovecot@dovecot.org <mailto:dovecot@dovecot.org>>> napísal(a):
Are there any other hints or tips regarding this „invalid uid ‚0‘"-message?
from fts-solr source: case SOLR_XML_CONTENT_STATE_UID:
if (uint32_parse(str, len, &ctx->uid) < 0 || ctx->uid == 0) {
i_error("fts_solr: received invalid uid '%s'", t_strndup(str, len));
It means that plugin tries to parse '0' when in state of
in xml response from solr or there is a bug in xml response
parsing uid parser.
I would enable rawlog_dir in config and look there for unusual response or extra tags near usual uid in xml response. For example: |plugin { fts = solr fts_solr = url=... rawlog_dir=/var/log/fts-solr/ }|
Thanks for this tip! The rawlog_dir option is not yet supported in our version 2.3.4. What we could provide now would be the XML-Response from Solr. The corresponding search query sometimes causes this error. But Solr's answer seems to always remain the same
- in my opinion this already indicates a bug of Dovecot?
The XML response can be downloaded here: https://owncloud.eideo.de/index.php/s/8oyJaf5HBK6zdbr
Based on the XML response above, I investigated this problem thoroughly and determined that this is a pretty severe bug in the Solr XML response parsing code. This occurs only when the response is rather large and the boundary between two read chunks falls in the middle of a numeric value (that happens to end in '0').
Tracking internally as DOP-1470.
I think this may also explain some of the weird reports about invalid UIDs that we've seen in recent history on this mailing list.
Preliminary fix is here: https://github.com/stephanbosch/dovecot-core/commits/fix-solr-xml-parser
It's big due to some code restructuring for making a proper unit test. I am not sure when this will be in a release.
Regards,
Stephan.
-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com <mailto:larryrtx@gmail.com> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
--
Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail:larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
participants (4)
-
Aki Tuomi
-
Fabian Kuran
-
Larry Rosenman
-
Stephan Bosch