Following up on my report of 22 September.
Please disregard the analysis and the patch I posted earlier in this thread. Both were wrong, and I would rather say so now than have someone build on them.
*1.* The mechanism I described is not what happens. I claimed that iter.get_document() throws DocNotFoundError outside the try block and that the local handler never sees it. That is not the case: Xapian::Document is a lazy proxy, so get_document() does not fetch anything, and the exception is actually thrown later by replace_document() inside the try, where the local catch does see it and returns -1. My description of how the exception escapes was simply incorrect.
*2.* The reproduction recipe contains a line that does nothing: doveadm -o fts_flatcurve_rotate_count=25 has no effect. Verified with doveconf — the setting is only picked up from the config file. Anyone who tried that recipe would not have reproduced what I described.
And the patch is not a fix. Skipping the documents that fail to copy treats them as unreadable leftovers of deleted messages, which they are not — the documents are intact. The failure is in how they are addressed during the rebuild, so skipping them silently drops live data from the optimized index. The patch makes the error go away while making the result worse.
The underlying cause is elsewhere, and we have since identified and reproduced it deterministically. We are testing a different fix in the affected code path. I do not want to post a second half-verified theory, so I will follow up with the corrected analysis and a patch once the testing is complete.
Apologies for the noise.
-- Best regards, Ihor Rusyn
Following up on my report of 22 September.
Please disregard the analysis and the patch I posted earlier in this thread. Both were wrong, and I would rather say so now than have someone build on them.
The mechanism I described is not what happens. I claimed that iter.get_document() throws DocNotFoundError outside the try block and that the local handler never sees it. That is not the case: Xapian::Document is a lazy proxy, so get_document() does not fetch anything, and the exception is actually thrown later by replace_document() inside the try, where the local catch does see it and returns -1. My description of how the exception escapes was simply incorrect.
The reproduction recipe contains a line that does nothing: doveadm -o fts_flatcurve_rotate_count=25 has no effect. Verified with doveconf -- the setting is only picked up from the config file. Anyone who tried that recipe would not have reproduced what I described.
And the patch is not a fix. Skipping the documents that fail to copy treats them as unreadable leftovers of deleted messages, which they are not -- the documents are intact. The failure is in how they are addressed during the rebuild, so skipping them silently drops live data from the optimized index. The patch makes the error go away while making the result worse.
The underlying cause is elsewhere, and we have since identified and reproduced it deterministically. We are testing a different fix in the affected code path. I do not want to post a second half-verified theory, so I will follow up with the corrected analysis and a patch once the testing is complete.
Apologies for the noise.
-- Best regards, Ihor Rusyn