How to use xapian with non-text attachments

[Joan Moreau]

It is now very out of date.

@Jello : Kindly update please

On 2021-03-21 12:58, André Rodier wrote:

> Hello,
> 
> The version packaged on Bullseye is slightly out of date, I have filled
> a bug report:
> 
> https://bugs.debian.org/985654
> 
> Thanks to the maintainers for their hard work!
> 
> André
> 
> On Sun, 2021-03-21 at 10:51 +0000, André Rodier wrote:
> 
>> Hello,
>> 
>> I am developing a hosting platform on Debian Bullseye, with Dovecot
>> amongst other tools.
>> 
>> I am trying to use the xapian full test search plugin, but I can see
>> the attachments are skipped:
>> 
>> This is what I have in the logs when running the indexing in verbose
>> mode:
>> 
>> -------------------------------------------------------------------
>> ----
>> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_check_access
>> doveadm(camille): Info: FTS Xapian: Memory stats : Used = 56 MB, Free
>> =
>> 66 MB
>> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_index_hdr
>> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_query
>> doveadm(camille): Info: FTS Xapian: Query= uid:"44"
>> doveadm(camille): Info: FTS Xapian: Ngram(S) -> 63 items (total 0 KB)
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_unset_build_key
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_set_build_key
>> doveadm(camille): Info: FTS Xapian: New part (Header=Message-
>> Id,Type=(null),Disposition=(null))
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_build_more
>> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_check_access
>> doveadm(camille): Info: FTS Xapian: Memory stats : Used = 56 MB, Free
>> =
>> 66 MB
>> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_index_hdr
>> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_query
>> doveadm(camille): Info: FTS Xapian: Query= uid:"44"
>> doveadm(camille): Info: FTS Xapian: Ngram(XMID) -> 4 items (total 0
>> KB)
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_unset_build_key
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_set_build_key
>> doveadm(camille): Info: FTS Xapian: New part (Header=X-
>> Mailer,Type=(null),Disposition=(null))
>> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
>> 'xmailer'
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_set_build_key
>> doveadm(camille): Info: FTS Xapian: New part (Header=MIME-
>> Version,Type=(null),Disposition=(null))
>> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
>> 'mimeversion'
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_set_build_key
>> doveadm(camille): Info: FTS Xapian: New part (Header=Content-
>> Type,Type=(null),Disposition=(null))
>> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
>> 'contenttype'
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_set_build_key
>> doveadm(camille): Info: FTS Xapian: New part (Header=Authentication-
>> Results,Type=(null),Disposition=(null))
>> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
>> 'authenticationresults'
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_set_build_key
>> doveadm(camille): Info: FTS Xapian: New part (Header=X-AV-
>> Checked,Type=(null),Disposition=(null))
>> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
>> 'xavchecked'
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_set_build_key
>> doveadm(camille): Info: FTS Xapian: New part (Header=Content-
>> Type,Type=(null),Disposition=(null))
>> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
>> 'contenttype'
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_set_build_key
>> doveadm(camille): Info: FTS Xapian: New part
>> (Header=(null),Type=text/plain,Disposition=(null))
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_build_more
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_unset_build_key
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_set_build_key
>> doveadm(camille): Info: FTS Xapian: New part (Header=Content-
>> Type,Type=(null),Disposition=(null))
>> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
>> 'contenttype'
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_set_build_key
>> doveadm(camille): Info: FTS Xapian: New part (Header=Content-
>> Description,Type=(null),Disposition=(null))
>> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
>> 'contentdescription'
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_set_build_key
>> doveadm(camille): Info: FTS Xapian: New part (Header=Content-
>> Disposition,Type=(null),Disposition=(null))
>> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
>> 'contentdisposition'
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_set_build_key
>> doveadm(camille): Info: FTS Xapian: New part (Header=Content-
>> Transfer-
>> Encoding,Type=(null),Disposition=(null))
>> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
>> 'contenttransferencoding'
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_set_build_key
>> doveadm(camille): Info: FTS Xapian: New part
>> (Header=(null),Type=text/csv,Disposition=attachment;
>> filename="file.csv")
>> doveadm(camille): Info: FTS Xapian: Skipping part of type 'text/csv'
>> and disposition 'attachment; filename="file.csv"'
>> doveadm(camille): Info: FTS Xapian:
>> fts_backend_xapian_update_set_mailbox
>> doveadm(camille): Info: FTS Xapian: Unset box 'INBOX'
>> (c0d4e304584e5460dae30000075d7e67)
>> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_oldbox
>> doveadm(camille): Info: FTS Xapian: Done indexing 'INBOX'
>> (c0d4e304584e5460dae30000075d7e67) (13 msgs in 261 ms, rate: 49.8)
>> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_release
>> (unset_box)
>> doveadm(camille): Info: FTS Xapian: Committed 'unset_box' in 17 ms
>> doveadm(camille): Info: FTS Xapian: Box is empty
>> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_deinit
>> (/home/users/camille/mails/indexes/xapian-indexes)
>> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_release
>> (update_deinit)
>> doveadm(camille): Info: FTS Xapian: Committed 'update_deinit' in 0 ms
>> doveadm(camille): Info: FTS Xapian: Deinit
>> /home/users/camille/mails/indexes/xapian-indexes)
>> -------------------------------------------------------------------
>> ----
>> 
>> Especially thos line:
>> 
>> doveadm(camille): Info: FTS Xapian: Skipping part of type 'text/csv'
>> and disposition 'attachment; filename="file.csv"'
>> 
>> I have seen this as a bug on fts-xapian:
>> https://github.com/grosjo/fts-xapian/issues/68
>> 
>> Which then references a PR on Dovecot core itself:
>> 
>> https://github.com/dovecot/core/pull/155
>> 
>> Any idea on how to use xapian with non-text attachments ?
>> 
>> Thanks a lot for your help!
>> 
>> André
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20210703/5d38c828/attachment-0001.html>