How to use xapian with non-text attachments

[André Rodier]

Hello,

The version packaged on Bullseye is slightly out of date, I have filled
a bug report:

https://bugs.debian.org/985654

Thanks to the maintainers for their hard work!

André

On Sun, 2021-03-21 at 10:51 +0000, André Rodier wrote:
> Hello,
> 
> I am developing a hosting platform on Debian Bullseye, with Dovecot
> amongst other tools.
> 
> I am trying to use the xapian full test search plugin, but I can see
> the attachments are skipped:
> 
> This is what I have in the logs when running the indexing in verbose
> mode:
> 
> -------------------------------------------------------------------
> ----
> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_check_access
> doveadm(camille): Info: FTS Xapian: Memory stats : Used = 56 MB, Free
> =
> 66 MB
> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_index_hdr
> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_query
> doveadm(camille): Info: FTS Xapian: Query= uid:"44"
> doveadm(camille): Info: FTS Xapian: Ngram(S) -> 63 items (total 0 KB)
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_unset_build_key
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_set_build_key
> doveadm(camille): Info: FTS Xapian: New part (Header=Message-
> Id,Type=(null),Disposition=(null))
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_build_more
> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_check_access
> doveadm(camille): Info: FTS Xapian: Memory stats : Used = 56 MB, Free
> =
> 66 MB
> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_index_hdr
> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_query
> doveadm(camille): Info: FTS Xapian: Query= uid:"44"
> doveadm(camille): Info: FTS Xapian: Ngram(XMID) -> 4 items (total 0
> KB)
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_unset_build_key
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_set_build_key
> doveadm(camille): Info: FTS Xapian: New part (Header=X-
> Mailer,Type=(null),Disposition=(null))
> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
> 'xmailer'
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_set_build_key
> doveadm(camille): Info: FTS Xapian: New part (Header=MIME-
> Version,Type=(null),Disposition=(null))
> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
> 'mimeversion'
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_set_build_key
> doveadm(camille): Info: FTS Xapian: New part (Header=Content-
> Type,Type=(null),Disposition=(null))
> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
> 'contenttype'
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_set_build_key
> doveadm(camille): Info: FTS Xapian: New part (Header=Authentication-
> Results,Type=(null),Disposition=(null))
> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
> 'authenticationresults'
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_set_build_key
> doveadm(camille): Info: FTS Xapian: New part (Header=X-AV-
> Checked,Type=(null),Disposition=(null))
> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
> 'xavchecked'
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_set_build_key
> doveadm(camille): Info: FTS Xapian: New part (Header=Content-
> Type,Type=(null),Disposition=(null))
> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
> 'contenttype'
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_set_build_key
> doveadm(camille): Info: FTS Xapian: New part
> (Header=(null),Type=text/plain,Disposition=(null))
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_build_more
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_unset_build_key
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_set_build_key
> doveadm(camille): Info: FTS Xapian: New part (Header=Content-
> Type,Type=(null),Disposition=(null))
> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
> 'contenttype'
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_set_build_key
> doveadm(camille): Info: FTS Xapian: New part (Header=Content-
> Description,Type=(null),Disposition=(null))
> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
> 'contentdescription'
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_set_build_key
> doveadm(camille): Info: FTS Xapian: New part (Header=Content-
> Disposition,Type=(null),Disposition=(null))
> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
> 'contentdisposition'
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_set_build_key
> doveadm(camille): Info: FTS Xapian: New part (Header=Content-
> Transfer-
> Encoding,Type=(null),Disposition=(null))
> doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
> 'contenttransferencoding'
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_set_build_key
> doveadm(camille): Info: FTS Xapian: New part
> (Header=(null),Type=text/csv,Disposition=attachment;
> filename="file.csv")
> doveadm(camille): Info: FTS Xapian: Skipping part of type 'text/csv'
> and disposition 'attachment; filename="file.csv"'
> doveadm(camille): Info: FTS Xapian:
> fts_backend_xapian_update_set_mailbox
> doveadm(camille): Info: FTS Xapian: Unset box 'INBOX'
> (c0d4e304584e5460dae30000075d7e67)
> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_oldbox
> doveadm(camille): Info: FTS Xapian: Done indexing 'INBOX'
> (c0d4e304584e5460dae30000075d7e67) (13 msgs in 261 ms, rate: 49.8)
> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_release
> (unset_box)
> doveadm(camille): Info: FTS Xapian: Committed 'unset_box' in 17 ms
> doveadm(camille): Info: FTS Xapian: Box is empty
> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_deinit
> (/home/users/camille/mails/indexes/xapian-indexes)
> doveadm(camille): Info: FTS Xapian: fts_backend_xapian_release
> (update_deinit)
> doveadm(camille): Info: FTS Xapian: Committed 'update_deinit' in 0 ms
> doveadm(camille): Info: FTS Xapian: Deinit
> /home/users/camille/mails/indexes/xapian-indexes)
> -------------------------------------------------------------------
> ----
> 
> Especially thos line:
> 
> doveadm(camille): Info: FTS Xapian: Skipping part of type 'text/csv'
> and disposition 'attachment; filename="file.csv"'
> 
> I have seen this as a bug on fts-xapian: 
> https://github.com/grosjo/fts-xapian/issues/68
> 
> Which then references a PR on Dovecot core itself:
> 
> https://github.com/dovecot/core/pull/155
> 
> Any idea on how to use xapian with non-text attachments ?
> 
> Thanks a lot for your help!
> 
> André
>