Re: How to use xapian with non-text attachments

21 Mar 2021

      Hello,
The version packaged on Bullseye is slightly out of date, I have filled
a bug report:
https://bugs.debian.org/985654
Thanks to the maintainers for their hard work!
André
On Sun, 2021-03-21 at 10:51 +0000, André Rodier wrote:
...
Hello,
I am developing a hosting platform on Debian Bullseye, with Dovecot
amongst other tools.
I am trying to use the xapian full test search plugin, but I can see
the attachments are skipped:
This is what I have in the logs when running the indexing in verbose
mode:

doveadm(camille): Info: FTS Xapian: fts_backend_xapian_check_access
doveadm(camille): Info: FTS Xapian: Memory stats : Used = 56 MB, Free
66 MB
doveadm(camille): Info: FTS Xapian: fts_backend_xapian_index_hdr
doveadm(camille): Info: FTS Xapian: fts_backend_xapian_query
doveadm(camille): Info: FTS Xapian: Query= uid:"44"
doveadm(camille): Info: FTS Xapian: Ngram(S) -> 63 items (total 0 KB)
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_unset_build_key
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_set_build_key
doveadm(camille): Info: FTS Xapian: New part (Header=Message-
Id,Type=(null),Disposition=(null))
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_build_more
doveadm(camille): Info: FTS Xapian: fts_backend_xapian_check_access
doveadm(camille): Info: FTS Xapian: Memory stats : Used = 56 MB, Free
66 MB
doveadm(camille): Info: FTS Xapian: fts_backend_xapian_index_hdr
doveadm(camille): Info: FTS Xapian: fts_backend_xapian_query
doveadm(camille): Info: FTS Xapian: Query= uid:"44"
doveadm(camille): Info: FTS Xapian: Ngram(XMID) -> 4 items (total 0
KB)
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_unset_build_key
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_set_build_key
doveadm(camille): Info: FTS Xapian: New part (Header=X-
Mailer,Type=(null),Disposition=(null))
doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
'xmailer'
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_set_build_key
doveadm(camille): Info: FTS Xapian: New part (Header=MIME-
Version,Type=(null),Disposition=(null))
doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
'mimeversion'
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_set_build_key
doveadm(camille): Info: FTS Xapian: New part (Header=Content-
Type,Type=(null),Disposition=(null))
doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
'contenttype'
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_set_build_key
doveadm(camille): Info: FTS Xapian: New part (Header=Authentication-
Results,Type=(null),Disposition=(null))
doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
'authenticationresults'
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_set_build_key
doveadm(camille): Info: FTS Xapian: New part (Header=X-AV-
Checked,Type=(null),Disposition=(null))
doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
'xavchecked'
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_set_build_key
doveadm(camille): Info: FTS Xapian: New part (Header=Content-
Type,Type=(null),Disposition=(null))
doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
'contenttype'
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_set_build_key
doveadm(camille): Info: FTS Xapian: New part
(Header=(null),Type=text/plain,Disposition=(null))
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_build_more
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_unset_build_key
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_set_build_key
doveadm(camille): Info: FTS Xapian: New part (Header=Content-
Type,Type=(null),Disposition=(null))
doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
'contenttype'
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_set_build_key
doveadm(camille): Info: FTS Xapian: New part (Header=Content-
Description,Type=(null),Disposition=(null))
doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
'contentdescription'
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_set_build_key
doveadm(camille): Info: FTS Xapian: New part (Header=Content-
Disposition,Type=(null),Disposition=(null))
doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
'contentdisposition'
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_set_build_key
doveadm(camille): Info: FTS Xapian: New part (Header=Content-
Transfer-
Encoding,Type=(null),Disposition=(null))
doveadm(camille): Info: FTS Xapian: Unknown header (indexing)
'contenttransferencoding'
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_set_build_key
doveadm(camille): Info: FTS Xapian: New part
(Header=(null),Type=text/csv,Disposition=attachment;
filename="file.csv")
doveadm(camille): Info: FTS Xapian: Skipping part of type 'text/csv'
and disposition 'attachment; filename="file.csv"'
doveadm(camille): Info: FTS Xapian:
fts_backend_xapian_update_set_mailbox
doveadm(camille): Info: FTS Xapian: Unset box 'INBOX'
(c0d4e304584e5460dae30000075d7e67)
doveadm(camille): Info: FTS Xapian: fts_backend_xapian_oldbox
doveadm(camille): Info: FTS Xapian: Done indexing 'INBOX'
(c0d4e304584e5460dae30000075d7e67) (13 msgs in 261 ms, rate: 49.8)
doveadm(camille): Info: FTS Xapian: fts_backend_xapian_release
(unset_box)
doveadm(camille): Info: FTS Xapian: Committed 'unset_box' in 17 ms
doveadm(camille): Info: FTS Xapian: Box is empty
doveadm(camille): Info: FTS Xapian: fts_backend_xapian_update_deinit
(/home/users/camille/mails/indexes/xapian-indexes)
doveadm(camille): Info: FTS Xapian: fts_backend_xapian_release
(update_deinit)
doveadm(camille): Info: FTS Xapian: Committed 'update_deinit' in 0 ms
doveadm(camille): Info: FTS Xapian: Deinit
/home/users/camille/mails/indexes/xapian-indexes)

Especially thos line:
doveadm(camille): Info: FTS Xapian: Skipping part of type 'text/csv'
and disposition 'attachment; filename="file.csv"'
I have seen this as a bug on fts-xapian:
https://github.com/grosjo/fts-xapian/issues/68
Which then references a PR on Dovecot core itself:
https://github.com/dovecot/core/pull/155
Any idea on how to use xapian with non-text attachments ?
Thanks a lot for your help!
André

Re: How to use xapian with non-text attachments

André Rodier

doveadm(camille): Info: FTS Xapian: fts_backend_xapian_check_access doveadm(camille): Info: FTS Xapian: Memory stats : Used = 56 MB, Free