Thank you for all tips. The design look more clear to me now.

I have one more question. I looked into fts_build_want_index_part() and I saw that I need to add some flags to message_part_flags, what values should I choose? My first approach was to follow your schema and set MESSAGE_PART_FLAG_ATTACHMENT = 0x16. There is any problem with this?

I already had changed parse_content_type() to set ctx->part->flags correctly but if i choose my custom flag dovecot assume that all attachment lines are headers. I already tried to set those ctx->part->flags as TEXT and the fts_backend was feeded correctly with all attachment lines.

I don't know if this is related with the value of MESSAGE_PART_FLAG_ATTACHMENT or if I am missing something (like setting block.hdr = NULL or some more code to handle new flags).

Thank you, Rui Carneiro

On Wed, Apr 15, 2009 at 11:23 PM, Timo Sirainen tss@iki.fi wrote:

On Mon, 2009-04-13 at 11:18 +0100, Rui Carneiro wrote:

...

I didn't understood yet what is the plugin's design and how the plugins are called from the core system and I was wondering if anyone could help me with that.

fts-storage.c hooks into all the functions in mail-storage API that it needs to. Currently indexing isn't done while messages are being saved, but instead just before searching. The searching functions are:

• fts_mailbox_search_init() tries to figure out if FTS can optimize the search. If it does, it tries to figure out if FTS index is up-to-date and if not, starts the search.

• fts_mailbox_search_next_nonblock() continues the indexing (or searching after indexing) for a while. The idea is that IMAP connection is able to process other commands while doing a long-running search. So fts plugin indexes FTS_SEARCH_NONBLOCK_COUNT (50) messages at a time. It would be nice if that value was dynamically calculated and also based on bytes instead of messages, but that's maybe too much trouble.

• fts_mailbox_search_next_update_seq() uses the fts search results and updates mail-storage's search stuff so that it doesn't go through messages that don't match.

• fts_build_mail() indexes a single mail. It parses the messages and returns the data in small blocks. For text/* and message/rfc822 parts those blocks are currently sent to FTS backend. This is where I think you should look into hooking your attachment parsing. Change fts_build_want_index_part() to look for more content-types that you're interested in and then before feeding the blocks to FTS backend put them through your own converter function, something like:

int attachment_extract_text(struct attachment_extract_context *ctx, const struct message_block *input, struct message_block *output);

-- mobile: +351 963446125 mail: rui.arc@gmail.com mail: ei04073@fe.up.pt website: http://paginas.fe.up.pt/~ei04073http://paginas.fe.up.pt/%7Eei04073

Hi again,

Anyone know some good libraries to handle the content of files like pdf, ppt, doc, etc? I am already indexing attachments all I need now is extract the text of them.

Regards, Rui Carneiro

On Mon, Apr 20, 2009 at 3:29 PM, Rui Carneiro rui.arc@gmail.com wrote:

Hi,

The problem was on the flag. My hexa to binary conversions was wrong.

Regards, Rui Carneiro

On Fri, Apr 17, 2009 at 10:03 AM, Rui Carneiro rui.arc@gmail.com wrote:

...

Thank you for all tips. The design look more clear to me now.

I have one more question. I looked into fts_build_want_index_part() and I saw that I need to add some flags to message_part_flags, what values should I choose? My first approach was to follow your schema and set MESSAGE_PART_FLAG_ATTACHMENT = 0x16. There is any problem with this?

I already had changed parse_content_type() to set ctx->part->flags correctly but if i choose my custom flag dovecot assume that all attachment lines are headers. I already tried to set those ctx->part->flags as TEXT and the fts_backend was feeded correctly with all attachment lines.

I don't know if this is related with the value of MESSAGE_PART_FLAG_ATTACHMENT or if I am missing something (like setting block.hdr = NULL or some more code to handle new flags).

Thank you, Rui Carneiro

On Wed, Apr 15, 2009 at 11:23 PM, Timo Sirainen tss@iki.fi wrote:

...

On Mon, 2009-04-13 at 11:18 +0100, Rui Carneiro wrote:

...

I didn't understood yet what is the plugin's design and how the plugins are called from the core system and I was wondering if anyone could help me with that.

fts-storage.c hooks into all the functions in mail-storage API that it needs to. Currently indexing isn't done while messages are being saved, but instead just before searching. The searching functions are:

• fts_mailbox_search_init() tries to figure out if FTS can optimize the search. If it does, it tries to figure out if FTS index is up-to-date and if not, starts the search.

• fts_mailbox_search_next_nonblock() continues the indexing (or searching after indexing) for a while. The idea is that IMAP connection is able to process other commands while doing a long-running search. So fts plugin indexes FTS_SEARCH_NONBLOCK_COUNT (50) messages at a time. It would be nice if that value was dynamically calculated and also based on bytes instead of messages, but that's maybe too much trouble.

• fts_mailbox_search_next_update_seq() uses the fts search results and updates mail-storage's search stuff so that it doesn't go through messages that don't match.

• fts_build_mail() indexes a single mail. It parses the messages and returns the data in small blocks. For text/* and message/rfc822 parts those blocks are currently sent to FTS backend. This is where I think you should look into hooking your attachment parsing. Change fts_build_want_index_part() to look for more content-types that you're interested in and then before feeding the blocks to FTS backend put them through your own converter function, something like:

int attachment_extract_text(struct attachment_extract_context *ctx, const struct message_block *input, struct message_block *output);

-- mobile: +351 963446125 mail: rui.arc@gmail.com mail: ei04073@fe.up.pt website: http://paginas.fe.up.pt/~ei04073http://paginas.fe.up.pt/%7Eei04073

-- mobile: +351 963446125 mail: rui.arc@gmail.com mail: ei04073@fe.up.pt website: http://paginas.fe.up.pt/~ei04073http://paginas.fe.up.pt/%7Eei04073

-- mobile: +351 963446125 mail: rui.arc@gmail.com mail: ei04073@fe.up.pt website: http://paginas.fe.up.pt/~ei04073

Great idea!

I will give news soon.

On Tue, Apr 21, 2009 at 5:32 PM, Timo Sirainen tss@iki.fi wrote:

I've no idea, but you could at least look at some of the other full text search engines. I remember them advertising indexing support for all kinds of formats. Maybe they're using some specific library or maybe it would be easy to extract their parsing code.

On Wed, 2009-04-22 at 15:51 +0100, Rui Carneiro wrote:

Hi,

Almost full text search engines (C/C++) I looked (Swish-E, Wumpus, Lemur and Xapian) do not use any kind of library or parser. Instead, they use other applications like pdftotext, catdoc, catppt (etc) and call them with execvp (or equivalent). Using this approach on my project have some pros and cons:

Pros:

• The existing libraries to extract the content of pdf, doc (etc) are not very stable.
• Easier to handle errors (even if those applications crash dovecot will be still running)

Hmm. I hadn't thought of this before. Yeah, if they're not stable it's probably not a good idea to run in the same process as the rest of Dovecot. But I guess there could be some kind of a separate text extracting process that fts plugin would talk to. If that process dies it could get restarted automatically and fts could maybe retry and if it it dies again log it and just skip over it.

• Some programs to parse special formats (p.e. catppt and pdftotext) do not accept input from stdin (we need to create temporary files).

Maybe those programs could be changed and just require the newer versions?..

What approach would be better? Using applications like pdftotext and catdoc or, on the other hand, use their libraries and do it almost from scratch?

I think the API that fts plugin uses to do the conversion should be generic enough that both approaches would work. Then it would be easier to implement one or another or both eventually.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Wed, 22 Apr 2009, Rui Carneiro wrote:

I will talk with the developers of those applications about the possibility of supporting stdin input (if not supported yet).

I think the API that fts plugin uses to do the conversion should be

...

generic enough that both approaches would work. Then it would be easier to implement one or another or both eventually.

I think I will try the external applications approach. My developing time available is not to much.

Actually, if I consider what the xls-to-HTML converter did lately to our webmail frontend, I suggest to index "alien" formats asynchroneously, maybe in low-priority process, not only to prevent potential long conversation time and resource requirement, but also to prevent MUAs re-initate the search and force the IMAP server to index the same file simultaneously.

Bye,

Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)

iQEVAwUBSfBGBXWSIuGy1ktrAQKrRwgAll5KRqG0tMwPYgt21cKR5F4r8mrnA9nJ 5zvdQgFGXJoT4NegpzJ15+V8l7a28Uaxx79hzrubRpJSTNI5gU08TkzdNkJwWLTu IA8gK/ZwQnnMqpQByF/pf7ERzMroZv3ZpYpkbEbI64MMSYOrI2hT92t3KSSnJ39f TUSdRN9sUhdA69uWlKCFMofhAEfaoP+U8N3pg1b/kc14+HzmTqrx/SWNHZkzU5qm clUmfa/uGMuv+gq+bKSEtos79Q1QOTqH9qRSRbNsxOVISM75C7dTpqIlcqz53iIg RsRHDxCtyIv/UJrfE9fniOYE6l/xs8iLgG69fOGUCzwmLjVx2j9dKA== =7O9D -----END PGP SIGNATURE-----

Hi again,

On Wed, Apr 15, 2009 at 11:23 PM, Timo Sirainen tss@iki.fi wrote:

• fts_build_mail() indexes a single mail. It parses the messages and returns the data in small blocks. For text/* and message/rfc822 parts those blocks are currently sent to FTS backend. This is where I think you should look into hooking your attachment parsing. Change fts_build_want_index_part() to look for more content-types that you're interested in and then before feeding the blocks to FTS backend put them through your own converter function, something like:

int attachment_extract_text(struct attachment_extract_context *ctx, const struct message_block *input, struct message_block *output);

Let's take the example of an application-pdf content-type. Before I converter all pdf data to text I need to gather all data before. The actual process is feeding FTS backend with small parts of data and appending them on "build_more" functions (e.g. fts_backend_solr_build_more()).

So where should I call attachment_extract_text()? In fts_backend_solr_build_more() and not making append to cmd until data is extracted? Or gather all information before (e.g. fts_build_mail()) and send all in once to FTS backend?

I hope I've made myself clear.

Regards, Rui Carneiro

Portugalmail, Comunicações S.A. www.portugalmail.net

Citando Timo Sirainen tss@iki.fi:

1. You notice a non-text/* content-type and initialize text extraction for the MIME part. Like:

struct attachment_extract_context * attachment_extract_init(const char *content_type);

2. After this you feed all the input belonging to that MIME part to:

int attachment_extract_add(struct attachment_extract_context *ctx, const struct message_block *input);

Don't output anything to FTS backend at this point. The attachment_extract_add() would probably just basically write to a temporary file.

3. Finally you'll notice that the MIME part ends (either you get headers for the next MIME part or the entire message ends). Then finish the extraction, which actually executes the whatever conversion binaries:

int attachment_extract_finish(struct attachment_extract_context *ctx);

4. Get the resulting text to fts_backend_build_more() somehow. Either some attachment_extract_add_to_fts() which internally adds it or some kind of an iterator that returns the text in smaller blocks. Either would work..

That kind of an API would also make it possible to pretty easily modify in future to not write temporary files for specific content types if it's not required.

I tried your approach and I think it is working pretty well. Now I only need to look carefully to the output of external programs and build the XML correctly to send to Solr.

Thanks Timo

Regards, Rui Carneiro

-- Portugalmail, Comunicações S.A. www.portugalmail.net

On May 18, 2009, at 6:42 AM, Rui Carneiro wrote:

I am having some troubles sending all data to a file. When I finish
to send all data to a file, I tried to open it and the file is
corrupted.

The first think I noticed is that all chars are capitalized what
destroy all the file format.

Where are the chars capitalized?

Hmm. I'll see about getting it fixed in a better way, but for now you
could just change:

decoder = message_decoder_init(TRUE);

to

decoder = message_decoder_init(FALSE);

I'm thinking about making message_decoder uppercase only text/* body
parts.

Any other idea why files are getting corrupted?

Nope. If you still see corruption, try with some simple test mails and
see if it's adding garbage, losing contents or adding more content.

On Mon, 2009-05-18 at 17:35 +0100, Rui Carneiro wrote:

I think binary data is being corrupted anywhere before fts_backend_build_more() and I don't have any idea where.

All the data comes from lib-mail/message-decoder.c. Hmm. Looks like it tries to force giving only valid UTF-8 output. I guess it should have some flag or something that makes it do that only for text/* parts, not for binary parts. OK, implemented, see if it works with this and using the flag:

http://hg.dovecot.org/dovecot-1.2/rev/44548a7fb10d

On Tue, 2009-05-19 at 14:40 +0100, Rui Carneiro wrote:

...

http://hg.dovecot.org/dovecot-1.2/rev/44548a7fb10d

It is working now but I needed to do some changes on your code.

OK.

Please see the attachment to checked any problem that may exist.

You forgot the attachment.

Now, with attachment.

On Fri, 2009-05-22 at 18:24 +0100, Rui Carneiro wrote:

Hi Timo,

I almost finish the changes on fts plugin. By now, it seems to work fine with attachments (extracting and sending them to Solr). I only have a problem with the max size of the command (cmd) that we can send to Solr:

#define SOLR_CMDBUF_SIZE (1024*64)

By now, if we send some message bigger than this value the fts-plugin crash.

The problem is something else. The Solr code simply tries to keep the send buffer smaller than that, nothing would break if you sent a larger buffer. Show gdb backtrace of the crash?

On Fri, 2009-05-22 at 18:57 +0100, Rui Carneiro wrote:

Citando Timo Sirainen tss@iki.fi:

...

The problem is something else. The Solr code simply tries to keep the send buffer smaller than that, nothing would break if you sent a larger buffer. Show gdb backtrace of the crash?

I said it was from the buff size because when I increased it Dovecot didn't crash.

I guess it works around some other bug then. If it's a memory-related bug you could also see if valgrind complains something:

protocol imap { .. mail_executable = /usr/bin/valgrind /usr/local/libexec/dovecot/imap }

On Mon, 2009-05-25 at 14:20 +0100, Rui Carneiro wrote:

Citando Timo Sirainen tss@iki.fi:

...

I guess it works around some other bug then. If it's a memory-related bug you could also see if valgrind complains something:

protocol imap { .. mail_executable = /usr/bin/valgrind /usr/local/libexec/dovecot/imap }

Here is the output (I cloned the http://hg.dovecot.org/dovecot-1.2 and made no changes to this test):

ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 123 from 2)

So valgrind didn't find anything wrong. What does gdb show as the backtrace?

On May 26, 2009, at 5:46 AM, Rui Carneiro wrote:

Citando Timo Sirainen tss@iki.fi:

...

So valgrind didn't find anything wrong.

We should ignore LEAK SUMMARY?

At least for now. Memory leaks don't cause crashes.

...

What does gdb show as the backtrace?

My gdb is not writing where he should (or not writing at all). This
shouldn't be enough?

mail_executable = /usr/local/libexec/dovecot/gdbhelper /usr/local/ libexec/dovecot/imap

It's not writing /tmp/gdbhelper* files when crashing? Anyway there's
also one guaranteed way to get backtrace. Remove the gdbhelper and
then run:

gdb -p pidof imap cont <make it crash> bt full