[Dovecot] FTS/squat search indexes built when?
I know that a given search index doesn't get built the first time until you do an actual search on it from an IMAP client. I don't much care about that; I'm the only user, and I don't create new folders too often. At worst, I could create a script to go search every folder once.
But what I'm not clear about: Does it then get updated on every delivery by LDA? Or does the index also need to get updated when I do a search, assuming there's been at least one message delivered since the last search?
Jay Levitt
On Sat, 2008-06-14 at 07:56 -0400, Jay Levitt wrote:
But what I'm not clear about: Does it then get updated on every delivery by LDA? Or does the index also need to get updated when I do a search, assuming there's been at least one message delivered since the last search?
Currently it gets updated only when searching. I wanted to have it updated immediately (optionally) while mails are being delivered, but never got around to implementing it.
Timo Sirainen wrote:
On Sat, 2008-06-14 at 07:56 -0400, Jay Levitt wrote:
But what I'm not clear about: Does it then get updated on every delivery by LDA? Or does the index also need to get updated when I do a search, assuming there's been at least one message delivered since the last search?
Currently it gets updated only when searching. I wanted to have it updated immediately (optionally) while mails are being delivered, but never got around to implementing it.
OK, some followups then:
Is the index rebuilt from scratch every time, or should updates be fairly quick? I just did a search on my inbox (about 10,000 messages), and I know I've done a search there in the past month - yet it either took many, many minutes, or it timed out.
If I wrote a script to log in and search for something, and ran it every night through cron, would that achieve daily indexing?
Jay
On Mon, 2008-07-14 at 08:07 -0400, Jay Levitt wrote:
Timo Sirainen wrote:
On Sat, 2008-06-14 at 07:56 -0400, Jay Levitt wrote:
But what I'm not clear about: Does it then get updated on every delivery by LDA? Or does the index also need to get updated when I do a search, assuming there's been at least one message delivered since the last search?
Currently it gets updated only when searching. I wanted to have it updated immediately (optionally) while mails are being delivered, but never got around to implementing it.
OK, some followups then:
- Is the index rebuilt from scratch every time, or should updates be fairly quick?
It's updated, not rebuilt.
I just did a search on my inbox (about 10,000 messages), and I know I've done a search there in the past month - yet it either took many, many minutes, or it timed out.
Indexing 10k messages from scratch shouldn't take all that many seconds with somewhat recent computers..
- If I wrote a script to log in and search for something, and ran it every night through cron, would that achieve daily indexing?
Yes.
Hi Jay,
On Monday 14 July 2008, Jay Levitt wrote:
- If I wrote a script to log in and search for something, and ran it every night through cron, would that achieve daily indexing?
I would be interested in that script, too - are you working on it?
Thanks, Patrick.
-- STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key: https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005
Patrick Nagel wrote:
Hi Jay,
On Monday 14 July 2008, Jay Levitt wrote:
- If I wrote a script to log in and search for something, and ran it every night through cron, would that achieve daily indexing?
I would be interested in that script, too - are you working on it?
It appears that you can run IMAP commands by piping them into the imap process - look at some of Timo's previous posts for examples. I should think it's a one line to type in the IMAP search command and have it execute? (implementation left as an exercise for the reader...)
To do it remotely, perl/ruby and most common scripting languages have an IMAP module which should work well (need to embed a password obviously)
Ed W
Hi
On Fri, 25 Jul 2008 12:00:05 +0100, Ed W lists@wildgooses.com wrote:
I would be interested in that script, too - are you working on it?
It appears that you can run IMAP commands by piping them into the imap process - look at some of Timo's previous posts for examples.
I'm aware of 'dovecot --exec-mail imap'.
I should think it's a one line to type in the IMAP search command and have it execute?
But I don't know how to wait for / react to answers from the IMAP server (after listing mailboxes, selecting the mailbox, etc.) in shell (bash) script, which would be my preferred choice as scripting language. Is it possible? How?
(implementation left as an exercise for the reader...)
Obviously I wrote this mail because I wanted to save some time and effort - something wrong with that? ;)
Did really nobody do this "exercise" before?
To do it remotely, perl/ruby and most common scripting languages have an IMAP module which should work well (need to embed a password obviously)
Yes, I'm sure it would be easy with those - after reading all the module's documentation, understanding all the caveats, implementing all the error handling and so on.
So, again, if somebody came up with a solution already, please share, I'd appreciate it.
Patrick.
P.S.: RoundCube Webmail is not a good (as in easy to use) client to read and reply to mailing lists (yet).
-- STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key: https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005
On Jul 26, 2008, at 9:45 AM, Patrick Nagel wrote:
I should think it's a one line to type in the IMAP search command and have it execute?
But I don't know how to wait for / react to answers from the IMAP
server (after listing mailboxes, selecting the mailbox, etc.) in shell (bash) script, which would be my preferred choice as scripting language. Is
it possible? How?
I don't think you need to do any of that. Just:
printf "1 select $mailbox\n2 search text x93hgdgd\n3 logout\n" |
dovecot --exec-mail imap
For getting the list of mailboxes:
mailboxes=printf "1 list "" *\n" | dovecot --exec-mail imap | perl magic
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
Timo Sirainen wrote:
| printf "1 select $mailbox\n2 search text x93hgdgd\n3 logout\n" |
| dovecot --exec-mail imap
|
| For getting the list of mailboxes:
|
| mailboxes=printf "1 list "" *\n" | dovecot --exec-mail imap | perl magic
Ok, looks easy. But I can't find information anywhere on how to specify the user. I tried with 'USER=username' in front of the dovecot call, and dovecot then said 'Logged in as username' instead of 'Logged in as root', but a list command doesn't show the mailboxes, so I think it's not accessing the maildir of that user. I guess it's because the users are not system users, and dovecot needs to retrieve info from the LDAP directory first. But how do I tell it to do that?
Logging in via netcat / telnet works, but then I don't know how to proceed...
Thanks for your help! Patrick.
STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key: https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFIkqF+7yMg/OiDoAURAii6AKCOpl9mSq8eWdwFEj4/LUOf4I3yEwCfcydS IOBDO6Q05JIARK9Cg/bbSlo= =kk4r -----END PGP SIGNATURE-----
Patrick Nagel wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
Timo Sirainen wrote: | printf "1 select $mailbox\n2 search text x93hgdgd\n3 logout\n" | | dovecot --exec-mail imap | | For getting the list of mailboxes: | | mailboxes=
printf "1 list "" *\n" | dovecot --exec-mail imap | perl magicOk, looks easy. But I can't find information anywhere on how to specify the user. I tried with 'USER=username' in front of the dovecot call, and dovecot then said 'Logged in as username' instead of 'Logged in as root', but a list command doesn't show the mailboxes, so I think it's not accessing the maildir of that user. I guess it's because the users are not system users, and dovecot needs to retrieve info from the LDAP directory first. But how do I tell it to do that?
Logging in via netcat / telnet works, but then I don't know how to proceed...
Thanks for your help! Patrick.
exporting the $USER variable seems to work - i.e. 'export USER=joetest'. The index is updated and you get all the lovely information for reporting.
This is great. Unfortunately when I run it, existing messages for mutt are marked as O (or old). Thunderbird works just fine. I'm betting to believe these clients hate me. Please let me know of a way to get around this. Thanks.
---Jack
On Aug 1, 2008, at 1:39 AM, Patrick Nagel wrote:
| mailboxes=
printf "1 list "" *\n" | dovecot --exec-mail imap | perl magicOk, looks easy. But I can't find information anywhere on how to
specify the user. I tried with 'USER=username' in front of the dovecot call, and
dovecot then said 'Logged in as username' instead of 'Logged in as root',
but a list command doesn't show the mailboxes, so I think it's not accessing
the maildir of that user. I guess it's because the users are not system users, and dovecot
needs to retrieve info from the LDAP directory first. But how do I tell it to
do that?
You can't do that with --exec-mail. It expects to be running using the
correct UID, GID, $USER and $HOME.
Logging in via netcat / telnet works, but then I don't know how to
proceed...
With v1.2 code you could use the same printf .. | nc localhost 143,
but unfortunately this doesn't work with older versions since the
input buffer is reset after login.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Timo Sirainen wrote:
| On Aug 1, 2008, at 1:39 AM, Patrick Nagel wrote:
|
|> | mailboxes=printf "1 list "" *\n" | dovecot --exec-mail imap | perl |> magic
|>
|> Ok, looks easy. But I can't find information anywhere on how to
|> specify the
|> user. I tried with 'USER=username' in front of the dovecot call, and
|> dovecot
|> then said 'Logged in as username' instead of 'Logged in as root', but
|> a list
|> command doesn't show the mailboxes, so I think it's not accessing the
|> maildir
|> of that user.
|> I guess it's because the users are not system users, and dovecot needs to
|> retrieve info from the LDAP directory first. But how do I tell it to
|> do that?
|
| You can't do that with --exec-mail. It expects to be running using the
| correct UID, GID, $USER and $HOME.
|
|> Logging in via netcat / telnet works, but then I don't know how to
|> proceed...
|
| With v1.2 code you could use the same printf .. | nc localhost 143, but
| unfortunately this doesn't work with older versions since the input
| buffer is reset after login.
|
I see. So now we're back to "I don't know how to react to dovecot's responses" ;)
To sum up: It's currently (with version 1.1) not possible to script IMAP operations with shell script when using virtual users. With perl or some other scripting language, using an IMAP module, it should be no problem though.
Version 1.2 will make it easier for scripts to interface via IMAP, because you can pipe a batch of commands into dovecot, and it won't just ignore everything after the login command.
Thanks for the clarification!
Patrick.
STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key: https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFIl8Er7yMg/OiDoAURAt4aAKCmYomljxapIJ1H/jS/YoOt0ygDiQCeK+kE 4r3nLKZ/cELZZX0eIg1xbEc= =g2dM -----END PGP SIGNATURE-----
participants (5)
-
Ed W
-
Jack Stewart
-
Jay Levitt
-
Patrick Nagel
-
Timo Sirainen