[Dovecot] Shared Squat index for public mailboxes
Hi,
I'm running Dovecot 1.2.17 for serving mail via IMAP as well as for providing access to a mailing list archive. The archive is implemented as a public read-only mailbox with per-user index files (i.e. the \Seen flags are per-user).
I recently enbled the Squat plugin to accelerate searches in the message bodies and noticed that every user (I'm using a virtual user setup) gets his own dovecot.index.search and dovecot.index.search.uids copies.
Is it possible to share those files among all users of the system? The squat plugin appears to store the search indices among the other index files (as explained on http://wiki.dovecot.org/Plugins/FTS/Squat) no matter what; I considered storing a central copy of the index files somewhere and then creating symlinks for all users. It should be ok as far as file-permissions go since all mail is owned by a single vmail system user, but I wonder whether the indices are really the same (I noticed their md5 checksums differ) and whether there may be file locking issues in case two users search message bodies simultaneously.
Can anybody shed some light?
-- Frerich Raabe - raabe@froglogic.com www.froglogic.com - Multi-Platform GUI Testing
Am 10.10.2012 09:29, schrieb Frerich Raabe:
Hi,
I'm running Dovecot 1.2.17 for serving mail via IMAP as well as for providing access to a mailing list archive. The archive is implemented as a public read-only mailbox with per-user index files (i.e. the \Seen flags are per-user).
i guess better upgrade to 2.1.x first
I recently enbled the Squat plugin to accelerate searches in the message bodies and noticed that every user (I'm using a virtual user setup) gets his own dovecot.index.search and dovecot.index.search.uids copies.
Is it possible to share those files among all users of the system? The squat plugin appears to store the search indices among the other index files (as explained on http://wiki.dovecot.org/Plugins/FTS/Squat) no matter what; I considered storing a central copy of the index files somewhere and then creating symlinks for all users. It should be ok as far as file-permissions go since all mail is owned by a single vmail system user, but I wonder whether the indices are really the same (I noticed their md5 checksums differ) and whether there may be file locking issues in case two users search message bodies simultaneously.
Can anybody shed some light?
after upgrade http://wiki2.dovecot.org/Plugins/FTS/Lucene may be better choice
this info might help
http://wiki2.dovecot.org/MailLocation
---snip Index files
Index files are by default stored under the same directory as mails. With maildir they are stored in the actual maildirs, with mbox they are stored under .imap/ directory. You may want to change the index file location if you're using NFS or if you're setting up shared mailboxes.
You can change the index file location by adding :INDEX=<path> to mail_location. For example:
mail_location = maildir:~/Maildir:INDEX=/var/indexes/%u --snip
after upgrade come back, ask again, or meanwhile Timo gives better advice
Best Regards MfG Robert Schetterer
Am 10.10.2012 09:49, schrieb Robert Schetterer:
Am 10.10.2012 09:29, schrieb Frerich Raabe:
I'm running Dovecot 1.2.17 for serving mail via IMAP as well as for providing access to a mailing list archive. The archive is implemented as a public read-only mailbox with per-user index files (i.e. the \Seen flags are per-user).
i guess better upgrade to 2.1.x first
Given that Dovecot 1.2.17 works fine for me, I actually didn't see the need to upgrade yet.
I recently enbled the Squat plugin to accelerate searches in the message bodies and noticed that every user (I'm using a virtual user setup) gets his own dovecot.index.search and dovecot.index.search.uids copies.
Is it possible to share those files among all users of the system? The squat plugin appears to store the search indices among the other index files (as explained on http://wiki.dovecot.org/Plugins/FTS/Squat) no matter what; I considered storing a central copy of the index files somewhere and then creating symlinks for all users. It should be ok as far as file-permissions go since all mail is owned by a single vmail system user, but I wonder whether the indices are really the same (I noticed their md5 checksums differ) and whether there may be file locking issues in case two users search message bodies simultaneously.
Can anybody shed some light?
after upgrade http://wiki2.dovecot.org/Plugins/FTS/Lucene may be better choice
Why?
this info might help
http://wiki2.dovecot.org/MailLocation
---snip Index files
Index files are by default stored under the same directory as mails. With maildir they are stored in the actual maildirs, with mbox they are stored under .imap/ directory. You may want to change the index file location if you're using NFS or if you're setting up shared mailboxes.
You can change the index file location by adding :INDEX=<path> to mail_location. For example:
mail_location = maildir:~/Maildir:INDEX=/var/indexes/%u --snip
I already use this; as I mentioned, the index files of the public readonly mailbox is stored per-user so that each user has his own set of \Seen flags. Here's my public namespace:
namespace public { separator = / prefix = Lists/ location = maildir:/home/vmail/lists/Maildir:CONTROL=~/Maildir/lists:INDEX=~/Maildir/lists subscriptions = no }
Alas, this means that *all* index files (including the Squat index) is stored per-user whereas I'd just to have just *some* of them per-user. :-)
after upgrade come back, ask again, or meanwhile Timo gives better advice
Does this imply that questions regarding Dovecot 1.2.17 are considered offtopic on this list? If so, I apologize - I'll look for another forum then.
-- Frerich Raabe - raabe@froglogic.com www.froglogic.com - Multi-Platform GUI Testing
Am 10.10.2012 10:06, schrieb Frerich Raabe:
I already use this; as I mentioned, the index files of the public readonly mailbox is stored per-user so that each user has his own set of \Seen flags. Here's my public namespace:
namespace public { separator = / prefix = Lists/ location = maildir:/home/vmail/lists/Maildir:CONTROL=~/Maildir/lists:INDEX=~/Maildir/lists
subscriptions = no }
Alas, this means that *all* index files (including the Squat index) is stored per-user whereas I'd just to have just *some* of them per-user. :-)
after upgrade come back, ask again, or meanwhile Timo gives better advice
Does this imply that questions regarding Dovecot 1.2.17 are considered offtopic on this list? If so, I apologize - I'll look for another forum then.
no wait till others will reply, indexing questions might be more complex to answer, then i know recent in short time
perhaps meanwhile this helps
-snip
http://wiki2.dovecot.org/SharedMailboxes/Public
With Maildir a dovecot-shared file controls if the \Seen flags are shared or private. The file must be created separately inside each Maildir, although if the file already exists in the Maildir root it's automatically copied for newly created mailboxes. If dovecot-shared file doesn't exist in Maildir, the \Seen flags are shared. If it exists, the \Seen flag state is stored only in the user's index files. By making each user have their own private index files, you can make the \Seen flag private for the users. For example:
namespace { type = public separator = / prefix = Public/ location = maildir:/var/mail/public:INDEX=~/Maildir/public subscriptions = no }
Now when accessing e.g. "Public/lkml" mailbox, Dovecot keeps its index files in ~/Maildir/public/lkml/ directory. If it ever gets deleted, the \Seen flags are lost.
If you want to change what flags are shared when dovecot-shared file exists, currently you'll have to modify the source code: src/lib-storage/index/maildir/maildir-storage.c maildir_open() has mbox->ibox.box.private_flags_mask = MAIL_SEEN; Change the MAIL_SEEN to any flag combination you want. See src/lib-mail/mail-types.h for list of valid flags.
--snip
however if there is a need for some new/debugged/better features relate to what you asked you have to upgrade to latest dovecot
Best Regards MfG Robert Schetterer
Am 10.10.2012 10:24, schrieb Robert Schetterer:
Am 10.10.2012 10:06, schrieb Frerich Raabe:
I already use this; as I mentioned, the index files of the public readonly mailbox is stored per-user so that each user has his own set of \Seen flags. Here's my public namespace:
namespace public { separator = / prefix = Lists/ location = maildir:/home/vmail/lists/Maildir:CONTROL=~/Maildir/lists:INDEX=~/Maildir/lists subscriptions = no }
[..]
perhaps meanwhile this helps
-snip
[..]
namespace { type = public separator = / prefix = Public/ location = maildir:/var/mail/public:INDEX=~/Maildir/public subscriptions = no }
Note how this is basically exactly the same as what I posted, except that it uses the Dovecot 2 configuration file format ('type = public') and that it calls the prefix/location "public" instead of "lists".
If you want to change what flags are shared when dovecot-shared file exists, currently you'll have to modify the source code: src/lib-storage/index/maildir/maildir-storage.c maildir_open() has mbox->ibox.box.private_flags_mask = MAIL_SEEN; Change the MAIL_SEEN to any flag combination you want. See src/lib-mail/mail-types.h for list of valid flags.
I don't think this is applicable to my case, and a check of the source code seems to confirm that: I'm not trying to change the set of flags stored for a given mail but rather the index file of the Squat plugin.
-- Frerich Raabe - raabe@froglogic.com www.froglogic.com - Multi-Platform GUI Testing
On 10.10.2012, at 11.06, Frerich Raabe wrote:
I already use this; as I mentioned, the index files of the public readonly mailbox is stored per-user so that each user has his own set of \Seen flags. Here's my public namespace:
namespace public { separator = / prefix = Lists/ location = maildir:/home/vmail/lists/Maildir:CONTROL=~/Maildir/lists:INDEX=~/Maildir/lists subscriptions = no }
Alas, this means that *all* index files (including the Squat index) is stored per-user whereas I'd just to have just *some* of them per-user. :-)
You'll need v2.2 and its INDEXPVT setting.
Am 11.10.2012 um 22:10 schrieb Timo Sirainen:
On 10.10.2012, at 11.06, Frerich Raabe wrote:
I already use this; as I mentioned, the index files of the public readonly mailbox is stored per-user so that each user has his own set of \Seen flags. Here's my public namespace:
namespace public { separator = / prefix = Lists/ location = maildir:/home/vmail/lists/Maildir:CONTROL=~/Maildir/lists:INDEX=~/Maildir/lists subscriptions = no }
Alas, this means that *all* index files (including the Squat index) is stored per-user whereas I'd just to have just *some* of them per-user. :-)
You'll need v2.2 and its INDEXPVT setting.
Hm, you mean the feature introduced by http://hg.dovecot.org/dovecot-2.2/rev/dbd42f7198eb ? Is there some discussion of the feature somewhere?
The commit log is a bit unclear to me, it says 'Per-user flags can now be stored in private index files.' however http://wiki2.dovecot.org/SharedMailboxes/Public says 'By making each user have their own private index files, you can make the \Seen flag private for the users.' (using the INDEX setting).
Makes me wonder - the Wiki talks about 'private index files' when talking about 'INDEX' and the commit says 'private index files' talking about INDEXPVT - what is the difference? :-)
-- Frerich Raabe - raabe@froglogic.com www.froglogic.com - Multi-Platform GUI Testing
On 11.10.2012, at 23.38, Frerich Raabe wrote:
Am 11.10.2012 um 22:10 schrieb Timo Sirainen:
On 10.10.2012, at 11.06, Frerich Raabe wrote:
I already use this; as I mentioned, the index files of the public readonly mailbox is stored per-user so that each user has his own set of \Seen flags. Here's my public namespace:
namespace public { separator = / prefix = Lists/ location = maildir:/home/vmail/lists/Maildir:CONTROL=~/Maildir/lists:INDEX=~/Maildir/lists subscriptions = no }
Alas, this means that *all* index files (including the Squat index) is stored per-user whereas I'd just to have just *some* of them per-user. :-)
You'll need v2.2 and its INDEXPVT setting.
Hm, you mean the feature introduced by http://hg.dovecot.org/dovecot-2.2/rev/dbd42f7198eb ?
Yes.
Is there some discussion of the feature somewhere?
http://markmail.org/message/45jxf363ffrubonv has some.
The commit log is a bit unclear to me, it says 'Per-user flags can now be stored in private index files.' however http://wiki2.dovecot.org/SharedMailboxes/Public says 'By making each user have their own private index files, you can make the \Seen flag private for the users.' (using the INDEX setting).
Makes me wonder - the Wiki talks about 'private index files' when talking about 'INDEX' and the commit says 'private index files' talking about INDEXPVT - what is the difference? :-)
You can have both! Shared indexes having the shared stuff (including squat indexes), while the private indexes only have the per-user flags, nothing else. For example with sdbox/mdbox you couldn't even have set per-user INDEX location or it would have just broken.
participants (3)
-
Frerich Raabe
-
Robert Schetterer
-
Timo Sirainen