[Dovecot] Virtual mailboxes not always up to date?
Hi,
I have been experimenting with the Virtual mailboxes plugin 1 recently.
This is my setup: the following dovecot-virtual file exists in a virtual mailbox called "todo":
------ dovecot-virtual ------ * -Trash -Trash/* -Spam -Spam/* OR KEYWORD $TODO KEYWORD todo
so if any mail in any mailbox except Trash+Spam has the keyword "$TODO" (that's what KMail sets when you mark a message as "action item") or "todo" (that's what Thunderbird sets when you put the Todo tag on a message), it should be shown in that 'todo' virtual mailbox.
This works for the first time after I edit the dovecot-virtual file and then access the virtual mailbox (all mails I expect to be shown in the todo mailbox are there) - but further accesses always show the same content, even though I removed the keyword from one of the mails (and I can confirm that the keyword has actually been removed by dovecot, since the filename doesn't contain the letter corresponding to the keyword anymore).
It seems that changing the dovecot-virtual file triggers the execution of the "search program" - but shouldn't simply accessing the virtual mailbox also trigger it? Or is there a (undocumented?) timeout between search program executions?
Patrick.
# dovecot -n # 1.2.6: /etc/dovecot/dovecot.conf # OS: Linux 2.6.22-vs2.2.0.7-gentoo x86_64 Gentoo Base System release 2.0.1 protocols: imaps managesieve listen: x.x.x.x ssl: required ssl_ca_file: /etc/ssl/dovecot/sub.class1.server.ca.crt ssl_cert_file: /etc/ssl/dovecot/ssl.crt ssl_key_file: /etc/ssl/dovecot/ssl.key login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(managesieve): /usr/libexec/dovecot/managesieve-login mail_max_userip_connections(default): 20 mail_max_userip_connections(imap): 20 mail_max_userip_connections(managesieve): 10 mail_location: maildir:~/.maildir mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(managesieve): /usr/libexec/dovecot/managesieve mail_process_size: 2048 mail_plugins(default): virtual fts fts_squat mail_plugins(imap): virtual fts fts_squat mail_plugins(managesieve): mail_plugin_dir(default): /usr/lib64/dovecot/imap mail_plugin_dir(imap): /usr/lib64/dovecot/imap mail_plugin_dir(managesieve): /usr/lib64/dovecot/managesieve imap_client_workarounds(default): tb-extra-mailbox-sep imap_client_workarounds(imap): tb-extra-mailbox-sep imap_client_workarounds(managesieve): namespace: type: private separator: . inbox: yes list: yes subscriptions: yes namespace: type: private separator: . prefix: virtual. location: virtual:~/.maildir/virtual list: yes subscriptions: yes lda: postmaster_address: postmaster@x mail_plugins: sieve virtual auth default: passdb: driver: pam args: * userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix plugin: fts: squat fts_squat: partial=4 full=10
-- STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key: E883A005 https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005
On Mon, 09 Nov 2009 08:03:59 +0800 Patrick Nagel <patrick.nagel@star-group.net> wrote:
Hi,
I have been experimenting with the Virtual mailboxes plugin [1] recently.
This is my setup: the following dovecot-virtual file exists in a virtual mailbox called "todo":
------ dovecot-virtual ------ * -Trash -Trash/* -Spam -Spam/* OR KEYWORD $TODO KEYWORD todo
so if any mail in any mailbox except Trash+Spam has the keyword "$TODO" (that's what KMail sets when you mark a message as "action item") or "todo" (that's what Thunderbird sets when you put the Todo tag on a message), it should be shown in that 'todo' virtual mailbox.
This works for the first time after I edit the dovecot-virtual file and then access the virtual mailbox (all mails I expect to be shown in the todo mailbox are there) - but further accesses always show the same content, even though I removed the keyword from one of the mails (and I can confirm that the keyword has actually been removed by dovecot, since the filename doesn't contain the letter corresponding to the keyword anymore).
It seems that changing the dovecot-virtual file triggers the execution of the "search program" - but shouldn't simply accessing the virtual mailbox also trigger it? Or is there a (undocumented?) timeout between search program executions?
namespace: type: private separator: . prefix: virtual. location: virtual:~/.maildir/virtual list: yes subscriptions: yes
Try to add :INDEX=MEMORY to location setting.
On Mon, 2009-11-09 at 08:03 +0800, Patrick Nagel wrote:
This works for the first time after I edit the dovecot-virtual file and then access the virtual mailbox (all mails I expect to be shown in the todo mailbox are there) - but further accesses always show the same content, even though I removed the keyword from one of the mails (and I can confirm that the keyword has actually been removed by dovecot, since the filename doesn't contain the letter corresponding to the keyword anymore).
What about if you get client to issue EXPUNGE command? New messages should show up in mailbox automatically, but existing ones aren't removed unless EXPUNGE is given or mailbox is re-selected.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Timo,
On 2009-11-10 01:56, Timo Sirainen wrote:
On Mon, 2009-11-09 at 08:03 +0800, Patrick Nagel wrote:
This works for the first time after I edit the dovecot-virtual file and then access the virtual mailbox (all mails I expect to be shown in the todo mailbox are there) - but further accesses always show the same content, even though I removed the keyword from one of the mails (and I can confirm that the keyword has actually been removed by dovecot, since the filename doesn't contain the letter corresponding to the keyword anymore).
What about if you get client to issue EXPUNGE command? New messages should show up in mailbox automatically, but existing ones aren't removed unless EXPUNGE is given or mailbox is re-selected.
Hmm... I did some more testing. Since the Thunderbird 3.0 beta 4 I'm using right now seems to choose '$label4' as IMAP keyword for the 'To Do' tag, I added it to the search command, which is now
OR (OR KEYWORD $TODO KEYWORD todo) KEYWORD $label4
(Off topic: what a mess, why do they change the keywords all the time? I also see KMAILTODO, maybe from another version of KMail?)
Anyway, I did the following:
- Set the '$label4' keyword on two mails in my inbox via Thunderbird
- Checked the virtual.todo mailbox in Thunderbird - both mails were shown
- Unset the '$label4' keyword on one of the two mails in my inbox via Thunderbird
- Closed Thunderbird and opened it again (this should definitely re-select the virtual.todo inbox, right?)
- Checked the virtual.todo mailbox in Thunderbird - still both mails were shown
And then I had the following talk with my dovecot server:
$ /usr/sbin/dovecot --exec-mail imap
- PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH] Logged in as patrick 1 select virtual.todo
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft $TODO KMAILFORWARDED KMAILTODO KMAILWATCHED KMAILIGNORED $FORWARDED $WATCHED $IGNORED $label4)
- OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $TODO KMAILFORWARDED KMAILTODO KMAILWATCHED KMAILIGNORED $FORWARDED $WATCHED $IGNORED $label4 \*)] Flags permitted.
- 2 EXISTS
- 0 RECENT
- OK [UIDVALIDITY 1257723001] UIDs valid
- OK [UIDNEXT 4] Predicted next UID
- OK [HIGHESTMODSEQ 4] Highest 1 OK [READ-WRITE] Select completed. 1 fetch 1:2 flags
- 1 FETCH (FLAGS (\Seen))
- 2 FETCH (FLAGS (\Seen $label4)) 1 OK Fetch completed. 1 expunge 1 OK Expunge completed. 1 fetch 1:2 flags
- 1 FETCH (FLAGS (\Seen))
- 2 FETCH (FLAGS (\Seen $label4)) 1 OK Fetch completed. 1 select virtual.todo
- OK [CLOSED] Previous mailbox closed.
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft $TODO KMAILFORWARDED KMAILTODO KMAILWATCHED KMAILIGNORED $FORWARDED $WATCHED $IGNORED $label4)
- OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $TODO KMAILFORWARDED KMAILTODO KMAILWATCHED KMAILIGNORED $FORWARDED $WATCHED $IGNORED $label4 \*)] Flags permitted.
- 2 EXISTS
- 0 RECENT
- OK [UIDVALIDITY 1257723001] UIDs valid
- OK [UIDNEXT 4] Predicted next UID
- OK [HIGHESTMODSEQ 4] Highest 1 OK [READ-WRITE] Select completed. 1 fetch 1:2 flags
- 1 FETCH (FLAGS (\Seen))
- 2 FETCH (FLAGS (\Seen $label4)) 1 OK Fetch completed.
So it seems that neither EXPUNGE nor re-SELECT seems to have any "refreshing" effect.
Nikita Koshikov told me to "Try to add :INDEX=MEMORY to location setting".
After my tests above I tried that, and suddenly everything works as expected. It seems logical to have the index of a virtual folder in memory only, since it is generated dynamically anyway, and is outdated before it's even written to disk (at least without some configurable "virtual search result max age" parameter). So maybe the easiest solution is, to imply ':INDEX=MEMORY' when a namespace location is a virtual folder?
Patrick.
STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key: E883A005 https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAkr5FGQACgkQ7yMg/OiDoAXY/wCgqJEw6yy3YH3//JBGDDBB+6Pj FCcAnREwazHZAEqqFyc7iGXIQiCz+mnP =u6rT -----END PGP SIGNATURE-----
On Nov 10, 2009, at 2:21 AM, Patrick Nagel wrote:
So it seems that neither EXPUNGE nor re-SELECT seems to have any
"refreshing" effect.
I'll see tomorrow if I can reproduce it.
Nikita Koshikov told me to "Try to add :INDEX=MEMORY to location
setting".After my tests above I tried that, and suddenly everything works as
expected. It seems logical to have the index of a virtual folder in memory
only, since it is generated dynamically anyway, and is outdated before it's even
written to disk (at least without some configurable "virtual search result max
age" parameter). So maybe the easiest solution is, to imply
':INDEX=MEMORY' when a namespace location is a virtual folder?
Well, that also makes the performance somewhat bad. And UIDVALIDITY
changes every time, so client's local caching won't work. And if
client uses multiple connections to the same mailbox, it might get
confused since each connection gets a different UIDVALIDITY.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Timo,
On 2009-11-10 15:30, Timo Sirainen wrote:
On Nov 10, 2009, at 2:21 AM, Patrick Nagel wrote:
So it seems that neither EXPUNGE nor re-SELECT seems to have any "refreshing" effect.
I'll see tomorrow if I can reproduce it.
Thanks and good night :)
Nikita Koshikov told me to "Try to add :INDEX=MEMORY to location setting".
After my tests above I tried that, and suddenly everything works as expected. It seems logical to have the index of a virtual folder in memory only, since it is generated dynamically anyway, and is outdated before it's even written to disk (at least without some configurable "virtual search result max age" parameter). So maybe the easiest solution is, to imply ':INDEX=MEMORY' when a namespace location is a virtual folder?
Well, that also makes the performance somewhat bad. And UIDVALIDITY changes every time, so client's local caching won't work. And if client uses multiple connections to the same mailbox, it might get confused since each connection gets a different UIDVALIDITY.
Ok, I wasn't aware of all those implications.
Patrick.
STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key: E883A005 https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAkr5GYsACgkQ7yMg/OiDoAXQUgCgrzG/aAH1oP/gju4cS4xedxRo T7YAoLbCN/O3d7YiCmR798mrWBlNw5IF =zvHa -----END PGP SIGNATURE-----
Finally fixed: http://hg.dovecot.org/dovecot-1.2/rev/9cccb8617820
On Tue, 2009-11-10 at 15:21 +0800, Patrick Nagel wrote:
So it seems that neither EXPUNGE nor re-SELECT seems to have any "refreshing" effect.
Nikita Koshikov told me to "Try to add :INDEX=MEMORY to location setting".
After my tests above I tried that, and suddenly everything works as expected. It seems logical to have the index of a virtual folder in memory only, since it is generated dynamically anyway, and is outdated before it's even written to disk (at least without some configurable "virtual search result max age" parameter). So maybe the easiest solution is, to imply ':INDEX=MEMORY' when a namespace location is a virtual folder?
That also makes it somewhat slower. Although if you're only searching flags, it doesn't matter much.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Timo,
On 2009-12-10 07:50, Timo Sirainen wrote:
Finally fixed: http://hg.dovecot.org/dovecot-1.2/rev/9cccb8617820
Great news, thanks! :)
STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key E883A005 https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/
iEYEARECAAYFAksgZwQACgkQ7yMg/OiDoAUzcgCgjLpN3VnjPrMMC95XXjM2cnwU ka4An2fmouXb9BOTr8aLGKPwWg4jWOx7 =22OC -----END PGP SIGNATURE-----
participants (3)
-
Nikita Koshikov
-
Patrick Nagel
-
Timo Sirainen