[Dovecot] IMAP views - and things to make Outlook Express not suck so much
Some of the things that keep being brought up SEEM TO ME to be a form of IMAP crack that can be accomplished in a much more generalized manner with a little bit of thought.
An idea I had a while back was for IMAP views. It accomplishes the four things I keep seeing on this list without being specific to any of them.
Consider new configuration variables:
rewrite Junk { folder = INBOX search = KEYWORD "Junk" post = +FLAGS (Junk) } rewrite Trash { folder = * search = DELETED fetch = -FLAGS (\Deleted) post = +FLAGS (\Deleted) }
Views would not be deletable (unless one wants to invent some nonstandard IMAP extensions today), but they would otherwise look like regular folders. One could post and search and fetch messages as normal (with some alterations to various IMAP fields).
In the examples I gave:
A001 SELECT Junk A002 SEARCH ALL A003 SELECT Trash A004 SEARCH ALL A005 FETCH 1 FLAGS
would be translated into:
A001 SELECT INBOX A002 SEARCH ALL KEYWORD Junk A003 SELECT {for each SUBSCRIBED folder} A004 SEARCH ALL DELETED A005 FETCH 1 FLAGS
Except, A005 would actually yield:
- 1 FETCH (FLAGS ())
Instead of:
- 1 FETCH (FLAGS (\Deleted))
It would probably be ideal if these were configurable per-user, but I'd be happy with system-wide rewriting.
Options for rewrite could include: folder = folders that could be selected using LSUB OR a valid (existing) folder name. If this is system wide, pretty much only INBOX, INBOX.% and INBOX.* and * and % are valid :) search = tags to append to SEARCH operations on the view fetch = IMAP fields that are acceptable to STORE that we will pretend occurred when fetching. recent = IMAP fields that should be STORE'd when the \Recent flag is still set. post = IMAP fields that are acceptable to STORE that we will perform when APPENDing or STOREing. append = argument to be passed to /bin/sh -c which will receive on standard input the APPENDed body, and produce on output a STOREed body. open = argument for /bin/sh -c for when this view is SELECTed or EXAMINEd close = argument to be passed to /bin/sh -c for when the view is closed. index = argument to be passed to /bin/sh -c for when dovecot wants to index a portion of the message. standard input of the argument is the message, whereas standard output is the portion to actually index. path = a regular dovecot mailbox path that would contain the real contents of this folder.
ro path =
like path, but would be decidedly read-only (except for
any side-effects generated by append =)Subprocesses that exit 99 would handle the task internally- thus any normal actions (such as for index = and append =) would not occur. Exit code 0 is for normal completion, and all other exit codes should generate an error.
If one allowed open = and close = to be used on REAL folders, this would allow people to actually have the behavior UW-IMAP does by MOVING mail from /v/s/m into ~/mbox...
One could use append = with a "Outbox" view that could write the queue at the same time as storing a copy into Sent Items. One could also use append for gpg-signing messages as they're being recorded. One could also use append for publishing to a mailing list.
One could use index = to spell-check messages beforehand thus improving my chances of actually finding messages from my PHB.
One could also use append =, recent = and ro path = to emulate shared folders support in a manner that would be useful for using IMAP folders as a kind of work-queue.
Thoughts?
On Wed, 2004-05-26 at 21:43, Geo Carncross wrote:
An idea I had a while back was for IMAP views. It accomplishes the four things I keep seeing on this list without being specific to any of them.
I've thought about something similiar before, but I've mostly just thought about performance and complexity issues with it. Creating a single view from many mailboxes could be slow without some special indexing.
Good ideas anyway, I'll think about them more once I get around to implementing the thing (unless someone else wants to? :), sometimes after v1.0.
Simple open/close/append triggers would be easier to implement, those could be done with a plugin.
On Wed, May 26, 2004 at 10:34:36PM +0300, Timo Sirainen wrote:
On Wed, 2004-05-26 at 21:43, Geo Carncross wrote:
An idea I had a while back was for IMAP views. It accomplishes the four things I keep seeing on this list without being specific to any of them.
I've thought about something similiar before, but I've mostly just thought about performance and complexity issues with it. Creating a single view from many mailboxes could be slow without some special indexing.
Just as an implementation point; it would be good if the specification language was Sieve.
J
-- Joshua Goodall "as modern as tomorrow afternoon" joshua@roughtrade.net - FW109
On Wed, 2004-05-26 at 17:42, Joshua Goodall wrote:
On Wed, May 26, 2004 at 10:34:36PM +0300, Timo Sirainen wrote:
On Wed, 2004-05-26 at 21:43, Geo Carncross wrote:
An idea I had a while back was for IMAP views. It accomplishes the four things I keep seeing on this list without being specific to any of them.
I've thought about something similiar before, but I've mostly just thought about performance and complexity issues with it. Creating a single view from many mailboxes could be slow without some special indexing.
Just as an implementation point; it would be good if the specification language was Sieve.
This could require the IMAP server fetch the entire contents of a message before processing something as simple as a FETCH FLAGS command.
Also, much of sieve would be otherwise useless (fileinto, keep, etc).
It'd also require sieve be available to dovecot, which it isn't (yet).
AS an implementation point, poisoning the IMAP queries would probably be easier- and currently, wouldn't require any architectural changes (except to support matching multiple folders).
On Wed, May 26, 2004 at 07:31:30PM -0400, Geo Carncross wrote:
Just as an implementation point; it would be good if the specification language was Sieve.
This could require the IMAP server fetch the entire contents of a message before processing something as simple as a FETCH FLAGS command.
That would be a naive implementation.
Also, much of sieve would be otherwise useless (fileinto, keep, etc).
The rationale is this: if a Sieve-based delivery agent exists, you'll be using some kind of frontend tool to write configuration filters. It is a big usability win if the delivery filter language, and thus the interface, has the same logic as the virtual folder specification language and interface.
It'd also require sieve be available to dovecot, which it isn't (yet).
Indeed. If there was a dovecot-API-based delivery agent, I hope it would speak Sieve. Such a thing lurks at the back of my mind in the stack marked "interesting projects".
J
-- Joshua Goodall "as modern as tomorrow afternoon" joshua@roughtrade.net - FW109
On Thu, 2004-05-27 at 20:07, Joshua Goodall wrote:
On Wed, May 26, 2004 at 07:31:30PM -0400, Geo Carncross wrote:
Just as an implementation point; it would be good if the specification language was Sieve.
This could require the IMAP server fetch the entire contents of a message before processing something as simple as a FETCH FLAGS command.
That would be a naive implementation.
I didn't say _would_, I said _could_. Read on...
Also, much of sieve would be otherwise useless (fileinto, keep, etc).
The rationale is this: if a Sieve-based delivery agent exists, you'll be using some kind of frontend tool to write configuration filters. It is a big usability win if the delivery filter language, and thus the interface, has the same logic as the virtual folder specification language and interface.
One might add an "appearin" extension to sieve that would work like "fileinto" but only be a reference to wherever the message actually went.
If this were the case, the task could actually be made simpler: "appearin" would add the message to the mailbox, but mark it with some special flags that Dovecot would know that actions on this message must be postprocessed by the sieve program (after mirroring to the original copy of the message-- wherever it was fileinto'd).
Whenever changes where made, the copies of the messages would be deleted in such a way that they were REALLY DELETED (without having to EXPUNGE) and they could be redelivered.
The obvious benefit to this manner is that sieve actually makes this behavior pleasant, and it requires only MINIMAL changes to dovecot.
_AND_ you get to pretend you can reuse all your helpful sieve scripts and script generators :)
[[ and in reality, you probably can, with minor adjustments to take advantage of appearin ]]
With this, the Outbox and Shared virtual folders could be implemented. One could also implement the PHB-filter I described.
However. This doesn't make it possible to implement the virtual Trashbox (unless sieve could ask questions about mailboxes instead of "the current message"), the virtual Junk folder (same reason). Sieve doesn't have a way to ask "FOR ALL DELETED MESSAGES", or "FOR ALL KEYWORD JUNK MESSAGES" -- instead you have to say "if this message is deleted", or "if this message is junk" -- something that unless the sieve implementation was optimized for, would require an extraneous amount of work.
Much more so than playing IMAP questions, anyway.
It'd also require sieve be available to dovecot, which it isn't (yet).
Indeed. If there was a dovecot-API-based delivery agent, I hope it would speak Sieve. Such a thing lurks at the back of my mind in the stack marked "interesting projects".
Sieve is not a complicated language, but adding the "appearin" extension wouldn't require a system so intricately tied to dovecot. One could use arbitrary scripts, with "appearin" implemented as a simple program. It might suit some peoples desires, but I don't think it would help mine out any...
On Wed, 2004-05-26 at 15:34, Timo Sirainen wrote:
On Wed, 2004-05-26 at 21:43, Geo Carncross wrote:
An idea I had a while back was for IMAP views. It accomplishes the four things I keep seeing on this list without being specific to any of them.
I've thought about something similiar before, but I've mostly just thought about performance and complexity issues with it. Creating a single view from many mailboxes could be slow without some special indexing.
It shouldn't be slower than performing the actions manually. The only "tricky" part would be the UIDL/ID matchups. If one is willing to implement 64-bit integers, or accept a 2^32 maximal limit on the number of messages accessible in a view, then the problem can be reduced to m=n*nfol+folnum where nfol is the number of subscribed folders, folnum is the relay counter, n is the original UIDL or ID number, and m is the resulting UIDL/ID passed to the client.
Am I missing something about dovecot's implementation?
On 27.5.2004, at 02:31, Geo Carncross wrote:
I've thought about something similiar before, but I've mostly just thought about performance and complexity issues with it. Creating a single view from many mailboxes could be slow without some special indexing.
It shouldn't be slower than performing the actions manually.
Sure, it's not, but clients aren't doing this manually now. Doing it in a simple way would be easy I guess, but I'm not sure if the performance would be good enough to be acceptable.. Especially if a view consists of tens of mailboxes. Each synchronization would require scanning through changes in all of them.
I had been thinking about making some global index about mailboxes and their states. So that by just reading one file would be able to tell what mailboxes have changed. With that kind of system it would work much better.
The only "tricky" part would be the UIDL/ID matchups. If one is willing to implement 64-bit integers, or accept a 2^32 maximal limit on the number of messages accessible in a view, then the problem can be reduced to m=n*nfol+folnum where nfol is the number of subscribed folders, folnum is the relay counter, n is the original UIDL or ID number, and m is the resulting UIDL/ID passed to the client.
IMAP spec only allows using 32bit numbers, so UIDs and message counts can be max. 2^32-1. But I don't think the UID lookup is too difficult, I could just create a new index of uid => {original mailbox id, uid} mapping with simple increasing UIDs.
On Wed, 2004-05-26 at 20:20, Timo Sirainen wrote:
On 27.5.2004, at 02:31, Geo Carncross wrote:
I've thought about something similiar before, but I've mostly just thought about performance and complexity issues with it. Creating a single view from many mailboxes could be slow without some special indexing.
It shouldn't be slower than performing the actions manually.
Sure, it's not, but clients aren't doing this manually now. Doing it in a simple way would be easy I guess, but I'm not sure if the performance would be good enough to be acceptable.. Especially if a view consists of tens of mailboxes. Each synchronization would require scanning through changes in all of them.
Perhaps another way would be to maintain separate indexes for each view and (see below).
I had been thinking about making some global index about mailboxes and their states. So that by just reading one file would be able to tell what mailboxes have changed. With that kind of system it would work much better.
This would be easier still with an LDA that could pull a trigger when a mailbox needs re-syncing. One could enable a configuration option called (say) lda_trigger = that would refer to a spec identifying a pipe (possibly per mailbox) that dovecot could select() on. any writes to it would fill the buffer and cause dovecot to rescan.
If dovecot wasn't watching the pipe, and the pipe buffer got full, "oh well", dovecot should expunge the pipe buffer when it starts up anyway. the LDA trigger-pull could be as simple as:
|echo '' > mailbox/lda_trigger 2>/dev/null; exit 0
being added to ones .qmail file. Similarly, .forwards could be altered similarly.
This watching of a pipe could solve other problems (like avoiding extra stat() calls)
participants (3)
-
Geo Carncross
-
Joshua Goodall
-
Timo Sirainen