[Dovecot] Manual thread breaking from Thunderbird
Hello
I'm looking for a way to manually break threads apart when independent threads are mixed up, mostly by users which aren't aware that they should not be replying to existing threads when starting a new one.
Since Thunderbird afaik does the threading itself, it would require a solution that actually changes the In-Reply-To header (and maybe also that other (non-standard?) header which is being sent by the M$ clients). For me modifying the actual emails on disk would be fine, i.e. renaming the In-Reply-To header to Old-In-Reply-To and such. (I'd actually *prefer* a solution that modifies the server side store, as opposed to one which keeps that info on the client side only like mutt seems to be doing.)
Is there a way to do this already? Seems like a task for a dovecot plugin--or for the xexec plugin and an external perl script or some such?
For triggering the renaming, a hacky solution could be to misuse labels (e.g. marking a message starred and "Later" and "Important" at the same time would trigger the modification and remove the labels).
When implementing such a thing: maybe also offer a way to modify subjects (for those cases where people are using short meaningless and recurring subjects--to both help breaking up the ensuing threading as well as giving them a useful replacement), maybe as an IMAP extension or such (to be used by Thunderbird plugins (?), or by web apps).
(I can code in C and Perl, but don't generally have much time.)
Christian.
Christian Jaeger wrote:
I'm looking for a way to manually break threads apart when independent threads are mixed up, mostly by users which aren't aware that they should not be replying to existing threads when starting a new one.
Sounds like a great idea!
I'm curious though, how do convince your Thunderbird and Dovecot to work like Gmail and keep everything threaded in the same folder in the first place?
(Assuming that you do this and apply labels to tell things apart..)
rosenfield.albert@gmail.com wrote:
Christian Jaeger wrote:
I'm looking for a way to manually break threads apart when independent threads are mixed up, mostly by users which aren't aware that they should not be replying to existing threads when starting a new one.
Sounds like a great idea!
I'm curious though, how do convince your Thunderbird and Dovecot to work like Gmail and keep everything threaded in the same folder in the first place?
I'm not sure whether I understand you correctly: are you asking how to turn on threading in Thunderbird? (That would be in the menu View -> Sort by -> Threaded.) But that's Thunderbird's doing, Dovecot isn't really involved afaik, I assume Thunderbird will look at the message-id and in-reply-to headers of the mails.
Or are you asking about the Gmail feature of not using subfolders but keep everything in the same place and only use search to find your stuff? (Well, I haven't used Gmail much, one of the reason being that when I tried it I thought how would you keep mailing lists separated sanely? (Without having to create separate custom queries for each of them or so.))
Christian.
Christian Jaeger wrote:
rosenfield.albert@gmail.com wrote:
Christian Jaeger wrote:
I'm looking for a way to manually break threads apart when independent threads are mixed up, mostly by users which aren't aware that they should not be replying to existing threads when starting a new one.
Sounds like a great idea!
I'm curious though, how do convince your Thunderbird and Dovecot to work like Gmail and keep everything threaded in the same folder in the first place?
I'm not sure whether I understand you correctly: are you asking how to turn on threading in Thunderbird? (That would be in the menu View -> Sort by -> Threaded.) But that's Thunderbird's doing, Dovecot isn't really involved afaik, I assume Thunderbird will look at the message-id and in-reply-to headers of the mails.
and References and few other heuristics.
Or are you asking about the Gmail feature of not using subfolders but keep everything in the same place and only use search to find your stuff?
I guess that's what he means. In short, how to have the same message appear in different folder (be them "virtual" or not). and I guess, if you delete a message, you no more see it in the said folders.
I believe the right place for this feature is in the MUA (thunderbird) to avoid downloading the same message N times.
(Well, I haven't used Gmail much, one of the reason being that when I tried it I thought how would you keep mailing lists separated sanely? (Without having to create separate custom queries for each of them or so.))
Christian.
At 1:41 PM +0200 5/7/08, rosenfield.albert@gmail.com wrote:
Christian Jaeger wrote:
I'm looking for a way to manually break threads apart when independent threads are mixed up, mostly by users which aren't aware that they should not be replying to existing threads when starting a new one.
Sounds like a great idea!
The IMAP extension that supports threading (http://tools.ietf.org/html/draft-ietf-imapext-sort-20) has no support for clients 'breaking' threads. In fact, it does not explicitly or implicitly require that the IMAP server preserve any state about threading that cannot be reconstructed from the messages, but rather implies that an IMAP server rebuilds the thread map from the messages for each THREAD command.
This means that if you want to break threads, it has to be entirely done in the client. Personally, I think that's where all threading belongs anyway.
I suppose one could also try to define a new IMAP threading extension or to convince Mark Crispin to make radical changes to the one he's been working on (and implementing in UW-IMAP) for 8 years, but I suspect that those approaches would be high-effort and low-yield.
I'm curious though, how do convince your Thunderbird and Dovecot to work like Gmail and keep everything threaded in the same folder in the first place?
Dovecot isn't really a player in that question, which is really about how to make TBird do what you want it to do with messages and present them in a particular organizational style.
-- Bill Cole bill@scconsult.com
On May 7, 2008, at 5:59 PM, Bill Cole wrote:
At 1:41 PM +0200 5/7/08, rosenfield.albert@gmail.com wrote:
Christian Jaeger wrote:
I'm looking for a way to manually break threads apart when
independent threads are mixed up, mostly by users which aren't aware that they
should not be replying to existing threads when starting a new one.Sounds like a great idea!
The IMAP extension that supports threading (http://tools.ietf.org/html/draft-ietf-imapext-sort-20 ) has no support for clients 'breaking' threads.
Thunderbird and most other clients don't use server-side threading, so
this doesn't matter there.
I suppose one could also try to define a new IMAP threading
extension or to convince Mark Crispin to make radical changes to the
one he's been working on (and implementing in UW-IMAP) for 8 years,
but I suspect that those approaches would be high-effort and low- yield.
The existing THREAD=REFERENCES can't be changed, but new ones can be
added. Dovecot v1.2 will have THREAD=X-REFERENCES2 which I'll probably
try to get standardized.
It would be possible to make a new threading algorithm that started a
new thread if a message had e.g. "newthread" keyword. But I don't
really see it being worth the trouble.
I'm curious though, how do convince your Thunderbird and Dovecot to work like Gmail and keep everything threaded in the same folder in
the first place?
Virtual mailboxes would do that trick. Dovecot v1.2 will have support
for that, although many clients also support it already.
Bill Cole:
Dovecot isn't really a player in that question, which is really about how to make TBird do what you want it to do with messages and present them in a particular organizational style.
Well, Dovecot "deliver" puts the messages in place, and you want incoming messages to land in the same folder as the conversation it is already is in, and also have other virtual labels applied depending on filter rules (eg dovecot-ml emails should have the "dovecot" label applied).
Christian Jaeger:
Or are you asking about the Gmail feature of not using subfolders but keep everything in the same place and only use search to find your stuff?
That's rather simplified - it's "using search", plus it's labels, an "archive" button, a new threading interface, combining multiple messages into one, etc. But yeah. I had assumed that you made your TBird behave like that..
Timo Sirainen:
Virtual mailboxes would do that trick. Dovecot v1.2 will have support for that, although many clients also support it already.
Awesome :-).. Any more information somewhere? Which clients?
On May 7, 2008, at 6:54 PM, rosenfield.albert@gmail.com wrote:
Timo Sirainen:
Virtual mailboxes would do that trick. Dovecot v1.2 will have support for that, although many clients also support it already.
Awesome :-).. Any more information somewhere?
http://dovecot.org/list/dovecot/2008-March/029546.html
Which clients?
Mail.app and Evolution at least. And doesn't Outlook also support them
nowadays?
On 5/7/2008, Timo Sirainen (tss@iki.fi) wrote:
Mail.app and Evolution at least. And doesn't Outlook also support them nowadays?
Thunderbird does - but the last time I tried to use them, the performance was terrible when large folders were included in the search...
Its been a while though, so maybe it has gotten better...
--
Best regards,
Charles
Mail.app and Evolution at least. And doesn't Outlook also support them nowadays?
Ah, you mean "saved search" containers in the tree view? Thunderbird has these too. Doesn't quite make it Gmail though.
I mean, it's a cute feature, but it does not modify the work flow drastically enough to be even remotely useful as anything else than a curiosity. Gmail modifies, or rather, fixes the entire work flow ;-).
On May 7, 2008, at 7:41 PM, rosenfield.albert@gmail.com wrote:
Mail.app and Evolution at least. And doesn't Outlook also support them nowadays?
Ah, you mean "saved search" containers in the tree view? Thunderbird has these too. Doesn't quite make it Gmail though.
I mean, it's a cute feature, but it does not modify the work flow drastically enough to be even remotely useful as anything else than a curiosity. Gmail modifies, or rather, fixes the entire work flow ;-).
I haven't used it myself, but .. Hmm. Wasn't it about showing entire
threads of messages in INBOX? With Dovecot it would be done by:
mailboxes = INBOX "Sent Messages" search = INTHREAD REFERENCES MAILBOX INBOX
So it matches entire threads of messages in INBOX. INTHREAD would need
adding support for INTHREAD extension.
I haven't used it myself, but ..
If you're interested in the client part of mailing, perhaps create a dummy account, join the dovecot and postfix mailing lists, create two new filter rules that apply either "postfix" or "dovecot" labels and archives (= unapplies "Inbox" label) messages depending on the to/cc line, and try it out :-).
Hmm. Wasn't it about showing entire threads of messages in INBOX?
Yes, or under any other label than "Inbox".
Also, the messages does not need to have the "Inbox" label to show up under Inbox - they can have only the "Sent" label, and they will still show up in the thread as long as they're part of it as per In-Reply-To: or whatever.
Obviously useful since if you send an item ("Sent" label applied) and receive an answer ("Inbox" label applied), the exact same conversation will show up under both labels, and with the full list of messages relevant to it.
With Dovecot it would be done by:
mailboxes = INBOX "Sent Messages" search = INTHREAD REFERENCES MAILBOX INBOX
So it matches entire threads of messages in INBOX.
So if I understand correctly, this will basically tell the IMAP client *all* the messages relating to a given conversation, regardless of whether they were sent or incoming?
(With the flip-side that software which expects to use different folders, eg say a Horde webmail alternative for the account, will stop working against this mailbox... Or is this all or none regarding which mailbox are affected?)
INTHREAD would need adding support for INTHREAD extension.
Ok, what does that do? (Apologies for being in learning mode.)
On Wed, 2008-05-07 at 19:49 +0200, rosenfield.albert@gmail.com wrote:
I haven't used it myself, but ..
If you're interested in the client part of mailing, perhaps create a dummy account,
I've a test account which I've used a couple of times, but haven't bothered too much with it.
Hmm. Wasn't it about showing entire threads of messages in INBOX?
Yes, or under any other label than "Inbox".
Also, the messages does not need to have the "Inbox" label to show up under Inbox - they can have only the "Sent" label, and they will still show up in the thread as long as they're part of it as per In-Reply-To: or whatever.
Right.
With Dovecot it would be done by:
mailboxes = INBOX "Sent Messages" search = INTHREAD REFERENCES MAILBOX INBOX
So it matches entire threads of messages in INBOX.
So if I understand correctly, this will basically tell the IMAP client *all* the messages relating to a given conversation, regardless of whether they were sent or incoming?
Yes, as long as at least one message in the thread exists in INBOX (the "MAILBOX INBOX" search term).
(With the flip-side that software which expects to use different folders, eg say a Horde webmail alternative for the account, will stop working against this mailbox... Or is this all or none regarding which mailbox are affected?)
I'm not sure what you mean. There are still different mailboxes. This just creates virtual mailboxes on top of the existing ones.
INTHREAD would need adding support for INTHREAD extension.
Ok, what does that do? (Apologies for being in learning mode.)
Just makes the search match all messages in a thread if at least one message in the thread matches the search terms.
I've a test account which I've used a couple of times, but haven't bothered too much with it.
Guess it's not for everyone.
(Personally, I dislike that my email is stored at Googleplex, and that I cannot fiddle with the server setup. Otherwise, it's pretty darn great.)
With Dovecot it would be done by:
mailboxes = INBOX "Sent Messages" search = INTHREAD REFERENCES MAILBOX INBOX ... Yes, as long as at least one message in the thread exists in INBOX (the "MAILBOX INBOX" search term).
So the above "mailboxes=, search=" is not a Dovecot configuration option but a search performed by an IMAP client, right.. (learning mode!)
I'm not sure what you mean.
I just misunderstood something basic (see above).
There are still different mailboxes. This just creates virtual mailboxes on top of the existing ones.
"Creates" as in "the client has to keep track of them client-side, the server knows nothing about them", right?
[INTHREAD] just makes the search match all messages in a thread if at least one message in the thread matches the search terms.
Niice.. Can an unmodified MUA be expected to correctly fetch the individual messages found with INTHREAD? Or does it need to "switch folders" or perform other IMAP magic.
On Wed, 2008-05-07 at 20:13 +0200, rosenfield.albert@gmail.com wrote:
With Dovecot it would be done by:
mailboxes = INBOX "Sent Messages" search = INTHREAD REFERENCES MAILBOX INBOX ... Yes, as long as at least one message in the thread exists in INBOX (the "MAILBOX INBOX" search term).
So the above "mailboxes=, search=" is not a Dovecot configuration option but a search performed by an IMAP client, right.. (learning mode!)
No, I did meant that they were Dovecot configuration options that describe a single virtual mailbox (although the actual configuration format is currently a bit different).
There are still different mailboxes. This just creates virtual mailboxes on top of the existing ones.
"Creates" as in "the client has to keep track of them client-side, the server knows nothing about them", right?
Clients could of course do this also, but I can't really do anything about them.
[INTHREAD] just makes the search match all messages in a thread if at least one message in the thread matches the search terms.
Niice.. Can an unmodified MUA be expected to correctly fetch the individual messages found with INTHREAD? Or does it need to "switch folders" or perform other IMAP magic.
The configured virtual mailbox would look exactly like a normal mailbox to the client.
The configured virtual mailbox would look exactly like a normal mailbox to the client.
The advantage being that Dovecot would index messages in all physical folders for super-fast access when asking via IMAP for messages relating to a specific thread-id. (Hope I got this right.)
In case I wanted to modify the MUA to support better threading and tags/labels, is that enough? Can I simply tell Dovecot via IMAP to create a new virtual folder that aggregates virtual messages from all the other folders and it will start indexing them for thread-locality?
(Would be great, because it's always easier when you only have to change either the MUA or the "post office" side of things.)
Is it possible to ask Dovecot via IMAP to look for other things in the virtual folder, for example all items with the flag "huey lewis" applied? Is this sort of query fast / indexed?
On Wed, 2008-05-07 at 20:37 +0200, rosenfield.albert@gmail.com wrote:
The configured virtual mailbox would look exactly like a normal mailbox to the client.
The advantage being that Dovecot would index messages in all physical folders for super-fast access when asking via IMAP for messages relating to a specific thread-id. (Hope I got this right.)
That could be one use case, but since no clients do that I doubt it's a very useful one for a while. :) I'm mostly thinking about using virtual mailboxes for custom webmails, at least initially.
In case I wanted to modify the MUA to support better threading and tags/labels, is that enough? Can I simply tell Dovecot via IMAP to create a new virtual folder that aggregates virtual messages from all the other folders and it will start indexing them for thread-locality?
Currently only admin can configure them, but it would be possible to create a new extension for configuring them via IMAP. There are already a couple drafts related to it.
Is it possible to ask Dovecot via IMAP to look for other things in the virtual folder, for example all items with the flag "huey lewis" applied?
Yes (other than spaces not being allowed in flags). It's built using a fully functional IMAP SEARCH query.
Is this sort of query fast / indexed?
Current implementation is a bit slow, but in future it will be fast.
That could be one use case, but since no clients do that I doubt it's a very useful one for a while. :) I'm mostly thinking about using virtual mailboxes for custom webmails, at least initially.
;-).. I find it to be a problem that you're the one who decides what it's useful for and what it's not :-P.
Is it possible to ask Dovecot via IMAP to look for other things in the virtual folder, for example all items with the flag "huey lewis" applied?
Yes (other than spaces not being allowed in flags). It's built using a fully functional IMAP SEARCH query.
Is this sort of query fast / indexed?
Current implementation is a bit slow, but in future it will be fast.
So at what point will it be possible to have Dovecot index whatever the client asks it to index? :-)
On Wed, 2008-05-07 at 20:51 +0200, rosenfield.albert@gmail.com wrote:
That could be one use case, but since no clients do that I doubt it's a very useful one for a while. :) I'm mostly thinking about using virtual mailboxes for custom webmails, at least initially.
;-).. I find it to be a problem that you're the one who decides what it's useful for and what it's not :-P.
I'm not deciding, just observing.
Is it possible to ask Dovecot via IMAP to look for other things in the virtual folder, for example all items with the flag "huey lewis" applied?
Yes (other than spaces not being allowed in flags). It's built using a fully functional IMAP SEARCH query.
Is this sort of query fast / indexed?
Current implementation is a bit slow, but in future it will be fast.
So at what point will it be possible to have Dovecot index whatever the client asks it to index? :-)
No idea. Would be at least useful to have a standard to base it on. Of course no-one prevents you from coding it sooner. :)
No idea. Would be at least useful to have a standard to base it on. Of course no-one prevents you from coding it sooner. :)
Gonna be tough to squeeze in between all the updates that needs to be done to the MUA. Hopefully it doesn't take more than 5 minutes to set up a Thunderbird build environment.
..... Mu-ha-ha...
At 5:54 PM +0200 5/7/08, rosenfield.albert@gmail.com wrote:
Bill Cole: [in regards to making TBird and Dovecot work like GMail]
Dovecot isn't really a player in that question, which is really about how to make TBird do what you want it to do with messages and present them in a particular organizational style.
Well, Dovecot "deliver" puts the messages in place, and you want incoming messages to land in the same folder as the conversation it is already is in, and also have other virtual labels applied depending on filter rules (eg dovecot-ml emails should have the "dovecot" label applied).
Yes, but 'deliver' does not itself make those decisions. If you want mail delivered anywhere other than INBOX, you need to either call deliver with a -m argument or use a plugin (e.g. sieve) and as far as I can tell, only the sieve plugin can set IMAP message flags on delivery. As for 'virtual labels,' I assume you are referring to TBird's 'Tags' which map loosely to IMAP custom message flags, (a.k.a. keywords.) The looseness is that TBird seems to ignore keywords that it has not been explicitly configured to show, so if sieve does add a keyword to a message, it has to be one known to TBird in order to be used by TBird.
There has to be user-specific and MUA-specific configuration for the usage model you seem to want. There really isn't much more that a LDA/IMAP combination can do to support a GMail-like presentation, that has to be done by the MUA, i.e TBird.
-- Bill Cole bill@scconsult.com
delivered anywhere other than INBOX, you need to either call deliver with a -m argument or use a plugin (e.g. sieve) and as far as I can tell, only the
Fair enough.
(Could never get sieve (cmusieve) to work with Dovecot - what does other people use that's simple to set up and works efficiently?)
labels,' I assume you are referring to TBird's 'Tags' which map loosely to IMAP custom message flags, (a.k.a. keywords.)
Sure, whatever. The principal difference between the two, I guess, is that a gmail label, when created, are automatically given an item in the tree view, which when clicked shows a list of all threads where at least one non-spam, non-trash mail item has the relevant label applied.
(How to configure TBird to do something like this with Tags is either impossible or beyond me, I'm afraid.)
There has to be user-specific and MUA-specific configuration for the usage model you seem to want.
That's a fair enough response, I'm willing to accept that ;-).
participants (6)
-
Bill Cole
-
Charles Marcus
-
Christian Jaeger
-
mouss
-
rosenfield.albert＠gmail.com
-
Timo Sirainen