Modify stored mail contents?
Is there a safe way to modify the contents of emails stored by Dovecot? I'll probably only want to change the message bodies, not the headers, if that matters. Looking for ways to do this both for existing emails and new emails as they are received (though anything that works for existing emails can probably just be run again for new emails.) My mail storage is currently mdbox, but I could migrate to another format if that helps.
On Sat, 22 Jul 2017 12:51:15 +0200, Evan Martin stated:
Is there a safe way to modify the contents of emails stored by Dovecot? I'll probably only want to change the message bodies, not the headers, if that matters. Looking for ways to do this both for existing emails and new emails as they are received (though anything that works for existing emails can probably just be run again for new emails.) My mail storage is currently mdbox, but I could migrate to another format if that helps.
You could just view the message in your MUA and then save it to another drive, or whatever. Then, using a text editor, you could modify it to your hearts content.
Exactly, what problem are you trying to address?
-- Jerry
Yes, obviously clients can save messages. I meant: to modify messages in bulk, on the server, replacing the existing message bodies stored by Dovecot.
On 22/07/2017 3:56 PM, Jerry wrote:
On Sat, 22 Jul 2017 12:51:15 +0200, Evan Martin stated:
Is there a safe way to modify the contents of emails stored by Dovecot? I'll probably only want to change the message bodies, not the headers, if that matters. Looking for ways to do this both for existing emails and new emails as they are received (though anything that works for existing emails can probably just be run again for new emails.) My mail storage is currently mdbox, but I could migrate to another format if that helps. You could just view the message in your MUA and then save it to another drive, or whatever. Then, using a text editor, you could modify it to your hearts content.
Exactly, what problem are you trying to address?
On Sat, 22 Jul 2017 12:51:15 +0200, Evan Martin stated:
Is there a safe way to modify the contents of emails stored by Dovecot? I'll probably only want to change the message bodies, not the headers, if that matters. Looking for ways to do this both for existing emails and new emails as they are received (though anything that works for existing emails can probably just be run again for new emails.) My mail storage is currently mdbox, but I could migrate to another format if that helps.
On 22/07/2017 3:56 PM, Jerry wrote:
You could just view the message in your MUA and then save it to another drive, or whatever. Then, using a text editor, you could modify it to your hearts content.
Exactly, what problem are you trying to address?
On Sat, 22 Jul 2017 16:05:19 +0200, Evan Martin stated:
Yes, obviously clients can save messages. I meant: to modify messages in bulk, on the server, replacing the existing message bodies stored by Dovecot.
I am not understanding what problem you are trying to alleviate. Are these your messages or those of your clients?
-- Jerry
On 22/07/2017 11:51, Evan Martin wrote:
Is there a safe way to modify the contents of emails stored by Dovecot? I'll probably only want to change the message bodies, not the headers, if that matters.
I use javamail[1] for this. The javamail API can be used to get and put emails with IMAP so dovecot is kept in sync with the changes you make between fetching and storing.
It looks like the mail filter plugin [https://wiki2.dovecot.org/Plugins/MailFilter] is almost exactly what I want, except for this:
Currently the filtering must not modify the message in any way: mail -> write filter -> read filter -> must produce exactly the original mail back. (TODO: Modifying the mail during writing would be possible with some code changes.)
Is there any prospect of those code changes being made, so that the filter can modify mail contents?
There's no indication in the docs or the code of what would break if the contents were modified, but I'm guessing indexes and caches would be out of date and would need to be rebuilt? Is it possible to just disable those? I don't need high performance.
On 22/07/2017 12:51 PM, Evan Martin wrote:
Is there a safe way to modify the contents of emails stored by Dovecot? I'll probably only want to change the message bodies, not the headers, if that matters. Looking for ways to do this both for existing emails and new emails as they are received (though anything that works for existing emails can probably just be run again for new emails.) My mail storage is currently mdbox, but I could migrate to another format if that helps.
On 23-07-17 13:07, Evan Martin wrote:
It looks like the mail filter plugin [https://wiki2.dovecot.org/Plugins/MailFilter] is almost exactly what I want, except for this:
Currently the filtering must not modify the message in any way: mail -> write filter -> read filter -> must produce exactly the original mail back. (TODO: Modifying the mail during writing would be possible with some code changes.)
Is there any prospect of those code changes being made, so that the filter can modify mail contents?
There's no indication in the docs or the code of what would break if the contents were modified, but I'm guessing indexes and caches would be out of date and would need to be rebuilt? Is it possible to just disable those? I don't need high performance.
On 22/07/2017 12:51 PM, Evan Martin wrote:
Is there a safe way to modify the contents of emails stored by Dovecot? I'll probably only want to change the message bodies, not the headers, if that matters. Looking for ways to do this both for existing emails and new emails as they are received (though anything that works for existing emails can probably just be run again for new emails.) My mail storage is currently mdbox, but I could migrate to another format if that helps.
In general, you should not do this. When a message is stored using IMAP, it is immutable. The IMAP server also remembers things like size and assigns messages a unique ID, so mail readers that have already downloaded the message with that ID, don't have to download the whole message again to verify whether it's contents have magically changed. What you're suggesting is not simply compatible with IMAP standards.
The normal way of applying changes to messages is just like a mail client connecting to IMAP: create a new message and save it to the store, then delete the old one.
Two people already asked you in this thread what the actual problem is you're trying to solve, but you failed to answer the question. I'm afraid that we can't help you any further without more detail.
Kind regards, Tom
On 23 Jul 2017, at 16.14, Tom Hendrikx tom@whyscream.net wrote:
In general, you should not do this. When a message is stored using IMAP, it is immutable. The IMAP server also remembers things like size and assigns messages a unique ID, so mail readers that have already downloaded the message with that ID, don't have to download the whole message again to verify whether it's contents have magically changed. What you're suggesting is not simply compatible with IMAP standards.
The normal way of applying changes to messages is just like a mail client connecting to IMAP: create a new message and save it to the store, then delete the old one.
Just like that. But instead of using IMAP interface to do it, you can also do it with doveadm:
- doveadm fetch mail
- doveadm delete mail
- modify fetched mail
- doveadm import modified mail back
Simple as that. There is no other supported way to do it. Editing mail objects on storage will break things.
Sami
On 23/07/2017 3:30 PM, Sami Ketola wrote:
On 23 Jul 2017, at 16.14, Tom Hendrikx tom@whyscream.net wrote:
In general, you should not do this. When a message is stored using IMAP, it is immutable. The IMAP server also remembers things like size and assigns messages a unique ID, so mail readers that have already downloaded the message with that ID, don't have to download the whole message again to verify whether it's contents have magically changed. What you're suggesting is not simply compatible with IMAP standards.
The normal way of applying changes to messages is just like a mail client connecting to IMAP: create a new message and save it to the store, then delete the old one. Just like that. But instead of using IMAP interface to do it, you can also do it with doveadm:
- doveadm fetch mail
- doveadm delete mail
- modify fetched mail
- doveadm import modified mail back
Simple as that. There is no other supported way to do it. Editing mail objects on storage will break things.
Sami
Thank you, that's an interesting idea. Exporting the mail and deleting it is easy enough, but I'm not sure where I would import the message from. doveadm import expects a mailbox store as source, so I'm back to the problem of safely writing emails in a mailbox store format that Dovecot understands!
I can export to Maildir format [doveadm backup -u USER "maildir:/mymaildir:LAYOUT=fs"], where each file contains one message and nothing else, but even that has extra files like dovecot-uidlist, dovecot.index.cache, etc. I'm not sure if it's safe to import from a Maildir where the message contents have been modified, but the other files haven't.
Also, would the import create new UIDs? I probably want UIDs to change, so that the IMAP client re-downloads the messages. I'd want to preserve IMAP flags like "Seen", though and, ideally, the sequence of messages inside a mailbox.
participants (5)
-
Evan Martin
-
James
-
Jerry
-
Sami Ketola
-
Tom Hendrikx