[trees-plugin] - Dovecot index gets corrupted, when using maildir and recievend and accessing mail at the same time
Hello dovecot team,
at first I want to thank you the awesome dovecot software.
I'm currently researching and developing the trees plugin (https://0xacab.org/riseuplabs/trees). In the future I would love to package this software for debian. I run into a problem, which I can not fix by myself, that's why I'm reaching out to you.
The plugin is producing a reproducible error, when using dovecots mailbox format maildir, and receiving and accessing mail at the same time. I'm searching for additional information to fix that.
I know that it's may not the scope of the mailing list to support external plugins, but since you are the experts, I thought I should ask at least.
Trees Plugin:
The plugin adds individually encrypted mail storage to the Dovecot IMAP server. It encrypts all (incoming and outgoing) mail on the mail server with a key, which can only be accessed with the users password. So the server operator is not able to see any mail content, if it's saved on the hard disk.
Bug Description:
I'm using a postfix dovecot setup with the enabled trees plugin with the mailboxformat maildir. When I'm receiving and accessing mail at the same time, the dovecot index gets corrupted. For accessing those mail I use Thunderbird.
Example corruption:
ug 4 09:55:15 trees dovecot: imap-login: Login: user=<treesenabled@trees.testing>, method=PLAIN, rip=192.168.22.1, lip=192.168.22.77, mpid=4643, TLS, session=<w0OCCplyhLLAqBYB> Aug 4 09:55:15 trees dovecot: imap(treesenabled@trees.testing): Debug: trees plugin initialized Aug 4 09:55:15 trees dovecot: imap(treesenabled@trees.testing): Connection closed in=506 out=495 Aug 4 09:55:15 trees dovecot: imap(treesenabled@trees.testing): Error: Maildir filename has wrong S value, renamed the file from /var/vmail/trees.testing/treesenabled/Maildir/cur/1533376515.M379290P4641.trees,S=1456,W=748:2, to /var/vmail/trees.testing/treesenabled/Maildir/cur/1533376515.M379290P4641.trees,S=783:2, Aug 4 09:55:15 trees dovecot: imap(treesenabled@trees.testing): Error: Corrupted index cache file /var/vmail/trees.testing/treesenabled/Maildir/dovecot.index.cache: Broken physical size for mail UID 1 in mailbox INBOX: read(/var/vmail/trees.testing/treesenabled/Maildir/cur/1533376515.M379290P4641.trees,S=1456,W=748:2,) failed: Cached message size larger than expected (1456 > 728, box=INBOX, UID=1, cached Message-Id=<4ae8ea2c-17ac-37d5-9b9e-b32c6edc777a@trees.testing>) Aug 4 09:55:15 trees dovecot: imap(treesenabled@trees.testing): Panic: file istream.c: line 175 (i_stream_read): assertion failed: (old_size <= _stream->pos - _stream->skip) Aug 4 09:55:15 trees dovecot: imap(treesenabled@trees.testing): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95e92) [0x7fa88e2afe92] -> /usr/lib/dovecot/libdovecot.so.0(+0x95f8d) [0x7fa88e2aff8d] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fa88e245a91] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x290) [0x7fa88e2baf20] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7fa88e2bb72d] -> /usr/lib/dovecot/libdovecot.so.0(message_get_body_size+0xfd) [0x7fa88e29ab6d] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_init_stream+0x220) [0x7fa88e5eb3d0] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x66fc0) [0x7fa88e5a2fc0] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_stream_because+0x5d) [0x7fa88e572e1d] -> /usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x536) [0x7fa88e6264f6] -> dovecot-trees.testing/imap(+0x1ebbc) [0x559b794f1bbc] -> dovecot-trees.testing/imap(+0x1cfb6) [0x559b794effb6] -> dovecot-trees.testing/imap(imap_fetch_more+0x39) [0x559b794f10e9] -> dovecot-trees.testing/imap(cmd_fetch+0x33b) [0x559b794e2d9b] -> dovecot-trees.testing/imap(command_exec+0xa5) [0x559b794ee735] -> dovecot-trees.testing/imap(+0x199c2) [0x559b794ec9c2] -> dovecot-trees.testing/imap(+0x19a4c) [0x559b794eca4c] -> dovecot-trees.testing/imap(client_handle_input+0x1b5) [0x559b794ece55] -> dovecot-trees.testing/imap(client_input+0x86) [0x559b794ed3c6] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7fa88e2c49f2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109) [0x7fa88e2c6029] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7fa88e2c4a8c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7fa88e2c4c38] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7fa88e24bfd3] -> dovecot-trees.testing/imap(main+0x328) [0x559b794dfe68] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7fa88de9b2e1] -> dovecot-trees.testing/imap(_start+0x2a) [0x559b794dffea] Aug 4 09:55:15 trees dovecot: imap(treesenabled@trees.testing): Fatal: master: service(imap): child 4633 killed with signal 6 (core dumps disabled)
The behavior does not occur, when I'm using maildir and access my mail after receiving it. It also does not occur with other mailbox formats such as mbox, sdbox and mdbox
I opened a bug report at the trees repository, which is more detailed ( https://0xacab.org/riseuplabs/trees/issues/24 ). There I described how it's possible to reproduce this error.
In the trees repo there is also a vagrant box, with which it's possible to spin up the exact same setup.
Questions:
1.) Do you know what could possibly lead to the the larger cached message size, when accessing this mail via imap?
2.) Why does these sizes differ from each other? Mail size from the log is: 'S=1456,W=748:2' or 'renamed to ...S=783:2'. But I do not see where the size of '728' come from?
3.) Do you have a hint what could lead to this behavior and how could I solve this error?
Unrelated but also interesting for trees usage: 4.) Is there a tool, with which you can export and import mail to the same inbox?
Background: After activation for an existing mail user, only new outgoing and incoming mail is encrypted. So it would be necessary to "re-save" the existing mail again in this mailbox, to pipe all mails through the encryption process.
I know that there is dsync, imapsync and several different tools, but I'm not sure what is the best tool for the job, or if I should use dsync to do a 3 way sync?
existing_mailbox@server.com > tmp_mailbox@server.com > existing_mailbox@server.com
Thank you:
Thank you for time reading this mail :)
Dovecot Config:
2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
Pigeonhole version 0.4.16 (fed8554)
OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.4 ext4
auth_mechanisms = plain login instance_name = trees.testing login_greeting = trees.testing ready. mail_gid = 5000 mail_location = maildir:/var/vmail/%Ld/%Ln/Maildir mail_plugins = " quota trees" mail_uid = 5000 mailbox_list_index = yes namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe autoexpunge = 30 days special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe autoexpunge = 30 days special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocols = " imap pop3" service auth { unix_listener /var/spool/postfix/private/auth { mode = 0666 } unix_listener auth-userdb { group = vmail mode = 0660 } } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } process_min_avail = 4 } service imap { executable = imap } service pop3 { executable = pop3 } ssl = required ssl_cert = </etc/ssl/certs/ssl-cert-snakeoil.pem ssl_cipher_list = ALL:!LOW:!RC4:!EXP:!aNULL ssl_dh_parameters_length = 2048 ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql }
Follow up:
cmouse from #dovecot pointed out, that this was a dovecot bug which is already fixed in newer dovecot versions. Thank you!
With dovecot 2.2.34 (current debian backports version), I can not reproduce the error.
So my last, "bug unrelated" question remains:
4.) Is there a tool, with which you can export and import mail to the same inbox?
Background: After activation for an existing mail user, only new outgoing and incoming mail is encrypted. So it would be necessary to "re-save" the existing mail again in this mailbox, to pipe all mails through the encryption process.
I know that there is dsync, imapsync and several different tools, but I'm not sure what is the best tool for the job, or if I should use dsync to do a 3 way sync?
existing_mailbox@server.com > tmp_mailbox@server.com > existing_mailbox@server.com
Thanks again to cmouse.
cheers neutron
Op 09/08/2018 om 19:29 schreef neutron:
Follow up:
cmouse from #dovecot pointed out, that this was a dovecot bug which is already fixed in newer dovecot versions. Thank you!
With dovecot 2.2.34 (current debian backports version), I can not reproduce the error.
So my last, "bug unrelated" question remains:
4.) Is there a tool, with which you can export and import mail to the same inbox?
Background: After activation for an existing mail user, only new outgoing and incoming mail is encrypted. So it would be necessary to "re-save" the existing mail again in this mailbox, to pipe all mails through the encryption process.
I know that there is dsync, imapsync and several different tools, but I'm not sure what is the best tool for the job, or if I should use dsync to do a 3 way sync?
Maybe this could help:
https://wiki1.dovecot.org/Plugins/Convert
No idea whether it would work with that plugin though.
Regards,
Stephan.
On Thu, 9 Aug 2018, neutron wrote:
The plugin adds individually encrypted mail storage to the Dovecot IMAP server. It encrypts all (incoming and outgoing) mail on the mail server with a key, which can only be accessed with the users password. So the server operator is not able to see any mail content, if it's saved on the hard disk.
Another privacy plugin that assumes the server operator is unmotivated or respects your privacy anyways, and won't just skim your password right off the top to look at your mail. A vault with steel walls and a dirt floor.
Joseph Tam <jtam.home@gmail.com>
Quoting Joseph Tam <jtam.home@gmail.com>:
Another privacy plugin that assumes the server operator is unmotivated or respects your privacy anyways, and won't just skim your password right off the top to look at your mail. A vault with steel walls and a dirt floor.
*SIGH* As usual, you're right on the money, Joseph.
I used to let things like this "slide", but somewhat recently I've had some clients badgering me to implement something like this. It takes longer than it should to explain how pointless the exercise is.
Given that:
Email transactions, from submission, to delivery, to final reception by a MUA, are done with plaintext contents. Those who want security, will undergo the additional steps and hassles with using PGP to encrypt the contents, providing the only demonstrably secure (against "Evil SysAdmins") means of cloaking your content. The submission, delivery, and final reception is still performed as "plaintext", albeit with an attachment that is encrypted, a process done (and undone) by the ultimate endpoint clients.
Even if the "Evil SysAdmin" doesn't scribble all of the users' passphrases into a log, it's trivial for various tools, many of which were hastily cobbled together during the fad of implementing Sarbanes-Oxley Act (SOX) compliance on mail servers. Tools like "milter-bcc" and friends which automatically clone all email submitted to or arriving through SMTP, etc. It doesn't matter if your SMTP software implements 65,536 Jiggabyte Key Quantum-Computing-Resistant crypto, when it has the decrypted contents in its spool.
I imagine this is an exercise in buzzword collection, and to be seen to be "doing something" to improve security and/or privacy.
If privacy is desired, there are only end-to-end encryption/signature schemes to ensure anything at all, and even there we're at the mercy of mathematical gods greater than we.
Looking to a "magical" oracle on your server to do it for you, whilst keeping all of the leaky, plaintext, and promiscuous protocols (DSN, bounces, intermediate MXer hosts that eruct contents to various envelope addresses, etc) that will betray you behind your back without a moment's notice is a Fool's Errand.
Think it over.
=M=
While this is true, it can be useful to encrypt messages in-rest at 3rd party storage. For end user, only PGP or similar provides sufficient security against admin. ---Aki TuomiDovecot oy -------- Original message --------From: "M. Balridge" <dovecot@r.paypc.com> Date: 11/08/2018 13:56 (GMT+02:00) To: Dovecot Mailing List <dovecot@dovecot.org> Subject: Re: [trees-plugin] - Dovecot index gets corrupted, when using maildir and recievend and accessing mail at the same time Quoting Joseph Tam <jtam.home@gmail.com>:
Another privacy plugin that assumes the server operator is unmotivated or respects your privacy anyways, and won't just skim your password right off the top to look at your mail. A vault with steel walls and a dirt floor.
*SIGH* As usual, you're right on the money, Joseph.
I used to let things like this "slide", but somewhat recently I've had some clients badgering me to implement something like this. It takes longer than it should to explain how pointless the exercise is.
Given that:
Email transactions, from submission, to delivery, to final reception by a MUA, are done with plaintext contents. Those who want security, will undergo the additional steps and hassles with using PGP to encrypt the contents, providing the only demonstrably secure (against "Evil SysAdmins") means of cloaking your content. The submission, delivery, and final reception is still performed as "plaintext", albeit with an attachment that is encrypted, a process done (and undone) by the ultimate endpoint clients.
Even if the "Evil SysAdmin" doesn't scribble all of the users' passphrases into a log, it's trivial for various tools, many of which were hastily cobbled together during the fad of implementing Sarbanes-Oxley Act (SOX) compliance on mail servers. Tools like "milter-bcc" and friends which automatically clone all email submitted to or arriving through SMTP, etc. It doesn't matter if your SMTP software implements 65,536 Jiggabyte Key Quantum-Computing-Resistant crypto, when it has the decrypted contents in its spool.
I imagine this is an exercise in buzzword collection, and to be seen to be "doing something" to improve security and/or privacy.
If privacy is desired, there are only end-to-end encryption/signature schemes to ensure anything at all, and even there we're at the mercy of mathematical gods greater than we.
Looking to a "magical" oracle on your server to do it for you, whilst keeping all of the leaky, plaintext, and promiscuous protocols (DSN, bounces, intermediate MXer hosts that eruct contents to various envelope addresses, etc) that will betray you behind your back without a moment's notice is a Fool's Errand.
Think it over.
=M=
participants (5)
-
Aki Tuomi
-
Joseph Tam
-
M. Balridge
-
neutron
-
Stephan Bosch