[trees-plugin] - Dovecot index gets corrupted, when using maildir and recievend and accessing mail at the same time

9 Aug 2018


      Hello dovecot team,
at first I want to thank you the awesome dovecot software.
I'm currently researching and developing the trees plugin
(https://0xacab.org/riseuplabs/trees). In the future I would love to
package this software for debian. I run into a problem, which I can not
fix by myself, that's why I'm reaching out to you.
The plugin is producing a reproducible error, when using dovecots
mailbox format maildir, and receiving and accessing mail at the same
time. I'm searching for additional information to fix that.
I know that it's may not the scope of the mailing list to support
external plugins, but since you are the experts, I thought I should ask
at least.
Trees Plugin:
The plugin adds individually encrypted mail storage to the Dovecot IMAP
server. It encrypts all (incoming and outgoing) mail on the mail server
with a key, which can only be accessed with the users password. So the
server operator is not able to see any mail content, if it's saved on
the hard disk.
Bug Description:
I'm using a postfix dovecot setup with the enabled trees plugin with the
mailboxformat maildir.
When I'm receiving and accessing mail at the same time, the dovecot
index gets corrupted. For accessing those mail I use Thunderbird.
Example corruption:
...
ug  4 09:55:15 trees dovecot: imap-login: Login: user=treesenabled@trees.testing, method=PLAIN, rip=192.168.22.1, lip=192.168.22.77, mpid=4643, TLS, session=<w0OCCplyhLLAqBYB>
Aug  4 09:55:15 trees dovecot: imap(treesenabled@trees.testing): Debug: trees plugin initialized
Aug  4 09:55:15 trees dovecot: imap(treesenabled@trees.testing): Connection closed in=506 out=495
Aug  4 09:55:15 trees dovecot: imap(treesenabled@trees.testing): Error: Maildir filename has wrong S value, renamed the file from /var/vmail/trees.testing/treesenabled/Maildir/cur/1533376515.M379290P4641.trees,S=1456,W=748:2, to /var/vmail/trees.testing/treesenabled/Maildir/cur/1533376515.M379290P4641.trees,S=783:2,
Aug  4 09:55:15 trees dovecot: imap(treesenabled@trees.testing): Error: Corrupted index cache file /var/vmail/trees.testing/treesenabled/Maildir/dovecot.index.cache: Broken physical size for mail UID 1 in mailbox INBOX: read(/var/vmail/trees.testing/treesenabled/Maildir/cur/1533376515.M379290P4641.trees,S=1456,W=748:2,) failed: Cached message size larger than expected (1456 > 728, box=INBOX, UID=1, cached Message-Id=4ae8ea2c-17ac-37d5-9b9e-b32c6edc777a@trees.testing)
Aug  4 09:55:15 trees dovecot: imap(treesenabled@trees.testing): Panic: file istream.c: line 175 (i_stream_read): assertion failed: (old_size <= _stream->pos - _stream->skip)
Aug  4 09:55:15 trees dovecot: imap(treesenabled@trees.testing): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95e92) [0x7fa88e2afe92] -> /usr/lib/dovecot/libdovecot.so.0(+0x95f8d) [0x7fa88e2aff8d] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fa88e245a91] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x290) [0x7fa88e2baf20] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7fa88e2bb72d] -> /usr/lib/dovecot/libdovecot.so.0(message_get_body_size+0xfd) [0x7fa88e29ab6d] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_init_stream+0x220) [0x7fa88e5eb3d0] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x66fc0) [0x7fa88e5a2fc0] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_stream_because+0x5d) [0x7fa88e572e1d] -> /usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x536) [0x7fa88e6264f6] -> dovecot-trees.testing/imap(+0x1ebbc) [0x559b794f1bbc] -> dovecot-trees.testing/imap(+0x1cfb6) [0x559b794effb6] -> dovecot-trees.testing/imap(imap_fetch_more+0x39) [0x559b794f10e9] -> dovecot-trees.testing/imap(cmd_fetch+0x33b) [0x559b794e2d9b] -> dovecot-trees.testing/imap(command_exec+0xa5) [0x559b794ee735] -> dovecot-trees.testing/imap(+0x199c2) [0x559b794ec9c2] -> dovecot-trees.testing/imap(+0x19a4c) [0x559b794eca4c] -> dovecot-trees.testing/imap(client_handle_input+0x1b5) [0x559b794ece55] -> dovecot-trees.testing/imap(client_input+0x86) [0x559b794ed3c6] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7fa88e2c49f2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109) [0x7fa88e2c6029] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7fa88e2c4a8c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7fa88e2c4c38] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7fa88e24bfd3] -> dovecot-trees.testing/imap(main+0x328) [0x559b794dfe68] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7fa88de9b2e1] -> dovecot-trees.testing/imap(_start+0x2a) [0x559b794dffea]
Aug  4 09:55:15 trees dovecot: imap(treesenabled@trees.testing): Fatal: master: service(imap): child 4633 killed with signal 6 (core dumps disabled)
The behavior does not occur, when I'm using maildir and access my mail
after receiving it. It also does not occur with other mailbox formats
such as mbox, sdbox and mdbox
I opened a bug report at the trees repository, which is more detailed (
https://0xacab.org/riseuplabs/trees/issues/24 ). There I described how
it's possible to reproduce this error.
In the trees repo there is also a vagrant box, with which it's possible
to spin up the exact same setup.
Questions:
1.) Do you know what could possibly lead to the the larger cached
message size, when accessing this mail via imap?
2.) Why does these sizes differ from each other? Mail size from the log
is: 'S=1456,W=748:2' or 'renamed to ...S=783:2'. But I do not see where
the size of '728' come from?
3.) Do you have a hint what could lead to this behavior and how could I
solve this error?
Unrelated but also interesting for trees usage:
4.) Is there a tool, with which you can export and import mail to the
same inbox?
Background: After activation for an existing mail user, only new
outgoing and incoming mail is encrypted. So it would be necessary to
"re-save" the existing mail again in this mailbox, to pipe all mails
through the encryption process.
I know that there is dsync, imapsync and several different tools, but
I'm not sure what is the best tool for the job, or if I should use dsync
to do a 3 way sync?
...
existing_mailbox@server.com  > tmp_mailbox@server.com > existing_mailbox@server.com
Thank you:
Thank you for time reading this mail :)
Dovecot Config:
# 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.16 (fed8554)
# OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.4 ext4
auth_mechanisms = plain login
instance_name = trees.testing
login_greeting = trees.testing ready.
mail_gid = 5000
mail_location = maildir:/var/vmail/%Ld/%Ln/Maildir
mail_plugins = " quota trees"
mail_uid = 5000
mailbox_list_index = yes
namespace inbox {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
autoexpunge = 30 days
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
auto = subscribe
autoexpunge = 30 days
special_use = \Trash
}
prefix =
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
protocols = " imap pop3"
service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0666
}
unix_listener auth-userdb {
group = vmail
mode = 0660
}
}
service imap-login {
inet_listener imap {
port = 143
}
inet_listener imaps {
port = 993
ssl = yes
}
process_min_avail = 4
}
service imap {
executable = imap
}
service pop3 {
executable = pop3
}
ssl = required
ssl_cert =

[trees-plugin] - Dovecot index gets corrupted, when using maildir and recievend and accessing mail at the same time

neutron

Trees Plugin:

Bug Description:

Questions:

Thank you:

Dovecot Config: