[trees-plugin] - Dovecot index gets corrupted, when using maildir and recievend and accessing mail at the same time
neutron
neutron at systemli.org
Thu Aug 9 19:11:29 EEST 2018
Hello dovecot team,
at first I want to thank you the awesome dovecot software.
I'm currently researching and developing the trees plugin
(https://0xacab.org/riseuplabs/trees). In the future I would love to
package this software for debian. I run into a problem, which I can not
fix by myself, that's why I'm reaching out to you.
The plugin is producing a reproducible error, when using dovecots
mailbox format maildir, and receiving and accessing mail at the same
time. I'm searching for additional information to fix that.
I know that it's may not the scope of the mailing list to support
external plugins, but since you are the experts, I thought I should ask
at least.
Trees Plugin:
===================
The plugin adds individually encrypted mail storage to the Dovecot IMAP
server. It encrypts all (incoming and outgoing) mail on the mail server
with a key, which can only be accessed with the users password. So the
server operator is not able to see any mail content, if it's saved on
the hard disk.
Bug Description:
===================
I'm using a postfix dovecot setup with the enabled trees plugin with the
mailboxformat maildir.
When I'm receiving and accessing mail at the same time, the dovecot
index gets corrupted. For accessing those mail I use Thunderbird.
Example corruption:
> ug 4 09:55:15 trees dovecot: imap-login: Login: user=<treesenabled at trees.testing>, method=PLAIN, rip=192.168.22.1, lip=192.168.22.77, mpid=4643, TLS, session=<w0OCCplyhLLAqBYB>
> Aug 4 09:55:15 trees dovecot: imap(treesenabled at trees.testing): Debug: trees plugin initialized
> Aug 4 09:55:15 trees dovecot: imap(treesenabled at trees.testing): Connection closed in=506 out=495
> Aug 4 09:55:15 trees dovecot: imap(treesenabled at trees.testing): Error: Maildir filename has wrong S value, renamed the file from /var/vmail/trees.testing/treesenabled/Maildir/cur/1533376515.M379290P4641.trees,S=1456,W=748:2, to /var/vmail/trees.testing/treesenabled/Maildir/cur/1533376515.M379290P4641.trees,S=783:2,
> Aug 4 09:55:15 trees dovecot: imap(treesenabled at trees.testing): Error: Corrupted index cache file /var/vmail/trees.testing/treesenabled/Maildir/dovecot.index.cache: Broken physical size for mail UID 1 in mailbox INBOX: read(/var/vmail/trees.testing/treesenabled/Maildir/cur/1533376515.M379290P4641.trees,S=1456,W=748:2,) failed: Cached message size larger than expected (1456 > 728, box=INBOX, UID=1, cached Message-Id=<4ae8ea2c-17ac-37d5-9b9e-b32c6edc777a at trees.testing>)
> Aug 4 09:55:15 trees dovecot: imap(treesenabled at trees.testing): Panic: file istream.c: line 175 (i_stream_read): assertion failed: (old_size <= _stream->pos - _stream->skip)
> Aug 4 09:55:15 trees dovecot: imap(treesenabled at trees.testing): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95e92) [0x7fa88e2afe92] -> /usr/lib/dovecot/libdovecot.so.0(+0x95f8d) [0x7fa88e2aff8d] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fa88e245a91] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x290) [0x7fa88e2baf20] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7fa88e2bb72d] -> /usr/lib/dovecot/libdovecot.so.0(message_get_body_size+0xfd) [0x7fa88e29ab6d] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_init_stream+0x220) [0x7fa88e5eb3d0] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x66fc0) [0x7fa88e5a2fc0] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_stream_because+0x5d) [0x7fa88e572e1d] -> /usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x536) [0x7fa88e6264f6] -> dovecot-trees.testing/imap(+0x1ebbc) [0x559b794f1bbc] -> dovecot-trees.testing/imap(+0x1cfb6) [0x559b794effb6] -> dovecot-trees.testing/imap(imap_fetch_more+0x39) [0x559b794f10e9] -> dovecot-trees.testing/imap(cmd_fetch+0x33b) [0x559b794e2d9b] -> dovecot-trees.testing/imap(command_exec+0xa5) [0x559b794ee735] -> dovecot-trees.testing/imap(+0x199c2) [0x559b794ec9c2] -> dovecot-trees.testing/imap(+0x19a4c) [0x559b794eca4c] -> dovecot-trees.testing/imap(client_handle_input+0x1b5) [0x559b794ece55] -> dovecot-trees.testing/imap(client_input+0x86) [0x559b794ed3c6] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7fa88e2c49f2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109) [0x7fa88e2c6029] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7fa88e2c4a8c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7fa88e2c4c38] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7fa88e24bfd3] -> dovecot-trees.testing/imap(main+0x328) [0x559b794dfe68] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7fa88de9b2e1] -> dovecot-trees.testing/imap(_start+0x2a) [0x559b794dffea]
> Aug 4 09:55:15 trees dovecot: imap(treesenabled at trees.testing): Fatal: master: service(imap): child 4633 killed with signal 6 (core dumps disabled)
The behavior does not occur, when I'm using maildir and access my mail
after receiving it. It also does not occur with other mailbox formats
such as mbox, sdbox and mdbox
I opened a bug report at the trees repository, which is more detailed (
https://0xacab.org/riseuplabs/trees/issues/24 ). There I described how
it's possible to reproduce this error.
In the trees repo there is also a vagrant box, with which it's possible
to spin up the exact same setup.
Questions:
===================
1.) Do you know what could possibly lead to the the larger cached
message size, when accessing this mail via imap?
2.) Why does these sizes differ from each other? Mail size from the log
is: 'S=1456,W=748:2' or 'renamed to ...S=783:2'. But I do not see where
the size of '728' come from?
3.) Do you have a hint what could lead to this behavior and how could I
solve this error?
Unrelated but also interesting for trees usage:
4.) Is there a tool, with which you can export and import mail to the
same inbox?
Background: After activation for an existing mail user, only new
outgoing and incoming mail is encrypted. So it would be necessary to
"re-save" the existing mail again in this mailbox, to pipe all mails
through the encryption process.
I know that there is dsync, imapsync and several different tools, but
I'm not sure what is the best tool for the job, or if I should use dsync
to do a 3 way sync?
> existing_mailbox at server.com > tmp_mailbox at server.com > existing_mailbox at server.com
Thank you:
===================
Thank you for time reading this mail :)
Dovecot Config:
===================
# 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.16 (fed8554)
# OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.4 ext4
auth_mechanisms = plain login
instance_name = trees.testing
login_greeting = trees.testing ready.
mail_gid = 5000
mail_location = maildir:/var/vmail/%Ld/%Ln/Maildir
mail_plugins = " quota trees"
mail_uid = 5000
mailbox_list_index = yes
namespace inbox {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
autoexpunge = 30 days
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
auto = subscribe
autoexpunge = 30 days
special_use = \Trash
}
prefix =
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
protocols = " imap pop3"
service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0666
}
unix_listener auth-userdb {
group = vmail
mode = 0660
}
}
service imap-login {
inet_listener imap {
port = 143
}
inet_listener imaps {
port = 993
ssl = yes
}
process_min_avail = 4
}
service imap {
executable = imap
}
service pop3 {
executable = pop3
}
ssl = required
ssl_cert = </etc/ssl/certs/ssl-cert-snakeoil.pem
ssl_cipher_list = ALL:!LOW:!RC4:!EXP:!aNULL
ssl_dh_parameters_length = 2048
ssl_key = # hidden, use -P to show it
ssl_prefer_server_ciphers = yes
userdb {
driver = prefetch
}
userdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://dovecot.org/pipermail/dovecot/attachments/20180809/522f9cb9/attachment.sig>
More information about the dovecot
mailing list