August 2018 - dovecot

Re: [trees-plugin] - Dovecot index gets corrupted,^M when using maildir and recievend and accessing mail at the same time
by Joseph Tam 15 Aug '18

15 Aug '18

On Sat, 11 Aug 2018, Aki Tuomi wrote: > While this is true, it can be useful to encrypt messages in-rest at 3rd > party storage. Sure, in this case, mail operator != storage operator. It's not apparent those who clamour for it know the difference. > For end user, only PGP or similar provides sufficient security against admin. For content. Metadata can also reveal a lot. If the goal is to ensure privacy using a third-party storage provider (and assuming mail handlers are implictly trusted), it seems more productive to invest effort into block/FS level encryption that comprehensively protects all data. Joseph Tam <jtam.home(a)gmail.com>

1 0

doveadm mailbox delete not working
by Ralf Becker 15 Aug '18

15 Aug '18

I have a user who has several folders in his mailbox, which we can not delete, neither via IMAP nor via doveadm: root@ka-nfs-mail:~# doveadm mailbox list -u <username> | grep hbereiche | cat -v INBOX/[Fachbereiche ^M INBOX/Fachbereiche ^M INBOX/hbereiche^M INBOX/hbereiche/LAGen]^M INBOX/hbereiche/LAG^M INBOX/[Fachbereiche^M INBOX/[Fachbereiche/LAGen]^M INBOX/[Fachbereiche]^M INBOX/[Fachbereiche]/LAGen]^M INBOX/[Fachbereiche]/LAGe^M root@ka-nfs-mail:~# doveadm mailbox delete -u <username> 'INBOX/Fachbereiche ' doveadm(<username>): Info: Mailbox deleted: INBOX/Fachbereiche root@ka-nfs-mail:~# doveadm mailbox list -u <username>| grep hbereiche | cat -v INBOX/[Fachbereiche ^M INBOX/Fachbereiche ^M INBOX/hbereiche^M INBOX/hbereiche/LAGen]^M INBOX/hbereiche/LAG^M INBOX/[Fachbereiche^M INBOX/[Fachbereiche/LAGen]^M INBOX/[Fachbereiche]^M INBOX/[Fachbereiche]/LAGen]^M INBOX/[Fachbereiche]/LAGe^M As far as I tried none of these folders can be deleted (I added single quotes for trailing space and tried to delete subfolders first). Mailbox is in mdbox format on a replication pair under Dovecot 2.2.36 and I tried both nodes of the replication with same result. Any ideas? Ralf -- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0

3 6

error while compiling dovecot
by Rajesh M 15 Aug '18

15 Aug '18

hello my server is centos 6 64 bit while installing dovecot i get this error Failed to run: ./test-compression what could be the reason. could someone guide me please Version dovecot-ce-2.3.0-1 0 / 7 tests failed ==21130== Conditional jump or move depends on uninitialised value(s) ==21130== at 0x349B00ACB2: ??? (in /usr/lib64/liblzma.so.0.0.0) ==21130== by 0x349B00AEA9: ??? (in /usr/lib64/liblzma.so.0.0.0) ==21130== by 0x349B003121: ??? (in /usr/lib64/liblzma.so.0.0.0) ==21130== by 0x349B0061A0: ??? (in /usr/lib64/liblzma.so.0.0.0) ==21130== by 0x349B005212: lzma_easy_encoder (in /usr/lib64/liblzma.so.0.0.0) ==21130== by 0x406ACF: o_stream_create_lzma (ostream-lzma.c:206) ==21130== by 0x403D89: test_compression_handler (test-compression.c:36) ==21130== by 0x4040BD: test_compression (test-compression.c:96) ==21130== by 0x408281: test_run_funcs (test-common.c:244) ==21130== by 0x4083E0: test_run (test-common.c:315) ==21130== by 0x40398B: main (test-compression.c:308) ==21130== Failed to run: ./test-compression make[3]: *** [check-local] Error 1 make[3]: Leaving directory `/root/rpmbuild/BUILD/dovecot-ce-2.3.0/src/lib-compression' make[2]: *** [check-am] Error 2 make[2]: Leaving directory `/root/rpmbuild/BUILD/dovecot-ce-2.3.0/src/lib-compression' make[1]: *** [check-recursive] Error 1 make[1]: Leaving directory `/root/rpmbuild/BUILD/dovecot-ce-2.3.0/src' make: *** [check-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.LCwYD9 (%check) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.LCwYD9 (%check) thanks, rajesh

2 1

Confusing search / caps - non caps different results
by Götz Reinicke 15 Aug '18

15 Aug '18

Hi, I have some troubles with searching and could not find an answer yet. The search term is „istop“ in the subject. The client was SOGo. We run the recent dovecot, with solr backend and milder with zlib compression. Searching for all small letters „istop“ shows up no hit. Searching with a capital „S“ „iStop“ shows all messages back 10 years ago. Sprachig for „istopmotion“ all messages show up, wether the subject contains caps or non caps at all. I did a telnet to our dovecot and searched by hand for all three terms: 8 search Subject istop * SEARCH 8 OK Search completed (0.025 + 0.000 + 0.022 secs). 9 search subject iStop * SEARCH 544 604 654 662 910 1394 1471 1474 5493 9 OK Search completed (0.028 + 0.000 + 0.023 secs). 10 search subject istopmotion * SEARCH 544 604 654 662 910 1471 1472 1474 5493 10 OK Search completed (0.027 + 0.000 + 0.023 secs). Any hints or suggestion is welcome as I m very confused and lost. Thanks and regards . Götz

1 0

Re: doveadm mailbox delete not working
by Ralf Becker 15 Aug '18

15 Aug '18

Hi Aki, thanks for looking into this :) Am 14.08.18 um 15:15 schrieb Aki Tuomi: > can you turn on mail_debug=yes and run doveadm -Dv mailbox delete and > provide output and logs from both servers? root@ka-nfs-mail:~# doveadm -Dv mailbox delete -u h 'INBOX/Fachbereiche ' Debug: Loading modules from directory: /usr/lib/dovecot Debug: Module loaded: /usr/lib/dovecot/lib01_acl_plugin.so Debug: Module loaded: /usr/lib/dovecot/lib10_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/lib15_notify_plugin.so Debug: Module loaded: /usr/lib/dovecot/lib20_mail_log_plugin.so Debug: Module loaded: /usr/lib/dovecot/lib20_replication_plugin.so Debug: Loading modules from directory: /usr/lib/dovecot/doveadm Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: expire_set_lookup: symbol not found (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: fts_backend_rescan: symbol not found (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: mail_crypt_box_get_public_key: symbol not found (this is usually intentional, so just ignore this message) doveadm(<username>): Debug: auth PASS input: doveadm(<username> 32679): Debug: auth USER input: <username> userdb_quota_rule=*:bytes=15728640000 master_user=<username> userdb_acl_groups=koakram@<domain>,wahlkampfnetzwerk@<domain>,wahlkalender 2017@<domain>,lgs@<domain> home=/var/dovecot/imap/<domain>/<user> doveadm(<username> 32679): Debug: Added userdb setting: plugin/master_user=<username> doveadm(<username> 32679): Debug: Added userdb setting: plugin/userdb_acl_groups=koakram@<domain>,wahlkampfnetzwerk@<domain>,wahlkalender 2017@<domain>,lgs@<domain> doveadm(<username> 32679): Debug: Added userdb setting: plugin/userdb_quota_rule=*:bytes=15728640000 doveadm(<username>): Debug: Effective uid=90, gid=101, home=/var/dovecot/imap/<domain>/<user> doveadm(<username>): Debug: Quota root: name=User quota backend=dict args=:ns=INBOX/:file:/var/dovecot/imap/<domain>/<user>/dovecot-quota doveadm(<username>): Debug: Quota rule: root=User quota mailbox=* bytes=107374182400 messages=0 doveadm(<username>): Debug: Quota grace: root=User quota bytes=10737418240 (10%) doveadm(<username>): Debug: dict quota: user=<username>, uri=file:/var/dovecot/imap/<domain>/<user>/dovecot-quota, noenforcing=0 doveadm(<username>): Debug: Namespace inboxes: type=private, prefix=INBOX/, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=no location=mdbox:~/mdbox doveadm(<username>): Debug: fs: root=/var/dovecot/imap/<domain>/<user>/mdbox, index=, indexpvt=, control=, inbox=, alt= doveadm(<username>): Debug: acl: initializing backend with data: vfile doveadm(<username>): Debug: acl: acl username = <username> doveadm(<username>): Debug: acl: owner = 1 doveadm(<username>): Debug: acl vfile: Global ACLs disabled doveadm(<username>): Debug: Namespace users: type=shared, prefix=user/%n/, sep=/, inbox=no, hidden=no, list=yes, subscriptions=no location=mdbox:%h/mdbox:INDEXPVT=~/shared/%u doveadm(<username>): Debug: shared: root=/run/dovecot, index=, indexpvt=, control=, inbox=, alt= doveadm(<username>): Debug: acl: initializing backend with data: vfile doveadm(<username>): Debug: acl: acl username = <username> doveadm(<username>): Debug: acl: owner = 0 doveadm(<username>): Debug: acl vfile: Global ACLs disabled doveadm(<username>): Debug: Namespace subs: type=private, prefix=, sep=/, inbox=no, hidden=yes, list=no, subscriptions=yes location=mdbox:~/mdbox doveadm(<username>): Debug: fs: root=/var/dovecot/imap/<domain>/<user>/mdbox, index=, indexpvt=, control=, inbox=, alt= doveadm(<username>): Debug: acl: initializing backend with data: vfile doveadm(<username>): Debug: acl: acl username = <username> doveadm(<username>): Debug: acl: owner = 1 doveadm(<username>): Debug: acl vfile: Global ACLs disabled doveadm(<username>): Debug: quota: quota_over_flag check: quota_over_script unset - skipping doveadm(<username>): Debug: INBOX/Fachbereiche : Mailbox opened because: mailbox delete doveadm(<username>): Debug: acl vfile: file /var/dovecot/imap/<domain>/<user>/mdbox/mailboxes/Fachbereiche /dbox-Mails/dovecot-acl not found doveadm(<username>): Debug: Namespace INBOX/: Using permissions from /var/dovecot/imap/<domain>/<user>/mdbox: mode=0700 gid=default doveadm(<username>): Debug: replication: Replication requested by 'mailbox delete', priority=1 doveadm(<username>): Info: Mailbox deleted: INBOX/Fachbereiche Output and logs are from the (less loaded) standby/backup node. I can get the logs from the active node tonight. I had to remove some folder-names for privacy reasons, but they all have the same output in the logs. Ralf > > > > --- > Aki Tuomi > Dovecot oy > > -------- Original message -------- > From: Ralf Becker <rb(a)egroupware.org> > Date: 14/08/2018 16:13 (GMT+02:00) > To: dovecot(a)dovecot.org > Subject: doveadm mailbox delete not working > > I have a user who has several folders in his mailbox, which we can not > delete, neither via IMAP nor via doveadm: > > root@ka-nfs-mail:~# doveadm mailbox list -u <username> | grep hbereiche > | cat -v > INBOX/[Fachbereiche ^M > INBOX/Fachbereiche ^M > INBOX/hbereiche^M > INBOX/hbereiche/LAGen]^M > INBOX/hbereiche/LAG^M > INBOX/[Fachbereiche^M > INBOX/[Fachbereiche/LAGen]^M > INBOX/[Fachbereiche]^M > INBOX/[Fachbereiche]/LAGen]^M > INBOX/[Fachbereiche]/LAGe^M > root@ka-nfs-mail:~# doveadm mailbox delete -u <username> > 'INBOX/Fachbereiche ' > doveadm(<username>): Info: Mailbox deleted: INBOX/Fachbereiche > root@ka-nfs-mail:~# doveadm mailbox list -u <username>| grep hbereiche | > cat -v > INBOX/[Fachbereiche ^M > INBOX/Fachbereiche ^M > INBOX/hbereiche^M > INBOX/hbereiche/LAGen]^M > INBOX/hbereiche/LAG^M > INBOX/[Fachbereiche^M > INBOX/[Fachbereiche/LAGen]^M > INBOX/[Fachbereiche]^M > INBOX/[Fachbereiche]/LAGen]^M > INBOX/[Fachbereiche]/LAGe^M > > As far as I tried none of these folders can be deleted (I added single > quotes for trailing space and tried to delete subfolders first). > > Mailbox is in mdbox format on a replication pair under Dovecot 2.2.36 > and I tried both nodes of the replication with same result. > > Any ideas? > > Ralf > > -- > Ralf Becker > EGroupware GmbH [www.egroupware.org] > Handelsregister HRB Kaiserslautern 3587 > Geschäftsführer Birgit und Ralf Becker > Leibnizstr. 17, 67663 Kaiserslautern, Germany > Telefon +49 631 31657-0 > > -- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0

3 7

Re: doveadm mailbox delete not working
by Ralf Becker 15 Aug '18

15 Aug '18

Hi Aki, Am 15.08.18 um 11:31 schrieb Aki Tuomi: > Such fun folders. Can you try doveadm -Dv mailbox delete -u username > folder and post it to the list? root@ka-nfs-mail:~# doveadm -Dv mailbox delete -u <username> 'INBOX/Fachbereiche ' Debug: Loading modules from directory: /usr/lib/dovecot Debug: Module loaded: /usr/lib/dovecot/lib01_acl_plugin.so Debug: Module loaded: /usr/lib/dovecot/lib10_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/lib15_notify_plugin.so Debug: Module loaded: /usr/lib/dovecot/lib20_mail_log_plugin.so Debug: Module loaded: /usr/lib/dovecot/lib20_replication_plugin.so Debug: Loading modules from directory: /usr/lib/dovecot/doveadm Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: expire_set_lookup: symbol not found (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: fts_backend_rescan: symbol not found (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: mail_crypt_box_get_public_key: symbol not found (this is usually intentional, so just ignore this message) doveadm(<username> 46922): Debug: Added userdb setting: plugin/master_user=<username> doveadm(<username> 46922): Debug: Added userdb setting: plugin/userdb_acl_groups=koakram@<domain>,wahlkampfnetzwerk@<domain>,wahlkalender 2017@<domain>,lgs@<domain> doveadm(<username> 46922): Debug: Added userdb setting: plugin/userdb_quota_rule=*:bytes=15728640000 doveadm(<username>): Debug: Effective uid=90, gid=101, home=/var/dovecot/imap/<domain>/<user> doveadm(<username>): Debug: Quota root: name=User quota backend=dict args=:ns=INBOX/:file:/var/dovecot/imap/<domain>/<user>/dovecot-quota doveadm(<username>): Debug: Quota rule: root=User quota mailbox=* bytes=107374182400 messages=0 doveadm(<username>): Debug: Quota grace: root=User quota bytes=10737418240 (10%) doveadm(<username>): Debug: dict quota: user=<username>, uri=file:/var/dovecot/imap/<domain>/<user>/dovecot-quota, noenforcing=0 doveadm(<username>): Debug: Namespace inboxes: type=private, prefix=INBOX/, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=no location=mdbox:~/mdbox doveadm(<username>): Debug: fs: root=/var/dovecot/imap/<domain>/<user>/mdbox, index=, indexpvt=, control=, inbox=, alt= doveadm(<username>): Debug: acl: initializing backend with data: vfile doveadm(<username>): Debug: acl: acl username = <username> doveadm(<username>): Debug: acl: owner = 1 doveadm(<username>): Debug: acl vfile: Global ACLs disabled doveadm(<username>): Debug: Namespace users: type=shared, prefix=user/%n/, sep=/, inbox=no, hidden=no, list=yes, subscriptions=no location=mdbox:%h/mdbox:INDEXPVT=~/shared/%u doveadm(<username>): Debug: shared: root=/run/dovecot, index=, indexpvt=, control=, inbox=, alt= doveadm(<username>): Debug: acl: initializing backend with data: vfile doveadm(<username>): Debug: acl: acl username = <username> doveadm(<username>): Debug: acl: owner = 0 doveadm(<username>): Debug: acl vfile: Global ACLs disabled doveadm(<username>): Debug: Namespace subs: type=private, prefix=, sep=/, inbox=no, hidden=yes, list=no, subscriptions=yes location=mdbox:~/mdbox doveadm(<username>): Debug: fs: root=/var/dovecot/imap/<domain>/<user>/mdbox, index=, indexpvt=, control=, inbox=, alt= doveadm(<username>): Debug: acl: initializing backend with data: vfile doveadm(<username>): Debug: acl: acl username = <username> doveadm(<username>): Debug: acl: owner = 1 doveadm(<username>): Debug: acl vfile: Global ACLs disabled doveadm(<username>): Debug: quota: quota_over_flag check: quota_over_script unset - skipping doveadm(<username>): Debug: INBOX/Fachbereiche : Mailbox opened because: mailbox delete doveadm(<username>): Debug: acl vfile: file /var/dovecot/imap/<domain>/<user>/mdbox/mailboxes/Fachbereiche /dbox-Mails/dovecot-acl not found doveadm(<username>): Debug: Namespace INBOX/: Using permissions from /var/dovecot/imap/<domain>/<user>/mdbox: mode=0700 gid=default doveadm(<username>): Debug: replication: Replication requested by 'mailbox delete', priority=1 doveadm(<username>): Info: Mailbox deleted: INBOX/Fachbereiche Event doveadm reporting the folder is deleted, it is not: root@ka-nfs-mail:~# doveadm mailbox list -u <username> 'INBOX/Fachbereiche ' INBOX/Fachbereiche But it can not get a status of it: root@ka-nfs-mail:~# doveadm mailbox status -u <username> all 'INBOX/Fachbereiche ' doveadm(<username>): Error: Mailbox INBOX/Fachbereiche : Failed to lookup mailbox status: Mailbox doesn't exist: INBOX/Fachbereiche Really wired Ralf > Aki > > On 15.08.2018 11:03, Ralf Becker wrote: >> Hi Aki, >> >> I respond to you only on purpose, as the a dont want to show all folders >> public, please remove them when you reply to the list. >> >> This is the folder my previous examples where about: >> >> "mailbox": "INBOX/[Fachbereiche " <-- trailing space >> >> >> Am 15.08.18 um 08:33 schrieb Aki Tuomi: >>> Can you try >>> >>> doveadm -fjson mailbox list -u user INBOX/* -- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0

1 0

Recommendations for backup methods
by Daniel Miller 14 Aug '18

14 Aug '18

I've been re-thinking my backup strategy - I wanted to see what input others have. At this time - I'm using sdbox as the primary storage format and running on a single server. Previously, all my backups were simple filesystem backups. Either inotify-based or cron-based. The whole mail folder structure would then be copied to a remote storage site. I've changed services - I now use OpenDrive for the remote storage and can access the remote drive as a mounted folder via WebDAV. I also now use a cron-based series of restic jobs for the backup processing. Because I have some large folders - both personal & business letters going back quite some time as well as mailing list archives - some mail folders can be quite sizable. Now I'm wondering, between sdbox's support for alternate locations and the ability to use different storage formats via namespaces - perhaps I should setup archive folders that are stored differently to improve backup performance. Maybe maildir or even mbox for the archive folders - I create one per year? Any comments? Experiences? -- Daniel

2 1

Move messages with doveadm recursively.
by Хлебалов Степан 14 Aug '18

14 Aug '18

Hello. I can move messages with doveadm like this: doveadm move -A Archive mailbox INBOX/* before 2018-02-01 but if inbox has a subfolders then subfolders structure does not move to Archive only messages. Is there a way to move messages to the same folder in new destination? Thanks.

2 1

"For end user, only PGP or similar provides sufficient security against admin." (was: [trees-plugin] - Dovecot index gets corrupted,^M when using maildir and recievend and accessing mail at the same time)
by Steffen Kaiser 13 Aug '18

13 Aug '18

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sat, 11 Aug 2018, Aki Tuomi wrote: > While this is true, it can be useful to encrypt messages in-rest at 3rd party storage. > For end user, only PGP or similar provides sufficient security against admin. Nice, short, pinpointed words I will file away for upcoming discussions. And I will file M's response, too, for the management. Thanks both of you. > -------- Original message --------From: "M. Balridge" <dovecot(a)r.paypc.com> Date: 11/08/2018 13:56 (GMT+02:00) To: Dovecot Mailing List <dovecot(a)dovecot.org> Subject: Re: [trees-plugin] - Dovecot index gets corrupted, > when using maildir and recievend and accessing mail at the same time > Quoting Joseph Tam <jtam.home(a)gmail.com>: > >> Another privacy plugin that assumes the server operator is unmotivated or >> respects your privacy anyways, and won't just skim your password right off >> the top to look at your mail. A vault with steel walls and a dirt floor. > > *SIGH* As usual, you're right on the money, Joseph. > > I used to let things like this "slide", but somewhat recently I've had some > clients badgering me to implement something like this. It takes longer than it > should to explain how pointless the exercise is. > > Given that: > > 1) Email transactions, from submission, to delivery, to final reception by a > MUA, are done with plaintext contents. Those who want security, will undergo > the additional steps and hassles with using PGP to encrypt the contents, > providing the only demonstrably secure (against "Evil SysAdmins") means of > cloaking your content. The submission, delivery, and final reception is still > performed as "plaintext", albeit with an attachment that is encrypted, a > process done (and undone) by the ultimate endpoint clients. > > 2) Even if the "Evil SysAdmin" doesn't scribble all of the users' passphrases > into a log, it's trivial for various tools, many of which were hastily cobbled > together during the fad of implementing Sarbanes-Oxley Act (SOX) compliance on > mail servers. Tools like "milter-bcc" and friends which automatically clone > all email submitted to or arriving through SMTP, etc. It doesn't matter if > your SMTP software implements 65,536 Jiggabyte Key Quantum-Computing-Resistant > crypto, when it has the decrypted contents in its spool. > > I imagine this is an exercise in buzzword collection, and to be seen to be > "doing something" to improve security and/or privacy. > > If privacy is desired, there are only end-to-end encryption/signature schemes > to ensure anything at all, and even there we're at the mercy of mathematical > gods greater than we. > > Looking to a "magical" oracle on your server to do it for you, whilst keeping > all of the leaky, plaintext, and promiscuous protocols (DSN, bounces, > intermediate MXer hosts that eruct contents to various envelope addresses, > etc) that will betray you behind your back without a moment's notice is a > Fool's Errand. > > Think it over. > > =M= > > - -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEVAwUBW3ErEMQnQQNheMxiAQJZ+Qf9ECwe0SZXwClaM+wHBVdsOPLPuL6rkSzV TAkPe7bV2jnqUL8J0I7F46MW4yV76ttbWMbZ3wP6Mom2roNOqGoQIxWsQLkgZvib Wdg29L0nsMkHY6A5zCRM/n4rvNi/xDHIUWIinZRUWvFr8J6WWkSaYneX2Xjvf6tF 24nj+tqcuYtFomsY802WySgovLZi5y0s8nSSkQ9nnPA44hpozfbQXXf/pO14D2BL vhsiqvLKnS/3wY83Y05RLCsojfQDG3Vbqgm6qV9qkpOtGN9sLV/ufXc8tui070UW FDmV5S/KnP8Z7ru9Hq83JEhxkaApPhcKqIQcpjUIeWyobIwvYr718A== =cbbG -----END PGP SIGNATURE-----

1 0

[trees-plugin] - Dovecot index gets corrupted, when using maildir and recievend and accessing mail at the same time
by neutron 11 Aug '18

11 Aug '18

Hello dovecot team, at first I want to thank you the awesome dovecot software. I'm currently researching and developing the trees plugin (https://0xacab.org/riseuplabs/trees) In the future I would love to package this software for debian. I run into a problem, which I can not fix by myself, that's why I'm reaching out to you. The plugin is producing a reproducible error, when using dovecots mailbox format maildir, and receiving and accessing mail at the same time. I'm searching for additional information to fix that. I know that it's may not the scope of the mailing list to support external plugins, but since you are the experts, I thought I should ask at least. Trees Plugin: =================== The plugin adds individually encrypted mail storage to the Dovecot IMAP server. It encrypts all (incoming and outgoing) mail on the mail server with a key, which can only be accessed with the users password. So the server operator is not able to see any mail content, if it's saved on the hard disk. Bug Description: =================== I'm using a postfix dovecot setup with the enabled trees plugin with the mailboxformat maildir. When I'm receiving and accessing mail at the same time, the dovecot index gets corrupted. For accessing those mail I use Thunderbird. Example corruption: > ug 4 09:55:15 trees dovecot: imap-login: Login: user=<treesenabled(a)trees.testing>, method=PLAIN, rip=192.168.22.1, lip=192.168.22.77, mpid=4643, TLS, session=<w0OCCplyhLLAqBYB> > Aug 4 09:55:15 trees dovecot: imap(treesenabled(a)trees.testing): Debug: trees plugin initialized > Aug 4 09:55:15 trees dovecot: imap(treesenabled(a)trees.testing): Connection closed in=506 out=495 > Aug 4 09:55:15 trees dovecot: imap(treesenabled(a)trees.testing): Error: Maildir filename has wrong S value, renamed the file from /var/vmail/trees.testing/treesenabled/Maildir/cur/1533376515.M379290P4641.trees,S=1456,W=748:2, to /var/vmail/trees.testing/treesenabled/Maildir/cur/1533376515.M379290P4641.trees,S=783:2, > Aug 4 09:55:15 trees dovecot: imap(treesenabled(a)trees.testing): Error: Corrupted index cache file /var/vmail/trees.testing/treesenabled/Maildir/dovecot.index.cache: Broken physical size for mail UID 1 in mailbox INBOX: read(/var/vmail/trees.testing/treesenabled/Maildir/cur/1533376515.M379290P4641.trees,S=1456,W=748:2,) failed: Cached message size larger than expected (1456 > 728, box=INBOX, UID=1, cached Message-Id=<4ae8ea2c-17ac-37d5-9b9e-b32c6edc777a(a)trees.testing>) > Aug 4 09:55:15 trees dovecot: imap(treesenabled(a)trees.testing): Panic: file istream.c: line 175 (i_stream_read): assertion failed: (old_size <= _stream->pos - _stream->skip) > Aug 4 09:55:15 trees dovecot: imap(treesenabled(a)trees.testing): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95e92) [0x7fa88e2afe92] -> /usr/lib/dovecot/libdovecot.so.0(+0x95f8d) [0x7fa88e2aff8d] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fa88e245a91] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x290) [0x7fa88e2baf20] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7fa88e2bb72d] -> /usr/lib/dovecot/libdovecot.so.0(message_get_body_size+0xfd) [0x7fa88e29ab6d] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_init_stream+0x220) [0x7fa88e5eb3d0] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x66fc0) [0x7fa88e5a2fc0] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_stream_because+0x5d) [0x7fa88e572e1d] -> /usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x536) [0x7fa88e6264f6] -> dovecot-trees.testing/imap(+0x1ebbc) [0x559b794f1bbc] -> dovecot-trees.testing/imap(+0x1cfb6) [0x559b794effb6] -> dovecot-trees.testing/imap(imap_fetch_more+0x39) [0x559b794f10e9] -> dovecot-trees.testing/imap(cmd_fetch+0x33b) [0x559b794e2d9b] -> dovecot-trees.testing/imap(command_exec+0xa5) [0x559b794ee735] -> dovecot-trees.testing/imap(+0x199c2) [0x559b794ec9c2] -> dovecot-trees.testing/imap(+0x19a4c) [0x559b794eca4c] -> dovecot-trees.testing/imap(client_handle_input+0x1b5) [0x559b794ece55] -> dovecot-trees.testing/imap(client_input+0x86) [0x559b794ed3c6] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7fa88e2c49f2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109) [0x7fa88e2c6029] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7fa88e2c4a8c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7fa88e2c4c38] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7fa88e24bfd3] -> dovecot-trees.testing/imap(main+0x328) [0x559b794dfe68] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7fa88de9b2e1] -> dovecot-trees.testing/imap(_start+0x2a) [0x559b794dffea] > Aug 4 09:55:15 trees dovecot: imap(treesenabled(a)trees.testing): Fatal: master: service(imap): child 4633 killed with signal 6 (core dumps disabled) The behavior does not occur, when I'm using maildir and access my mail after receiving it. It also does not occur with other mailbox formats such as mbox, sdbox and mdbox I opened a bug report at the trees repository, which is more detailed ( https://0xacab.org/riseuplabs/trees/issues/24 ). There I described how it's possible to reproduce this error. In the trees repo there is also a vagrant box, with which it's possible to spin up the exact same setup. Questions: =================== 1.) Do you know what could possibly lead to the the larger cached message size, when accessing this mail via imap? 2.) Why does these sizes differ from each other? Mail size from the log is: 'S=1456,W=748:2' or 'renamed to ...S=783:2'. But I do not see where the size of '728' come from? 3.) Do you have a hint what could lead to this behavior and how could I solve this error? Unrelated but also interesting for trees usage: 4.) Is there a tool, with which you can export and import mail to the same inbox? Background: After activation for an existing mail user, only new outgoing and incoming mail is encrypted. So it would be necessary to "re-save" the existing mail again in this mailbox, to pipe all mails through the encryption process. I know that there is dsync, imapsync and several different tools, but I'm not sure what is the best tool for the job, or if I should use dsync to do a 3 way sync? > existing_mailbox(a)server.com > tmp_mailbox(a)server.com > existing_mailbox(a)server.com Thank you: =================== Thank you for time reading this mail :) Dovecot Config: =================== # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.4 ext4 auth_mechanisms = plain login instance_name = trees.testing login_greeting = trees.testing ready. mail_gid = 5000 mail_location = maildir:/var/vmail/%Ld/%Ln/Maildir mail_plugins = " quota trees" mail_uid = 5000 mailbox_list_index = yes namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe autoexpunge = 30 days special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe autoexpunge = 30 days special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocols = " imap pop3" service auth { unix_listener /var/spool/postfix/private/auth { mode = 0666 } unix_listener auth-userdb { group = vmail mode = 0660 } } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } process_min_avail = 4 } service imap { executable = imap } service pop3 { executable = pop3 } ssl = required ssl_cert = </etc/ssl/certs/ssl-cert-snakeoil.pem ssl_cipher_list = ALL:!LOW:!RC4:!EXP:!aNULL ssl_dh_parameters_length = 2048 ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql }

5 5