June 2019 - dovecot - dovecot.org

flags not synced correctly with dovecot sync (dsync)
by Dan Christensen 31 Aug '19

31 Aug '19

I'm running dovecot 2.3.4.1 from https://repo.dovecot.org/ on Ubuntu 18.04 on three machines that I'll call server, laptop1 and laptop2. Both laptop1 and laptop2 run dovecot sync against server to keep local copies of my imap folders. Even when I initially had only two machines, laptop1 and server, I occasionally noticed that flags were lost, usually custom flags used by Gnus, but I couldn't reliably reproduce the problem. Now that I have two laptops syncing against the server, the problem has gotten worse and I figured out a way to reproduce it: - on server: create new IMAP folder test, and put two read messages in it - on laptop1: doveadm sync -u user -l 10 -m test -f user@server - on laptop2: doveadm sync -u user -l 10 -m test -f user@server At this point, all three machines show the two messages M1 and M2 as being read. - on laptop1: mark message M1 unread - on laptop2: mark message M2 unread - on laptop1: doveadm sync -u user -l 10 -m test -f user@server Both laptop1 and server have M1 unread, M2 read, as expected. - on laptop2: doveadm sync -u user -l 10 -m test -f user@server Now laptop2 and server have M1 *read*, M2 unread. - on laptop1: doveadm sync -u user -l 10 -m test -f user@server Now laptop1 and the server have both M1 and M2 *read*. - on laptop2: doveadm sync -u user -l 10 -m test -f user@server Now laptop2 has both read as well. The two lines that say "*read*" are wrong in my opinion. dsync propagated a read mark to an unread message, even though that message was marked unread more recently than it was marked read. I usually use stateful sync, and get many related problems. I just did a test in which M1 and M2 started out read, and I started with empty files named dstate.test on laptop1 and laptop2. Then I did the above procedure, using the command doveadm sync -u user -l 10 -m test -s "`cat dstate.test`" user@server > dstate.test At the end, laptop2 and server had both messages unread (which is good), but laptop1 had only M1 unread, and repeated runs of the sync command did not correct this. So the stateful sync failed to detect a change. Are these bugs in dovecot? Is there more information that I can provide? The output of doveconf -n on one machine is below, and the others are almost identical. Thanks for any help! Dan # 2.3.4.1 (3c0b8769e): /etc/dovecot/dovecot.conf # OS: Linux 4.15.0-45-generic x86_64 Ubuntu 18.04.1 LTS # Hostname: laptop2 auth_mechanisms = plain login listen = 127.0.0.1 mail_index_log2_max_age = 10 days mail_index_log_rotate_min_age = 1 days mail_index_log_rotate_min_size = 300 k mail_location = maildir:~/Maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = scheme=CRYPT username_format=%u /etc/dovecot/users driver = passwd-file } protocols = imap service imap-login { inet_listener imap { address = * port = 143 } inet_listener imaps { address = * port = 943 ssl = yes } } service imap { process_limit = 25 } ssl_cert = </etc/dovecot/private/dovecot.pem ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { args = username_format=%u /etc/dovecot/users driver = passwd-file } protocol lda { postmaster_address = [elided] } protocol imap { mail_max_userip_connections = 20 }

3 8

Emails not visible after renaming folders
by Aleksandr 14 Aug '19

14 Aug '19

Hello, I have strange problem with "losing" emails after rename mail folder(s) (via imap client: thunderbird, roundcude, etc..) How to reproduce: 1. Create some folder name, like TEST 2. Create sub-folder under TEST (like SUBTEST) Structure: TEST |--SUBTEST # doveadm mailbox list -u postmaster@testmailbox Spam Trash Sent Drafts INBOX TEST TEST/SUBTEST 3. Move (or copy) mails from INBOX to SUBTEST (all looks fine, and mails visible under SUBTEST) 4. Rename TEST folder to any new name, NEWTEST Let`s try to view mails in mail client in NEWTEST-SUBTEST, folder have no emails :( mailsrv# doveadm -f table mailbox status -u postmaster@testmailbox "messages vsize" NEWTEST* mailbox messages vsize NEWTEST 0 0 NEWTEST/SUBTEST 0 0 If doveadm force-resync postmaster@testmailbox, mails will be visible in INBOX mailsrv# doveadm -f table mailbox status -u postmaster@testmailbox "messages vsize" INBOX* mailbox messages vsize INBOX 2 28 Dovecot installation: CentOS x86_64 Linux 7.5.1804 Storage: HDD Local Partition - XFS filesystem / multi-dbox (mdbox) as mail_storage (this problem is not reproduced with the settings as Maildir storage !) somthing wrong with mapping indices. ---- [start] ---- # dovecot -n # 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.21 (92477967) # OS: Linux 3.10.0-862.2.3.el7.x86_64 x86_64 CentOS Linux release 7.5.1804 (Core) # Hostname: xxxx auth_mechanisms = plain login digest-md5 cram-md5 base_dir = /var/run/dovecot/ default_client_limit = 20000 default_login_user = dovecot default_process_limit = 10000 dict { quota = redis:host=127.0.0.1:prefix=user/:timeout_msecs=1000 } disable_plaintext_auth = no first_valid_gid = 90 first_valid_uid = 90 imapc_features = rfc822.size fetch-headers imapc_host = xxxx imapc_user = %u lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes login_greeting = xxxx. login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c login_trusted_networks = 10.0.1.0/24 mail_access_groups = mail mail_debug = yes mail_fsync = never mail_gid = 97 mail_location = mdbox:~/mail/mailboxes:FULLDIRNAME=mBoX-MeSsAgEs:INDEX=~/mail/index:CONTROL=~/mail/control:INBOX=~/mail/mailboxes/inbox mail_log_prefix = "%{session} %Us(%u): " mail_max_lock_timeout = 30 secs mail_plugins = quota zlib mail_prefetch_count = 20 mail_privileged_group = mail mail_uid = 97 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables enotify environment mailbox date index ihave duplicate mime foreverypart extracttext vacation-seconds editheader mbox_lock_timeout = 30 secs mbox_very_dirty_syncs = yes mbox_write_locks = fcntl namespace inbox { inbox = yes list = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Spam { auto = subscribe } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { cgroup_basedir = /usr/sys/cgroup hostingAccount = default quota = dict:User quota::proxy::quota quota_grace = 0%% quota_over_flag_value = TRUE quota_over_script = account-quota mismatch %u quota_rule = *:storage=4T quota_warning = storage=101%% account-quota block %u quota_warning1 = -storage=100%% account-quota unblock %u sieve = file:~/sieve;active=~/.dovecot.sieve sieve_before = /home/sieve/%d/ sieve_extensions = +editheader +vacation-seconds -body sieve_max_redirects = 200 sieve_vacation_min_period = 1s trash_folder = Trash zlib_save = lz4 zlib_save_level = 6 } pop3_delete_type = expunge protocols = imap pop3 sieve service account-quota { executable = script /usr/local/scripts/account-quota unix_listener account-quota { mode = 0666 user = dovecot } user = dovecot } service anvil { chroot = /var/empty client_limit = 0 drop_priv_before_exec = no executable = anvil extra_groups = group = $default_internal_user idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 1 protocol = service_count = 0 type = anvil unix_listener anvil-auth-penalty { group = $default_internal_user mode = 0660 user = dovecot } unix_listener anvil { group = $default_internal_user mode = 0660 user = dovecot } user = $default_internal_user vsz_limit = 18446744073709551615 B } service auth-worker { chroot = /var/empty user = $default_internal_user } service auth { chroot = /var/empty executable = /usr/libexec/dovecot/auth unix_listener auth-client { mode = 0660 } unix_listener auth-master { mode = 0666 } unix_listener auth-userdb { mode = 0777 } user = $default_internal_user } service dict { executable = /usr/libexec/dovecot/dict unix_listener dict { mode = 0600 user = $default_internal_user } } service imap-login { executable = /usr/libexec/dovecot/imap-login inet_listener imap { address = xxxxxxx port = 143 } process_limit = 10000 process_min_avail = 1 service_count = 0 user = dovecot } service imap { executable = /usr/libexec/dovecot/imap process_limit = 18000 process_min_avail = 1 service_count = 0 vsz_limit = 256 M } service lmtp { process_limit = 10 } service pop3-login { executable = /usr/libexec/dovecot/pop3-login inet_listener pop3 { address = xxxxxx port = 110 } process_limit = 1000 process_min_avail = 1 service_count = 0 user = dovecot vsz_limit = 256 M } service pop3 { executable = /usr/libexec/dovecot/pop3 process_limit = 1000 process_min_avail = 1 service_count = 0 } service send-warning { executable = script /usr/local/scripts/send-warning unix_listener send-warning { mode = 0666 user = dovecot } user = dovecot } shutdown_clients = no ssl_cert = </etc/dovecot/xxxx.crt ssl_key = # hidden, use -P to show it userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } verbose_proctitle = yes protocol lda { mail_plugins = sieve quota cgroups zlib } protocol imap { imap_client_workarounds = delay-newmail tb-extra-mailbox-sep mail_max_userip_connections = 50 mail_plugins = quota imap_quota mail_log notify cgroups zlib } protocol pop3 { mail_max_userip_connections = 20 mail_plugins = quota cgroups zlib pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_enable_last = no pop3_no_flag_updates = yes pop3_uidl_format = %08Xu%08Xv } ---- [end] ---- Regards, Aleksandr.

3 4

LastLogin update
by Jorge Bastos 13 Aug '19

13 Aug '19

Howdy, I'd like an adive on what to use. I have sql users, and i want to have the lastlogin of them, what is the native option for this: * Using last_login plugin which will give me the last login? * Use postlogin for imap and pop3n which will give me besides lastlogin the IP and protocol? Thanks in advanced,

8 13

[BUG?] Double quota calulation when special folder is present
by Mark Moseley 07 Aug '19

07 Aug '19

Just hoping to get some dev eyes on this. I'm incredibly reluctant to throw the word 'bug' around (since 99 times out of 100, it's not -- it's almost always the config), but I can't think of any way that this could be a config issue, esp when the pre-2.2.34 version works as expected. I noticed during troubleshooting that dovecot errors out if I try to create a subfolder called 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse (i.e. a folder called INBOX.INBOX.SomethingElse - resulting in a directory called .INBOX.SomethingElse on the filesystem, and leading to the problem described below). Is that sub-subfolder creation (where the top level subfolder matches the namespace name) supposed to be allowed? It seems odd that 'INBOX' (as a subfolder of INBOX) would be blocked but INBOX.SomethingElse (as a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse (i.e. INBOX.INBOX.SomethingElse) would be blocked as well. On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <dovecot(a)dovecot.org> wrote: > Hello, > > we're operating dovecot on a small server. Some years ago, we migrated > from courier IMAP to dovecot. Therefore, we defined our default > Namespace "inbox" with prefix "INBOX." to have this compatible. I found > this in some migration docs those days. Generally, everything worked as > expected. > > Our only namespace is configured like this: > > namespace inbox { > separator = . > prefix = INBOX. > inbox = yes > } > > Regularly, there is no folder named INBOX or .INBOX in the file system, > I suppose this is correct. But I found a special corner case today when > it comes to quota calculation. > > When - for whatever reason - a folder .INBOX.foo (for arbitrary values > of foo) exists, the whole mailbox is counted twice in quota > recalculation. Just creating .INBOX does nothing but a subfolder > triggers the problem. > > This is my shell view (replaced username and file path and deleted > unnecessary debug output) > > $ cat maildirsize > 268435456S > 14697 17 > $ maildirmake .INBOX.foo > $ sudo doveadm -D quota recalc -u <user> > [...] > doveadm(<user>): Debug: Namespace inbox: type=private, prefix=INBOX., > sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes > location=maildir:/home/.../test > doveadm(<user>): Debug: maildir++: root=/home/.../test, index=, > indexpvt=, control=, inbox=/home/.../test, alt= > doveadm(<user>): Debug: Namespace : type=private, prefix=, sep=, > inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none > doveadm(<user>): Debug: none: root=, index=, indexpvt=, control=, > inbox=, alt= > doveadm(<user>): Debug: quota: quota_over_flag check: quota_over_script > unset - skipping > doveadm(<user>): Debug: Quota root User quota: Recalculated relative > rules with bytes=268435456 count=0. Now grace=26843545 > doveadm(<user>): Debug: Namespace INBOX.: Using permissions from > /home/.../test: mode=0700 gid=default > > $ cat maildirsize > 268435456S > 29394 34 > > > So the used quota has exactly been doubled by just creating an empty > subfolder. > > Do you have any pointers for fixing my configuration or is this a bug in > dovecot? > > I coincidentally resurrected a months-old thread with this same issue a few days ago. I'm seeing the exact same after upgrading from 2.2.32 to 2.2.36. The original poster (who also narrowed it down to something in 2.2.34) mentioned a workaround that does indeed work, namely setting mailbox_list_index=no: > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c or src/lib-storage/list/mailbox-list-fs-iter.c or src/lib-storage/list/mailbox-list-index-iter.c or src/lib-storage/list/mailbox-list-index.c? The latter few have some added strcmp's against "INBOX". Then again, there's a lot of new code in the diffs under src/lib-storage that references INBOX specifically.

3 5

Dovecot replication and userdb "noreplicate".
by Reio Remma 07 Aug '19

07 Aug '19

Hello! I finally took the time and spent two days to set up replication for my server and now I have a question or two. I initially set noreplicate userdb field to 1 for all but a test user, but I could still see in the logs that all mailboxes were trying to connect to the other server via SSH. Is that normal? Jun 22 16:55:22 host dovecot: dsync-local(user(a)host.ee)<>: Error: Remote command returned error 84: ssh -i /home/vmail/.ssh/vmail.pem -l vmail backup.host.ee doveadm dsync-server -D -u user(a)host.ee Then I ended up setting mail_replica in userdb for only my test user, but I could still see in the logs that it was trying to sync the others as well, despite mail_replica being 0 for the rest. Jun 22 20:52:59 host dovecot: doveadm(user(a)host.ee): Fatal: -N parameter requires syncing with remote host I also notice (and read from recent posts) that sieve script replication doesn't work at all. Dovecot v2.3.6 and Pigeonhole from the official Dovecot CentOS repo. Thanks, Reio PS: Getting SSH for Dovecot to work with SELinux on CentOS 7 was fun as usual. :)

3 6

GSSAPI vs group check
by Németh Ákos Ferenc 05 Aug '19

05 Aug '19

Dear All, Is it possible to make any authorization (eg. checking of group membership) in case of GSSAPI authentication? Our dovecot authenticates the users against PAM and GSSAPI. In the PAM file I'm able to check if a user is a member of a selected (e.g mailreader) group. If the user is member, he can login otherwise not (see below). If the user has a valid Kerberos ticket and he tries to login via GSSAPI, I can't restrict him if he is not a member of the selected group. How can I overcome this issue? My config: passdb { driver = pam # [session=yes] [setcred=yes] [failure_show_msg=yes] [max_requests=<n>] # [cache_key=<key>] [<service name>] #args = dovecot } userdb { # <doc/wiki/AuthDatabase.Passwd.txt> driver = passwd # [blocking=no] #args = # Override fields from passwd #override_fields = home=/home/virtual/%u } ...in PAM file: auth [success=1 default=ignore] pam_succeed_if.so user ingroup mailreader auth [success=ignore default=2] pam_succeed_if.so user ingroup admins auth [success=ignore default=1] pam_succeed_if.so uid >= 1000 auth [success=3 default=ignore] pam_winbind.so krb5_auth krb5_ccache_type=FILE cached_login auth [success=ignore default=1] pam_succeed_if.so uid < 1000 auth [success=1 default=ignore] pam_unix.so nullok_secure try_first_pass auth requisite pam_deny.so auth required pam_permit.so Thank you. Br, Ákos

2 3

Bug with latest GCC 9
by Martin Liška 29 Jul '19

29 Jul '19

Hi. As mentioned here: https://bugzilla.opensuse.org/show_bug.cgi?id=1123136 there's a new issue with GCC 9, it's related to: https://gcc.gnu.org/gcc-9/porting_to.html#complit in: /* Not const! Never return this as a result directly! */ #define SMTP_ADDRESS_LITERAL(localpart, domain) \ &((struct smtp_address){ (localpart), (domain) }) Thanks, Martin

2 4

sieve scripts not synching for 2.3.5.1 pre-built
by Jan-Pieter Cornet 29 Jul '19

29 Jul '19

Hi, We're synching mailboxes, changing format from maildir to mdbox, using doveadm backup/doveadm sync. When still running 2.2.36, 'doveadm backup' also synched the sieve scripts, without issues. After the upgrade to 2.3.5.1, the sieve sync stopped working. We're using the pre-built 2.3 packages from https://repo.dovecot.org/ce-2.3-latest/debian/stretch In fact, when I now remove the dovecot-2.3 packages, and use my own old 2.2.36 packages, using the exact same config, the sieve scripts are being copied again. In 2.2.36, when I migrate a mailbox, I do see Pigeonhole being initialized. Eg: userimap-dev1:~ # doveadm -D backup -u xtra30 "doveadm -D dsync-server -u xtra30+mdbox=.8d1&4&03&xtra30" (note: that strange destination user signals to the auth proxy that we want the new mdbox storage instead of the default for this user) Debug: Loading modules from directory: /usr/lib/dovecot/modules Debug: Module loaded: /usr/lib/dovecot/modules/lib02_lazy_expunge_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so [...skipping a bunch of lines...] dsync-local(xtra30): Debug: brain M: Mailbox Spam: local=88b39d12998d9e5cbb700000436edcce/0/1, remote=00000000000000000000000000000000/0/0: mailbox not selectable yet dsync-local(xtra30): Debug: brain M: Mailbox Test: local=801bb932e98d9e5c3d6d0000436edcce/0/1, remote=00000000000000000000000000000000/0/0: mailbox not selectable yet dsync-local(xtra30): Debug: brain M: Mailbox Trash: local=fc9b162b86479f5cc85a0000436edcce/0/1, remote=00000000000000000000000000000000/0/0: mailbox not selectable yet dsync-remote(xtra30): Debug: brain S: Skipping unchanged mailbox e7c1961d84479f5c885a0000436edcce dsync-local(xtra30): Debug: doveadm-sieve: Iterating Sieve mailbox attributes dsync-local(xtra30): Debug: sieve: Pigeonhole version 0.4.24 () initializing dsync-local(xtra30): Debug: sieve: include: sieve_global is not set; it is currently not possible to include `:global' scripts. dsync-local(xtra30): Debug: sieve: file storage: Using active Sieve script path: /var/mail/.8d1/index/7/oldsieve/x/xt/xtra30/dovecot.sieve [... skipping a lot of sieve debug lines that seem fine ...] dsync-remote(xtra30): Debug: sieve: file script: Opened script `xtra30-testfilter' from `/var/mail/.8d1/index/4/03/xtra30/sieve/scripts/xtra30-testfilter.sieve' dsync-remote(xtra30): Debug: doveadm-sieve: Assigned value for key `vendor/vendor.dovecot/pvt/server/sieve/default' (last change: 2019-03-29 22:27:41) dsync-remote(xtra30): Debug: brain S: Import INBOX: Import attribute vendor/vendor.dovecot/pvt/server/sieve/default: Nonexistent locally dsync-remote(xtra30): Debug: brain S: Import INBOX: Last common UID=0. Delayed expunges= dsync-remote(xtra30): Debug: brain S: Import INBOX: Import change type=save GUID=1553894809.M254847P28859.userimap11.xs4all.net,S=1311,W=1342 UID=1 hdr_hash= result=Mail's UID is above local UIDNEXT - No more local mails found [...] Afterwards, the sieve scripts have been transferred, even including last-modified time of any scripts. However, with 2.3.5.1, it seems doveadm-sieve is never called from dsync. Output includes: userimap-dev1:~ # doveadm -D backup -u xtra30 "doveadm -D dsync-server -u xtra30+mdbox=.8d1&4&03&xtra30" Debug: Loading modules from directory: /usr/lib/dovecot/modules Debug: Module loaded: /usr/lib/dovecot/modules/lib02_lazy_expunge_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_lookup_dict_iterate_visible_next (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so [... skipping some lines again ..] dsync-local(xtra30)<SCSfIc5Wo1zTKAAAsPW7qw>: Debug: brain M: Mailbox Spam: local=88b39d12998d9e5cbb700000436edcce/0/1, remote=00000000000000000000000000000000/0/0: mailbox not selectable yet dsync-local(xtra30)<SCSfIc5Wo1zTKAAAsPW7qw>: Debug: brain M: Mailbox Test: local=801bb932e98d9e5c3d6d0000436edcce/0/1, remote=00000000000000000000000000000000/0/0: mailbox not selectable yet dsync-local(xtra30)<SCSfIc5Wo1zTKAAAsPW7qw>: Debug: brain M: Mailbox Trash: local=fc9b162b86479f5cc85a0000436edcce/0/1, remote=00000000000000000000000000000000/0/0: mailbox not selectable yet dsync-remote(xtra30)<h8hfIs5Wo1zWKAAAsPW7qw>: Debug: brain S: Skipping unchanged mailbox e7c1961d84479f5c885a0000436edcce dsync-remote(xtra30)<h8hfIs5Wo1zWKAAAsPW7qw>: Debug: brain S: Import INBOX: Last common UID=0. Delayed expunges= dsync-remote(xtra30)<h8hfIs5Wo1zWKAAAsPW7qw>: Debug: brain S: Import INBOX: Import change type=save GUID=1553894809.M254847P28859.userimap11.xs4all.net,S=1311,W=1342 UID=1 hdr_hash= result=Mail's UID is above local UIDNEXT - No more local mails found [...] In the above output from 2.3.5.1, all lines starting with "Debug: doveadm-sieve: Iterating Sieve mailbox attributes" are absent. I have no idea why. We can "work around" it by explicitly doing an rsync of the sieve storage after the migration, but that's a bit of a hack. Attached is a doveconf -n output for 2.3.5.1. -- Jan-Pieter Cornet <johnpc(a)xs4all.net> Systeembeheer XS4ALL Internet bv www.xs4all.nl

6 7

IMAP copy stopped copying flags
by Andrzej A. Filip 29 Jul '19

29 Jul '19

Is it intended behavior? It seems to be caused by upgrade to 1:2.3.2-2 on Debian/Testing. -- A. Filip

4 6

Re: Best mail encryption solution for per-user
by Aki Tuomi 12 Jul '19

12 Jul '19

On 27.05.2018 21:16, mail(a)sjemm.net wrote: > May 27, 2018 8:52 AM, "Aki Tuomi" <aki.tuomi(a)dovecot.fi> wrote: >>> On 26 May 2018 at 10:36 mail(a)sjemm.net wrote: >>> >>> May 23, 2018 10:10 AM, mail(a)sjemm.net wrote: >>> May 23, 2018 9:46 AM, "Aki Tuomi" <aki.tuomi(a)dovecot.fi> wrote: >>> >>> On 23.05.2018 10:15, mail(a)sjemm.net wrote: >>> >>> May 23, 2018 8:31 AM, "Aki Tuomi" <aki.tuomi(a)dovecot.fi> wrote: >>> >>> On 23.05.2018 09:13, mail(a)sjemm.net wrote: >>> May 20, 2018 8:01 PM, mail(a)sjemm.net wrote: >>> May 20, 2018 2:47 PM, "Aki Tuomi" <aki.tuomi(a)dovecot.fi> wrote: >>> On 19 May 2018 at 16:40 mail(a)sjemm.net wrote: >>> >>> May 18, 2018 10:01 PM, "Aki Tuomi" <aki.tuomi(a)dovecot.fi> wrote: >>> On 18 May 2018 at 21:44 mail(a)sjemm.net wrote: >>> >>> May 18, 2018 4:43 PM, "Aki Tuomi" <aki.tuomi(a)dovecot.fi> wrote: >>> On 18 May 2018 at 17:38 mail(a)sjemm.net wrote: >>> >>> May 18, 2018 4:05 PM, "Aki Tuomi" <aki.tuomi(a)dovecot.fi> wrote: >>> On 18 May 2018 at 16:43 mail(a)sjemm.net wrote: >>> >>> Hi Tai74 and Aki, >>> I followed your conversation with interest on how to setup per user encryption in dovecot. >>> I have setup my dovecot with the following in a conf file: >>> >>> ============== >>> >>> mail_attribute_dict = file:%h/Maildir/dovecot-attributes >>> mail_plugins = $mail_plugins mail_crypt >>> plugin { >>> >>> mail_crypt_curve = secp521r1 >>> >>> mail_crypt_save_version = 2 >>> >>> } >>> >>> ============== >>> >>> This works nice, all emails are being encrypted and every user/folder has keys. >>> But as I understood from your conversation these keys are not protected. And I want them to be >>> protected by the users password used by imap. >>> >>> Those passwords are stored in a mysql DB file. ( I used a guide from workaround [dot] org to set up >>> the DB and postfix/dovecot) >>> >>> but how would i set it so, that the users password from the DB is used to encrypt the keys? >>> >>> should i use mail_crypt_private_password = ? >>> how do i point it to the mysql db then? >>> im unsure about this >>> >>> Do you have any hints on this? >>> >>> Kind regards, >>> Zjemm >>> >>> The passwords in your MySQL database are, hopefully, not in plaintext. If you want to secure your >>> user's keys using user's login password, you must have a TOOL that manages this. >>> >>> You can use mail_crypt_private_password = %w in (mysql) passdb fields to provide the user's login >>> password as private password. You might want to run it thru some hash, so %{sha1:password} might be >>> a good option. >>> >>> You can change the key password using 'doveadm mailbox cryptokey', this needs to be done every time >>> user changes his password. >>> >>> Also note that if you go down this road, and the user forgets his password, you will not be able to >>> recover the emails without backup copy of the private key. >>> >>> Aki >>> >>> Hi Aki >>> >>> I used the following command: >>> dovecot pw -s SHA256-CRYPT >>> >>> the output on the chosen password looks like: {SHA256-CRYPT}$5$Rokc06a7In4SF3bO$OQpGQWqg........ >>> >>> This output is used to store in the password fields in the database. So no plain text passwords no >>> :) >>> >>> You can use mail_crypt_private_password = %w in (mysql) passdb fields to provide the user's login >>> password as private password. >>> >>> can you explain this a bit more for me? >>> >>> for now i have in the 10-auth.conf file the following: >>> ============== >>> passdb { >>> driver = sql >>> >>> # Path for SQL configuration file, see example-config/dovecot-sql.conf.ext >>> args = /etc/dovecot/dovecot-sql.conf.ext >>> } >>> >>> and: >>> >>> userdb { >>> driver = static >>> args = uid=vmail gid=vmail home=/var/vmail/%d/%n >>> } >>> ============== >>> >>> then i have in dovecot-sql.conf.ext >>> ============== >>> driver = mysql >>> connect = host=x.x.x.x dbname=mailserver user=mailuser password=mailpasswordexample >>> default_pass_scheme = SHA256-CRYPT >>> password_query = SELECT email as user, password FROM virtual_users WHERE email='%u'; >>> ============== >>> Where do i need to set : mail_crypt_private_password = %w ? >>> >>> password as private password. You might want to run it thru some hash, so %{sha1:password} might be >>> a good option. >>> >>> the passwords are allready hashed in the DB using: dovecot pw -s SHA256-CRYPT to genereate the has. >>> so this step isnt nesesary anymore am i right? >>> >>> Thank you for your quick response, very helpfull >>> >>> Zjemm >>> >>> You misunderstood a bit. The idea is to use the *plaintext* password as the password for the >>> private key. Otherwise anyone could just decrypt it by looking at your database where the hashed >>> password is.. >>> >>> So: >>> >>> password_query = SELECT email as user, password, '%w' AS userdb_mail_crypt_private_password FROM >>> virtual_users WHERE email='%u' >>> >>> Aki >>> >>> Hi Aki, >>> >>> Thank you very much for your help, i realy appreciate that. >>> >>> Ok so if i understand it correctly i'll have to use: >>> >>> password_query = SELECT email as user, password, '%w' AS userdb_mail_crypt_private_password FROM >>> virtual_users WHERE email='%u' >>> >>> in my dovecot-sql.conf.ext file >>> >>> This query selects the user, the password, and %w >>> >>> if i run a little query myself: >>> MariaDB [mailserver]> SELECT email as user, password, '%w' AS userdb_mail_crypt_private_password >>> FROM virtual_users; >>> +------------------+----------------------------------+------------------------------------+ >>> | user | password | userdb_mail_crypt_private_password | >>> +------------------+----------------------------------+------------------------------------+ >>> | john(a)example.org | {SHA256-CRYPT}$5$M/GWzmtjsLroRWI | %w | >>> +------------------+----------------------------------+------------------------------------+ >>> >>> %w is a dovecot variable, and stands for the plaintext password, but the password is not stored as >>> plaintext in the DB, %w get filled with the actual plaintext password by dovecot upon the user that >>> is typing in the password when authenticating. >>> >>> is this correct? >>> >>> yes. >>> >>> so then i have the username the hashed password en the plaintext password as a result of the query. >>> >>> yes >>> >>> now userdb_mail_crypt_private_password = the plaintext password >>> do i need to reference it somewhere? or is userdb_mail_crypt_private_password autmatically used by >>> the dovecot mail_crypt plugin to encrypt the keys? or should it be mail_crypt_private_password? >>> >>> It gets injected into the mail process as 'mail_crypt_private_password', as if it was set in plugin >>> {} section. >>> >>> if i have this setup working i'm going to write a blog post on this topic to share this knowledge >>> >>> Thanks again and have a great weekend. >>> >>> Zjemm >>> >>> Aki >>> >>> Hi Aki, >>> >>> Cool i'm testing it right now. >>> I have set up a new mailserver (life is great with lxc containers :) ) >>> >>> postfix and dovecot are working like normal >>> >>> next i enable mail_crypt >>> >>> i did create a file: /etc/dovecot/conf.d/10-mailcrypt.conf >>> ========================== >>> mail_attribute_dict = file:%h/Maildir/dovecot-attributes >>> >>> mail_plugins = $mail_plugins mail_crypt >>> >>> plugin { >>> mail_crypt_curve = secp521r1 >>> mail_crypt_save_version = 2 >>> } >>> ========================== >>> >>> and then i changed the file: /etc/dovecot/dovecot-sql.conf.ext >>> >>> so the query is now the new query: >>> password_query = SELECT email as user, password, '%w' AS userdb_mail_crypt_private_password FROM >>> virtual_users WHERE email='%u'; >>> >>> then i restarted dovecot and postfix and send a test email to the one and only testuser that is in >>> there. >>> >>> when i open the mailbox with the tool mutt, i can see the new email, and when openening the email >>> the mutt client drops the connection. >>> >>> in the log i can see: >>> >>> May 19 13:34:48 mailserver1.example.local dovecot[600]: imap-login: Login: user=<john(a)example.org>, >>> method=PLAIN, rip=::1, lip=::1, mpid=713, TLS, session=<E3PnIY9sNM4AAAAAAAAAAAAAAAAAAAAB> >>> May 19 13:34:49 mailserver1.example.local dovecot[600]: imap(john(a)example.org): Error: read() >>> failed: >>> read(/var/vmail/example.org/john/Maildir/cur/1526736378.M161472P641.mailser…, >>> =571:2,) failed: Private key not available: Cannot decrypt key >>> bfc5bb25b1bf64290eea6dc14b516c6a0a25b64551b6e4f0f8677ba7274887cb: error:03070068:bignum >>> routines:BN_mpi2bn:encoding error (FETCH BODY[] for mailbox INBOX UID 8) >>> >>> i think i missed a step, but witch one? >>> >>> the userpassword hasnt been changed (that would be the next step in the testing process) >>> >>> should i've use doveadm first to encrypt the key with that userpassword? i thought it would do that >>> on the fly, because the initial keys where only just created when enabling the mail_crypt plugin >>> >>> please let me know your thougts >>> Zjemm >>> I noticed you replied directly to me, and not to the list, too... fixed that for you. >>> >>> mail_crypt_private_password is used when key is created, but if you have created it before using >>> password, you'll need to encrypt it before turning the setting on. >>> >>> Aki >>> Hi, Ok i'm sorry about that, thank you for fixing that. i'm a bit new to mailing lists :) >>> >>> are you a dovecot devoloper? or a enthusiastic user? >>> >>> anyway, i managed to get it working now with all your help :) >>> >>> to start fresh i did a: >>> rm -rf /var/vmail/example.org >>> >>> so all email is gone, and i assume all dovecot keys are gone that way. >>> >>> Then i tried to set a password first: >>> >>> root@mailserver1:/var/vmail# doveadm mailbox cryptokey password -u john(a)example.org -n summersun >>> result: Changed password for 0 key(s) >>> >>> (summersun = the password for this user) >>> >>> but then it says there are 0 keys, so then i send the test user an email and try to open the >>> mailbox, i'll get: >>> >>> imap(john(a)example.org): Error: read() failed: >>> read(/var/vmail/example.org/john/Maildir/cur/1526827992.M149712P883.mailser…, >>> =561:2,) failed: Private key not available: Cannot decrypt key >>> 71849013a70b0c631c06112077e7c2fe39b0b2737b4933b219793841209d4e7f: error:03070068:bignum >>> routines:BN_mpi2bn:encoding error >>> >>> but hey, now the key's are created, so i tried to set the password again: >>> >>> root@mailserver1:/var/vmail# doveadm mailbox cryptokey password -u john(a)example.org -n summersun >>> result: dcrypt_key_load_private(787701bd677dd69a26842547a3926cbaa625b0b5a91751f06678c3e9708343d9) >>> failed: password missing >>> ========= >>> >>> Lets start over again, i did a new: >>> rm -rf /var/vmail/example.org >>> >>> i disabled the mail_crypt_private_password in the sql query and send an email to the test user. >>> now again new keys are generated. >>> >>> then i did set the password via doveadm: >>> >>> root@mailserver1:/var/vmail# doveadm mailbox cryptokey password -u john(a)example.org -n summersun >>> result: Changed password for 1 key(s) >>> >>> i enabled the mail_crypt_private_password again in the sql query part and now it works :) >>> >>> YESSS!!! >>> >>> if i now change the password for the user, i cannot read the email anymore. that is what should >>> happen. >>> so now i changed the password for this user also with doveadm: >>> - doveadm mailbox cryptokey password -u john(a)example.org -n qwerty -o summersun >>> >>> and now i'm able to read the emails again. so it all works now as expected. >>> >>> i have now only one thing remaining. >>> >>> when i want to add a new user, i have to somehow: >>> - create the new user in the mysql DB >>> - start with dovecot not using mail_crypt_private_password (so disabled) >>> - send an initial email to let the users mailbox be created and generate the keys >>> - set the users password with doveadm >>> - restart dovecot again with mail_crypt_private_password (so enabled) >>> >>> but this would be disruptive to existing users, and i would have to script the whole thing as a >>> "add-new-user" script to make it usable i guess. >>> >>> do you know of a more easy way of acomplishing this? >>> could i use doveadm for all steps, and get around the mail_crypt_private_password disable part? >>> >>> or could i add a column in the mysql db called userdb_mail_crypt_private_password and fill that >>> column with %w for users that allready exist, and empty for new users. >>> when being empty, it might not set the mail_crypt_private_password for new users >>> >>> or is this the wrong way of thinking? >>> >>> let me know your thoughts on this. >>> >>> And last but not least, i really want to thank you for all your help. It is a really nice >>> experience. >>> Bless you! >>> >>> Zjemm >>> Hi Aki, >>> >>> You did fix the posting on the mail archive, i can see our conversation on there, all but the last >>> email i send to you. >>> Maybe you did not received the last email? >>> >>> would you be willing to take a look at my last question in the email See below: >>> Thank you very much. >>> >>> ================================ >>> i have now only one thing remaining. >>> >>> when i want to add a new user, i have to somehow: >>> - create the new user in the mysql DB >>> - start with dovecot not using mail_crypt_private_password (so disabled) >>> - send an initial email to let the users mailbox be created and generate the keys >>> - set the users password with doveadm >>> - restart dovecot again with mail_crypt_private_password (so enabled) >>> >>> but this would be disruptive to existing users, and i would have to script the whole thing as a >>> "add-new-user" script to make it usable i guess. >>> >>> do you know of a more easy way of acomplishing this? >>> could i use doveadm for all steps, and get around the mail_crypt_private_password disable part? >>> >>> or could i add a column in the mysql db called userdb_mail_crypt_private_password and fill that >>> column with %w for users that allready exist, and empty for new users. >>> when being empty, it might not set the mail_crypt_private_password for new users >>> >>> or is this the wrong way of thinking? >>> >>> let me know your thoughts on this. >>> >>> And last but not least, i really want to thank you for all your help. It is a really nice >>> experience. >>> Bless you! >>> >>> Zjemm >>> >>> If you have mail_crypt_private_password set *when* keys are generated, >>> dovecot should use that password to encrypt the user key. >>> >>> Aki >>> Hi Aki, >>> >>> well, if there is a new email user setup in the mysql DB, and no email has been send to that new >>> user. at that point there are no keys. >>> >>> So trying to set a password on those key's wont work. see: >>> root@mailserver1:/var/vmail/example.org# doveadm mailbox cryptokey password -u john2(a)example.org -n >>> qwerty >>> result: Changed password for 0 key(s) >>> >>> the keys need to be generated first. And that only happens when sending a mail to that new user. >>> Also, at the point of creating the keys, the mail_crypt_private_password should be disabled >>> otherwise the keys are generated and also encrypted with no user password, and then im unable to >>> set a new pasword for that key >>> >>> How can i set the password before the keys are generated? >>> or can i manually generate key's and then set the password? >>> >>> so at this point i have to do the following: >>> >>> You can use doveadm mailbox cryptokey generate -u user -U >>> >>> to generate a user key. >>> >>> Aki >>> >>> - create the new user in the mysql DB >>> - start with dovecot not using mail_crypt_private_password (so disabled) >>> - send an initial email to let the users mailbox be created and generate the keys >>> - set the users password with doveadm >>> - restart dovecot again with mail_crypt_private_password (so enabled) >>> can i somehow enable the mail_crypt_private_password per user? >>> because this mysql query always returns: %w >>> and thus allways sets: userdb_mail_crypt_private_password >>> SELECT email as user, password, '%w' AS userdb_mail_crypt_private_password FROM virtual_users; >>> >>> Hi Aki, >>> >>> You are correct >>> >>> doveadm mailbox cryptokey generate -u user -U generates the keys, but i have to first remove: >>> '%w' AS userdb_mail_crypt_private_password >>> from dovecot configuration and restart dovecot. >>> >>> if i don't do that, the keys are generated, but also somehow encrypted using: no >>> password/empty/something else (i don't know) >>> >>> here you can see that happening: >>> >>> root@mailserver1:/var/vmail/example.org# doveadm mailbox cryptokey generate -u john2(a)example.org -U >>> Folder Public ID >>> ✓ <userkey> 53004acf9886f887ca081c83d7392a5aa3fac4b5a20ce013db91a341fb4644c4 >>> root@mailserver1:/var/vmail/example.org# doveadm mailbox cryptokey password -u john2(a)example.org -n >>> qwerty >>> result: dcrypt_key_load_private(53004acf9886f887ca081c83d7392a5aa3fac4b5a20ce013db91a341fb4644c4) >>> failed: password missing >>> >>> it expects an old password wich is never set in my opinion. >>> >>> so the last part of this setup would be to figure out a way to disable '%w' AS >>> userdb_mail_crypt_private_password for new users without manually removing that part from the query >>> and restarting dovecot. As that would mean connection interruption while adding new users >>> >>> if we could add new users to the DB and somehow not give '%w' AS userdb_mail_crypt_private_password >>> for that new user in the query, we can add the password afterward, and then enable '%w' AS >>> userdb_mail_crypt_private_password in the query for this user >>> >>> could we add a column in the user db lets say called: encrypted (yes/no) >>> >>> and then change the existing query: SELECT email as user, password, '%w' AS >>> userdb_mail_crypt_private_password FROM virtual_users; >>> >>> to firrt check if encryption is set to 'yes', if so retrun: >>> SELECT email as user, password, '%w' AS userdb_mail_crypt_private_password FROM virtual_users; >>> >>> if not return: >>> SELECT email as user, password FROM virtual_users; >>> >>> but then in one mysql query so dovecot can work with it >>> >>> ? >>> >>> Hi Aki, >>> >>> would you be willing to respond to my last question i had send to you last week? >>> its this one: >>> ================================== >>> doveadm mailbox cryptokey generate -u user -U generates the keys, but i have to first remove: >>> '%w' AS userdb_mail_crypt_private_password >>> from dovecot configuration and restart dovecot. >>> >>> if i don't do that, the keys are generated, but also somehow encrypted using: no >>> password/empty/something else (i don't know) >>> >>> here you can see that happening: >>> >>> root@mailserver1:/var/vmail/example.org# doveadm mailbox cryptokey generate -u john2(a)example.org -U >>> Folder Public ID >>> ✓ <userkey> 53004acf9886f887ca081c83d7392a5aa3fac4b5a20ce013db91a341fb4644c4 >>> root@mailserver1:/var/vmail/example.org# doveadm mailbox cryptokey password -u john2(a)example.org -n >>> qwerty >>> result: dcrypt_key_load_private(53004acf9886f887ca081c83d7392a5aa3fac4b5a20ce013db91a341fb4644c4) >>> failed: password missing >>> >>> it expects an old password wich is never set in my opinion. >>> >>> could this be a bug? or am i doing it wrong? if needed i could send in a bug report? >>> >>> or might we need to enable the userdb_mail_crypt_private_password "per user" ? >>> >>> so the last part of this setup would be to figure out a way to disable '%w' AS >>> userdb_mail_crypt_private_password for new users without manually removing that part from the query >>> and restarting dovecot. As that would mean connection interruption while adding new users >>> >>> if we could add new users to the DB and somehow not give '%w' AS userdb_mail_crypt_private_password >>> for that new user in the query, we can add the password afterward, and then enable '%w' AS >>> userdb_mail_crypt_private_password in the query for this user >>> >>> could we add a column in the user db lets say called: encrypted (yes/no) >>> >>> and then change the existing query: SELECT email as user, password, '%w' AS >>> userdb_mail_crypt_private_password FROM virtual_users; >>> >>> to firrt check if encryption is set to 'yes', if so retrun: >>> SELECT email as user, password, '%w' AS userdb_mail_crypt_private_password FROM virtual_users; >>> >>> if not return: >>> SELECT email as user, password FROM virtual_users; >>> >>> but then in one mysql query so dovecot can work with it >>> >>> Please let me know your thougts, after this last thing im done :) >>> also let me know if i can support this project in some way, im willing to contribute >>> >>> Zjemm >> You can select it as userdb_mail_crypt_save_version=0 to disable encryption (it won't disable >> decryption). You can use value '2' for enabling encryption. DO NOT USE 1. >> >> Also I tried the SQL based thing, and if you had EMPTY mail_crypt_private_password, try these to >> change the password. If the one that works is the second one, let me know =) >> >> doveadm mailbox cryptokey password -u testuser -nnewpass -o "" >> >> doveadm mailbox cryptokey password -u testuser -nnewpass -o "yes" >> >> Aki > Hi Aki, > > Thanks for replying > > So if i understand you correctly i could enable encryption per user by using: userdb_mail_crypt_save_version= in the database as an extra column. the column would be called "userdb_mail_crypt_save_version" and the value is "0" or "2" > > if a user has: userdb_mail_crypt_save_version=0, encryption is disabled for that user > if a user has: userdb_mail_crypt_save_version=2, encryption is enabled for that user > > would this also override the setting: mail_crypt_save_version = 2 > in the mailcryp.conf file where that is a global setting? or should i delete it there and only use userdb_mail_crypt_save_version= from the database? > > for the password set command i hve testest some things: > > new user with no keys yet: > doveadm -o plugin/mail_crypt_private_password=password mailbox cryptokey generate -u john2(a)example.org -U > > i can set a password with: > new password: > doveadm mailbox cryptokey password -u john2(a)example.org -n newpass -o password > > > if i do not use new user doveadm: (doveadm -o plugin/mail_crypt_private_password=password mailbox cryptokey generate -u john2(a)example.org -U) > then use this: > doveadm mailbox cryptokey password -u john2(a)example.org -n newpass -o "yes" > > > so yes it is the second one > > Kind regards, > > Zjemm It's unfortunate problem with 2.2 that it defaults empty userdb variables into yes. This has been remedied in 2.3. Unfortunately there is no clean fix to this, they all resort to various levels of trickery. Variables set in passdb/userdb overwrite the ones in plugin {} section. Aki

2 2