Help with imapc and Shared Folder in a Cluster

Wed Apr 28 17:34:48 EEST 2021

> On 28/04/2021 14:29 Alessio Cecchi <alessio at skye.it> wrote:
> 
>  
> Il 28/04/21 11:49, Markus Valentin ha scritto:
> >> On 27/04/2021 22:04 Alessio Cecchi <alessio at skye.it> wrote:
> >> Il 23/04/21 09:29, Markus Valentin ha scritto:
> >>> On 4/22/21 11:49 PM, Alessio Cecchi wrote:> I'm tryng to setup Shared
> >>> Mailboxes in Dovecot (2.3.14) Cluster as
> >>>> explained here:
> >>>>
> >>>> https://doc.dovecot.org/configuration_manual/shared_mailboxes/cluster_setup/
> >>>>
> >>>>
> >>>> but I'm not happy:
> >>>>
> >>>> # doveadm acl debug -u test1 at emailtest.net shared/test2/Sent
> >>>>
> >>>> doveadm(test1 at emailtest.net): Info: imapc(10.0.0.202:143): Connected to
> >>>> 10.0.0.202:143 (local 10.0.0.203:58054)
> >>>> doveadm(test1 at emailtest.net): Info: imapc(10.0.0.202:143): Connected to
> >>>> 10.0.0.202:143 (local 10.0.0.203:58056)
> >>>> doveadm(test1 at emailtest.net): Error: imapc(10.0.0.202:143):
> >>>> Authentication failed: [AUTHENTICATIONFAILED] Authentication failed.
> >>>> doveadm(test1 at emailtest.net): Error: Can't open mailbox
> >>>> shared/test2/Sent: Authentication failed: [AUTHENTICATIONFAILED]
> >>>> Authentication failed.
> >>>>
> >>>> ACL, master-user, master-password works fine because with regular
> >>>> configuration shared folders works fine and also with master-user or
> >>>> with master-password I can login and see and access to shared/ namespace
> >>>> and shared folders.
> >>>>
> >>>> But when I try to switch location from
> >>>>
> >>>> location = maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u
> >>>>
> >>>> to
> >>>>
> >>>> location = imapc:~/Maildir/shared/%%u/
> >>>> [...]
> >>>> imapc_host = 10.0.0.202
> >>>> imapc_master_user = %u
> >>>> #imapc_user = %u
> >>>> imapc_password = Password
> >>>> imapc_features = search
> >>>>
> >>>> stop working.
> >>>>
> >>>> The relevant error is this:
> >>>>
> >>>> Apr 22 22:57:14 doveadm(test1 at testemail.net): Info:
> >>>> imapc(10.0.0.203:143): Connected to 10.0.0.202:143 (local 10.0.0.203:58070)
> >>>> Apr 22 22:57:14 doveadm(test1 at testemail.net): Debug:
> >>>> imapc(10.0.0.203:143): Server capabilities: IMAP4rev1 SASL-IR
> >>>> LOGIN-REFERRALS ID ENABLE IDLE XLIST LITERAL+ AUTH=PLAIN AUTH=LOGIN
> >>>> Apr 22 22:57:14 doveadm(test1 at testemail.net): Debug:
> >>>> imapc(10.0.0.203:143): Authenticating as test1 at testemail.net for user
> >>>> test2 at testemail.net
> >>>> Apr 22 22:57:16 doveadm(test1 at testemail.net): Error:
> >>>> imapc(10.0.0.203:143): Authentication failed: [AUTHENTICATIONFAILED]
> >>>> Authentication failed.
> >>>> Apr 22 22:57:16 doveadm(test1 at testemail.net): Debug:
> >>>> imapc(10.0.0.203:143): Disconnected
> >>>> Apr 22 22:57:16 doveadm(test1 at testemail.net): Error: Can't open mailbox
> >>>> shared/test2/Sent: Authentication failed: [AUTHENTICATIONFAILED]
> >>>> Authentication failed.
> >>>>
> >>>> Please note "Authenticating as test1 at testemail.net for user
> >>>> test2 at testemail.net" failed.
> >>>>
> >>>> So my question is, the documentation page is update and right or I
> >>>> missing something?
> >>> Hi,
> >>>
> >>> from my perspective it is likely that test1 at testemail.net can't be
> >>> authenticated as a master user which is required for this setup to work.
> >>>
> >>>   From the cluster setup page:
> >>>
> >>> "You’ll need to setup master user logins to work for all the users. The
> >>> logged in user becomes the master user. The master user doesn’t actually
> >>> have any special privileges. "
> >>>
> >> Hi,
> >>
> >> after some days of debug I have found a solution to have shared folders
> >> works via imapc, even if partially.
> >>
> >> First, in the documentation page there is an error, the right "location"
> >> should be like this:
> >>
> >> location = imapc:%%h/Maildir
> >>
> >> with %%h/ instead of ~/
> >>
> >> After I have setup two passdb like these:
> >>
> >> passdb {
> >>      driver = static
> >>      args = password=P4ssw0rd
> >>      result_success = continue
> >> }
> >>
> >> passdb {
> >>     driver = sql
> >>     args = /etc/dovecot/dovecot-sql-master.conf.ext
> >>     master = yes
> >>     result_success = continue
> >> }
> >>
> >> where the first is required (only on backend dovecot) when the sharing
> >> user (test2) need to login (with imapc_password) and the second (both in
> >> director and backend dovecot) when the "test1" need to login into
> >> sharing (test2) account like master user.
> >>
> >> So acl debug works fine:
> >>
> >> # doveadm acl debug -u test1 at emailtest.net shared/test2/Sent
> >> doveadm(test1 at emailtest.net): Info: imapc(10.0.0.202:143): Connected to
> >> 10.0.0.202:143 (local 10.0.0.203:39698)
> >> doveadm(test1 at emailtest.net): Info: imapc(10.0.0.202:143): Connected to
> >> 10.0.0.202:143 (local 10.0.0.203:39700)
> >> doveadm(test1 at emailtest.net): Info: Mailbox 'Sent' is in namespace
> >> 'shared/test2/'
> >> doveadm(test1 at emailtest.net): Info: Mailbox path:
> >> /home/vmail/domains/emailtest.net/test2/Maildir/.Sent
> >> doveadm(test1 at emailtest.net): Info: All message flags are shared across
> >> users in mailbox
> >> doveadm(test1 at emailtest.net): Info: User test1 at emailtest.net has rights:
> >> lookup read write write-seen write-deleted insert expunge
> >> doveadm(test1 at emailtest.net): Info: Mailbox found from dovecot-acl-list
> >> doveadm(test1 at emailtest.net): Info: User test2 at emailtest.net found from
> >> ACL shared dict
> >> doveadm(test1 at emailtest.net): Info: Mailbox shared/test2/Sent is visible
> >> in LIST
> >>
> >> But the are still some issues, if the sharing ring is like "test2 share
> >> a folder with test1 that share a folder with test3 that share a folder
> >> test2" dovecot have a loop until max_user_connections is reached.
> >> Probably until option "acl_ignore_namespace" will be available we cannot
> >> solve this.
> >>
> >> Moreover, if both test1 and test2 mark as read/unread the same message
> >> in a shared folder dovecot have some indexes issue:
> >>
> >> Apr 25 21:03:56 Error: imap(test2 at emailtest.net)
> >> session=<H5EBptDA3IgKQgTL>: Mailbox Sent: Timeout (180s) while waiting
> >> for lock for transaction log file
> >> /home/vmail/domains/emailtest.net/test2/Maildir/.Sent/dovecot.index.log
> >> (WRITE lock held by pid 6181)
> >> Apr 25 21:03:56 Warning: imap(test1 at emailtest.net)
> >> session=<6Kn/pdDAHKxT07oG>: Transaction log file
> >> /home/vmail/domains/emailtest.net/test2/Maildir/.Sent/dovecot.index.log
> >> was locked for 180 seconds (Mailbox was synchronized)
> >> Apr 25 21:03:56 Warning: imap(test2 at emailtest.net)
> >> session=<7d2ppdDAFKxT07oG>: Locking transaction log file
> >> /home/vmail/domains/emailtest.net/test2/Maildir/.Sent/dovecot.index.log
> >> took 178 seconds (syncing)
> >>
> >> probably because INDEXPVT is not supported in imapc?
> >>
> >> So, I should wait until 2.3.15 to have shared mailboxes via imapc work
> >> fine or I missing something?
> > Hi,
> >
> > when configuring the shared namespace like this:
> >
> > location = imapc:%%h/Maildir
> >
> > It only works because imapc using Maildir format internally too. This way Dovecot is told to use the sharing users home as the location for imapc indexes for the current user. That is probably the reason for the troubles with the lock transaction file. You should change it to something like this:
> >
> > location = imapc:~/shared/%%u
> >
> > When configured like this, the current user will store indexes for accessing the shared mailbox under the "shared" prefix in it's home directory and create folders for each user that shared mailboxes to the current user.
> 
> Hi Markus,
> 
> I understand and agree that with "imapc:~/shared/%%u" indexes are stored 
> in the right home directory for each user but I still found an issue 
> with this configuration, and is that dovecot is looking for the 
> "dovecot-acl" file in the wrong path, so was unable to reading rights 
> for the mailbox, that is not visibile to user.
> 
> Please look these tests where I run:
> 
> doveadm -D acl debug -u test1 at emailtest.net shared/test2/Sent
> 
> with location = imapc:~/shared/%%u dovecot-acl was not found:
> 
> Apr 28 12:31:54 doveadm(test1 at emailtest.net): Debug: 
> imapc(10.66.4.202:143): Authenticated successfully
> Apr 28 12:31:54 doveadm(test1 at emailtest.net): Info: Mailbox 'Sent' is in 
> namespace 'shared/test2/'
> Apr 28 12:31:54 doveadm(test1 at emailtest.net): Info: Mailbox path: 
> /home/vmail/domains/emailtest.net/test1/shared/test2 at emailtest.net/.Sent
> Apr 28 12:31:54 doveadm(test1 at emailtest.net): Info: All message flags 
> are shared across users in mailbox
> Apr 28 12:31:54 doveadm(test1 at emailtest.net): Debug: acl vfile: file 
> /home/vmail/domains/emailtest.net/test1/shared/test2 at emailtest.net/.Sent/dovecot-acl 
> not found
> [...]
> Apr 28 12:31:54 doveadm(test1 at emailpnl.net): Info: Mailbox 
> shared/test2/Sent is NOT visible in LIST
> 
> with location = imapc:~/Maildir/shared/%%u dovecot-acl was not found:
> 
> Apr 28 12:40:01 doveadm(test1 at emailtest.net): Debug: 
> imapc(10.66.4.202:143): Authenticated successfully
> Apr 28 12:40:01 doveadm(test1 at emailtest.net): Info: Mailbox 'Sent' is in 
> namespace 'shared/test2/'
> Apr 28 12:40:01 doveadm(test1 at emailtest.net): Info: Mailbox path: 
> /home/vmail/domains/emailtest.net/test1/Maildir/shared/test2 at emailtest.net/.Sent
> Apr 28 12:40:01 doveadm(test1 at emailtest.net): Info: All message flags 
> are shared across users in mailbox
> Apr 28 12:40:01 doveadm(test1 at emailtest.net): Debug: acl vfile: file 
> /home/vmail/domains/emailtest.net/test1/Maildir/shared/test2 at emailtest.net/.Sent/dovecot-acl 
> not found
> [...]
> Apr 28 12:40:01 doveadm(test1 at emailpnl.net): Info: Mailbox 
> shared/test2/Sent is NOT visible in LIST
> 
> with location = imapc:%%h/Maildir dovecot-acl was found:
> 
> Apr 28 12:34:41 doveadm(test1 at emailtest.net): Debug: 
> imapc(10.66.4.202:143): Authenticated successfully
> Apr 28 12:34:41 doveadm(test1 at emailtest.net): Info: Mailbox 'Sent' is in 
> namespace 'shared/test2/'
> Apr 28 12:34:41 doveadm(test1 at emailtest.net): Info: Mailbox path: 
> /home/vmail/domains/emailtest.net/test2/Maildir/.Sent
> Apr 28 12:34:41 doveadm(test1 at emailtest.net): Info: All message flags 
> are shared across users in mailbox
> Apr 28 12:34:41 doveadm(test1 at emailtest.net): Debug: acl vfile: reading 
> file /home/vmail/domains/emailtest.net/test2/Maildir/.Sent/dovecot-acl
> [...]
> Apr 28 12:34:41 doveadm(test1 at emailpnl.net): Info: Mailbox 
> shared/test2/Sent is visible in LIST
> 
> with traditional mailbox sharing location = 
> maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u  dovecot-acl was found:
> 
> Apr 28 14:03:41 doveadm(test1 at emailtest.net): Info: Mailbox 'Sent' is in 
> namespace 'shared/test2/'
> Apr 28 14:03:41 doveadm(test1 at emailtest.net): Info: Mailbox path: 
> /home/vmail/domains/emailtest.net/test2/Maildir/.Sent
> Apr 28 14:03:41 doveadm(test1 at emailtest.net): Info: All message flags 
> are shared across users in mailbox
> Apr 28 14:03:41 doveadm(test1 at emailtest.net): Debug: acl vfile: reading 
> file /home/vmail/domains/emailtest.net/test2/Maildir/.Sent/dovecot-acl
> [...]
> Apr 28 14:03:41 doveadm(test1 at emailpnl.net): Info: Mailbox 
> shared/test2/Sent is visible in LIST
> 
> So, what is exactly the argument after "imapc:"?
> 
> Where dovecot store indexes for user test1 or where dovecot looks for 
> "dovecot-acl", that is under test2 home? Cannot be both since are things 
> in different user's home directory.
> 
> Or am I wrong?

Hi,

i think you are right, but the problem here is that Dovecot attempts to access the ACL locally because it does not yet have the "acl_ignore_namespace" setting. This is used to tell Dovecot to not check for ACLs for the given namespace prefix. So if no ACL locally is preventing the access to the shared namespace, an imapc connection will be established and ACLs will be checked on the backend where the connection arrives. Which can be the current one or another one. The access to the other backend then will be happening with the current user as master user, trying to access the sharing users mailbox, this is the situation where ACLs will/should be applied.

Markus

> 
> Thanks
> 
> -- 
> Alessio Cecchi
> Postmaster @ http://www.qboxmail.it
> https://www.linkedin.com/in/alessice