AW: Problem with copy e-mails via doveadm

[Christian Küppers]

(sorry for that, missed to change from html to text version )
-----------------------

Further help appreciated on this topic.

further question/option:
If the proxy enable part in password_query is the main problem. Would it
work if I create a single new dovecot instance (in new vm) with same
configuration as our proxy&director instances but without the "'y' AS proxy"
part? Would this change copy the e-mails from backend servers "shard1" to
backend servers "shard2" without proxing command to "shard2"?
 

Christian

 



----- Ursprüngliche Nachricht -----
Von: Christian Küppers c.kueppers at onoffice.de
Gesendet: Freitag, 20. August 2021 14:41:26
An: aki.tuomi at open-xchange.com
Cc: dovecot at dovecot.org
Betreff: Re: AW: Problem with copy e-mails via doveadm

That doesn't work / has no visible effect.

mail/location/mail_location gets overwritten by "-o mail_location=imapc:" in
doveadm command.

Is it possible to "bind" options/values to users in doveadm command, like
"doveadm -o target_mail_location=imapc: -o
source_mail_location=mbox:~/mail:INBOX=/var/mail/user -o
target_mail_host=<dns> ..."?

Other optional way:
Is doveadm capable of handling different configurations (e.g. backends,
received from userdb) for different given users in one command?
Is it possible to disable proxy of my doveadm command to backend if I run it
on a production/actively used director&proxy server without changing running
configuration? I tried adding "-o proxy=n" to my doveadm command but without
effect.

I ask these questions because from my point of view the director is the only
server to run this copy command on and has the knowledge of both users
servers to connect to (if command wouldn't get proxied).

Christian

----- Ursprüngliche Nachricht -----
Von: Aki Tuomi aki.tuomi at open-xchange.com
Gesendet: Freitag, 20. August 2021 12:46:01
An: c.kueppers at onoffice.de
Cc: dovecot at dovecot.org
Betreff: Re: AW: Problem with copy e-mails via doveadm

Run the command on the target host, and change

user_query = SELECT '/vmail/%Ld/%Ln' AS home,
'mbox:~/mail:INBOX=/var/mail/%u' AS mail, 10000 AS uid, 10000 AS gid FROM
users WHERE email = '%Lu'

Aki

> On 20/08/2021 13:39 Christian Küppers c.kueppers at onoffice.de wrote:
>
>
> Please explain in more detail how I can do this.
>
> > Try targeting your director instead.
> in cmd of director
> /usr/bin/doveadm -Dv -o mail_location=imapc: -o imapc_host=<director-dns>
-o imapc_user="source_user_shard1" -o
imapc_password="source_user_shard1_password" copy -u "dest_user_shard2"
"dest_folder" user "source_user_shard1" mailbox "source_folder" ALL
> does no change, like i said.
>
> in cmd of backend shard2
> /usr/bin/doveadm -Dv -o mail_location=imapc: -o imapc_host=<director-dns>
-o imapc_user="source_user_shard1" -o
imapc_password="source_user_shard1_password" copy -u "dest_user_shard2"
"dest_folder" user "source_user_shard1" mailbox "source_folder" ALL
> results in same output than targeting backend shard1 direct.
>
> > The problem actually is that you are now targeting the source user into
the source user as well. You need to, somehow, make dovecot return
mail=imapc: for the target user. Or you can try to do the copying on the
target backend instead, so that you can return `mail=whatever your mail
location is` from userdb lookup easier.
>
> mysql dovecot configuration part:
> user_query = SELECT '/vmail/%Ld/%Ln' AS home, 10000 AS uid, 10000 AS gid
FROM users WHERE email = '%Lu'
> password_query = SELECT email AS user, password, 10000 AS userdb_uid,
10000 AS userdb_gid, '/vmail/%Ld/%Ln' AS userdb_home FROM users WHERE email
= '%Lu' AND active = '1'
>
> Isn't my described attempt on backend of dest_user_shard2 exactly what you
described as possible next try? If not what has to be changed?
>
> Christian
>
>
>
> ----- Ursprüngliche Nachricht -----
> Von: Aki Tuomi aki.tuomi at open-xchange.com
> Gesendet: Freitag, 20. August 2021 12:14:54
> An: ckueppers at onoffice.de, dovecot at dovecot.org
> Betreff: Re: AW: Problem with copy e-mails via doveadm
>
> The problem actually is that you are now targeting the source user into
the
> source user as well. You need to, somehow, make dovecot return mail=imapc:
> for the target user. Or you can try to do the copying on the target
backend
> instead, so that you can return `mail=whatever your mail location is` from
> userdb lookup easier.
>
> Aki
>
> > On 20/08/2021 13:09 Aki Tuomi aki.tuomi at open-xchange.com wrote:
> >
> >
> > Try targeting your director instead.
> >
> > Aki
> >
> > > On 20/08/2021 12:45 Christian Küppers c.kueppers at onoffice.de wrote:
> > >
> > >
> > > Okay, i need some further help. 
> > >
> > > What i've tried with your hint:
> > >
> > > Executing on cmd of director&proxy server:
> > > /usr/bin/doveadm -Dv -o mail_location=imapc: copy -u
"dest_user_shard2"
> "dest_folder" user "source_user_shard1" mailbox "source_folder" ALL
> > > /usr/bin/doveadm -Dv -o mail_location=imapc: -o
> imapc_host=<director-dns> -o imapc_user="source_user_shard1" -o
> imapc_password="source_user_shard1_password" copy -u "dest_user_shard2"
> "dest_folder" user "source_user_shard1" mailbox "source_folder" ALL
> > >
> > > I've also tried to execute "doveadm -c" with copied and modified
> configuration on director&proxy server without "'y' AS proxy" in sql
> password_query configuration part - without luck.
> > > This leads all to same debug output and result as command in first
post.
> > >
> > > After that i changed to cmd of backend server of dest_user_shard2 and
> tried:
> > > /usr/bin/doveadm -Dv -o mail_location=imapc: -o
> imapc_host=<backend-shard1-dns> -o imapc_user="source_user_shard1" -o
> imapc_password="source_user_shard1_password" copy -u "dest_user_shard2"
> "dest_folder" user "source_user_shard1" mailbox "source_folder" ALL
> > > [..]
> > > Aug 20 08:52:38
> doveadm(source_user_shard1)<2442101><qAs8KVZtH2F1QyUA5QDIzw>: Debug:
> imapc(<backend-shard1-dns>:143): Authenticated successfully
> > > Aug 20 08:52:38
> doveadm(source_user_shard1)<2442101><qAs8KVZtH2F1QyUA5QDIzw>: Debug:
imapc:
> root=, index=, indexpvt=, control=, inbox=, alt=
> > > Aug 20 08:52:38
> doveadm(source_user_shard1)<2442101><qAs8KVZtH2F1QyUA5QDIzw>: Debug:
quota:
> quota_over_flag check: quota_over_script unset - skipping
> > > Aug 20 08:52:38 doveadm(dest_user_shard2): Debug: Mailbox dest_folder:
> Mailbox opened because: copy
> > > Aug 20 08:52:38 doveadm(dest_user_shard2): Debug: Mailbox
source_folder:
> Mailbox opened because: copy
> > > Aug 20 08:52:38 doveadm(dest_user_shard2): Debug: Mailbox
source_folder:
> UID 1: Opened mail because: copying
> > > Aug 20 08:52:38 doveadm(dest_user_shard2): Error: Copying message UID
1
> from 'source_folder' failed: Mailbox doesn't exist: dest_folder (0.001 +
> 0.000 secs).
> > > Aug 20 08:52:38 doveadm(dest_user_shard2): Debug: Mailbox
source_folder:
> UID 2: Opened mail because: copying
> > > Aug 20 08:52:38 doveadm(dest_user_shard2): Error: Copying message UID
2
> from 'source_folder' failed: Mailbox doesn't exist: dest_folder (0.001 +
> 0.000 secs).
> > > Aug 20 08:52:38 doveadm(dest_user_shard2): Debug: Mailbox
source_folder:
> UID 3: Opened mail because: copying
> > > Aug 20 08:52:38 doveadm(dest_user_shard2): Error: Copying message UID
3
> from 'source_folder' failed: Mailbox doesn't exist: dest_folder (0.001 +
> 0.000 secs).
> > > Aug 20 08:52:38 doveadm(dest_user_shard2): Debug: Mailbox
source_folder:
> UID 4: Opened mail because: copying
> > > Aug 20 08:52:38 doveadm(dest_user_shard2): Error: Copying message UID
4
> from 'source_folder' failed: Mailbox doesn't exist: dest_folder (0.001 +
> 0.000 secs).
> > > Aug 20 08:52:38 doveadm(dest_user_shard2): Debug: Mailbox
source_folder:
> UID 5: Opened mail because: copying
> > > Aug 20 08:52:38 doveadm(dest_user_shard2): Error: Copying message UID
5
> from 'source_folder' failed: Mailbox doesn't exist: dest_folder (0.001 +
> 0.000 secs).
> > > Aug 20 08:52:38 doveadm(dest_user_shard2): Error: Syncing mailbox
> 'dest_folder' failed: Mailbox doesn't exist: dest_folder (0.001 + 0.000
> secs).
> > > Aug 20 08:52:38 doveadm(dest_user_shard2): Debug:
> imapc(<backend-shard1-dns>:143): Disconnected
> > > Aug 20 08:52:38 doveadm(dest_user_shard2): Debug:
> imapc(<backend-shard1-dns>:143): Disconnected
> > > Aug 20 08:52:38 doveadm(2442101): Debug: auth-master: conn
> unix:/var/run/dovecot/auth-userdb (pid=2630417,uid=0): Disconnected:
> Connection closed (fd=9)
> > >
> > > after creating dest_folder in source_user_shard1 mailbox output
changed
> to:
> > > [..]
> > > Aug 20 08:56:11
> doveadm(source_user_shard1)<2446702><0AmaCStuH2FuVSUA5QDIzw>: Debug:
> imapc(<backend-shard1-dns>:143): Authenticated successfully
> > > Aug 20 08:56:11
> doveadm(source_user_shard1)<2446702><0AmaCStuH2FuVSUA5QDIzw>: Debug:
imapc:
> root=, index=, indexpvt=, control=, inbox=, alt=
> > > Aug 20 08:56:11
> doveadm(source_user_shard1)<2446702><0AmaCStuH2FuVSUA5QDIzw>: Debug:
quota:
> quota_over_flag check: quota_over_script unset - skipping
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Debug: Mailbox dest_folder:
> Mailbox opened because: copy
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Debug: Mailbox
source_folder:
> Mailbox opened because: copy
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Debug: Mailbox
source_folder:
> UID 1: Opened mail because: copying
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Debug: Mailbox dest_folder:
> saving UID 1: Opened mail
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Debug: Mailbox
source_folder:
> UID 2: Opened mail because: copying
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Debug: Mailbox dest_folder:
> saving UID 2: Opened mail
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Debug: Mailbox
source_folder:
> UID 3: Opened mail because: copying
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Debug: Mailbox dest_folder:
> saving UID 3: Opened mail
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Debug: Mailbox
source_folder:
> UID 4: Opened mail because: copying
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Debug: Mailbox dest_folder:
> saving UID 4: Opened mail
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Debug: Mailbox
source_folder:
> UID 5: Opened mail because: copying
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Debug: Mailbox dest_folder:
> saving UID 5: Opened mail
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Debug: replication:
> Replication requested by 'cmd_copy_box', priority=2
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Error: Syncing mailbox
> 'dest_folder' failed: BUG: Unknown internal error
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Debug:
> imapc(<backend-shard1-dns>:143): Disconnected
> > > Aug 20 08:56:11 doveadm(dest_user_shard2): Debug:
> imapc(<backend-shard1-dns>:143): Disconnected
> > > Aug 20 08:56:11 doveadm(2446702): Debug: auth-master: conn
> unix:/var/run/dovecot/auth-userdb (pid=2630417,uid=0): Disconnected:
> Connection closed (fd=9)
> > >
> > > This time e-mails get copied in source_user_shard1 from source_folder
to
> dest_folder although dest_user_shard2 is given in doveadm command and is
> different from source_user_shard1.
> > > It is not what I expected and desired.
> > >
> > > So yeah i haven't get the whole picture right now.
> > > Maybe you can give me some details.
> > >
> > > Christian
> > >
> > >
> > >
> > > ----- Ursprüngliche Nachricht -----
> > > Von: Aki Tuomi aki.tuomi at open-xchange.com
> > > Gesendet: Freitag, 20. August 2021 09:06:31
> > > An: c.kueppers at onoffice.de
> > > Cc: dovecot at dovecot.org
> > > Betreff: Re: AW: Problem with copy e-mails via doveadm
> > >
> > > The copy command gets proxied to the remote server because you used
> proxy=y.
> > > To do this kind of copying, you need to specify mail_location=imapc:
and
> > > target to the director.
> > >
> > > Aki
> > >
> > > > On 20/08/2021 09:53 Christian Küppers c.kueppers at onoffice.de wrote:
> > > >
> > > >
> > > > Maybe i missunderstood this message. For me this indicates that
> doveadm
> > > is searching on the wrong server (backend) for "source_user_shard1"
> 's 
> > > and
> > > > folder "source_folder". From my understanding doveadm has to look
> > > on 192.168.0.11 or 192.168.0.12 (according to provided
configuration),
> but
> > > the
> > > > message says 192.168.0.21 which is backend for "dest_user_shard2".
> > > >
> > > >
> > > >
> > > > ----- Ursprüngliche Nachricht -----
> > > > Von: Aki Tuomi aki.tuomi at open-xchange.com
> > > > Gesendet: Freitag, 20. August 2021 08:39:41
> > > > An: c.kueppers at onoffice.de, dovecot at dovecot.org
> > > > Betreff: Re: AW: Problem with copy e-mails via doveadm
> > > >
> > > > Aug 17 10:22:05 doveadm(dest_user_shard2): Debug: auth-master:
passdb
> > > > lookup(dest_user_shard2): Finished passdb lookup
> (user=dest_user_shard2
> > > > proxy=y director_tag=shard2 proxy_refresh=450 host=192.168.0.21)
> > > > Aug 17 10:22:05 doveadm(dest_user_shard2): Error:
> > > > remote(192.168.0.21:24245): Mailbox source_folder: Mailbox sync
> failed:
> > > > Mailbox doesn't exist: source_folder
> > > >
> > > > Does this help?
> > > >
> > > > Aki
> > > >
> > > > > On 20/08/2021 09:12 Christian Küppers c.kueppers at onoffice.de
wrote:
> > > > >
> > > > >
> > > > > Try to republish due to unreadable message (in html format) in
> mailing
> > > > list archive
> > > > > maybe someone can take a better look at it now
> > > > >
> > > > >
> > > >
> > >
>
-------------------------------------------------------------------------------------------
> > > > >
> > > > > Hi,
> > > > >
> > > > > i want to copy e-mails from user "source_user_shard1" and folder
> > > > "source_folder" lying on backend servers (shard1) to another
> > > > > user "dest_user_shard2" and folder "dest_folder" lying on
different
> > > > backend servers (shard2) via command line tool doveadm copy
> > > > > executed on dovecot proxy&director servers. But doveadm seems to
> look
> > > only
> > > > on "dest_user_shard2"'s backend for folder "source_folder"
> > > > > of user "source_user_shard1" and can not find it. To confirm this,
> i've
> > > > checked directories on filesystem on backend of "dest_user_shard2"
> > > > > and a folder for "source_user_shard1" was created including
> one folder
> > > > "mdbox" and only one file "dovecot.list.index.log" in it. Folder and
> file
> > > > > timestamps match the command debug output time.
> > > > > "doveadm list" executed for both users show the right folders
> > > > (source_folder and dest_folder exist in respective mailbox).
> > > > >
> > > > > Is it a bug or wrong usage of tool? Please advice.
> > > > >
> > > > > command executed on dovecot proxy&director server with debug
output:
> > > > > /usr/bin/doveadm -Dv copy -u "dest_user_shard2" "dest_folder" user
> > > > "source_user_shard1" mailbox "source_folder" ALL
> > > > > Aug 17 10:22:05 Debug: Loading modules from directory:
> > > > /usr/lib/dovecot/modules
> > > > > Aug 17 10:22:05 Debug: Module loaded:
> > > > /usr/lib/dovecot/modules/lib10_quota_plugin.so
> > > > > Aug 17 10:22:05 Debug: Loading modules from directory:
> > > > /usr/lib/dovecot/modules/doveadm
> > > > > Aug 17 10:22:05 Debug: Skipping module doveadm_acl_plugin, because
> > > > dlopen() failed:
> > > > /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so:
> > > > > undefined symbol: acl_user_module (this is usually intentional, so
> just
> > > > ignore this message)
> > > > > Aug 17 10:22:05 Debug: Module loaded:
> > > > /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so
> > > > > Aug 17 10:22:05 Debug: Module loaded:
> > > > /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so
> > > > > Aug 17 10:22:05 Debug: Skipping module doveadm_fts_plugin, because
> > > > dlopen() failed:
> > > > /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so:
> > > > > undefined symbol: fts_user_get_language_list (this is usually
> > > intentional,
> > > > so just ignore this message)
> > > > > Aug 17 10:22:05 doveadm(dest_user_shard2): Debug: auth-master:
> passdb
> > > > lookup(dest_user_shard2): Started passdb lookup
> > > > > Aug 17 10:22:05 doveadm(dest_user_shard2): Debug: auth-master:
conn
> > > > unix:/var/run/dovecot/director-userdb: Connecting
> > > > > Aug 17 10:22:05 doveadm(dest_user_shard2): Debug: auth-master:
conn
> > > > unix:/var/run/dovecot/director-userdb (pid=647,uid=0): Client
> connected
> > > > > (fd=9)
> > > > > Aug 17 10:22:05 doveadm(dest_user_shard2): Debug: auth-master:
> passdb
> > > > lookup(dest_user_shard2): auth PASS input: user=dest_user_shard2
> proxy=y
> > > > > director_tag=shard2 proxy_refresh=450 host=192.168.0.21
> > > > > Aug 17 10:22:05 doveadm(dest_user_shard2): Debug: auth-master:
> passdb
> > > > lookup(dest_user_shard2): Finished passdb lookup
> (user=dest_user_shard2
> > > > > proxy=y director_tag=shard2 proxy_refresh=450 host=192.168.0.21)
> > > > > Aug 17 10:22:05 doveadm(dest_user_shard2): Error:
> > > > remote(192.168.0.21:24245): Mailbox source_folder: Mailbox sync
> failed:
> > > > > Mailbox doesn't exist: source_folder
> > > > > Aug 17 10:22:05 doveadm(dest_user_shard2): Debug: auth-master:
conn
> > > > unix:/var/run/dovecot/director-userdb (pid=647,uid=0): Disconnected:
> > > > > Connection closed (fd=9)
> > > > >
> > > <snip/>
> > >
> > > > >
> > > > > Kind regards,
> > > > >
> > > > >  Christian Küppers
> > > > > Expert Administrator
> > > > >
> > > > >  
> > > > >
> > > > > onOffice GmbH
> > > > > Charlottenburger Allee 5 | 52068 Aachen
> > > > > Tel. +49 (0)241 446 86-0 | Fax. +49 (0)241 446 86-250
> > > > > E-Mail:c.kueppers at onoffice.de| Web:www.onOffice.com
> > > > >
> > > > >  
> > > > > Registergericht: Amtsgericht Aachen, HRB 21420
> > > > > Geschäftsführer: Dipl.-Kfm. Stefan Mantl
> > > > > Prokuristen: Janosch Reuschenbach, Kristina Andresen, Christian
> > > Mähringer