At a customer we run dovecot-2.3.10 along with postfix and mariadb, using postfixadmin as GUI.
Now I plan to migrate to a new hardware (bigger machine) and in the same step to migrate from Gentoo Linux to Debian Linux.
So I think about how to get the mailboxes from A to B.
We use one UID afaik:
mail_home = /home/vmail/%d/%u mail_location = maildir:~
I read about dsync and wonder:
does it make sense in this case to set up replication and let the servers replicate at first while still running on the old server?
do I have advantages over doing some rsync-jobs?
I somewhere read about not being able to sync sieve-scripts below the maildirs?
We have:
plugin { sieve = /home/vmail/%d/%u/sieve/.dovecot.sieve sieve_before = /etc/dovecot/sieve/global/default.sieve sieve_dir = /home/vmail/%d/%u/sieve/scripts/ sieve_extensions = +imapflags }
- would the replication sync over all the existing mails/mailboxes or only the ones created *after* starting the replication? Stupid question maybe, but I just want to be sure before I screw up the installation.
- sure, backups backups all the way
;-)
thanks for any feedback on this!
Stefan
- Stefan G. Weichinger:
does it make sense in this case to set up replication and let the servers replicate at first while still running on the old server?
If you use "dsync over TCP connections" [1], you can set up your new server without users noticing it (with the exception of extra server load for replication). It will take some time to sync all the data, but it takes no manual action once replication is fully set up.
[1] https://wiki.dovecot.org/Replication
The docs state one should be able to use a "noreplicate" field to limit the accounts, but I was not able to get it to work with a LDAP user DB. Regardless what value I provided in LDAP, replication would happen, and I could not find detailed documentation.
do I have advantages over doing some rsync-jobs?
Oh, you know, some small stuff, like data integrity checks, retry on failures, event-driven replication. ;-)
-Ralph
Am 15.04.20 um 20:36 schrieb Ralph Seichter:
- Stefan G. Weichinger:
does it make sense in this case to set up replication and let the servers replicate at first while still running on the old server?
If you use "dsync over TCP connections" [1], you can set up your new server without users noticing it (with the exception of extra server load for replication). It will take some time to sync all the data, but it takes no manual action once replication is fully set up.
[1] https://wiki.dovecot.org/Replication
The docs state one should be able to use a "noreplicate" field to limit the accounts, but I was not able to get it to work with a LDAP user DB. Regardless what value I provided in LDAP, replication would happen, and I could not find detailed documentation.
do I have advantages over doing some rsync-jobs?
Oh, you know, some small stuff, like data integrity checks, retry on failures, event-driven replication. ;-)
Ah, sure, nice ;-)
I have the config ready on the "old" server, so we're waiting for the new box to be delivered. As always with new sync tools I am cautious to not lose data by doing something wrong (like syncing the empty server to the productive one ;-) ). As far as I see that can't happen here ...
For sure I already read [1] ... and I have this block prepared now (as far as I understand I need all of this):
## 2020-04 replication, @sgw # mail_plugins = $mail_plugins notify replication
service aggregator { fifo_listener replication-notify-fifo { user = vmail mode = 0666 } unix_listener replication-notify { user = vmail mode = 0666 } }
service config { unix_listener config { user = vmail } }
service replicator { unix_listener replicator-doveadm { mode = 0666 } }
service doveadm { inet_listener { port = 12345 } }
doveadm_port = 12345 doveadm_password = aadCdsfxxxx
plugin { mail_replica = tcp:anotherhost.example.com:12345 # use port 12345 explicitly }
-- at least it didn't throw errors at a first test ;-) (dry run)
On 15 Apr 2020, at 07:50, Stefan G. Weichinger <lists@xunil.at> wrote:
So I think about how to get the mailboxes from A to B.
We use one UID afaik:
mail_home = /home/vmail/%d/%u mail_location = maildir:~
I read about dsync and wonder:
- does it make sense in this case to set up replication and let the servers replicate at first while still running on the old server?
I ave done this (dovevot mailers under a email user) and I simply used rsync to copy the mail to the new machine.
Once I had synced, I then synced again, shutdown the mail server, synced on last time, and started the mail server on the new hardware. The last sync only had to catch up a few emails, so all told I was down for a shorter time than a simple reboot.
That said, doing it via replication would have been “better” but in my case wasn’t worth the time it would have taken to setup, test, test, backup all the mail an extra time anyway, and then implement.
I’m about to do it all over again since old machine running the Mailserver cannot seem to boot properly to install FreeNSD amd64 despite being a 64bit Intel (It’s a BIOS/EFI issue), so I am dropping in a new, albeit still old, replacement with a zpool of three SSD drives (up from the current setup of 2 SSDs with backups on the second drive and on a second machine and on a remote machine)
—
Am 17.04.20 um 03:37 schrieb @lbutlr:
On 15 Apr 2020, at 07:50, Stefan G. Weichinger <lists@xunil.at> wrote:
So I think about how to get the mailboxes from A to B.
We use one UID afaik:
mail_home = /home/vmail/%d/%u mail_location = maildir:~
I read about dsync and wonder:
- does it make sense in this case to set up replication and let the servers replicate at first while still running on the old server?
I ave done this (dovevot mailers under a email user) and I simply used rsync to copy the mail to the new machine.
Once I had synced, I then synced again, shutdown the mail server, synced on last time, and started the mail server on the new hardware. The last sync only had to catch up a few emails, so all told I was down for a shorter time than a simple reboot.
That said, doing it via replication would have been “better” but in my case wasn’t worth the time it would have taken to setup, test, test, backup all the mail an extra time anyway, and then implement.
thanks for that feedback
Today I will setup the new server and try to enable replication.
I will connect both dovecots to the same instance of mariadb which should let them know about all the existing mailboxes ... and then I will see.
My question around the sieve-scripts is still open ... I assume I will maybe try to replicate a sub-set of the mailboxes, check things ... and then decide how to proceed.
Am 21.04.20 um 11:49 schrieb Stefan G. Weichinger:
I will connect both dovecots to the same instance of mariadb which should let them know about all the existing mailboxes ... and then I will see.
My question around the sieve-scripts is still open ... I assume I will maybe try to replicate a sub-set of the mailboxes, check things ... and then decide how to proceed.
So far it works great, only two mailboxes marked inactive via postfixadmin are skipped at sync.
Do I have to enable them to get them synced as well?
My queries:
user_query = SELECT '/home/vmail/%d/%u' as home, 'maildir:/home/vmail/%d/%u' as mail, 5000 AS uid, 5000 AS gid, concat('dirsize:storage=', quota) AS quota FROM mailbox WHERE (username = '%u' OR username = '%u@mydomain.tld') AND active = '1'
password_query = SELECT username as user, password, '/home/vmail/%d/%u' as userdb_home, 'maildir:/home/vmail/%d/%u' as userdb_mail, 5000 as userdb_uid, 5000 as userdb_gid FROM mailbox WHERE (username = '%u' OR username = '%u@mydomain.tld') AND active = '1'
iterate_query = SELECT username as user FROM mailbox
Am 22.04.20 um 09:41 schrieb Stefan G. Weichinger:
iterate_query = SELECT username as user FROM mailbox
Ok, we enabled the 2 users and they synced (after a restart of one dovecot, btw).
What irritates me:
on the new server I find less files in /home/vmail than on the old server.
So something is skipped or so.
I try to compare now ..
On 2020-04-22 10:35, Stefan G. Weichinger wrote:
Am 22.04.20 um 09:41 schrieb Stefan G. Weichinger:
iterate_query = SELECT username as user FROM mailbox
Ok, we enabled the 2 users and they synced (after a restart of one dovecot, btw).
What irritates me:
on the new server I find less files in /home/vmail than on the old server.
So something is skipped or so.
I try to compare now ..
# on the old doveadm quota recalc -A doveadm quota get -A
# on the new doveadm quota recalc -A doveadm quota get -A
then you can see with users missing data
i dont know how to sync missing data, all i know is i use aid4mail that solves it for me, just one bug in it that it makes dupe mails if running aid4mail twice
Am 22.04.20 um 10:43 schrieb Benny Pedersen:
On 2020-04-22 10:35, Stefan G. Weichinger wrote:
Am 22.04.20 um 09:41 schrieb Stefan G. Weichinger:
iterate_query = SELECT username as user FROM mailbox
Ok, we enabled the 2 users and they synced (after a restart of one dovecot, btw).
What irritates me:
on the new server I find less files in /home/vmail than on the old server.
So something is skipped or so.
I try to compare now ..
# on the old doveadm quota recalc -A doveadm quota get -A
# on the new doveadm quota recalc -A doveadm quota get -A
then you can see with users missing data
ok, I try.
On the new machine the results are there instantly (SSDs and fast CPU etc), the old one still calculates.
"quota" plugin wasn't enabled, that might add some initial work. Heavy load now ... ~45 : I had to stop that now
I saw the first few lines of output and the values all are slightly different between the 2 servers.
I will ask the admin there to compare mailboxes via client, in thunderbird for example.
On 22/04/2020 11:35 Stefan G. Weichinger <lists@xunil.at> wrote:
Am 22.04.20 um 09:41 schrieb Stefan G. Weichinger:
iterate_query = SELECT username as user FROM mailbox
Ok, we enabled the 2 users and they synced (after a restart of one dovecot, btw).
What irritates me:
on the new server I find less files in /home/vmail than on the old server.
So something is skipped or so.
I try to compare now ..
You can also try running doveadm sync by hand.
doveadm -Dv sync -u victim tcp:otherserver:port
Aki
On 22/04/2020 17:13 Benny Pedersen <me@junc.eu> wrote:
On 2020-04-22 10:45, Aki Tuomi wrote:
You can also try running doveadm sync by hand. doveadm -Dv sync -u victim tcp:otherserver:port
what doeveconf config makes that command line work ?
Something like this:
on target host
doveadm_password=foobar
service doveadm { inet_listener { port = 12345 } }
on source host
doveadm_password=foobar
Aki
Am 22.04.20 um 10:35 schrieb Stefan G. Weichinger:
Am 22.04.20 um 09:41 schrieb Stefan G. Weichinger:
iterate_query = SELECT username as user FROM mailbox
Ok, we enabled the 2 users and they synced (after a restart of one dovecot, btw).
All users synced.
What if I disable the 2 users now (= inactive in postfixadmin) ?
I assume the 2 dovecots would simply keep the mailboxes "as is" and not remove anything, correct?
thanks ;-)
participants (5)
-
＠lbutlr
-
Aki Tuomi
-
Benny Pedersen
-
Ralph Seichter
-
Stefan G. Weichinger