Hi,
I recently performed a mail server migration for a client under fairly serious time constraints (ahem, yes one of those jobs). I would normally use imapsync to get all the mail copied to the new server, but under the circumstances, I had to write a script to rsync each of the mailboxes over ssh, from one server to another.
This all worked fine, except for the fact that if a client was using a POP email client, when they connected to the new server, they re-downloaded all their mail, creating duplicates of everything in their Outlook client. Of course they weren't happy about this. IMAP email clients were OK. No duplicates, and everything was fine.
So while I was migrating the mail, I did try for a while to understand the format of the UID files, but failed to do so in the available time, so the client just had to deal with duplicate emails. But now the smoke has cleared, I'd like to understand the problem a little better, and I was hoping someone on this forum could explain it to me, and the changes I'd need to make to the files so that the POP client DIDN'T download the duplicate emails.
Here's what I understand. The list of dovecot files in an account looks like this: -rw-r----- 1 admin mail 0 Jul 11 2018 dovecot-acl-list -rw-r----- 1 admin mail 14K Sep 20 08:36 dovecot.index -rw-r----- 1 admin mail 216K Sep 20 17:24 dovecot.index.cache -rw-r----- 1 admin mail 29K Sep 20 17:24 dovecot.index.log -rw-r----- 1 admin mail 388 Dec 17 2017 dovecot.list.index -rw-r----- 1 admin mail 1.6K Jul 11 2018 dovecot.list.index.log -rw-r----- 1 admin mail 24 Jul 11 2018 dovecot.mailbox.log -rw-r----- 1 admin mail 53 Sep 12 09:42 dovecot-quota -rw-r----- 1 admin mail 52K Sep 20 17:24 dovecot-uidlist -rw-r----- 1 admin mail 8 Jul 11 2018 dovecot-uidvalidity -rw-r----- 1 admin mail 0 Jul 4 2016 dovecot-uidvalidity.5779bbeb
It seems that the dovecot-uidlist is an index of all the mail files in the account. As far as I can gather, this should be the same on both servers after I've rsynced them over ssh: The files and the list of them in dovecot-uidlist should correspond. So what is it that the POP client looks at and decides to re-download all the emails? And how could I have stopped it from doing that, and get it to realize that the list of emails it holds internally is the same as the list on the server? What would I need to change in the dovecot files?
Thanks for any insight into the process.
As a secondary question -- and perhaps I should put this in a separate message -- I did notice that "doveadm sync" would apparently have helped me with this, but I wasn't able to get that to work either. I believe it was something to do with the fact that all the mailboxes were under the same linux user account on the target server, and I couldn't figure out all the paths and permissions in time. Would 'doveadm sync' have fixed all the UID and duplicate POP email issues?
P.
On Sep 20, 2019, at 10:38 PM, Plutocrat plutocrat@gmail.com wrote:
I recently performed a mail server migration for a client under fairly serious time constraints (ahem, yes one of those jobs). I would normally use imapsync to get all the mail copied to the new server, but under the circumstances, I had to write a script to rsync each of the mailboxes over ssh, from one server to another.
This all worked fine, except for the fact that if a client was using a POP email client, when they connected to the new server, they re-downloaded all their mail, creating duplicates of everything in their Outlook client. Of course they weren't happy about this. IMAP email clients were OK. No duplicates, and everything was fine.
Dealing with weird POP3 things h is why I disabled it on my server more than a decade ago; users have to beg for POP access and promise they will ONLY use it to get their mail into gmail.
So while I was migrating the mail, I did try for a while to understand the format of the UID files, but failed to do so in the available time, so the client just had to deal with duplicate emails. But now the smoke has cleared, I'd like to understand the problem a little better, and I was hoping someone on this forum could explain it to me, and the changes I'd need to make to the files so that the POP client DIDN'T download the duplicate emails.
Did you check https://wiki.dovecot.org/Migration? It has a lot of info on this.
Seems dsync would hav been the best way to do this?
https://wiki.dovecot.org/Migration/Dsync
As a secondary question -- and perhaps I should put this in a separate message -- I did notice that "doveadm sync" would apparently have helped me with this, but I wasn't able to get that to work either. I believe it was something to do with the fact that all the mailboxes were under the same linux user account on the target server, and I couldn't figure out all the paths and permissions in time. Would 'doveadm sync' have fixed all the UID and duplicate POP email issues?
Don’t know, but dsync says it does this:
"The pop3-migration plugin is used to preserve POP3 UIDLs. When dsync is handling IMAP INBOX and requests a POP3 UIDL, the plugin connects to the POP3 server and figures out which IMAP messages match which POP3 messages and then returns the appropriate POP3 UIDL.”
Trouble is, if you are migrating POP and the server is not up, I am not sure what you can do with dsync?
-- @mdhughes: One of the few regrets I have about lawn-less apartments: Shallow graves are so much harder to come by.
On 22/09/2019 4:08 AM, @lbutlr via dovecot wrote:
So while I was migrating the mail, I did try for a while to understand the format of the UID files, but failed to do so in the available time, so the client just had to deal with duplicate emails. But now the smoke has cleared, I'd like to understand the problem a little better, and I was hoping someone on this forum could explain it to me, and the changes I'd need to make to the files so that the POP client DIDN'T download the duplicate emails.
Did you check https://wiki.dovecot.org/Migration? It has a lot of info on this.
I did see that page, but was unable to figure out which of the three UID related files I needed to keep, and which to edit in order to 'trick' the email client into not re-downloading the mails.
The migration was from dovecot 2.x to 2.x, so I don't think there was any translation issue between servers. The POP clients were Outlook (my old nemesis). The only information I could find is that the dovecot UIDL format is pop3_uidl_format = %v.%u But that didn't really take me anywhere. So its all a bit of a mystery. I was hoping there would be a UID guru on this list who could walk me through it, or point me to some reference.
Don’t know, but dsync says it does this: "The pop3-migration plugin is used to preserve POP3 UIDLs. When dsync is handling IMAP INBOX and requests a POP3 UIDL, the plugin connects to the POP3 server and figures out which IMAP messages match which POP3 messages and then returns the appropriate POP3 UIDL.” Trouble is, if you are migrating POP and the server is not up, I am not sure what you can do with dsync?
I did try to use dsync (aka doveadm sync) for a few hours, but couldn't get a test account to migrate across. It seems like this would have been the solution but I was unable to confirm. At times it seemed to login and complete without error, but I could never find where it had put the mail! Other times it refused to login or gave permission errors. I looked around for real life examples, but wasn't able to find any.
Had I got it to work, I could maybe have figured out the UID problem. Maybe that will be a project for me in the future, when I have a spare few hours!
Maybe if I gave a few details of the directory structure that would help. I was migrating from a Cpanel installation to a standalone mail server. In Cpanel, there was the admin account login, and under that ~/mail/domain1.com/mailbox1 ~/mail/domain1.com/mailbox2 ~/mail/domain2.com/mailbox1 ~/mail/domain2.com/mailbox2
On the target server the same structure existed. One login, admin, and under that account ~/mail/domain1.com/mailbox1 ~/mail/domain1.com/mailbox2 ~/mail/domain2.com/mailbox1 ~/mail/domain2.com/mailbox2 All files and directories were chown admin:mail
The command I used was doveadm sync -u test@domain.com ssh -i id_rsa -o "StrictHostKeyChecking=no" admin@100.110.120.130
I tried a few options to tell it where to put the mail, but dovecot on the target server didn't seem to know where each account was located.
P.
On 23 Sep 2019, at 8.20, Plutocrat via dovecot dovecot@dovecot.org wrote:
The command I used was doveadm sync -u test@domain.com ssh -i id_rsa -o "StrictHostKeyChecking=no" admin@100.110.120.130
I tried a few options to tell it where to put the mail, but dovecot on the target server didn't seem to know where each account was located.
doveadm backup -u test@domain.com mailto:test@domain.com -R ssh sshuser@remote.com mailto:sshuser@remote.com "sudo /usr/bin/doveadm dsync-server -u test@domain.com mailto:test@domain.com"
run this on the target system to which you are migrating to.
HTH
Sami
On 23/09/2019 5:01 PM, Sami Ketola wrote:
doveadm backup -u test@domain.com mailto:test@domain.com -R ssh sshuser@remote.com mailto:sshuser@remote.com "sudo /usr/bin/doveadm dsync-server -u test@domain.com mailto:test@domain.com" run this on the target system to which you are migrating to.
Seems counter-intuitive to run the command on the target system! Pull, rather than push. However that wasn't an option for me at the time. I could only make a connection outbound from the source server, as ssh was locked down, and I was forced to use Cpanel's 'Terminal' App.
So maybe that was the problem them. I could only push and not pull.
P.
On Sep 22, 2019, at 11:20 PM, Plutocrat plutocrat@gmail.com wrote:
doveadm sync -u
Did the target machine already have the user setup? I think dsync wants to sync mailboxes between configured and working servers with users already defined.
-- "Back off, man. I'm a scientist.”
On 24 Sep 2019, at 17.14, @lbutlr via dovecot dovecot@dovecot.org wrote:
On Sep 22, 2019, at 11:20 PM, Plutocrat plutocrat@gmail.com wrote:
doveadm sync -u
Did the target machine already have the user setup? I think dsync wants to sync mailboxes between configured and working servers with users already defined.
Yes. Target machine should be configured and tested before starting any migration. That is the thing with *all* migrations.
Sami
On 24/09/2019 10:14 PM, @lbutlr via dovecot wrote:
Did the target machine already have the user setup? I think dsync wants to sync mailboxes between configured and working servers with users already defined.
Yes, of course. And the directory structure was pretty much identical between the two. Which was how I was able to write and run the rsync script.
I think at this point, I'm past the dsync problems. That window has closed, and the migration is done, albeit with this irritating re-downloading of email with POP problem. However I'm still keen to get to the bottom of how this whole UID/UIDL/POP3 transaction takes place, so I can understand it for the future, and prevent it happening again. I'm guessing it would be a useful addition to the dovecot documentation as well.
P.
On 25 Sep 2019, at 4.52, Plutocrat via dovecot dovecot@dovecot.org wrote:
On 24/09/2019 10:14 PM, @lbutlr via dovecot wrote:
Did the target machine already have the user setup? I think dsync wants to sync mailboxes between configured and working servers with users already defined.
Yes, of course. And the directory structure was pretty much identical between the two. Which was how I was able to write and run the rsync script.
I think at this point, I'm past the dsync problems. That window has closed, and the migration is done, albeit with this irritating re-downloading of email with POP problem. However I'm still keen to get to the bottom of how this whole UID/UIDL/POP3 transaction takes place, so I can understand it for the future, and prevent it happening again. I'm guessing it would be a useful addition to the dovecot documentation as well.
If you want to keep the data complete then the only proper way to do the migration is dsync. Specially since you now were migrating dovecot to dovecot.
imapsync does not retain imap uids or pop3 uidls. all users would need to clear their local caches.
Sami
On 25.09.19 12:29, Sami Ketola via dovecot wrote:
On 25 Sep 2019, at 4.52, Plutocrat via dovecot dovecot@dovecot.org wrote:
On 24/09/2019 10:14 PM, @lbutlr via dovecot wrote:
Did the target machine already have the user setup? I think dsync wants to sync mailboxes between configured and working servers with users already defined.
Yes, of course. And the directory structure was pretty much identical between the two. Which was how I was able to write and run the rsync script.
I think at this point, I'm past the dsync problems. That window has closed, and the migration is done, albeit with this irritating re-downloading of email with POP problem. However I'm still keen to get to the bottom of how this whole UID/UIDL/POP3 transaction takes place, so I can understand it for the future, and prevent it happening again. I'm guessing it would be a useful addition to the dovecot documentation as well.
If you want to keep the data complete then the only proper way to do the migration is dsync. Specially since you now were migrating dovecot to dovecot.
imapsync does not retain imap uids or pop3 uidls. all users would need to clear their local caches.
Sami
Very interesting thread, since I was about to perform the migration in the next few days. I was operating under the assumption that imapsync would retain both imap uids and pop3 uidls. According to Sami's post this is not the case. This begs the question which path to take. The source server is dovecot, albeit a very old version with no dsync (we're talking dovecot 1.x.x).
Any ideas?
Thanks Ionel
On 25 Sep 2019, at 15.14, Ionel via dovecot dovecot@dovecot.org wrote:
On 25.09.19 12:29, Sami Ketola via dovecot wrote:
On 25 Sep 2019, at 4.52, Plutocrat via dovecot dovecot@dovecot.org wrote:
On 24/09/2019 10:14 PM, @lbutlr via dovecot wrote:
Did the target machine already have the user setup? I think dsync wants to sync mailboxes between configured and working servers with users already defined.
Yes, of course. And the directory structure was pretty much identical between the two. Which was how I was able to write and run the rsync script.
I think at this point, I'm past the dsync problems. That window has closed, and the migration is done, albeit with this irritating re-downloading of email with POP problem. However I'm still keen to get to the bottom of how this whole UID/UIDL/POP3 transaction takes place, so I can understand it for the future, and prevent it happening again. I'm guessing it would be a useful addition to the dovecot documentation as well.
If you want to keep the data complete then the only proper way to do the migration is dsync. Specially since you now were migrating dovecot to dovecot.
imapsync does not retain imap uids or pop3 uidls. all users would need to clear their local caches.
Sami
Very interesting thread, since I was about to perform the migration in the next few days. I was operating under the assumption that imapsync would retain both imap uids and pop3 uidls. According to Sami's post this is not the case. This begs the question which path to take. The source server is dovecot, albeit a very old version with no dsync (we're talking dovecot 1.x.x).
Any ideas?
Native doveadm protocol between two major versions might not be compatible so I would not use that. What I would do in your case is to use imapc connector to pull the emails with dovecot dsync and pop3_migration -plugin to fetch the POP3 UIDLs from the old server over pop3 connection.
https://wiki2.dovecot.org/Migration/Dsync
Sami
On Sep 25, 2019, at 07:16, Sami Ketola Sami.Ketola@open-xchange.com wrote:
What I would do in your case is to use imapc connector to pull the emails with dovecot dsync and pop3_migration -plugin to fetch the POP3 UIDLs from the old server over pop3 connection.
Also, consider not using POP3 at all. It was designed for an entirely different world than the one we live in.
-- My main job is trying to come up with new and innovative and effective ways to reject even more mail. I'm up to about 97% now.
participants (6)
-
＠lbutlr
-
Ionel
-
LuKreme
-
Plutocrat
-
Sami Ketola
-
Sami Ketola