[Dovecot] Server Migration Attempt - new messages DELETED after secondary rsyncs

27 Dec 2013 · */Charles/*

      Starting a new thread, as I've got a lot more details now...
First, I know this may be (or is even likely) something weird happening
with Thunderbird, but the bottom line is, I have to find a resolution or
a workaround or something, or I will be unable to pull the trigger on
the server migration. It is simply unacceptable to lose ANY mails doing
this.
Goal: migrate mailstore from old server to new server using rsync
Thunderbird (24.2) is set up to access the new account for testing
Problem: after rsync, ls shows new messages (11 for the 27th), but when
Thunderbird accesses the maildir again after another rsync, MOST (but
not all) of the new messages are deleted (only one for the 27th remains)
Timo - I'd be happy to do anything necessary to track this down if
you'll just tell me what you need...
Note: if you are pretty confident that upgrading dovecot on the old
server and using dsync to do the migration will solve the problem,
obviously I'm willing to do that if there is no other choice, but... I
hope you'll want to get to the bottom of this regardless, as, according
to you and everything I've read, it should work without losing any emails...
Ok, as to details...
Source (ol/active server): dovecot 2.1.17, postfix virtual for delivery,
vmail owned by postfix:postfix
Note: using prefix=INBOX. courier compatibility namespace
Target: dovecot 2.2.10, dovecot LDA, vmail owned by vmail:vmail, nothing
being actively delivered (yet)
Note: NOT using prefix=INBOX. courier compatiblity namespace
Both doveconf -n outputs at end of this message
*** Could the different namespaces, or differing dovecot versions cause
this?
Current /snapshot/ being used was taken at about 3:48am this morning
rsync command (dovecot is STOPPED before performing, source is the
latest snapshot of the active server):
rsync -rltgovDHP --delete --delete-excluded --exclude-from
'/path/to/excludes.txt' /snapshot/.../vmail/example.com/user/
/.../vmail/example.com/user/
I'm only syncing the one user until I get this problem resolved as it is
faster.
excludes.txt contains only (just to eliminate these which are still on
the source):
courierimap*
Right after the rsync:
ls -al /.../user
(changes depending on what is in includes.txt)
-rw-------   1 vmail vmail     51 Dec 18 10:13 dovecot-keywords
-rw-------   1 vmail vmail 221638 Dec 27 02:42 dovecot-uidlist
-rw-------   1 vmail vmail      8 Dec 18 09:56 dovecot-uidvalidity
-r--r--r--   1 vmail vmail      0 Jun  9  2012 dovecot-uidvalidity.4fd32805
ls -al /.../user/cur
(copy/pasted newest thru 12/25 - this list is always the same after a sync)
-rw-------   1 vmail vmail    24904 Dec 25 15:47
1388004452.Vfe02Ic72976M281736.oldhost.example.com:2,
-rw-------   1 vmail vmail  5825360 Dec 25 22:52
1388029948.Vfe02Ic7306fM556006.oldhost.example.com:2,
-rw-------   1 vmail vmail    27718 Dec 26 02:19
1388042391.Vfe02Ic33b40M543019.oldhost.example.com:2,
-rw-------   1 vmail vmail    28083 Dec 26 07:11
1388059893.Vfe02Ic76388M193423.oldhost.example.com:2,S
-rw-------   1 vmail vmail   100651 Dec 26 08:02
1388062961.Vfe02Ic763c2M415005.oldhost.example.com:2,
-rw-------   1 vmail vmail     5286 Dec 26 09:41
1388068882.Vfe02Ic7642dM83715.oldhost.example.com:2,S
-rw-------   1 vmail vmail    53203 Dec 26 09:55
1388069721.Vfe02Ic7646fM716189.oldhost.example.com:2,S
-rw-------   1 vmail vmail    12039 Dec 26 12:48
1388080111.Vfe02Ic76742M972942.oldhost.example.com:2,RS
-rw-------   1 vmail vmail    13531 Dec 26 13:14
1388081648.Vfe02Ic7674cM68343.oldhost.example.com:2,S
-rw-------   1 vmail vmail    25161 Dec 26 15:47
1388090846.Vfe02Ic768acM489505.oldhost.example.com:2,
-rw-------   1 vmail vmail     5037 Dec 26 16:27
1388093257.Vfe02Ic76918M766778.oldhost.example.com:2,
-rw-------   1 vmail vmail     3157 Dec 26 17:01
1388095312.Vfe02Ic76997M703744.oldhost.example.com:2,
-rw-------   1 vmail vmail    22480 Dec 26 17:50
1388098239.Vfe02Ic769deM225078.oldhost.example.com:2,
-rw-------   1 vmail vmail    10043 Dec 26 19:59
1388105984.Vfe02Ic76a25M12505.oldhost.example.com:2,
-rw-------   1 vmail vmail    10635 Dec 26 21:11
1388110298.Vfe02Ic76a65M649895.oldhost.example.com:2,
-rw-------   1 vmail vmail     8695 Dec 26 21:11
1388110319.Vfe02Ic76a6fM696948.oldhost.example.com:2,
-rw-------   1 vmail vmail    11045 Dec 26 21:56
1388112965.Vfe02Ic76a57M185504.oldhost.example.com:2,
-rw-------   1 vmail vmail      823 Dec 26 22:40
1388115600.Vfe02Ib61cf5M913598.oldhost.example.com:2,ST
-rw-------   1 vmail vmail      815 Dec 26 22:40
1388115627.Vfe02Ib63443M811945.oldhost.example.com:2,ST
-rw-------   1 vmail vmail      785 Dec 26 22:40
1388115628.Vfe02Ib659c9M617314.oldhost.example.com:2,ST
-rw-------   1 vmail vmail     6156 Dec 27 00:05
1388120705.Vfe02Ib7aa31M58188.oldhost.example.com:2,ST
-rw-------   1 vmail vmail      869 Dec 27 00:05
1388120713.Vfe02Ib8506aM185504.oldhost.example.com:2,ST
-rw-------   1 vmail vmail      925 Dec 27 00:29
1388122155.Vfe02Ibb2a1fM764275.oldhost.example.com:2,ST
-rw-------   1 vmail vmail      823 Dec 27 00:38
1388122687.Vfe02Ic2f69aM168075.oldhost.example.com:2,ST
-rw-------   1 vmail vmail      801 Dec 27 00:38
1388122696.Vfe02Ic31befM507147.oldhost.example.com:2,ST
-rw-------   1 vmail vmail      801 Dec 27 00:38
1388122709.Vfe02Ic3ac2bM981625.oldhost.example.com:2,ST
-rw-------   1 vmail vmail     4576 Dec 27 00:57
1388123820.Vfe02Ic769e8M270744.oldhost.example.com:2,ST
-rw-------   1 vmail vmail     8379 Dec 27 01:33
1388126003.Vfe02Ib7fd87M510118.oldhost.example.com:2,ST
-rw-------   1 vmail vmail    27647 Dec 27 02:20
1388128838.Vfe02Ic7610fM483481.oldhost.example.com:2,
-rw-------   1 vmail vmail     6373 Dec 27 02:42
1388130144.Vfe02Ic76a99M631316.oldhost.example.com:2,ST
-rw-------   1 vmail vmail     3906 Dec 27 03:01
1388131281.Vfe02Ic76a9cM599830.oldhost.example.com:2,ST
Start dovecot, select INBOX in Thunderbird for this account, it only
displays these (I got this ls output from AFTER the messages in the cur
directory are deleted per below - before doing any of the 3 aqctions
below, ALL of the messages show in the ls output):
ls -al /.../user/cur
-rw-------   1 vmail vmail    24904 Dec 25 15:47
1388004452.Vfe02Ic72976M281736.oldhost.example.com:2,
-rw-------   1 vmail vmail  5825360 Dec 25 22:52
1388029948.Vfe02Ic7306fM556006.oldhost.example.com:2,
-rw-------   1 vmail vmail    27718 Dec 26 02:19
1388042391.Vfe02Ic33b40M543019.oldhost.example.com:2,
-rw-------   1 vmail vmail    28083 Dec 26 07:11
1388059893.Vfe02Ic76388M193423.oldhost.example.com:2,S
-rw-------   1 vmail vmail   100651 Dec 26 08:02
1388062961.Vfe02Ic763c2M415005.oldhost.example.com:2,
-rw-------   1 vmail vmail     5286 Dec 26 09:41
1388068882.Vfe02Ic7642dM83715.oldhost.example.com:2,S
-rw-------   1 vmail vmail    53203 Dec 26 09:55
1388069721.Vfe02Ic7646fM716189.oldhost.example.com:2,S
-rw-------   1 vmail vmail    12039 Dec 26 12:48
1388080111.Vfe02Ic76742M972942.oldhost.example.com:2,RS
-rw-------   1 vmail vmail    13531 Dec 26 13:14
1388081648.Vfe02Ic7674cM68343.oldhost.example.com:2,S
-rw-------   1 vmail vmail    25161 Dec 26 15:47
1388090846.Vfe02Ic768acM489505.oldhost.example.com:2,
-rw-------   1 vmail vmail     5037 Dec 26 16:27
1388093257.Vfe02Ic76918M766778.oldhost.example.com:2,
-rw-------   1 vmail vmail     3157 Dec 26 17:01
1388095312.Vfe02Ic76997M703744.oldhost.example.com:2,
-rw-------   1 vmail vmail    22480 Dec 26 17:50
1388098239.Vfe02Ic769deM225078.oldhost.example.com:2,
-rw-------   1 vmail vmail    10043 Dec 26 19:59
1388105984.Vfe02Ic76a25M12505.oldhost.example.com:2,
-rw-------   1 vmail vmail    10635 Dec 26 21:11
1388110298.Vfe02Ic76a65M649895.oldhost.example.com:2,
-rw-------   1 vmail vmail     8695 Dec 26 21:11
1388110319.Vfe02Ic76a6fM696948.oldhost.example.com:2,
-rw-------   1 vmail vmail    11045 Dec 26 21:56
1388112965.Vfe02Ic76a57M185504.oldhost.example.com:2,
-rw-------   1 vmail vmail    27647 Dec 27 02:20
1388128838.Vfe02Ic7610fM483481.oldhost.example.com:2,
This is weird, because it does show ONE new message from the 27th.
And again, ls -al on the cur dir at this point still shows ALL of the
messages (11 for the 27th, and 18 for the 26th) shown in the FIRST ls
output, ABOVE this second listing (which is shown only to show you what
Thunderbird is seeing, and what shows in the ls output AFTER doing any
of the 3 actions below).
Now...
Doing ANY of the following results in MOST (BUT NOT ALL) OF THE NEW
MESSAGES BEING DELETED ON THE SERVER - ie, ls -al output changes to the
match what is immediately above (what was showing in Thunderbird right
after selecting the Inbox right after the rsync).

Right-click on the Inbox > Properties > Repair Folder (in Thunderbird)
Right-click on INBOX > Compact (in Thunderbird)
Manually deleting the .msf file for the INBOX

Obviously, I must find a resolution for this before I can pull the
trigger on this migration.

old server dovecon -n:
oldhost : Fri Dec 27, 05:59:12 : ~
# doveconf -n
# 2.1.17: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.7-gentoo-r1 x86_64 Gentoo Base System release 2.2
auth_default_realm = example.com
auth_mechanisms = plain login
first_valid_uid = 207
last_valid_uid = 207
listen = *
login_log_format_elements = user=<%u> method=%m rip=%r lport=%{lport}
mpid=%e %c session=<%{session}>
mail_gid = 207
mail_location = maildir:~/:INDEX=MEMORY
mail_uid = 207
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix = INBOX.
separator = .
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
}
protocols = imap lmtp
service auth-worker {
user = $default_internal_user
}
service auth {
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
}
unix_listener auth-userdb {
group = postfix
mode = 0600
user = postfix
}
}
service imap-login {
service_count = 0
}
service imap {
process_limit = 2048
}
ssl = required
ssl_cert = 

new server dovecot -n:
host : Fri Dec 27, 08:13:34 : ~
# doveconf -n
# 2.2.10: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.17-gentoo x86_64 Gentoo Base System release 2.2
auth_default_realm = example.com
auth_mechanisms = plain login
first_valid_uid = 200
last_valid_uid = 200
listen = *
login_log_format_elements = user=<%u> method=%m %c rip=%r lport=%{lport}
mail_gid = vmail
mail_location = maildir:~
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
separator = .
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
}
protocols = imap lmtp
service auth-worker {
user = $default_internal_user
}
service auth {
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
}
unix_listener auth-userdb {
group = vmail
mode = 0600
user = vmail
}
}
service imap-login {
service_count = 0
}
service imap {
process_limit = 2048
}
ssl = required
ssl_cert = 
--
Best regards,
*/Charles/*

[Dovecot] Server Migration Attempt - new messages DELETED after secondary rsyncs

Charles Marcus