Starting a new thread, as I've got a lot more details now...

First, I know this may be (or is even likely) something weird happening with Thunderbird, but the bottom line is, I have to find a resolution or a workaround or something, or I will be unable to pull the trigger on the server migration. It is simply unacceptable to lose ANY mails doing this.

Goal: migrate mailstore from old server to new server using rsync

Thunderbird (24.2) is set up to access the new account for testing

Problem: after rsync, ls shows new messages (11 for the 27th), but when Thunderbird accesses the maildir again after another rsync, MOST (but not all) of the new messages are deleted (only one for the 27th remains)

Timo - I'd be happy to do anything necessary to track this down if you'll just tell me what you need...

Note: if you are pretty confident that upgrading dovecot on the old server and using dsync to do the migration will solve the problem, obviously I'm willing to do that if there is no other choice, but... I hope you'll want to get to the bottom of this regardless, as, according to you and everything I've read, it should work without losing any emails...

Ok, as to details...

Source (ol/active server): dovecot 2.1.17, postfix virtual for delivery, vmail owned by postfix:postfix Note: using prefix=INBOX. courier compatibility namespace

Target: dovecot 2.2.10, dovecot LDA, vmail owned by vmail:vmail, nothing being actively delivered (yet) Note: NOT using prefix=INBOX. courier compatiblity namespace

Both doveconf -n outputs at end of this message

*** Could the different namespaces, or differing dovecot versions cause this?

Current /snapshot/ being used was taken at about 3:48am this morning

rsync command (dovecot is STOPPED before performing, source is the latest snapshot of the active server):

rsync -rltgovDHP --delete --delete-excluded --exclude-from '/path/to/excludes.txt' /snapshot/.../vmail/example.com/user/ /.../vmail/example.com/user/

I'm only syncing the one user until I get this problem resolved as it is faster.

excludes.txt contains only (just to eliminate these which are still on the source): courierimap*

Right after the rsync:

ls -al /.../user (changes depending on what is in includes.txt) -rw------- 1 vmail vmail 51 Dec 18 10:13 dovecot-keywords -rw------- 1 vmail vmail 221638 Dec 27 02:42 dovecot-uidlist -rw------- 1 vmail vmail 8 Dec 18 09:56 dovecot-uidvalidity -r--r--r-- 1 vmail vmail 0 Jun 9 2012 dovecot-uidvalidity.4fd32805

ls -al /.../user/cur (copy/pasted newest thru 12/25 - this list is always the same after a sync) -rw------- 1 vmail vmail 24904 Dec 25 15:47 1388004452.Vfe02Ic72976M281736.oldhost.example.com:2, -rw------- 1 vmail vmail 5825360 Dec 25 22:52 1388029948.Vfe02Ic7306fM556006.oldhost.example.com:2, -rw------- 1 vmail vmail 27718 Dec 26 02:19 1388042391.Vfe02Ic33b40M543019.oldhost.example.com:2, -rw------- 1 vmail vmail 28083 Dec 26 07:11 1388059893.Vfe02Ic76388M193423.oldhost.example.com:2,S -rw------- 1 vmail vmail 100651 Dec 26 08:02 1388062961.Vfe02Ic763c2M415005.oldhost.example.com:2, -rw------- 1 vmail vmail 5286 Dec 26 09:41 1388068882.Vfe02Ic7642dM83715.oldhost.example.com:2,S -rw------- 1 vmail vmail 53203 Dec 26 09:55 1388069721.Vfe02Ic7646fM716189.oldhost.example.com:2,S -rw------- 1 vmail vmail 12039 Dec 26 12:48 1388080111.Vfe02Ic76742M972942.oldhost.example.com:2,RS -rw------- 1 vmail vmail 13531 Dec 26 13:14 1388081648.Vfe02Ic7674cM68343.oldhost.example.com:2,S -rw------- 1 vmail vmail 25161 Dec 26 15:47 1388090846.Vfe02Ic768acM489505.oldhost.example.com:2, -rw------- 1 vmail vmail 5037 Dec 26 16:27 1388093257.Vfe02Ic76918M766778.oldhost.example.com:2, -rw------- 1 vmail vmail 3157 Dec 26 17:01 1388095312.Vfe02Ic76997M703744.oldhost.example.com:2, -rw------- 1 vmail vmail 22480 Dec 26 17:50 1388098239.Vfe02Ic769deM225078.oldhost.example.com:2, -rw------- 1 vmail vmail 10043 Dec 26 19:59 1388105984.Vfe02Ic76a25M12505.oldhost.example.com:2, -rw------- 1 vmail vmail 10635 Dec 26 21:11 1388110298.Vfe02Ic76a65M649895.oldhost.example.com:2, -rw------- 1 vmail vmail 8695 Dec 26 21:11 1388110319.Vfe02Ic76a6fM696948.oldhost.example.com:2, -rw------- 1 vmail vmail 11045 Dec 26 21:56 1388112965.Vfe02Ic76a57M185504.oldhost.example.com:2, -rw------- 1 vmail vmail 823 Dec 26 22:40 1388115600.Vfe02Ib61cf5M913598.oldhost.example.com:2,ST -rw------- 1 vmail vmail 815 Dec 26 22:40 1388115627.Vfe02Ib63443M811945.oldhost.example.com:2,ST -rw------- 1 vmail vmail 785 Dec 26 22:40 1388115628.Vfe02Ib659c9M617314.oldhost.example.com:2,ST -rw------- 1 vmail vmail 6156 Dec 27 00:05 1388120705.Vfe02Ib7aa31M58188.oldhost.example.com:2,ST -rw------- 1 vmail vmail 869 Dec 27 00:05 1388120713.Vfe02Ib8506aM185504.oldhost.example.com:2,ST -rw------- 1 vmail vmail 925 Dec 27 00:29 1388122155.Vfe02Ibb2a1fM764275.oldhost.example.com:2,ST -rw------- 1 vmail vmail 823 Dec 27 00:38 1388122687.Vfe02Ic2f69aM168075.oldhost.example.com:2,ST -rw------- 1 vmail vmail 801 Dec 27 00:38 1388122696.Vfe02Ic31befM507147.oldhost.example.com:2,ST -rw------- 1 vmail vmail 801 Dec 27 00:38 1388122709.Vfe02Ic3ac2bM981625.oldhost.example.com:2,ST -rw------- 1 vmail vmail 4576 Dec 27 00:57 1388123820.Vfe02Ic769e8M270744.oldhost.example.com:2,ST -rw------- 1 vmail vmail 8379 Dec 27 01:33 1388126003.Vfe02Ib7fd87M510118.oldhost.example.com:2,ST -rw------- 1 vmail vmail 27647 Dec 27 02:20 1388128838.Vfe02Ic7610fM483481.oldhost.example.com:2, -rw------- 1 vmail vmail 6373 Dec 27 02:42 1388130144.Vfe02Ic76a99M631316.oldhost.example.com:2,ST -rw------- 1 vmail vmail 3906 Dec 27 03:01 1388131281.Vfe02Ic76a9cM599830.oldhost.example.com:2,ST

Start dovecot, select INBOX in Thunderbird for this account, it only displays these (I got this ls output from AFTER the messages in the cur directory are deleted per below - before doing any of the 3 aqctions below, ALL of the messages show in the ls output): ls -al /.../user/cur -rw------- 1 vmail vmail 24904 Dec 25 15:47 1388004452.Vfe02Ic72976M281736.oldhost.example.com:2, -rw------- 1 vmail vmail 5825360 Dec 25 22:52 1388029948.Vfe02Ic7306fM556006.oldhost.example.com:2, -rw------- 1 vmail vmail 27718 Dec 26 02:19 1388042391.Vfe02Ic33b40M543019.oldhost.example.com:2, -rw------- 1 vmail vmail 28083 Dec 26 07:11 1388059893.Vfe02Ic76388M193423.oldhost.example.com:2,S -rw------- 1 vmail vmail 100651 Dec 26 08:02 1388062961.Vfe02Ic763c2M415005.oldhost.example.com:2, -rw------- 1 vmail vmail 5286 Dec 26 09:41 1388068882.Vfe02Ic7642dM83715.oldhost.example.com:2,S -rw------- 1 vmail vmail 53203 Dec 26 09:55 1388069721.Vfe02Ic7646fM716189.oldhost.example.com:2,S -rw------- 1 vmail vmail 12039 Dec 26 12:48 1388080111.Vfe02Ic76742M972942.oldhost.example.com:2,RS -rw------- 1 vmail vmail 13531 Dec 26 13:14 1388081648.Vfe02Ic7674cM68343.oldhost.example.com:2,S -rw------- 1 vmail vmail 25161 Dec 26 15:47 1388090846.Vfe02Ic768acM489505.oldhost.example.com:2, -rw------- 1 vmail vmail 5037 Dec 26 16:27 1388093257.Vfe02Ic76918M766778.oldhost.example.com:2, -rw------- 1 vmail vmail 3157 Dec 26 17:01 1388095312.Vfe02Ic76997M703744.oldhost.example.com:2, -rw------- 1 vmail vmail 22480 Dec 26 17:50 1388098239.Vfe02Ic769deM225078.oldhost.example.com:2, -rw------- 1 vmail vmail 10043 Dec 26 19:59 1388105984.Vfe02Ic76a25M12505.oldhost.example.com:2, -rw------- 1 vmail vmail 10635 Dec 26 21:11 1388110298.Vfe02Ic76a65M649895.oldhost.example.com:2, -rw------- 1 vmail vmail 8695 Dec 26 21:11 1388110319.Vfe02Ic76a6fM696948.oldhost.example.com:2, -rw------- 1 vmail vmail 11045 Dec 26 21:56 1388112965.Vfe02Ic76a57M185504.oldhost.example.com:2, -rw------- 1 vmail vmail 27647 Dec 27 02:20 1388128838.Vfe02Ic7610fM483481.oldhost.example.com:2,

This is weird, because it does show ONE new message from the 27th.

And again, ls -al on the cur dir at this point still shows ALL of the messages (11 for the 27th, and 18 for the 26th) shown in the FIRST ls output, ABOVE this second listing (which is shown only to show you what Thunderbird is seeing, and what shows in the ls output AFTER doing any of the 3 actions below).

Now...

Doing ANY of the following results in MOST (BUT NOT ALL) OF THE NEW MESSAGES BEING DELETED ON THE SERVER - ie, ls -al output changes to the match what is immediately above (what was showing in Thunderbird right after selecting the Inbox right after the rsync).

1. Right-click on the Inbox > Properties > Repair Folder (in Thunderbird)
2. Right-click on INBOX > Compact (in Thunderbird)
3. Manually deleting the .msf file for the INBOX

Obviously, I must find a resolution for this before I can pull the trigger on this migration.

old server dovecon -n:

oldhost : Fri Dec 27, 05:59:12 : ~ # doveconf -n # 2.1.17: /etc/dovecot/dovecot.conf # OS: Linux 3.10.7-gentoo-r1 x86_64 Gentoo Base System release 2.2 auth_default_realm = example.com auth_mechanisms = plain login first_valid_uid = 207 last_valid_uid = 207 listen = * login_log_format_elements = user=<%u> method=%m rip=%r lport=%{lport} mpid=%e %c session=<%{session}> mail_gid = 207 mail_location = maildir:~/:INDEX=MEMORY mail_uid = 207 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = INBOX. separator = . } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } protocols = imap lmtp service auth-worker { user = $default_internal_user } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = postfix mode = 0600 user = postfix } } service imap-login { service_count = 0 } service imap { process_limit = 2048 } ssl = required ssl_cert = </etc/ssl/myCerts/imap.pem ssl_key = </etc/ssl/myCerts/imap_key.pem userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol smtp { auth_verbose = yes } protocol lda { mail_plugins = sieve } protocol imap { mail_max_userip_connections = 20 } oldhost : Fri Dec 27, 08:22:04 : ~

new server dovecot -n:

host : Fri Dec 27, 08:13:34 : ~ # doveconf -n # 2.2.10: /etc/dovecot/dovecot.conf # OS: Linux 3.10.17-gentoo x86_64 Gentoo Base System release 2.2 auth_default_realm = example.com auth_mechanisms = plain login first_valid_uid = 200 last_valid_uid = 200 listen = * login_log_format_elements = user=<%u> method=%m %c rip=%r lport=%{lport} mail_gid = vmail mail_location = maildir:~ mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = . } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } protocols = imap lmtp service auth-worker { user = $default_internal_user } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service imap-login { service_count = 0 } service imap { process_limit = 2048 } ssl = required ssl_cert = </etc/ssl/myCerts/imap.pem ssl_key = </etc/ssl/myCerts/imap_key.pem userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol smtp { auth_verbose = yes } protocol lda { auth_verbose = yes mail_plugins = sieve } protocol imap { mail_max_userip_connections = 20 } host : Fri Dec 27, 08:19:59 : ~

--

Best regards,

*/Charles/*