‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, March 17, 2020 11:46 AM, Aki Tuomi aki.tuomi@open-xchange.com wrote:

Trydoveadm -Dv expunge -u username\\@domain.com mailbox INBOX savedbefore 90

It might give insight what goes wrong.

Here would be the output but I still can't see where the problem is:

Debug: Loading modules from directory: /usr/lib/dovecot/modules Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(admin@domain.tld)<5312><>: Debug: auth USER input: admin@domain.tld home=/var/vmail/domain.tld/admin uid=150 gid=8 quota_rule=*:bytes=1024000000 doveadm(admin@domain.tld)<5312><>: Debug: Added userdb setting: plugin/quota_rule=*:bytes=1024000000 doveadm(admin@domain.tld): Debug: Effective uid=150, gid=8, home=/var/vmail/domain.tld/admin doveadm(admin@domain.tld): Debug: Quota root: name=User quota backend=maildir args= doveadm(admin@domain.tld): Debug: Quota rule: root=User quota mailbox=* bytes=1024000000 messages=0 doveadm(admin@domain.tld): Debug: Quota rule: root=User quota mailbox=INBOX.Trash bytes=+104857600 messages=0 doveadm(admin@domain.tld): Debug: Quota warning: bytes=972800000 (95%) messages=0 reverse=no command=quota-warning 95 admin@domain.tld doveadm(admin@domain.tld): Debug: Quota warning: bytes=819200000 (80%) messages=0 reverse=no command=quota-warning 80 admin@domain.tld doveadm(admin@domain.tld): Debug: Quota grace: root=User quota bytes=102400000 (10%) doveadm(admin@domain.tld): Debug: Namespace inbox: type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/var/vmail/domain.tld/admin/Maildir doveadm(admin@domain.tld): Debug: maildir++: root=/var/vmail/domain.tld/admin/Maildir, index=, indexpvt=, control=, inbox=/var/vmail/domain.tld/admin/Maildir, alt= doveadm(admin@domain.tld): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none doveadm(admin@domain.tld): Debug: none: root=, index=, indexpvt=, control=, inbox=, alt= doveadm(admin@domain.tld): Debug: quota: quota_over_flag check: quota_over_script unset - skipping doveadm(admin@domain.tld): Debug: Mailbox INBOX: Mailbox opened because: expunge

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, March 17, 2020 12:45 PM, Aki Tuomi aki.tuomi@open-xchange.com wrote:

doveadm search -u username\\\\@domain.com mailbox INBOX savedbefore 90

return any mails?

No it does not return any mails, although there are mails older than 90 days in the INBOX.

How did you do the migration?

I copied over the mails with rsync from the old mailbox server to the new one.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, March 17, 2020 1:05 PM, Aki Tuomi aki.tuomi@open-xchange.com wrote:

Did you add sufficient flags to rsync to retain file mtime and ctime? These are used to determine the mail saved date.

Aki

I have been using "rsync -av". As far as I know the "-a or --archive" parameter should preserve almost everything, at least the rsync man page says that.

On 17/03/2020 14:15 mabi mabi@protonmail.ch wrote:

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, March 17, 2020 1:10 PM, mabi mabi@protonmail.ch wrote:

...

I have been using "rsync -av". As far as I know the "-a or --archive" parameter should preserve almost everything, at least the rsync man page says that.

If that is of any help I did a "stat" on one of the oldest mails in that folder (mail from 17th October 2019) and here is the output:

Size: 5363 Blocks: 8 IO Block: 1048576 regular file Device: 2ch/44d Inode: 274504 Links: 1 Access: (0600/-rw-------) Uid: ( 150/ vmail) Gid: ( 8/ mail) Access: 2020-01-16 18:21:53.887373904 +0100 Modify: 2019-10-17 23:02:05.000000000 +0200 Change: 2020-01-16 18:21:53.887373904 +0100 Birth: -

Can you try doveadm fetch -u user date.saved UID mail-uid To see if it matches this?

Aki

On 17/03/2020 14:32 mabi mabi@protonmail.ch wrote:

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, March 17, 2020 1:19 PM, Aki Tuomi aki.tuomi@open-xchange.com wrote:

...

Can you trydoveadm fetch -u user date.saved UID mail-uid To see if it matches this?

How can I find out the "mail-uid" of this specific message? Is that the "Message-ID" header of that particular mail?

If you have message-id you can use

doveadm search -u user HEADER MESSAGE-ID "messageid" to find out the UID.

Aki

On 17/03/2020 14:52 mabi mabi@protonmail.ch wrote:

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, March 17, 2020 1:38 PM, Aki Tuomi aki.tuomi@open-xchange.com wrote:

...

If you have message-id you can use

doveadm search -u user HEADER MESSAGE-ID "messageid" to find out the UID.

I tried the following command:

sudo doveadm search -u user admin\@domain.tld HEADER MESSAGE-ID "SAWH8n6hDRbXHO@mgw.subdom.domain.tld"

But I get back the following error:

Fatal: Unknown argument ADMIN\@DOMAIN.TLD

maybe try

sudo doveadm search -u admin\@domain.tld HEADER MESSAGE-ID "SAWH8n6hDRbXHO@mgw.subdom.domain.tld"

Aki

On 17/03/2020 15:16 mabi mabi@protonmail.ch wrote:

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, March 17, 2020 2:00 PM, Aki Tuomi aki.tuomi@open-xchange.com wrote:

...

sudo doveadm search -u admin\\@domain.tld HEADER MESSAGE-ID "SAWH8n6hDRbXHO@mgw.subdom.domain.tld"

Ah yes, I had to remove the "user" part of course... Thanks!

After running this command, the only output I get is that:

3c37122a9327e159902500001c4caf4d 130863

UID is the second number. Now you can use 130863 as UID.

So doveadm fetch -u admin\\@domain.tld "mailbox date.saved" UID 130863

Aki

On 17 Mar 2020, at 15.40, mabi mabi@protonmail.ch wrote:

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, March 17, 2020 2:18 PM, Aki Tuomi aki.tuomi@open-xchange.com wrote:

...

So doveadm fetch -u admin\\@domain.tld "mailbox date.saved" UID 130863

Aki

This is the ouput:

mailbox: INBOX date.saved: 2020-01-16 18:21:53

I think that might be the date where I have done the migration from the old to the new mailbox server...

So your rsync did not preserve mail object ctimes. Dovecot uses that for internal date if correct value is not in dovecot.index.cache.

Sami

On 17 Mar 2020, at 16.43, mabi mabi@protonmail.ch wrote:

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, March 17, 2020 3:26 PM, Sami Ketola sami.ketola@dovecot.fi wrote:

...

So your rsync did not preserve mail object ctimes. Dovecot uses that for internal date if correct value is not in dovecot.index.cache.

Oh I see, so that's quite unfortunate that rsync does not preserve the ctime. They should have a big warning in their man page...

So what would be the correct process for moving/copying mails from one mailbox server to another in order to avoid this issue?

Since you were migrating dovecot->dovecot I would have just used Dsync.

Sami

On 17 Mar 2020, at 17.27, mabi mabi@protonmail.ch wrote:

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, March 17, 2020 3:57 PM, Sami Ketola sami.ketola@dovecot.fi wrote:

...

Since you were migrating dovecot->dovecot I would have just used Dsync.

Thanks for the hint about dsync. It seems quite complex to setup and to use. Isn't there an easier method? What about tar'ing the files? Will this preserve the ctime?

what is so complex in it?

doveadm backup -u <uid> -R ssh old-server sudo /usr/bin/doveadm dsync-server -u <uid>

or just use doveadm over tcp if no channel encryption is required. I've used to do this over insecure internet hence ssh. Sami