Outlook 365 MUA produces dovecot changing filename on some mailbox

Wed Sep 21 15:22:52 UTC 2022

>>>>> "Emilio" == Emilio Augusto Lazo Zaia <emiliolazozaia at gmail.com> writes:

> Hello John! Thanks for your answer.

You're welcome, but I'm not sure I'm going to be able to help here.  I
suspect this is a client issue, just like I hate how the Apple Mail
client marks messages SEEN for no reason at all because it does
"SELECT INBOX" vs "EXAMINE INBOX". 

> On 12/9/22 12:30, John Stoffel wrote:

>         I have a setup on which dovecot is the server in a domain on which mailboxes are >20GB, some of
>         them are 150GB or more.

>     It looks like you're using Maildir format?  In any case, please post
>     more details of your configuration. 

> Sure. It is a simple setup, with Maildir.

> # doveconf -N

> auth_debug = no
> auth_mechanisms = plain login
> auth_verbose = no
> debug_log_path = /var/log/syslog/mail.debug
> disable_plaintext_auth = no
> login_greeting = Hermes.
> mail_debug = yes
> mail_index_rewrite_max_log_bytes = 1 M
> mail_location = maildir:~/Maildir
> namespace inbox {
>  inbox = yes
>  location =  
>  mailbox Drafts {
>    special_use = \Drafts
>  }
>  mailbox Junk {
>    special_use = \Junk
>  }
>  mailbox Sent {
>    special_use = \Sent
>  }
>  mailbox "Sent Messages" {
>    special_use = \Sent
>  }
>  mailbox Trash {
>    special_use = \Trash
>  }
>  prefix =  
> }
> passdb {
>  driver = pam
>  name =  
> }
> protocols = " imap"
> service auth {
>  unix_listener auth-client {
>    group = Debian-exim
>    mode = 0660
>    user = Debian-exim
>  }
> }
> service imap {
>  vsz_limit = 2 G
> }
> ssl = yes
> ssl_cert = </etc/dovecot/dovecot-cert.pem
> ssl_client_ca_dir = /etc/ssl/certs
> ssl_dh = # hidden, use -P to show it
> ssl_key = # hidden, use -P to show it
> userdb {
>  driver = passwd
>  name =  
> }
> verbose_proctitle = yes
> protocol imap {
>  mail_max_userip_connections = 20
> }

>         When reconfiguring _some_ mailboxes with Outlook 365 (configuring a previously used mailbox on a
>         new PC), the process takes too long even if mail visibility on Outlook is 3 months.

>     So the outlook client can't download and parse the number of messages
>     on there without breaking?  How many messages are there?  It's
>     probably not the size of the mailbox, but the number of messages.

> 50K mails on some folders... total 5.5M on the whole server. I did discover which was the problem
> but I don't have a solution.

Yeah, that's alot.  You might need to push your clients to start
putting more emails into sub-folders, maybe split up by year say, so
they don't have quite so many emails sitting around.  

>         Inspecting at INBOX cur directory I did note that the 'ls' command shows the last file listed is a
>         file from year 2020, 2019 or whatever not being the last received mail; that means the
>         alphabetical ordering of 'ls' is showing past years after the last received mail, i.e. dovecot
>         changed the epoch in filename. This only happens with Outlook. Following is an excerpt of ls:

>     Sure, 'ls -l' doesn't do any sorting, it just reads the directory
>     information as returned from the disk and show you the results.  If
>     you want it by time, you need to do:

>         ls -ltr

>     to have the newest files be at the end.  But how 'ls' sees the
>     directry entries doen't matter to dovecot.  

> Newest files are at the end; 'ls' without '-U' or '--sort=none' lists files ordered
> alphabetically, so the last entry would be the last mail placed on the folder because the epoch
> time is part of the filename in Maildir structure...

>         -rw-r--r-- 1 coord.ventas ventas   544765 sep  9 10:57 1662769412.M329925P1259441.xyz.com,S=
>         544765,W=552045:2,
>         -rw-r--r-- 1 coord.ventas ventas   163491 sep  5 12:14 1662769412.M74257P1259441.xyz.com,S=
>         163491,W=165846:2,S
>         -rw-r--r-- 1 coord.ventas ventas  3043536 sep  9 12:02 1662769412.M777084P1259441.xyz.com,S=
>         3043536,W=3083246:2,
>         -rw-r--r-- 1 coord.ventas ventas   161002 feb 27  2020 1662874173.M608062P1282222.xyz.com,S=
>         161002,W=163373:2,S
>         -rw-r--r-- 1 coord.ventas ventas  2230491 feb 27  2020 1662874176.M281294P1282222.xyz.com,S=
>         2230491,W=2259506:2,S
>         -rw-r--r-- 1 coord.ventas ventas   167925 feb 27  2020 1662874176.M741229P1282222.xyz.com,S=
>         167925,W=170373:2,S

>         The process of renaming files is continuous until all folder was
>         reread and renamed every file on it. If I issue 'lsof' (with grep) I
>         see the file that is being renamed passing through tmp/ directory
>         and after that, the file is placed under new/ directory (I guess)
>         and Outlook sees a new email to sync, making the sync process too
>         long... mails that have been synced are resynced...

>     Wait what?  Can you explain exactly what you're doing here?  Why would
>     O365 client be renaming the file  

> Ok, I disabled TLS options and inspected the traffic with tcpdump. I did note unnecesary STORE,
> EXPUNGE and APPEND operations on previously synced folder. I guess the whole mailbox is analyzed
> (from the local cache on .pst file) and O365 tags as SPAM a large number of innocent emails
> (including local mails from the same domain which does not travel through internet to reach the
> destination).

So this is really a client issue.  You might need to ask around on the
Microsoft site to see if there are any settings you can change when
it's accessing a server using IMAP.  Also, look in the dovecot
archives.  

> Also I'm using port 144 instead of 143 to bypass GData antivirus
> SPAM checks. Anyway I completely disabled GData antivirus and none
> of its services remains running on these workstations; also starting
> Outlook with safe mode exhibits this behavior, so I don't know which
> feature is this on O365 that rescan all stored emails but it is
> beyond the scope of this list because there is no feature on dovecot
> that can be enabled to workaround this. I guess this is a bug on
> O365.

Yup, sorry!

> All mails are retransmitted to the server with the subject modified (tagging them as SPAM or SPAM
> suspected) and obviously dovecot stores them with a new epoch timestamp on filename and that's why
> I have mails with older timestamp but with recent epoch time on filename...

>         Take a look at the timestamp on filenames: 1662769412 corresponds to
>         "Fri Sep  9 20:23:32 -04 2022" and 1662874176 corresponds to Sun Sep
>         11 01:29:36 -04 2022 but mails were neither sent nor received on
>         that dates.

>         What is Outlook requesting imap server to do and how to avoid this?

>     What is the outlook client version?  What are your connection
>     settings?  Have you looked in the logs on the dovecot server side to
>     see what they say?

> Outlook 365 is version 16.15601.20148 compilation 2208 on one workstation that exhibits the issue
> and this same version on another PC on which this issue is not present...

>     What working clients are your users using today?  Are they working
>     properly with older clients?  

> Basically I'm the mail server administrator not the sysadmin on the network. All users prefer
> working with O365. I've installed Thunderbird on every PC but users refuses to use it in favor of
> using Outlook, unfortunately.

>     Basically, you have to assume we know nothing and explain your setup
>     and configuration to us like we're idiots who don't know anything.
>     Spell out all the details please.  :-)

> Yes, I'm sorry. I hope with this mail I've clarified the scenario.

> Thanks again!

> Emilio