Outlook 365 MUA produces dovecot changing filename on some mailbox
Hello
I have a setup on which dovecot is the server in a domain on which mailboxes are >20GB, some of them are 150GB or more.
When reconfiguring _some_ mailboxes with Outlook 365 (configuring a previously used mailbox on a new PC), the process takes too long even if mail visibility on Outlook is 3 months.
Inspecting at INBOX cur directory I did note that the 'ls' command shows the last file listed is a file from year 2020, 2019 or whatever not being the last received mail; that means the alphabetical ordering of 'ls' is showing past years after the last received mail, i.e. dovecot changed the epoch in filename. This only happens with Outlook. Following is an excerpt of ls:
-rw-r--r-- 1 coord.ventas ventas 544765 sep 9 10:57 1662769412.M329925P1259441.xyz.com,S=544765,W=552045:2, -rw-r--r-- 1 coord.ventas ventas 163491 sep 5 12:14 1662769412.M74257P1259441.xyz.com,S=163491,W=165846:2,S -rw-r--r-- 1 coord.ventas ventas 3043536 sep 9 12:02 1662769412.M777084P1259441.xyz.com,S=3043536,W=3083246:2, -rw-r--r-- 1 coord.ventas ventas 161002 feb 27 2020 1662874173.M608062P1282222.xyz.com,S=161002,W=163373:2,S -rw-r--r-- 1 coord.ventas ventas 2230491 feb 27 2020 1662874176.M281294P1282222.xyz.com,S=2230491,W=2259506:2,S -rw-r--r-- 1 coord.ventas ventas 167925 feb 27 2020 1662874176.M741229P1282222.xyz.com,S=167925,W=170373:2,S
The process of renaming files is continuous until all folder was reread and renamed every file on it. If I issue 'lsof' (with grep) I see the file that is being renamed passing through tmp/ directory and after that, the file is placed under new/ directory (I guess) and Outlook sees a new email to sync, making the sync process too long... mails that have been synced are resynced...
Take a look at the timestamp on filenames: 1662769412 corresponds to "Fri Sep 9 20:23:32 -04 2022" and 1662874176 corresponds to Sun Sep 11 01:29:36 -04 2022 but mails were neither sent nor received on that dates.
What is Outlook requesting imap server to do and how to avoid this?
Thanks
--
Emilio Augusto Lazo Zaia
"Emilio" == Emilio Augusto Lazo Zaia <emiliolazozaia@gmail.com> writes:
I have a setup on which dovecot is the server in a domain on which mailboxes are >20GB, some of them are 150GB or more.
It looks like you're using Maildir format? In any case, please post more details of your configuration.
When reconfiguring _some_ mailboxes with Outlook 365 (configuring a previously used mailbox on a new PC), the process takes too long even if mail visibility on Outlook is 3 months.
So the outlook client can't download and parse the number of messages on there without breaking? How many messages are there? It's probably not the size of the mailbox, but the number of messages.
Inspecting at INBOX cur directory I did note that the 'ls' command shows the last file listed is a file from year 2020, 2019 or whatever not being the last received mail; that means the alphabetical ordering of 'ls' is showing past years after the last received mail, i.e. dovecot changed the epoch in filename. This only happens with Outlook. Following is an excerpt of ls:
Sure, 'ls -l' doesn't do any sorting, it just reads the directory information as returned from the disk and show you the results. If you want it by time, you need to do:
ls -ltrto have the newest files be at the end. But how 'ls' sees the directry entries doen't matter to dovecot.
-rw-r--r-- 1 coord.ventas ventas 544765 sep 9 10:57 1662769412.M329925P1259441.xyz.com,S= 544765,W=552045:2, -rw-r--r-- 1 coord.ventas ventas 163491 sep 5 12:14 1662769412.M74257P1259441.xyz.com,S= 163491,W=165846:2,S -rw-r--r-- 1 coord.ventas ventas 3043536 sep 9 12:02 1662769412.M777084P1259441.xyz.com,S= 3043536,W=3083246:2, -rw-r--r-- 1 coord.ventas ventas 161002 feb 27 2020 1662874173.M608062P1282222.xyz.com,S= 161002,W=163373:2,S -rw-r--r-- 1 coord.ventas ventas 2230491 feb 27 2020 1662874176.M281294P1282222.xyz.com,S= 2230491,W=2259506:2,S -rw-r--r-- 1 coord.ventas ventas 167925 feb 27 2020 1662874176.M741229P1282222.xyz.com,S= 167925,W=170373:2,S
The process of renaming files is continuous until all folder was reread and renamed every file on it. If I issue 'lsof' (with grep) I see the file that is being renamed passing through tmp/ directory and after that, the file is placed under new/ directory (I guess) and Outlook sees a new email to sync, making the sync process too long... mails that have been synced are resynced...
Wait what? Can you explain exactly what you're doing here? Why would O365 client be renaming the file
Take a look at the timestamp on filenames: 1662769412 corresponds to "Fri Sep 9 20:23:32 -04 2022" and 1662874176 corresponds to Sun Sep 11 01:29:36 -04 2022 but mails were neither sent nor received on that dates.
What is Outlook requesting imap server to do and how to avoid this?
What is the outlook client version? What are your connection settings? Have you looked in the logs on the dovecot server side to see what they say?
What working clients are your users using today? Are they working properly with older clients?
Basically, you have to assume we know nothing and explain your setup and configuration to us like we're idiots who don't know anything. Spell out all the details please. :-)
John
Dnia 12.09.2022 o godz. 12:30:29 John Stoffel pisze:
Sure, 'ls -l' doesn't do any sorting, it just reads the directory information as returned from the disk and show you the results. If you want it by time, you need to do:
ls -ltrto have the newest files be at the end.
'ls' (and 'ls -l' as well) by default sorts files alphabetically. It has always been so. For unsorted list, one needs to use 'ls -lU' (that applies to GNU version of 'ls'; other versions may not recognize the '-U' switch).
Regards, Jaroslaw Rafa raj@rafa.eu.org
"In a million years, when kids go to school, they're gonna know: once there was a Hushpuppy, and she lived with her daddy in the Bathtub."
"Jaroslaw" == Jaroslaw Rafa <raj@rafa.eu.org> writes:
Dnia 12.09.2022 o godz. 12:30:29 John Stoffel pisze:
Sure, 'ls -l' doesn't do any sorting, it just reads the directory information as returned from the disk and show you the results. If you want it by time, you need to do:
ls -ltr
to have the newest files be at the end.
'ls' (and 'ls -l' as well) by default sorts files alphabetically. It has always been so.
Ooops, you're right! I was blanking on that since I was still thinking of the opendir() and readdir() calls which don't sort directory entries.
For unsorted list, one needs to use 'ls -lU' (that applies to GNU version of 'ls'; other versions may not recognize the '-U' switch).
Yup!
On 2022-09-13 01:06, John Stoffel wrote:
"Jaroslaw" == Jaroslaw Rafa <raj@rafa.eu.org> writes:
Dnia 12.09.2022 o godz. 12:30:29 John Stoffel pisze:
Sure, 'ls -l' doesn't do any sorting, it just reads the directory information as returned from the disk and show you the results. If you want it by time, you need to do:
ls -ltr
to have the newest files be at the end.
'ls' (and 'ls -l' as well) by default sorts files alphabetically. It has always been so.
Ooops, you're right! I was blanking on that since I was still thinking of the opendir() and readdir() calls which don't sort directory entries.
For unsorted list, one needs to use 'ls -lU' (that applies to GNU version of 'ls'; other versions may not recognize the '-U' switch).
Yup!
Also, ls -lt the other way round.
Hello John! Thanks for your answer.
On 12/9/22 12:30, John Stoffel wrote:
I have a setup on which dovecot is the server in a domain on which mailboxes are >20GB, some of them are 150GB or more. It looks like you're using Maildir format? In any case, please post more details of your configuration.
Sure. It is a simple setup, with Maildir.
# doveconf -N
auth_debug = no auth_mechanisms = plain login auth_verbose = no debug_log_path = /var/log/syslog/mail.debug disable_plaintext_auth = no login_greeting = Hermes. mail_debug = yes mail_index_rewrite_max_log_bytes = 1 M mail_location = maildir:~/Maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam name = } protocols = " imap" service auth { unix_listener auth-client { group = Debian-exim mode = 0660 user = Debian-exim } } service imap { vsz_limit = 2 G } ssl = yes ssl_cert = </etc/dovecot/dovecot-cert.pem ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = passwd name = } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 20 }
When reconfiguring _some_ mailboxes with Outlook 365 (configuring a previously used mailbox on a new PC), the process takes too long even if mail visibility on Outlook is 3 months. So the outlook client can't download and parse the number of messages on there without breaking? How many messages are there? It's probably not the size of the mailbox, but the number of messages. 50K mails on some folders... total 5.5M on the whole server. I did discover which was the problem but I don't have a solution.
Inspecting at INBOX cur directory I did note that the 'ls' command shows the last file listed is a file from year 2020, 2019 or whatever not being the last received mail; that means the alphabetical ordering of 'ls' is showing past years after the last received mail, i.e. dovecot changed the epoch in filename. This only happens with Outlook. Following is an excerpt of ls: Sure, 'ls -l' doesn't do any sorting, it just reads the directory information as returned from the disk and show you the results. If you want it by time, you need to do:
ls -ltrto have the newest files be at the end. But how 'ls' sees the directry entries doen't matter to dovecot. Newest files are at the end; 'ls' without '-U' or '--sort=none' lists files ordered alphabetically, so the last entry would be the last mail placed on the folder because the epoch time is part of the filename in Maildir structure...
-rw-r--r-- 1 coord.ventas ventas 544765 sep 9 10:57 1662769412.M329925P1259441.xyz.com,S= 544765,W=552045:2, -rw-r--r-- 1 coord.ventas ventas 163491 sep 5 12:14 1662769412.M74257P1259441.xyz.com,S= 163491,W=165846:2,S -rw-r--r-- 1 coord.ventas ventas 3043536 sep 9 12:02 1662769412.M777084P1259441.xyz.com,S= 3043536,W=3083246:2, -rw-r--r-- 1 coord.ventas ventas 161002 feb 27 2020 1662874173.M608062P1282222.xyz.com,S= 161002,W=163373:2,S -rw-r--r-- 1 coord.ventas ventas 2230491 feb 27 2020 1662874176.M281294P1282222.xyz.com,S= 2230491,W=2259506:2,S -rw-r--r-- 1 coord.ventas ventas 167925 feb 27 2020 1662874176.M741229P1282222.xyz.com,S= 167925,W=170373:2,S The process of renaming files is continuous until all folder was reread and renamed every file on it. If I issue 'lsof' (with grep) I see the file that is being renamed passing through tmp/ directory and after that, the file is placed under new/ directory (I guess) and Outlook sees a new email to sync, making the sync process too long... mails that have been synced are resynced... Wait what? Can you explain exactly what you're doing here? Why would O365 client be renaming the file
Ok, I disabled TLS options and inspected the traffic with tcpdump. I did note unnecesary STORE, EXPUNGE and APPEND operations on previously synced folder. I guess the whole mailbox is analyzed (from the local cache on .pst file) and O365 tags as SPAM a large number of innocent emails (including local mails from the same domain which does not travel through internet to reach the destination).
Also I'm using port 144 instead of 143 to bypass GData antivirus SPAM checks. Anyway I completely disabled GData antivirus and none of its services remains running on these workstations; also starting Outlook with safe mode exhibits this behavior, so I don't know which feature is this on O365 that rescan all stored emails but it is beyond the scope of this list because there is no feature on dovecot that can be enabled to workaround this. I guess this is a bug on O365.
All mails are retransmitted to the server with the subject modified (tagging them as SPAM or SPAM suspected) and obviously dovecot stores them with a new epoch timestamp on filename and that's why I have mails with older timestamp but with recent epoch time on filename...
Take a look at the timestamp on filenames: 1662769412 corresponds to "Fri Sep 9 20:23:32 -04 2022" and 1662874176 corresponds to Sun Sep 11 01:29:36 -04 2022 but mails were neither sent nor received on that dates. What is Outlook requesting imap server to do and how to avoid this? What is the outlook client version? What are your connection settings? Have you looked in the logs on the dovecot server side to see what they say? Outlook 365 is version 16.15601.20148 compilation 2208 on one workstation that exhibits the issue and this same version on another PC on which this issue is not present... What working clients are your users using today? Are they working properly with older clients?
Basically I'm the mail server administrator not the sysadmin on the network. All users prefer working with O365. I've installed Thunderbird on every PC but users refuses to use it in favor of using Outlook, unfortunately.
Basically, you have to assume we know nothing and explain your setup and configuration to us like we're idiots who don't know anything. Spell out all the details please. :-)
Yes, I'm sorry. I hope with this mail I've clarified the scenario.
Thanks again!
Emilio
"Emilio" == Emilio Augusto Lazo Zaia <emiliolazozaia@gmail.com> writes:
Hello John! Thanks for your answer.
You're welcome, but I'm not sure I'm going to be able to help here. I suspect this is a client issue, just like I hate how the Apple Mail client marks messages SEEN for no reason at all because it does "SELECT INBOX" vs "EXAMINE INBOX".
On 12/9/22 12:30, John Stoffel wrote:
I have a setup on which dovecot is the server in a domain on which mailboxes are >20GB, some of them are 150GB or more.
It looks like you're using Maildir format? In any case, please post more details of your configuration.
Sure. It is a simple setup, with Maildir.
# doveconf -N
auth_debug = no auth_mechanisms = plain login auth_verbose = no debug_log_path = /var/log/syslog/mail.debug disable_plaintext_auth = no login_greeting = Hermes. mail_debug = yes mail_index_rewrite_max_log_bytes = 1 M mail_location = maildir:~/Maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam name = } protocols = " imap" service auth { unix_listener auth-client { group = Debian-exim mode = 0660 user = Debian-exim } } service imap { vsz_limit = 2 G } ssl = yes ssl_cert = </etc/dovecot/dovecot-cert.pem ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = passwd name = } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 20 }
When reconfiguring _some_ mailboxes with Outlook 365 (configuring a previously used mailbox on a new PC), the process takes too long even if mail visibility on Outlook is 3 months.
So the outlook client can't download and parse the number of messages on there without breaking? How many messages are there? It's probably not the size of the mailbox, but the number of messages.
50K mails on some folders... total 5.5M on the whole server. I did discover which was the problem but I don't have a solution.
Yeah, that's alot. You might need to push your clients to start putting more emails into sub-folders, maybe split up by year say, so they don't have quite so many emails sitting around.
Inspecting at INBOX cur directory I did note that the 'ls' command shows the last file listed is a file from year 2020, 2019 or whatever not being the last received mail; that means the alphabetical ordering of 'ls' is showing past years after the last received mail, i.e. dovecot changed the epoch in filename. This only happens with Outlook. Following is an excerpt of ls:
Sure, 'ls -l' doesn't do any sorting, it just reads the directory information as returned from the disk and show you the results. If you want it by time, you need to do:
ls -ltr
to have the newest files be at the end. But how 'ls' sees the directry entries doen't matter to dovecot.
Newest files are at the end; 'ls' without '-U' or '--sort=none' lists files ordered alphabetically, so the last entry would be the last mail placed on the folder because the epoch time is part of the filename in Maildir structure...
-rw-r--r-- 1 coord.ventas ventas 544765 sep 9 10:57 1662769412.M329925P1259441.xyz.com,S= 544765,W=552045:2, -rw-r--r-- 1 coord.ventas ventas 163491 sep 5 12:14 1662769412.M74257P1259441.xyz.com,S= 163491,W=165846:2,S -rw-r--r-- 1 coord.ventas ventas 3043536 sep 9 12:02 1662769412.M777084P1259441.xyz.com,S= 3043536,W=3083246:2, -rw-r--r-- 1 coord.ventas ventas 161002 feb 27 2020 1662874173.M608062P1282222.xyz.com,S= 161002,W=163373:2,S -rw-r--r-- 1 coord.ventas ventas 2230491 feb 27 2020 1662874176.M281294P1282222.xyz.com,S= 2230491,W=2259506:2,S -rw-r--r-- 1 coord.ventas ventas 167925 feb 27 2020 1662874176.M741229P1282222.xyz.com,S= 167925,W=170373:2,S
The process of renaming files is continuous until all folder was reread and renamed every file on it. If I issue 'lsof' (with grep) I see the file that is being renamed passing through tmp/ directory and after that, the file is placed under new/ directory (I guess) and Outlook sees a new email to sync, making the sync process too long... mails that have been synced are resynced...
Wait what? Can you explain exactly what you're doing here? Why would O365 client be renaming the file
Ok, I disabled TLS options and inspected the traffic with tcpdump. I did note unnecesary STORE, EXPUNGE and APPEND operations on previously synced folder. I guess the whole mailbox is analyzed (from the local cache on .pst file) and O365 tags as SPAM a large number of innocent emails (including local mails from the same domain which does not travel through internet to reach the destination).
So this is really a client issue. You might need to ask around on the Microsoft site to see if there are any settings you can change when it's accessing a server using IMAP. Also, look in the dovecot archives.
Also I'm using port 144 instead of 143 to bypass GData antivirus SPAM checks. Anyway I completely disabled GData antivirus and none of its services remains running on these workstations; also starting Outlook with safe mode exhibits this behavior, so I don't know which feature is this on O365 that rescan all stored emails but it is beyond the scope of this list because there is no feature on dovecot that can be enabled to workaround this. I guess this is a bug on O365.
Yup, sorry!
All mails are retransmitted to the server with the subject modified (tagging them as SPAM or SPAM suspected) and obviously dovecot stores them with a new epoch timestamp on filename and that's why I have mails with older timestamp but with recent epoch time on filename...
Take a look at the timestamp on filenames: 1662769412 corresponds to "Fri Sep 9 20:23:32 -04 2022" and 1662874176 corresponds to Sun Sep 11 01:29:36 -04 2022 but mails were neither sent nor received on that dates.
What is Outlook requesting imap server to do and how to avoid this?
What is the outlook client version? What are your connection settings? Have you looked in the logs on the dovecot server side to see what they say?
Outlook 365 is version 16.15601.20148 compilation 2208 on one workstation that exhibits the issue and this same version on another PC on which this issue is not present...
What working clients are your users using today? Are they working properly with older clients?
Basically I'm the mail server administrator not the sysadmin on the network. All users prefer working with O365. I've installed Thunderbird on every PC but users refuses to use it in favor of using Outlook, unfortunately.
Basically, you have to assume we know nothing and explain your setup and configuration to us like we're idiots who don't know anything. Spell out all the details please. :-)
Yes, I'm sorry. I hope with this mail I've clarified the scenario.
Thanks again!
Emilio
participants (4)
-
Emilio Augusto Lazo Zaia
-
hi＠zakaria.website
-
Jaroslaw Rafa
-
John Stoffel