Users losing mail from their mailboxes
Hello,
We are on Dovecot 2.3.18.
Some users are occasionally complaining that all mail vanishes from their inbox.
However, we cannot find any indication of deletions or expunge in the log.
One explanation could be that mail files are deleted externally (at the file system level), but I can't imagine what could cause such deletions.
How would you suggest us to troubleshoot this further?
We could enable "mail_debug = yes", but this would flood the logs and cause performance drop, and the issue is happening rarely.
Any other ideas to troubleshoot these cases in the past (through logs) or monitor (whatever) for the future?
(The config is listed at the end.)
Thanks in advance.
Best regards, Nick
The config follows:
=======================================================================
protocols = imap pop3 sieve lmtp
login_greeting = Dovecot OUR ORG
log_path = /var/log/dove.log
mail_location = maildir:~/Maildir/
mail_gid = 500
mail_uid = 500
auth_mechanisms = plain login
auth_username_format = %Ln
auth_verbose = no
auth_debug = no
mail_debug = no
disable_plaintext_auth = no
mail_plugins = quota mail_log notify
protocol imap {
imap_client_workarounds = "delay-newmail"
mail_plugins = quota imap_quota mail_log notify
mail_max_userip_connections = 400
namespace inbox {
mailbox Trash {
autoexpunge = 15d
}
}
}
protocol pop3 {
mail_max_userip_connections = 3
mail_plugins = quota notify
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_uidl_format = %08Xu%08Xv
namespace inbox {
mailbox Trash {
autoexpunge = 15d
}
}
}
protocol lda {
auth_socket_path = /var/run/dovecot/auth-master
mail_plugins = quota notify sieve
postmaster_address = sysadmin@noa.gr
sendmail_path = /usr/lib/sendmail
}
protocol lmtp {
auth_socket_path = /var/run/dovecot/auth-master
postmaster_address = sysadmin@noa.gr
mail_plugins = quota notify sieve
sendmail_path = /usr/lib/sendmail
}
protocol sieve {
managesieve_max_line_length = 65536
mail_max_userip_connections = 10
managesieve_logout_format = bytes=%i/%o
managesieve_max_compile_errors = 10
}
userdb {
args = /etc/dovecot/dovecot-usrdb-ldap.conf
driver = ldap
}
passdb {
args = /etc/dovecot/dovecot-passdb-ldap.conf
driver = ldap
}
plugin {
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename flag_change save mailbox_create
mail_log_fields = uid box msgid size flags vsize from subject
quota = maildir:User quota
quota_rule = *:storage=15G
quota_rule2 = Trash:storage=+3%%
quota_warning = storage=75%% quota-warning 75 %u
quota_warning2 = storage=90%% quota-warning 90 %u
# Used by both the Sieve plugin and the ManageSieve protocol
sieve = file:~/sieve;active=~/.dovecot.sieve
sieve_max_script_size = 0
sieve_max_actions = 0
sieve_max_redirects = 2
}
service quota-warning {
executable = script /opt/mail1.sh
user = vmail
unix_listener quota-warning {
user = vmail
}
}
service auth {
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
}
unix_listener auth-master {
group = vmail
mode = 0660
user = vmail
}
user = root
}
service imap-login {
service_count = 100
vsz_limit = 64 M
process_limit = 500
}
service pop3-login {
service_count = 100
vsz_limit = 64 M
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
service_count = 100
process_min_avail = 0
vsz_limit = 64M
}
service managesieve {
process_limit = 1024
}
service imap {
executable = imap postlogin
process_limit = 2048
}
service pop3 {
executable = pop3 postlogin
}
service postlogin {
executable = script-login -d rawlog
unix_listener postlogin {
}
}
service lmtp {
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
}
ssl = yes
ssl_cert = </etc/pki/tls/certs/servercert.crt
ssl_key = </etc/pki/tls/private/serverkey.key
namespace inbox {
separator = .
prefix =
inbox = yes
mailbox Drafts {
special_use = \Drafts
auto = subscribe
}
mailbox Junk {
special_use = \Junk
auto = subscribe
}
mailbox Trash {
special_use = \Trash
auto = subscribe
}
mailbox Sent {
special_use = \Sent
auto = subscribe
}
}=======================================================================
Hello, We are on Dovecot 2.3.18. Some users are occasionally complaining that all mail vanishes from their inbox. However, we cannot find any indication of deletions or expunge in the log. One explanation could be that mail files are deleted externally (at the file system level), but I can't imagine what could cause such deletions. How would you suggest us to troubleshoot this further? We could enable "mail_debug = yes", but this would flood the logs and cause performance drop, and the issue is happening rarely. Any other ideas to troubleshoot these cases in the past (through logs) or monitor (whatever) for the future? (The config is listed at the end.) Thanks in advance. Best regards, Nick The config follows:
protocols = imap pop3 sieve lmtp
login_greeting = Dovecot OUR ORG
log_path = /var/log/dove.log
mail_location = maildir:~/Maildir/
mail_gid = 500
mail_uid = 500
auth_mechanisms = plain login
auth_username_format = %Ln
auth_verbose = no
auth_debug = no
mail_debug = no
disable_plaintext_auth = no
mail_plugins = quota mail_log notify
protocol imap {
imap_client_workarounds = "delay-newmail"
mail_plugins = quota imap_quota mail_log notify
mail_max_userip_connections = 400
namespace inbox {
mailbox Trash {
autoexpunge = 15d
}
}
}
protocol pop3 {
mail_max_userip_connections = 3
mail_plugins = quota notify
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_uidl_format = %08Xu%08Xv
namespace inbox {
mailbox Trash {
autoexpunge = 15d
}
}
}
protocol lda {
auth_socket_path = /var/run/dovecot/auth-master
mail_plugins = quota notify sieve
postmaster_address = sysadmin@noa.gr
sendmail_path = /usr/lib/sendmail
}
protocol lmtp {
auth_socket_path = /var/run/dovecot/auth-master
postmaster_address = sysadmin@noa.gr
mail_plugins = quota notify sieve
sendmail_path = /usr/lib/sendmail
}
protocol sieve {
managesieve_max_line_length = 65536
mail_max_userip_connections = 10
managesieve_logout_format = bytes=%i/%o
managesieve_max_compile_errors = 10
}
userdb {
args = /etc/dovecot/dovecot-usrdb-ldap.conf
driver = ldap
}
passdb {
args = /etc/dovecot/dovecot-passdb-ldap.conf
driver = ldap
}
plugin {
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename flag_change save mailbox_create
mail_log_fields = uid box msgid size flags vsize from subject
quota = maildir:User quota
quota_rule = *:storage=15G
quota_rule2 = Trash:storage=+3%%
quota_warning = storage=75%% quota-warning 75 %u
quota_warning2 = storage=90%% quota-warning 90 %u
# Used by both the Sieve plugin and the ManageSieve protocol
sieve = file:~/sieve;active=~/.dovecot.sieve
sieve_max_script_size = 0
sieve_max_actions = 0
sieve_max_redirects = 2
}
service quota-warning {
executable = script /opt/mail1.sh
user = vmail
unix_listener quota-warning {
user = vmail
}
}
service auth {
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
}
unix_listener auth-master {
group = vmail
mode = 0660
user = vmail
}
user = root
}
service imap-login {
service_count = 100
vsz_limit = 64 M
process_limit = 500
}
service pop3-login {
service_count = 100
vsz_limit = 64 M
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
service_count = 100
process_min_avail = 0
vsz_limit = 64M
}
service managesieve {
process_limit = 1024
}
service imap {
executable = imap postlogin
process_limit = 2048
}
service pop3 {
executable = pop3 postlogin
}
service postlogin {
executable = script-login -d rawlog
unix_listener postlogin {
}
}
service lmtp {
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
}
ssl = yes
ssl_cert = </etc/pki/tls/certs/servercert.crt
ssl_key = </etc/pki/tls/private/serverkey.key
namespace inbox {
separator = .
prefix =
inbox = yes
mailbox Drafts {
special_use = \Drafts
auto = subscribe
}
mailbox Junk {
special_use = \Junk
auto = subscribe
}
mailbox Trash {
special_use = \Trash
auto = subscribe
}
mailbox Sent {
special_use = \Sent
auto = subscribe
}
}=======================================================================
Hi,
In a similar situation, the reason was Outlook - account using IMAP, deleting the emails by the most important client, without any rules or Junk filters.
I would suggest before anything else:
- Create "recipient bcc copy" (like "forward a copy to another mailbox"). You can compare both mailboxes if needed;
- Use the mailbox only by webmail temporary if possible, for the test period;
- Of course, enable the logging...
Cheers A.
On 5/16/25 11:06 AM, Nikolaos Milas via dovecot wrote:
I am on the Debian build 2.3.19.1, and there are outright bugs in that code. I am doing replication between two servers and I have made my own C source changes and recompiled to make it work as well as it does—though I can look at the code and still see clear problems.
I conclude that Dovecot's maintainers decided this code is too broken for them to try to fix, and so they have pulled the replication feature from 2.4.
In my logwatch e-mail this morning the machine that initiates the replication had a crash and a stacktrace (/much/ better than what the stock Debian build was doing, it was reliably blowing up the stack; most days my modified code does not crash even once).
I am running a small personal site, I do a complete backup of all user maildirs every night, and keep 15-days worth of backups¹. I do not trust Dovecot.
All public support for 2.3 has ended. I conclude that 2.3 is bad news and users of 2.3 need to get off 2.3.
There is the current (very recently released) 2.4.
From what I have seen here migrating to 2.4 is tricky, and sometimes 2.4 simply crashes when it doesn't like some configuration detail. Software that crashes on a bad config is really unnerving.
And, 2.4 doesn't have replication.
So both 2.3 and 2.4 look bad to me. I need a new IMAP server, and I don't know what it is.
Someone started writing an IMAP server in Rust a while back². The server is Stalwart, it /might/ be very good, but it seems to be a complete, all-sing all-dancing, do-everything solution. I can't drop Stalwart in as a new IMAP server, I would need to redo everything because Stalwart and its web GUI wants to do everything. (If I do do Stalwart, what if I discover it has its own problems? Stalwart isn't as new as 2.4, but the whole thing is pretty new.)
There are other Linux IMAP servers, but they all seem obscure in the face of Dovecot being the standard everyone uses. Except "everyone" doesn't use /e-mail/ that much anymore, and when they do use e-mail it is via something like Gmail and their web GUI. It feels like IMAP itself is getting pretty obscure.
Grrr.
-kb, the Kent who, in using replication, knows he is using an obscure feature of the (more and more) obscure Dovecot, implementing the (more and more) obscure protocol IMAP.
Using the
--link-destoption of rsync files that haven't changed from one incremental backup to the next do not get duplicated, rather both backups will have hard links to the same data. Not as fast as file systems with snapshot features, but still very cool.Rust simply could not have the bugs I was seeing in the Dovecot sources, the compiler would error all over the place on code like that. Yes, the Rust compiler can be annoying, but once it /is/ happy with code there are legions of bugs that simply cannot exist.
On 5/16/25 11:06 AM, Nikolaos Milas via dovecot wrote: We are on Dovecot 2.3.18. Some users are occasionally complaining that all mail vanishes from their inbox. I am on the Debian build 2.3.19.1, and there are outright bugs in that code. I am doing replication between two servers and I have made my own C source changes and recompiled to make it work as well as it does—though I can look at the code and still see clear problems. I conclude that Dovecot's maintainers decided this code is too broken for them to try to fix, and so they have pulled the replication feature from 2.4. In my logwatch e-mail this morning the machine that initiates the replication had a crash and a stacktrace (much better than what the stock Debian build was doing, it was reliably blowing up the stack; most days my modified code does not crash even once). I am running a small personal site, I do a complete backup of all user maildirs every night, and keep 15-days worth of backups¹. I do not trust Dovecot. All public support for 2.3 has ended. I conclude that 2.3 is bad news and users of 2.3 need to get off 2.3. There is the current (very recently released) 2.4. From what I have seen here migrating to 2.4 is tricky, and sometimes 2.4 simply crashes when it doesn't like some configuration detail. Software that crashes on a bad config is really unnerving. And, 2.4 doesn't have replication. So both 2.3 and 2.4 look bad to me. I need a new IMAP server, and I don't know what it is. Someone started writing an IMAP server in Rust a while back². The server is Stalwart, it might be very good, but it seems to be a complete, all-sing all- dancing, do-everything solution. I can't drop Stalwart in as a new IMAP server, I would need to redo everything because Stalwart and its web GUI wants to do everything. (If I do do Stalwart, what if I discover it has its own problems? Stalwart isn't as new as 2.4, but the whole thing is pretty new.) There are other Linux IMAP servers, but they all seem obscure in the face of Dovecot being the standard everyone uses. Except "everyone" doesn't use e-mail that much anymore, and when they do use e-mail it is via something like Gmail and their web GUI. It feels like IMAP itself is getting pretty obscure. Grrr.
-kb, the Kent who, in using replication, knows he is using an obscure feature of the (more and more) obscure Dovecot, implementing the (more and more) obscure protocol IMAP.
- Using the
--link-destoption of rsync files that haven't changed from one incremental backup to the next do not get duplicated, rather both backups will have hard links to the same data. Not as fast as file systems with snapshot features, but still very cool. - Rust simply could not have the bugs I was seeing in the Dovecot sources, the compiler would error all over the place on code like that. Yes, the Rust compiler can be annoying, but once it is happy with code there are legions of bugs that simply cannot exist.
On 16/5/2025 7:37 μ.μ., Kent Borg wrote:
Hi Kent,
Thanks for your really insightful philosophical AND technical review!
Your insight seems deep as you obviously are comfortable with source code analysis and development.
I am a simple admin setting things up (a poor man's mail server) and expecting them to run smoothly. Dovecot has been fine in our use case for many years as far as I can tell.
Now, I don't know if the issue I have faced has somehow been caused by a Dovecot bug or not; I would really appreciate if someone could point to some known bug (fixed or not) as a possible cause.
I am not looking to migrate away from Dovecot, but I would really like to be pointed to the right direction in isolating the cause of the issue. For example, could it be caused by storage performance issues which could cause high iowait? (We occasionally have such issues on our VPS server hosting the mail services.)
As I am preparing a new VPS server with -hopefully- better storage performance, I will migrate to 2.4 and see how it goes. Yet, our production environment is still on 2.3 and it will take some time to move to the new server.
Incidentally, the Stalwart project seems quite interesting, even if not so widely adopted or reviewed! If you have time to put it in a semi-production / sem-test environment, it might be worth the effort.
Best, Nick
On 16/5/2025 7:37 μ.μ., Kent Borg wrote: ...
So both 2.3 and 2.4 look bad to me. I need a new IMAP server, and I don't know what it is. Someone started writing an IMAP server in Rust a while back². The server is Stalwart, it might be very good, but it seems to be a complete, all-sing all- dancing, do-everything solution. I can't drop Stalwart in as a new IMAP server, I would need to redo everything because Stalwart and its web GUI wants to do everything. (If I do do Stalwart, what if I discover it has its own problems? Stalwart isn't as new as 2.4, but the whole thing is pretty new.) There are other Linux IMAP servers, but they all seem obscure in the face of Dovecot being the standard everyone uses. Except "everyone" doesn't use e-mail that much anymore, and when they do use e-mail it is via something like Gmail and their web GUI. It feels like IMAP itself is getting pretty obscure. Grrr. -kb, the Kent who, in using replication, knows he is using an obscure feature of the (more and more) obscure Dovecot, implementing the (more and more) obscure protocol IMAP. ... Hi Kent, Thanks for your really insightful philosophical AND technical review! Your insight seems deep as you obviously are comfortable with source code analysis and development. I am a simple admin setting things up (a poor man's mail server) and expecting them to run smoothly. Dovecot has been fine in our use case for many years as far as I can tell. Now, I don't know if the issue I have faced has somehow been caused by a Dovecot bug or not; I would really appreciate if someone could point to some known bug (fixed or not) as a possible cause. I am not looking to migrate away from Dovecot, but I would really like to be pointed to the right direction in isolating the cause of the issue. For example, could it be caused by storage performance issues which could cause high iowait? (We occasionally have such issues on our VPS server hosting the mail services.) As I am preparing a new VPS server with -hopefully- better storage performance, I will migrate to 2.4 and see how it goes. Yet, our production environment is still on 2.3 and it will take some time to move to the new server. Incidentally, the Stalwart project seems quite interesting, even if not so widely adopted or reviewed! If you have time to put it in a semi-production / sem-test environment, it might be worth the effort. Best, Nick
On 16/05/2025 17:06, Nikolaos Milas via dovecot wrote:
deliver_log_format = msgid=%m, storage_id=%{storage_id}
so you will know file name for stored email and then
inotifywatch to recursively monitor maildir. You will know what deletes that file and you could focus on debugging that.
Best regards,> Nick
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
On 16/5/2025 10:28 μ.μ., Arkadiusz Miśkiewicz via dovecot wrote:
Thank you for the hint.
Could you suggest a specific setup to run inotifywatch or inotifywait as a service and monitor the whole of /home/vmail for file deletions in a way that both the deleted filename and the deleting process are logged?
It seems that inotifywait cannot identify the deleting process because it operates at the filesystem level and doesn’t have insight into process/user metadata.
I have seen suggestions to use inotifywait and auditd combined, but setups are not that straightforward. Moreover, I think that the number of file events/deletions is probably large so we might end up in huge log files.
Please advise!
Thanks, Nick
On 16/05/2025 18:06 EEST Nikolaos Milas via dovecot <dovecot@dovecot.org> wrote: Hello, We are on Dovecot 2.3.18. Some users are occasionally complaining that all mail vanishes from their inbox. However, we cannot find any indication of deletions or expunge in the log. One explanation could be that mail files are deleted externally (at the file system level), but I can't imagine what could cause such deletions. How would you suggest us to troubleshoot this further? We could enable "mail_debug = yes", but this would flood the logs and cause performance drop, and the issue is happening rarely. Any other ideas to troubleshoot these cases in the past (through logs) or monitor (whatever) for the future? (The config is listed at the end.) Thanks in advance. Best regards, Nick The config follows: I would strongly suggest enabling mail_log plugin globally and not just for protocol imap. This should enable you to see why the mails are deleted. Aki
On 17/5/2025 10:56 μ.μ., Aki Tuomi via dovecot wrote:
I would strongly suggest enabling mail_log plugin globally and not just for protocol imap. This should enable you to see why the mails are deleted.
Hi Aki,
I thought I have done so by including:
mail_plugins = quota mail_log notify
at the "root" config level. (I have posted my whole config.) If I should do something more to enable the mail_log plugin globally, please correct me. Thanks a lot, Nick
On 17/5/2025 10:56 μ.μ., Aki Tuomi via dovecot wrote: I would strongly suggest enabling mail_log plugin globally and not just for protocol imap. This should enable you to see why the mails are deleted. Hi Aki, I thought I have done so by including: mail_plugins = quota mail_log notify at the "root" config level. (I have posted my whole config.)
If I should do something more to enable the mail_log plugin globally, please correct me.
Thanks a lot, Nick
On 18/05/2025 00:18 EEST Nikolaos Milas via dovecot
<dovecot@dovecot.org> wrote:
On 17/5/2025 10:56 μ.μ., Aki Tuomi via dovecot wrote:
I would strongly suggest enabling mail_log plugin globally
and not just for
protocol imap. This should enable you to see why the mails
are deleted.
Hi Aki,
I thought I have done so by including:
mail_plugins = quota mail_log notify
at the "root" config level. (I have posted my whole config.) If I
should
do something more to enable the mail_log plugin globally, please
correct
me. Thanks a lot, Nick
On 17/5/2025 10:56 μ.μ., Aki Tuomi via dovecot wrote:
I would strongly suggest enabling mail_log plugin globally and not
just for
protocol imap. This should enable you to see why the mails are
deleted.
Hi Aki,
I thought I have done so by including:
mail_plugins = quota mail_log notify
at the "root" config level. (I have posted my whole config.)
If I should do something more to enable the mail_log plugin globally,
please
correct me.
Thanks a lot,
Nick
_______________________________________________
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-leave@dovecot.org
Make sure you use mail_plugins = $mail_plugins foo bar when doing per-
protocol plugins to avoid overriding global plugins. And ensure global plugins
is done first before any !include or !try_include statements.
Aki
participants (5)
-
Aki Tuomi
-
alex＠anilex.com
-
Arkadiusz Miśkiewicz
-
Kent Borg
-
Nikolaos Milas