Dear Dovecot community,
As discussing on IRC with Aki, I have the following thing happening:
I got an alert from my mailservice last night that I was running over quota. My quota reached almost 91% (coming from 30%) in one day. I do not recall receiving so much email (we talk about multiple gb’s additional space usage), so something was acting up.
When I logged into my mailservers (I’ll describe my setup later on), it mentioned that my FreeBSD mailbox was reaching 12gb of space usage. While the seeming replicated machine (named A) only had 100mb of space in use (which, with the amount of messages in the various mailboxes is consistent with what I would expect).
I found out that the “Junk” mailbox was the ‘major' issue. I vaporized the replicated (12GB) mailbox and issued a ‘doveadm sync -u $user -N -U tcps:hostB` so that I would expect the hostB to reach around 100mb-ish of email. I let it go for a little and saw that the mail was done. When I issued a simple ‘df -h’ on the directory, it was already multiple 100mb’s large and growing. It grew back to multiple GB in little time. Again the Junk mailbox was the culprit. In the mean time I noticed my mailclient downloading 44k mails in that mailbox (where there were only a few in the Junk mailbox normally).
I again vaporized the entire mailbox on hostB and cleaned out Junk and issued a manual synchronisation. It again grew quickly.
Aki did an investigation with me and noticed that my rspamc script was not very well written. I rewrote it as suggested (As written on the site).
This morning I continued with the investigation and it was again large on hostB where it was still 100mb-ish on hostA. I vaporized host B again and issued the manual sync with -Dv included per request of Aki. After letting it run for a little hostA grew to 300mb and hostB restarted to reach 1.2GB and remained static there. After half an hour or so the mailbox on hostA grew to 900mb and stayed there.
I investigated the mailboxes and saw 8 times the same email. Cleaning them up with doveadm deduplicate -u $user mailbox ‘*’ made sure a lot of email was cut from both hosts. I am now in the state where:
hostA: 268MB hostB: 579MB
Both are for the same mailbox (I suspect that other mailboxes have similiar issues). When I look at the amount of messages counted by Apple Mail in the particular mailbox I see 1399 emails. The amount of ‘u’ files is 14722 on hostA and 23590 on hostB.
I do not understand the difference from my mail client and the amount of u files.
from mutt: Mutt: =mailbox [Msgs:1415 New:1 Flag:1 24M]-, which is roughly the same as Apple mail reports.
I am a bit lost on where to look next, I would suspect that the synchronised mailboxes would be similar in size, but not the one 2 as large, or before that it grew and grew and grew.
My personal feeling is that the sync takes place, but somehow it is not registering that a message had been synchronised and does that again, and again, and again till the message gets synchronised right and the acknowledge is received and it stays the same. That would at least support the 8times the same email that I saw at first and might also support the roughly double as big size of the other host (one sync goes wrong, the next sync went right and got acked). At the bottom I also referenced two previous emails of mine which report similar oddities. Messages that were removed on hostA got back within seconds as “new message” . My feeling there is that the hostB has the ‘failed sync copy with a different uid/guid’ and tells hostA that there is a new message and sends it over.
My doveconf -n for hostA (hostB follows), I did sanitize a bit wrt. IP addresses and hostnames:
2.2.33.2 (d6601f4ec): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.21 (92477967) # OS: FreeBSD 11.1-RELEASE amd64 auth_mechanisms = plain login disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes lmtp_save_to_detail_mailbox = yes login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c %k mail_debug = yes mail_fsync = always mail_location = sdbox:~/sdbox mail_plugins = " quota notify replication" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve namespace { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { auto = subscribe special_use = \Junk } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = . } passdb { driver = pam } plugin { imapsieve_mailbox1_before = file:/usr/local/lib/dovecot/sieve/report-spam.sieve imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_name = Spam imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = Spam imapsieve_mailbox2_name = * mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size mail_replica = tcps:hostB:12346 sieve = ~/.dovecot.sieve sieve_after = /usr/local/etc/dovecot/sieve/after/ sieve_before = /usr/local/etc/dovecot/sieve/global/ sieve_dir = ~/sieve sieve_execute_bin_dir = /usr/local/lib/dovecot/sieve sieve_global = /usr/local/etc/dovecot/sieve/global/ sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.execute sieve_global_path = /usr/local/etc/dovecot/sieve/global/dovecot.sieve sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve sieve_plugins = sieve_imapsieve sieve_extprograms sieve_vacation_dont_check_recipient = yes } postmaster_address = postmaster@isp protocols = imap pop3 lmtp sieve service aggregator { fifo_listener replication-notify-fifo { mode = 0666 } unix_listener replication-notify { mode = 0666 } } service auth { unix_listener /var/spool/postfix/private/auth { mode = 0666 } } service doveadm { inet_listener { port = 12346 ssl = yes } } service imap-login { service_count = 1 } service imap { process_limit = 1024 } service lmtp { inet_listener lmtp { address = external ipv4 and ipv6 127.0.0.1 ::1 port = 24 } unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service pop3 { process_limit = 1024 } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } ssl_ca =
for hostB:
# 2.2.33.2 (d6601f4ec): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.21 (92477967) # OS: FreeBSD 11.1-RELEASE-p1 amd64 auth_mechanisms = plain login disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes lmtp_save_to_detail_mailbox = yes login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c %k mail_debug = yes mail_fsync = always mail_location = sdbox:~/sdbox mail_plugins = " quota notify replication" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve namespace { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { auto = subscribe special_use = \Junk } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = . } passdb { driver = pam } plugin { imapsieve_mailbox1_before = file:/usr/local/lib/dovecot/sieve/report-spam.sieve imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_name = Spam imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = Spam imapsieve_mailbox2_name = * mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size mail_replica = tcps:hostA:12346 sieve = ~/.dovecot.sieve sieve_after = /usr/local/etc/dovecot/sieve/after/ sieve_before = /usr/local/etc/dovecot/sieve/global/ sieve_dir = ~/sieve sieve_execute_bin_dir = /usr/local/lib/dovecot/sieve sieve_global = /usr/local/etc/dovecot/sieve/global/ sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.execute sieve_global_path = /usr/local/etc/dovecot/sieve/global/dovecot.sieve sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve sieve_plugins = sieve_imapsieve sieve_extprograms sieve_vacation_dont_check_recipient = yes } postmaster_address = postmaster@isp protocols = imap pop3 lmtp sieve service aggregator { fifo_listener replication-notify-fifo { mode = 0666 } unix_listener replication-notify { mode = 0666 } } service auth { unix_listener /var/spool/postfix/private/auth { mode = 0666 } } service doveadm { inet_listener { port = 12346 ssl = yes } } service imap-login { service_count = 1 } service imap { process_limit = 1024 } service lmtp { inet_listener lmtp { address = ipv4 and ipv6 address 127.0.0.1 ::1 port = 24 } unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service pop3 { process_limit = 1024 } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } ssl_ca = https://dovecot.org/list/dovecot/2016-July/104873.html and https://www.dovecot.org/list/dovecot/2017-March/107459.html