<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I'm using haproxy because of convenience and familiarity, the setup also seemed easier then director.
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The manual does state to use a stick-table with dovecot IMAP.... my issue is with Postfix & Dovecot LMTP though(along with replication)
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Using stick-tables (based on source IP) in the same way wouldn't solve the issue. Since different IPs can send emails to the same mailbox and because replication occurs simultaneously on different servers you might still run into the problem.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
the haproxy postfix backend would need to be setup in an active/passive way... or using stick-tables to ensure only 1 server is active at a time.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The following configuration balances the servers so they switch off every minute that passes without a connection coming through. Ensuring only one server is receiving mail.
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
backend smtp_10025
<div> balance roundrobin</div>
<div> option tcp-check</div>
<div> server mail01 10.0.0.100:10025 check check-send-proxy send-proxy</div>
<div> server mail02 10.0.0.101:10025 check check-send-proxy send-proxy</div>
<div> stick on int(1)</div>
<div> stick-table type integer size 1 nopurge expire 60s</div>
<div> tcp-check expect rstring ^220</div>
<div> tcp-check send QUIT\r\n</div>
timeout server 600s<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The purging issue was a result of a round-robin'd connection to postfix where every connection alternated the server that email was coming through. Postfix stored the mail locally to dovecot and dovecot replicated it across instances. In theory this should've
worked, (and for multiple months it did!)<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Marc Roos <M.Roos@f1-outsourcing.eu><br>
<b>Sent:</b> Wednesday, August 26, 2020 11:58 PM<br>
<b>To:</b> Zelic Bojan <bojan.zelic@kudelskisecurity.com>; dovecot <dovecot@dovecot.org><br>
<b>Subject:</b> RE: Expuning & Purging doesn't fully remove emails?</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText"><br>
But is that not adviced in the manual[1]? I am also in the process of <br>
setting up dovecot behind a haproxy. Still testing with this. I was <br>
wondering why one would choose haproxy and not eg dovecot proxy (with <br>
director)<br>
<br>
<br>
<br>
[1]<br>
<a href="https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki2.dovecot.org%2FHAProxy&data=02%7C01%7Cbojan.zelic%40kudelskisecurity.com%7Ca3c2c2eaabe444ef253008d84a5693ff%7Ce573309544254f08b6ba487b9a46a425%7C0%7C1%7C637341083000478508&sdata=zKRtkWbI6VzCpUJmdLudYA9M9noilPr%2Bsx48DY1x1mE%3D&reserved=0">https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki2.dovecot.org%2FHAProxy&data=02%7C01%7Cbojan.zelic%40kudelskisecurity.com%7Ca3c2c2eaabe444ef253008d84a5693ff%7Ce573309544254f08b6ba487b9a46a425%7C0%7C1%7C637341083000478508&sdata=zKRtkWbI6VzCpUJmdLudYA9M9noilPr%2Bsx48DY1x1mE%3D&reserved=0</a><br>
<br>
<br>
<br>
-----Original Message-----<br>
From: Zelic Bojan [<a href="mailto:bojan.zelic@kudelskisecurity.com">mailto:bojan.zelic@kudelskisecurity.com</a>]
<br>
Sent: woensdag 26 augustus 2020 23:39<br>
To: dovecot@dovecot.org<br>
Subject: Re: Expuning & Purging doesn't fully remove emails?<br>
<br>
I managed to figure this out.<br>
<br>
<br>
Just wanted to follow up if anyone in the future encounters the same <br>
issue.<br>
<br>
I am using HAProxy along with dovecot replication. When an email comes <br>
in... it is round robin'd to each of the 2 dovecot/postfix servers. I <br>
have postfix running on each server and it uses the local dovecot LMTP <br>
service for storing the mails. <br>
<br>
We sometimes get surges of emails(hundreds or thousands in a couple of <br>
seconds). Each alternative request hits a different mail server and then <br>
should be replicated. The replication works and we don't encounter <br>
missing or duplicated emails.<br>
<br>
However, when I purge expunged emails, somehow this does not fully <br>
remove them when it's setup in this fashion. <br>
<br>
I ran 2 tests:<br>
1) I send 1000 emails to the loadbalancer(round-robin), expunge on both <br>
servers & purge them on both servers, recreate the indexes... ~200 <br>
emails come back.<br>
2) I send 1000 emails to one of the mail servers, expunge & purge them, <br>
recreate the indexes... 0 emails come back.<br>
<br>
My fix is to remove the round robin load balancing and use sticky tables <br>
in HAproxy.<br>
<br>
<br>
<br>
________________________________<br>
<br>
From: Zelic Bojan<br>
Sent: Thursday, August 20, 2020 1:42 PM<br>
To: dovecot@dovecot.org <dovecot@dovecot.org><br>
Subject: Expuning & Purging doesn't fully remove emails? <br>
<br>
Hello, I'm facing an issue where deleted emails keep re-appearing after <br>
my mailbox index gets recreated. I'm running version 2.2.36 of dovecot, <br>
but I tested the same scenario under 2.3.10. I'm also using mdbox, <br>
autoexpunge, and using dovecot replication.<br>
<br>
I've had several instances now where some expunged emails show up again <br>
in a mailbox. I noticed this error:<br>
<br>
doveadm: Error: Corrupted dbox file <br>
/var/mail/virtual/mailbox@domain.com/mdbox/storage/m.3228 (around <br>
offset=1988744): msg header has bad magic value<br>
<br>
which caused the index to get rebuilt... however several times now, the <br>
indexes got rebuilt even though there doesnt seem like there was any <br>
error... so im not sure why that is. <br>
<br>
lmtp(13910): Warning: fscking index file <br>
/var/mail/virtual/mailbox@domain/mdbox/storage/dovecot.map.index<br>
lmtp(13910): Warning: fscking index file <br>
/var/mail/virtual/mailbox@domain/mdbox/storage/dovecot.map.index<br>
lmtp(13910): Warning: mdbox <br>
/var/mail/virtual/mailbox@domain/mdbox/storage: rebuilding indexes<br>
<br>
I'm not sure why these mails keep coming back though... or if there's <br>
anything that I can do to limit the number of emails that get restored.<br>
<br>
I want to make sure expunged & purged emails stay expunged & purged. If <br>
I run a purge and then force index recreation... why would expunged <br>
emails come back? Shouldn't I expect them all to be deleted & purged? <br>
Does expunge not expunge all emails? (In production, I'm running <br>
autoexpunge but this test below shows what happens when I attempt to <br>
expunge everything)<br>
<br>
doveadm search -u email@domain all | wc -l<br>
# output 22096<br>
<br>
doveadm expunge -u email@domain mailbox '*' all<br>
doveadm search -u email@domain all | wc -l<br>
# output: 0<br>
<br>
doveadm purge -u email@domain<br>
doveadm dump /var/mail/virtual/email@domain/mdbox/storage/ | grep -c <br>
'ref.*\b0\b' <br>
# output: 0<br>
<br>
doveadm force-resync -u email@domain Inbox<br>
# output:<br>
# doveadm(email@domain): Warning: fscking index file <br>
/var/mail/virtual/email@domain/mdbox/storage/dovecot.map.index<br>
# doveadm(email@domain): Warning: mdbox <br>
/var/mail/virtual/email@domain/mdbox/storage: rebuilding indexes<br>
# doveadm(email@domain): Warning: fscking index file <br>
/var/mail/virtual/email@domain/mdbox/storage/dovecot.map.index<br>
<br>
doveadm search -u email@domain all | wc -l<br>
# output: 843<br>
<br>
I would expect the output to be 0. Theoretically I deleted all emails <br>
and purged all emails. Nothing should be left on the disk? However... I <br>
can see there are still m.* files in mdbox/storage for the mailbox. <br>
<br>
Overall, I'm not sure why the index got recreated... but I'm trying to <br>
limit the impact of mailbox corruption so that deleted emails do not <br>
come back if the index is somehow recreated again. <br>
<br>
If I were to re-run expunge, purge, and force-resync a 2nd time... it <br>
does get emptied out, but I'm not looking to run force-resync <br>
intentially since it causes dataloss with mdbox, and re-running only <br>
expunge & purge doesn't seem to do anything. <br>
<br>
Bojan Zelic<br>
Sr. IT Infrastructure Engineer<br>
<br>
<br>
</div>
</span></font></div>
</body>
</html>