Out of memory on lmtp vsz_limit

[Terence Lau]

Looks like its working. Haven't had any recurrence of the issue since scheduling a move operation daily.

Thanks for your help.

Cheers.

From: Aki Tuomi [mailto:aki.tuomi at dovecot.fi]
Sent: Friday, 16 February 2018 3:17 PM
To: Terence Lau <terence.lau at royhill.com.au>; dovecot at dovecot.org
Subject: Re: Out of memory on lmtp vsz_limit


How about you try moving the mail into another folder on daily basis, this way the INBOX would stay nice and empty.

doveadm move -u archive at company.com<mailto:archive at company.com> Archive MAILBOX INBOX SENTBEFORE todays-date
Aki
On 16.02.2018 06:19, Terence Lau wrote:
Bump.

Any advice would be most appreciated.

Thanks.

From: Terence Lau
Sent: Wednesday, 24 January 2018 9:59 AM
To: 'dovecot at dovecot.org<mailto:dovecot at dovecot.org>' <dovecot at dovecot.org><mailto:dovecot at dovecot.org>
Subject: Out of memory on lmtp vsz_limit

Hi,

We've been getting these types or errors for quite a while now ...

Fatal: master: service(lmtp): child 63477 returned error 83 (Out of memory (service lmtp { vsz_limit=4096 MB }, you may need to increase it)

... and these errors have been decreasing in occurrence as we increased the default_vsz_limit.  Which is good but I would like to get some advice on how I could possibly eliminate the errors.

We have an internal smtp server (postfix 3.1.0) that has the config "always_bcc=archive at company.com<mailto:always_bcc=archive at company.com>" over lmtp.  This mailbox is on a separate dovecot server with the following config (please let me know if the full config is required):

# 2.2.22 (fe789d2): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.13 (7b14904)
# OS: Linux 4.4.0-109-generic x86_64 Ubuntu 16.04.1 LTS ext4
default_vsz_limit = 4 G
mail_location = maildir:/home/vmail/%d/%n
protocols = " imap lmtp pop3"
service lmtp {
  inet_listener lmtp {
    port = 24
  }
}
userdb {
  args = username_format=%u /etc/dovecot/users
  default_fields = uid=vmail gid=vmail home=/home/vmail/%d/%n
  driver = passwd-file
}
protocol lmtp {
  mail_plugins =
}

Since we discovered the errors, we've been increasing the default_vsz_limit to 1G, then 2G and now 4G (Server has 6GB of memory).  These errors occur whenever a large number of emails get sent around the same time to our smtp server.  This causes the dovecot server to start crunching CPU and Memory.  Load average goes through the roof and takes some time to come back down as the smtp queue clears itself.

This mailbox is obviously very large but we have a script that runs daily to delete any emails older than a month:

find /home/vmail/company.com/archive/new/ -type f -mtime +30 -exec rm {} \;
find /home/vmail/company.com/archive/cur/ -type f -mtime +30 -exec rm {} \;

Still, the mailbox has on average of just under 300,000 emails.  No one accesses this mailbox with an email client, not until we need to dig something up.  And this has only happen once.  So the emails pretty much never get read/process by a user.

Now that we've increased the default_vsz_limit to 4G, the occurrence of these errors has reduced considerably.  But they still happen occasionally.  Short of increasing the memory further, are there any other options I have?

Thanks.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20180228/f88f27f7/attachment-0001.html>