replicator: Panic: data stack: Out of memory when allocating 268435496 bytes

Sat Jan 7 20:03:09 UTC 2023

Hello,

try updating to Dovecot v2.3.20 // I had the same in earlier versions, 
but it doesn't happen any longer now.

Increasing the limit didn't help me at that time, even the 0 didn't help.

Am 07.01.23 um 11:17 schrieb Paul Kudla:
>
> ok a little bit more info ....
>
> My servers all run under FreeBSD 12.xx
>
> which was also the base for the apple operating system origionally.
>
> setting default_vsz_limit = 0 i seem to remember trying with no so 
> great results
>
> setting to zero can cause memory over runs (espically with 
> replication) etc i found that when i used the config i sent eariler 
> (vsz_limit is defaulted (not set )) everything worked
>
> I generally find that tweaking the memory alloted to the individual 
> services a more balanced approach but it does take a lot of trial and 
> error
>
> also note memory in the system is also a factor my mail servers have 
> 32G dedicated to them which is what the settings were based on that 
> seem to work pretty good at the moment.
>
> I am running without any setting thus the default
>
> I got this info from :
>
> https://doc.dovecot.org/configuration_manual/service_configuration/
>
> vsz_limit
>
> Limit the process’s address space (both RLIMIT_DATA and RLIMIT_AS if 
> available). When the space is reached, some memory allocations may 
> start failing with “Out of memory”, or the kernel may kill the process 
> with signal 9. This setting is mainly intended to prevent memory leaks 
> from eating up all of the memory, but there can be also legitimate 
> reasons why the process reaches this limit. For example a huge mailbox 
> may not be accessed if this limit is too low. The default value 
> (18446744073709551615=2^64-1) sets the limit to default_vsz_limit, 
> while 0 disables the limit entirely.
>
> There are 3 types of services that need to be optimized in different 
> ways:
>
>     Master services (e.g. auth, anvil, indexer, director, log):
>
>         Currently there isn’t any easy way to optimize these. If these 
> become a bottleneck, typically you need to run another Dovecot server. 
> In some cases it may be possible to create multiple master processes 
> and have each one be responsible for only specific users/processes, 
> although this may also require some extra development.
>
>     Services that do disk I/O or other blocking operations (e.g. imap, 
> pop3, lmtp):
>
>         These should have client_limit=1, because any blocking 
> operation will block all the other clients and cause unnecessary 
> delays and even timeouts. This means that process_limit specifies the 
> maximum number of available parallel connections.
>
>     Services that have no blocking operations (e.g. imap-login, 
> pop3-login):
>
>         For best performance (but a bit less safety), these should 
> have process_limit and process_min_avail set to the number of CPU 
> cores, so each CPU will be busy serving the process but without 
> unnecessary context switches. Then client_limit needs to be set high 
> enough to be able to serve all the needed connections (max 
> connections=process_limit * client_limit). service_count is commonly 
> set to unlimited (0) for these services. Otherwise when the 
> service_count is beginning to be reached, the total number of 
> available connections will shrink. With very bad luck that could mean 
> that all the processes are simply waiting for the existing connections 
> to die away before the process can die and a new one can be created. 
> Although this could be made less likely by setting process_limit 
> higher than process_min_avail, but that’s still not a guarantee since 
> each process could get a very long running connection and the 
> process_limit would be eventually reached.
>
>
>
>
>
> Happy Saturday !!!
> Thanks - paul
>
> Paul Kudla
>
>
> Scom.ca Internet Services <http://www.scom.ca>
> 004-1009 Byron Street South
> Whitby, Ontario - Canada
> L1N 4S3
>
> Toronto 416.642.7266
> Main 1.866.411.7266
> Fax 1.888.892.7266
> Email paul at scom.ca
>
> On 1/6/2023 5:20 PM, Gerben Wierda wrote:
>> How problematic is it to have
>>
>> default_vsz_limit = 0
>>
>> in dovecot.conf? macOS+MacPorts had this as a requirement even.
>>
>> Gerben
>>
>>> On 6 Jan 2023, at 16:49, Paul Kudla <paul at scom.ca 
>>> <mailto:paul at scom.ca>> wrote:
>>>
>>>
>>> i ran into this as well
>>>
>>> here is the full config for mine with replication
>>>
>>> # cat dovecot.conf
>>> # 2.3.14 (cee3cbc0d): /usr/local/etc/dovecot/dovecot.conf
>>> # OS: FreeBSD 12.1-RELEASE amd64
>>> # Hostname: mail18.scom.ca <http://mail18.scom.ca>
>>>
>>> auth_debug = no
>>> auth_debug_passwords = no
>>>
>>> default_process_limit = 16384
>>>
>>> mail_debug = no
>>>
>>> #lock_method = dotlock
>>> #mail_max_lock_timeout = 300s
>>>
>>> #mbox_read_locks = dotlock
>>> #mbox_write_locks = dotlock
>>>
>>> mmap_disable = yes
>>> dotlock_use_excl = no
>>> mail_fsync = always
>>> mail_nfs_storage = no
>>> mail_nfs_index = no
>>>
>>> auth_mechanisms = plain login
>>> auth_verbose = yes
>>> base_dir = /data/dovecot/run/
>>> debug_log_path = syslog
>>> disable_plaintext_auth = no
>>> dsync_features = empty-header-workaround
>>>
>>> info_log_path = syslog
>>> login_greeting = SCOM.CA Internet Services Inc. - Dovecot ready
>>> login_log_format_elements = user=<%u> method=%m rip=%r lip=%l 
>>> mpid=%e %c
>>>
>>>
>>> mail_location = maildir:~/
>>>
>>> mail_plugins = " virtual notify replication fts fts_lucene "
>>> mail_prefetch_count = 20
>>>
>>> protocols = imap pop3 lmtp sieve
>>>
>>>
>>> protocol lmtp {
>>>  mail_plugins = $mail_plugins sieve
>>>  postmaster_address =
>>> }
>>>
>>> service lmtp {
>>>  process_limit=1000
>>>  vsz_limit = 512m
>>>  client_limit=1
>>>   unix_listener /usr/home/postfix.local/private/dovecot-lmtp {
>>>     group = postfix
>>>     mode = 0600
>>>     user = postfix
>>>  }
>>> }
>>>
>>> protocol lda {
>>>  mail_plugins = $mail_plugins sieve
>>> }
>>>
>>> service lda {
>>>  process_limit=1000
>>>  vsz_limit = 512m
>>> }
>>>
>>> service imap {
>>>  process_limit=4096
>>>  vsz_limit = 2g
>>>  client_limit=1
>>> }
>>>
>>> service pop3 {
>>>  process_limit=1000
>>>  vsz_limit = 512m
>>>  client_limit=1
>>> }
>>>
>>> namespace inbox {
>>>  inbox = yes
>>>  location =
>>>  mailbox Drafts {
>>>    auto = subscribe
>>>    special_use = \Drafts
>>>  }
>>>  mailbox Sent {
>>>    auto = subscribe
>>>    special_use = \Sent
>>>  }
>>>  mailbox Trash {
>>>    auto = subscribe
>>>    special_use = \Trash
>>>  }
>>>  prefix =
>>>  separator = /
>>> }
>>>
>>> passdb {
>>>  args = /usr/local/etc/dovecot/dovecot-pgsql.conf
>>>  driver = sql
>>> }
>>>
>>> doveadm_port = 12345
>>> doveadm_password = secretxxxx
>>>
>>> service doveadm {
>>>  process_limit = 0
>>>  process_min_avail = 0
>>>  idle_kill = 0
>>>  client_limit = 1
>>>  user = vmail
>>>  inet_listener {
>>>    port = 12345
>>>  }
>>> }
>>>
>>> service config {
>>>  unix_listener config {
>>>    user = vmail
>>>    }
>>> }
>>>
>>> dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u
>>> #dsync_remote_cmd = doveadm sync -d -u%u
>>>
>>> replication_dsync_parameters = -d -N -l 300 -U
>>>
>>> plugin {
>>>  mail_log_events = delete undelete expunge copy mailbox_delete 
>>> mailbox_rename
>>>  mail_log_fields = uid, box, msgid, from, subject, size, vsize, flags
>>>  push_notification_driver = dlog
>>>
>>>  sieve = file:~/sieve;active=~/sieve/.dovecot.sieve
>>>  #sieve = ~/.dovecot.sieve
>>>  sieve_duplicate_default_period = 1h
>>>  sieve_duplicate_max_period = 1h
>>>  sieve_extensions = +duplicate +notify +imapflags +vacation-seconds
>>>  sieve_global_dir = /usr/local/etc/dovecot/sieve
>>>  sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve
>>>
>>>
>>>  mail_replica = tcp:10.221.0.19:12345
>>>  #mail_replica = remote:vmail at 10.221.0.19 <mailto:vmail at 10.221.0.19>
>>>  #replication_sync_timeout = 2
>>>
>>>  fts = lucene
>>>  fts_lucene = whitespace_chars=@.
>>>        fts_autoindex = yes
>>>        fts_languages = en
>>> }
>>>
>>> #sieve_extensions = vnd.dovecot.duplicate
>>>
>>> #sieve_plugins = vnd.dovecot.duplicate
>>>
>>> service anvil {
>>>  process_limit = 1
>>>  client_limit=5000
>>>  vsz_limit = 512m
>>>  unix_listener anvil {
>>>    group = vmail
>>>    mode = 0666
>>>  }
>>> }
>>>
>>> service indexer-worker {
>>>        vsz_limit = 2g
>>> }
>>>
>>>
>>>
>>> service auth {
>>>   process_limit = 1
>>>   client_limit=5000
>>>   vsz_limit = 1g
>>>
>>>   unix_listener auth-userdb {
>>>      mode = 0660
>>>      user = vmail
>>>      group = vmail
>>>   }
>>>   unix_listener /var/spool/postfix/private/auth {
>>>      mode = 0666
>>>   }
>>>
>>> }
>>>
>>> service stats {
>>>  process_limit = 1000
>>>  vsz_limit = 1g
>>>  unix_listener stats-reader {
>>>    group = vmail
>>>    mode = 0666
>>>  }
>>>  unix_listener stats-writer {
>>>    group = vmail
>>>    mode = 0666
>>>  }
>>> }
>>> userdb {
>>>  args = /usr/local/etc/dovecot/dovecot-pgsql.conf
>>>  driver = sql
>>>
>>> }
>>>
>>> protocol imap {
>>>  mail_max_userip_connections = 50
>>>  mail_plugins = $mail_plugins notify replication
>>> }
>>>
>>> protocol pop3 {
>>>  mail_max_userip_connections = 50
>>>  mail_plugins = $mail_plugins notify replication
>>> }
>>>
>>> protocol imaps {
>>>  mail_max_userip_connections = 25
>>>  mail_plugins = $mail_plugins notify replication
>>> }
>>>
>>> protocol pop3s {
>>>  mail_max_userip_connections = 25
>>>  mail_plugins = $mail_plugins notify replication
>>> }
>>>
>>>
>>> service managesieve-login {
>>>  process_limit = 1000
>>>  vsz_limit = 1g
>>>  inet_listener sieve {
>>>    port = 4190
>>>  }
>>> }
>>>
>>> verbose_proctitle = yes
>>>
>>> replication_max_conns = 100
>>>
>>> replication_full_sync_interval = 1d
>>>
>>> service replicator {
>>>  client_limit = 0
>>>  drop_priv_before_exec = no
>>>  idle_kill = 4294967295s
>>>  process_limit = 1
>>>  process_min_avail = 0
>>>  service_count = 0
>>>  vsz_limit = 8g
>>>        unix_listener replicator-doveadm {
>>>    mode = 0600
>>>    user = vmail
>>>  }
>>>  vsz_limit = 8192M
>>> }
>>>
>>>
>>> service aggregator {
>>>  process_limit = 1000
>>>  #vsz_limit = 1g
>>>  fifo_listener replication-notify-fifo {
>>>    user = vmail
>>>    group = vmail
>>>    mode = 0666
>>>  }
>>>
>>> }
>>>
>>> service pop3-login {
>>>  process_limit = 1000
>>>  client_limit = 100
>>>  vsz_limit = 512m
>>> }
>>>
>>>
>>> service imap-urlauth-login {
>>>  process_limit = 1000
>>>  client_limit = 1000
>>>  vsz_limit = 1g
>>> }
>>>
>>>
>>> service imap-login {
>>>  process_limit=1000
>>>  client_limit = 1000
>>>  vsz_limit = 1g
>>> }
>>>
>>>
>>> protocol sieve {
>>>  managesieve_implementation_string = Dovecot Pigeonhole
>>>  managesieve_max_line_length = 65536
>>> }
>>>
>>>
>>>
>>>
>>> #Addition ssl config
>>> !include sni.conf
>>>
>>> with sni cert support (examples)
>>>
>>> # cat sni.conf
>>> #sni.conf
>>> ssl = yes
>>> verbose_ssl = yes
>>> ssl_dh =</usr/local/etc/dovecot/dh-4096.pem
>>> ssl_prefer_server_ciphers = yes
>>> #ssl_min_protocol = TLSv1.2
>>>
>>> #Default *.scom.ca <http://scom.ca>
>>> ssl_key =</usr/local/etc/dovecot/scom.pem
>>> ssl_cert =</usr/local/etc/dovecot/scom.pem
>>> ssl_ca =</usr/local/etc/dovecot/scom.pem
>>>
>>> local_name .scom.ca <http://scom.ca> {
>>> ssl_key =</usr/local/etc/dovecot/scom.pem
>>> ssl_cert =</usr/local/etc/dovecot/scom.pem
>>> ssl_ca =</usr/local/etc/dovecot/scom.pem
>>>
>>> }
>>>
>>> local_name mail.clancyca.com <http://mail.clancyca.com> {
>>>  ssl_key =</usr/local/etc/dovecot/cert/mail.clancyca.com 
>>> <http://mail.clancyca.com>
>>>  ssl_cert =</usr/local/etc/dovecot/cert/mail.clancyca.com 
>>> <http://mail.clancyca.com>
>>>  ssl_ca =</usr/local/etc/dovecot/cert/mail.clancyca.com 
>>> <http://mail.clancyca.com>
>>> }
>>>
>>> local_name mail.paulkudla.net <http://mail.paulkudla.net> {
>>>  ssl_key =</usr/local/etc/dovecot/cert/mail.paulkudla.net 
>>> <http://mail.paulkudla.net>
>>>  ssl_cert =</usr/local/etc/dovecot/cert/mail.paulkudla.net 
>>> <http://mail.paulkudla.net>
>>>  ssl_ca =</usr/local/etc/dovecot/cert/mail.paulkudla.net 
>>> <http://mail.paulkudla.net>
>>> }
>>>
>>> local_name mail.ekst.ca <http://mail.ekst.ca> {
>>>  ssl_key =</usr/local/etc/dovecot/cert/mail.ekst.ca 
>>> <http://mail.ekst.ca>
>>>  ssl_cert =</usr/local/etc/dovecot/cert/mail.ekst.ca 
>>> <http://mail.ekst.ca>
>>>  ssl_ca =</usr/local/etc/dovecot/cert/mail.ekst.ca 
>>> <http://mail.ekst.ca>
>>> }
>>>
>>> local_name mail.hamletdevelopments.ca 
>>> <http://mail.hamletdevelopments.ca> {
>>>  ssl_key =</usr/local/etc/dovecot/cert/mail.hamletdevelopments.ca 
>>> <http://mail.hamletdevelopments.ca>
>>>  ssl_cert =</usr/local/etc/dovecot/cert/mail.hamletdevelopments.ca 
>>> <http://mail.hamletdevelopments.ca>
>>>  ssl_ca =</usr/local/etc/dovecot/cert/mail.hamletdevelopments.ca 
>>> <http://mail.hamletdevelopments.ca>
>>> }
>>>
>>> pg sql support supporting replication
>>>
>>> # cat dovecot-pgsql.conf
>>> driver = pgsql
>>> connect = host=localhost port=5433 dbname=scom_billing user=pgsql 
>>> password=
>>> default_pass_scheme = PLAIN
>>>
>>> password_query = SELECT username as user, password FROM email_users 
>>> WHERE username = '%u' and password <> 'alias' and status = True and 
>>> destination = '%u'
>>>
>>> user_query = SELECT home, uid, gid FROM email_users WHERE username = 
>>> '%u' and password <> 'alias' and status = True and destination = '%u'
>>>
>>> #iterate_query = SELECT user, password FROM email_users WHERE 
>>> username = '%u' and password <> 'alias' and status = True and 
>>> destination = '%u'
>>>
>>> iterate_query = SELECT "username" as user, domain FROM email_users 
>>> WHERE status = True and alias_flag = False
>>>
>>>
>>>
>>>
>>>
>>> Happy Friday !!!
>>> Thanks - paul
>>>
>>> Paul Kudla
>>>
>>>
>>> Scom.ca <http://Scom.ca> Internet Services <http://www.scom.ca 
>>> <http://www.scom.ca>>
>>> 004-1009 Byron Street South
>>> Whitby, Ontario - Canada
>>> L1N 4S3
>>>
>>> Toronto 416.642.7266
>>> Main 1.866.411.7266
>>> Fax 1.888.892.7266
>>> Email paul at scom.ca <mailto:paul at scom.ca>
>>>
>>> On 1/6/2023 5:32 AM, Gerben Wierda wrote:
>>>>> On 6 Jan 2023, at 08:53, Aki Tuomi <aki.tuomi at open-xchange.com 
>>>>> <mailto:aki.tuomi at open-xchange.com>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On January 6, 2023 3:56:39 AM GMT+02:00, Gerben Wierda 
>>>>> <gerben.wierda at rna.nl <mailto:gerben.wierda at rna.nl>> wrote:
>>>>>> One step further in my quest to create a replacement mail server.
>>>>>>
>>>>>> I now have my old mail server (2.3.19.1, macOS + MacPorts) and my 
>>>>>> new (2.3.20, Alpine Linux, Docker, apk package). When I turn on 
>>>>>> replication it works, but, after a while I see:
>>>>>>
>>>>>> Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when 
>>>>>> allocating 268435496 bytes
>>>>>> Jan 06 00:50:32 replicator: Fatal: master: service(replicator): 
>>>>>> child 133 killed with signal 6 (core dumped)
>>>>>> Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: 
>>>>>> replication(sysbh): Sync failure:
>>>>>> Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: 
>>>>>> replication(sysbh): Remote sent invalid input: -
>>>>>>
>>>>>> I've removed synchronous operation for now (found a message on 
>>>>>> the net suggesting that) but is this known and what does it mean?
>>>>>>
>>>>>> Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda 
>>>>>> <https://www.linkedin.com/in/gerbenwierda>>)
>>>>>> R&A IT Strategy <https://ea.rna.nl/ <https://ea.rna.nl/>> (main 
>>>>>> site)
>>>>>> Book: Chess and the Art of Enterprise Architecture 
>>>>>> <https://ea.rna.nl/the-book/ <https://ea.rna.nl/the-book/>>
>>>>>> Book: Mastering ArchiMate 
>>>>>> <https://ea.rna.nl/the-book-edition-iii/ 
>>>>>> <https://ea.rna.nl/the-book-edition-iii/>>
>>>>>>
>>>>>
>>>>> Dovecot default memory limit is 256M. You should probably set
>>>>>
>>>>> service replicator {
>>>>>  vsz_limit = 2G
>>>>> }
>>>>>
>>>>> because replicator might have to use more memory, especially for 
>>>>> larger indexes.
>>>>>
>>>>> Aki
>>>> That is a good tip as well.
>>>> I had followed this bit of experience from someone else: 
>>>> https://marc.info/?l=dovecot&m=164438199727640 
>>>> <https://marc.info/?l=dovecot&m=164438199727640>, haven't seen any 
>>>> err message since. But that might be because they are in sync now 
>>>> and both sides are aware. Can I trigger full replication again so I 
>>>> can test?
>>>> Gerben
>>>> -- 
>>>> This message has been scanned for viruses and
>>>> dangerous content by *MailScanner* <http://www.mailscanner.info/>, 
>>>> and is
>>>> believed to be clean.
>>
>>
>> -- 
>> This message has been scanned for viruses and
>> dangerous content by *MailScanner* <http://www.mailscanner.info/>, 
>> and is
>> believed to be clean.