replicator: Panic: data stack: Out of memory when allocating 268435496 bytes
One step further in my quest to create a replacement mail server.
I now have my old mail server (2.3.19.1, macOS + MacPorts) and my new (2.3.20, Alpine Linux, Docker, apk package). When I turn on replication it works, but, after a while I see:
Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when allocating 268435496 bytes Jan 06 00:50:32 replicator: Fatal: master: service(replicator): child 133 killed with signal 6 (core dumped) Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Sync failure: Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Remote sent invalid input: -
I've removed synchronous operation for now (found a message on the net suggesting that) but is this known and what does it mean?
Gerben Wierda (LinkedIn https://www.linkedin.com/in/gerbenwierda) R&A IT Strategy https://ea.rna.nl/ (main site) Book: Chess and the Art of Enterprise Architecture https://ea.rna.nl/the-book/ Book: Mastering ArchiMate https://ea.rna.nl/the-book-edition-iii/
On January 6, 2023 3:56:39 AM GMT+02:00, Gerben Wierda gerben.wierda@rna.nl wrote:
One step further in my quest to create a replacement mail server.
I now have my old mail server (2.3.19.1, macOS + MacPorts) and my new (2.3.20, Alpine Linux, Docker, apk package). When I turn on replication it works, but, after a while I see:
Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when allocating 268435496 bytes Jan 06 00:50:32 replicator: Fatal: master: service(replicator): child 133 killed with signal 6 (core dumped) Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Sync failure: Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Remote sent invalid input: -
I've removed synchronous operation for now (found a message on the net suggesting that) but is this known and what does it mean?
Gerben Wierda (LinkedIn https://www.linkedin.com/in/gerbenwierda) R&A IT Strategy https://ea.rna.nl/ (main site) Book: Chess and the Art of Enterprise Architecture https://ea.rna.nl/the-book/ Book: Mastering ArchiMate https://ea.rna.nl/the-book-edition-iii/
Dovecot default memory limit is 256M. You should probably set
service replicator { vsz_limit = 2G }
because replicator might have to use more memory, especially for larger indexes.
Aki
On January 6, 2023 3:56:39 AM GMT+02:00, Gerben Wierda gerben.wierda@rna.nl wrote:
Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when allocating 268435496 bytes Jan 06 00:50:32 replicator: Fatal: master: ... service replicator { vsz_limit = 2G }
because replicator might have to use more memory, especially for larger indexes.
Aki That's probably as good a short-term fix as any, but a longer term fix will
On Thursday, January 5, 2023 10:53:13 PM AKST Aki Tuomi wrote: probably require effectively "going on a diet," losing weight, cracking down on memory leaks, matching up every malloc() and free() and getting leaner and meaner with the memory allocation and Big-O time & space complexity of algorithms.
On January 6, 2023 12:21:45 PM GMT+02:00, justina colmena ~biz justina@colmena.biz wrote:
On January 6, 2023 3:56:39 AM GMT+02:00, Gerben Wierda gerben.wierda@rna.nl wrote:
Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when allocating 268435496 bytes Jan 06 00:50:32 replicator: Fatal: master: ... service replicator { vsz_limit = 2G }
because replicator might have to use more memory, especially for larger indexes.
Aki That's probably as good a short-term fix as any, but a longer term fix will
On Thursday, January 5, 2023 10:53:13 PM AKST Aki Tuomi wrote: probably require effectively "going on a diet," losing weight, cracking down on memory leaks, matching up every malloc() and free() and getting leaner and meaner with the memory allocation and Big-O time & space complexity of algorithms.
mmaps are counted against vsz_limit so if you got index+cache over 256M you will run against the limit.
Aki
On 6 Jan 2023, at 08:53, Aki Tuomi aki.tuomi@open-xchange.com wrote:
On January 6, 2023 3:56:39 AM GMT+02:00, Gerben Wierda gerben.wierda@rna.nl wrote:
One step further in my quest to create a replacement mail server.
I now have my old mail server (2.3.19.1, macOS + MacPorts) and my new (2.3.20, Alpine Linux, Docker, apk package). When I turn on replication it works, but, after a while I see:
Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when allocating 268435496 bytes Jan 06 00:50:32 replicator: Fatal: master: service(replicator): child 133 killed with signal 6 (core dumped) Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Sync failure: Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Remote sent invalid input: -
I've removed synchronous operation for now (found a message on the net suggesting that) but is this known and what does it mean?
Gerben Wierda (LinkedIn https://www.linkedin.com/in/gerbenwierda) R&A IT Strategy https://ea.rna.nl/ (main site) Book: Chess and the Art of Enterprise Architecture https://ea.rna.nl/the-book/ Book: Mastering ArchiMate https://ea.rna.nl/the-book-edition-iii/
Dovecot default memory limit is 256M. You should probably set
service replicator { vsz_limit = 2G }
because replicator might have to use more memory, especially for larger indexes.
Aki
That is a good tip as well.
I had followed this bit of experience from someone else: https://marc.info/?l=dovecot&m=164438199727640 https://marc.info/?l=dovecot&m=164438199727640, haven't seen any err message since. But that might be because they are in sync now and both sides are aware. Can I trigger full replication again so I can test?
Gerben
i ran into this as well
here is the full config for mine with replication
# cat dovecot.conf # 2.3.14 (cee3cbc0d): /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 12.1-RELEASE amd64 # Hostname: mail18.scom.ca
auth_debug = no auth_debug_passwords = no
default_process_limit = 16384
mail_debug = no
#lock_method = dotlock #mail_max_lock_timeout = 300s
#mbox_read_locks = dotlock #mbox_write_locks = dotlock
mmap_disable = yes dotlock_use_excl = no mail_fsync = always mail_nfs_storage = no mail_nfs_index = no
auth_mechanisms = plain login auth_verbose = yes base_dir = /data/dovecot/run/ debug_log_path = syslog disable_plaintext_auth = no dsync_features = empty-header-workaround
info_log_path = syslog login_greeting = SCOM.CA Internet Services Inc. - Dovecot ready login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
mail_location = maildir:~/
mail_plugins = " virtual notify replication fts fts_lucene " mail_prefetch_count = 20
protocols = imap pop3 lmtp sieve
protocol lmtp { mail_plugins = $mail_plugins sieve postmaster_address = }
service lmtp { process_limit=1000 vsz_limit = 512m client_limit=1 unix_listener /usr/home/postfix.local/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } }
protocol lda { mail_plugins = $mail_plugins sieve }
service lda { process_limit=1000 vsz_limit = 512m }
service imap { process_limit=4096 vsz_limit = 2g client_limit=1 }
service pop3 { process_limit=1000 vsz_limit = 512m client_limit=1 }
namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / }
passdb { args = /usr/local/etc/dovecot/dovecot-pgsql.conf driver = sql }
doveadm_port = 12345 doveadm_password = secretxxxx
service doveadm { process_limit = 0 process_min_avail = 0 idle_kill = 0 client_limit = 1 user = vmail inet_listener { port = 12345 } }
service config { unix_listener config { user = vmail } }
dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u #dsync_remote_cmd = doveadm sync -d -u%u
replication_dsync_parameters = -d -N -l 300 -U
plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid, box, msgid, from, subject, size, vsize, flags push_notification_driver = dlog
sieve = file:~/sieve;active=~/sieve/.dovecot.sieve #sieve = ~/.dovecot.sieve sieve_duplicate_default_period = 1h sieve_duplicate_max_period = 1h sieve_extensions = +duplicate +notify +imapflags +vacation-seconds sieve_global_dir = /usr/local/etc/dovecot/sieve sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve
mail_replica = tcp:10.221.0.19:12345 #mail_replica = remote:vmail@10.221.0.19 #replication_sync_timeout = 2
fts = lucene fts_lucene = whitespace_chars=@. fts_autoindex = yes fts_languages = en }
#sieve_extensions = vnd.dovecot.duplicate
#sieve_plugins = vnd.dovecot.duplicate
service anvil { process_limit = 1 client_limit=5000 vsz_limit = 512m unix_listener anvil { group = vmail mode = 0666 } }
service indexer-worker { vsz_limit = 2g }
service auth { process_limit = 1 client_limit=5000 vsz_limit = 1g
unix_listener auth-userdb {
mode = 0660
user = vmail
group = vmail
}
unix_listener /var/spool/postfix/private/auth {
mode = 0666
}
}
service stats { process_limit = 1000 vsz_limit = 1g unix_listener stats-reader { group = vmail mode = 0666 } unix_listener stats-writer { group = vmail mode = 0666 } } userdb { args = /usr/local/etc/dovecot/dovecot-pgsql.conf driver = sql
}
protocol imap { mail_max_userip_connections = 50 mail_plugins = $mail_plugins notify replication }
protocol pop3 { mail_max_userip_connections = 50 mail_plugins = $mail_plugins notify replication }
protocol imaps { mail_max_userip_connections = 25 mail_plugins = $mail_plugins notify replication }
protocol pop3s { mail_max_userip_connections = 25 mail_plugins = $mail_plugins notify replication }
service managesieve-login { process_limit = 1000 vsz_limit = 1g inet_listener sieve { port = 4190 } }
verbose_proctitle = yes
replication_max_conns = 100
replication_full_sync_interval = 1d
service replicator { client_limit = 0 drop_priv_before_exec = no idle_kill = 4294967295s process_limit = 1 process_min_avail = 0 service_count = 0 vsz_limit = 8g unix_listener replicator-doveadm { mode = 0600 user = vmail } vsz_limit = 8192M }
service aggregator { process_limit = 1000 #vsz_limit = 1g fifo_listener replication-notify-fifo { user = vmail group = vmail mode = 0666 }
}
service pop3-login { process_limit = 1000 client_limit = 100 vsz_limit = 512m }
service imap-urlauth-login { process_limit = 1000 client_limit = 1000 vsz_limit = 1g }
service imap-login { process_limit=1000 client_limit = 1000 vsz_limit = 1g }
protocol sieve { managesieve_implementation_string = Dovecot Pigeonhole managesieve_max_line_length = 65536 }
#Addition ssl config !include sni.conf
with sni cert support (examples)
# cat sni.conf #sni.conf ssl = yes verbose_ssl = yes ssl_dh =
#Default *.scom.ca ssl_key =
local_name .scom.ca { ssl_key =
}
local_name mail.clancyca.com { ssl_key =
local_name mail.paulkudla.net { ssl_key =
local_name mail.ekst.ca { ssl_key =
local_name mail.hamletdevelopments.ca { ssl_key =
pg sql support supporting replication
# cat dovecot-pgsql.conf driver = pgsql connect = host=localhost port=5433 dbname=scom_billing user=pgsql password= default_pass_scheme = PLAIN
password_query = SELECT username as user, password FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
user_query = SELECT home, uid, gid FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
#iterate_query = SELECT user, password FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
iterate_query = SELECT "username" as user, domain FROM email_users WHERE status = True and alias_flag = False
Happy Friday !!! Thanks - paul
Paul Kudla
Scom.ca Internet Services http://www.scom.ca 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3
Toronto 416.642.7266 Main 1.866.411.7266 Fax 1.888.892.7266 Email paul@scom.ca
On 1/6/2023 5:32 AM, Gerben Wierda wrote:
On 6 Jan 2023, at 08:53, Aki Tuomi
mailto:aki.tuomi@open-xchange.com> wrote: On January 6, 2023 3:56:39 AM GMT+02:00, Gerben Wierda
mailto:gerben.wierda@rna.nl> wrote: One step further in my quest to create a replacement mail server.
I now have my old mail server (2.3.19.1, macOS + MacPorts) and my new (2.3.20, Alpine Linux, Docker, apk package). When I turn on replication it works, but, after a while I see:
Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when allocating 268435496 bytes Jan 06 00:50:32 replicator: Fatal: master: service(replicator): child 133 killed with signal 6 (core dumped) Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Sync failure: Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Remote sent invalid input: -
I've removed synchronous operation for now (found a message on the net suggesting that) but is this known and what does it mean?
Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda https://www.linkedin.com/in/gerbenwierda>) R&A IT Strategy <https://ea.rna.nl/ https://ea.rna.nl/> (main site) Book: Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/ https://ea.rna.nl/the-book/> Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/ https://ea.rna.nl/the-book-edition-iii/>
Dovecot default memory limit is 256M. You should probably set
service replicator { vsz_limit = 2G }
because replicator might have to use more memory, especially for larger indexes.
Aki
That is a good tip as well.
I had followed this bit of experience from someone else: https://marc.info/?l=dovecot&m=164438199727640 https://marc.info/?l=dovecot&m=164438199727640, haven't seen any err message since. But that might be because they are in sync now and both sides are aware. Can I trigger full replication again so I can test?
Gerben
-- This message has been scanned for viruses and dangerous content by *MailScanner* http://www.mailscanner.info/, and is believed to be clean.
How problematic is it to have
default_vsz_limit = 0
in dovecot.conf? macOS+MacPorts had this as a requirement even.
Gerben
On 6 Jan 2023, at 16:49, Paul Kudla paul@scom.ca wrote:
i ran into this as well
here is the full config for mine with replication
# cat dovecot.conf # 2.3.14 (cee3cbc0d): /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 12.1-RELEASE amd64 # Hostname: mail18.scom.ca
auth_debug = no auth_debug_passwords = no
default_process_limit = 16384
mail_debug = no
#lock_method = dotlock #mail_max_lock_timeout = 300s
#mbox_read_locks = dotlock #mbox_write_locks = dotlock
mmap_disable = yes dotlock_use_excl = no mail_fsync = always mail_nfs_storage = no mail_nfs_index = no
auth_mechanisms = plain login auth_verbose = yes base_dir = /data/dovecot/run/ debug_log_path = syslog disable_plaintext_auth = no dsync_features = empty-header-workaround
info_log_path = syslog login_greeting = SCOM.CA Internet Services Inc. - Dovecot ready login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
mail_location = maildir:~/
mail_plugins = " virtual notify replication fts fts_lucene " mail_prefetch_count = 20
protocols = imap pop3 lmtp sieve
protocol lmtp { mail_plugins = $mail_plugins sieve postmaster_address = }
service lmtp { process_limit=1000 vsz_limit = 512m client_limit=1 unix_listener /usr/home/postfix.local/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } }
protocol lda { mail_plugins = $mail_plugins sieve }
service lda { process_limit=1000 vsz_limit = 512m }
service imap { process_limit=4096 vsz_limit = 2g client_limit=1 }
service pop3 { process_limit=1000 vsz_limit = 512m client_limit=1 }
namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / }
passdb { args = /usr/local/etc/dovecot/dovecot-pgsql.conf driver = sql }
doveadm_port = 12345 doveadm_password = secretxxxx
service doveadm { process_limit = 0 process_min_avail = 0 idle_kill = 0 client_limit = 1 user = vmail inet_listener { port = 12345 } }
service config { unix_listener config { user = vmail } }
dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u #dsync_remote_cmd = doveadm sync -d -u%u
replication_dsync_parameters = -d -N -l 300 -U
plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid, box, msgid, from, subject, size, vsize, flags push_notification_driver = dlog
sieve = file:~/sieve;active=~/sieve/.dovecot.sieve #sieve = ~/.dovecot.sieve sieve_duplicate_default_period = 1h sieve_duplicate_max_period = 1h sieve_extensions = +duplicate +notify +imapflags +vacation-seconds sieve_global_dir = /usr/local/etc/dovecot/sieve sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve
mail_replica = tcp:10.221.0.19:12345 #mail_replica = remote:vmail@10.221.0.19 #replication_sync_timeout = 2
fts = lucene fts_lucene = whitespace_chars=@. fts_autoindex = yes fts_languages = en }
#sieve_extensions = vnd.dovecot.duplicate
#sieve_plugins = vnd.dovecot.duplicate
service anvil { process_limit = 1 client_limit=5000 vsz_limit = 512m unix_listener anvil { group = vmail mode = 0666 } }
service indexer-worker { vsz_limit = 2g }
service auth { process_limit = 1 client_limit=5000 vsz_limit = 1g
unix_listener auth-userdb { mode = 0660 user = vmail group = vmail } unix_listener /var/spool/postfix/private/auth { mode = 0666 }
}
service stats { process_limit = 1000 vsz_limit = 1g unix_listener stats-reader { group = vmail mode = 0666 } unix_listener stats-writer { group = vmail mode = 0666 } } userdb { args = /usr/local/etc/dovecot/dovecot-pgsql.conf driver = sql
}
protocol imap { mail_max_userip_connections = 50 mail_plugins = $mail_plugins notify replication }
protocol pop3 { mail_max_userip_connections = 50 mail_plugins = $mail_plugins notify replication }
protocol imaps { mail_max_userip_connections = 25 mail_plugins = $mail_plugins notify replication }
protocol pop3s { mail_max_userip_connections = 25 mail_plugins = $mail_plugins notify replication }
service managesieve-login { process_limit = 1000 vsz_limit = 1g inet_listener sieve { port = 4190 } }
verbose_proctitle = yes
replication_max_conns = 100
replication_full_sync_interval = 1d
service replicator { client_limit = 0 drop_priv_before_exec = no idle_kill = 4294967295s process_limit = 1 process_min_avail = 0 service_count = 0 vsz_limit = 8g unix_listener replicator-doveadm { mode = 0600 user = vmail } vsz_limit = 8192M }
service aggregator { process_limit = 1000 #vsz_limit = 1g fifo_listener replication-notify-fifo { user = vmail group = vmail mode = 0666 }
}
service pop3-login { process_limit = 1000 client_limit = 100 vsz_limit = 512m }
service imap-urlauth-login { process_limit = 1000 client_limit = 1000 vsz_limit = 1g }
service imap-login { process_limit=1000 client_limit = 1000 vsz_limit = 1g }
protocol sieve { managesieve_implementation_string = Dovecot Pigeonhole managesieve_max_line_length = 65536 }
#Addition ssl config !include sni.conf
with sni cert support (examples)
# cat sni.conf #sni.conf ssl = yes verbose_ssl = yes ssl_dh =
#Default *.scom.ca ssl_key =
local_name .scom.ca { ssl_key =
}
local_name mail.clancyca.com { ssl_key =
local_name mail.paulkudla.net { ssl_key =
local_name mail.ekst.ca { ssl_key =
local_name mail.hamletdevelopments.ca { ssl_key =
pg sql support supporting replication
# cat dovecot-pgsql.conf driver = pgsql connect = host=localhost port=5433 dbname=scom_billing user=pgsql password= default_pass_scheme = PLAIN
password_query = SELECT username as user, password FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
user_query = SELECT home, uid, gid FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
#iterate_query = SELECT user, password FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
iterate_query = SELECT "username" as user, domain FROM email_users WHERE status = True and alias_flag = False
Happy Friday !!! Thanks - paul
Paul Kudla
Scom.ca Internet Services http://www.scom.ca 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3
Toronto 416.642.7266 Main 1.866.411.7266 Fax 1.888.892.7266 Email paul@scom.ca
On 1/6/2023 5:32 AM, Gerben Wierda wrote:
On 6 Jan 2023, at 08:53, Aki Tuomi
mailto:aki.tuomi@open-xchange.com> wrote: On January 6, 2023 3:56:39 AM GMT+02:00, Gerben Wierda
mailto:gerben.wierda@rna.nl> wrote: One step further in my quest to create a replacement mail server.
I now have my old mail server (2.3.19.1, macOS + MacPorts) and my new (2.3.20, Alpine Linux, Docker, apk package). When I turn on replication it works, but, after a while I see:
Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when allocating 268435496 bytes Jan 06 00:50:32 replicator: Fatal: master: service(replicator): child 133 killed with signal 6 (core dumped) Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Sync failure: Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Remote sent invalid input: -
I've removed synchronous operation for now (found a message on the net suggesting that) but is this known and what does it mean?
Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda https://www.linkedin.com/in/gerbenwierda>) R&A IT Strategy <https://ea.rna.nl/ https://ea.rna.nl/> (main site) Book: Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/ https://ea.rna.nl/the-book/> Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/ https://ea.rna.nl/the-book-edition-iii/>
Dovecot default memory limit is 256M. You should probably set
service replicator { vsz_limit = 2G }
because replicator might have to use more memory, especially for larger indexes.
Aki That is a good tip as well. I had followed this bit of experience from someone else: https://marc.info/?l=dovecot&m=164438199727640 https://marc.info/?l=dovecot&m=164438199727640, haven't seen any err message since. But that might be because they are in sync now and both sides are aware. Can I trigger full replication again so I can test? Gerben
This message has been scanned for viruses and dangerous content by *MailScanner* http://www.mailscanner.info/, and is believed to be clean.
ok a little bit more info ....
My servers all run under FreeBSD 12.xx
which was also the base for the apple operating system origionally.
setting default_vsz_limit = 0 i seem to remember trying with no so great results
setting to zero can cause memory over runs (espically with replication) etc i found that when i used the config i sent eariler (vsz_limit is defaulted (not set )) everything worked
I generally find that tweaking the memory alloted to the individual services a more balanced approach but it does take a lot of trial and error
also note memory in the system is also a factor my mail servers have 32G dedicated to them which is what the settings were based on that seem to work pretty good at the moment.
I am running without any setting thus the default
I got this info from :
https://doc.dovecot.org/configuration_manual/service_configuration/
vsz_limit
Limit the process’s address space (both RLIMIT_DATA and RLIMIT_AS if available). When the space is reached, some memory allocations may start failing with “Out of memory”, or the kernel may kill the process with signal 9. This setting is mainly intended to prevent memory leaks from eating up all of the memory, but there can be also legitimate reasons why the process reaches this limit. For example a huge mailbox may not be accessed if this limit is too low. The default value (18446744073709551615=2^64-1) sets the limit to default_vsz_limit, while 0 disables the limit entirely.
There are 3 types of services that need to be optimized in different ways:
Master services (e.g. auth, anvil, indexer, director, log):
Currently there isn’t any easy way to optimize these. If these
become a bottleneck, typically you need to run another Dovecot server. In some cases it may be possible to create multiple master processes and have each one be responsible for only specific users/processes, although this may also require some extra development.
Services that do disk I/O or other blocking operations (e.g. imap,
pop3, lmtp):
These should have client_limit=1, because any blocking
operation will block all the other clients and cause unnecessary delays and even timeouts. This means that process_limit specifies the maximum number of available parallel connections.
Services that have no blocking operations (e.g. imap-login,
pop3-login):
For best performance (but a bit less safety), these should have
process_limit and process_min_avail set to the number of CPU cores, so each CPU will be busy serving the process but without unnecessary context switches. Then client_limit needs to be set high enough to be able to serve all the needed connections (max connections=process_limit these services. Otherwise when the service_count is beginning to be
- client_limit). service_count is commonly set to unlimited (0) for
reached, the total number of available connections will shrink. With very bad luck that could mean that all the processes are simply waiting for the existing connections to die away before the process can die and a new one can be created. Although this could be made less likely by setting process_limit higher than process_min_avail, but that’s still not a guarantee since each process could get a very long running connection and the process_limit would be eventually reached.
Happy Saturday !!! Thanks - paul
Paul Kudla
Scom.ca Internet Services http://www.scom.ca 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3
Toronto 416.642.7266 Main 1.866.411.7266 Fax 1.888.892.7266 Email paul@scom.ca
On 1/6/2023 5:20 PM, Gerben Wierda wrote:
How problematic is it to have
default_vsz_limit = 0
in dovecot.conf? macOS+MacPorts had this as a requirement even.
Gerben
On 6 Jan 2023, at 16:49, Paul Kudla
mailto:paul@scom.ca> wrote: i ran into this as well
here is the full config for mine with replication
# cat dovecot.conf # 2.3.14 (cee3cbc0d): /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 12.1-RELEASE amd64 # Hostname: mail18.scom.ca http://mail18.scom.ca
auth_debug = no auth_debug_passwords = no
default_process_limit = 16384
mail_debug = no
#lock_method = dotlock #mail_max_lock_timeout = 300s
#mbox_read_locks = dotlock #mbox_write_locks = dotlock
mmap_disable = yes dotlock_use_excl = no mail_fsync = always mail_nfs_storage = no mail_nfs_index = no
auth_mechanisms = plain login auth_verbose = yes base_dir = /data/dovecot/run/ debug_log_path = syslog disable_plaintext_auth = no dsync_features = empty-header-workaround
info_log_path = syslog login_greeting = SCOM.CA Internet Services Inc. - Dovecot ready login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
mail_location = maildir:~/
mail_plugins = " virtual notify replication fts fts_lucene " mail_prefetch_count = 20
protocols = imap pop3 lmtp sieve
protocol lmtp { mail_plugins = $mail_plugins sieve postmaster_address = }
service lmtp { process_limit=1000 vsz_limit = 512m client_limit=1 unix_listener /usr/home/postfix.local/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } }
protocol lda { mail_plugins = $mail_plugins sieve }
service lda { process_limit=1000 vsz_limit = 512m }
service imap { process_limit=4096 vsz_limit = 2g client_limit=1 }
service pop3 { process_limit=1000 vsz_limit = 512m client_limit=1 }
namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / }
passdb { args = /usr/local/etc/dovecot/dovecot-pgsql.conf driver = sql }
doveadm_port = 12345 doveadm_password = secretxxxx
service doveadm { process_limit = 0 process_min_avail = 0 idle_kill = 0 client_limit = 1 user = vmail inet_listener { port = 12345 } }
service config { unix_listener config { user = vmail } }
dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u #dsync_remote_cmd = doveadm sync -d -u%u
replication_dsync_parameters = -d -N -l 300 -U
plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid, box, msgid, from, subject, size, vsize, flags push_notification_driver = dlog
sieve = file:~/sieve;active=~/sieve/.dovecot.sieve #sieve = ~/.dovecot.sieve sieve_duplicate_default_period = 1h sieve_duplicate_max_period = 1h sieve_extensions = +duplicate +notify +imapflags +vacation-seconds sieve_global_dir = /usr/local/etc/dovecot/sieve sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve
mail_replica = tcp:10.221.0.19:12345 #mail_replica = remote:vmail@10.221.0.19 mailto:vmail@10.221.0.19 #replication_sync_timeout = 2
fts = lucene fts_lucene = whitespace_chars=@. fts_autoindex = yes fts_languages = en }
#sieve_extensions = vnd.dovecot.duplicate
#sieve_plugins = vnd.dovecot.duplicate
service anvil { process_limit = 1 client_limit=5000 vsz_limit = 512m unix_listener anvil { group = vmail mode = 0666 } }
service indexer-worker { vsz_limit = 2g }
service auth { process_limit = 1 client_limit=5000 vsz_limit = 1g
unix_listener auth-userdb { mode = 0660 user = vmail group = vmail } unix_listener /var/spool/postfix/private/auth { mode = 0666 }
}
service stats { process_limit = 1000 vsz_limit = 1g unix_listener stats-reader { group = vmail mode = 0666 } unix_listener stats-writer { group = vmail mode = 0666 } } userdb { args = /usr/local/etc/dovecot/dovecot-pgsql.conf driver = sql
}
protocol imap { mail_max_userip_connections = 50 mail_plugins = $mail_plugins notify replication }
protocol pop3 { mail_max_userip_connections = 50 mail_plugins = $mail_plugins notify replication }
protocol imaps { mail_max_userip_connections = 25 mail_plugins = $mail_plugins notify replication }
protocol pop3s { mail_max_userip_connections = 25 mail_plugins = $mail_plugins notify replication }
service managesieve-login { process_limit = 1000 vsz_limit = 1g inet_listener sieve { port = 4190 } }
verbose_proctitle = yes
replication_max_conns = 100
replication_full_sync_interval = 1d
service replicator { client_limit = 0 drop_priv_before_exec = no idle_kill = 4294967295s process_limit = 1 process_min_avail = 0 service_count = 0 vsz_limit = 8g unix_listener replicator-doveadm { mode = 0600 user = vmail } vsz_limit = 8192M }
service aggregator { process_limit = 1000 #vsz_limit = 1g fifo_listener replication-notify-fifo { user = vmail group = vmail mode = 0666 }
}
service pop3-login { process_limit = 1000 client_limit = 100 vsz_limit = 512m }
service imap-urlauth-login { process_limit = 1000 client_limit = 1000 vsz_limit = 1g }
service imap-login { process_limit=1000 client_limit = 1000 vsz_limit = 1g }
protocol sieve { managesieve_implementation_string = Dovecot Pigeonhole managesieve_max_line_length = 65536 }
#Addition ssl config !include sni.conf
with sni cert support (examples)
# cat sni.conf #sni.conf ssl = yes verbose_ssl = yes ssl_dh =
#Default *.scom.ca http://scom.ca ssl_key =
local_name .scom.ca http://scom.ca { ssl_key =
}
local_name mail.clancyca.com http://mail.clancyca.com { ssl_key =http://mail.clancyca.com ssl_cert =http://mail.clancyca.com ssl_ca =http://mail.clancyca.com }
local_name mail.paulkudla.net http://mail.paulkudla.net { ssl_key =http://mail.paulkudla.net ssl_cert =http://mail.paulkudla.net ssl_ca =http://mail.paulkudla.net }
local_name mail.ekst.ca http://mail.ekst.ca { ssl_key =http://mail.ekst.ca ssl_cert =http://mail.ekst.ca ssl_ca =http://mail.ekst.ca }
local_name mail.hamletdevelopments.ca http://mail.hamletdevelopments.ca { ssl_key =http://mail.hamletdevelopments.ca ssl_cert =http://mail.hamletdevelopments.ca ssl_ca =http://mail.hamletdevelopments.ca }
pg sql support supporting replication
# cat dovecot-pgsql.conf driver = pgsql connect = host=localhost port=5433 dbname=scom_billing user=pgsql password= default_pass_scheme = PLAIN
password_query = SELECT username as user, password FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
user_query = SELECT home, uid, gid FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
#iterate_query = SELECT user, password FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
iterate_query = SELECT "username" as user, domain FROM email_users WHERE status = True and alias_flag = False
Happy Friday !!! Thanks - paul
Paul Kudla
Scom.ca http://Scom.ca Internet Services <http://www.scom.ca http://www.scom.ca> 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3
Toronto 416.642.7266 Main 1.866.411.7266 Fax 1.888.892.7266 Email paul@scom.ca mailto:paul@scom.ca
On 1/6/2023 5:32 AM, Gerben Wierda wrote:
On 6 Jan 2023, at 08:53, Aki Tuomi
mailto:aki.tuomi@open-xchange.com> wrote: On January 6, 2023 3:56:39 AM GMT+02:00, Gerben Wierda
mailto:gerben.wierda@rna.nl> wrote: One step further in my quest to create a replacement mail server.
I now have my old mail server (2.3.19.1, macOS + MacPorts) and my new (2.3.20, Alpine Linux, Docker, apk package). When I turn on replication it works, but, after a while I see:
Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when allocating 268435496 bytes Jan 06 00:50:32 replicator: Fatal: master: service(replicator): child 133 killed with signal 6 (core dumped) Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Sync failure: Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Remote sent invalid input: -
I've removed synchronous operation for now (found a message on the net suggesting that) but is this known and what does it mean?
Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda https://www.linkedin.com/in/gerbenwierda>) R&A IT Strategy <https://ea.rna.nl/ https://ea.rna.nl/> (main site) Book: Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/ https://ea.rna.nl/the-book/> Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/ https://ea.rna.nl/the-book-edition-iii/>
Dovecot default memory limit is 256M. You should probably set
service replicator { vsz_limit = 2G }
because replicator might have to use more memory, especially for larger indexes.
Aki That is a good tip as well. I had followed this bit of experience from someone else: https://marc.info/?l=dovecot&m=164438199727640 https://marc.info/?l=dovecot&m=164438199727640, haven't seen any err message since. But that might be because they are in sync now and both sides are aware. Can I trigger full replication again so I can test? Gerben
This message has been scanned for viruses and dangerous content by *MailScanner* http://www.mailscanner.info/, and is believed to be clean.
-- This message has been scanned for viruses and dangerous content by *MailScanner* http://www.mailscanner.info/, and is believed to be clean.
Hello,
try updating to Dovecot v2.3.20 // I had the same in earlier versions, but it doesn't happen any longer now.
Increasing the limit didn't help me at that time, even the 0 didn't help.
Am 07.01.23 um 11:17 schrieb Paul Kudla:
ok a little bit more info ....
My servers all run under FreeBSD 12.xx
which was also the base for the apple operating system origionally.
setting default_vsz_limit = 0 i seem to remember trying with no so great results
setting to zero can cause memory over runs (espically with replication) etc i found that when i used the config i sent eariler (vsz_limit is defaulted (not set )) everything worked
I generally find that tweaking the memory alloted to the individual services a more balanced approach but it does take a lot of trial and error
also note memory in the system is also a factor my mail servers have 32G dedicated to them which is what the settings were based on that seem to work pretty good at the moment.
I am running without any setting thus the default
I got this info from :
https://doc.dovecot.org/configuration_manual/service_configuration/
vsz_limit
Limit the process’s address space (both RLIMIT_DATA and RLIMIT_AS if available). When the space is reached, some memory allocations may start failing with “Out of memory”, or the kernel may kill the process with signal 9. This setting is mainly intended to prevent memory leaks from eating up all of the memory, but there can be also legitimate reasons why the process reaches this limit. For example a huge mailbox may not be accessed if this limit is too low. The default value (18446744073709551615=2^64-1) sets the limit to default_vsz_limit, while 0 disables the limit entirely.
There are 3 types of services that need to be optimized in different ways:
Master services (e.g. auth, anvil, indexer, director, log):
Currently there isn’t any easy way to optimize these. If these become a bottleneck, typically you need to run another Dovecot server. In some cases it may be possible to create multiple master processes and have each one be responsible for only specific users/processes, although this may also require some extra development.
Services that do disk I/O or other blocking operations (e.g. imap, pop3, lmtp):
These should have client_limit=1, because any blocking operation will block all the other clients and cause unnecessary delays and even timeouts. This means that process_limit specifies the maximum number of available parallel connections.
Services that have no blocking operations (e.g. imap-login, pop3-login):
For best performance (but a bit less safety), these should have process_limit and process_min_avail set to the number of CPU cores, so each CPU will be busy serving the process but without unnecessary context switches. Then client_limit needs to be set high enough to be able to serve all the needed connections (max connections=process_limit * client_limit). service_count is commonly set to unlimited (0) for these services. Otherwise when the service_count is beginning to be reached, the total number of available connections will shrink. With very bad luck that could mean that all the processes are simply waiting for the existing connections to die away before the process can die and a new one can be created. Although this could be made less likely by setting process_limit higher than process_min_avail, but that’s still not a guarantee since each process could get a very long running connection and the process_limit would be eventually reached.
Happy Saturday !!! Thanks - paul
Paul Kudla
Scom.ca Internet Services http://www.scom.ca 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3
Toronto 416.642.7266 Main 1.866.411.7266 Fax 1.888.892.7266 Email paul@scom.ca
On 1/6/2023 5:20 PM, Gerben Wierda wrote:
How problematic is it to have
default_vsz_limit = 0
in dovecot.conf? macOS+MacPorts had this as a requirement even.
Gerben
On 6 Jan 2023, at 16:49, Paul Kudla
mailto:paul@scom.ca> wrote: i ran into this as well
here is the full config for mine with replication
# cat dovecot.conf # 2.3.14 (cee3cbc0d): /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 12.1-RELEASE amd64 # Hostname: mail18.scom.ca http://mail18.scom.ca
auth_debug = no auth_debug_passwords = no
default_process_limit = 16384
mail_debug = no
#lock_method = dotlock #mail_max_lock_timeout = 300s
#mbox_read_locks = dotlock #mbox_write_locks = dotlock
mmap_disable = yes dotlock_use_excl = no mail_fsync = always mail_nfs_storage = no mail_nfs_index = no
auth_mechanisms = plain login auth_verbose = yes base_dir = /data/dovecot/run/ debug_log_path = syslog disable_plaintext_auth = no dsync_features = empty-header-workaround
info_log_path = syslog login_greeting = SCOM.CA Internet Services Inc. - Dovecot ready login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
mail_location = maildir:~/
mail_plugins = " virtual notify replication fts fts_lucene " mail_prefetch_count = 20
protocols = imap pop3 lmtp sieve
protocol lmtp { mail_plugins = $mail_plugins sieve postmaster_address = }
service lmtp { process_limit=1000 vsz_limit = 512m client_limit=1 unix_listener /usr/home/postfix.local/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } }
protocol lda { mail_plugins = $mail_plugins sieve }
service lda { process_limit=1000 vsz_limit = 512m }
service imap { process_limit=4096 vsz_limit = 2g client_limit=1 }
service pop3 { process_limit=1000 vsz_limit = 512m client_limit=1 }
namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / }
passdb { args = /usr/local/etc/dovecot/dovecot-pgsql.conf driver = sql }
doveadm_port = 12345 doveadm_password = secretxxxx
service doveadm { process_limit = 0 process_min_avail = 0 idle_kill = 0 client_limit = 1 user = vmail inet_listener { port = 12345 } }
service config { unix_listener config { user = vmail } }
dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u #dsync_remote_cmd = doveadm sync -d -u%u
replication_dsync_parameters = -d -N -l 300 -U
plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid, box, msgid, from, subject, size, vsize, flags push_notification_driver = dlog
sieve = file:~/sieve;active=~/sieve/.dovecot.sieve #sieve = ~/.dovecot.sieve sieve_duplicate_default_period = 1h sieve_duplicate_max_period = 1h sieve_extensions = +duplicate +notify +imapflags +vacation-seconds sieve_global_dir = /usr/local/etc/dovecot/sieve sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve
mail_replica = tcp:10.221.0.19:12345 #mail_replica = remote:vmail@10.221.0.19 mailto:vmail@10.221.0.19 #replication_sync_timeout = 2
fts = lucene fts_lucene = whitespace_chars=@. fts_autoindex = yes fts_languages = en }
#sieve_extensions = vnd.dovecot.duplicate
#sieve_plugins = vnd.dovecot.duplicate
service anvil { process_limit = 1 client_limit=5000 vsz_limit = 512m unix_listener anvil { group = vmail mode = 0666 } }
service indexer-worker { vsz_limit = 2g }
service auth { process_limit = 1 client_limit=5000 vsz_limit = 1g
unix_listener auth-userdb { mode = 0660 user = vmail group = vmail } unix_listener /var/spool/postfix/private/auth { mode = 0666 }
}
service stats { process_limit = 1000 vsz_limit = 1g unix_listener stats-reader { group = vmail mode = 0666 } unix_listener stats-writer { group = vmail mode = 0666 } } userdb { args = /usr/local/etc/dovecot/dovecot-pgsql.conf driver = sql
}
protocol imap { mail_max_userip_connections = 50 mail_plugins = $mail_plugins notify replication }
protocol pop3 { mail_max_userip_connections = 50 mail_plugins = $mail_plugins notify replication }
protocol imaps { mail_max_userip_connections = 25 mail_plugins = $mail_plugins notify replication }
protocol pop3s { mail_max_userip_connections = 25 mail_plugins = $mail_plugins notify replication }
service managesieve-login { process_limit = 1000 vsz_limit = 1g inet_listener sieve { port = 4190 } }
verbose_proctitle = yes
replication_max_conns = 100
replication_full_sync_interval = 1d
service replicator { client_limit = 0 drop_priv_before_exec = no idle_kill = 4294967295s process_limit = 1 process_min_avail = 0 service_count = 0 vsz_limit = 8g unix_listener replicator-doveadm { mode = 0600 user = vmail } vsz_limit = 8192M }
service aggregator { process_limit = 1000 #vsz_limit = 1g fifo_listener replication-notify-fifo { user = vmail group = vmail mode = 0666 }
}
service pop3-login { process_limit = 1000 client_limit = 100 vsz_limit = 512m }
service imap-urlauth-login { process_limit = 1000 client_limit = 1000 vsz_limit = 1g }
service imap-login { process_limit=1000 client_limit = 1000 vsz_limit = 1g }
protocol sieve { managesieve_implementation_string = Dovecot Pigeonhole managesieve_max_line_length = 65536 }
#Addition ssl config !include sni.conf
with sni cert support (examples)
# cat sni.conf #sni.conf ssl = yes verbose_ssl = yes ssl_dh =
#Default *.scom.ca http://scom.ca ssl_key =
local_name .scom.ca http://scom.ca { ssl_key =
}
local_name mail.clancyca.com http://mail.clancyca.com { ssl_key =http://mail.clancyca.com ssl_cert =http://mail.clancyca.com ssl_ca =http://mail.clancyca.com }
local_name mail.paulkudla.net http://mail.paulkudla.net { ssl_key =http://mail.paulkudla.net ssl_cert =http://mail.paulkudla.net ssl_ca =http://mail.paulkudla.net }
local_name mail.ekst.ca http://mail.ekst.ca { ssl_key =http://mail.ekst.ca ssl_cert =http://mail.ekst.ca ssl_ca =http://mail.ekst.ca }
local_name mail.hamletdevelopments.ca http://mail.hamletdevelopments.ca { ssl_key =http://mail.hamletdevelopments.ca ssl_cert =http://mail.hamletdevelopments.ca ssl_ca =http://mail.hamletdevelopments.ca }
pg sql support supporting replication
# cat dovecot-pgsql.conf driver = pgsql connect = host=localhost port=5433 dbname=scom_billing user=pgsql password= default_pass_scheme = PLAIN
password_query = SELECT username as user, password FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
user_query = SELECT home, uid, gid FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
#iterate_query = SELECT user, password FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
iterate_query = SELECT "username" as user, domain FROM email_users WHERE status = True and alias_flag = False
Happy Friday !!! Thanks - paul
Paul Kudla
Scom.ca http://Scom.ca Internet Services <http://www.scom.ca http://www.scom.ca> 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3
Toronto 416.642.7266 Main 1.866.411.7266 Fax 1.888.892.7266 Email paul@scom.ca mailto:paul@scom.ca
On 1/6/2023 5:32 AM, Gerben Wierda wrote:
On 6 Jan 2023, at 08:53, Aki Tuomi
mailto:aki.tuomi@open-xchange.com> wrote: On January 6, 2023 3:56:39 AM GMT+02:00, Gerben Wierda
mailto:gerben.wierda@rna.nl> wrote: One step further in my quest to create a replacement mail server.
I now have my old mail server (2.3.19.1, macOS + MacPorts) and my new (2.3.20, Alpine Linux, Docker, apk package). When I turn on replication it works, but, after a while I see:
Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when allocating 268435496 bytes Jan 06 00:50:32 replicator: Fatal: master: service(replicator): child 133 killed with signal 6 (core dumped) Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Sync failure: Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Remote sent invalid input: -
I've removed synchronous operation for now (found a message on the net suggesting that) but is this known and what does it mean?
Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda https://www.linkedin.com/in/gerbenwierda>) R&A IT Strategy <https://ea.rna.nl/ https://ea.rna.nl/> (main site) Book: Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/ https://ea.rna.nl/the-book/> Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/ https://ea.rna.nl/the-book-edition-iii/>
Dovecot default memory limit is 256M. You should probably set
service replicator { vsz_limit = 2G }
because replicator might have to use more memory, especially for larger indexes.
Aki That is a good tip as well. I had followed this bit of experience from someone else: https://marc.info/?l=dovecot&m=164438199727640 https://marc.info/?l=dovecot&m=164438199727640, haven't seen any err message since. But that might be because they are in sync now and both sides are aware. Can I trigger full replication again so I can test? Gerben
This message has been scanned for viruses and dangerous content by *MailScanner* http://www.mailscanner.info/, and is believed to be clean.
-- This message has been scanned for viruses and dangerous content by *MailScanner* http://www.mailscanner.info/, and is believed to be clean.
On 6 Jan 2023, at 08:53, Aki Tuomi aki.tuomi@open-xchange.com wrote:
On January 6, 2023 3:56:39 AM GMT+02:00, Gerben Wierda gerben.wierda@rna.nl wrote:
One step further in my quest to create a replacement mail server.
I now have my old mail server (2.3.19.1, macOS + MacPorts) and my new (2.3.20, Alpine Linux, Docker, apk package). When I turn on replication it works, but, after a while I see:
Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when allocating 268435496 bytes Jan 06 00:50:32 replicator: Fatal: master: service(replicator): child 133 killed with signal 6 (core dumped) Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Sync failure: Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Remote sent invalid input: -
I've removed synchronous operation for now (found a message on the net suggesting that) but is this known and what does it mean?
Gerben Wierda (LinkedIn https://www.linkedin.com/in/gerbenwierda) R&A IT Strategy https://ea.rna.nl/ (main site) Book: Chess and the Art of Enterprise Architecture https://ea.rna.nl/the-book/ Book: Mastering ArchiMate https://ea.rna.nl/the-book-edition-iii/
Dovecot default memory limit is 256M. You should probably set
service replicator { vsz_limit = 2G }
because replicator might have to use more memory, especially for larger indexes.
Aki
Coming back to this one. doveadm config says everything is vsz_limit = 18446744073709551615 B
Isn't that more than 2G already (nd certainly more than 256M?)?
I also tried putting
default_vsz_limit = 2G
in dovecot.conf but that doesn't change anything either.
What am I missing?
i ran into this as well
here is the full config for mine with replication
# cat dovecot.conf # 2.3.14 (cee3cbc0d): /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 12.1-RELEASE amd64 # Hostname: mail18.scom.ca
auth_debug = no auth_debug_passwords = no
default_process_limit = 16384
mail_debug = no
#lock_method = dotlock #mail_max_lock_timeout = 300s
#mbox_read_locks = dotlock #mbox_write_locks = dotlock
mmap_disable = yes dotlock_use_excl = no mail_fsync = always mail_nfs_storage = no mail_nfs_index = no
auth_mechanisms = plain login auth_verbose = yes base_dir = /data/dovecot/run/ debug_log_path = syslog disable_plaintext_auth = no dsync_features = empty-header-workaround
info_log_path = syslog login_greeting = SCOM.CA Internet Services Inc. - Dovecot ready login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
mail_location = maildir:~/
mail_plugins = " virtual notify replication fts fts_lucene " mail_prefetch_count = 20
protocols = imap pop3 lmtp sieve
protocol lmtp { mail_plugins = $mail_plugins sieve postmaster_address = }
service lmtp { process_limit=1000 vsz_limit = 512m client_limit=1 unix_listener /usr/home/postfix.local/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } }
protocol lda { mail_plugins = $mail_plugins sieve }
service lda { process_limit=1000 vsz_limit = 512m }
service imap { process_limit=4096 vsz_limit = 2g client_limit=1 }
service pop3 { process_limit=1000 vsz_limit = 512m client_limit=1 }
namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / }
passdb { args = /usr/local/etc/dovecot/dovecot-pgsql.conf driver = sql }
doveadm_port = 12345 doveadm_password = secretxxxx
service doveadm { process_limit = 0 process_min_avail = 0 idle_kill = 0 client_limit = 1 user = vmail inet_listener { port = 12345 } }
service config { unix_listener config { user = vmail } }
dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u #dsync_remote_cmd = doveadm sync -d -u%u
replication_dsync_parameters = -d -N -l 300 -U
plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid, box, msgid, from, subject, size, vsize, flags push_notification_driver = dlog
sieve = file:~/sieve;active=~/sieve/.dovecot.sieve #sieve = ~/.dovecot.sieve sieve_duplicate_default_period = 1h sieve_duplicate_max_period = 1h sieve_extensions = +duplicate +notify +imapflags +vacation-seconds sieve_global_dir = /usr/local/etc/dovecot/sieve sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve
mail_replica = tcp:10.221.0.19:12345 #mail_replica = remote:vmail@10.221.0.19 #replication_sync_timeout = 2
fts = lucene fts_lucene = whitespace_chars=@. fts_autoindex = yes fts_languages = en }
#sieve_extensions = vnd.dovecot.duplicate
#sieve_plugins = vnd.dovecot.duplicate
service anvil { process_limit = 1 client_limit=5000 vsz_limit = 512m unix_listener anvil { group = vmail mode = 0666 } }
service indexer-worker { vsz_limit = 2g }
service auth { process_limit = 1 client_limit=5000 vsz_limit = 1g
unix_listener auth-userdb { mode = 0660 user = vmail group = vmail } unix_listener /var/spool/postfix/private/auth { mode = 0666 }
}
service stats { process_limit = 1000 vsz_limit = 1g unix_listener stats-reader { group = vmail mode = 0666 } unix_listener stats-writer { group = vmail mode = 0666 } } userdb { args = /usr/local/etc/dovecot/dovecot-pgsql.conf driver = sql
}
protocol imap { mail_max_userip_connections = 50 mail_plugins = $mail_plugins notify replication }
protocol pop3 { mail_max_userip_connections = 50 mail_plugins = $mail_plugins notify replication }
protocol imaps { mail_max_userip_connections = 25 mail_plugins = $mail_plugins notify replication }
protocol pop3s { mail_max_userip_connections = 25 mail_plugins = $mail_plugins notify replication }
service managesieve-login { process_limit = 1000 vsz_limit = 1g inet_listener sieve { port = 4190 } }
verbose_proctitle = yes
replication_max_conns = 100
replication_full_sync_interval = 1d
service replicator { client_limit = 0 drop_priv_before_exec = no idle_kill = 4294967295s process_limit = 1 process_min_avail = 0 service_count = 0 vsz_limit = 8g unix_listener replicator-doveadm { mode = 0600 user = vmail } vsz_limit = 8192M }
service aggregator { process_limit = 1000 #vsz_limit = 1g fifo_listener replication-notify-fifo { user = vmail group = vmail mode = 0666 }
}
service pop3-login { process_limit = 1000 client_limit = 100 vsz_limit = 512m }
service imap-urlauth-login { process_limit = 1000 client_limit = 1000 vsz_limit = 1g }
service imap-login { process_limit=1000 client_limit = 1000 vsz_limit = 1g }
protocol sieve { managesieve_implementation_string = Dovecot Pigeonhole managesieve_max_line_length = 65536 }
#Addition ssl config !include sni.conf
with sni cert support (examples)
# cat sni.conf #sni.conf ssl = yes verbose_ssl = yes ssl_dh =
#Default *.scom.ca ssl_key =
local_name .scom.ca { ssl_key =
}
local_name mail.clancyca.com { ssl_key =
local_name mail.paulkudla.net { ssl_key =
local_name mail.ekst.ca { ssl_key =
local_name mail.hamletdevelopments.ca { ssl_key =
pg sql support supporting replication
# cat dovecot-pgsql.conf driver = pgsql connect = host=localhost port=5433 dbname=scom_billing user=pgsql password= default_pass_scheme = PLAIN
password_query = SELECT username as user, password FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
user_query = SELECT home, uid, gid FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
#iterate_query = SELECT user, password FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
iterate_query = SELECT "username" as user, domain FROM email_users WHERE status = True and alias_flag = False
Happy Friday !!! Thanks - paul
Paul Kudla
Scom.ca Internet Services http://www.scom.ca 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3
Toronto 416.642.7266 Main 1.866.411.7266 Fax 1.888.892.7266 Email paul@scom.ca On 1/5/2023 8:56 PM, Gerben Wierda wrote:
One step further in my quest to create a replacement mail server.
I now have my old mail server (2.3.19.1, macOS + MacPorts) and my new (2.3.20, Alpine Linux, Docker, apk package). When I turn on replication it works, but, after a while I see:
Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when allocating 268435496 bytes Jan 06 00:50:32 replicator: Fatal: master: service(replicator): child 133 killed with signal 6 (core dumped) Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Sync failure: Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Remote sent invalid input: -
I've removed synchronous operation for now (found a message on the net suggesting that) but is this known and what does it mean?
Gerben Wierda (LinkedIn https://www.linkedin.com/in/gerbenwierda) R&A IT Strategy https://ea.rna.nl/ (main site) Book: Chess and the Art of Enterprise Architecture https://ea.rna.nl/the-book/ Book: Mastering ArchiMate https://ea.rna.nl/the-book-edition-iii/
-- This message has been scanned for viruses and dangerous content by *MailScanner* http://www.mailscanner.info/, and is believed to be clean.
Hi all, I've been pounding my head against the sand for a while here trying to figure out why I can't get:
doveadm user '*'
working properly. I've got a Debian 11 VPS runnig dovecot version 2.3.21-1+debian10 and it works great. But not I'm trying to add in simple replication to a home dovecot instance over a wireguard tunnel so I can do backups and have a little better resiliency. Maybe.
In any case, my sqlite schema looks like this:
sqlite> .schema virtual_users
CREATE TABLE `virtual_users` (
`id` integer NOT NULL PRIMARY KEY AUTOINCREMENT
, `domain_id` integer NOT NULL
, `password` varchar(106) NOT NULL
, `email` varchar(100) NOT NULL
, UNIQUE (`email`)
, CONSTRAINT `virtual_users_ibfk_1` FOREIGN KEY (`domain_id`) REFERENCES `virtual_domains` (`id`) E
);
CREATE INDEX "idx_virtual_users_domain_id" ON "virtual_users" (`domain_id`);
and I don't have any other tables. The 'domain_id' was/is a leftover from my thinking I needed it for extra testing of other domains and such.
I can do 'doveadm user john@stoffel.org' and it works just fine. When I do "doveadm user '*'" it fails and I get:
doveadm user '*'
Error: auth-master: userdb list: User listing returned failure
Fatal: user listing failed
So my config looks like this:
root@mail:/etc/dovecot/conf.d# cat auth-sql.conf.ext
# Authentication for SQL users. Included from 10-auth.conf.
#
#
passdb { driver = sql
# Path for SQL configuration file, see
example-config/dovecot-sql.conf.ext
args = /etc/dovecot/dovecot-sql.conf.ext
}
userdb { driver = static args = uid=mail gid=mail home=/var/mail/%d/%n }
My /etc/dovecot/dovecot-sql.conf.ext has the following:
driver = sqlite connect = /etc/dovecot/private/virtual_users.sqlite3
default_pass_scheme = SHA512-CRYPT
password_query = SELECT '/var/mail/%d/%u' AS userdb_home, 'mail' AS userdb_uid, 'mail' AS userdb_gid, email as user, password FROM virtual_users WHERE email='%u';
iterate_query = SELECT email AS user from virtual_users;
And my general doveadm config output is this, slightly edited down to remove stuff I don't think I need to show is at the end. Any hints on what I've done wrong here? Do I need a more complete sqlite3 schema? I wish I could get more debugging info on what query it's trying to run and the error(s) it's getting.
Thanks, John
# 2.3.21 (47349e2482): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.21 (f6cd4b8e) # OS: Linux 5.10.0-26-amd64 x86_64 Debian 11.8 ext4 # Hostname: localhost # NOTE: Send doveconf -n output instead when asking for help. auth_anonymous_username = anonymous auth_cache_negative_ttl = 1 hours auth_cache_size = 0 auth_cache_ttl = 1 hours auth_cache_verify_password_with_worker = no auth_debug = no auth_debug_passwords = no auth_failure_delay = 2 secs auth_gssapi_hostname = auth_krb5_keytab = auth_master_user_separator = auth_mechanisms = plain login auth_policy_check_after_auth = yes auth_policy_check_before_auth = yes auth_policy_hash_mech = sha256 auth_policy_hash_nonce = auth_policy_hash_truncate = 12 auth_policy_log_only = no auth_policy_reject_on_fail = no auth_policy_report_after_auth = yes auth_policy_request_attributes = login=%{requested_username} pwhash=%{hashed_password} remote=%{rip} device_id=%{client_id} protocol=%s session_id=%{session} auth_policy_server_api_header = auth_policy_server_timeout_msecs = 2000 auth_policy_server_url = auth_proxy_self = auth_realms = auth_socket_path = auth-userdb auth_ssl_require_client_cert = no auth_ssl_username_from_cert = no auth_stats = no auth_use_winbind = no auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@ auth_username_format = %Lu auth_username_translation = auth_verbose = no auth_verbose_passwords = no auth_winbind_helper_path = /usr/bin/ntlm_auth auth_worker_max_count = 30 base_dir = /run/dovecot config_cache_size = 1 M debug_log_path = default_client_limit = 1000 default_idle_kill = 1 mins default_internal_group = dovecot default_internal_user = dovecot default_login_user = dovenull default_process_limit = 100 default_vsz_limit = 256 M deliver_log_format = msgid=%m: %$ dict_db_config = disable_plaintext_auth = yes dotlock_use_excl = yes doveadm_allowed_commands = doveadm_api_key = doveadm_http_rawlog_dir = doveadm_password = doveadm_port = 0 doveadm_socket_path = doveadm-server doveadm_ssl = no doveadm_username = doveadm doveadm_worker_count = 0 first_valid_gid = 1 first_valid_uid = 0 import_environment = TZ CORE_OUTOFMEM CORE_ERROR LISTEN_PID LISTEN_FDS NOTIFY_SOCKET info_log_path = libexec_dir = /usr/lib/dovecot listen = * log_core_filter = log_debug = log_path = /var/log/dovecot.log log_timestamp = "%b %d %H:%M:%S " mail_access_groups = mail_always_cache_fields = mail_attachment_detection_options = mail_attachment_dir = mail_attachment_fs = sis posix mail_attachment_hash = %{sha1} mail_attachment_min_size = 128 k mail_attribute_dict = mail_cache_fields = flags mail_chroot = mail_debug = no mail_fsync = optimized mail_full_filesystem_access = no mail_gid = mail_home = mail_location = maildir:/var/mail/%d/%n/Maildir mail_log_prefix = "%s(%u)<%{pid}><%{session}>: " mail_max_keyword_length = 50 mail_max_lock_timeout = 0 mail_max_userip_connections = 10 mail_never_cache_fields = imap.envelope mail_nfs_index = no mail_nfs_storage = no mail_plugin_dir = /usr/lib/dovecot/modules mail_plugins = " notify replication" mail_prefetch_count = 0 mail_privileged_group = mail mail_save_crlf = no mail_server_admin = mail_server_comment = mail_shared_explicit_inbox = no mail_sort_max_read_count = 0 mail_temp_dir = /tmp mail_temp_scan_interval = 1 weeks mail_uid = mail_vsize_bg_after_count = 0 passdb { args = /etc/dovecot/dovecot-sql.conf.ext auth_verbose = default default_fields = deny = no driver = sql master = no mechanisms = name = override_fields = pass = no result_failure = continue result_internalfail = continue result_success = return-ok skip = never username_filter = } protocols = imap lmtp sieve recipient_delimiter = +_ service auth-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = auth -w extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = worker unix_listener auth-worker { group = mode = 0600 user = $default_internal_user } user = mail vsz_limit = 18446744073709551615 B } service auth { chroot = client_limit = 0 drop_priv_before_exec = no executable = auth extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-client { group = mode = 0600 user = $default_internal_user } unix_listener auth-login { group = mode = 0600 user = $default_internal_user } unix_listener auth-master { group = mode = 0600 user = } unix_listener auth-userdb { group = mode = 0666 user = mail } unix_listener login/login { group = mode = 0666 user = } unix_listener token-login/tokenlogin { group = mode = 0666 user = } user = dovecot vsz_limit = 18446744073709551615 B } service doveadm { chroot = client_limit = 1 drop_priv_before_exec = no executable = doveadm-server extra_groups = $default_internal_group group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 1 type = unix_listener doveadm-server { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service log { chroot = client_limit = 0 drop_priv_before_exec = no executable = log extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = log unix_listener log-errors { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } state_dir = /var/lib/dovecot stats_http_rawlog_dir = syslog_facility = mail userdb { args = uid=mail gid=mail home=/var/mail/%d/%n auth_verbose = default default_fields = driver = static name = override_fields = result_failure = continue result_internalfail = continue result_success = return-ok skip = never }
"John" == John Stoffel john@stoffel.org writes:
Do I think I'm on the right track here, since I removed the following from /etc/dovecot/conf.d/auth-sql.conf.ext
#userdb {
# driver = static
# args = uid=mail gid=mail home=/var/mail/%d/%n
#}
So now my error is as follows:
# doveadm user -u '*'
Error: auth-master: userdb list: User listing returned failure
Fatal: user listing failed
Because now when I restart dovecot, I see the following in the log:
Dec 01 16:55:14 master: Info: Dovecot v2.3.21 (47349e2482) starting up
for imap, lmtp, sieve (core dumps disabled)
Dec 01 16:55:14 auth: Warning: sql: Ignoring changed iterate_query in
/etc/dovecot/dovecot-sql.conf.ext, because userdb sql not used. (If
this is intentional, set userdb_warning_disable=yes)
Dec 01 16:55:14 auth: Error: auth-master client: Trying to iterate
users, but userdbs don't support it (created 0 msecs ago, handshake 0
msecs ago)
So I commented out my 'iterate_query = ...' (see below) from /etc/postfix/dovecot-sql.conf.ext and now I get the error on startup which says:
Dec 01 16:57:42 master: Info: Dovecot v2.3.21 (47349e2482) starting up for imap, lmtp, sieve (core dumps disabled) Dec 01 16:57:42 auth: Error: auth-master client: Trying to iterate users, but userdbs don't support it (created 0 msecs ago, handshake 0 msecs ago) Dec 01 16:57:42 replicator: Error: auth-master: userdb list: User listing returned failure Dec 01 16:57:42 replicator: Error: listing users failed, can't replicate existing data
Which tells me I need the iteracte_users setting, but I've got a bogus query in there. So I think I should be using something like this:
iterate_query = SELECT email AS user from virtual_users;
where 'virtual_users' is the one and only table in my sqlite db file. And I'm just returning the 'email' column as 'user', since that's what it seems to expect.
Hmmm...
I've been pounding my head against the sand for a while here trying to figure out why I can't get:
doveadm user '*'
working properly. I've got a Debian 11 VPS runnig dovecot version 2.3.21-1+debian10 and it works great. But not I'm trying to add in simple replication to a home dovecot instance over a wireguard tunnel so I can do backups and have a little better resiliency. Maybe.
In any case, my sqlite schema looks like this:
sqlite> .schema virtual_users
CREATE TABLE `virtual_users` ( `id` integer NOT NULL PRIMARY KEY AUTOINCREMENT , `domain_id` integer NOT NULL , `password` varchar(106) NOT NULL , `email` varchar(100) NOT NULL , UNIQUE (`email`) , CONSTRAINT `virtual_users_ibfk_1` FOREIGN KEY (`domain_id`) REFERENCES `virtual_domains` (`id`) E ); CREATE INDEX "idx_virtual_users_domain_id" ON "virtual_users" (`domain_id`);
and I don't have any other tables. The 'domain_id' was/is a leftover from my thinking I needed it for extra testing of other domains and such.
I can do 'doveadm user john@stoffel.org' and it works just fine. When I do "doveadm user '*'" it fails and I get:
doveadm user '*' Error: auth-master: userdb list: User listing returned failure Fatal: user listing failed
So my config looks like this:
root@mail:/etc/dovecot/conf.d# cat auth-sql.conf.ext # Authentication for SQL users. Included from 10-auth.conf. # #
passdb { driver = sql
# Path for SQL configuration file, see example-config/dovecot-sql.conf.ext args = /etc/dovecot/dovecot-sql.conf.ext
}
userdb { driver = static args = uid=mail gid=mail home=/var/mail/%d/%n }
My /etc/dovecot/dovecot-sql.conf.ext has the following:
driver = sqlite connect = /etc/dovecot/private/virtual_users.sqlite3
default_pass_scheme = SHA512-CRYPT
password_query = SELECT '/var/mail/%d/%u' AS userdb_home, 'mail' AS userdb_uid, 'mail' AS userdb_gid, email as user, password FROM virtual_users WHERE email='%u';
iterate_query = SELECT email AS user from virtual_users;
And my general doveadm config output is this, slightly edited down to remove stuff I don't think I need to show is at the end. Any hints on what I've done wrong here? Do I need a more complete sqlite3 schema? I wish I could get more debugging info on what query it's trying to run and the error(s) it's getting.
Thanks, John
# 2.3.21 (47349e2482): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.21 (f6cd4b8e) # OS: Linux 5.10.0-26-amd64 x86_64 Debian 11.8 ext4 # Hostname: localhost # NOTE: Send doveconf -n output instead when asking for help. auth_anonymous_username = anonymous auth_cache_negative_ttl = 1 hours auth_cache_size = 0 auth_cache_ttl = 1 hours auth_cache_verify_password_with_worker = no auth_debug = no auth_debug_passwords = no auth_failure_delay = 2 secs auth_gssapi_hostname = auth_krb5_keytab = auth_master_user_separator = auth_mechanisms = plain login auth_policy_check_after_auth = yes auth_policy_check_before_auth = yes auth_policy_hash_mech = sha256 auth_policy_hash_nonce = auth_policy_hash_truncate = 12 auth_policy_log_only = no auth_policy_reject_on_fail = no auth_policy_report_after_auth = yes auth_policy_request_attributes = login=%{requested_username} pwhash=%{hashed_password} remote=%{rip} device_id=%{client_id} protocol=%s session_id=%{session} auth_policy_server_api_header = auth_policy_server_timeout_msecs = 2000 auth_policy_server_url = auth_proxy_self = auth_realms = auth_socket_path = auth-userdb auth_ssl_require_client_cert = no auth_ssl_username_from_cert = no auth_stats = no auth_use_winbind = no auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@ auth_username_format = %Lu auth_username_translation = auth_verbose = no auth_verbose_passwords = no auth_winbind_helper_path = /usr/bin/ntlm_auth auth_worker_max_count = 30 base_dir = /run/dovecot config_cache_size = 1 M debug_log_path = default_client_limit = 1000 default_idle_kill = 1 mins default_internal_group = dovecot default_internal_user = dovecot default_login_user = dovenull default_process_limit = 100 default_vsz_limit = 256 M deliver_log_format = msgid=%m: %$ dict_db_config = disable_plaintext_auth = yes dotlock_use_excl = yes doveadm_allowed_commands = doveadm_api_key = doveadm_http_rawlog_dir = doveadm_password = doveadm_port = 0 doveadm_socket_path = doveadm-server doveadm_ssl = no doveadm_username = doveadm doveadm_worker_count = 0 first_valid_gid = 1 first_valid_uid = 0 import_environment = TZ CORE_OUTOFMEM CORE_ERROR LISTEN_PID LISTEN_FDS NOTIFY_SOCKET info_log_path = libexec_dir = /usr/lib/dovecot listen = * log_core_filter = log_debug = log_path = /var/log/dovecot.log log_timestamp = "%b %d %H:%M:%S " mail_access_groups = mail_always_cache_fields = mail_attachment_detection_options = mail_attachment_dir = mail_attachment_fs = sis posix mail_attachment_hash = %{sha1} mail_attachment_min_size = 128 k mail_attribute_dict = mail_cache_fields = flags mail_chroot = mail_debug = no mail_fsync = optimized mail_full_filesystem_access = no mail_gid = mail_home = mail_location = maildir:/var/mail/%d/%n/Maildir mail_log_prefix = "%s(%u)<%{pid}><%{session}>: " mail_max_keyword_length = 50 mail_max_lock_timeout = 0 mail_max_userip_connections = 10 mail_never_cache_fields = imap.envelope mail_nfs_index = no mail_nfs_storage = no mail_plugin_dir = /usr/lib/dovecot/modules mail_plugins = " notify replication" mail_prefetch_count = 0 mail_privileged_group = mail mail_save_crlf = no mail_server_admin = mail_server_comment = mail_shared_explicit_inbox = no mail_sort_max_read_count = 0 mail_temp_dir = /tmp mail_temp_scan_interval = 1 weeks mail_uid = mail_vsize_bg_after_count = 0 passdb { args = /etc/dovecot/dovecot-sql.conf.ext auth_verbose = default default_fields = deny = no driver = sql master = no mechanisms = name = override_fields = pass = no result_failure = continue result_internalfail = continue result_success = return-ok skip = never username_filter = } protocols = imap lmtp sieve recipient_delimiter = +_ service auth-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = auth -w extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = worker unix_listener auth-worker { group = mode = 0600 user = $default_internal_user } user = mail vsz_limit = 18446744073709551615 B } service auth { chroot = client_limit = 0 drop_priv_before_exec = no executable = auth extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-client { group = mode = 0600 user = $default_internal_user } unix_listener auth-login { group = mode = 0600 user = $default_internal_user } unix_listener auth-master { group = mode = 0600 user = } unix_listener auth-userdb { group = mode = 0666 user = mail } unix_listener login/login { group = mode = 0666 user = } unix_listener token-login/tokenlogin { group = mode = 0666 user = } user = dovecot vsz_limit = 18446744073709551615 B } service doveadm { chroot = client_limit = 1 drop_priv_before_exec = no executable = doveadm-server extra_groups = $default_internal_group group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 1 type = unix_listener doveadm-server { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service log { chroot = client_limit = 0 drop_priv_before_exec = no executable = log extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = log unix_listener log-errors { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } state_dir = /var/lib/dovecot stats_http_rawlog_dir = syslog_facility = mail userdb { args = uid=mail gid=mail home=/var/mail/%d/%n auth_verbose = default default_fields = driver = static name = override_fields = result_failure = continue result_internalfail = continue result_success = return-ok skip = never }
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
iterate_query only works with
userdb { driver = sql args = /path/to/auth-sql.conf.ext }
Aki
On 02/12/2023 00:02 EET John Stoffel john@stoffel.org wrote:
"John" == John Stoffel john@stoffel.org writes:
Do I think I'm on the right track here, since I removed the following from /etc/dovecot/conf.d/auth-sql.conf.ext
#userdb { # driver = static # args = uid=mail gid=mail home=/var/mail/%d/%n #}
So now my error is as follows:
# doveadm user -u '*' Error: auth-master: userdb list: User listing returned failure Fatal: user listing failed
Because now when I restart dovecot, I see the following in the log:
Dec 01 16:55:14 master: Info: Dovecot v2.3.21 (47349e2482) starting up for imap, lmtp, sieve (core dumps disabled) Dec 01 16:55:14 auth: Warning: sql: Ignoring changed iterate_query in /etc/dovecot/dovecot-sql.conf.ext, because userdb sql not used. (If this is intentional, set userdb_warning_disable=yes) Dec 01 16:55:14 auth: Error: auth-master client: Trying to iterate users, but userdbs don't support it (created 0 msecs ago, handshake 0 msecs ago)
So I commented out my 'iterate_query = ...' (see below) from /etc/postfix/dovecot-sql.conf.ext and now I get the error on startup which says:
Dec 01 16:57:42 master: Info: Dovecot v2.3.21 (47349e2482) starting up for imap, lmtp, sieve (core dumps disabled) Dec 01 16:57:42 auth: Error: auth-master client: Trying to iterate users, but userdbs don't support it (created 0 msecs ago, handshake 0 msecs ago) Dec 01 16:57:42 replicator: Error: auth-master: userdb list: User listing returned failure Dec 01 16:57:42 replicator: Error: listing users failed, can't replicate existing data
Which tells me I need the iteracte_users setting, but I've got a bogus query in there. So I think I should be using something like this:
iterate_query = SELECT email AS user from virtual_users;
where 'virtual_users' is the one and only table in my sqlite db file. And I'm just returning the 'email' column as 'user', since that's what it seems to expect.
Hmmm...
I've been pounding my head against the sand for a while here trying to figure out why I can't get:
doveadm user '*'
working properly. I've got a Debian 11 VPS runnig dovecot version 2.3.21-1+debian10 and it works great. But not I'm trying to add in simple replication to a home dovecot instance over a wireguard tunnel so I can do backups and have a little better resiliency. Maybe.
In any case, my sqlite schema looks like this:
sqlite> .schema virtual_users
CREATE TABLE `virtual_users` ( `id` integer NOT NULL PRIMARY KEY AUTOINCREMENT , `domain_id` integer NOT NULL , `password` varchar(106) NOT NULL , `email` varchar(100) NOT NULL , UNIQUE (`email`) , CONSTRAINT `virtual_users_ibfk_1` FOREIGN KEY (`domain_id`) REFERENCES `virtual_domains` (`id`) E ); CREATE INDEX "idx_virtual_users_domain_id" ON "virtual_users" (`domain_id`);
and I don't have any other tables. The 'domain_id' was/is a leftover from my thinking I needed it for extra testing of other domains and such.
I can do 'doveadm user john@stoffel.org' and it works just fine. When I do "doveadm user '*'" it fails and I get:
doveadm user '*' Error: auth-master: userdb list: User listing returned failure Fatal: user listing failed
So my config looks like this:
root@mail:/etc/dovecot/conf.d# cat auth-sql.conf.ext # Authentication for SQL users. Included from 10-auth.conf. # #
passdb { driver = sql
# Path for SQL configuration file, see example-config/dovecot-sql.conf.ext args = /etc/dovecot/dovecot-sql.conf.ext
}
userdb { driver = static args = uid=mail gid=mail home=/var/mail/%d/%n }
My /etc/dovecot/dovecot-sql.conf.ext has the following:
driver = sqlite connect = /etc/dovecot/private/virtual_users.sqlite3
default_pass_scheme = SHA512-CRYPT
password_query = SELECT '/var/mail/%d/%u' AS userdb_home, 'mail' AS userdb_uid, 'mail' AS userdb_gid, email as user, password FROM virtual_users WHERE email='%u';
iterate_query = SELECT email AS user from virtual_users;
And my general doveadm config output is this, slightly edited down to remove stuff I don't think I need to show is at the end. Any hints on what I've done wrong here? Do I need a more complete sqlite3 schema? I wish I could get more debugging info on what query it's trying to run and the error(s) it's getting.
Thanks, John
# 2.3.21 (47349e2482): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.21 (f6cd4b8e) # OS: Linux 5.10.0-26-amd64 x86_64 Debian 11.8 ext4 # Hostname: localhost # NOTE: Send doveconf -n output instead when asking for help. auth_anonymous_username = anonymous auth_cache_negative_ttl = 1 hours auth_cache_size = 0 auth_cache_ttl = 1 hours auth_cache_verify_password_with_worker = no auth_debug = no auth_debug_passwords = no auth_failure_delay = 2 secs auth_gssapi_hostname = auth_krb5_keytab = auth_master_user_separator = auth_mechanisms = plain login auth_policy_check_after_auth = yes auth_policy_check_before_auth = yes auth_policy_hash_mech = sha256 auth_policy_hash_nonce = auth_policy_hash_truncate = 12 auth_policy_log_only = no auth_policy_reject_on_fail = no auth_policy_report_after_auth = yes auth_policy_request_attributes = login=%{requested_username} pwhash=%{hashed_password} remote=%{rip} device_id=%{client_id} protocol=%s session_id=%{session} auth_policy_server_api_header = auth_policy_server_timeout_msecs = 2000 auth_policy_server_url = auth_proxy_self = auth_realms = auth_socket_path = auth-userdb auth_ssl_require_client_cert = no auth_ssl_username_from_cert = no auth_stats = no auth_use_winbind = no auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@ auth_username_format = %Lu auth_username_translation = auth_verbose = no auth_verbose_passwords = no auth_winbind_helper_path = /usr/bin/ntlm_auth auth_worker_max_count = 30 base_dir = /run/dovecot config_cache_size = 1 M debug_log_path = default_client_limit = 1000 default_idle_kill = 1 mins default_internal_group = dovecot default_internal_user = dovecot default_login_user = dovenull default_process_limit = 100 default_vsz_limit = 256 M deliver_log_format = msgid=%m: %$ dict_db_config = disable_plaintext_auth = yes dotlock_use_excl = yes doveadm_allowed_commands = doveadm_api_key = doveadm_http_rawlog_dir = doveadm_password = doveadm_port = 0 doveadm_socket_path = doveadm-server doveadm_ssl = no doveadm_username = doveadm doveadm_worker_count = 0 first_valid_gid = 1 first_valid_uid = 0 import_environment = TZ CORE_OUTOFMEM CORE_ERROR LISTEN_PID LISTEN_FDS NOTIFY_SOCKET info_log_path = libexec_dir = /usr/lib/dovecot listen = * log_core_filter = log_debug = log_path = /var/log/dovecot.log log_timestamp = "%b %d %H:%M:%S " mail_access_groups = mail_always_cache_fields = mail_attachment_detection_options = mail_attachment_dir = mail_attachment_fs = sis posix mail_attachment_hash = %{sha1} mail_attachment_min_size = 128 k mail_attribute_dict = mail_cache_fields = flags mail_chroot = mail_debug = no mail_fsync = optimized mail_full_filesystem_access = no mail_gid = mail_home = mail_location = maildir:/var/mail/%d/%n/Maildir mail_log_prefix = "%s(%u)<%{pid}><%{session}>: " mail_max_keyword_length = 50 mail_max_lock_timeout = 0 mail_max_userip_connections = 10 mail_never_cache_fields = imap.envelope mail_nfs_index = no mail_nfs_storage = no mail_plugin_dir = /usr/lib/dovecot/modules mail_plugins = " notify replication" mail_prefetch_count = 0 mail_privileged_group = mail mail_save_crlf = no mail_server_admin = mail_server_comment = mail_shared_explicit_inbox = no mail_sort_max_read_count = 0 mail_temp_dir = /tmp mail_temp_scan_interval = 1 weeks mail_uid = mail_vsize_bg_after_count = 0 passdb { args = /etc/dovecot/dovecot-sql.conf.ext auth_verbose = default default_fields = deny = no driver = sql master = no mechanisms = name = override_fields = pass = no result_failure = continue result_internalfail = continue result_success = return-ok skip = never username_filter = } protocols = imap lmtp sieve recipient_delimiter = +_ service auth-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = auth -w extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = worker unix_listener auth-worker { group = mode = 0600 user = $default_internal_user } user = mail vsz_limit = 18446744073709551615 B } service auth { chroot = client_limit = 0 drop_priv_before_exec = no executable = auth extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-client { group = mode = 0600 user = $default_internal_user } unix_listener auth-login { group = mode = 0600 user = $default_internal_user } unix_listener auth-master { group = mode = 0600 user = } unix_listener auth-userdb { group = mode = 0666 user = mail } unix_listener login/login { group = mode = 0666 user = } unix_listener token-login/tokenlogin { group = mode = 0666 user = } user = dovecot vsz_limit = 18446744073709551615 B } service doveadm { chroot = client_limit = 1 drop_priv_before_exec = no executable = doveadm-server extra_groups = $default_internal_group group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 1 type = unix_listener doveadm-server { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service log { chroot = client_limit = 0 drop_priv_before_exec = no executable = log extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = log unix_listener log-errors { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } state_dir = /var/lib/dovecot stats_http_rawlog_dir = syslog_facility = mail userdb { args = uid=mail gid=mail home=/var/mail/%d/%n auth_verbose = default default_fields = driver = static name = override_fields = result_failure = continue result_internalfail = continue result_success = return-ok skip = never }
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
"Aki" == Aki Tuomi aki.tuomi@open-xchange.com writes:
iterate_query only works with userdb { driver = sql args = /path/to/auth-sql.conf.ext }
Thanks, that was the key part I was missing! Can you maybe think to update the code to give a more useful error message, or even a warning on startup which says something like: 'static driver does not support iterating users'? I tried looking at the source code, but it's going to take me quite a while to wrap my brain around how it's structured and how error messages propogate.
Even just listing which userdb block failed would be a help, since you can have multiple ones defined.
In any case, I've got it working now once I updated both areas in my configuration which referred to the 'static' driver.
Thanks, really appreciate your help and all the work you guys do on this software!
John
On 02/12/2023 00:02 EET John Stoffel john@stoffel.org wrote:
> "John" == John Stoffel john@stoffel.org writes:
Do I think I'm on the right track here, since I removed the following from /etc/dovecot/conf.d/auth-sql.conf.ext
#userdb { # driver = static # args = uid=mail gid=mail home=/var/mail/%d/%n #}
So now my error is as follows:
# doveadm user -u '*' Error: auth-master: userdb list: User listing returned failure Fatal: user listing failed
Because now when I restart dovecot, I see the following in the log:
Dec 01 16:55:14 master: Info: Dovecot v2.3.21 (47349e2482) starting up for imap, lmtp, sieve (core dumps disabled) Dec 01 16:55:14 auth: Warning: sql: Ignoring changed iterate_query in /etc/dovecot/dovecot-sql.conf.ext, because userdb sql not used. (If this is intentional, set userdb_warning_disable=yes) Dec 01 16:55:14 auth: Error: auth-master client: Trying to iterate users, but userdbs don't support it (created 0 msecs ago, handshake 0 msecs ago)
So I commented out my 'iterate_query = ...' (see below) from /etc/postfix/dovecot-sql.conf.ext and now I get the error on startup which says:
Dec 01 16:57:42 master: Info: Dovecot v2.3.21 (47349e2482) starting up for imap, lmtp, sieve (core dumps disabled) Dec 01 16:57:42 auth: Error: auth-master client: Trying to iterate users, but userdbs don't support it (created 0 msecs ago, handshake 0 msecs ago) Dec 01 16:57:42 replicator: Error: auth-master: userdb list: User listing returned failure Dec 01 16:57:42 replicator: Error: listing users failed, can't replicate existing data
Which tells me I need the iteracte_users setting, but I've got a bogus query in there. So I think I should be using something like this:
iterate_query = SELECT email AS user from virtual_users;
where 'virtual_users' is the one and only table in my sqlite db file. And I'm just returning the 'email' column as 'user', since that's what it seems to expect.
Hmmm...
I've been pounding my head against the sand for a while here trying to figure out why I can't get:
doveadm user '*'
working properly. I've got a Debian 11 VPS runnig dovecot version 2.3.21-1+debian10 and it works great. But not I'm trying to add in simple replication to a home dovecot instance over a wireguard tunnel so I can do backups and have a little better resiliency. Maybe.
In any case, my sqlite schema looks like this:
sqlite> .schema virtual_users
CREATE TABLE `virtual_users` ( `id` integer NOT NULL PRIMARY KEY AUTOINCREMENT , `domain_id` integer NOT NULL , `password` varchar(106) NOT NULL , `email` varchar(100) NOT NULL , UNIQUE (`email`) , CONSTRAINT `virtual_users_ibfk_1` FOREIGN KEY (`domain_id`) REFERENCES `virtual_domains` (`id`) E ); CREATE INDEX "idx_virtual_users_domain_id" ON "virtual_users" (`domain_id`);
and I don't have any other tables. The 'domain_id' was/is a leftover from my thinking I needed it for extra testing of other domains and such.
I can do 'doveadm user john@stoffel.org' and it works just fine. When I do "doveadm user '*'" it fails and I get:
doveadm user '*' Error: auth-master: userdb list: User listing returned failure Fatal: user listing failed
So my config looks like this:
root@mail:/etc/dovecot/conf.d# cat auth-sql.conf.ext # Authentication for SQL users. Included from 10-auth.conf. # #
passdb { driver = sql
# Path for SQL configuration file, see example-config/dovecot-sql.conf.ext args = /etc/dovecot/dovecot-sql.conf.ext
}
userdb { driver = static args = uid=mail gid=mail home=/var/mail/%d/%n }
My /etc/dovecot/dovecot-sql.conf.ext has the following:
driver = sqlite connect = /etc/dovecot/private/virtual_users.sqlite3
default_pass_scheme = SHA512-CRYPT
password_query = SELECT '/var/mail/%d/%u' AS userdb_home, 'mail' AS userdb_uid, 'mail' AS userdb_gid, email as user, password FROM virtual_users WHERE email='%u';
iterate_query = SELECT email AS user from virtual_users;
And my general doveadm config output is this, slightly edited down to remove stuff I don't think I need to show is at the end. Any hints on what I've done wrong here? Do I need a more complete sqlite3 schema? I wish I could get more debugging info on what query it's trying to run and the error(s) it's getting.
Thanks, John
# 2.3.21 (47349e2482): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.21 (f6cd4b8e) # OS: Linux 5.10.0-26-amd64 x86_64 Debian 11.8 ext4 # Hostname: localhost # NOTE: Send doveconf -n output instead when asking for help. auth_anonymous_username = anonymous auth_cache_negative_ttl = 1 hours auth_cache_size = 0 auth_cache_ttl = 1 hours auth_cache_verify_password_with_worker = no auth_debug = no auth_debug_passwords = no auth_failure_delay = 2 secs auth_gssapi_hostname = auth_krb5_keytab = auth_master_user_separator = auth_mechanisms = plain login auth_policy_check_after_auth = yes auth_policy_check_before_auth = yes auth_policy_hash_mech = sha256 auth_policy_hash_nonce = auth_policy_hash_truncate = 12 auth_policy_log_only = no auth_policy_reject_on_fail = no auth_policy_report_after_auth = yes auth_policy_request_attributes = login=%{requested_username} pwhash=%{hashed_password} remote=%{rip} device_id=%{client_id} protocol=%s session_id=%{session} auth_policy_server_api_header = auth_policy_server_timeout_msecs = 2000 auth_policy_server_url = auth_proxy_self = auth_realms = auth_socket_path = auth-userdb auth_ssl_require_client_cert = no auth_ssl_username_from_cert = no auth_stats = no auth_use_winbind = no auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@ auth_username_format = %Lu auth_username_translation = auth_verbose = no auth_verbose_passwords = no auth_winbind_helper_path = /usr/bin/ntlm_auth auth_worker_max_count = 30 base_dir = /run/dovecot config_cache_size = 1 M debug_log_path = default_client_limit = 1000 default_idle_kill = 1 mins default_internal_group = dovecot default_internal_user = dovecot default_login_user = dovenull default_process_limit = 100 default_vsz_limit = 256 M deliver_log_format = msgid=%m: %$ dict_db_config = disable_plaintext_auth = yes dotlock_use_excl = yes doveadm_allowed_commands = doveadm_api_key = doveadm_http_rawlog_dir = doveadm_password = doveadm_port = 0 doveadm_socket_path = doveadm-server doveadm_ssl = no doveadm_username = doveadm doveadm_worker_count = 0 first_valid_gid = 1 first_valid_uid = 0 import_environment = TZ CORE_OUTOFMEM CORE_ERROR LISTEN_PID LISTEN_FDS NOTIFY_SOCKET info_log_path = libexec_dir = /usr/lib/dovecot listen = * log_core_filter = log_debug = log_path = /var/log/dovecot.log log_timestamp = "%b %d %H:%M:%S " mail_access_groups = mail_always_cache_fields = mail_attachment_detection_options = mail_attachment_dir = mail_attachment_fs = sis posix mail_attachment_hash = %{sha1} mail_attachment_min_size = 128 k mail_attribute_dict = mail_cache_fields = flags mail_chroot = mail_debug = no mail_fsync = optimized mail_full_filesystem_access = no mail_gid = mail_home = mail_location = maildir:/var/mail/%d/%n/Maildir mail_log_prefix = "%s(%u)<%{pid}><%{session}>: " mail_max_keyword_length = 50 mail_max_lock_timeout = 0 mail_max_userip_connections = 10 mail_never_cache_fields = imap.envelope mail_nfs_index = no mail_nfs_storage = no mail_plugin_dir = /usr/lib/dovecot/modules mail_plugins = " notify replication" mail_prefetch_count = 0 mail_privileged_group = mail mail_save_crlf = no mail_server_admin = mail_server_comment = mail_shared_explicit_inbox = no mail_sort_max_read_count = 0 mail_temp_dir = /tmp mail_temp_scan_interval = 1 weeks mail_uid = mail_vsize_bg_after_count = 0 passdb { args = /etc/dovecot/dovecot-sql.conf.ext auth_verbose = default default_fields = deny = no driver = sql master = no mechanisms = name = override_fields = pass = no result_failure = continue result_internalfail = continue result_success = return-ok skip = never username_filter = } protocols = imap lmtp sieve recipient_delimiter = +_ service auth-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = auth -w extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = worker unix_listener auth-worker { group = mode = 0600 user = $default_internal_user } user = mail vsz_limit = 18446744073709551615 B } service auth { chroot = client_limit = 0 drop_priv_before_exec = no executable = auth extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-client { group = mode = 0600 user = $default_internal_user } unix_listener auth-login { group = mode = 0600 user = $default_internal_user } unix_listener auth-master { group = mode = 0600 user = } unix_listener auth-userdb { group = mode = 0666 user = mail } unix_listener login/login { group = mode = 0666 user = } unix_listener token-login/tokenlogin { group = mode = 0666 user = } user = dovecot vsz_limit = 18446744073709551615 B } service doveadm { chroot = client_limit = 1 drop_priv_before_exec = no executable = doveadm-server extra_groups = $default_internal_group group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 1 type = unix_listener doveadm-server { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service log { chroot = client_limit = 0 drop_priv_before_exec = no executable = log extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = log unix_listener log-errors { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } state_dir = /var/lib/dovecot stats_http_rawlog_dir = syslog_facility = mail userdb { args = uid=mail gid=mail home=/var/mail/%d/%n auth_verbose = default default_fields = driver = static name = override_fields = result_failure = continue result_internalfail = continue result_success = return-ok skip = never }
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
"Aki" == Aki Tuomi via dovecot dovecot@dovecot.org writes:
That did the trick. Thanks! Maybe the docs can be updated to make this crystal clear? Especially the section which talks about using the static driver with userdb and how it doesn't allow the use of interate_query.
iterate_query only works with userdb { driver = sql args = /path/to/auth-sql.conf.ext }
Aki
On 02/12/2023 00:02 EET John Stoffel john@stoffel.org wrote:
> "John" == John Stoffel john@stoffel.org writes:
Do I think I'm on the right track here, since I removed the following from /etc/dovecot/conf.d/auth-sql.conf.ext
#userdb { # driver = static # args = uid=mail gid=mail home=/var/mail/%d/%n #}
So now my error is as follows:
# doveadm user -u '*' Error: auth-master: userdb list: User listing returned failure Fatal: user listing failed
Because now when I restart dovecot, I see the following in the log:
Dec 01 16:55:14 master: Info: Dovecot v2.3.21 (47349e2482) starting up for imap, lmtp, sieve (core dumps disabled) Dec 01 16:55:14 auth: Warning: sql: Ignoring changed iterate_query in /etc/dovecot/dovecot-sql.conf.ext, because userdb sql not used. (If this is intentional, set userdb_warning_disable=yes) Dec 01 16:55:14 auth: Error: auth-master client: Trying to iterate users, but userdbs don't support it (created 0 msecs ago, handshake 0 msecs ago)
So I commented out my 'iterate_query = ...' (see below) from /etc/postfix/dovecot-sql.conf.ext and now I get the error on startup which says:
Dec 01 16:57:42 master: Info: Dovecot v2.3.21 (47349e2482) starting up for imap, lmtp, sieve (core dumps disabled) Dec 01 16:57:42 auth: Error: auth-master client: Trying to iterate users, but userdbs don't support it (created 0 msecs ago, handshake 0 msecs ago) Dec 01 16:57:42 replicator: Error: auth-master: userdb list: User listing returned failure Dec 01 16:57:42 replicator: Error: listing users failed, can't replicate existing data
Which tells me I need the iteracte_users setting, but I've got a bogus query in there. So I think I should be using something like this:
iterate_query = SELECT email AS user from virtual_users;
where 'virtual_users' is the one and only table in my sqlite db file. And I'm just returning the 'email' column as 'user', since that's what it seems to expect.
Hmmm...
I've been pounding my head against the sand for a while here trying to figure out why I can't get:
doveadm user '*'
working properly. I've got a Debian 11 VPS runnig dovecot version 2.3.21-1+debian10 and it works great. But not I'm trying to add in simple replication to a home dovecot instance over a wireguard tunnel so I can do backups and have a little better resiliency. Maybe.
In any case, my sqlite schema looks like this:
sqlite> .schema virtual_users
CREATE TABLE `virtual_users` ( `id` integer NOT NULL PRIMARY KEY AUTOINCREMENT , `domain_id` integer NOT NULL , `password` varchar(106) NOT NULL , `email` varchar(100) NOT NULL , UNIQUE (`email`) , CONSTRAINT `virtual_users_ibfk_1` FOREIGN KEY (`domain_id`) REFERENCES `virtual_domains` (`id`) E ); CREATE INDEX "idx_virtual_users_domain_id" ON "virtual_users" (`domain_id`);
and I don't have any other tables. The 'domain_id' was/is a leftover from my thinking I needed it for extra testing of other domains and such.
I can do 'doveadm user john@stoffel.org' and it works just fine. When I do "doveadm user '*'" it fails and I get:
doveadm user '*' Error: auth-master: userdb list: User listing returned failure Fatal: user listing failed
So my config looks like this:
root@mail:/etc/dovecot/conf.d# cat auth-sql.conf.ext # Authentication for SQL users. Included from 10-auth.conf. # #
passdb { driver = sql
# Path for SQL configuration file, see example-config/dovecot-sql.conf.ext args = /etc/dovecot/dovecot-sql.conf.ext
}
userdb { driver = static args = uid=mail gid=mail home=/var/mail/%d/%n }
My /etc/dovecot/dovecot-sql.conf.ext has the following:
driver = sqlite connect = /etc/dovecot/private/virtual_users.sqlite3
default_pass_scheme = SHA512-CRYPT
password_query = SELECT '/var/mail/%d/%u' AS userdb_home, 'mail' AS userdb_uid, 'mail' AS userdb_gid, email as user, password FROM virtual_users WHERE email='%u';
iterate_query = SELECT email AS user from virtual_users;
And my general doveadm config output is this, slightly edited down to remove stuff I don't think I need to show is at the end. Any hints on what I've done wrong here? Do I need a more complete sqlite3 schema? I wish I could get more debugging info on what query it's trying to run and the error(s) it's getting.
Thanks, John
# 2.3.21 (47349e2482): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.21 (f6cd4b8e) # OS: Linux 5.10.0-26-amd64 x86_64 Debian 11.8 ext4 # Hostname: localhost # NOTE: Send doveconf -n output instead when asking for help. auth_anonymous_username = anonymous auth_cache_negative_ttl = 1 hours auth_cache_size = 0 auth_cache_ttl = 1 hours auth_cache_verify_password_with_worker = no auth_debug = no auth_debug_passwords = no auth_failure_delay = 2 secs auth_gssapi_hostname = auth_krb5_keytab = auth_master_user_separator = auth_mechanisms = plain login auth_policy_check_after_auth = yes auth_policy_check_before_auth = yes auth_policy_hash_mech = sha256 auth_policy_hash_nonce = auth_policy_hash_truncate = 12 auth_policy_log_only = no auth_policy_reject_on_fail = no auth_policy_report_after_auth = yes auth_policy_request_attributes = login=%{requested_username} pwhash=%{hashed_password} remote=%{rip} device_id=%{client_id} protocol=%s session_id=%{session} auth_policy_server_api_header = auth_policy_server_timeout_msecs = 2000 auth_policy_server_url = auth_proxy_self = auth_realms = auth_socket_path = auth-userdb auth_ssl_require_client_cert = no auth_ssl_username_from_cert = no auth_stats = no auth_use_winbind = no auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@ auth_username_format = %Lu auth_username_translation = auth_verbose = no auth_verbose_passwords = no auth_winbind_helper_path = /usr/bin/ntlm_auth auth_worker_max_count = 30 base_dir = /run/dovecot config_cache_size = 1 M debug_log_path = default_client_limit = 1000 default_idle_kill = 1 mins default_internal_group = dovecot default_internal_user = dovecot default_login_user = dovenull default_process_limit = 100 default_vsz_limit = 256 M deliver_log_format = msgid=%m: %$ dict_db_config = disable_plaintext_auth = yes dotlock_use_excl = yes doveadm_allowed_commands = doveadm_api_key = doveadm_http_rawlog_dir = doveadm_password = doveadm_port = 0 doveadm_socket_path = doveadm-server doveadm_ssl = no doveadm_username = doveadm doveadm_worker_count = 0 first_valid_gid = 1 first_valid_uid = 0 import_environment = TZ CORE_OUTOFMEM CORE_ERROR LISTEN_PID LISTEN_FDS NOTIFY_SOCKET info_log_path = libexec_dir = /usr/lib/dovecot listen = * log_core_filter = log_debug = log_path = /var/log/dovecot.log log_timestamp = "%b %d %H:%M:%S " mail_access_groups = mail_always_cache_fields = mail_attachment_detection_options = mail_attachment_dir = mail_attachment_fs = sis posix mail_attachment_hash = %{sha1} mail_attachment_min_size = 128 k mail_attribute_dict = mail_cache_fields = flags mail_chroot = mail_debug = no mail_fsync = optimized mail_full_filesystem_access = no mail_gid = mail_home = mail_location = maildir:/var/mail/%d/%n/Maildir mail_log_prefix = "%s(%u)<%{pid}><%{session}>: " mail_max_keyword_length = 50 mail_max_lock_timeout = 0 mail_max_userip_connections = 10 mail_never_cache_fields = imap.envelope mail_nfs_index = no mail_nfs_storage = no mail_plugin_dir = /usr/lib/dovecot/modules mail_plugins = " notify replication" mail_prefetch_count = 0 mail_privileged_group = mail mail_save_crlf = no mail_server_admin = mail_server_comment = mail_shared_explicit_inbox = no mail_sort_max_read_count = 0 mail_temp_dir = /tmp mail_temp_scan_interval = 1 weeks mail_uid = mail_vsize_bg_after_count = 0 passdb { args = /etc/dovecot/dovecot-sql.conf.ext auth_verbose = default default_fields = deny = no driver = sql master = no mechanisms = name = override_fields = pass = no result_failure = continue result_internalfail = continue result_success = return-ok skip = never username_filter = } protocols = imap lmtp sieve recipient_delimiter = +_ service auth-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = auth -w extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = worker unix_listener auth-worker { group = mode = 0600 user = $default_internal_user } user = mail vsz_limit = 18446744073709551615 B } service auth { chroot = client_limit = 0 drop_priv_before_exec = no executable = auth extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-client { group = mode = 0600 user = $default_internal_user } unix_listener auth-login { group = mode = 0600 user = $default_internal_user } unix_listener auth-master { group = mode = 0600 user = } unix_listener auth-userdb { group = mode = 0666 user = mail } unix_listener login/login { group = mode = 0666 user = } unix_listener token-login/tokenlogin { group = mode = 0666 user = } user = dovecot vsz_limit = 18446744073709551615 B } service doveadm { chroot = client_limit = 1 drop_priv_before_exec = no executable = doveadm-server extra_groups = $default_internal_group group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 1 type = unix_listener doveadm-server { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service log { chroot = client_limit = 0 drop_priv_before_exec = no executable = log extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = log unix_listener log-errors { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } state_dir = /var/lib/dovecot stats_http_rawlog_dir = syslog_facility = mail userdb { args = uid=mail gid=mail home=/var/mail/%d/%n auth_verbose = default default_fields = driver = static name = override_fields = result_failure = continue result_internalfail = continue result_success = return-ok skip = never }
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
participants (6)
-
Aki Tuomi
-
Gerben Wierda
-
John Stoffel
-
justina colmena ~biz
-
Jörg Schulz
-
Paul Kudla