- dovecot - dovecot.org

Fwd: Some mails do not get replicated anymore after memory-exhaust
by Christoph Kluge 02 Mar '17

02 Mar '17

Hey guys, overall I have an working dovecot replication between 2 servers running on amazon cloud. Sadly I had some messages that my server ran out of memory. After investigating a little bit further I realized that some mails didn't got replicated, but I'm not sure if this was related to the memory exhaust. I was expecting that the full-sync would catch them up but sadly it's not. Attached I'm adding: * /etc/dovecot/dovecot.conf from both servers * one sample of my memory-exhaust exception * maildir directory listing of one mailbox on both servers * commands + outpot of manual attempt for full-replication * grep information of missing mail inside Maildir on both servers Here is my configuration from both servers. The configugration is 1:1 the same except the mail_replica server. Please note one server runs on debian 8.7 and the other one on 7.11. ---- SERVER A > # dovecot -n > # 2.2.13: /etc/dovecot/dovecot.conf > # OS: Linux 3.2.0-4-amd64 x86_64 Debian 8.7 > ---- SERVER B > # dovecot -n > # 2.2.13: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.32-34-pve i686 Debian 7.11 > auth_mechanisms = plain login > disable_plaintext_auth = no > doveadm_password = **** > doveadm_port = 12345 > listen = *,[::] > log_timestamp = "%Y-%m-%d %H:%M:%S " > mail_max_userip_connections = 100 > mail_plugins = notify replication quota > mail_privileged_group = vmail > passdb { > args = /etc/dovecot/dovecot-sql.conf > driver = sql > } > plugin { > mail_replica = tcp:*.****.de > quota = dict:user::file:/var/vmail/%d/%n/.quotausage > replication_full_sync_interval = 1 hours > sieve = /var/vmail/%d/%n/.sieve > sieve_max_redirects = 25 > } > protocols = imap > replication_max_conns = 2 > service aggregator { > fifo_listener replication-notify-fifo { > mode = 0666 > user = vmail > } > unix_listener replication-notify { > mode = 0666 > user = vmail > } > } > service auth { > unix_listener /var/spool/postfix/private/auth { > group = postfix > mode = 0660 > user = postfix > } > unix_listener auth-userdb { > group = vmail > mode = 0600 > user = vmail > } > user = root > } > service config { > unix_listener config { > user = vmail > } > } > service doveadm { > inet_listener { > port = 12345 > } > user = vmail > } > service imap-login { > client_limit = 1000 > process_limit = 512 > } > service lmtp { > unix_listener /var/spool/postfix/private/dovecot-lmtp { > group = postfix > mode = 0600 > user = postfix > } > } > service replicator { > process_min_avail = 1 > unix_listener replicator-doveadm { > mode = 0666 > } > } > ssl_cert = </etc/postfix/smtpd.cert > ssl_key = </etc/postfix/smtpd.key > ssl_protocols = !SSLv2 !SSLv3 > userdb { > driver = prefetch > } > userdb { > args = /etc/dovecot/dovecot-sql.conf > driver = sql > } > protocol imap { > mail_plugins = notify replication quota imap_quota > } > protocol pop3 { > mail_plugins = quota > pop3_uidl_format = %08Xu%08Xv > } > protocol lda { > mail_plugins = notify replication quota sieve > postmaster_address = webmaster@localhost > } > protocol lmtp { > mail_plugins = notify replication quota sieve > postmaster_address = webmaster@localhost > } This is the exception which I got several times: Feb 26 16:16:39 mx dovecot: replicator: Panic: data stack: Out of memory > when allocating 268435496 bytes > Feb 26 16:16:39 mx dovecot: replicator: Error: Raw backtrace: > /usr/lib/dovecot/libdovecot.so.0(+0x6b6fe) [0x7f7ca2b0a6fe] -> > /usr/lib/dovecot/libdovecot.so.0(+0x6b7ec) [0x7f7ca2b0a7ec] -> > /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f7ca2ac18fb] -> > /usr/lib/dovecot/libdovecot.so.0(+0x6977e) [0x7f7ca2b0877e] -> > /usr/lib/dovecot/libdovecot.so.0(+0x699db) [0x7f7ca2b089db] -> > /usr/lib/dovecot/libdovecot.so.0(+0x82198) [0x7f7ca2b21198] -> > /usr/lib/dovecot/libdovecot.so.0(+0x6776d) [0x7f7ca2b0676d] -> > /usr/lib/dovecot/libdovecot.so.0(buffer_write+0x6c) [0x7f7ca2b069dc] -> > dovecot/replicator(replicator_queue_push+0x14e) [0x7f7ca2fa17ae] -> > dovecot/replicator(+0x4f9e) [0x7f7ca2fa0f9e] -> dovecot/replicator(+0x4618) > [0x7f7ca2fa0618] -> dovecot/replicator(+0x4805) [0x7f7ca2fa0805] -> > /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x3f) [0x7f7ca2b1bd0f] > -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xf9) > [0x7f7ca2b1cd09] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9) > [0x7f7ca2b1bd79] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) > [0x7f7ca2b1bdf8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) > [0x7f7ca2ac6dc3] -> dovecot/replicator(main+0x195) [0x7f7ca2f9f8b5] -> > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f7ca2715b45] > -> dovecot/replicator(+0x395d) [0x7f7ca2f9f95d] > Feb 26 16:16:39 mx dovecot: imap(***.com): Warning: replication(***.com): > Sync failure: > Feb 26 16:16:39 mx dovecot: replicator: Fatal: master: > service(replicator): child 24012 killed with signal 6 (core dumps disabled) This is the current maildir listing on Server A # ls -la /var/vmail/*.eu/*h/Maildir/new/ > total 24 > drwx------ 2 vmail vmail 4096 Feb 27 18:12 . > drwx------ 15 vmail vmail 4096 Feb 27 21:47 .. > -rw------- 1 vmail vmail 3600 Feb 27 14:49 1488206976.M277562P25620.mail, > S=3600,W=3671 > -rw------- 1 vmail vmail 4390 Feb 27 15:17 1488208642.M513542P27111.mail, > S=4390,W=4478:2,S > -rw------- 1 vmail vmail 3577 Feb 27 16:32 1488213157.M307300P30773.mail, > S=3577,W=3648:2,S This is the current maildir listing on Server B # ls -la /var/vmail/*.eu/*h/Maildir/new/ > total 16 > drwx------ 2 vmail vmail 12288 Feb 27 16:45 . > drwx------ 15 vmail vmail 4096 Feb 27 21:47 .. This is how I tried to manually sync it doveadm -v sync -u *h(a)*.eu -f tcp:mx.***.de:12345 This is the users sync status # doveadm replicator status 'cheecoh(a)ragequit.eu' > username priority fast sync full sync failed > *h(a)*.eu none 00:24:47 10:57:04 - Then I tried to lookup for the mail-id which is also the same on both servers # grep -ri "M277562P25620" /var/vmail/*.eu/*h/ > /var/vmail/*.eu/*h/Maildir/dovecot-uidlist:493 :1488206976.M277562P25620. > mail,S=3600,W=3671 I have no idea what else I could do. I could also pass a "doveadm -Dv sync" output but this one is really huge.. Best Regards Christoph Kluge

1 1

dovecot-lda crash after upgrade to 2.2.28
by Max Kostikov 01 Mar '17

01 Mar '17

Hi! I posted this problem few days ago in FreeBSD bugtracker https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217364 So, the problem is in crash of dovecot-lda client while local mailbox delivery with this message in log > dovecot: lda(my@email): Panic: file mail-namespace.c: line 709 > (mail_namespace_find): assertion failed: (ns != NULL) My Dovecot config stills the same from previous 2.2.27 installation to I believe that problem in new Dovecot code related namespace. System: > # uname -v > FreeBSD 11.0-RELEASE-p8 #0: Wed Feb 22 06:12:04 UTC 2017 > root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC Options: > root@beta:/usr/ports/mail/dovecot2 # make showconfig | grep =on > DOCS=on: Build and/or install documentation > EXAMPLES=on: Build and/or install examples > KQUEUE=on: kqueue(2) support > LIBWRAP=on: TCP wrapper support > LZ4=on: LZ4 compression support > MYSQL=on: MySQL database support > GSSAPI_BASE=on: Use GSSAPI from base Config: # doveconf -n # 2.2.27 (c0f36b0): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: FreeBSD 11.0-RELEASE-p8 amd64 ufs auth_mechanisms = plain login default_login_user = dovecot dict { sqluserquota = mysql:/usr/local/etc/dovecot/dovecot-dict-sql-user.conf } disable_plaintext_auth = no first_valid_gid = 0 first_valid_uid = 25 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes login_access_sockets = tcpwrap login_greeting = Dovecot ready! login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c mail_access_groups = mail mail_gid = 6 mail_location = maildir:/var/mail/%d/%n mail_plugins = acl quota trash mail_privileged_group = mail mail_uid = 26 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { location = maildir:/var/mail/%d/.public:INDEXPVT=/var/mail/%d/%n/public prefix = public. separator = . subscriptions = no type = public } namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = . type = private } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf driver = sql } plugin { acl = vfile::cache_secs=300 antispam_backend = mailtrain antispam_mail_notspam = %u-revoke antispam_mail_sendmail = /usr/local/etc/dovecot/move-cmd.sh antispam_mail_sendmail_args = antispam_mail_spam = %u-report antispam_spam = Junk antispam_trash = Trash quota = dict:user_quota::proxy::sqluserquota quota_grace = 10%% quota_rule2 = Trash:storage=+10%% quota_rule3 = Junk:storage=+10%% quota_warning = storage=100%% quota-exceeded 100 %u quota_warning2 = storage=95%% quota-warning 95 %u quota_warning3 = storage=90%% quota-warning 90 %u quota_warning4 = storage=75%% quota-warning 75 %u sieve = ~/dovecot.sieve sieve_before = /usr/local/etc/dovecot/default.sieve sieve_dir = ~/sieve sieve_global_path = /usr/local/etc/dovecot/default.sieve trash = /usr/local/etc/dovecot/dovecot-trash.conf } protocols = imap pop3 lmtp sieve service auth { unix_listener auth-client { mode = 0660 user = mailnull } unix_listener auth-master { mode = 0660 user = mailnull } user = root } service dict { unix_listener dict { mode = 0660 user = mailnull } } service imap-login { chroot = login client_limit = 64 executable = /usr/local/libexec/dovecot/imap-login inet_listener imap { port = 143 } inet_listener imaps { port = 993 } process_limit = 32 process_min_avail = 8 service_count = 1 user = $default_login_user vsz_limit = 64 M } service imap { executable = /usr/local/libexec/dovecot/imap } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 1 service_count = 1 user = $default_login_user vsz_limit = 64 M } service managesieve { process_limit = 10 } service pop3-login { chroot = login client_limit = 64 executable = /usr/local/libexec/dovecot/pop3-login inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 } process_limit = 32 process_min_avail = 8 service_count = 1 user = $default_login_user vsz_limit = 64 M } service pop3 { executable = /usr/local/libexec/dovecot/pop3 } service quota-warning { executable = script /usr/local/etc/dovecot/quota_warning.sh unix_listener quota-warning { mode = 0660 user = mailnull } } service tcpwrap { unix_listener login/tcpwrap { mode = 0600 user = $default_login_user } } ssl_ca = </usr/local/etc/letsencrypt/live/peek.ru/chain.pem ssl_cert = </usr/local/etc/letsencrypt/live/peek.ru/cert.pem ssl_cipher_list = ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS ssl_dh_parameters_length = 2048 ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { args = /usr/local/etc/dovecot/dovecot-sql.conf driver = sql } protocol sieve { managesieve_implementation_string = dovecot managesieve_max_line_length = 65536 } protocol imap { imap_client_workarounds = delay-newmail tb-extra-mailbox-sep mail_plugins = acl quota trash antispam imap_acl imap_quota } protocol lmtp { mail_plugins = acl quota trash sieve } protocol pop3 { pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_uidl_format = %08Xu%08Xv } protocol lda { auth_socket_path = /var/run/dovecot/auth-master mail_plugins = acl quota trash sieve postmaster_address = postmaster(a)peek.ru sendmail_path = /usr/sbin/sendmail } local_name *.domain.ru { ssl_ca = </usr/local/etc/letsencrypt/live/domain.ru/chain.pem ssl_cert = </usr/local/etc/letsencrypt/live/domain.ru/cert.pem ssl_key = # hidden, use -P to show it } -- With best regards, Max Kostikov BBM: 24CA5DF8 | W: https://kostikov.co

3 11

Exclude namespace from replication
by George Kontostanos 01 Mar '17

01 Mar '17

Hello list, I am using replication with dsync and I am facing some issues with shared folders. More particularly expunged message reappearing. I have searched a bit and it was suggested that shared folders should not be synced. My namespace looks like that: namespace { inbox = no list = children location = maildir:/usr/local/vhosts/mail/%%d/%%n:LAYOUT=fs:INDEX=/usr/local/vhosts/indexes/%d/%n/shared/%%u:INDEXPVT=/usr/local/vhosts/indexes/%d/%n/shared/%%u prefix = shared/%%d/%%n/ separator = / subscriptions = no type = shared } I was wondering if defining: replication_dsync_parameters = -d -N -l 30 -U -x shared Is enough. Or do I need to change the syntax of the namespace like that: namespace shared { .... .... type = shared } Thanks for your help

1 0

Replacement for antispam plugin
by Aki Tuomi 01 Mar '17

01 Mar '17

Hi! Since antispam plugin is deprecated and we would really prefer people not to use it, we wrote instructions on how to replace it with IMAPSieve. Comments and suggestions are most welcome. https://wiki.dovecot.org/HowTo/AntispamWithSieve --- Aki Tuomi Dovecot oy

18 42

Expunged message reappeared, giving a new UID
by Γιώργος Δημακόπουλος 01 Mar '17

01 Mar '17

Dovecot 2.2.27 dsync-server(account_name): Warning: Maildir /usr/local/vhosts/mail/*******/info/Sent: Expunged message reappeared, giving a new UID (old uid=29787, file=*******) What triggers this replication for ever ?? Thank you George

2 1

Preserve environment variables for doveadm
by Teodor Milkov 01 Mar '17

01 Mar '17

Hello, I noticed there are ways to instruct various dovecot processes to preserve particular environment variables. One way is to use import_environment in dovecot.conf and yet another is to export DOVECOT_PRESERVE_ENVS variable – both can contain space delimited list of variables to preserve. I found report that this is supposed to work for dsync: http://www.dovecot.org/list/dovecot/2012-October/138659.html Unfortunately I can't it to work for "doveadm quota recalc". Is it supposed to work or am I doing something wrong? I did strace on the doveadm process and noticed it does exec() doveconf which in turn does exec() doveadm again. If I replace doveconf with a shell script which exports my environment variables then they become available to doveadm: # cat /dovecot/bin/doveconf.wrapper #!/bin/sh export LD_PRELOAD='dovecot-preload.so' exec /dovecot/bin/doveconf.orig "$@"

1 0

Expunged message reappeared, giving a new UID
by Γιώργος Δημακόπουλος 01 Mar '17

01 Mar '17

Please help. How to stop email replication is on going. dsync-server : Warning Expunged message reappeared, giving a new UID

1 0

Query about list policy?
by Andy R 01 Mar '17

01 Mar '17

Greetings for a second time today, As a quick query, what is the policy about linking to user-scripts and small addons ? I've used a couple of perl scripts to help me with dspam training from dovecot-antispam and imapsieve triggering for sorting and feeding messages to dspam in the background. They are most likely not the greatest but seem to work at least. I've posted them to github, but just wanted to ask whether it was an issue if i linked them here, or if they would need any review before posting or any other answer? Many thanks, and to Aki Tuomi and the dovecot team also. Andy R.

1 0

Quota usage value shows 140% of actual disk usage
by Umut Erol Kaçar 28 Feb '17

28 Feb '17

Hello everyone, Our server has these installed: dovecot-2.0.21-2.el6.x86_64 dovecot-pigeonhole-2.0.21-2.el6.x86_64 dovecot-mysql-2.0.21-2.el6.x86_64 ...and has been running for quite a long time, with several hundred domains and thousands of accounts on it. My colleagues reported that it's been showing quota usage values that are more than actual disk usage, so I investigated and found that Dovecot shows a quota usage value roughly around 140% of actual disk usage. It's also valid on newly created accounts. My test account for example: doveadm quota get -u test(a)example.local Quota name Type Value Limit % User quota STORAGE 4359 512000 0 User quota MESSAGE 7 - 0 du -sc /home/vmail/example.local/test/Maildir/{*,.[!.]*} 1044 /home/vmail/example.local/test/Maildir/cur 28 /home/vmail/example.local/test/Maildir/dovecot.index.cache 8 /home/vmail/example.local/test/Maildir/dovecot.index.log 4 /home/vmail/example.local/test/Maildir/dovecot.mailbox.log 4 /home/vmail/example.local/test/Maildir/dovecot-uidlist 4 /home/vmail/example.local/test/Maildir/dovecot-uidvalidity 0 /home/vmail/example.local/test/Maildir/dovecot-uidvalidity.56a4dc8e 4 /home/vmail/example.local/test/Maildir/new 4 /home/vmail/example.local/test/Maildir/subscriptions 4 /home/vmail/example.local/test/Maildir/tmp 24 /home/vmail/example.local/test/Maildir/.Junk 1932 /home/vmail/example.local/test/Maildir/.Sent 44 /home/vmail/example.local/test/Maildir/.Trash 3104 total 4359/3104=1,40431701 So it shows roughly around 1,4*actualDiskUsage. The ratio is mostly the same for almost all the other accounts. It can vary between like 1,3-1,6. So, the gap gets insane when more disk space is used, say like with 2GB disk usage, Dovecot thinks 3,5GB quota is used... dovecot quota recalc does not fix the issue, it only sets the same value again (I've checked with tcpdump and saw the query with the same quota usage value). The method is Dictionary quota with SQL. I'm attaching the dovecot -n output with some other config files. I've tried setting messages and bytes value to -1 on the MariaDB database to force recalculation. But as soon as I run doveadm quota recalc, it gets the same wrong value again. What can I do to fix this? Thanks in advance.

3 3

Index queue
by azurit＠pobox.sk 28 Feb '17

28 Feb '17

Hi, can i, somehow, list mailboxes which are to be indexed by indexer-worker (=index queue?) ? How can i know what part of all mailboxes was indexed so far? Are there any statistics about Solr data dir size based on emails amount? For example, we have about 5TB of emails, what should i except about index size in Solr? Thank you. azur

1 0