April 2017 - dovecot - dovecot.org

System load spike on dovecot reload
by dave＠evilcigi.eu 24 Apr '17

24 Apr '17

Hi everyone, I'm running dovecot with quite a lot of users and lots of active imap connections (like 20'000). I'm using different user IDs for users, so I need to have imap {service_count=1} - i.e. I have a lots of imap processes running. Everything works fine, until I reload dovecot configuration. When that happen, every client is forced to relogin in the same time and that causes a huge system load spike (2-3000 5 min load). I was thinking that it would be great, if dovecot wouldn't kick all the users in the same time during reload, but somehow gradually, during specified interval. I'm aware of the shutdown_clients directive that could help, but I don't like it - I do want the clients get disconnected on dovecot shutdown and also I want them to relogin in reasonably short time after reload. Is something like that possible with dovecot or does it make sense to implement that in the future versions? Thank you. Dave.

5 4

Feature Request - Director Balance
by Webert de Souza Lima 24 Apr '17

24 Apr '17

Hi, often I run into the situation where a dovecot server goes down for maintenance, and all users get concentrated in the remaining dovecot server (considering I have 2 dovecot servers only). When that dovecot server comes back online, director server will send new users to it, but the dovecot server that was up all the time will still have tons of clients mapped to it. I suggest the director servers to always try to balance load between servers, in the way: - if a server has several more connections than other, mark it to re-balance - when a user connected to this loaded server disconnects, map it to another server (that is per definition not the same server) immediately. that way it would gracefully re-balance, not killing existing connections, just waiting for them to finish. Thank you for your time. Webert Lima MAV Tecnologia Belo Horizonte, Brasil.

2 2

Dovecot 2.3 ?
by Reuben Farrelly 24 Apr '17

24 Apr '17

Hi, Is anyone here running dovecot-2.3 from hg? I'm just interested in finding out how people are going with it, how stable is it and are there any interesting features that make it worth spending time looking at. I'm in a position where I can test with semi-live data (backed up and moderately important but not mission critical) but before I take the plunge I'd like to find out what the general consensus is. I see pretty much all bugfixes from 2.2 are going into 2.3 as well... Is there any timeframe on a beta release? Thanks, Reuben

3 4

Public Folder Problems
by Kevin 24 Apr '17

24 Apr '17

Hi, I've been trying to get a Public Folder namespace working, so far without any luck. I've tried in my home folder, outside my home folder, with and without acl. I can see the namespace (PUBLIC) in Roundcube, but that's as far as I get. I've put a .FirstFolder under the location, but it never appears. I've added dirs cur, new, and tmp but no difference. No hints from the maillog. Could some kind soul please put me out of my misery! # 2.2.28 (bed8434): /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-642.15.1.el6.x86_64 x86_64 CentOS release 6.9 (Final) auth_cache_size = 8 k auth_mechanisms = plain login auth_username_chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!#$-=?^_{}~./@+%" dict { expire = sqlite:/usr/local/cpanel/etc/dovecot/dovecot-dict-expire.conf.ext } first_valid_uid = 201 lda_mailbox_autocreate = yes lmtp_save_to_detail_mailbox = yes mail_access_groups = dovecot mail_location = maildir:~/mail mail_plugins = quota quota_clone mail_prefetch_count = 20 mailbox_list_index = yes maildir_very_dirty_syncs = yes namespace inbox { inbox = yes location = mailbox Archives { auto = subscribe special_use = \Archive } mailbox "Deleted Messages" { auto = no special_use = \Trash } mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Spam { auto = create special_use = \Junk } mailbox Trash { auto = no special_use = \Trash } mailbox not-Spam { auto = create } prefix = INBOX. separator = . type = private } namespace one { list = yes location = maildir:/var/spool/pubmail prefix = PUBLIC. separator = . type = public } passdb { args = /usr/local/cpanel/bin/dovecot-wrap driver = checkpassword } plugin { acl = vfile expire = Trash expire2 = Deleted Messages expire3 = INBOX.Deleted Messages expire4 = INBOX.Trash expire_cache = yes expire_dict = proxy::expire quota_exceeded_message = Mailbox is full / Blocks limit exceeded / Inode limit exceeded zlib_save = gz } protocols = lmtp imap service auth { unix_listener auth-client { mode = 0666 } } service config { vsz_limit = 2 G } service dict { unix_listener dict { group = dovecot mode = 0660 } } service imap-login { client_limit = 500 inet_listener imap { address = *,:: } inet_listener imaps { address = *,:: } process_limit = 50 process_min_avail = 2 service_count = 0 vsz_limit = 128 M } service imap { process_limit = 512 vsz_limit = 512 M } service lmtp { client_limit = 1 process_limit = 500 unix_listener lmtp { group = mail mode = 0660 user = mailnull } vsz_limit = 512 M } service managesieve-login { client_limit = 500 process_limit = 50 process_min_avail = 2 service_count = 0 vsz_limit = 128 M } service managesieve { process_limit = 512 vsz_limit = 512 M } service pop3-login { client_limit = 500 inet_listener pop3 { address = *,:: } inet_listener pop3s { address = *,:: } process_limit = 50 process_min_avail = 2 service_count = 0 vsz_limit = 128 M } service pop3 { process_limit = 512 vsz_limit = 512 M } service quota-status { executable = quota-status -p postfix unix_listener quota-status { mode = 0666 } } ssl_cert = </etc/dovecot/ssl/dovecot.crt ssl_cipher_list = ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS ssl_key = # hidden, use -P to show it ssl_protocols = !SSLv2 !SSLv3 userdb { driver = prefetch } userdb { args = /usr/local/cpanel/etc/dovecot/cpauthd-dict.conf driver = dict } userdb { args = /usr/local/cpanel/bin/dovecot-wrap driver = checkpassword } protocol imap { imap_capability = +NAMESPACE imap_idle_notify_interval = 24 mins imap_logout_format = in=%i, out=%o, bytes=%i/%o mail_max_userip_connections = 20 mail_plugins = acl quota imap_quota expire imap_zlib quota_clone virtual namespace sent { hidden = yes list = no location = virtual:/usr/local/cpanel/etc/dovecot/virtual/sent:INDEX=~/mail/virtual/%u prefix = sent separator = . } namespace spam { hidden = yes list = no location = virtual:/usr/local/cpanel/etc/dovecot/virtual/spam:INDEX=~/mail/virtual/%u prefix = spam separator = . } } protocol pop3 { mail_max_userip_connections = 3 mail_plugins = quota expire quota quota_clone virtual namespace sent { hidden = yes list = no location = virtual:/usr/local/cpanel/etc/dovecot/virtual/sent:INDEX=~/mail/virtual/%u prefix = sent separator = . } namespace spam { hidden = yes list = no location = virtual:/usr/local/cpanel/etc/dovecot/virtual/spam:INDEX=~/mail/virtual/%u prefix = spam separator = . } pop3_logout_format = top=%t/%p, retr=%r/%b, del=%d/%m, size=%s, bytes=%i/%o pop3_uidl_format = UID%u-%v } protocol lmtp { mail_plugins = quota quota_clone postmaster_address = root quota_full_tempfail = no } protocol lda { mail_plugins = quota quota_clone postmaster_address = root quota_full_tempfail = no }

1 1

[BUG] client state / Message count mismatch with imap-hibernate and mixed POP3/IMAP access
by Christian Balzer 24 Apr '17

24 Apr '17

Hello, this is on Debian Jessie box, dovecot 2.2.27 from backports, imap-hibernate obviously enabled. I've been seeing a few of these since starting this cluster (see previous mail), they all follow the same pattern, a user who accesses their mailbox with both POP3 and IMAP deletes mails with POP3 and the IMAP (imap-hibernate really) is getting confused and upset about this: --- Apr 6 09:55:49 mbx11 dovecot: imap-login: Login: user=<redacted(a)gol.com>, method=PLAIN, rip=xxx.xxx.x.46, lip=xxx.xxx.x.113, mpid=951561, secured, session=<2jBV+HRM1Pbc9w8u> Apr 6 10:01:06 mbx11 dovecot: pop3-login: Login: user=<redacted(a)gol.com>, method=PLAIN, rip=xxx.xxx.x.46, lip=xxx.xxx.x.41, mpid=35447, secured, session=<uZw7C3VM9vfc9w8u> Apr 6 10:01:07 mbx11 dovecot: pop3(redacted(a)gol.com): Disconnected: Logged out top=0/0, retr=0/0, del=1/1, size=20674 session=<uZw7C3VM9vfc9w8u> Apr 6 10:01:07 mbx11 dovecot: imap(redacted(a)gol.com): Error: imap-master: Failed to import client state: Message count mismatch after handling expunges (0 != 1) Apr 6 10:01:07 mbx11 dovecot: imap(redacted(a)gol.com): Client state initialization failed in=0 out=0 head=<0> del=<0> exp=<0> trash=<0> session=<2jBV+HRM1Pbc9w8u> Apr 6 10:01:15 mbx11 dovecot: imap-login: Login: user=<redacted(a)gol.com>, method=PLAIN, rip=xxx.xxx.x.46, lip=xxx.xxx.x.113, mpid=993303, secured, session=<6QC6C3VMF/jc9w8u> Apr 6 10:07:42 mbx11 dovecot: imap-hibernate(redacted(a)gol.com): Connection closed in=85 out=1066 head=<0> del=<0> exp=<0> trash=<0> session=<6QC6C3VMF/jc9w8u> --- Didn't see these errors before introducing imap-hibernate, but then again this _could_ be something introduced between 2.2.13 (previous generation of servers) and .27, but I highly doubt it. My reading of the log is that the original IMAP session (<2jBV+HRM1Pbc9w8u>) fell over and terminated, resulting in the client to start up a new session. If so and with no false data/state transmitted to the client it would be not ideal, but not a critical problem either. Would be delighted if Aki or Timo could comment on this. If you need any further data, let me know. Christian -- Christian Balzer Network/Systems Engineer chibi(a)gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/

2 5

How to query for headers by regexp?
by Ralf Hildebrandt 23 Apr '17

23 Apr '17

I need to find all mails with a certain numerical range in my quarantine: X-Spam-Status: No, score=4.5 tagged_above=2 required=4.6 ... Assume I'd be looking for 4.5-6.0; so I tried searching for doveadm import -u restore mdbox:/home/quarantine/mdbox "" \ mailbox INBOX header X-Spam-Status "No, score=4.5" doveadm import -u restore mdbox:/home/quarantine/mdbox "" \ mailbox INBOX header X-Spam-Status "No, score=4.6" etc. Which would only return 2 Mails, with score score=4.5 What am I doing wrong here? Can I search using regular expressions? -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | https://www.charite.de

2 1

Is there a tool similar to pflogsumm for Dovecot?
by Paolo 22 Apr '17

22 Apr '17

I tried to find a similar tool for Dovecot, but couldn't. Is there such a thing? Also tried to hack a little the logwatch plugin, but I am not a per programmer ... Thanks, P.

1 0

imap service count>1 (was System load spike ...)
by Joseph Tam 21 Apr '17

21 Apr '17

> I'm running dovecot with quite a lot of users and lots of active imap > connections (like 20'000). I'm using different user IDs for users, so I > need to have imap {service_count=1} - i.e. I have a lots of imap > processes running. I had a misconception this service's service_count was non-adjustable. (https://wiki.dovecot.org/Services on imap service) service_count can be changed from 1 if only a single UID is used for mail users. This is improves performance, but it's less secure, because bugs in code may leak email data from another user's earlier connection. How easy would it be to implement a hybrid solution that maps all imap service for a unique UID to a single worker process. It would result in modest performance gain for busy servers, and data leakage security would be minimal. Joseph Tam <jtam.home(a)gmail.com>

1 0

Pipe to PHP script using Sieve Extprograms
by Thomas 21 Apr '17

21 Apr '17

Hello, I seem to be having some issues trying to setup a pipe to PHP scripts using Sieve Extprograms. I am trying to allow each individual mailbox to setup its own PHP pipe rules (in other words, each user should be able to pipe incoming emails to a user defined PHP script). I have the following lines in my main Sieve configuration file (/etc/dovecot/conf.d/90-sieve.conf): sieve_extensions = +vnd.dovecot.pipe sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.execute sieve_plugins = sieve_extprograms I have configured a Sieve service that looks like this (/etc/dovecot/conf.d/90-sieve-extprograms.conf): service php { # Define the executed script as parameter to the sieve service executable = /var/run/dovecot/sieve-pipe/php.sh # Use some unprivileged user for executing the program user = dovenull # The unix socket located in the sieve_pipe_socket_dir (as defined in the # plugin {} section above) unix_listener sieve-pipe/php { # LDA/LMTP must have access user = vmail mode = 0600 } } Contents of /var/run/dovecot/sieve-pipe/php.sh: #!/usr/bin/php Sieve script for one of my mailboxes: require ["fileinto", "regex", "date", "relational", "vacation"]; require ["mailbox", "imap4flags", "vnd.dovecot.pipe"]; # Move spam to spam folder if header :contains "X-Spam-Flag" "YES" { fileinto "Junk"; # Stop here so that we do not reply on spams stop; } pipe :try "php /var/www/web2/piping.php support(a)example.com" keep; This results however in the following error in my log file: main script: line 5: error: require command: unknown Sieve capability `vnd.dovecot.pipe'. main script: line 14: error: unknown command 'pipe' (only reported once at first occurence). main script: error: validation failed. I am also not quite confident I have set this up correctly. Am I heading in the right direction for what I am trying to achieve? Dovecot version: 2.2.10 dovecot -n: # 2.2.10: /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-514.16.1.el7.x86_64 x86_64 CentOS Linux release 7.3.1611 (Core) auth_mechanisms = plain login disable_plaintext_auth = no listen = *,[::] log_timestamp = "%Y-%m-%d %H:%M:%S " mail_plugins = " quota" mail_privileged_group = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave passdb { args = /etc/dovecot-sql.conf driver = sql } plugin { quota = dict:user::file:/var/vmail/%d/%n/.quotausage sieve = /var/vmail/%d/%n/.sieve } protocols = imap pop3 service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service imap-login { client_limit = 1000 process_limit = 500 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } ssl_cert = </etc/postfix/smtpd.cert ssl_key = </etc/postfix/smtpd.key ssl_protocols = !SSLv2 !SSLv3 userdb { driver = prefetch } userdb { args = /etc/dovecot-sql.conf driver = sql } protocol imap { mail_plugins = quota imap_quota } protocol pop3 { mail_plugins = quota pop3_uidl_format = %08Xu%08Xv } protocol lda { mail_plugins = sieve quota postmaster_address = root@localhost } protocol lmtp { mail_plugins = quota sieve postmaster_address = webmaster@localhost }

2 2

namespace assertion failed
by Elisamuel Resto 21 Apr '17

21 Apr '17

Hello, I'm trying to figure out if this is a configuration issue or an actual bug. Dovecot does start, but if anybody tries to log in or if you run sievec you get an assertion fail and a coredump. --- sievec(root): Panic: file mail-namespace.c: line 709 (mail_namespace_find): assertion failed: (ns != NULL) sievec(root): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x9c3a2) [0x7ff70e2133a2] -> /usr/lib/dovecot/libdovecot.so.0(+0x9c429) [0x7ff70e213429] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7ff70e1a5e41] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_namespace_find+0xc8) [0x7ff70e4e03a8] -> /usr/lib/dovecot/modules/lib11_trash_plugin.so(+0x1e56) [0x7ff70d5ace56] -> /usr/lib/dovecot/libdovecot-storage.so.0(hook_mail_user_created+0x1eb) [0x7ff70e4ecfeb] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_user_init+0x206) [0x7ff70e4f1fb6] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_storage_service_next+0x438) [0x7ff70e4f00c8] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_storage_service_lookup_next+0x4c) [0x7ff70e4f068c] -> sievec(sieve_tool_init_finish+0xc1) [0x403371] -> sievec(main+0xd2) [0x402cb2] -> /usr/lib/libc.so.6(__libc_start_main+0xf1) [0x7ff70ddf3511] -> sievec(_start+0x2a) [0x402f3a] Aborted (core dumped) --- Coredump as follows: --- #0 0x00007ff70de06a10 in raise () from /usr/lib/libc.so.6 #1 0x00007ff70de0813a in abort () from /usr/lib/libc.so.6 #2 0x00007ff70e21339a in ?? () from /usr/lib/dovecot/libdovecot.so.0 #3 0x00007ff70e213429 in default_fatal_handler () from /usr/lib/dovecot/libdovecot.so.0 #4 0x00007ff70e1a5e41 in i_panic () from /usr/lib/dovecot/libdovecot.so.0 #5 0x00007ff70e4e03a8 in mail_namespace_find () from /usr/lib/dovecot/libdovecot-storage.so.0 #6 0x00007ff70d5ace56 in ?? () from /usr/lib/dovecot/modules/lib11_trash_plugin.so #7 0x00007ff70e4ecfeb in hook_mail_user_created () from /usr/lib/dovecot/libdovecot-storage.so.0 #8 0x00007ff70e4f1fb6 in mail_user_init () from /usr/lib/dovecot/libdovecot-storage.so.0 #9 0x00007ff70e4f00c8 in mail_storage_service_next () from /usr/lib/dovecot/libdovecot-storage.so.0 #10 0x00007ff70e4f068c in mail_storage_service_lookup_next () from /usr/lib/dovecot/libdovecot-storage.so.0 #11 0x0000000000403371 in sieve_tool_init_finish () #12 0x0000000000402cb2 in main () --- Any help appreciated! -Sam

3 3