- dovecot - dovecot.org

Streaming out?
by Michael Segel 26 Apr '17

26 Apr '17

Hi, Has anyone tried to write out a mail stream to Kafka? Are there any good write ups on this feature? Can you split the mail so that you still write to a MailDir and also write to a stream? Thx -Mike

1 0

fts error recovery
by Daniel Miller 26 Apr '17

26 Apr '17

When a message is received, that is supposed to be added to the fts index, and an fts error occurs (like the Solr server is offline for some reason) - Dovecot logs that error and stores the message. Later, the fts system is available and new messages are added to the fts index without issue. Will the "missed" messages get added? Or would this require a manual re-indexing? -- Daniel

1 0

namespace alias issues?
by Kelsey Cummings 25 Apr '17

25 Apr '17

The Courier IMAP backwards compatibility section in the Namespaces wiki suggests that hidden aliased namespace can cause problems for some clients but doesn't quantify the extent or likelihood of running into these problems. Does anyone with real world experience with similar configurations have any feedback? -- kelsey.cummings(a)sonic.com sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407

1 0

Host ... is being updated before previous update had finished
by Mark Moseley 24 Apr '17

24 Apr '17

We just had a bunch of backend boxes go down due to a DDoS in our director cluster. When the DDoS died down, our director ring was a mess. Each box had thousands (and hundreds per second, which is a bit much) of log lines like the following: Apr 03 19:59:29 director: Warning: director(10.1.20.10:9090/left): Host 10.1.17.15 is being updated before previous update had finished (up -> down) - setting to state=down vhosts=100 Apr 03 19:59:29 director: Warning: director(10.1.20.10:9090/left): Host 10.1.17.15 is being updated before previous update had finished (down -> up) - setting to state=up vhosts=100 Apr 03 19:59:29 director: Warning: director(10.1.20.10:9090/left): Host 10.1.17.15 is being updated before previous update had finished (up -> down) - setting to state=down vhosts=100 Apr 03 19:59:29 director: Warning: director(10.1.20.10:9090/left): Host 10.1.17.15 is being updated before previous update had finished (down -> up) - setting to state=up vhosts=100 Apr 03 19:59:29 director: Warning: director(10.1.20.10:9090/left): Host 10.1.17.15 is being updated before previous update had finished (up -> down) - setting to state=down vhosts=100 Apr 03 19:59:29 director: Warning: director(10.1.20.2:9090/right): Host 10.1.17.15 is being updated before previous update had finished (down -> up) - setting to state=up vhosts=100 Apr 03 19:59:29 director: Warning: director(10.1.20.2:9090/right): Host 10.1.17.15 is being updated before previous update had finished (up -> down) - setting to state=down vhosts=100 Apr 03 19:59:29 director: Warning: director(10.1.20.2:9090/right): Host 10.1.17.15 is being updated before previous update had finished (down -> up) - setting to state=up vhosts=100 Apr 03 19:59:29 director: Warning: director(10.1.20.2:9090/right): Host 10.1.17.15 is being updated before previous update had finished (up -> down) - setting to state=down vhosts=100 Apr 03 19:59:29 director: Warning: director(10.1.20.2:9090/right): Host 10.1.17.15 is being updated before previous update had finished (down -> up) - setting to state=up vhosts=100 This was on every director box and the status of all of the directors in 'doveadm director ring status' was 'handshaking'. Here's a sample packet between directors: 19:51:23.552280 IP 10.1.20.10.56670 > 10.1.20.1.9090: Flags [P.], seq 4147:5128, ack 0, win 0, options [nop,nop,TS val 1373505883 ecr 1721203906], length 981 Q. [f.|.HOST 10.1.20.10 9090 1006732 10.1.17.15 100 D1491260800 HOST 10.1.20.10 9090 1006733 10.1.17.15 100 U1491260800 HOST 10.1.20.10 9090 1006734 10.1.17.15 100 D1491260800 HOST 10.1.20.10 9090 1006735 10.1.17.15 100 U1491260800 HOST 10.1.20.10 9090 1006736 10.1.17.15 100 D1491260800 HOST 10.1.20.10 9090 1006737 10.1.17.15 100 U1491260800 HOST 10.1.20.10 9090 1006738 10.1.17.15 100 D1491260800 HOST 10.1.20.10 9090 1006739 10.1.17.15 100 U1491260800 HOST 10.1.20.10 9090 1006740 10.1.17.15 100 D1491260800 HOST 10.1.20.10 9090 1006741 10.1.17.15 100 U1491260800 HOST 10.1.20.10 9090 1006742 10.1.17.15 100 D1491260800 HOST 10.1.20.10 9090 1006743 10.1.17.15 100 U1491260800 HOST 10.1.20.10 9090 1006744 10.1.17.15 100 D1491260800 HOST 10.1.20.10 9090 1006745 10.1.17.15 100 U1491260800 HOST 10.1.20.10 9090 1006746 10.1.17.15 100 D1491260800 HOST 10.1.20.10 9090 1006747 10.1.17.15 100 U1491260800 SYNC 10.1.20.10 9090 1011840 7 1491263483 3377546382 I'm guessing that D1491260800 is the user hash (with D for down), and the U version is for 'up'. I'm happy to provide the full tcpdump (and/or doveconf -a), though the tcpdump is basically all identical the one I pasted (same hash, same host). This seems pretty fragile. There should be some sort of tie break for that, instead of bringing the entire cluster to its knees. Or just drop the backend host completely. Or something, anything besides hosing things pretty badly. This is 2.2.27, on both the directors and backend. If the answer is upgrade to 2.2.28, then I'll upgrade immediately. I see commit a9ade104616bbb81c34cc6f8bfde5dab0571afac mentions the same error but the commit predates 2.2.27 by a month and a half. In the meantime, is there any doveadm command I could've done to fix this? I tried removing the host (doveadm director remove 10.1.17.15) but that didn't do anything. I didn't think to try to flush the mapping for that user till just now. I suspect that with the ring unsync'd, flushing the user wouldn't have helped. The only remedy was to kill dovecot on every box in the director cluster and then (with dovecot down on *all* of them) start dovecot back up. Restarting each director's dovecot (with other directors' dovecots still running) did nothing. Only by brining the entire cluster down did dovecot stop furiously logging "Host ... is being updated before previous update had finished" on every director host. Any advice is most welcome.

2 3

System load spike on dovecot reload
by dave＠evilcigi.eu 24 Apr '17

24 Apr '17

Hi everyone, I'm running dovecot with quite a lot of users and lots of active imap connections (like 20'000). I'm using different user IDs for users, so I need to have imap {service_count=1} - i.e. I have a lots of imap processes running. Everything works fine, until I reload dovecot configuration. When that happen, every client is forced to relogin in the same time and that causes a huge system load spike (2-3000 5 min load). I was thinking that it would be great, if dovecot wouldn't kick all the users in the same time during reload, but somehow gradually, during specified interval. I'm aware of the shutdown_clients directive that could help, but I don't like it - I do want the clients get disconnected on dovecot shutdown and also I want them to relogin in reasonably short time after reload. Is something like that possible with dovecot or does it make sense to implement that in the future versions? Thank you. Dave.

5 4

Feature Request - Director Balance
by Webert de Souza Lima 24 Apr '17

24 Apr '17

Hi, often I run into the situation where a dovecot server goes down for maintenance, and all users get concentrated in the remaining dovecot server (considering I have 2 dovecot servers only). When that dovecot server comes back online, director server will send new users to it, but the dovecot server that was up all the time will still have tons of clients mapped to it. I suggest the director servers to always try to balance load between servers, in the way: - if a server has several more connections than other, mark it to re-balance - when a user connected to this loaded server disconnects, map it to another server (that is per definition not the same server) immediately. that way it would gracefully re-balance, not killing existing connections, just waiting for them to finish. Thank you for your time. Webert Lima MAV Tecnologia Belo Horizonte, Brasil.

2 2

Dovecot 2.3 ?
by Reuben Farrelly 24 Apr '17

24 Apr '17

Hi, Is anyone here running dovecot-2.3 from hg? I'm just interested in finding out how people are going with it, how stable is it and are there any interesting features that make it worth spending time looking at. I'm in a position where I can test with semi-live data (backed up and moderately important but not mission critical) but before I take the plunge I'd like to find out what the general consensus is. I see pretty much all bugfixes from 2.2 are going into 2.3 as well... Is there any timeframe on a beta release? Thanks, Reuben

3 4

Public Folder Problems
by Kevin 24 Apr '17

24 Apr '17

Hi, I've been trying to get a Public Folder namespace working, so far without any luck. I've tried in my home folder, outside my home folder, with and without acl. I can see the namespace (PUBLIC) in Roundcube, but that's as far as I get. I've put a .FirstFolder under the location, but it never appears. I've added dirs cur, new, and tmp but no difference. No hints from the maillog. Could some kind soul please put me out of my misery! # 2.2.28 (bed8434): /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-642.15.1.el6.x86_64 x86_64 CentOS release 6.9 (Final) auth_cache_size = 8 k auth_mechanisms = plain login auth_username_chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!#$-=?^_{}~./@+%" dict { expire = sqlite:/usr/local/cpanel/etc/dovecot/dovecot-dict-expire.conf.ext } first_valid_uid = 201 lda_mailbox_autocreate = yes lmtp_save_to_detail_mailbox = yes mail_access_groups = dovecot mail_location = maildir:~/mail mail_plugins = quota quota_clone mail_prefetch_count = 20 mailbox_list_index = yes maildir_very_dirty_syncs = yes namespace inbox { inbox = yes location = mailbox Archives { auto = subscribe special_use = \Archive } mailbox "Deleted Messages" { auto = no special_use = \Trash } mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Spam { auto = create special_use = \Junk } mailbox Trash { auto = no special_use = \Trash } mailbox not-Spam { auto = create } prefix = INBOX. separator = . type = private } namespace one { list = yes location = maildir:/var/spool/pubmail prefix = PUBLIC. separator = . type = public } passdb { args = /usr/local/cpanel/bin/dovecot-wrap driver = checkpassword } plugin { acl = vfile expire = Trash expire2 = Deleted Messages expire3 = INBOX.Deleted Messages expire4 = INBOX.Trash expire_cache = yes expire_dict = proxy::expire quota_exceeded_message = Mailbox is full / Blocks limit exceeded / Inode limit exceeded zlib_save = gz } protocols = lmtp imap service auth { unix_listener auth-client { mode = 0666 } } service config { vsz_limit = 2 G } service dict { unix_listener dict { group = dovecot mode = 0660 } } service imap-login { client_limit = 500 inet_listener imap { address = *,:: } inet_listener imaps { address = *,:: } process_limit = 50 process_min_avail = 2 service_count = 0 vsz_limit = 128 M } service imap { process_limit = 512 vsz_limit = 512 M } service lmtp { client_limit = 1 process_limit = 500 unix_listener lmtp { group = mail mode = 0660 user = mailnull } vsz_limit = 512 M } service managesieve-login { client_limit = 500 process_limit = 50 process_min_avail = 2 service_count = 0 vsz_limit = 128 M } service managesieve { process_limit = 512 vsz_limit = 512 M } service pop3-login { client_limit = 500 inet_listener pop3 { address = *,:: } inet_listener pop3s { address = *,:: } process_limit = 50 process_min_avail = 2 service_count = 0 vsz_limit = 128 M } service pop3 { process_limit = 512 vsz_limit = 512 M } service quota-status { executable = quota-status -p postfix unix_listener quota-status { mode = 0666 } } ssl_cert = </etc/dovecot/ssl/dovecot.crt ssl_cipher_list = ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS ssl_key = # hidden, use -P to show it ssl_protocols = !SSLv2 !SSLv3 userdb { driver = prefetch } userdb { args = /usr/local/cpanel/etc/dovecot/cpauthd-dict.conf driver = dict } userdb { args = /usr/local/cpanel/bin/dovecot-wrap driver = checkpassword } protocol imap { imap_capability = +NAMESPACE imap_idle_notify_interval = 24 mins imap_logout_format = in=%i, out=%o, bytes=%i/%o mail_max_userip_connections = 20 mail_plugins = acl quota imap_quota expire imap_zlib quota_clone virtual namespace sent { hidden = yes list = no location = virtual:/usr/local/cpanel/etc/dovecot/virtual/sent:INDEX=~/mail/virtual/%u prefix = sent separator = . } namespace spam { hidden = yes list = no location = virtual:/usr/local/cpanel/etc/dovecot/virtual/spam:INDEX=~/mail/virtual/%u prefix = spam separator = . } } protocol pop3 { mail_max_userip_connections = 3 mail_plugins = quota expire quota quota_clone virtual namespace sent { hidden = yes list = no location = virtual:/usr/local/cpanel/etc/dovecot/virtual/sent:INDEX=~/mail/virtual/%u prefix = sent separator = . } namespace spam { hidden = yes list = no location = virtual:/usr/local/cpanel/etc/dovecot/virtual/spam:INDEX=~/mail/virtual/%u prefix = spam separator = . } pop3_logout_format = top=%t/%p, retr=%r/%b, del=%d/%m, size=%s, bytes=%i/%o pop3_uidl_format = UID%u-%v } protocol lmtp { mail_plugins = quota quota_clone postmaster_address = root quota_full_tempfail = no } protocol lda { mail_plugins = quota quota_clone postmaster_address = root quota_full_tempfail = no }

1 1

[BUG] client state / Message count mismatch with imap-hibernate and mixed POP3/IMAP access
by Christian Balzer 24 Apr '17

24 Apr '17

Hello, this is on Debian Jessie box, dovecot 2.2.27 from backports, imap-hibernate obviously enabled. I've been seeing a few of these since starting this cluster (see previous mail), they all follow the same pattern, a user who accesses their mailbox with both POP3 and IMAP deletes mails with POP3 and the IMAP (imap-hibernate really) is getting confused and upset about this: --- Apr 6 09:55:49 mbx11 dovecot: imap-login: Login: user=<redacted(a)gol.com>, method=PLAIN, rip=xxx.xxx.x.46, lip=xxx.xxx.x.113, mpid=951561, secured, session=<2jBV+HRM1Pbc9w8u> Apr 6 10:01:06 mbx11 dovecot: pop3-login: Login: user=<redacted(a)gol.com>, method=PLAIN, rip=xxx.xxx.x.46, lip=xxx.xxx.x.41, mpid=35447, secured, session=<uZw7C3VM9vfc9w8u> Apr 6 10:01:07 mbx11 dovecot: pop3(redacted(a)gol.com): Disconnected: Logged out top=0/0, retr=0/0, del=1/1, size=20674 session=<uZw7C3VM9vfc9w8u> Apr 6 10:01:07 mbx11 dovecot: imap(redacted(a)gol.com): Error: imap-master: Failed to import client state: Message count mismatch after handling expunges (0 != 1) Apr 6 10:01:07 mbx11 dovecot: imap(redacted(a)gol.com): Client state initialization failed in=0 out=0 head=<0> del=<0> exp=<0> trash=<0> session=<2jBV+HRM1Pbc9w8u> Apr 6 10:01:15 mbx11 dovecot: imap-login: Login: user=<redacted(a)gol.com>, method=PLAIN, rip=xxx.xxx.x.46, lip=xxx.xxx.x.113, mpid=993303, secured, session=<6QC6C3VMF/jc9w8u> Apr 6 10:07:42 mbx11 dovecot: imap-hibernate(redacted(a)gol.com): Connection closed in=85 out=1066 head=<0> del=<0> exp=<0> trash=<0> session=<6QC6C3VMF/jc9w8u> --- Didn't see these errors before introducing imap-hibernate, but then again this _could_ be something introduced between 2.2.13 (previous generation of servers) and .27, but I highly doubt it. My reading of the log is that the original IMAP session (<2jBV+HRM1Pbc9w8u>) fell over and terminated, resulting in the client to start up a new session. If so and with no false data/state transmitted to the client it would be not ideal, but not a critical problem either. Would be delighted if Aki or Timo could comment on this. If you need any further data, let me know. Christian -- Christian Balzer Network/Systems Engineer chibi(a)gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/

2 5

How to query for headers by regexp?
by Ralf Hildebrandt 23 Apr '17

23 Apr '17

I need to find all mails with a certain numerical range in my quarantine: X-Spam-Status: No, score=4.5 tagged_above=2 required=4.6 ... Assume I'd be looking for 4.5-6.0; so I tried searching for doveadm import -u restore mdbox:/home/quarantine/mdbox "" \ mailbox INBOX header X-Spam-Status "No, score=4.5" doveadm import -u restore mdbox:/home/quarantine/mdbox "" \ mailbox INBOX header X-Spam-Status "No, score=4.6" etc. Which would only return 2 Mails, with score score=4.5 What am I doing wrong here? Can I search using regular expressions? -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | https://www.charite.de

2 1