February 2017 - dovecot

[Sieve] Is the way to run external script to get attachment?
by Konstantin Shalygin 22 Mar '17

22 Mar '17

Hello. Read all about Extprograms, but have a question. Use case: Every day we have couple messages with attachment (from one sender, this is one-way communication). This attachment is uploads to ownCloud. By hand... need some automation. As I see, pipe cat execute scripts with text data USER/FROM/SUBJECT - and I think with variables it can be any text data, how about attachment? How I see this: script executes and as arg receives from sieve attachment file, or message file and we can parse it for attachment. When we have attachment - upload to ownCloud via RESTapi and purge message. Found some realization for encrypt messages https://github.com/EtiennePerot/gpgit/blob/master/encmaildir.sh but for me more easy just connect to IMAP and do what I want (but I love sieve). Thanks. <http://wiki.dovecot.org/Pigeonhole/Sieve/Plugins/Extprograms>

2 2

Is it possible to use sieve when delivering via pipe to the dovecot deliver command ?
by Thomas Carrié 21 Mar '17

21 Mar '17

Hello, I am using Exim with this delivery configuration: virtual_delivery: driver = pipe command = /usr/lib/dovecot/deliver -d $local_part@$domain -f ... I have enabled manage-sieve and uploaded a simple sieve script that I can see on the server. man page of /usr/lib/dovecot/deliver says it supports sieve. Mail debug is enabled: $ dovecot -n | grep debug mail_debug = yes Delivering a mail shows only this line in dovecot log: Jan 21 17:38:51 myserver dovecot: lda(joe(a)something.com): msgid=<20170121183537.7313f219@homenetwork>: saved mail to INBOX According to page http://wiki2.dovecot.org/Pigeonhole/Sieve/Troubleshooting#Sieve_Scripts_are… I should have a log saying that the sieve plugin is loaded Here is the sieve relevant parts of dovecot -n: # 2.2.13: /etc/dovecot/dovecot.conf ... plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } ... protocols = " imap sieve pop3 sieve" ... protocol lda { mail_plugins = " sieve" } Is there a requirement to use delivery via service (unix socket) so that sieve is enabled? Or is there a configuration thing I forgot ? Thank you for your help Thomas

2 2

[Bug] Mailbox aliases still broken
by azurit＠pobox.sk 20 Mar '17

20 Mar '17

Hi, about an year ago i was reporting a bug in mailbox aliases, which remains unfixed and unasnwered (probably totally ignored, don't understand why). I thought it was because the bug is old and already fixed but yesterday i upgraded to Dovecot 2.2.24 and problem persists. Here is the original report, everything, except the Dovecot version, is still correct: http://dovecot.org/list/dovecot/2015-June/101176.html Will this be fixed? Thanks for info. azur

4 7

Replication Troubles
by Wolfgang Hennerbichler 16 Mar '17

16 Mar '17

Hi Dovecot Users, I’ve configured dovecot dsync replication and I see troubles in the logs and get user complaints which I can’t explain. I found similar threads on this mailinglist, but I couldn’t find a solution anywhere. Does anybody have dsync running without problems on a high volume mailserver? I see the following logs, examples given: Feb 17 18:16:49 dovecot dovecot: imap(zoechi): Warning: /var/mail/zoechi/dovecot-uidlist: Duplicate file entry at line 10395: 1487350019.M138380P28563.dovecot.wogri.at,S=18930,W=19377 (uid 41092 -> 41093) - retrying by re-reading from beginning with this one I’m not sure - it might be that this is completely OK because due to replication UIDs clash. Maybe that’s OK, but I couldn’t find a confirmation. Feb 17 18:16:49 dovecot dovecot: imap(zoechi): Warning: Maildir /var/mail/zoechi: Expunged message reappeared, giving a new UID (old uid=41092, file=1487350019.M138380P28563.dovecot.wogri.at,S=18930,W=19377) This one is definitely a problem, deleted messages re-appear in the mailbox. I made sure that the 2 hosts doing replication have different hostnames. I run 2.2.27 (from debian-jessie backports). Config below. Thanks for hints (or pointers to other example configs where dsync works without problems) # dovecot -n # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.8.14 x86_64 Debian 8.7 ext4 auth_verbose = yes debug_log_path = /var/log/dovecot.debug doveadm_password = # hidden, use -P to show it first_valid_gid = 106 first_valid_uid = 104 hostname = localhost last_valid_gid = 106 last_valid_uid = 104 mail_gid = dovecot mail_location = maildir:/var/mail/%n mail_plugins = quota fts fts_lucene virtual notify replication mail_temp_dir = /var/lib/dovecot/tmp mail_uid = dovecot managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext editheader namespace { list = children location = virtual:/var/mail/%n/virtual prefix = virtual. separator = . } namespace inbox { inbox = yes list = yes location = mailbox "Deleted Messages" { auto = subscribe special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Spam { auto = subscribe special_use = \Junk } mailbox Trash { special_use = \Trash } prefix = separator = . subscriptions = yes type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { default_language = de fts = lucene fts_lucene = whitespace_chars=@. mail_replica = tcp:172.16.1.1:12345 quota = maildir:User quota quota_rule = *:storage=5G quota_rule2 = Trash:storage=+200M quota_rule3 = Spam:ignore quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = /etc/sieve/%n.sieve sieve_default = /etc/sieve/default.sieve sieve_dir = ~/sieve sieve_extensions = +editheader } pop3_deleted_flag = $POP3Deleted postmaster_address = postmaster(a)wogri.at protocols = " imap lmtp sieve pop3" service aggregator { fifo_listener replication-notify-fifo { user = dovecot } unix_listener replication-notify { user = dovecot } } service doveadm { inet_listener { port = 12345 } } service imap { process_limit = 1024 } service lmtp { inet_listener lmtp { port = 2003 } unix_listener lmtp { user = dovecot } user = dovecot } service managesieve-login { inet_listener sieve { port = 4190 } service_count = 1 } service pop3 { process_limit = 1024 } service quota-warning { executable = script /usr/local/sbin/quota-warning.sh unix_listener quota-warning { user = dovecot } user = dovecot } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } ssl = required ssl_cert = </etc/dovecot/server.cert.pem ssl_key = # hidden, use -P to show it userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } protocol lmtp { mail_plugins = quota sieve virtual notify replication } protocol lda { mail_plugins = quota fts fts_lucene virtual notify replication sieve } protocol imap { mail_max_userip_connections = 50 mail_plugins = quota fts fts_lucene virtual notify replication imap_quota } protocol pop3 { mail_plugins = quota fts fts_lucene virtual notify replication } local 2a00:XXX::/64 { doveadm_password = # hidden, use -P to show it }

3 4

Users with extra mailbox: namespace/mailbox in userdb lookup?
by MRob 14 Mar '17

14 Mar '17

Hello, We have a need for some users to have an additional folder created and subscribed for them. Is is possible to return an override for the default "namespace inbox" containing the needed mailbox definitions in the userdb lookup for such users? If so, how would the userdb lookup result be formatted? Presumably as just one long string, but how would the setting be named given there can't be a space in it? Can I replace the space with an underscore, e.g. userdb_namespace_inbox = '{ mailbox Drafts { auto=subscribe special_use = \Drafts } mailbox Trash { auto=subscribe special_use = \Trash } }' Or is there some other way to do this?

4 11

Dict protocol changes string
by Nagy, Attila 13 Mar '17

13 Mar '17

Hi, I use pass and userdb with dict protocol in a similar way: key passdb { key = passdb^MAuth-User: %u^MAuth-Pass: %w^MAuth-Protocol: %s^MClient-IP: %r format = json } (^M is an \r character, inserted with vi CTRL-v + enter) Until 2.2.24 this has worked, but 2.2.25 seems to convert that ASCII 13 into an ASCII 1 and an "r". Python printout from what I get with 2.2.25: 'Lshared/passdb\x01rAuth-User: user\x01rAuth-Pass: pass\x01rAuth-Protocol: pop3\x01rClient-IP: 1.2.3.4' Is this change intentional? Why?

2 4

v2.2.28 released
by Timo Sirainen 10 Mar '17

10 Mar '17

http://dovecot.org/releases/2.2/dovecot-2.2.28.tar.gz http://dovecot.org/releases/2.2/dovecot-2.2.28.tar.gz.sig * director: "doveadm director move" to same host now refreshes user's timeout. This allows keeping user constantly in the same backend by just periodically moving the user there. * When new mailbox is created, use initially INBOX's dovecot.index.cache caching decisions. * Expunging mails writes GUID to dovecot.index.log now only if the GUID is quickly available from index/cache. * pop3c: Increase timeout for PASS command to 5 minutes. * Mail access errors are no longer ignored when searching or sorting. With IMAP the untagged SEARCH/SORT reply is still sent the same as before, but NO reply is returned instead of OK. + Make dovecot.list.index's filename configurable. This is needed when there are multiple namespaces pointing to the same mail root (e.g. lazy_expunge namespace for mdbox). + Add size.virtual to dovecot.index when folder vsizes are accessed (e.g. quota=count). This is mainly a workaround to avoid slow quota recalculation performance when message sizes get lost from dovecot.index.cache due to corruption or some other reason. + auth: Support OAUTHBEARER and XOAUTH2 mechanisms. Also support them in lib-dsasl for client side. + auth: Support filtering by SASL mechanism: passdb { mechanisms } + Shrink the mail processes' memory usage by not storing settings duplicated unnecessarily many times. + imap: Add imap_fetch_failure setting to control what happens when FETCH fails for some mails (see example-config). + imap: Include info about last command in disconnection log line. + imap: Created new SEARCH=X-MIMEPART extension. It's currently not advertised by default, since it's not fully implemented. + fts-solr: Add support for basic authentication. + Cassandra: Support automatically retrying failed queries if execution_retry_interval and execution_retry_times are set. + doveadm: Added "mailbox path" command. + mail_log plugin: If plugin { mail_log_cached_only=yes }, log the wanted fields only if it doesn't require opening the email. + mail_vsize_bg_after_count setting added (see example-config). + mail_sort_max_read_count setting added (see example-config). + pop3c: Added pop3c_features=no-pipelining setting to prevent using PIPELINING extension even though it's advertised. - Index files: day_first_uid wasn't updated correctly since v2.2.26. This caused dovecot.index.cache to be non-optimal. - imap: SEARCH/SORT may have assert-crashed in client_check_command_hangs - imap: FETCH X-MAILBOX may have assert-crashed in virtual mailboxes. - imap: Running time in tagged command reply was often wrongly 0. - search: Using NOT n:* or NOT UID n:* wasn't handled correctly - director: doveadm director kick was broken - director: Fix crash when using director_flush_socket - director: Fix some bugs when moving users between backends - imapc: Various error handling fixes and improvements - master: doveadm process status output had a lot of duplicates. - autoexpunge: If mailbox's rename timestamp is newer than mail's save-timestamp, use it instead. This is useful when autoexpunging e.g. Trash/* and an entire mailbox is deleted by renaming it under Trash to prevent it from being autoexpunged too early. - autoexpunge: Multiple processes may have been trying to expunge the same mails simultaneously. This was problematic especially with lazy_expunge plugin. - auth: %{passdb:*} was empty in auth-worker processes - auth-policy: hashed_password was always sent empty. - dict-sql: Merge multiple UPDATEs to a single statement if possible. - fts-solr: Escape {} chars when sending queries - fts: fts_autoindex_exclude = \Special-use caused crashes - doveadm-server: Fix leaks and other problems when process is reused for multiple requests (service_count != 1) - sdbox: Fix assert-crash on mailbox create race - lda/lmtp: deliver_log_format values weren't entirely correct if Sieve was used. especially %{storage_id} was broken. - lmtp_user_concurrency_limit didn't work if userdb changed username

6 7

Maildirsize not updated
by chaouche yacine 08 Mar '17

08 Mar '17

Hello dovecot Problem : maildirsize and dovecot quota get have different values dovecot version : 2.2.13 happens with : some mailboxes This issue happens only to some mailboxes and only for a specific period of time until maildirsize gets updated by something (not me). I can't reproduce this now because it happened this morning and now its 2 PM and the problem is gone (maildirsize was updated). Anybody experienced the same situation ? -- Yassine.

3 6

fts_solr and connection via https://
by Jan Vonde 04 Mar '17

04 Mar '17

Hi, I am trying to get fts_solr working and my index server is available via HTTPS only. Dovecot is running on a Debian Jessie system and the Solr server has a letsencrypt certificate. My dovecot version is: 2.2.devel (a9ed8ae) The current setup is: 10-mail.conf: mail_plugins = fts fts_solr 90-fts.conf: plugin { fts = solr fts_autoindex = yes fts_solr = url=https://foo.example.com/solr/dovecot/ } When I try to index the mailboxes I am getting error messages like this: doveadm(user@host): Error: fts_solr: Lookup failed: 9002 Couldn't initialize SSL context: Can't verify remote server certs without trusted CAs (ssl_client_ca_* settings) doveadm(user@host): Error: Mailbox INBOX: Status lookup failed: Internal error occurred. Refer to server log for more information. [2017-01-22 09:52:38] Segmentation fault Contacting the index server via curl on the command line on the same host works, it returns HTTP 200: user@host ~ $ curl -s -o /dev/null -w "%{http_code}" https://foo.example.com/solr/ 200 user@host ~ $ Currently I have the following ssl related settings: user@host ~ $ doveconf -n -P | grep -i ssl ssl_cert = </etc/ssl/certs/mail.example.org.crt ssl_cipher_list = ALL:!LOW:!SSLv2:ALL:!aNULL:!ADH:!eNULL:!EXP:RC4+RSA:+HIGH:+MEDIUM ssl_key = </etc/ssl/private/mail.example.org.key ssl_protocols = !SSLv2 !SSLv3 I tried adding the following settings but that didn't help: ssl_ca = < /etc/ssl/certs/ca-certificates.crt ssl_client_ca_dir = /etc/ssl/certs Can you give me a hint how I can get the ssl certificate accepted? Thanks in advance and have a nice day, Jan :-)

2 10

Director+NFS Experiences
by Mark Moseley 03 Mar '17

03 Mar '17

As someone who is about to begin the process of moving from maildir to mdbox on NFS (and therefore just about to start the 'director-ization' of everything) for ~6.5m mailboxes, I'm curious if anyone can share any experiences with it. The list is surprisingly quiet about this subject, and articles on google are mainly just about setting director up. I've yet to stumble across an article about someone's experiences with it. * How big of a director cluster do you use? I'm going to have millions of mailboxes behind 10 directors. I'm guessing that's plenty. It's actually split over two datacenters. In the larger, we've got about 200k connections currently, so in a perfectly-balanced world, each director would have 20k connections on it. I'm guessing that's child's play. Any good rule of thumb for ratio of 'backend servers::director servers'? In my larger DC, it's about 5::1. * Do you use the perl poolmon script or something else? The perl script was being weird for me, so I rewrote it in python but it basically does the exact same things. * Seen any issues with director? In testing, I managed to wedge things by having my poolmon script running on all the cluster boxes (I think). I've since rewritten it to run *only* on the lowest-numbered director. When it wedged, I had piles (read: hundreds per second) of log entries that said: Feb 12 06:25:03 director: Warning: director(10.1.20.5:9090/right): Host 10.1.17.3 is being updated before previous update had finished (down -> up) - setting to state=up vhosts=0 Feb 12 06:25:03 director: Warning: director(10.1.20.5:9090/right): Host 10.1.17.3 is being updated before previous update had finished (up -> down) - setting to state=down vhosts=0 Feb 12 06:25:03 director: Warning: director(10.1.20.3:9090/left): Host 10.1.17.3 is being updated before previous update had finished (down -> up) - setting to state=up vhosts=0 Feb 12 06:25:03 director: Warning: director(10.1.20.3:9090/left): Host 10.1.17.3 is being updated before previous update had finished (up -> down) - setting to state=down vhosts=0 Because it was in testing, I didn't notice it and it was like this for several days till dovecot was restarted on all the director nodes. I'm not 100% on what happened, but my *guess* is that two boxes tried to update the status of the same backend server in rapid succession. * Assuming you're using NFS, do you still see non-trivial amounts of indexes getting corrupted? * Again, assuming NFS and assuming at least some corrupted indexes, what's your guess for success rate % for dovecot recovering them automatically? And how about success rate % for ones that dovecot wasn't able to do automatically but you had to use doveadm to repair it? Really what I'm trying to figure out is 1) how often sysops will need to manually recover indexes; and 2) how often admins *can't* manually recover indexes? * if you have unrecoverable indexes (and assuming you have snapshots on your NFS server), does grabbing the most recent indexes from the snapshots always work for recovery (obviously, up till the point that the snapshot was taken)? * Any gotchas you've seen anywhere in a director-fied stack? I realize that's a broad question :) * Does one of your director nodes going down cause any issues? E.g. issues with the left and right nodes syncing with each other? Or when the director node comes back up? * Does a backend node going down cause a storm of reconnects? In the time between deploying director and getting mailboxes converted to mdbox, reconnects for us will mean cold local-disk dovecot caches. But hopefully consistent hashing helps with that? * Do you have consistent hashing turned on? I can't think of any reason not to have it turned on, but who knows * Any other configuration knobs (including sysctl) that you needed to futz with, vs the default? I appreciate any feedback!

8 14