December 2016 - dovecot

Reappearing emails
by Ron Cleven 17 Jan '17

17 Jan '17

We have been running Dovecot 2.2.10 with a pair of CentOS 7 boxes with replications for the past year. We have been quite happy with the performance and reliability. Recently we received a report from an Android IMAP user that emails that he deleted without reading would often reappear in his INBOX. At first I thought it might be some quirky incompatibility with his email client, but I found that I could easily reproduce it in Roundcube by dragging an unread email to Trash. I have been monitoring this forum fairly religiously and don't recall seeing any similar reports. I plan to capture the IMAP wire interactions, but before doing that, I thought I would submit this to the IMAP gods to see if anyone had heard of this and/or if there is some newer version that might resolve the problem. This particular customer contacted his email vendor and asked them to analyze the problem. The actual log data is gigantic, so I just pasted in the vendor's narrative below in case it helps. Otherwise, as I said, I will capture a similar interaction at the wire level and follow up here. Vendor's narrative: You had 8 messages in the Inbox, and moved two to Trash. The server notified the app on the push connection (for Inbox) that there are now 6 messages. By the time the app got around to catch up with that (it did the Trash folder first), the server said "oh wait, there are 8 messages, not 6" (in the Inbox), and so the app synced them in, just as reported. app selecting the Inbox, server saying there are 8 messages there app moving UID 729 and 730 to Trash (push) server saying Trash now has 34 messages (push) server saying two messages gone from Inbox, 6 left (push) 6 messages left in Inbox becoming 8 (push) and here they are (the two new ones) Thanks, Ron

3 3

fts-solr: Returning 400 on searches; unescaped braces
by Michael Welsh Duggan 17 Jan '17

17 Jan '17

Using Debian, dovecot-solr 1:2.2.26.0-4, and solr-tomcat 3.6.2+dfsg-9, I am getting 400 errors when doing searches. Here is an example search query from dovecot that failed (captured with wireshark): Frame 23: 338 bytes on wire (2704 bits), 338 bytes captured (2704 bits) on interface 0 Linux cooked capture Internet Protocol Version 6, Src: ::1, Dst: ::1 Transmission Control Protocol, Src Port: 56860, Dst Port: 8080, Seq: 1, Ack: 1, Len: 250 Hypertext Transfer Protocol GET /solr/select?fl=uid,score&rows=2664&sort=uid+asc&q={!lucene+q.op%3dAND}(hdr:test+OR+body:test)&fq=%2Bbox:6d5de009f991854df726000012cf7b9c+%2Buser:md5i HTTP/1.1\r\n Host: localhost:8080\r\n Date: Mon, 19 Dec 2016 00:25:56 GMT\r\n Connection: Keep-Alive\r\n \r\n [Full request URI: http://localhost:8080/solr/select?fl=uid,score&rows=2664&sort=uid+asc&q={!l…] [HTTP request 1/1] [Response in frame: 25] Here is the same query from firefox, which succeeds: Frame 66: 646 bytes on wire (5168 bits), 646 bytes captured (5168 bits) on interface 0 Linux cooked capture Internet Protocol Version 6, Src: ::1, Dst: ::1 Transmission Control Protocol, Src Port: 56862, Dst Port: 8080, Seq: 1, Ack: 1, Len: 558 Hypertext Transfer Protocol GET /solr/select?fl=uid,score&rows=2664&sort=uid+asc&q=%7B!lucene+q.op%3DAND%7D(hdr:test+OR+body:test)&fq=%2Bbox:6d5de009f991854df726000012cf7b9c+%2Buser:md5i HTTP/1.1\r\n Host: localhost:8080\r\n Connection: keep-alive\r\n Cache-Control: max-age=0\r\n Upgrade-Insecure-Requests: 1\r\n User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.75 Safari/537.36\r\n Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\n DNT: 1\r\n Accept-Encoding: gzip, deflate, sdch, br\r\n Accept-Language: en-US,en;q=0.8\r\n \r\n [Full request URI: http://localhost:8080/solr/select?fl=uid,score&rows=2664&sort=uid+asc&q=%7B…] [HTTP request 1/1] [Response in frame: 86] The salient difference seems to be the encoding of the braces. Indeed in the tomcat 8 logs, I find the following which seems to corroborate my hypothesis: java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986 at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:467) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:667) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:789) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1437) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Indeed the braces are listed in fts-backend-solr.c as part of solr_escape_chars, so don't know how the braces are making it through unencoded. -- Michael Welsh Duggan (md5i(a)md5i.com)

2 3

Replication: Can't unsubscribe from shared mailbox
by Karsten Heiken 16 Jan '17

16 Jan '17

Hi, I am running two dovecot servers active/active. Everything runs pretty great, except for the replication of subscriptions in a shared namespace. When I unsubscribe from a folder the subscription instantly re-appears. The timestamp on the subscriptions file is updated, but the entry is still in there. If the other node is shut down, everything works as expected, which leads me to believe that the subscription is re-applied upon a replicator-run. I attached a tcpdump of a dsync run. The only action that was executed was to unsubscribe from the folder shared/weinrot(a)luis.uni-hannover.de/INBOX. Is there any other way I might help debugging this? Thanks, Karsten # 2.2.24 (a82c823): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.13 (7b14904) # OS: Linux 2.6.32-44-pve x86_64 Debian 8.5 default_vsz_limit = 512 M dict { acl = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } imap_max_line_length = 2 M lmtp_rcpt_check_quota = yes mail_attribute_dict = file:%h/Maildir/dovecot-attributes mail_gid = 7777 mail_location = maildir:%h/Maildir:LAYOUT=fs:DIRNAME=maiLdir mail_plugins = " zlib quota acl notify replication " mail_shared_explicit_inbox = yes mail_uid = 7777 mailbox_list_index = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext imapflags notify namespace { hidden = no ignore_on_failure = no inbox = no list = children location = maildir:%%h/Maildir:INDEXPVT=%h/shared/%%u:LAYOUT=fs:DIRNAME=maiLdir prefix = shared/%%u/ separator = / subscriptions = yes type = shared } namespace inbox { hidden = no inbox = yes list = yes location = mailbox 30dTrash { auto = subscribe autoexpunge = 30 days special_use = \Junk } mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / subscriptions = yes type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile acl_shared_dict = proxy::acl mail_replica = tcp:10.6.1.10:24245 quota = maildir:Postfach-Limit quota_grace = 2%% quota_rule = *:storage=8G quota_status_nouser = DUNNO quota_status_overquota = 552 5.2.2 Mailbox is full quota_status_success = DUNNO sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_extensions = +notify +imapflags sieve_max_actions = 250 sieve_max_redirects = 200 zlib_save = gz zlib_save_level = 6 } protocols = " imap lmtp sieve pop3 sieve" replication_dsync_parameters = -d -l 300 -U -N service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } } service auth { unix_listener auth-userdb { group = vmail user = vmail } } service dict { unix_listener dict { group = vmail mode = 0660 user = vmail } } service doveadm { inet_listener { port = 24245 } } service imap-login { process_min_avail = 16 service_count = 0 } service imap { executable = imap postlogin process_limit = 30000 } service lmtp { inet_listener lmtp { port = 24 } vsz_limit = 1 G } service managesieve-login { inet_listener sieve { port = 4190 } } service managesieve { process_limit = 1024 } service pop3-login { process_min_avail = 4 service_count = 0 } service pop3 { process_limit = 2500 } service postlogin { executable = script-login -d rawlog /usr/local/bin/dovecot-postlogin.sh } service quota-status { client_limit = 1 executable = quota-status -p postfix inet_listener { port = 12340 } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 user = vmail } } shutdown_clients = no ssl_cert = </etc/dovecot/private/fullchain.pem ssl_key = </etc/dovecot/private/privkey.pem syslog_facility = local6 userdb { args = /etc/dovecot/dovecot-ldap.conf.ext default_fields = home=/var/vmail/%Ld/%Ln quota_rule=*:bytes=8589934592 driver = ldap } verbose_proctitle = yes protocol lmtp { mail_plugins = " zlib quota acl notify replication sieve quota" } protocol lda { mail_plugins = " zlib quota acl notify replication sieve" } protocol imap { imap_metadata = yes mail_max_userip_connections = 100 mail_plugins = " zlib quota acl notify replication imap_quota imap_acl" } protocol pop3 { mail_max_userip_connections = 10 mail_plugins = " zlib quota acl notify replication" }

2 2

Virtual POP3 namespaces and quota-status
by Aurélien Beaujean 09 Jan '17

09 Jan '17

Hi, I've tried to do configure Virtual POP3 INBOX (dovecot version 2.2.18) as explained here http://wiki.dovecot.org/Plugins/Virtual with a static userdb and nopassword=y so I dont have any database on the dovecot server without the possibility to distinguish POP3 or IMAP in the userdb extra field to change the namespace on the fly. So I've put the namespace virtual informations inside the protocol pop3 { ... }, I did the same for the protocol imap { }: protocol pop3 { mail_plugins = $mail_plugins last_login virtual namespace inbox { prefix = separator = / list = yes } namespace virtual { inbox = yes prefix = virtual/ separator = / location = virtual:/etc/dovecot/virtual:INDEX=MEMORY list = no hidden = yes } namespace real { prefix = RealMails/ separator = / list = no hidden = yes } } protocol imap { mail_plugins = $mail_plugins imap_quota last_login namespace inbox { inbox = yes } } protocol lda { mail_plugins = " quota sieve" namespace inbox { inbox = yes } } passdb { args = nopassword=y uid=vmail gid=vmail home=/var/mail/%2Mn/%2.2Mn/%n driver = static } userdb { args = nopassword=y uid=vmail gid=vmail home=/var/mail/%2Mn/%2.2Mn/%n driver = static } service quota-status { executable = quota-status -p postfix inet_listener { port = 12340 } process_min_avail = 8 } But the service quota-status getting lost with a message: namespace configuration error: inbox=yes namespace missing I didn't find a way to tell quota-status which namespace to use as you can't setup a default one. If I declare a "global" namespace inbox pop3 deamon fails because of duplicated namespace inbox in the configuration files: pop3(xxx): Error: User initialization failed: namespace configuration error: There can be only one namespace with inbox=yes Any idea before I will patch some piece of code to disable the unique restriction of inbox namespace ? Thanks, -- Auré

1 2

dsync for subscription on public/shared folder
by Mike Fröhner 09 Jan '17

09 Jan '17

Hello, I already wrote regarding that, but maybe it sounded too complex. I will write it a bit softer now. The replication through dsnyc works perfectly with one exception. I am having an issue with the replication of the subscriptions file on shared folder. We have a public mailbox defined: namespace public-test { location = maildir:/opt/mail/_public/test prefix = public/test/ separator = / subscriptions = no type = public } There 2 subfolder .Test1 and .Test2 which I would like to subscribe on my user "ldaptestuser" which is part of the group "ldaptestgroup" The dovecot-acl file on /opt/mail/_public/test and /opt/mail/_public/test/.Test{1,2} are the same and are containing: group=ldaptestgroup lrwstipekxa Now if I login through my client (Thunderbird or Roundcubemail) I can see the following debug log: dovecot: imap(ldaptestuser): Debug: acl: acl username = ldaptestuser dovecot: imap(ldaptestuser): Debug: acl: owner = 0 dovecot: imap(ldaptestuser): Debug: acl: group added: domain users dovecot: imap(ldaptestuser): Debug: acl: group added: ldaptestgroup dovecot: imap(ldaptestuser): Debug: acl: group added: all_mailuser I created one new private folder on the mailbox of my test user and also subscribed it. This subscription is successfully replicated. After that I am trying to subscribe the public folder. I can see the folder in my client after I subscribed them. But on the 2nd dovecot server the subscription file is not updated. imap-1 # cat subscription Private1 public/test/Test1 imap-2 # cat subscription Private1 The debug log on both dovecot server tells me: dovecot: doveadm(<ip>,ldaptestuser): Debug: Effective uid=834603987, gid=991, home=/opt/mail/ldaptestuser dovecot: doveadm(<ip>,ldaptestuser): Debug: acl: No acl_shared_dict setting - shared mailbox listing is disabled dovecot: doveadm(<ip>,ldaptestuser): Debug: maildir++: root=/opt/mail/_public/test, index=, indexpvt=, control=, inbox=, alt= dovecot: doveadm(<ip>,ldaptestuser): Debug: acl: initializing backend with data: vfile dovecot: doveadm(<ip>,ldaptestuser): Debug: acl: acl username = ldaptestuser dovecot: doveadm(<ip>,ldaptestuser): Debug: acl: owner = 0 dovecot: doveadm(<ip>,ldaptestuser): Debug: acl vfile: Global ACLs disabled It looks like the doveadm() cannot find the group or is not checking/initalizing the groups. Anybody any idea if I missconfigured something or is this just a bug? I really would appreciate some help. kind regards, Mike;

1 1

expunging all mailboxes
by Michael Fox 03 Jan '17

03 Jan '17

Evidently, the 'doveadm expunge' command requires a "mailbox" to be defined in the search string. But is there a way tell it all mailboxes for a user? For example, something like: doveadm expunge -u user@domain mailbox ALL savedbefore 30d Thanks, Michael

4 10

two listeners with different "driver = " configs
by mj 03 Jan '17

03 Jan '17

Hi, I would like to have two seperate imap listeners, with different authentication settings, but the mailstore and userbase etc will be identical. I know I can do this: > service imap-login { > inet_listener imap { > port = 143 > } > inet_listener imap2 { > port = 144 > } > } But I'm unsure how to configure imap/143 with "driver = ldap" and imap2/144 with "driver = pam" Just to explain why i would like this: I am using pam-script-saml (https://github.com/ck-ws/pam-script-saml) to enable saml-based access to dovecot. I would like to have one listener 144 to only serve this saml authentication listener, and the regular 143 listener with driver = ldap. Is that config possible? Best regards, MJ

5 10

Re: FreeBSD / dovecot 2.2.27 / libwrap
by Jim Pazarena 30 Dec '16

30 Dec '16

It works ! It was THAT easy ! Can you suggest how to replace the hair I pulled out ? :-) On 2016-12-29 5:27 PM, Larry Rosenman wrote: > login_access_sockets = tcpwrap > > service tcpwrap { > unix_listener login/tcpwrap { > group = $default_login_user > mode = 0600 > user = $default_login_user > } > } > > > > On Thu, Dec 29, 2016 at 7:21 PM, Jim Pazarena <dovecot(a)paz.bz> wrote: > >> I have compiled dovecot2 for FreeBSD with the tcpwrap option. >> >> A tcpwrap binary gets built and resides in the FreeBSD directory >> /usr/local/libexec/dovecot >> >> an examination of the compiled options (using the FreeBSD pkg install >> dovecot2) confirms: LIBWRAP : on >> >> yet, when I adjust dovecot.conf with: login_access_sockets = tcpwrap >> >> I get the following logged error message: >> >> 20161229 17:02:49 imap-login: Error: connect(tcpwrap) failed: No such file >> or directory >> >> Is there any way to turn up some super logging so that I can find just >> what dovecot feels is the failure ? >> >> Or, does anyone have an idea how to figure this out? What little hair I >> have is rapidly getting pulled out in frustration! >> >> Thank you. >> >> >> >> -- >> Jim Pazarena dovecot(a)paz.bz >> > > -- Jim Pazarena dovecot(a)paz.bz

2 1

FreeBSD / dovecot 2.2.27 / libwrap
by Jim Pazarena 30 Dec '16

30 Dec '16

I have compiled dovecot2 for FreeBSD with the tcpwrap option. A tcpwrap binary gets built and resides in the FreeBSD directory /usr/local/libexec/dovecot an examination of the compiled options (using the FreeBSD pkg install dovecot2) confirms: LIBWRAP : on yet, when I adjust dovecot.conf with: login_access_sockets = tcpwrap I get the following logged error message: 20161229 17:02:49 imap-login: Error: connect(tcpwrap) failed: No such file or directory Is there any way to turn up some super logging so that I can find just what dovecot feels is the failure ? Or, does anyone have an idea how to figure this out? What little hair I have is rapidly getting pulled out in frustration! Thank you. -- Jim Pazarena dovecot(a)paz.bz

2 1

doveadm expunge -A Error: Dictionary commit failed
by Alessio Cecchi 29 Dec '16

29 Dec '16

Hi, with the latest dovecot-ee version (dovecot-ee-2.2.26.1-10) if I run “doveadm expunge -A mailbox Spam savedbefore 30d” dovecot an error: doveadm: Error: dict-client: Commit failed: Dict server timeout: No input for 1916.209 secs (1 commands pending, oldest sent 0.000 secs ago: C1) (reply took 0.000 secs) doveadm: Error: expire: Dictionary commit failed Probably also "doveadm quota recalc -A" fail. Everything worked fine up to version 2.2.24 I hope can be fixed. Thanks -- Alessio Cecchi Postmaster AT http://www.qboxmail.it http://www.linkedin.com/in/alessice

3 6