- dovecot - dovecot.org

No CRLF in Pigeonhole's header?
by Wojciech Sychut 15 Apr '19

15 Apr '19

Hello, I have been told on Freenode's #dovecot to ask here. I have Exim 4.92 complaining about "Non-CRLF-terminated header, under CHUNKING: message abandoned" with redirects made by Pigeonhole 0.5.5 (with Dovecot 2.3.5.1). Please find relevant part of the traffic below, and help me determine whether Exim is correct on this. 0x0000: 4500 0954 0000 4000 8006 0000 7f00 0001 E..T..@......... 0x0010: 7f00 0001 6ea2 0019 8982 e279 d090 b820 ....n......y.... 0x0020: 8018 021e 0749 0000 0101 080a c345 2b51 .....I.......E+Q 0x0030: b382 853c 4d41 494c 2046 524f 4d3a 3c72 ...<MAIL.FROM:<r 0x0040: 4073 3066 2e6d 642e 6465 7664 6576 6465 @sxf.md.devdevde 0x0050: 762e 6e65 743e 0d0a 5243 5054 2054 4f3a v.net>..RCPT.TO: 0x0060: 3c63 6f6e 7461 6374 4072 6567 6973 2e74 <contact(a)regis.t 0x0070: 6563 683e 0d0a 4244 4154 2032 3235 3420 ech>..BDAT.2254. 0x0080: 4c41 5354 0d0a 582d 5369 6576 653a 2050 LAST..X-Sieve:.P 0x0090: 6967 656f 6e68 6f6c 6520 5369 6576 6520 igeonhole.Sieve. 0x00a0: 302e 352e 3520 2832 3438 3362 3038 3529 0.5.5.(2483b085) 0x00b0: 0a58 2d53 6965 7665 2d52 6564 6972 6563 .X-Sieve-Redirec 0x00c0: 7465 642d 4672 6f6d 3a20 7240 7330 662e ted-From:.r@sxf. With Dovecot 2.2.36 + Pigeonhole 0.4.24.1 and required minimum of version-related configuration changes - redirection works properly. Related Exim code fragment: https://github.com/Exim/exim/blob/master/src/src/receive.c#L2222 Thank you, -- Wojtek Sychut

2 1

rsync sdbox, no new mails?
by Jakob Curdes 15 Apr '19

15 Apr '19

Hello, I had to restore a mailserver from a backup to replace a system with problems that is still live. I did a 1:1 restore as far as possible from a backup, then rsynced the /var/vmail/ directory, where we have all the mailboxes, in sdbox format. So both directories should be the same. After syncing, I started dovecot. I can see the mails until to the point of the backup, but not the newer ones ,although they seem to be present in the "cur" directory. I thought with sdbox it is all in the index files that have been transferred with the mails? Or am I missing a place that has to be synced? Thank you for hints, Jakob

2 2

LDAP iterate (dovecoc 2.2.27)
by Debreczeni Attila 15 Apr '19

15 Apr '19

Dear Dovecot Team, We would like to use the LDAP iterate for dovecot replication, but I got this error: dovecot: auth-worker(10149): Error: ldap(*): ldap_search(base=ou=mail,o=asd,c=TLD filter=(objectClass=inetMailUser) failed: Size limit exceeded dovecot: replicator: Error: User listing returned failure dovecot: replicator: Error: listing users failed, can't replicate existing data When I check the "doveadm replicator status": ... Total number of known users 1200 But we have more than 10.000 users. When I run the ldapsearch manually, I get the full list of the mail addresses. Who can I fix it? Thank you, Attila # dovecot --version 2.2.27 (c0f36b0) # dovecot -n | filtered # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.8 auth_worker_max_count = 150 default_vsz_limit = 512 M doveadm_password = # hidden, use -P to show it doveadm_port = *** mail_gid = vmail mail_plugins = " quota notify replication" namespace inbox { ... } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { mail_replica = tcp:REMOTE_IP } replication_dsync_parameters = -d -N -1 -l 30 -U service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } } service auth { inet_listener { address = LOCAL_IP port = LOCAL_PORT } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service dict { unix_listener dict { group = vmail mode = 0600 user = vmail } } service lmtp { inet_listener lmtp { address = LOCAL_IP port = LOCAL_PORT } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } userdb { args = /etc/dovecot/dovecot-domain-ldap.conf.ext driver = ldap result_success = continue-ok } userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap result_failure = return-fail } protocol imap { ... } protocol pop3 { ... } # grep -v '^ *$#.*$\?$' dovecot-ldap.conf.ext uris = ldaps://ldap.URL dn = uid=auth,o=domain,c=TLD dnpass = **** sasl_bind = no tls_ca_cert_dir = /etc/ssl/certs tls_require_cert = demand ldap_version = 3 base = ou=mail,o=asd,c=TLD deref = never scope = subtree user_attrs = =home=/dev/null/%Ld/%L{ldap:uid}, mailQuota=quota_rule=*:bytes=%$ user_filter = (objectClass=inetMailUser) pass_attrs = userPassword=password pass_filter = (objectClass=inetMailUser) iterate_attrs = mail=user iterate_filter = (objectClass=inetMailUser) default_pass_scheme = PLAIN

1 0

LDAP iterate (dovecoc 2.2.27)
by Debreczeni Attila 15 Apr '19

15 Apr '19

Dear Dovecot Team, We would like to use the LDAP iterate for dovecot replication, but I got this error: dovecot: auth-worker(10149): Error: ldap(*): ldap_search(base=ou=mail,o=asd,c=TLD filter=(objectClass=inetMailUser) failed: Size limit exceeded dovecot: replicator: Error: User listing returned failure dovecot: replicator: Error: listing users failed, can't replicate existing data When I check the "doveadm replicator status": ... Total number of known users 1200 But we have more than 10.000 users. When I run the ldapsearch manually, I get the full list of the mail addresses. Who can I fix it? Thank you, Attila # dovecot --version 2.2.27 (c0f36b0) # dovecot -n | filtered # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.8 auth_worker_max_count = 150 default_vsz_limit = 512 M doveadm_password = # hidden, use -P to show it doveadm_port = *** mail_gid = vmail mail_plugins = " quota notify replication" namespace inbox { ... } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { mail_replica = tcp:REMOTE_IP } replication_dsync_parameters = -d -N -1 -l 30 -U service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } } service auth { inet_listener { address = LOCAL_IP port = LOCAL_PORT } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service dict { unix_listener dict { group = vmail mode = 0600 user = vmail } } service lmtp { inet_listener lmtp { address = LOCAL_IP port = LOCAL_PORT } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } userdb { args = /etc/dovecot/dovecot-domain-ldap.conf.ext driver = ldap result_success = continue-ok } userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap result_failure = return-fail } protocol imap { ... } protocol pop3 { ... } # grep -v '^ *$#.*$\?$' dovecot-ldap.conf.ext uris = ldaps://ldap.URL dn = uid=auth,o=domain,c=TLD dnpass = **** sasl_bind = no tls_ca_cert_dir = /etc/ssl/certs tls_require_cert = demand ldap_version = 3 base = ou=mail,o=asd,c=TLD deref = never scope = subtree user_attrs = =home=/dev/null/%Ld/%L{ldap:uid}, mailQuota=quota_rule=*:bytes=%$ user_filter = (objectClass=inetMailUser) pass_attrs = userPassword=password pass_filter = (objectClass=inetMailUser) iterate_attrs = mail=user iterate_filter = (objectClass=inetMailUser) default_pass_scheme = PLAIN

1 0

Re: SOLR/Index?
by Larry Rosenman 15 Apr '19

15 Apr '19

full solr.log at: https://www.lerctr.org/~ler/solr.log The search DOES make it to SOLR: ⌂77% [ler@thebighonker.lerctr.org:~] 130 $ grep sysutils /var/log/solr/solr.log 2019-04-14 18:31:34.749 INFO (qtp349420578-7538) [ x:dovecot] o.a.s.c.S.Request [dovecot] webapp=/solr path=/select params={q={!lucene+q.op%3DAND}(hdr:sysutils\/+OR+body:sysutils\/)&fl=uid,score&sort=uid+asc&fq=%2Bbox:8097632f69627b5b58950000bbe98eac+%2Buser:ler&rows=22644&wt=xml} hits=0 status=0 QTime=460 Pick showing subjects: https://www.lerctr.org/~ler/sysutils_mail.png What else? I'm happy to provide access. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

3 11

SOLR/Index?
by Larry Rosenman 15 Apr '19

15 Apr '19

Given all the discussion on FTS/Solr, etc, I have a question: I have autoindex set, and searching in roundcube most of the time does NOT find things, HOWEVER if I do: doveadm fts rescan doveadm index I can find things in the mailboxes. WHY? (doveconf -n attached). -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

2 1

Re: dovecot Digest, Vol 192, Issue 52
by Peter Mogensen 15 Apr '19

15 Apr '19

On 4/14/19 4:04 PM, dovecot-request(a)dovecot.org wrote: >> Solr ships with autoCommit set to 15 seconds and openSearcher set to >> false on the autoCommit.? The autoSoftCommit setting is not enabled by >> default, but depending on how the index was created, Solr might try to >> set autoSoftCommit to 3 seconds ... which is WAY too short. I just run with the default. 15s autoCommit and no autoSoftCommit >> This thread says that dovecot is sending explicit commits.? I see explicit /update req. with softCommit and waitSearcer=true in a tcpdump. >> One thing >> that might be happening to exceed 60 seconds is an extremely long >> commit, which is usually caused by excessive cache autowarming, but >> might be related to insufficient memory.? The max heap setting on an >> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY >> small, and it doesn't take much index data before a much larger heap >> is required. I run with SOLR_JAVA_MEM="-Xmx8g -Xms2g" > I looked into the code (version 2.3.5.1): This is 2.2.35. I haven't checked the source difference to 2.3.x I must admit. > I immagine that one of the reasons dovecot sends softCommits is because > without autoindex active and even if mailboxes are periodically indexed > from cron, the last emails received with be indexed at the moment of the > search.? I expect that dovecot has to because of it's default behavior by only bringing the index up-to-date just before search. So it has towait for the index result to be available if there's been any new mails indexed. > 1) a configurable batch size would enable to tune the number of emails > per request and help stay under the 60 seconds hard coded http request > timeout. A configurable http timeout would be less useful, since this > will potentially run into other timeouts on solr side. Being able to configure it is great. But I don't think it solves much. I recompiled with 100 as batch size and it still ended in timeouts. Then I recompiled with 10min timeout and now I see all the batches completing and their processesing time is mostly between 1 and 2 minutes (so all would have failed). To me it looks like Solr just takes too long time to index. This is no small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz and for this test it's not doing anything else, so I'm a bit surprised that even with only a few users this takes so long time. /Peter

2 1

Re: Solr connection timeout hardwired to 60s
by Peter Mogensen 15 Apr '19

15 Apr '19

On 4/4/19 6:47 PM, dovecot-request(a)dovecot.org wrote: > For a typical Solr index, 60 seconds is an eternity. Most people aim > for query times of 100 milliseconds or less, and they often achieve > that goal. I'm pretty sure I get these while indexing, not querying. Apr 04 16:44:50 host dovecot[114690]: indexer-worker(me(a)example.com): Error: fts_solr: Indexing failed: Request timed out (Request queued 66.015 secs ago, 1 attempts in 66.005 secs, 63.146 in http ioloop, 0.000 in other ioloops, connected 94.903 secs ago) /Peter

5 8

Fwd: Problem with solr working, but not indexing
by TG Servers 14 Apr '19

14 Apr '19

2 2

Problem with solr working, but not indexing
by TG Servers 14 Apr '19

14 Apr '19

2 1