- dovecot - dovecot.org

rsync sdbox, no new mails?
by Jakob Curdes 15 Apr '19

15 Apr '19

Hello, I had to restore a mailserver from a backup to replace a system with problems that is still live. I did a 1:1 restore as far as possible from a backup, then rsynced the /var/vmail/ directory, where we have all the mailboxes, in sdbox format. So both directories should be the same. After syncing, I started dovecot. I can see the mails until to the point of the backup, but not the newer ones ,although they seem to be present in the "cur" directory. I thought with sdbox it is all in the index files that have been transferred with the mails? Or am I missing a place that has to be synced? Thank you for hints, Jakob

2 2

LDAP iterate (dovecoc 2.2.27)
by Debreczeni Attila 15 Apr '19

15 Apr '19

Dear Dovecot Team, We would like to use the LDAP iterate for dovecot replication, but I got this error: dovecot: auth-worker(10149): Error: ldap(*): ldap_search(base=ou=mail,o=asd,c=TLD filter=(objectClass=inetMailUser) failed: Size limit exceeded dovecot: replicator: Error: User listing returned failure dovecot: replicator: Error: listing users failed, can't replicate existing data When I check the "doveadm replicator status": ... Total number of known users 1200 But we have more than 10.000 users. When I run the ldapsearch manually, I get the full list of the mail addresses. Who can I fix it? Thank you, Attila # dovecot --version 2.2.27 (c0f36b0) # dovecot -n | filtered # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.8 auth_worker_max_count = 150 default_vsz_limit = 512 M doveadm_password = # hidden, use -P to show it doveadm_port = *** mail_gid = vmail mail_plugins = " quota notify replication" namespace inbox { ... } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { mail_replica = tcp:REMOTE_IP } replication_dsync_parameters = -d -N -1 -l 30 -U service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } } service auth { inet_listener { address = LOCAL_IP port = LOCAL_PORT } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service dict { unix_listener dict { group = vmail mode = 0600 user = vmail } } service lmtp { inet_listener lmtp { address = LOCAL_IP port = LOCAL_PORT } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } userdb { args = /etc/dovecot/dovecot-domain-ldap.conf.ext driver = ldap result_success = continue-ok } userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap result_failure = return-fail } protocol imap { ... } protocol pop3 { ... } # grep -v '^ *$#.*$\?$' dovecot-ldap.conf.ext uris = ldaps://ldap.URL dn = uid=auth,o=domain,c=TLD dnpass = **** sasl_bind = no tls_ca_cert_dir = /etc/ssl/certs tls_require_cert = demand ldap_version = 3 base = ou=mail,o=asd,c=TLD deref = never scope = subtree user_attrs = =home=/dev/null/%Ld/%L{ldap:uid}, mailQuota=quota_rule=*:bytes=%$ user_filter = (objectClass=inetMailUser) pass_attrs = userPassword=password pass_filter = (objectClass=inetMailUser) iterate_attrs = mail=user iterate_filter = (objectClass=inetMailUser) default_pass_scheme = PLAIN

1 0

LDAP iterate (dovecoc 2.2.27)
by Debreczeni Attila 15 Apr '19

15 Apr '19

Dear Dovecot Team, We would like to use the LDAP iterate for dovecot replication, but I got this error: dovecot: auth-worker(10149): Error: ldap(*): ldap_search(base=ou=mail,o=asd,c=TLD filter=(objectClass=inetMailUser) failed: Size limit exceeded dovecot: replicator: Error: User listing returned failure dovecot: replicator: Error: listing users failed, can't replicate existing data When I check the "doveadm replicator status": ... Total number of known users 1200 But we have more than 10.000 users. When I run the ldapsearch manually, I get the full list of the mail addresses. Who can I fix it? Thank you, Attila # dovecot --version 2.2.27 (c0f36b0) # dovecot -n | filtered # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.8 auth_worker_max_count = 150 default_vsz_limit = 512 M doveadm_password = # hidden, use -P to show it doveadm_port = *** mail_gid = vmail mail_plugins = " quota notify replication" namespace inbox { ... } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { mail_replica = tcp:REMOTE_IP } replication_dsync_parameters = -d -N -1 -l 30 -U service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } } service auth { inet_listener { address = LOCAL_IP port = LOCAL_PORT } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service dict { unix_listener dict { group = vmail mode = 0600 user = vmail } } service lmtp { inet_listener lmtp { address = LOCAL_IP port = LOCAL_PORT } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } userdb { args = /etc/dovecot/dovecot-domain-ldap.conf.ext driver = ldap result_success = continue-ok } userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap result_failure = return-fail } protocol imap { ... } protocol pop3 { ... } # grep -v '^ *$#.*$\?$' dovecot-ldap.conf.ext uris = ldaps://ldap.URL dn = uid=auth,o=domain,c=TLD dnpass = **** sasl_bind = no tls_ca_cert_dir = /etc/ssl/certs tls_require_cert = demand ldap_version = 3 base = ou=mail,o=asd,c=TLD deref = never scope = subtree user_attrs = =home=/dev/null/%Ld/%L{ldap:uid}, mailQuota=quota_rule=*:bytes=%$ user_filter = (objectClass=inetMailUser) pass_attrs = userPassword=password pass_filter = (objectClass=inetMailUser) iterate_attrs = mail=user iterate_filter = (objectClass=inetMailUser) default_pass_scheme = PLAIN

1 0

Re: SOLR/Index?
by Larry Rosenman 15 Apr '19

15 Apr '19

full solr.log at: https://www.lerctr.org/~ler/solr.log The search DOES make it to SOLR: ⌂77% [ler@thebighonker.lerctr.org:~] 130 $ grep sysutils /var/log/solr/solr.log 2019-04-14 18:31:34.749 INFO (qtp349420578-7538) [ x:dovecot] o.a.s.c.S.Request [dovecot] webapp=/solr path=/select params={q={!lucene+q.op%3DAND}(hdr:sysutils\/+OR+body:sysutils\/)&fl=uid,score&sort=uid+asc&fq=%2Bbox:8097632f69627b5b58950000bbe98eac+%2Buser:ler&rows=22644&wt=xml} hits=0 status=0 QTime=460 Pick showing subjects: https://www.lerctr.org/~ler/sysutils_mail.png What else? I'm happy to provide access. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

3 11

SOLR/Index?
by Larry Rosenman 15 Apr '19

15 Apr '19

Given all the discussion on FTS/Solr, etc, I have a question: I have autoindex set, and searching in roundcube most of the time does NOT find things, HOWEVER if I do: doveadm fts rescan doveadm index I can find things in the mailboxes. WHY? (doveconf -n attached). -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

2 1

Re: dovecot Digest, Vol 192, Issue 52
by Peter Mogensen 15 Apr '19

15 Apr '19

On 4/14/19 4:04 PM, dovecot-request(a)dovecot.org wrote: >> Solr ships with autoCommit set to 15 seconds and openSearcher set to >> false on the autoCommit.? The autoSoftCommit setting is not enabled by >> default, but depending on how the index was created, Solr might try to >> set autoSoftCommit to 3 seconds ... which is WAY too short. I just run with the default. 15s autoCommit and no autoSoftCommit >> This thread says that dovecot is sending explicit commits.? I see explicit /update req. with softCommit and waitSearcer=true in a tcpdump. >> One thing >> that might be happening to exceed 60 seconds is an extremely long >> commit, which is usually caused by excessive cache autowarming, but >> might be related to insufficient memory.? The max heap setting on an >> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY >> small, and it doesn't take much index data before a much larger heap >> is required. I run with SOLR_JAVA_MEM="-Xmx8g -Xms2g" > I looked into the code (version 2.3.5.1): This is 2.2.35. I haven't checked the source difference to 2.3.x I must admit. > I immagine that one of the reasons dovecot sends softCommits is because > without autoindex active and even if mailboxes are periodically indexed > from cron, the last emails received with be indexed at the moment of the > search.? I expect that dovecot has to because of it's default behavior by only bringing the index up-to-date just before search. So it has towait for the index result to be available if there's been any new mails indexed. > 1) a configurable batch size would enable to tune the number of emails > per request and help stay under the 60 seconds hard coded http request > timeout. A configurable http timeout would be less useful, since this > will potentially run into other timeouts on solr side. Being able to configure it is great. But I don't think it solves much. I recompiled with 100 as batch size and it still ended in timeouts. Then I recompiled with 10min timeout and now I see all the batches completing and their processesing time is mostly between 1 and 2 minutes (so all would have failed). To me it looks like Solr just takes too long time to index. This is no small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz and for this test it's not doing anything else, so I'm a bit surprised that even with only a few users this takes so long time. /Peter

2 1

Re: Solr connection timeout hardwired to 60s
by Peter Mogensen 15 Apr '19

15 Apr '19

On 4/4/19 6:47 PM, dovecot-request(a)dovecot.org wrote: > For a typical Solr index, 60 seconds is an eternity. Most people aim > for query times of 100 milliseconds or less, and they often achieve > that goal. I'm pretty sure I get these while indexing, not querying. Apr 04 16:44:50 host dovecot[114690]: indexer-worker(me(a)example.com): Error: fts_solr: Indexing failed: Request timed out (Request queued 66.015 secs ago, 1 attempts in 66.005 secs, 63.146 in http ioloop, 0.000 in other ioloops, connected 94.903 secs ago) /Peter

5 8

Fwd: Problem with solr working, but not indexing
by TG Servers 14 Apr '19

14 Apr '19

2 2

Problem with solr working, but not indexing
by TG Servers 14 Apr '19

14 Apr '19

2 1

Re: Solr connection timeout hardwired to 60s
by Peter Mogensen 14 Apr '19

14 Apr '19

sorry... I got distracted half way and forgot to put a meaningfull subject so the archive could figure out the thread. - resending. On 4/14/19 4:04 PM, dovecot-request(a)dovecot.org wrote: >> Solr ships with autoCommit set to 15 seconds and openSearcher set to >> false on the autoCommit.? The autoSoftCommit setting is not enabled by >> default, but depending on how the index was created, Solr might try to >> set autoSoftCommit to 3 seconds ... which is WAY too short. I just run with the default. 15s autoCommit and no autoSoftCommit >> This thread says that dovecot is sending explicit commits.? I see explicit /update req. with softCommit and waitSearcer=true in a tcpdump. >> One thing >> that might be happening to exceed 60 seconds is an extremely long >> commit, which is usually caused by excessive cache autowarming, but >> might be related to insufficient memory.? The max heap setting on an >> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY >> small, and it doesn't take much index data before a much larger heap >> is required. I run with SOLR_JAVA_MEM="-Xmx8g -Xms2g" > I looked into the code (version 2.3.5.1): This is 2.2.35. I haven't checked the source difference to 2.3.x I must admit. > I immagine that one of the reasons dovecot sends softCommits is because > without autoindex active and even if mailboxes are periodically indexed > from cron, the last emails received with be indexed at the moment of the > search.? I expect that dovecot has to because of it's default behavior by only bringing the index up-to-date just before search. So it has towait for the index result to be available if there's been any new mails indexed. > 1) a configurable batch size would enable to tune the number of emails > per request and help stay under the 60 seconds hard coded http request > timeout. A configurable http timeout would be less useful, since this > will potentially run into other timeouts on solr side. Being able to configure it is great. But I don't think it solves much. I recompiled with 100 as batch size and it still ended in timeouts. Then I recompiled with 10min timeout and now I see all the batches completing and their processesing time is mostly between 1 and 2 minutes (so all would have failed). To me it looks like Solr just takes too long time to index. This is no small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz and for this test it's not doing anything else, so I'm a bit surprised that even with only a few users this takes so long time. /Peter

2 2