- dovecot - dovecot.org

Re: SOLR/Index?
by Larry Rosenman 15 Apr '19

15 Apr '19

full solr.log at: https://www.lerctr.org/~ler/solr.log The search DOES make it to SOLR: ⌂77% [ler@thebighonker.lerctr.org:~] 130 $ grep sysutils /var/log/solr/solr.log 2019-04-14 18:31:34.749 INFO (qtp349420578-7538) [ x:dovecot] o.a.s.c.S.Request [dovecot] webapp=/solr path=/select params={q={!lucene+q.op%3DAND}(hdr:sysutils\/+OR+body:sysutils\/)&fl=uid,score&sort=uid+asc&fq=%2Bbox:8097632f69627b5b58950000bbe98eac+%2Buser:ler&rows=22644&wt=xml} hits=0 status=0 QTime=460 Pick showing subjects: https://www.lerctr.org/~ler/sysutils_mail.png What else? I'm happy to provide access. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

3 11

SOLR/Index?
by Larry Rosenman 14 Apr '19

14 Apr '19

Given all the discussion on FTS/Solr, etc, I have a question: I have autoindex set, and searching in roundcube most of the time does NOT find things, HOWEVER if I do: doveadm fts rescan doveadm index I can find things in the mailboxes. WHY? (doveconf -n attached). -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

2 1

Re: dovecot Digest, Vol 192, Issue 52
by Peter Mogensen 14 Apr '19

14 Apr '19

On 4/14/19 4:04 PM, dovecot-request(a)dovecot.org wrote: >> Solr ships with autoCommit set to 15 seconds and openSearcher set to >> false on the autoCommit.? The autoSoftCommit setting is not enabled by >> default, but depending on how the index was created, Solr might try to >> set autoSoftCommit to 3 seconds ... which is WAY too short. I just run with the default. 15s autoCommit and no autoSoftCommit >> This thread says that dovecot is sending explicit commits.? I see explicit /update req. with softCommit and waitSearcer=true in a tcpdump. >> One thing >> that might be happening to exceed 60 seconds is an extremely long >> commit, which is usually caused by excessive cache autowarming, but >> might be related to insufficient memory.? The max heap setting on an >> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY >> small, and it doesn't take much index data before a much larger heap >> is required. I run with SOLR_JAVA_MEM="-Xmx8g -Xms2g" > I looked into the code (version 2.3.5.1): This is 2.2.35. I haven't checked the source difference to 2.3.x I must admit. > I immagine that one of the reasons dovecot sends softCommits is because > without autoindex active and even if mailboxes are periodically indexed > from cron, the last emails received with be indexed at the moment of the > search.? I expect that dovecot has to because of it's default behavior by only bringing the index up-to-date just before search. So it has towait for the index result to be available if there's been any new mails indexed. > 1) a configurable batch size would enable to tune the number of emails > per request and help stay under the 60 seconds hard coded http request > timeout. A configurable http timeout would be less useful, since this > will potentially run into other timeouts on solr side. Being able to configure it is great. But I don't think it solves much. I recompiled with 100 as batch size and it still ended in timeouts. Then I recompiled with 10min timeout and now I see all the batches completing and their processesing time is mostly between 1 and 2 minutes (so all would have failed). To me it looks like Solr just takes too long time to index. This is no small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz and for this test it's not doing anything else, so I'm a bit surprised that even with only a few users this takes so long time. /Peter

2 1

Re: Solr connection timeout hardwired to 60s
by Peter Mogensen 14 Apr '19

14 Apr '19

On 4/4/19 6:47 PM, dovecot-request(a)dovecot.org wrote: > For a typical Solr index, 60 seconds is an eternity. Most people aim > for query times of 100 milliseconds or less, and they often achieve > that goal. I'm pretty sure I get these while indexing, not querying. Apr 04 16:44:50 host dovecot[114690]: indexer-worker(me(a)example.com): Error: fts_solr: Indexing failed: Request timed out (Request queued 66.015 secs ago, 1 attempts in 66.005 secs, 63.146 in http ioloop, 0.000 in other ioloops, connected 94.903 secs ago) /Peter

5 8

Fwd: Problem with solr working, but not indexing
by TG Servers 14 Apr '19

14 Apr '19

2 2

Problem with solr working, but not indexing
by TG Servers 14 Apr '19

14 Apr '19

2 1

Re: Solr connection timeout hardwired to 60s
by Peter Mogensen 14 Apr '19

14 Apr '19

sorry... I got distracted half way and forgot to put a meaningfull subject so the archive could figure out the thread. - resending. On 4/14/19 4:04 PM, dovecot-request(a)dovecot.org wrote: >> Solr ships with autoCommit set to 15 seconds and openSearcher set to >> false on the autoCommit.? The autoSoftCommit setting is not enabled by >> default, but depending on how the index was created, Solr might try to >> set autoSoftCommit to 3 seconds ... which is WAY too short. I just run with the default. 15s autoCommit and no autoSoftCommit >> This thread says that dovecot is sending explicit commits.? I see explicit /update req. with softCommit and waitSearcer=true in a tcpdump. >> One thing >> that might be happening to exceed 60 seconds is an extremely long >> commit, which is usually caused by excessive cache autowarming, but >> might be related to insufficient memory.? The max heap setting on an >> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY >> small, and it doesn't take much index data before a much larger heap >> is required. I run with SOLR_JAVA_MEM="-Xmx8g -Xms2g" > I looked into the code (version 2.3.5.1): This is 2.2.35. I haven't checked the source difference to 2.3.x I must admit. > I immagine that one of the reasons dovecot sends softCommits is because > without autoindex active and even if mailboxes are periodically indexed > from cron, the last emails received with be indexed at the moment of the > search.? I expect that dovecot has to because of it's default behavior by only bringing the index up-to-date just before search. So it has towait for the index result to be available if there's been any new mails indexed. > 1) a configurable batch size would enable to tune the number of emails > per request and help stay under the 60 seconds hard coded http request > timeout. A configurable http timeout would be less useful, since this > will potentially run into other timeouts on solr side. Being able to configure it is great. But I don't think it solves much. I recompiled with 100 as batch size and it still ended in timeouts. Then I recompiled with 10min timeout and now I see all the batches completing and their processesing time is mostly between 1 and 2 minutes (so all would have failed). To me it looks like Solr just takes too long time to index. This is no small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz and for this test it's not doing anything else, so I'm a bit surprised that even with only a few users this takes so long time. /Peter

2 2

Re: Mail account brute force / harassment
by Odhiambo Washington 14 Apr '19

14 Apr '19

On Thu, 11 Apr 2019 at 13:24, Marc Roos via dovecot <dovecot(a)dovecot.org> wrote: > > > Say for instance you have some one trying to constantly access an > account > > > Has any of you made something creative like this: > > * configure that account to allow to login with any password > * link that account to something like /dev/zero that generates infinite > amount of messages > (maybe send an archive of virusses?) > * transferring TB's of data to this harassing client. > > I think it would be interesting to be able to do such a thing. > > Instead of being evil, just use fail2ban to address this problem :-) -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", grep ^[^#] :-)

10 24

permissions for quota-status
by Andreas Meyer 14 Apr '19

14 Apr '19

Hello! doveconf --version 2.2.36 I want to move to a new server with dovecot but get an error Error: service(quota-status): listen(*, 12340) failed: Permission denied when I define service quota-status { executable = quota-status -p postfix inet_listener { port = 12340 } client_limit = 1 } private]# ll insgesamt 0 srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 anvil srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 bounce srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 defer srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 discard srw-------. 1 postfix postfix 0 14. Apr 15:08 dovecot-lmtp srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 error srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 lmtp srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 local srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 proxymap srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 proxywrite srw-rw----. 1 postfix postfix 0 14. Apr 14:58 quota-status srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 relay srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 retry srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 rewrite srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 scache srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 smtp srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 tlsmgr srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 trace srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 verify srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 virtual Don't know what's wrong. Can someone help? Kind regards Andreas -- PGP-Fingerprint: D392 5D21 0299 63D7 5BAE 4562 1E56 B2EA 81A2 59F1

2 1

nginx configuration to pass x-originating-ip
by André Rodier 14 Apr '19

14 Apr '19

Hello, There is a bug in SOGo, as it sends the original IP after successful login, and not before the login process. I traced the bug to the source code. https://sogo.nu/bugs/view.php?id=2979. Then, in my research, I found this old thread: https://forum.nginx.org/read.php?29,237299,237367#msg-237367 Can I use Nginx as an IMAP proxy to add the missing ID. I suspect this is something that can be achieved? I already have an Nginx for the SOGo front-end, so I would add another one for the IMAP proxy. With a little bit of luck, I should be able to pass the original IP between the two servers Does anyone has a working example? Thanks, André -- André Rodier HomeBox: https://github.com/progmaticltd/homebox

1 0