- dovecot - dovecot.org

SOLR/Index?
by Larry Rosenman 15 Apr '19

15 Apr '19

Given all the discussion on FTS/Solr, etc, I have a question: I have autoindex set, and searching in roundcube most of the time does NOT find things, HOWEVER if I do: doveadm fts rescan doveadm index I can find things in the mailboxes. WHY? (doveconf -n attached). -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

2 1

Re: dovecot Digest, Vol 192, Issue 52
by Peter Mogensen 15 Apr '19

15 Apr '19

On 4/14/19 4:04 PM, dovecot-request(a)dovecot.org wrote: >> Solr ships with autoCommit set to 15 seconds and openSearcher set to >> false on the autoCommit.? The autoSoftCommit setting is not enabled by >> default, but depending on how the index was created, Solr might try to >> set autoSoftCommit to 3 seconds ... which is WAY too short. I just run with the default. 15s autoCommit and no autoSoftCommit >> This thread says that dovecot is sending explicit commits.? I see explicit /update req. with softCommit and waitSearcer=true in a tcpdump. >> One thing >> that might be happening to exceed 60 seconds is an extremely long >> commit, which is usually caused by excessive cache autowarming, but >> might be related to insufficient memory.? The max heap setting on an >> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY >> small, and it doesn't take much index data before a much larger heap >> is required. I run with SOLR_JAVA_MEM="-Xmx8g -Xms2g" > I looked into the code (version 2.3.5.1): This is 2.2.35. I haven't checked the source difference to 2.3.x I must admit. > I immagine that one of the reasons dovecot sends softCommits is because > without autoindex active and even if mailboxes are periodically indexed > from cron, the last emails received with be indexed at the moment of the > search.? I expect that dovecot has to because of it's default behavior by only bringing the index up-to-date just before search. So it has towait for the index result to be available if there's been any new mails indexed. > 1) a configurable batch size would enable to tune the number of emails > per request and help stay under the 60 seconds hard coded http request > timeout. A configurable http timeout would be less useful, since this > will potentially run into other timeouts on solr side. Being able to configure it is great. But I don't think it solves much. I recompiled with 100 as batch size and it still ended in timeouts. Then I recompiled with 10min timeout and now I see all the batches completing and their processesing time is mostly between 1 and 2 minutes (so all would have failed). To me it looks like Solr just takes too long time to index. This is no small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz and for this test it's not doing anything else, so I'm a bit surprised that even with only a few users this takes so long time. /Peter

2 1

Re: Solr connection timeout hardwired to 60s
by Peter Mogensen 15 Apr '19

15 Apr '19

On 4/4/19 6:47 PM, dovecot-request(a)dovecot.org wrote: > For a typical Solr index, 60 seconds is an eternity. Most people aim > for query times of 100 milliseconds or less, and they often achieve > that goal. I'm pretty sure I get these while indexing, not querying. Apr 04 16:44:50 host dovecot[114690]: indexer-worker(me(a)example.com): Error: fts_solr: Indexing failed: Request timed out (Request queued 66.015 secs ago, 1 attempts in 66.005 secs, 63.146 in http ioloop, 0.000 in other ioloops, connected 94.903 secs ago) /Peter

5 8

Fwd: Problem with solr working, but not indexing
by TG Servers 14 Apr '19

14 Apr '19

2 2

Problem with solr working, but not indexing
by TG Servers 14 Apr '19

14 Apr '19

2 1

Re: Solr connection timeout hardwired to 60s
by Peter Mogensen 14 Apr '19

14 Apr '19

sorry... I got distracted half way and forgot to put a meaningfull subject so the archive could figure out the thread. - resending. On 4/14/19 4:04 PM, dovecot-request(a)dovecot.org wrote: >> Solr ships with autoCommit set to 15 seconds and openSearcher set to >> false on the autoCommit.? The autoSoftCommit setting is not enabled by >> default, but depending on how the index was created, Solr might try to >> set autoSoftCommit to 3 seconds ... which is WAY too short. I just run with the default. 15s autoCommit and no autoSoftCommit >> This thread says that dovecot is sending explicit commits.? I see explicit /update req. with softCommit and waitSearcer=true in a tcpdump. >> One thing >> that might be happening to exceed 60 seconds is an extremely long >> commit, which is usually caused by excessive cache autowarming, but >> might be related to insufficient memory.? The max heap setting on an >> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY >> small, and it doesn't take much index data before a much larger heap >> is required. I run with SOLR_JAVA_MEM="-Xmx8g -Xms2g" > I looked into the code (version 2.3.5.1): This is 2.2.35. I haven't checked the source difference to 2.3.x I must admit. > I immagine that one of the reasons dovecot sends softCommits is because > without autoindex active and even if mailboxes are periodically indexed > from cron, the last emails received with be indexed at the moment of the > search.? I expect that dovecot has to because of it's default behavior by only bringing the index up-to-date just before search. So it has towait for the index result to be available if there's been any new mails indexed. > 1) a configurable batch size would enable to tune the number of emails > per request and help stay under the 60 seconds hard coded http request > timeout. A configurable http timeout would be less useful, since this > will potentially run into other timeouts on solr side. Being able to configure it is great. But I don't think it solves much. I recompiled with 100 as batch size and it still ended in timeouts. Then I recompiled with 10min timeout and now I see all the batches completing and their processesing time is mostly between 1 and 2 minutes (so all would have failed). To me it looks like Solr just takes too long time to index. This is no small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz and for this test it's not doing anything else, so I'm a bit surprised that even with only a few users this takes so long time. /Peter

2 2

Re: Mail account brute force / harassment
by Odhiambo Washington 14 Apr '19

14 Apr '19

On Thu, 11 Apr 2019 at 13:24, Marc Roos via dovecot <dovecot(a)dovecot.org> wrote: > > > Say for instance you have some one trying to constantly access an > account > > > Has any of you made something creative like this: > > * configure that account to allow to login with any password > * link that account to something like /dev/zero that generates infinite > amount of messages > (maybe send an archive of virusses?) > * transferring TB's of data to this harassing client. > > I think it would be interesting to be able to do such a thing. > > Instead of being evil, just use fail2ban to address this problem :-) -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", grep ^[^#] :-)

10 24

permissions for quota-status
by Andreas Meyer 14 Apr '19

14 Apr '19

Hello! doveconf --version 2.2.36 I want to move to a new server with dovecot but get an error Error: service(quota-status): listen(*, 12340) failed: Permission denied when I define service quota-status { executable = quota-status -p postfix inet_listener { port = 12340 } client_limit = 1 } private]# ll insgesamt 0 srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 anvil srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 bounce srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 defer srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 discard srw-------. 1 postfix postfix 0 14. Apr 15:08 dovecot-lmtp srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 error srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 lmtp srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 local srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 proxymap srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 proxywrite srw-rw----. 1 postfix postfix 0 14. Apr 14:58 quota-status srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 relay srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 retry srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 rewrite srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 scache srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 smtp srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 tlsmgr srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 trace srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 verify srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 virtual Don't know what's wrong. Can someone help? Kind regards Andreas -- PGP-Fingerprint: D392 5D21 0299 63D7 5BAE 4562 1E56 B2EA 81A2 59F1

2 1

nginx configuration to pass x-originating-ip
by André Rodier 14 Apr '19

14 Apr '19

Hello, There is a bug in SOGo, as it sends the original IP after successful login, and not before the login process. I traced the bug to the source code. https://sogo.nu/bugs/view.php?id=2979. Then, in my research, I found this old thread: https://forum.nginx.org/read.php?29,237299,237367#msg-237367 Can I use Nginx as an IMAP proxy to add the missing ID. I suspect this is something that can be achieved? I already have an Nginx for the SOGo front-end, so I would add another one for the IMAP proxy. With a little bit of luck, I should be able to pass the original IP between the two servers Does anyone has a working example? Thanks, André -- André Rodier HomeBox: https://github.com/progmaticltd/homebox

1 0

Post login scripts environment
by André Rodier 14 Apr '19

14 Apr '19

Dear Dovecot users, I am running Dovecot 2.2.27 on Debian Stretch, no issue so far. I wonder if there is a way to pass the remote IP address, in an environment variable, in the post login script. My Post login scripts are working well, except that when the server is accessed through a webmail (Roundcube or SOGo), the remote IP address is systematically 127.0.0.1. The other question I have is, is it possible to pass the user agent of the email client used to access the server? I know this can be easily forged, but I would like to log it. Thanks for your insight. -- André Rodier

2 6