- dovecot - dovecot.org

Questions about how fts-solr works
by Radu Gheorghe 08 Aug '16

08 Aug '16

Hello, I'm quite a Dovecot newbie, so please be gentle :) Though I did my homework as well as I could, I still have some questions regarding the Solr plugin. Specifically: - I understand that by default, a mailbox is indexed on the first search and then deltas are indexed in subsequent searches. Are Emails indexed in batches or one by one? Looking at the code, I see a hardcoded limit of 1000, and I'm guessing if the mailbox is done and there are <1000 Emails in the buffer, it just flushes them, right? - if I set fts_autoindex=yes, does it mean that as soon as the Email is delivered by the MTA, it will be indexed in Solr? or does it have to read by the user or touched in any way? - also, with fts_autoindex=yes, are Emails indexed in batches? if yes, is there also a time limit besides the size limit? e.g. if only 100 messages were received - I have the same question about deletes: when do they happen, and are they batched? - what happens if Solr is unavailable? I know Dovecot keeps track of what indexed in dovecot.indexed files, but does it retry? if yes, what's the retry policy and can it be configured? Also, does it behave the same if Solr is actually available and throws an error? - the same question is for attachments, though I think this is general FTS - what if Tika fails to parse the attachment? Does Dovecot still index the Email metadata? As a side, I'm also wondering if I could use the Tika that comes with Solr (https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Solr+C…) Can fts-solr handle attachments at all? I'm asking because I don't see that field in the default schema.xml - can I add arbitrary URI parameters to the Solr request? I see that one could fiddle with the path, which I assume will let one have one collection per mailbox (though I'm curious how that works with batches - I'm guessing one batch/indexing thread per mailbox?). Specifically, I'm interested in using the mailbox as a routing value - if I read the code well, Dovecot does a soft commit when it's done with the specific mailbox. For indexing at search time, I see why it makes sense. If I do "autoindex", can I disable that and let Solr autoSoftCommit every N seconds? That should improve indexing throughput and reduce load. I see that one can already do this for hard commits (I'd use autoCommit there, though a hard commit is also triggered when ramBufferSizeMB gets hit) - when querying, can I sort by an arbitrary field, such as the date? I saw I can sort by score, but I can't find anything in the code that will suggest it's supported - also when querying, can I specify which fields to return? I see that the plugin asks for Email ID, so I'm guessing it fetches things like from/to from the Email itself. I'm thinking that if I want to sort by those values I need to set docValues=true on them, to save memory. In that case, I might as well retrieve the original string from docValues, which should be a whole lot faster My plan is to do all sorts of tests, but having a better background on how it works will certainly help. Any pointers, feedback, encouragement, etc is certainly welcome - thanks in advance! Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/

1 0

Dovecot book available again
by Peer Heinlein 08 Aug '16

08 Aug '16

Hi, after my publisher has to shut down his business at the end of last year, it took several months to organize everything. But it's done! I'm happy to announce: The Dovecot book is available again. You can order it at Createspace: https://www.createspace.com/5942312 Or Amazon: https://www.amazon.com/dp/1534895701 https://www.amazon.co.uk/dp/1534895701 https://www.amazon.de/dp/1534895701 And in some weeks in every book store by ISBN-13 978-1534895706. If you want to support the author and if you help us to make good books you should think about ordering it from Createspace instead of Amazon. There's a 10% discount for users of this mailinglist if you use the discount code VR5YF7W2 at Createspace. Valid until August, 21th. Feedback and comments by mail is appreciated, also comments and ratings at Createspace and Amazon :-) Thanks again to Timo and everybody from Dovecot Oy / Open-Xchange for their support. Peer P.S.: The German Dovecot-Book is also available at: http://www.dovecot-buch.de/online-shop/buch-bestellung/ -- Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-42 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

6 5

Dovecot password policy
by Robert Blayzor 06 Aug '16

06 Aug '16

Is there a way to configure Dovecot to perhaps filter/enforce which passwords are accepted before authenticating? Ie: Reject immediately (without a database lookup) if password is not X characters in length? ? -- Robert inoc.net!rblayzor XMPP: rblayzor.AT.inoc.net PGP Key: 78BEDCE1 @ pgp.mit.edu

4 5

Re: Dovecot password policy
by Joseph Tam 05 Aug '16

05 Aug '16

Robert Blayzor <rblayzor.bulk(a)inoc.net> writes: > Is there a way to configure Dovecot to perhaps filter/enforce which > passwords are accepted before authenticating? Ie: Reject immediately > (without a database lookup) if password is not X characters in length? Yes, use the checkpassword hook. http://wiki.dovecot.org/AuthDatabase/CheckPassword I think there also some PAM module that you can stack into your system that will enforce password policies. Joseph Tam <jtam.home(a)gmail.com>

1 0

Dovecot and Solr 6
by KSB 05 Aug '16

05 Aug '16

Hi! Dovecot 2.2.24 Had set up solr and new schema collection. Copied dovecot provided schema. There was an error with booleans (while getting schema via http), which I "solved" by removing "add-unknown-fields-to-the-schema" from solrconfig.xml. It is correct way to solve this? Anyway, I run tcpdump to see network activity between dovecot and solr: #tcpdump -i lo port 8983 and see nothing while doing: #doveadm fts rescan -u username(a)domain.tld no output also from doveadm. Noticed that dovecot.index.log reapears in user mail root if deleted. plugin { fts = solr fts_solr = url=http://localhost:8983/solr/dovecot/ debug .. } -- KSB

3 4

Problem with sieve configuration on dovecot+debian_jessie+ISPC3
by Filip Loncar [WarpMax | FederationServers] 05 Aug '16

05 Aug '16

I'm new-ish to dovecot and sieve (haven't used it extensively till now) and I'm not sure that I configured everything propperly. The problem is that we're trying to get the following sieve code to run: require ["fileinto", "regex", "date", "relational", "vacation", "duplicate"]; require "duplicate"; if duplicate :header "message-id" { discard; } else { redirect "<somemail1>"; redirect "<somemail2>"; redirect "<somemail3>"; redirect "<somemail4>"; redirect "<somemail5>"; redirect "<somemail6>"; keep; } keep; But we're constantly getting the following error: sieve: info: started log at Aug 02 08:13:21. main script: line 5: error: require command: unknown Sieve capability `duplicate'. main script: line 6: error: unknown test 'duplicate' (only reported once at first occurence). main script: error: validation failed. Both me and my collegue can't find what's wrong with the system. I would appreciate any kind of help on this matter. Below's my configuration. # dovecot --version 2.2.13 # dovecot -n # 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-45-pve x86_64 Debian 8.5 auth_mechanisms = plain login disable_plaintext_auth = no listen = *,[::] log_timestamp = "%Y-%m-%d %H:%M:%S " mail_privileged_group = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave vnd.dovecot.duplicate passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } plugin { quota = dict:user::file:/var/vmail/%d/%n/.quotausage sieve = /var/vmail/%d/%n/.sieve sieve_duplicate_default_period = 14d sieve_extensions = +vnd.dovecot.duplicate sieve_max_redirects = 6 sieve_plugins = sieve_extprograms } protocols = imap pop3 service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service imap-login { client_limit = 1000 process_limit = 500 } ssl_cert = </etc/postfix/smtpd.cert ssl_key = </etc/postfix/smtpd.key ssl_protocols = !SSLv2 !SSLv3 userdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } protocol imap { mail_plugins = quota imap_quota } protocol pop3 { mail_plugins = quota pop3_uidl_format = %08Xu%08Xv } protocol lda { mail_plugins = sieve quota } # uname -a Linux fero 2.6.32-45-pve #1 SMP Wed Mar 30 12:22:08 CEST 2016 x86_64 GNU/Linux It's a Debain Jessie VM running ISPC3. -- Srdac(an pozdrav | Best regards Filip Lonc(ar ___________________________________________ WarpMax hosting *WarpMax hosting* *A* Tehnolos(ki park Varaz(din, Zagrebac(ka 89, Varaz(din, Croatia - Europe *T * +385 42 492 129, 490 090, +385 99 5896 123 ___________________________________ Ova elektronic(ka poruka, kao i svi eventualni privici, su povjerljivi i namijenjeni iskljuc(ivo naznac(enom primatelju(ima). Ukoliko niste naznac(eni primatelj(i) ove elektronic(ke poruke, molimo da odmah obavijestite pos(iljatelja, a sve kopije ove elektronic(ke poruke (ukljuc(ujuc'i i privitke) trajno obris(ete. Svaka neovlas(tena upotreba, distribucija, reprodukcija ili priopc'avanje ove poruke zabranjeno je. WarpMax hosting ne preuzima odgovornost za sadrz(aj ove poruke i eventualnu s(tetu nastalu primitkom ove poruke i priloga sadrz(anih u poruci. Ne jamc(imo da je ova elektronic(ka poruka slobodna od virusa. WarpMax hosting ne preuzima odgovornost za bilo kakav gubitak ili s(tetu koja nastane zbog prisutnosti virusa u elektronic(koj poruci. This e-mail, including all attached files, if any, is confidential and intended for the sole use by the person(s) mentioned as recipient(s). If you are not the named recipient(s) indicated in this message, please notify the sender immediately, and permanently delete all copies of this message (including any attachments). Any review, use, distribution, disclosure, printing or copying of this e-mail other than by the intended recipient(s) is prohibited. WarpMax hosting accepts no liability for content of this message and any potential damage caused by this message and files attached to it. No warranty is given that this email is free of viruses. WarpMax hosting accepts no liability for any loss or damage caused by presence of a virus.

4 3

New password hashing scheme as plugin
by Andreas Meyer 05 Aug '16

05 Aug '16

Hi, I want to add a new password hashing scheme as plugin and provide it for the dovecot project, so that it will be included as optional plugin in future releases. Yet the plugin compiles fine and the .so file gets created. My approach is to call the functions password_scheme_register() and password_scheme_unregister() (src/auth/password-scheme.c) inside the plugin's _init() and _deinit() functions. When a client tries to login via imap the log shows an error message: Error: Couldn't load required plugin /usr/local/lib/dovecot/lib20_newauth_plugin.so: dlopen() failed: /usr/local/lib/dovecot/lib20_newauth_plugin.so: undefined symbol: password_scheme_unregister That error obviously occurs, because the required lib is not linked into the plugin. Now my problem is to understand, whether this approach (via .so file) can be successful at all and if so, how? Do I need to link libs into it? Which would that be and how do I specify them in the Makefile.am? I assume, I don't need to link a lib, but need to use a hook to register the new hashing scheme, since the plugin is loaded into an already existing process, which needs to get extended. If this is true, which hook would that be and how do I register the new scheme? Or is there no way to accomplish this via a separated .so plugin? Should I extend the sources in /src/auth instead? (separat file for the hashing scheme with preprocessor instructions to include it only on demand) Thank you very much, Andreas

3 8

file/folder perms permissions
by Jim Pazarena 05 Aug '16

05 Aug '16

Forgive me if I am blind, but I cannot find default folder permissions. For the home directories, with the sub of mail: and deeper, I see some files with 660, some with 600, some folders with 770 some with 660 I am a bit confused. I manually messed with some files, and my MUA complained about permissions. So I got that mess fixed, and in doing so I noticed the seemingly contradictory permissions. Thanks!

3 2

SSL connection reset by peer
by Vince42 04 Aug '16

04 Aug '16

Hi, I am running a dovecot server and have set up an external monitoring, where every five minutes a login with SSL on port 993 is done. I usually get once a day an error "connection reset by peer - SSL connect", which goes away until the next monitor is executed. Initially I thought that raising the mail_max_userip_connections in protocol imap in 20-imap.conf to 256 should do the trick - but the error stays. What could be the reason for this error and which configuration values could be changed in order to avoid this error? I read a lot of different suggestions - but did not find a plausible explanation and recommendation. Thanks in advance! -- Cheers, \\|// Vince (o o) ----------------------------ooO-(_)-Ooo------------------------- ''' (o)_(o) [ ][0][ ] ô¿ô (=°o°=) World Domination by Copy and Paste [ ][ ][0] - (")_(") [0][0][0] () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments Ooo. ---------------------------.ooO----( )------------------------- ( ) (_/ \_)

5 11

Dovecot 2.2.25 test failure
by Michael A. Peters 04 Aug '16

04 Aug '16

Operating system - 64 bit CentOS 7 gcc-4.8.5-4.el7.x86_64 Building against LibreSSL which has been fine for other releases, but it is a crypto test that is fails. Tried with LibreSSL 2.4.2 and 2.3.6 - both the build completes but fails the make check Dovecot 2.2.24 passes make check on both. This is where it fails: Making check in lib-dcrypt make[2]: Entering directory `/home/alice/rpmbuild/BUILD/dovecot-2.2.25/src/lib-dcrypt' for bin in test-crypto test-stream; do \ if ! /bin/sh ../../run-test.sh ../.. ./$bin; then exit 1; fi; \ done ../../run-test.sh: line 21: 22369 Segmentation fault (core dumped) valgrind -q --trace-children=yes --leak-check=full --suppressions="$supp_path" --log-file=test.out.$$ $* ==22369== Invalid read of size 8 ==22369== at 0x113A47: dcrypt_ctx_sym_create (dcrypt.c:61) ==22369== by 0x119194: test_cipher_test_vectors (test-crypto.c:60) ==22369== by 0x1196F0: test_run_funcs (test-common.c:354) ==22369== by 0x119FC0: test_run (test-common.c:404) ==22369== by 0x113461: main (test-crypto.c:554) ==22369== Address 0x8 is not stack'd, malloc'd or (recently) free'd ==22369== ==22369== ==22369== Process terminating with default action of signal 11 (SIGSEGV) ==22369== Access not within mapped region at address 0x8 ==22369== at 0x113A47: dcrypt_ctx_sym_create (dcrypt.c:61) ==22369== by 0x119194: test_cipher_test_vectors (test-crypto.c:60) ==22369== by 0x1196F0: test_run_funcs (test-common.c:354) ==22369== by 0x119FC0: test_run (test-common.c:404) ==22369== by 0x113461: main (test-crypto.c:554) ==22369== If you believe this happened as a result of a stack ==22369== overflow in your program's main thread (unlikely but ==22369== possible), you can try to increase the size of the ==22369== main thread stack using the --main-stacksize= flag. ==22369== The main thread stack size used in this run was 8388608. Failed to run: ./test-crypto make[2]: *** [check-test] Error 1 make[2]: Leaving directory `/home/alice/rpmbuild/BUILD/dovecot-2.2.25/src/lib-dcrypt' make[1]: *** [check-recursive] Error 1 make[1]: Leaving directory `/home/alice/rpmbuild/BUILD/dovecot-2.2.25/src' make: *** [check-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.Il5fdU (%check) Thanks for suggestions.

3 7