- dovecot - dovecot.org

Fwd: SOLR/Index?
by Larry Rosenman 15 Apr '19

15 Apr '19

forgot to reply all. ---------- Forwarded message --------- From: Larry Rosenman <larryrtx(a)gmail.com> Date: Mon, Apr 15, 2019 at 5:58 AM Subject: Re: SOLR/Index? To: John Fawcett <john(a)voipsupport.it> the local users (myself, my wife, a friend) can authenticate EITHER as <username> or <username>@lerctr.org. switching to all virtual users is NOT going to happen. If I login to roundcube with <user>@lerctr.org it finds the autoindexed mail. So, if I make everyone always authenticate as <user>@lerctr.org we should be fine. and change my script to do doveadm -u <user>@<domain> instead of depending on the local user running the cron job. question: Is there some way to have dovecot change what it sees to be <user>@lerctr.org when they login as <user>? On Mon, Apr 15, 2019 at 5:54 AM John Fawcett via dovecot < dovecot(a)dovecot.org> wrote: > On 15/04/2019 11:38, Larry Rosenman via dovecot wrote: > > ⌂63% [ler@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX > ⌂67% [ler@thebighonker.lerctr.org:~] 1 $ mail -s "test fts1970" > ler(a)lerctr.org > test fts1970 > > test fts1970 > . > EOT > [ler@thebighonker.lerctr.org:~] $ mailq > [ler@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX > Subject: test fts1970 > test fts1970 > test fts1970 > > > Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <= > ler(a)lerctr.org U=ler P=local S=388 > Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect from local > Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler(a)lerctr.org/49364): > save: box=INBOX, uid=175402, msgid=< > E1hFxvD-000Csq-P6(a)thebighonker.lerctr.org>, size=640, vsize=660, > from=Larry Rosenman <ler(a)lerctr.org>, subject=test fts1970, flags=() > Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler(a)lerctr.org/49364): > sieve: msgid=<E1hFxvD-000Csq-P6(a)thebighonker.lerctr.org>: stored mail > into mailbox 'INBOX' (subject=test fts1970 from=ler(a)lerctr.org size=660) > Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect from > local: Client has quit the connection (state=READY) > Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler < > ler(a)lerctr.org> R=localuser T=dovecot_lmtp S=404 C="250 2.0.0 < > ler(a)lerctr.org> 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s > Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 Completed QT=0s > Apr 15 04:29:03 thebighonker dovecot[2507]: indexer-worker( > ler(a)lerctr.org/49366) Indexed 1 messages in INBOX (UIDs 175402..175402) > > > ⌂81% [ler@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX body > 'fts1970' > ⌂83% [ler@thebighonker.lerctr.org:~] $ > > > ⌂65% [ler@thebighonker.lerctr.org:~] 75 $ doveadm search -u ler(a)lerctr.org > mailbox INBOX body 'fts1970' > a53a143be44bda5bd4830000bbe98eac 175402 > [ler@thebighonker.lerctr.org:~] $ doveadm index -q INBOX > [ler@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX body > 'fts1970' > [ler@thebighonker.lerctr.org:~] $ doveadm fts rescan > [ler@thebighonker.lerctr.org:~] $ doveadm index -q INBOX > [ler@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX body > 'fts1970' > a53a143be44bda5bd4830000bbe98eac 175402 > [ler@thebighonker.lerctr.org:~] $ doveadm search -u ler(a)lerctr.org > mailbox INBOX body 'fts1970' > a53a143be44bda5bd4830000bbe98eac 175402 > [ler@thebighonker.lerctr.org:~] $ > > So, yes, your hypothesis is correct. > > Question: How can I make it consistent? > > I have a script that runs on the first of the month that does archiving, > and I have similar issues in that namespace: > ⌂67% [ler@thebighonker.lerctr.org:~] $ cat bin/archive-mail > #!/bin/sh > PATH=$PATH:/usr/local/bin > #Expects to be run after midnight on the first of the month > # to archive all the previous months mail > #Date Run: > TODAY=`date "+%Y-%m-%d"` > #last month in YYYY/MM > YEAR_LAST_MONTH=`date -v-1d "+%Y/%m"` > #1st of last month as 01-Mon-YYYY > FIRST_LAST_MONTH=`date -v-1d "+01-%b-%Y"` > echo 'TODAY=' ${TODAY} > echo 'YEAR_LAST_MONTH=' ${YEAR_LAST_MONTH} > echo 'FIRST_LAST_MONTH=' ${FIRST_LAST_MONTH} > # get a list of all the mailboxes with at least one real message > doveadm -f tab mailbox status vsize \* 2>/dev/null | > sed -e 1d | sort -k 1,1 | > awk 'BEGIN {FS="\t"} {if ($2 > 0) print $1}' | > while read i > do > echo `date` start ${i} > doveadm mailbox create "ARCHIVE/${YEAR_LAST_MONTH}/${i}" > doveadm -f tab mailbox status messages "${i}" > doveadm move "ARCHIVE/${YEAR_LAST_MONTH}/${i}" mailbox \ > "${i}" BEFORE ${TODAY} SINCE ${FIRST_LAST_MONTH} > doveadm -f tab mailbox status messages "${i}" > echo `date` done ${i} > done > ⌂64% [ler@thebighonker.lerctr.org:~] $ > > > The Exim config can be provided as well if necessary. > > ler & ler(a)lerctr.org *ARE THE SAME MAILBOX* > > At the moment it looks as though you have two sets of emails indexed in > solr. One is indexed under username (the one you are running mannually and > apparently the one used by roundcube too, but that's to be verified) and > another set being indexed by autoindex = yes option using the full email > address. Once you've got it working as you require, then you may want to > clean out solr and reindex with just one of them just to reduce volumes. > > Your setup seems to have a mix of users from mysql and from /etc/passwd. > Not sure if your mysql users are all mapped to real users or they have > their own mailboxes with domain included. Your solution will depend on what > you really need and if the setup is working correctly you may not want to > tweak it too much or other things may start breaking. > > The simplest thing that comes to mind (providing your manually indexed > mails show up in roundcube searches) is just to turn off autoindex and > schedule indexing from cron. As things stand I believe autoindex works well > with virtual users username@domain. You'll probably need to start off > with a rescan if you do this. > > If you can swith to all virtual users without local users then probably > you can use autoindex, but I can understand that may not be possible. For > sure I'd recommend doing it in a test environment first so you can be sure > of the configuration and then look at migrating your existing mailboxes > over to virtual users. > > John > > > > -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106 -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

4 11

No CRLF in Pigeonhole's header?
by Wojciech Sychut 15 Apr '19

15 Apr '19

Hello, I have been told on Freenode's #dovecot to ask here. I have Exim 4.92 complaining about "Non-CRLF-terminated header, under CHUNKING: message abandoned" with redirects made by Pigeonhole 0.5.5 (with Dovecot 2.3.5.1). Please find relevant part of the traffic below, and help me determine whether Exim is correct on this. 0x0000: 4500 0954 0000 4000 8006 0000 7f00 0001 E..T..@......... 0x0010: 7f00 0001 6ea2 0019 8982 e279 d090 b820 ....n......y.... 0x0020: 8018 021e 0749 0000 0101 080a c345 2b51 .....I.......E+Q 0x0030: b382 853c 4d41 494c 2046 524f 4d3a 3c72 ...<MAIL.FROM:<r 0x0040: 4073 3066 2e6d 642e 6465 7664 6576 6465 @sxf.md.devdevde 0x0050: 762e 6e65 743e 0d0a 5243 5054 2054 4f3a v.net>..RCPT.TO: 0x0060: 3c63 6f6e 7461 6374 4072 6567 6973 2e74 <contact(a)regis.t 0x0070: 6563 683e 0d0a 4244 4154 2032 3235 3420 ech>..BDAT.2254. 0x0080: 4c41 5354 0d0a 582d 5369 6576 653a 2050 LAST..X-Sieve:.P 0x0090: 6967 656f 6e68 6f6c 6520 5369 6576 6520 igeonhole.Sieve. 0x00a0: 302e 352e 3520 2832 3438 3362 3038 3529 0.5.5.(2483b085) 0x00b0: 0a58 2d53 6965 7665 2d52 6564 6972 6563 .X-Sieve-Redirec 0x00c0: 7465 642d 4672 6f6d 3a20 7240 7330 662e ted-From:.r@sxf. With Dovecot 2.2.36 + Pigeonhole 0.4.24.1 and required minimum of version-related configuration changes - redirection works properly. Related Exim code fragment: https://github.com/Exim/exim/blob/master/src/src/receive.c#L2222 Thank you, -- Wojtek Sychut

2 1

rsync sdbox, no new mails?
by Jakob Curdes 15 Apr '19

15 Apr '19

Hello, I had to restore a mailserver from a backup to replace a system with problems that is still live. I did a 1:1 restore as far as possible from a backup, then rsynced the /var/vmail/ directory, where we have all the mailboxes, in sdbox format. So both directories should be the same. After syncing, I started dovecot. I can see the mails until to the point of the backup, but not the newer ones ,although they seem to be present in the "cur" directory. I thought with sdbox it is all in the index files that have been transferred with the mails? Or am I missing a place that has to be synced? Thank you for hints, Jakob

2 2

LDAP iterate (dovecoc 2.2.27)
by Debreczeni Attila 15 Apr '19

15 Apr '19

Dear Dovecot Team, We would like to use the LDAP iterate for dovecot replication, but I got this error: dovecot: auth-worker(10149): Error: ldap(*): ldap_search(base=ou=mail,o=asd,c=TLD filter=(objectClass=inetMailUser) failed: Size limit exceeded dovecot: replicator: Error: User listing returned failure dovecot: replicator: Error: listing users failed, can't replicate existing data When I check the "doveadm replicator status": ... Total number of known users 1200 But we have more than 10.000 users. When I run the ldapsearch manually, I get the full list of the mail addresses. Who can I fix it? Thank you, Attila # dovecot --version 2.2.27 (c0f36b0) # dovecot -n | filtered # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.8 auth_worker_max_count = 150 default_vsz_limit = 512 M doveadm_password = # hidden, use -P to show it doveadm_port = *** mail_gid = vmail mail_plugins = " quota notify replication" namespace inbox { ... } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { mail_replica = tcp:REMOTE_IP } replication_dsync_parameters = -d -N -1 -l 30 -U service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } } service auth { inet_listener { address = LOCAL_IP port = LOCAL_PORT } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service dict { unix_listener dict { group = vmail mode = 0600 user = vmail } } service lmtp { inet_listener lmtp { address = LOCAL_IP port = LOCAL_PORT } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } userdb { args = /etc/dovecot/dovecot-domain-ldap.conf.ext driver = ldap result_success = continue-ok } userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap result_failure = return-fail } protocol imap { ... } protocol pop3 { ... } # grep -v '^ *$#.*$\?$' dovecot-ldap.conf.ext uris = ldaps://ldap.URL dn = uid=auth,o=domain,c=TLD dnpass = **** sasl_bind = no tls_ca_cert_dir = /etc/ssl/certs tls_require_cert = demand ldap_version = 3 base = ou=mail,o=asd,c=TLD deref = never scope = subtree user_attrs = =home=/dev/null/%Ld/%L{ldap:uid}, mailQuota=quota_rule=*:bytes=%$ user_filter = (objectClass=inetMailUser) pass_attrs = userPassword=password pass_filter = (objectClass=inetMailUser) iterate_attrs = mail=user iterate_filter = (objectClass=inetMailUser) default_pass_scheme = PLAIN

1 0

LDAP iterate (dovecoc 2.2.27)
by Debreczeni Attila 15 Apr '19

15 Apr '19

Dear Dovecot Team, We would like to use the LDAP iterate for dovecot replication, but I got this error: dovecot: auth-worker(10149): Error: ldap(*): ldap_search(base=ou=mail,o=asd,c=TLD filter=(objectClass=inetMailUser) failed: Size limit exceeded dovecot: replicator: Error: User listing returned failure dovecot: replicator: Error: listing users failed, can't replicate existing data When I check the "doveadm replicator status": ... Total number of known users 1200 But we have more than 10.000 users. When I run the ldapsearch manually, I get the full list of the mail addresses. Who can I fix it? Thank you, Attila # dovecot --version 2.2.27 (c0f36b0) # dovecot -n | filtered # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.8 auth_worker_max_count = 150 default_vsz_limit = 512 M doveadm_password = # hidden, use -P to show it doveadm_port = *** mail_gid = vmail mail_plugins = " quota notify replication" namespace inbox { ... } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { mail_replica = tcp:REMOTE_IP } replication_dsync_parameters = -d -N -1 -l 30 -U service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } } service auth { inet_listener { address = LOCAL_IP port = LOCAL_PORT } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service dict { unix_listener dict { group = vmail mode = 0600 user = vmail } } service lmtp { inet_listener lmtp { address = LOCAL_IP port = LOCAL_PORT } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } userdb { args = /etc/dovecot/dovecot-domain-ldap.conf.ext driver = ldap result_success = continue-ok } userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap result_failure = return-fail } protocol imap { ... } protocol pop3 { ... } # grep -v '^ *$#.*$\?$' dovecot-ldap.conf.ext uris = ldaps://ldap.URL dn = uid=auth,o=domain,c=TLD dnpass = **** sasl_bind = no tls_ca_cert_dir = /etc/ssl/certs tls_require_cert = demand ldap_version = 3 base = ou=mail,o=asd,c=TLD deref = never scope = subtree user_attrs = =home=/dev/null/%Ld/%L{ldap:uid}, mailQuota=quota_rule=*:bytes=%$ user_filter = (objectClass=inetMailUser) pass_attrs = userPassword=password pass_filter = (objectClass=inetMailUser) iterate_attrs = mail=user iterate_filter = (objectClass=inetMailUser) default_pass_scheme = PLAIN

1 0

Re: SOLR/Index?
by Larry Rosenman 15 Apr '19

15 Apr '19

full solr.log at: https://www.lerctr.org/~ler/solr.log The search DOES make it to SOLR: ⌂77% [ler@thebighonker.lerctr.org:~] 130 $ grep sysutils /var/log/solr/solr.log 2019-04-14 18:31:34.749 INFO (qtp349420578-7538) [ x:dovecot] o.a.s.c.S.Request [dovecot] webapp=/solr path=/select params={q={!lucene+q.op%3DAND}(hdr:sysutils\/+OR+body:sysutils\/)&fl=uid,score&sort=uid+asc&fq=%2Bbox:8097632f69627b5b58950000bbe98eac+%2Buser:ler&rows=22644&wt=xml} hits=0 status=0 QTime=460 Pick showing subjects: https://www.lerctr.org/~ler/sysutils_mail.png What else? I'm happy to provide access. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

3 11

SOLR/Index?
by Larry Rosenman 14 Apr '19

14 Apr '19

Given all the discussion on FTS/Solr, etc, I have a question: I have autoindex set, and searching in roundcube most of the time does NOT find things, HOWEVER if I do: doveadm fts rescan doveadm index I can find things in the mailboxes. WHY? (doveconf -n attached). -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

2 1

Re: dovecot Digest, Vol 192, Issue 52
by Peter Mogensen 14 Apr '19

14 Apr '19

On 4/14/19 4:04 PM, dovecot-request(a)dovecot.org wrote: >> Solr ships with autoCommit set to 15 seconds and openSearcher set to >> false on the autoCommit.? The autoSoftCommit setting is not enabled by >> default, but depending on how the index was created, Solr might try to >> set autoSoftCommit to 3 seconds ... which is WAY too short. I just run with the default. 15s autoCommit and no autoSoftCommit >> This thread says that dovecot is sending explicit commits.? I see explicit /update req. with softCommit and waitSearcer=true in a tcpdump. >> One thing >> that might be happening to exceed 60 seconds is an extremely long >> commit, which is usually caused by excessive cache autowarming, but >> might be related to insufficient memory.? The max heap setting on an >> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY >> small, and it doesn't take much index data before a much larger heap >> is required. I run with SOLR_JAVA_MEM="-Xmx8g -Xms2g" > I looked into the code (version 2.3.5.1): This is 2.2.35. I haven't checked the source difference to 2.3.x I must admit. > I immagine that one of the reasons dovecot sends softCommits is because > without autoindex active and even if mailboxes are periodically indexed > from cron, the last emails received with be indexed at the moment of the > search.? I expect that dovecot has to because of it's default behavior by only bringing the index up-to-date just before search. So it has towait for the index result to be available if there's been any new mails indexed. > 1) a configurable batch size would enable to tune the number of emails > per request and help stay under the 60 seconds hard coded http request > timeout. A configurable http timeout would be less useful, since this > will potentially run into other timeouts on solr side. Being able to configure it is great. But I don't think it solves much. I recompiled with 100 as batch size and it still ended in timeouts. Then I recompiled with 10min timeout and now I see all the batches completing and their processesing time is mostly between 1 and 2 minutes (so all would have failed). To me it looks like Solr just takes too long time to index. This is no small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz and for this test it's not doing anything else, so I'm a bit surprised that even with only a few users this takes so long time. /Peter

2 1

Re: Solr connection timeout hardwired to 60s
by Peter Mogensen 14 Apr '19

14 Apr '19

On 4/4/19 6:47 PM, dovecot-request(a)dovecot.org wrote: > For a typical Solr index, 60 seconds is an eternity. Most people aim > for query times of 100 milliseconds or less, and they often achieve > that goal. I'm pretty sure I get these while indexing, not querying. Apr 04 16:44:50 host dovecot[114690]: indexer-worker(me(a)example.com): Error: fts_solr: Indexing failed: Request timed out (Request queued 66.015 secs ago, 1 attempts in 66.005 secs, 63.146 in http ioloop, 0.000 in other ioloops, connected 94.903 secs ago) /Peter

5 8

Fwd: Problem with solr working, but not indexing
by TG Servers 14 Apr '19

14 Apr '19

2 2