Re: SOLR/Index?

Larry Rosenman

15 Apr 2019 15 Apr '19

2:39 a.m.

full solr.log at: https://www.lerctr.org/~ler/solr.log

The search DOES make it to SOLR: ⌂77% [ler@thebighonker.lerctr.org:~] 130 $ grep sysutils /var/log/solr/solr.log 2019-04-14 18:31:34.749 INFO (qtp349420578-7538) [ x:dovecot] o.a.s.c.S.Request [dovecot] webapp=/solr path=/select params={q={!lucene+q.op%3DAND}(hdr:sysutils\/+OR+body:sysutils\/)&fl=uid,score&sort=uid+asc&fq=%2Bbox:8097632f69627b5b58950000bbe98eac+%2Buser:ler&rows=22644&wt=xml} hits=0 status=0 QTime=460

Pick showing subjects: https://www.lerctr.org/~ler/sysutils_mail.png

What else?

I'm happy to provide access.

-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Attachments:

attachment.html (text/html — 1.7 KB)

Show replies by date

John Fawcett

15 Apr 15 Apr

3:27 a.m.

New subject: SOLR/Index?

On 15/04/2019 01:39, Larry Rosenman via dovecot wrote:

...

Larry

so the search is returning no hits as you said. But can you show that there is data in the index that should match?

doveadm search -u user@example.com mailbox inbox body "sysutils/"

Can you do a controlled test and send yourself a test message with that string and show the solr log where it is being inserted into the index and then search for it with doveadm (just to rule out roundcube for the moment) and show solr log for that search?

John

Larry Rosenman

3:33 a.m.

New subject: SOLR/Index?

⌂72% [ler@thebighonker.lerctr.org:~] $ doveadm search mailbox lists/freebsd/ports-commiters body 'sysutils' [ler@thebighonker.lerctr.org:~] $ doveadm fts rescan [ler@thebighonker.lerctr.org:~] $ doveadm index -q lists/freebsd/ports-commiters ⌂64% [ler@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected (auth failed, 1 attempts in 2 secs): user=<gwc>, method=PLAIN, rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed, session=<EAk5woaGV4S0tNl8> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login: user=<ler>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS, session=<lr1mwoaGwYMgAQRwHw8DrQC73P/+UNkA> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP address 23.100.68.192 Apr 14 19:30:55 thebighonker exim[14846]: H=(DaVinci-MWare.prophet21lab.com) [23.100.68.192]:52130 I=[192.147.25.65]:25 sender verify defer for < info@duke.org>: host lookup did not complete Apr 14 19:30:55 thebighonker exim[14846]: H=(DaVinci-MWare.prophet21lab.com) [23.100.68.192]:52130 I=[192.147.25.65]:25 F=<info@duke.org> temporarily rejected RCPT <jpotyh@why.net>: Could not complete sender verify Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login: user=<ctr>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS, session=<bVWMxIaGJtogAQRwHw8DrQC73P/+UNkA> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP address 80.253.235.35 Apr 14 19:31:19 thebighonker dovecot[2507]: indexer-worker(ler/14919): Indexed 1578 messages in lists/freebsd/ports-commiters (UIDs 21067..22644) ^C [ler@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox lists/freebsd/ports-commiters body 'sysutils/' 8097632f69627b5b58950000bbe98eac 21077 8097632f69627b5b58950000bbe98eac 21082 8097632f69627b5b58950000bbe98eac 21083 8097632f69627b5b58950000bbe98eac 21086 8097632f69627b5b58950000bbe98eac 21118 8097632f69627b5b58950000bbe98eac 21119 8097632f69627b5b58950000bbe98eac 21121 8097632f69627b5b58950000bbe98eac 21124 8097632f69627b5b58950000bbe98eac 21125 8097632f69627b5b58950000bbe98eac 21126 8097632f69627b5b58950000bbe98eac 21127 8097632f69627b5b58950000bbe98eac 21128 8097632f69627b5b58950000bbe98eac 21141 8097632f69627b5b58950000bbe98eac 21142 8097632f69627b5b58950000bbe98eac 21168 8097632f69627b5b58950000bbe98eac 21175 8097632f69627b5b58950000bbe98eac 21180 8097632f69627b5b58950000bbe98eac 21184 8097632f69627b5b58950000bbe98eac 21186 8097632f69627b5b58950000bbe98eac 21188 8097632f69627b5b58950000bbe98eac 21195 8097632f69627b5b58950000bbe98eac 21196 8097632f69627b5b58950000bbe98eac 21198 8097632f69627b5b58950000bbe98eac 21292 8097632f69627b5b58950000bbe98eac 21312 8097632f69627b5b58950000bbe98eac 21313 8097632f69627b5b58950000bbe98eac 21323 8097632f69627b5b58950000bbe98eac 21330 8097632f69627b5b58950000bbe98eac 21344 8097632f69627b5b58950000bbe98eac 21345 8097632f69627b5b58950000bbe98eac 21348 8097632f69627b5b58950000bbe98eac 21353 8097632f69627b5b58950000bbe98eac 21354 8097632f69627b5b58950000bbe98eac 21359 8097632f69627b5b58950000bbe98eac 21367 8097632f69627b5b58950000bbe98eac 21368 8097632f69627b5b58950000bbe98eac 21369 8097632f69627b5b58950000bbe98eac 21370 8097632f69627b5b58950000bbe98eac 21371 8097632f69627b5b58950000bbe98eac 21380 8097632f69627b5b58950000bbe98eac 21390 8097632f69627b5b58950000bbe98eac 21391 8097632f69627b5b58950000bbe98eac 21392 8097632f69627b5b58950000bbe98eac 21393 8097632f69627b5b58950000bbe98eac 21394 8097632f69627b5b58950000bbe98eac 21395 8097632f69627b5b58950000bbe98eac 21439 8097632f69627b5b58950000bbe98eac 21440 8097632f69627b5b58950000bbe98eac 21480 8097632f69627b5b58950000bbe98eac 21518 8097632f69627b5b58950000bbe98eac 21538 8097632f69627b5b58950000bbe98eac 21539 8097632f69627b5b58950000bbe98eac 21593 8097632f69627b5b58950000bbe98eac 21610 8097632f69627b5b58950000bbe98eac 21612 8097632f69627b5b58950000bbe98eac 21615 8097632f69627b5b58950000bbe98eac 21682 8097632f69627b5b58950000bbe98eac 21696 8097632f69627b5b58950000bbe98eac 21697 8097632f69627b5b58950000bbe98eac 21700 8097632f69627b5b58950000bbe98eac 21701 8097632f69627b5b58950000bbe98eac 21710 8097632f69627b5b58950000bbe98eac 21743 8097632f69627b5b58950000bbe98eac 21856 8097632f69627b5b58950000bbe98eac 21858 8097632f69627b5b58950000bbe98eac 21882 8097632f69627b5b58950000bbe98eac 21883 8097632f69627b5b58950000bbe98eac 21886 8097632f69627b5b58950000bbe98eac 21887 8097632f69627b5b58950000bbe98eac 21900 8097632f69627b5b58950000bbe98eac 21910 8097632f69627b5b58950000bbe98eac 21918 8097632f69627b5b58950000bbe98eac 21930 8097632f69627b5b58950000bbe98eac 21931 8097632f69627b5b58950000bbe98eac 21955 8097632f69627b5b58950000bbe98eac 21971 8097632f69627b5b58950000bbe98eac 21986 8097632f69627b5b58950000bbe98eac 22027 8097632f69627b5b58950000bbe98eac 22031 8097632f69627b5b58950000bbe98eac 22045 8097632f69627b5b58950000bbe98eac 22050 8097632f69627b5b58950000bbe98eac 22051 8097632f69627b5b58950000bbe98eac 22062 8097632f69627b5b58950000bbe98eac 22106 8097632f69627b5b58950000bbe98eac 22112 8097632f69627b5b58950000bbe98eac 22113 8097632f69627b5b58950000bbe98eac 22116 8097632f69627b5b58950000bbe98eac 22119 8097632f69627b5b58950000bbe98eac 22151 8097632f69627b5b58950000bbe98eac 22155 8097632f69627b5b58950000bbe98eac 22192 8097632f69627b5b58950000bbe98eac 22196 8097632f69627b5b58950000bbe98eac 22204 8097632f69627b5b58950000bbe98eac 22205 8097632f69627b5b58950000bbe98eac 22219 8097632f69627b5b58950000bbe98eac 22237 8097632f69627b5b58950000bbe98eac 22241 8097632f69627b5b58950000bbe98eac 22294 8097632f69627b5b58950000bbe98eac 22309 8097632f69627b5b58950000bbe98eac 22317 8097632f69627b5b58950000bbe98eac 22331 8097632f69627b5b58950000bbe98eac 22338 8097632f69627b5b58950000bbe98eac 22343 8097632f69627b5b58950000bbe98eac 22344 8097632f69627b5b58950000bbe98eac 22404 8097632f69627b5b58950000bbe98eac 22409 8097632f69627b5b58950000bbe98eac 22461 8097632f69627b5b58950000bbe98eac 22466 8097632f69627b5b58950000bbe98eac 22553 8097632f69627b5b58950000bbe98eac 22556 8097632f69627b5b58950000bbe98eac 22564 8097632f69627b5b58950000bbe98eac 22565 8097632f69627b5b58950000bbe98eac 22599 8097632f69627b5b58950000bbe98eac 22605 8097632f69627b5b58950000bbe98eac 22618 8097632f69627b5b58950000bbe98eac 22619 8097632f69627b5b58950000bbe98eac 22620 8097632f69627b5b58950000bbe98eac 22621 8097632f69627b5b58950000bbe98eac 22622 8097632f69627b5b58950000bbe98eac 22623 8097632f69627b5b58950000bbe98eac 22624 8097632f69627b5b58950000bbe98eac 22625 8097632f69627b5b58950000bbe98eac 22626 8097632f69627b5b58950000bbe98eac 22627 8097632f69627b5b58950000bbe98eac 22628 8097632f69627b5b58950000bbe98eac 22629 8097632f69627b5b58950000bbe98eac 22630 8097632f69627b5b58950000bbe98eac 22631 8097632f69627b5b58950000bbe98eac 22632 8097632f69627b5b58950000bbe98eac 22633 8097632f69627b5b58950000bbe98eac 22634 8097632f69627b5b58950000bbe98eac 22635 8097632f69627b5b58950000bbe98eac 22636 8097632f69627b5b58950000bbe98eac 22637 8097632f69627b5b58950000bbe98eac 22638 8097632f69627b5b58950000bbe98eac 22639 8097632f69627b5b58950000bbe98eac 22640 8097632f69627b5b58950000bbe98eac 22641 8097632f69627b5b58950000bbe98eac 22642 8097632f69627b5b58950000bbe98eac 22643 [ler@thebighonker.lerctr.org:~] $

So, why does the FTS rescan catch it, but the autoindex doesn't? mail is stored in this mailbox by pigeonhole scripts using lmtp.

-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Aki Tuomi

8:55 a.m.

New subject: SOLR/Index?

On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:

...

⌂72% [ler@thebighonker.lerctr.org:~] $ doveadm search mailbox lists/freebsd/ports-commiters body 'sysutils' [ler@thebighonker.lerctr.org:~] $ doveadm fts rescan [ler@thebighonker.lerctr.org:~] $ doveadm index -q lists/freebsd/ports-commiters ⌂64% [ler@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected (auth failed, 1 attempts in 2 secs): user=<gwc>, method=PLAIN, rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed, session=<EAk5woaGV4S0tNl8> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login: user=<ler>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS, session=<lr1mwoaGwYMgAQRwHw8DrQC73P/+UNkA> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP address 23.100.68.192 Apr 14 19:30:55 thebighonker exim[14846]: H=(DaVinci-MWare.prophet21lab.com <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130 I=[192.147.25.65]:25 sender verify defer for <info@duke.org <mailto:info@duke.org>>: host lookup did not complete Apr 14 19:30:55 thebighonker exim[14846]: H=(DaVinci-MWare.prophet21lab.com <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130 I=[192.147.25.65]:25 F=<info@duke.org <mailto:info@duke.org>> temporarily rejected RCPT <jpotyh@why.net <mailto:jpotyh@why.net>>: Could not complete sender verify Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login: user=<ctr>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS, session=<bVWMxIaGJtogAQRwHw8DrQC73P/+UNkA> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP address 80.253.235.35 Apr 14 19:31:19 thebighonker dovecot[2507]: indexer-worker(ler/14919): Indexed 1578 messages in lists/freebsd/ports-commiters (UIDs 21067..22644) ^C [ler@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox lists/freebsd/ports-commiters body 'sysutils/'

Just minor nit, but you are searching for 'sysutils' first, then 'sysutils/'. FTS does not do substring searches by default.

Aki

...

Larry Rosenman

9:09 a.m.

New subject: SOLR/Index?

Note the hits after the fts rescan/index.

Get Outlook for Android<https://aka.ms/ghei36>

From: Aki Tuomi <aki.tuomi@open-xchange.com> Sent: Monday, April 15, 2019 12:55:07 AM To: Larry Rosenman; John Fawcett Cc: Dovecot Mailing List Subject: Re: SOLR/Index?

On 15.4.2019 3.33, Larry Rosenman via dovecot wrote: ⌂72% [ler@thebighonker.lerctr.org:~<mailto:ler@thebighonker.lerctr.org:~>] $ doveadm search mailbox lists/freebsd/ports-commiters body 'sysutils' [ler@thebighonker.lerctr.org:~<mailto:ler@thebighonker.lerctr.org:~>] $ doveadm fts rescan [ler@thebighonker.lerctr.org:~<mailto:ler@thebighonker.lerctr.org:~>] $ doveadm index -q lists/freebsd/ports-commiters ⌂64% [ler@thebighonker.lerctr.org:~<mailto:ler@thebighonker.lerctr.org:~>] $ tail -f /var/log/maillog Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected (auth failed, 1 attempts in 2 secs): user=<gwc>, method=PLAIN, rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed, session=<EAk5woaGV4S0tNl8> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login: user=<ler>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS, session=<lr1mwoaGwYMgAQRwHw8DrQC73P/+UNkA> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP address 23.100.68.192 Apr 14 19:30:55 thebighonker exim[14846]: H=(DaVinci-MWare.prophet21lab.com<http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130 I=[192.147.25.65]:25 sender verify defer for <info@duke.org<mailto:info@duke.org>>: host lookup did not complete Apr 14 19:30:55 thebighonker exim[14846]: H=(DaVinci-MWare.prophet21lab.com<http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130 I=[192.147.25.65]:25 F=<info@duke.org<mailto:info@duke.org>> temporarily rejected RCPT <jpotyh@why.net<mailto:jpotyh@why.net>>: Could not complete sender verify Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login: user=<ctr>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS, session=<bVWMxIaGJtogAQRwHw8DrQC73P/+UNkA> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP address 80.253.235.35 Apr 14 19:31:19 thebighonker dovecot[2507]: indexer-worker(ler/14919): Indexed 1578 messages in lists/freebsd/ports-commiters (UIDs 21067..22644) ^C [ler@thebighonker.lerctr.org:~<mailto:ler@thebighonker.lerctr.org:~>] 130 $ doveadm search mailbox lists/freebsd/ports-commiters body 'sysutils/'

Just minor nit, but you are searching for 'sysutils' first, then 'sysutils/'. FTS does not do substring searches by default.

Aki

-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com<mailto:larryrtx@gmail.com> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

John Fawcett

10:06 a.m.

New subject: SOLR/Index?

On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:

...

Note the hits after the fts rescan/index.

Get Outlook for Android <https://aka.ms/ghei36>

*From:* Aki Tuomi <aki.tuomi@open-xchange.com> *Sent:* Monday, April 15, 2019 12:55:07 AM *To:* Larry Rosenman; John Fawcett *Cc:* Dovecot Mailing List *Subject:* Re: SOLR/Index?

On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:

...
⌂72% [ler@thebighonker.lerctr.org:~] $ doveadm search mailbox lists/freebsd/ports-commiters body 'sysutils' [ler@thebighonker.lerctr.org:~] $ doveadm fts rescan [ler@thebighonker.lerctr.org:~] $ doveadm index -q lists/freebsd/ports-commiters ⌂64% [ler@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected (auth failed, 1 attempts in 2 secs): user=<gwc>, method=PLAIN, rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed, session=<EAk5woaGV4S0tNl8> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login: user=<ler>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS, session=<lr1mwoaGwYMgAQRwHw8DrQC73P/+UNkA> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP address 23.100.68.192 Apr 14 19:30:55 thebighonker exim[14846]: H=(DaVinci-MWare.prophet21lab.com <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130 I=[192.147.25.65]:25 sender verify defer for <info@duke.org <mailto:info@duke.org>>: host lookup did not complete Apr 14 19:30:55 thebighonker exim[14846]: H=(DaVinci-MWare.prophet21lab.com <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130 I=[192.147.25.65]:25 F=<info@duke.org <mailto:info@duke.org>> temporarily rejected RCPT <jpotyh@why.net <mailto:jpotyh@why.net>>: Could not complete sender verify Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login: user=<ctr>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS, session=<bVWMxIaGJtogAQRwHw8DrQC73P/+UNkA> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP address 80.253.235.35 Apr 14 19:31:19 thebighonker dovecot[2507]: indexer-worker(ler/14919): Indexed 1578 messages in lists/freebsd/ports-commiters (UIDs 21067..22644) ^C [ler@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox lists/freebsd/ports-commiters body 'sysutils/'

Just minor nit, but you are searching for 'sysutils' first, then 'sysutils/'. FTS does not do substring searches by default.

Aki

...
-- Larry Rosenman http://www.lerctr.org/~ler <http://www.lerctr.org/~ler> Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com <mailto:larryrtx@gmail.com> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

just to be sure: are you running a standard unmodified 2.3.5.1 version which you built from source code?

I can see that first you search for sysutils, then do a rescan and reindex (which is shown in the log) and then you are able to find sysutils/.

It is better when doing these tests to search for the same string before and after, just to eliminate too many different factors in the test.

Nevertheless I did not see your logging for what happens when you receive a test message containing sysutils/. Dovecot should be outputing info about autoindexing given your setup. Does it do that or does it give some other message? Can you show those logs?

John

Larry Rosenman

11:31 a.m.

New subject: SOLR/Index?

It always shows the autoindex. And yes built from sources. I'm the FreeBSD port maintainer for mail/docecot. This has been happening for several releases.

Get Outlook for Android<https://aka.ms/ghei36>

From: dovecot <dovecot-bounces@dovecot.org> on behalf of John Fawcett via dovecot <dovecot@dovecot.org> Sent: Monday, April 15, 2019 2:06:55 AM To: dovecot@dovecot.org Subject: Re: SOLR/Index?

On 15/04/2019 08:09, Larry Rosenman via dovecot wrote: Note the hits after the fts rescan/index.

Get Outlook for Android<https://aka.ms/ghei36>

From: Aki Tuomi <aki.tuomi@open-xchange.com><mailto:aki.tuomi@open-xchange.com> Sent: Monday, April 15, 2019 12:55:07 AM To: Larry Rosenman; John Fawcett Cc: Dovecot Mailing List Subject: Re: SOLR/Index?

Just minor nit, but you are searching for 'sysutils' first, then 'sysutils/'. FTS does not do substring searches by default.

Aki

-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com<mailto:larryrtx@gmail.com> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

just to be sure: are you running a standard unmodified 2.3.5.1 version which you built from source code?

I can see that first you search for sysutils, then do a rescan and reindex (which is shown in the log) and then you are able to find sysutils/.

It is better when doing these tests to search for the same string before and after, just to eliminate too many different factors in the test.

John

John Fawcett

11:57 a.m.

New subject: SOLR/Index?

On 15/04/2019 10:31, Larry Rosenman via dovecot wrote:

...

It always shows the autoindex. And yes built from sources. I'm the FreeBSD port maintainer for mail/docecot. This has been happening for several releases.

Get Outlook for Android <https://aka.ms/ghei36>

*From:* dovecot <dovecot-bounces@dovecot.org> on behalf of John Fawcett via dovecot <dovecot@dovecot.org> *Sent:* Monday, April 15, 2019 2:06:55 AM *To:* dovecot@dovecot.org *Subject:* Re: SOLR/Index? On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:

...
Note the hits after the fts rescan/index.

Get Outlook for Android <https://aka.ms/ghei36>

*From:* Aki Tuomi <aki.tuomi@open-xchange.com> *Sent:* Monday, April 15, 2019 12:55:07 AM *To:* Larry Rosenman; John Fawcett *Cc:* Dovecot Mailing List *Subject:* Re: SOLR/Index?

On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:

...
⌂72% [ler@thebighonker.lerctr.org:~] $ doveadm search mailbox lists/freebsd/ports-commiters body 'sysutils' [ler@thebighonker.lerctr.org:~] $ doveadm fts rescan [ler@thebighonker.lerctr.org:~] $ doveadm index -q lists/freebsd/ports-commiters ⌂64% [ler@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected (auth failed, 1 attempts in 2 secs): user=<gwc>, method=PLAIN, rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed, session=<EAk5woaGV4S0tNl8> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login: user=<ler>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS, session=<lr1mwoaGwYMgAQRwHw8DrQC73P/+UNkA> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP address 23.100.68.192 Apr 14 19:30:55 thebighonker exim[14846]: H=(DaVinci-MWare.prophet21lab.com <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130 I=[192.147.25.65]:25 sender verify defer for <info@duke.org <mailto:info@duke.org>>: host lookup did not complete Apr 14 19:30:55 thebighonker exim[14846]: H=(DaVinci-MWare.prophet21lab.com <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130 I=[192.147.25.65]:25 F=<info@duke.org <mailto:info@duke.org>> temporarily rejected RCPT <jpotyh@why.net <mailto:jpotyh@why.net>>: Could not complete sender verify Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login: user=<ctr>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS, session=<bVWMxIaGJtogAQRwHw8DrQC73P/+UNkA> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP address 80.253.235.35 Apr 14 19:31:19 thebighonker dovecot[2507]: indexer-worker(ler/14919): Indexed 1578 messages in lists/freebsd/ports-commiters (UIDs 21067..22644) ^C [ler@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox lists/freebsd/ports-commiters body 'sysutils/'

Just minor nit, but you are searching for 'sysutils' first, then 'sysutils/'. FTS does not do substring searches by default.

Aki

...
-- Larry Rosenman http://www.lerctr.org/~ler <http://www.lerctr.org/~ler> Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com <mailto:larryrtx@gmail.com> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

just to be sure: are you running a standard unmodified 2.3.5.1 version which you built from source code?

I can see that first you search for sysutils, then do a rescan and reindex (which is shown in the log) and then you are able to find sysutils/.

It is better when doing these tests to search for the same string before and after, just to eliminate too many different factors in the test.

Nevertheless I did not see your logging for what happens when you receive a test message containing sysutils/. Dovecot should be outputing info about autoindexing given your setup. Does it do that or does it give some other message? Can you show those logs?

John

Larry

Did you notice any difference between the logging for auto indexing and the logging for indexing that you triggered manually? Would you mind posting the auto indexing logging for a message to that same user (ler)?

best regards

John

Larry Rosenman

11:59 a.m.

New subject: SOLR/Index?

I'll run a full test when I'm back in front of areal computer vs. My phone.(in a few hours)

Get Outlook for Android<https://aka.ms/ghei36>

From: dovecot <dovecot-bounces@dovecot.org> on behalf of John Fawcett via dovecot <dovecot@dovecot.org> Sent: Monday, April 15, 2019 3:57:08 AM To: Dovecot Mailing List Subject: Re: SOLR/Index?

On 15/04/2019 10:31, Larry Rosenman via dovecot wrote: It always shows the autoindex. And yes built from sources. I'm the FreeBSD port maintainer for mail/docecot. This has been happening for several releases.

Get Outlook for Android<https://aka.ms/ghei36>

From: dovecot <dovecot-bounces@dovecot.org><mailto:dovecot-bounces@dovecot.org> on behalf of John Fawcett via dovecot <dovecot@dovecot.org><mailto:dovecot@dovecot.org> Sent: Monday, April 15, 2019 2:06:55 AM To: dovecot@dovecot.org<mailto:dovecot@dovecot.org> Subject: Re: SOLR/Index?

On 15/04/2019 08:09, Larry Rosenman via dovecot wrote: Note the hits after the fts rescan/index.

Get Outlook for Android<https://aka.ms/ghei36>

Just minor nit, but you are searching for 'sysutils' first, then 'sysutils/'. FTS does not do substring searches by default.

Aki

-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com<mailto:larryrtx@gmail.com> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

just to be sure: are you running a standard unmodified 2.3.5.1 version which you built from source code?

I can see that first you search for sysutils, then do a rescan and reindex (which is shown in the log) and then you are able to find sysutils/.

It is better when doing these tests to search for the same string before and after, just to eliminate too many different factors in the test.

John

Larry

best regards

John

John Fawcett

12:05 p.m.

New subject: SOLR/Index?

On 15/04/2019 10:59, Larry Rosenman via dovecot wrote:

...

I'll run a full test when I'm back in front of areal computer vs. My phone.(in a few hours)

Get Outlook for Android <https://aka.ms/ghei36>

*From:* dovecot <dovecot-bounces@dovecot.org> on behalf of John Fawcett via dovecot <dovecot@dovecot.org> *Sent:* Monday, April 15, 2019 3:57:08 AM *To:* Dovecot Mailing List *Subject:* Re: SOLR/Index? On 15/04/2019 10:31, Larry Rosenman via dovecot wrote:

...
It always shows the autoindex. And yes built from sources. I'm the FreeBSD port maintainer for mail/docecot. This has been happening for several releases.

Get Outlook for Android <https://aka.ms/ghei36>

*From:* dovecot <dovecot-bounces@dovecot.org> on behalf of John Fawcett via dovecot <dovecot@dovecot.org> *Sent:* Monday, April 15, 2019 2:06:55 AM *To:* dovecot@dovecot.org *Subject:* Re: SOLR/Index? On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:

...
Note the hits after the fts rescan/index.

Get Outlook for Android <https://aka.ms/ghei36>

*From:* Aki Tuomi <aki.tuomi@open-xchange.com> *Sent:* Monday, April 15, 2019 12:55:07 AM *To:* Larry Rosenman; John Fawcett *Cc:* Dovecot Mailing List *Subject:* Re: SOLR/Index?

On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:

...
⌂72% [ler@thebighonker.lerctr.org:~] $ doveadm search mailbox lists/freebsd/ports-commiters body 'sysutils' [ler@thebighonker.lerctr.org:~] $ doveadm fts rescan [ler@thebighonker.lerctr.org:~] $ doveadm index -q lists/freebsd/ports-commiters ⌂64% [ler@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected (auth failed, 1 attempts in 2 secs): user=<gwc>, method=PLAIN, rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed, session=<EAk5woaGV4S0tNl8> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login: user=<ler>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS, session=<lr1mwoaGwYMgAQRwHw8DrQC73P/+UNkA> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP address 23.100.68.192 Apr 14 19:30:55 thebighonker exim[14846]: H=(DaVinci-MWare.prophet21lab.com <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130 I=[192.147.25.65]:25 sender verify defer for <info@duke.org <mailto:info@duke.org>>: host lookup did not complete Apr 14 19:30:55 thebighonker exim[14846]: H=(DaVinci-MWare.prophet21lab.com <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130 I=[192.147.25.65]:25 F=<info@duke.org <mailto:info@duke.org>> temporarily rejected RCPT <jpotyh@why.net <mailto:jpotyh@why.net>>: Could not complete sender verify Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login: user=<ctr>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS, session=<bVWMxIaGJtogAQRwHw8DrQC73P/+UNkA> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP address 80.253.235.35 Apr 14 19:31:19 thebighonker dovecot[2507]: indexer-worker(ler/14919): Indexed 1578 messages in lists/freebsd/ports-commiters (UIDs 21067..22644) ^C [ler@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox lists/freebsd/ports-commiters body 'sysutils/'

Just minor nit, but you are searching for 'sysutils' first, then 'sysutils/'. FTS does not do substring searches by default.

Aki

...
-- Larry Rosenman http://www.lerctr.org/~ler <http://www.lerctr.org/~ler> Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com <mailto:larryrtx@gmail.com> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

just to be sure: are you running a standard unmodified 2.3.5.1 version which you built from source code?

I can see that first you search for sysutils, then do a rescan and reindex (which is shown in the log) and then you are able to find sysutils/.

It is better when doing these tests to search for the same string before and after, just to eliminate too many different factors in the test.

Nevertheless I did not see your logging for what happens when you receive a test message containing sysutils/. Dovecot should be outputing info about autoindexing given your setup. Does it do that or does it give some other message? Can you show those logs?

John

Larry

Did you notice any difference between the logging for auto indexing and the logging for indexing that you triggered manually? Would you mind posting the auto indexing logging for a message to that same user (ler)?

best regards

John

ok, the hypothesis I'm trying to confirm is that auto indexing is taking place under the full username ler@somedomain whereas in the command line tests you did your are using user ler (as presumably is roundcube).

John

Larry Rosenman

12:38 p.m.

New subject: SOLR/Index?

⌂63% [ler@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX ⌂67% [ler@thebighonker.lerctr.org:~] 1 $ mail -s "test fts1970" ler@lerctr.org test fts1970

test fts1970 . EOT [ler@thebighonker.lerctr.org:~] $ mailq [ler@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX Subject: test fts1970 test fts1970 test fts1970

Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <= ler@lerctr.org U=ler P=local S=388 Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect from local Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler@lerctr.org/49364): save: box=INBOX, uid=175402, msgid=< E1hFxvD-000Csq-P6@thebighonker.lerctr.org>, size=640, vsize=660, from=Larry Rosenman <ler@lerctr.org>, subject=test fts1970, flags=() Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler@lerctr.org/49364): sieve: msgid=<E1hFxvD-000Csq-P6@thebighonker.lerctr.org>: stored mail into mailbox 'INBOX' (subject=test fts1970 from=ler@lerctr.org size=660) Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect from local: Client has quit the connection (state=READY) Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler < ler@lerctr.org> R=localuser T=dovecot_lmtp S=404 C="250 2.0.0 < ler@lerctr.org> 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 Completed QT=0s Apr 15 04:29:03 thebighonker dovecot[2507]: indexer-worker( ler@lerctr.org/49366): Indexed 1 messages in INBOX (UIDs 175402..175402)

⌂81% [ler@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX body 'fts1970' ⌂83% [ler@thebighonker.lerctr.org:~] $

⌂65% [ler@thebighonker.lerctr.org:~] 75 $ doveadm search -u ler@lerctr.org mailbox INBOX body 'fts1970' a53a143be44bda5bd4830000bbe98eac 175402 [ler@thebighonker.lerctr.org:~] $ doveadm index -q INBOX [ler@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX body 'fts1970' [ler@thebighonker.lerctr.org:~] $ doveadm fts rescan [ler@thebighonker.lerctr.org:~] $ doveadm index -q INBOX [ler@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX body 'fts1970' a53a143be44bda5bd4830000bbe98eac 175402 [ler@thebighonker.lerctr.org:~] $ doveadm search -u ler@lerctr.org mailbox INBOX body 'fts1970' a53a143be44bda5bd4830000bbe98eac 175402 [ler@thebighonker.lerctr.org:~] $

So, yes, your hypothesis is correct.

Question: How can I make it consistent?

I have a script that runs on the first of the month that does archiving, and I have similar issues in that namespace: ⌂67% [ler@thebighonker.lerctr.org:~] $ cat bin/archive-mail #!/bin/sh PATH=$PATH:/usr/local/bin #Expects to be run after midnight on the first of the month # to archive all the previous months mail #Date Run: TODAY=date "+%Y-%m-%d" #last month in YYYY/MM YEAR_LAST_MONTH=date -v-1d "+%Y/%m" #1st of last month as 01-Mon-YYYY FIRST_LAST_MONTH=date -v-1d "+01-%b-%Y" echo 'TODAY=' ${TODAY} echo 'YEAR_LAST_MONTH=' ${YEAR_LAST_MONTH} echo 'FIRST_LAST_MONTH=' ${FIRST_LAST_MONTH} # get a list of all the mailboxes with at least one real message doveadm -f tab mailbox status vsize \* 2>/dev/null | sed -e 1d | sort -k 1,1 | awk 'BEGIN {FS="\t"} {if ($2 > 0) print $1}' | while read i do echo date start ${i} doveadm mailbox create "ARCHIVE/${YEAR_LAST_MONTH}/${i}" doveadm -f tab mailbox status messages "${i}" doveadm move "ARCHIVE/${YEAR_LAST_MONTH}/${i}" mailbox
"${i}" BEFORE ${TODAY} SINCE ${FIRST_LAST_MONTH} doveadm -f tab mailbox status messages "${i}" echo date done ${i} done ⌂64% [ler@thebighonker.lerctr.org:~] $

The Exim config can be provided as well if necessary.

ler & ler@lerctr.org *ARE THE SAME MAILBOX*

On Mon, Apr 15, 2019 at 4:05 AM John Fawcett via dovecot < dovecot@dovecot.org> wrote:

...

On 15/04/2019 10:59, Larry Rosenman via dovecot wrote:

I'll run a full test when I'm back in front of areal computer vs. My phone.(in a few hours)

Get Outlook for Android <https://aka.ms/ghei36>

*From:* dovecot <dovecot-bounces@dovecot.org> <dovecot-bounces@dovecot.org> on behalf of John Fawcett via dovecot <dovecot@dovecot.org> <dovecot@dovecot.org> *Sent:* Monday, April 15, 2019 3:57:08 AM *To:* Dovecot Mailing List *Subject:* Re: SOLR/Index?

On 15/04/2019 10:31, Larry Rosenman via dovecot wrote:

It always shows the autoindex. And yes built from sources. I'm the FreeBSD port maintainer for mail/docecot. This has been happening for several releases.

Get Outlook for Android <https://aka.ms/ghei36>

*From:* dovecot <dovecot-bounces@dovecot.org> <dovecot-bounces@dovecot.org> on behalf of John Fawcett via dovecot <dovecot@dovecot.org> <dovecot@dovecot.org> *Sent:* Monday, April 15, 2019 2:06:55 AM *To:* dovecot@dovecot.org *Subject:* Re: SOLR/Index?

On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:

Note the hits after the fts rescan/index.

Get Outlook for Android <https://aka.ms/ghei36>

*From:* Aki Tuomi <aki.tuomi@open-xchange.com> <aki.tuomi@open-xchange.com> *Sent:* Monday, April 15, 2019 12:55:07 AM *To:* Larry Rosenman; John Fawcett *Cc:* Dovecot Mailing List *Subject:* Re: SOLR/Index?

On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:

⌂72% [ler@thebighonker.lerctr.org:~] $ doveadm search mailbox lists/freebsd/ports-commiters body 'sysutils' [ler@thebighonker.lerctr.org:~] $ doveadm fts rescan [ler@thebighonker.lerctr.org:~] $ doveadm index -q lists/freebsd/ports-commiters ⌂64% [ler@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected (auth failed, 1 attempts in 2 secs): user=<gwc>, method=PLAIN, rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed, session=<EAk5woaGV4S0tNl8> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login: user=<ler>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS, session=<lr1mwoaGwYMgAQRwHw8DrQC73P/+UNkA> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP address 23.100.68.192 Apr 14 19:30:55 thebighonker exim[14846]: H=( DaVinci-MWare.prophet21lab.com) [23.100.68.192]:52130 I=[192.147.25.65]:25 sender verify defer for <info@duke.org>: host lookup did not complete Apr 14 19:30:55 thebighonker exim[14846]: H=( DaVinci-MWare.prophet21lab.com) [23.100.68.192]:52130 I=[192.147.25.65]:25 F=<info@duke.org> temporarily rejected RCPT < jpotyh@why.net>: Could not complete sender verify Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login: user=<ctr>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS, session=<bVWMxIaGJtogAQRwHw8DrQC73P/+UNkA> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP address 80.253.235.35 Apr 14 19:31:19 thebighonker dovecot[2507]: indexer-worker(ler/14919): Indexed 1578 messages in lists/freebsd/ports-commiters (UIDs 21067..22644) ^C [ler@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox lists/freebsd/ports-commiters body 'sysutils/'

Just minor nit, but you are searching for 'sysutils' first, then 'sysutils/'. FTS does not do substring searches by default.

Aki

-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

just to be sure: are you running a standard unmodified 2.3.5.1 version which you built from source code?

I can see that first you search for sysutils, then do a rescan and reindex (which is shown in the log) and then you are able to find sysutils/.

It is better when doing these tests to search for the same string before and after, just to eliminate too many different factors in the test.

Nevertheless I did not see your logging for what happens when you receive a test message containing sysutils/. Dovecot should be outputing info about autoindexing given your setup. Does it do that or does it give some other message? Can you show those logs?

John

Larry

Did you notice any difference between the logging for auto indexing and the logging for indexing that you triggered manually? Would you mind posting the auto indexing logging for a message to that same user (ler)?

best regards

John

ok, the hypothesis I'm trying to confirm is that auto indexing is taking place under the full username ler@somedomain whereas in the command line tests you did your are using user ler (as presumably is roundcube).

John

-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

John Fawcett

1:53 p.m.

New subject: SOLR/Index?

On 15/04/2019 11:38, Larry Rosenman via dovecot wrote:

...

⌂63% [ler@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX ⌂67% [ler@thebighonker.lerctr.org:~] 1 $ mail -s "test fts1970" ler@lerctr.org <mailto:ler@lerctr.org> test fts1970

test fts1970 . EOT [ler@thebighonker.lerctr.org:~] $ mailq [ler@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX Subject: test fts1970 test fts1970 test fts1970

Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <= ler@lerctr.org <mailto:ler@lerctr.org> U=ler P=local S=388 Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect from local Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler@lerctr.org/49364 <http://ler@lerctr.org/49364>): save: box=INBOX, uid=175402, msgid=<E1hFxvD-000Csq-P6@thebighonker.lerctr.org <mailto:E1hFxvD-000Csq-P6@thebighonker.lerctr.org>>, size=640, vsize=660, from=Larry Rosenman <ler@lerctr.org <mailto:ler@lerctr.org>>, subject=test fts1970, flags=() Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler@lerctr.org/49364 <http://ler@lerctr.org/49364>): sieve: msgid=<E1hFxvD-000Csq-P6@thebighonker.lerctr.org <mailto:E1hFxvD-000Csq-P6@thebighonker.lerctr.org>>: stored mail into mailbox 'INBOX' (subject=test fts1970 from=ler@lerctr.org <mailto:ler@lerctr.org> size=660) Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect from local: Client has quit the connection (state=READY) Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler <ler@lerctr.org <mailto:ler@lerctr.org>> R=localuser T=dovecot_lmtp S=404 C="250 2.0.0 <ler@lerctr.org <mailto:ler@lerctr.org>> 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 Completed QT=0s Apr 15 04:29:03 thebighonker dovecot[2507]: indexer-worker(ler@lerctr.org/49366 <http://ler@lerctr.org/49366>): Indexed 1 messages in INBOX (UIDs 175402..175402)

⌂81% [ler@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX body 'fts1970' ⌂83% [ler@thebighonker.lerctr.org:~] $

⌂65% [ler@thebighonker.lerctr.org:~] 75 $ doveadm search -u ler@lerctr.org <mailto:ler@lerctr.org> mailbox INBOX body 'fts1970' a53a143be44bda5bd4830000bbe98eac 175402 [ler@thebighonker.lerctr.org:~] $ doveadm index -q INBOX [ler@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX body 'fts1970' [ler@thebighonker.lerctr.org:~] $ doveadm fts rescan [ler@thebighonker.lerctr.org:~] $ doveadm index -q INBOX [ler@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX body 'fts1970' a53a143be44bda5bd4830000bbe98eac 175402 [ler@thebighonker.lerctr.org:~] $ doveadm search -u ler@lerctr.org <mailto:ler@lerctr.org> mailbox INBOX body 'fts1970' a53a143be44bda5bd4830000bbe98eac 175402 [ler@thebighonker.lerctr.org:~] $

So, yes, your hypothesis is correct.

Question: How can I make it consistent?

I have a script that runs on the first of the month that does archiving, and I have similar issues in that namespace: ⌂67% [ler@thebighonker.lerctr.org:~] $ cat bin/archive-mail #!/bin/sh PATH=$PATH:/usr/local/bin #Expects to be run after midnight on the first of the month # to archive all the previous months mail #Date Run: TODAY=date "+%Y-%m-%d" #last month in YYYY/MM YEAR_LAST_MONTH=date -v-1d "+%Y/%m" #1st of last month as 01-Mon-YYYY FIRST_LAST_MONTH=date -v-1d "+01-%b-%Y" echo 'TODAY=' ${TODAY} echo 'YEAR_LAST_MONTH=' ${YEAR_LAST_MONTH} echo 'FIRST_LAST_MONTH=' ${FIRST_LAST_MONTH} # get a list of all the mailboxes with at least one real message doveadm -f tab mailbox status vsize \* 2>/dev/null | sed -e 1d | sort -k 1,1 | awk 'BEGIN {FS="\t"} {if ($2 > 0) print $1}' | while read i do echo date start ${i} doveadm mailbox create "ARCHIVE/${YEAR_LAST_MONTH}/${i}" doveadm -f tab mailbox status messages "${i}" doveadm move "ARCHIVE/${YEAR_LAST_MONTH}/${i}" mailbox
"${i}" BEFORE ${TODAY} SINCE ${FIRST_LAST_MONTH} doveadm -f tab mailbox status messages "${i}" echo date done ${i} done ⌂64% [ler@thebighonker.lerctr.org:~] $

The Exim config can be provided as well if necessary.

ler & ler@lerctr.org <mailto:ler@lerctr.org> *ARE THE SAME MAILBOX*

At the moment it looks as though you have two sets of emails indexed in solr. One is indexed under username (the one you are running mannually and apparently the one used by roundcube too, but that's to be verified) and another set being indexed by autoindex = yes option using the full email address. Once you've got it working as you require, then you may want to clean out solr and reindex with just one of them just to reduce volumes.

Your setup seems to have a mix of users from mysql and from /etc/passwd. Not sure if your mysql users are all mapped to real users or they have their own mailboxes with domain included. Your solution will depend on what you really need and if the setup is working correctly you may not want to tweak it too much or other things may start breaking.

The simplest thing that comes to mind (providing your manually indexed mails show up in roundcube searches) is just to turn off autoindex and schedule indexing from cron. As things stand I believe autoindex works well with virtual users username@domain. You'll probably need to start off with a rescan if you do this.

If you can swith to all virtual users without local users then probably you can use autoindex, but I can understand that may not be possible. For sure I'd recommend doing it in a test environment first so you can be sure of the configuration and then look at migrating your existing mailboxes over to virtual users.

John

2151

Age (days ago)

2152

Last active (days ago)

List overview

11 comments

3 participants

participants (3)

Aki Tuomi
John Fawcett
Larry Rosenman

Re: SOLR/Index?

tags

participants (3)