SOLR/Index?

[John Fawcett]

On 15/04/2019 11:38, Larry Rosenman via dovecot wrote:
> ⌂63% [ler at thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX
> ⌂67% [ler at thebighonker.lerctr.org:~] 1 $ mail -s "test fts1970"
> ler at lerctr.org <mailto:ler at lerctr.org>
> test fts1970
>
> test fts1970
> .
> EOT
> [ler at thebighonker.lerctr.org:~] $ mailq
> [ler at thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX
> Subject: test fts1970
> test fts1970
> test fts1970
>
>
> Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <=
> ler at lerctr.org <mailto:ler at lerctr.org> U=ler P=local S=388
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect from
> local
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler at lerctr.org/49364
> <http://ler@lerctr.org/49364>): save: box=INBOX, uid=175402,
> msgid=<E1hFxvD-000Csq-P6 at thebighonker.lerctr.org
> <mailto:E1hFxvD-000Csq-P6 at thebighonker.lerctr.org>>, size=640,
> vsize=660, from=Larry Rosenman <ler at lerctr.org
> <mailto:ler at lerctr.org>>, subject=test fts1970, flags=()
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler at lerctr.org/49364
> <http://ler@lerctr.org/49364>): sieve:
> msgid=<E1hFxvD-000Csq-P6 at thebighonker.lerctr.org
> <mailto:E1hFxvD-000Csq-P6 at thebighonker.lerctr.org>>: stored mail into
> mailbox 'INBOX' (subject=test fts1970 from=ler at lerctr.org
> <mailto:ler at lerctr.org> size=660)
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect
> from local: Client has quit the connection (state=READY)
> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler
> <ler at lerctr.org <mailto:ler at lerctr.org>> R=localuser T=dovecot_lmtp
> S=404 C="250 2.0.0 <ler at lerctr.org <mailto:ler at lerctr.org>>
> 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s
> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 Completed QT=0s
> Apr 15 04:29:03 thebighonker dovecot[2507]:
> indexer-worker(ler at lerctr.org/49366 <http://ler@lerctr.org/49366>):
> Indexed 1 messages in INBOX (UIDs 175402..175402)
>
>
> ⌂81% [ler at thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX 
> body 'fts1970'
> ⌂83% [ler at thebighonker.lerctr.org:~] $
>
>
> ⌂65% [ler at thebighonker.lerctr.org:~] 75 $ doveadm search -u
> ler at lerctr.org <mailto:ler at lerctr.org>  mailbox INBOX body 'fts1970'
> a53a143be44bda5bd4830000bbe98eac 175402
> [ler at thebighonker.lerctr.org:~] $ doveadm index -q INBOX
> [ler at thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX  body
> 'fts1970'
> [ler at thebighonker.lerctr.org:~] $ doveadm fts rescan
> [ler at thebighonker.lerctr.org:~] $ doveadm index -q INBOX
> [ler at thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX  body
> 'fts1970'
> a53a143be44bda5bd4830000bbe98eac 175402
> [ler at thebighonker.lerctr.org:~] $ doveadm search -u ler at lerctr.org
> <mailto:ler at lerctr.org>  mailbox INBOX body 'fts1970'
> a53a143be44bda5bd4830000bbe98eac 175402
> [ler at thebighonker.lerctr.org:~] $
>
> So, yes, your hypothesis is correct.
>
> Question: How can I make it consistent?  
>
> I have a script that runs on the first of the month that does
> archiving, and I have similar issues in that namespace:
> ⌂67% [ler at thebighonker.lerctr.org:~] $ cat bin/archive-mail
> #!/bin/sh
> PATH=$PATH:/usr/local/bin
> #Expects to be run after midnight on the first of the month
> #  to archive all the previous months mail
> #Date Run:
> TODAY=`date "+%Y-%m-%d"`
> #last month in YYYY/MM
> YEAR_LAST_MONTH=`date -v-1d "+%Y/%m"`
> #1st of last month as 01-Mon-YYYY
> FIRST_LAST_MONTH=`date -v-1d "+01-%b-%Y"`
> echo 'TODAY=' ${TODAY}
> echo 'YEAR_LAST_MONTH=' ${YEAR_LAST_MONTH}
> echo 'FIRST_LAST_MONTH=' ${FIRST_LAST_MONTH}
> # get a list of all the mailboxes with at least one real message
> doveadm -f tab mailbox status vsize \* 2>/dev/null |
>         sed -e 1d | sort -k 1,1 |
>         awk  'BEGIN {FS="\t"} {if ($2 > 0)  print $1}' |
> while read i
> do
>    echo `date` start ${i}
>    doveadm mailbox create "ARCHIVE/${YEAR_LAST_MONTH}/${i}"
>    doveadm -f tab mailbox status messages "${i}"
>    doveadm move "ARCHIVE/${YEAR_LAST_MONTH}/${i}" mailbox \
>             "${i}" BEFORE ${TODAY} SINCE ${FIRST_LAST_MONTH}
>    doveadm -f tab mailbox status messages "${i}"
>    echo `date` done  ${i}
> done
> ⌂64% [ler at thebighonker.lerctr.org:~] $
>
>
> The Exim config can be provided as well if necessary.
>
> ler & ler at lerctr.org <mailto:ler at lerctr.org> *ARE THE SAME MAILBOX*
>
At the moment it looks as though you have two sets of emails indexed in
solr. One is indexed under username (the one you are running mannually
and apparently the one used by roundcube too, but that's to be verified)
and another set being indexed by autoindex = yes option using the full
email address. Once you've got it working as you require, then you may
want to clean out solr and reindex with just one of them just to reduce
volumes.

Your setup seems to have a mix of users from mysql and from /etc/passwd.
Not sure if your mysql users are all mapped to real users or they have
their own mailboxes with domain included. Your solution will depend on
what you really need and if the setup is working correctly you may not
want to tweak it too much or other things may start breaking.

The simplest thing that comes to mind (providing your manually indexed
mails show up in roundcube searches) is just to turn off autoindex and
schedule indexing from cron. As things stand I believe autoindex works
well with virtual users username at domain. You'll probably need to start
off with a rescan if you do this.

If you can swith to all virtual users without local users then probably
you can use autoindex, but I can understand that may not be possible.
For sure I'd recommend doing it in a test environment first so you can
be sure of the configuration and then look at migrating your existing
mailboxes over to virtual users.

John



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20190415/6cf94d0a/attachment.html>