Fwd: SOLR/Index?

[Larry Rosenman]

These are REAL local users, authenticated via PAM....


On Mon, Apr 15, 2019 at 6:26 AM Johan Huldtgren via dovecot <
dovecot at dovecot.org> wrote:

>
>
> On 2019/04/15 06:59, Larry Rosenman via dovecot wrote:
> > forgot to reply all.
> >
> > ---------- Forwarded message ---------
> > From: *Larry Rosenman* <larryrtx at gmail.com <mailto:larryrtx at gmail.com>>
> > Date: Mon, Apr 15, 2019 at 5:58 AM
> > Subject: Re: SOLR/Index?
> > To: John Fawcett <john at voipsupport.it <mailto:john at voipsupport.it>>
> >
> >
> > the local users (myself, my wife, a friend) can authenticate EITHER as
> <username> or <username>@lerctr.org <http://lerctr.org>.
> >
> > switching to all virtual users is NOT going to happen.
> >
> > If I login to roundcube with <user>@lerctr.org <http://lerctr.org> it
> finds the autoindexed mail.
> >
> > So, if I make everyone always authenticate as <user>@lerctr.org <
> http://lerctr.org> we should be fine.
> >
> > and change my script to do doveadm -u <user>@<domain> instead of
> depending on the local user running the cron job.
> >
> > question: Is there some way to have dovecot change what it sees to be
> <user>@lerctr.org <http://lerctr.org> when they login as <user>?
>
> FWIW, when I switched all users to be virtual to let the handful of people
> who had been logging in as <username>
> not have to switch to <username>@example.com I created a second passdb
> entry which points to different sql.conf file
> which automatically appends this specific domain, thereby letting them
> type in <username> but getting logged in as
> <username>@example.com so my dovecot logs always show <username>@
> example.com. Not sure if this helps you get around
> your issue.
>
> .jh
>
> > On Mon, Apr 15, 2019 at 5:54 AM John Fawcett via dovecot <
> dovecot at dovecot.org <mailto:dovecot at dovecot.org>> wrote:
> >
> >     On 15/04/2019 11:38, Larry Rosenman via dovecot wrote:
> >>     ⌂63% [ler at thebighonker.lerctr.org:~ <mailto:
> ler at thebighonker.lerctr.org:~>] $ grep fts1970 mail/INBOX
> >>     ⌂67% [ler at thebighonker.lerctr.org:~ <mailto:
> ler at thebighonker.lerctr.org:~>] 1 $ mail -s "test fts1970" ler at lerctr.org
> <mailto:ler at lerctr.org>
> >>     test fts1970
> >>
> >>     test fts1970
> >>     .
> >>     EOT
> >>     [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>]
> $ mailq
> >>     [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>]
> $ grep fts1970 mail/INBOX
> >>     Subject: test fts1970
> >>     test fts1970
> >>     test fts1970
> >>
> >>
> >>     Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <=
> ler at lerctr.org <mailto:ler at lerctr.org> U=ler P=local S=388
> >>     Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect
> from local
> >>     Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(
> ler at lerctr.org/49364 <http://ler@lerctr.org/49364>): save: box=INBOX,
> uid=175402, msgid=<E1hFxvD-000Csq-P6 at thebighonker.lerctr.org <mailto:
> E1hFxvD-000Csq-P6 at thebighonker.lerctr.org>>, size=640, vsize=660,
> from=Larry Rosenman <ler at lerctr.org <mailto:ler at lerctr.org>>,
> subject=test fts1970, flags=()
> >>     Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(
> ler at lerctr.org/49364 <http://ler@lerctr.org/49364>): sieve: msgid=<
> E1hFxvD-000Csq-P6 at thebighonker.lerctr.org <mailto:
> E1hFxvD-000Csq-P6 at thebighonker.lerctr.org>>: stored mail into mailbox
> 'INBOX' (subject=test fts1970 from=ler at lerctr.org <mailto:ler at lerctr.org>
> size=660)
> >>     Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect
> from local: Client has quit the connection (state=READY)
> >>     Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler <
> ler at lerctr.org <mailto:ler at lerctr.org>> R=localuser T=dovecot_lmtp S=404
> C="250 2.0.0 <ler at lerctr.org <mailto:ler at lerctr.org>>
> 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s
> >>     Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6
> Completed QT=0s
> >>     Apr 15 04:29:03 thebighonker dovecot[2507]: indexer-worker(
> ler at lerctr.org/49366 <http://ler@lerctr.org/49366>): Indexed 1 messages
> in INBOX (UIDs 175402..175402)
> >>
> >>
> >>     ⌂81% [ler at thebighonker.lerctr.org:~ <mailto:
> ler at thebighonker.lerctr.org:~>] $ doveadm search mailbox INBOX  body
> 'fts1970'
> >>     ⌂83% [ler at thebighonker.lerctr.org:~ <mailto:
> ler at thebighonker.lerctr.org:~>] $
> >>
> >>
> >>     ⌂65% [ler at thebighonker.lerctr.org:~ <mailto:
> ler at thebighonker.lerctr.org:~>] 75 $ doveadm search -u ler at lerctr.org
> <mailto:ler at lerctr.org>  mailbox INBOX body 'fts1970'
> >>     a53a143be44bda5bd4830000bbe98eac 175402
> >>     [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>]
> $ doveadm index -q INBOX
> >>     [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>]
> $ doveadm search mailbox INBOX  body 'fts1970'
> >>     [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>]
> $ doveadm fts rescan
> >>     [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>]
> $ doveadm index -q INBOX
> >>     [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>]
> $ doveadm search mailbox INBOX  body 'fts1970'
> >>     a53a143be44bda5bd4830000bbe98eac 175402
> >>     [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>]
> $ doveadm search -u ler at lerctr.org <mailto:ler at lerctr.org>  mailbox INBOX
> body 'fts1970'
> >>     a53a143be44bda5bd4830000bbe98eac 175402
> >>     [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>]
> $
> >>
> >>     So, yes, your hypothesis is correct.
> >>
> >>     Question: How can I make it consistent?
> >>
> >>     I have a script that runs on the first of the month that does
> archiving, and I have similar issues in that namespace:
> >>     ⌂67% [ler at thebighonker.lerctr.org:~ <mailto:
> ler at thebighonker.lerctr.org:~>] $ cat bin/archive-mail
> >>     #!/bin/sh
> >>     PATH=$PATH:/usr/local/bin
> >>     #Expects to be run after midnight on the first of the month
> >>     #  to archive all the previous months mail
> >>     #Date Run:
> >>     TODAY=`date "+%Y-%m-%d"`
> >>     #last month in YYYY/MM
> >>     YEAR_LAST_MONTH=`date -v-1d "+%Y/%m"`
> >>     #1st of last month as 01-Mon-YYYY
> >>     FIRST_LAST_MONTH=`date -v-1d "+01-%b-%Y"`
> >>     echo 'TODAY=' ${TODAY}
> >>     echo 'YEAR_LAST_MONTH=' ${YEAR_LAST_MONTH}
> >>     echo 'FIRST_LAST_MONTH=' ${FIRST_LAST_MONTH}
> >>     # get a list of all the mailboxes with at least one real message
> >>     doveadm -f tab mailbox status vsize \* 2>/dev/null |
> >>             sed -e 1d | sort -k 1,1 |
> >>             awk  'BEGIN {FS="\t"} {if ($2 > 0)  print $1}' |
> >>     while read i
> >>     do
> >>        echo `date` start ${i}
> >>        doveadm mailbox create "ARCHIVE/${YEAR_LAST_MONTH}/${i}"
> >>        doveadm -f tab mailbox status messages "${i}"
> >>        doveadm move "ARCHIVE/${YEAR_LAST_MONTH}/${i}" mailbox \
> >>                 "${i}" BEFORE ${TODAY} SINCE ${FIRST_LAST_MONTH}
> >>        doveadm -f tab mailbox status messages "${i}"
> >>        echo `date` done  ${i}
> >>     done
> >>     ⌂64% [ler at thebighonker.lerctr.org:~ <mailto:
> ler at thebighonker.lerctr.org:~>] $
> >>
> >>
> >>     The Exim config can be provided as well if necessary.
> >>
> >>     ler & ler at lerctr.org <mailto:ler at lerctr.org> *ARE THE SAME MAILBOX*
> >>
> >     At the moment it looks as though you have two sets of emails indexed
> in solr. One is indexed under username (the one you are running mannually
> and apparently the one used by roundcube too, but that's to be verified)
> and another set being indexed by autoindex = yes option using the full
> email address. Once you've got it working as you require, then you may want
> to clean out solr and reindex with just one of them just to reduce volumes.
> >
> >     Your setup seems to have a mix of users from mysql and from
> /etc/passwd. Not sure if your mysql users are all mapped to real users or
> they have their own mailboxes with domain included. Your solution will
> depend on what you really need and if the setup is working correctly you
> may not want to tweak it too much or other things may start breaking.
> >
> >     The simplest thing that comes to mind (providing your manually
> indexed mails show up in roundcube searches) is just to turn off autoindex
> and schedule indexing from cron. As things stand I believe autoindex works
> well with virtual users username at domain. You'll probably need to start
> off with a rescan if you do this.
> >
> >     If you can swith to all virtual users without local users then
> probably you can use autoindex, but I can understand that may not be
> possible. For sure I'd recommend doing it in a test environment first so
> you can be sure of the configuration and then look at migrating your
> existing mailboxes over to virtual users.
> >
> >     John
> >
> >
> >
> >
> >
> > --
> > Larry Rosenman                     http://www.lerctr.org/~ler
> > Phone: +1 214-642-9640 (c)     E-Mail: larryrtx at gmail.com <mailto:
> larryrtx at gmail.com>
> > US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
> >
> >
> > --
> > Larry Rosenman                     http://www.lerctr.org/~ler
> > Phone: +1 214-642-9640 (c)     E-Mail: larryrtx at gmail.com <mailto:
> larryrtx at gmail.com>
> > US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
>


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c)     E-Mail: larryrtx at gmail.com
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20190415/e738ec1d/attachment-0001.html>