Mail Archiving/Lucene Indexes/mbox
Larry Rosenman
larryrtx at gmail.com
Wed Aug 20 12:25:45 UTC 2014
Any new ideas on this?
On Fri, Aug 1, 2014 at 9:22 AM, Larry Rosenman <larryrtx at gmail.com> wrote:
> Yes, I do.
>
> thebighonker.lerctr.org /home/ler $ doveconf -n
> # 2.2.13: /usr/local/etc/dovecot/dovecot.conf
> # OS: FreeBSD 10.0-STABLE amd64
> auth_default_realm = lerctr.org
> auth_mechanisms = plain login
> auth_realms = lerctr.org thebighonker.lerctr.org tbh.lerctr.org
> auth_username_format = %Ln
> lda_mailbox_autocreate = yes
> lmtp_save_to_detail_mailbox = yes
> login_access_sockets = tcpwrap
> mail_debug = yes
> mail_location = mbox:~/mail:INBOX=~/mail/INBOX
> mail_plugins = fts fts_lucene stats
> mail_privileged_group = mail
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope encoded-character
> vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
> copy include variables body enotify environment mailbox date ihave
> duplicate vacation-seconds editheader
> namespace archive {
> hidden = no
> inbox = no
> list = no
> location = mbox:~/MAILARCHIVE
> prefix = "#ARCHIVE/"
> separator = /
> }
> namespace inbox {
> inbox = yes
> location =
> mailbox Drafts {
> special_use = \Drafts
> }
> mailbox INBOX {
> auto = create
> }
> mailbox Junk {
> special_use = \Junk
> }
> mailbox Sent {
> special_use = \Sent
> }
> mailbox "Sent Messages" {
> special_use = \Sent
> }
> mailbox Trash {
> special_use = \Trash
> }
> prefix =
> }
> passdb {
> args = failure_show_msg=yes session=yes max_requests=20
> driver = pam
> }
> plugin {
> fts = lucene
> fts_autoindex = yes
> fts_lucene = whitespace_chars=@. normalize no_snowball
> recipient_delimiter = +
> sieve = ~/.dovecot.sieve
> sieve_dir = ~/sieve
> sieve_extensions = +editheader +vacation-seconds
> stats_command_min_time = 1 mins
> stats_domain_min_time = 12 hours
> stats_ip_min_time = 12 hours
> stats_memory_limit = 16 M
> stats_refresh = 5s
> stats_session_min_time = 15 mins
> stats_track_cmds = yes
> stats_user_min_time = 1 hours
> }
> protocols = imap pop3 lmtp sieve
> service auth {
> unix_listener auth-client {
> mode = 0666
> }
> unix_listener auth-master {
> mode = 0666
> }
> }
> service stats {
> chroot = empty
> client_limit = 0
> drop_priv_before_exec = no
> executable = stats
> extra_groups =
> fifo_listener stats-mail {
> group =
> mode = 0666
> user =
> }
> group =
> idle_kill = 4294967295 secs
> privileged_group =
> process_limit = 1
> process_min_avail = 0
> protocol =
> service_count = 0
> type =
> unix_listener stats {
> group =
> mode = 0666
> user =
> }
> user = $default_internal_user
> vsz_limit = 18446744073709551615 B
> }
> service tcpwrap {
> unix_listener login/tcpwrap {
> group = $default_login_user
> mode = 0600
> user = $default_login_user
> }
> }
> ssl_cert = </etc/ssl/certs/CHAIN.crt
> ssl_key = </etc/ssl/certs/thebighonker.lerctr.org.key.open
> userdb {
> driver = passwd
> }
> verbose_proctitle = yes
> protocol lmtp {
> mail_plugins = fts fts_lucene sieve
> }
> protocol lda {
> mail_plugins = fts fts_lucene sieve
> }
> protocol imap {
> imap_client_workarounds = tb-extra-mailbox-sep tb-lsub-flags
> mail_max_userip_connections = 50
> mail_plugins = fts fts_lucene stats imap_stats
> }
> thebighonker.lerctr.org /home/ler $
>
>
> The following was put out while the script was running:
> Aug 1 00:05:00 thebighonker dovecot: indexer-worker(ler): Error: lucene:
> Failed to sync mailbox INBOX: Mailbox isn't selectable
> Aug 1 00:05:31 thebighonker last message repeated 24 times
> Aug 1 00:05:45 thebighonker last message repeated 95 times
> Aug 1 00:05:45 thebighonker exim[69950]: 1XD51Y-000ICE-2D <=
> ler at lerctr.org U=ler P=local S=11222
> Aug 1 00:05:45 thebighonker dovecot: lda(ler): sieve: msgid=<
> E1XD51Y-000ICE-2D at thebighonker.lerctr.org>: stored mail into mailbox
> 'INBOX'
> Aug 1 00:05:45 thebighonker exim[70321]: 1XD51Y-000ICE-2D => ler <
> ler at lerctr.org> R=localuser T=local_delivery_dovecot S=11327 QT=45s DT=0s
> Aug 1 00:05:45 thebighonker exim[70321]: 1XD51Y-000ICE-2D Completed QT=45s
> Aug 1 00:05:46 thebighonker dovecot: indexer-worker(ler): Indexed 1
> messages in INBOX
>
> the script:
> thebighonker.lerctr.org /home/ler $ cat bin/archive-mail
> #!/bin/sh
> PATH=$PATH:/usr/local/bin
> #Expects to be run after midnight on the first of the month
> # to archive all the previous months mail
> #Date Run:
> TODAY=`date "+%Y-%m-%d"`
> #last month in YYYY/MM
> YEAR_LAST_MONTH=`date -v-1d "+%Y/%m"`
> #1st of last month as 01-Mon-YYYY
> FIRST_LAST_MONTH=`date -v-1d "+01-%b-%Y"`
> echo 'TODAY=' ${TODAY}
> echo 'YEAR_LAST_MONTH=' ${YEAR_LAST_MONTH}
> echo 'FIRST_LAST_MONTH=' ${FIRST_LAST_MONTH}
> # get a list of all the mailboxes with at least one real message
> doveadm -f table mailbox status vsize \* 2>/dev/null | \
> awk '{if ($2 > 0) print $1}' >/tmp/ler-mailbox-list.$$ 2>/dev/null
> # archive all the mails
> for i in `cat /tmp/ler-mailbox-list.$$`
> do
> echo `date` start ${i}
> doveadm mailbox create \#ARCHIVE/${YEAR_LAST_MONTH}/${i}
> doveadm -f tab mailbox status messages ${i}
> doveadm move \#ARCHIVE/${YEAR_LAST_MONTH}/${i} mailbox \
> ${i} BEFORE ${TODAY} SINCE ${FIRST_LAST_MONTH}
> doveadm -f tab mailbox status messages ${i}
> echo `date` done ${i}
> done
> rm /tmp/ler-mailbox-list.$$
> thebighonker.lerctr.org /home/ler $
>
>
> I am running the hg patch that fixes the filenames to be in the .imap
> space, but that is the only patch I have applied on top of 2.2.13.
>
>
>
>
>
> On Fri, Aug 1, 2014 at 9:09 AM, Timo Sirainen <tss at iki.fi> wrote:
>
>> On 01 Aug 2014, at 08:40, Larry Rosenman <larryrtx at gmail.com> wrote:
>>
>> > I have a script (available on demand) that moves all my mail for a given
>> > month to a separate NAMESPACE (#ARCHIVE).
>> >
>> > When I do that, the lucene-indexes folder in the PRIMARY namespace
>> seems to
>> > get cleaned out, and NOTHING gets put in the lucene-indexes folder
>> > (although it gets created) in the #ARCHIVE namespace.
>> >
>> > Is this to spec?
>> >
>> > the archiving is done via doveadm move commands.
>>
>> Do you have fts_autoindex=yes? If not, the Lucene index update is delayed
>> until the first search is issued.
>>
>>
>
>
> --
> Larry Rosenman http://www.lerctr.org/~ler
> Phone: +1 214-642-9640 (c) E-Mail: larryrtx at gmail.com
> US Mail: 108 Turvey Cove, Hutto, TX 78634-5688
>
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: larryrtx at gmail.com
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688
More information about the dovecot
mailing list