Solr FTS - message deletes not working as expected

[Shawn Heisey]

I have Solr FTS enabled in dovecot.  It is behaving differently than I 
would expect.

When I delete messages in Thunderbird normally, using IMAP to talk to 
dovecot, the deletion hits Solr right away.  If I then ask Thunderbird 
to empty the trash, that delete also hits Solr immediately.

But if I use shift-delete in Thunderbird, which deletes the message 
immediately without going through Trash, things are different.  One 
delete that I did (6 messages) took at least a minute before Solr saw 
the request.  And on other tests that I ran, deleting one message, I 
never did see the delete request hit Solr.  I waited for several minutes.

I use the shift-delete mechanism a lot more than the method that sends 
to trash.  That way I do not need to remember to empty the trash.

Dovecot version is 2:2.3.16-2+ubuntu20.04 installed from the dovecot 
repo, not from distro packages.  I don't think the Solr version matters, 
but just in case I am wrong, that is 8.10.1.  I don't recall exactly 
what I changed in the solrconfig.xml and managed-schema files, but it 
should not be anything that affects how Solr FTS works.  The problem 
seems to be that Solr FTS isn't sending deletes, not that Solr is 
behaving incorrectly.

I just discovered that trying to get the dovecot config failed:

root at bilbo:~# doveconf -n
# 2.3.16 (7e2e900c1a): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.16 (09c29328)
doveconf: Fatal: Error in configuration file 
/etc/dovecot/conf.d/10-master.conf line 64: Invalid size: $default_vsz_limit

After fixing that, this is what I get for the config:

root at bilbo:/etc/dovecot# doveconf -n
# 2.3.16 (7e2e900c1a): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.16 (09c29328)
# OS: Linux 5.11.0-1020-aws x86_64 Ubuntu 20.04.3 LTS ext4
# Hostname: REDACTED
default_vsz_limit = 1 G
dict {
   quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext
}
first_valid_uid = 150
last_valid_uid = 150
listen = *
mail_gid = mail
mail_location = maildir:/var/vmail/%d/%u
mail_plugins = " quota fts fts_solr"
mail_privileged_group = mail
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date index ihave duplicate mime foreverypart extracttext
namespace inbox {
   inbox = yes
   location =
   mailbox Drafts {
     special_use = \Drafts
   }
   mailbox Junk {
     special_use = \Junk
   }
   mailbox Sent {
     special_use = \Sent
   }
   mailbox "Sent Messages" {
     special_use = \Sent
   }
   mailbox Trash {
     special_use = \Trash
   }
   prefix =
}
passdb {
   args = /etc/dovecot/dovecot-sql.conf.ext
   driver = sql
}
plugin {
   fts = solr
   fts_autoindex = yes
   fts_solr = url=http://localhost:8983/solr/dovecot/
   quota = dict:User quota::proxy::quota
   quota_grace = 10%%
   quota_max_mail_size = 100M
   quota_status_nouser = DUNNO
   quota_status_overquota = 552 5.2.2 Mailbox is full
   quota_status_success = DUNNO
   sieve = 
file:/var/vmail/sieve/%d/%u/sieve;active=/var/vmail/sieve/%d/%u/.dovecot.sieve
   sieve_default = file:/var/vmail/global.sieve
}
postmaster_address = postmaster at REDACTED
protocols = " imap lmtp sieve pop3"
service auth {
   unix_listener /var/spool/postfix/private/auth {
     group = postfix
     mode = 0666
     user = postfix
   }
   unix_listener auth-userdb {
     group = mail
     mode = 0666
     user = vmail
   }
}
service dict {
   unix_listener dict {
     group = postfix
     mode = 0660
     user = vmail
   }
}
service imap {
   vsz_limit = 1 G
}
service stats {
   unix_listener stats-reader {
     group = vmail
     mode = 0660
     user = vmail
   }
   unix_listener stats-writer {
     group = vmail
     mode = 0660
     user = vmail
   }
}
ssl_cert = </etc/ssl/certs/local/mainwildcards.pem
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
userdb {
   args = /etc/dovecot/dovecot-sql.conf.ext
   driver = sql
}
protocol lmtp {
   mail_plugins = " quota fts fts_solr sieve"
}
protocol lda {
   mail_plugins = " quota fts fts_solr sieve"
}
protocol imap {
   mail_max_userip_connections = 20
   mail_plugins = " quota fts fts_solr imap_quota"
}

I repeated the delete test after fixing the config and there was no 
change -- the delete request never made it to Solr.

Reindexing completely for my dovecot install takes about 8 minutes.  I 
have less than 150K messages.  Reindexing for some could take hours or 
days.  This is what I use to initiate a full reindex:

root at bilbo:~# cat /usr/local/sbin/reindexsolr
#!/bin/sh
DOVECOT_SOLR_URL_BASE="http://localhost:8983/solr/dovecot"
DEL_ALL_QUERY_XML="<delete><query>*:*</query></delete>"
MAIL_ROOT=/var/vmail

service dovecot stop
curl \
   "${DOVECOT_SOLR_URL_BASE}/update?commit=true&optimize=true" \
   -H "Content-Type: text/xml" \
   --data-binary "${DEL_ALL_QUERY_XML}"
find ${MAIL_ROOT} -name "dovecot.index*" -print0 | xargs -0 rm -f
service dovecot start
doveadm index -A -q '*'