How to Analyze Missing UIDs (and Files) in Maildirs?

[Jochen Bern]

Hello everyone,

I would like pointers on how to analyze the following situation, please:

I'm running one test and one production dovecot IMAPS server for one of
our platforms. The clients are essentially appliances we distribute,
auth by client cert, virtual users only, mailboxes in maildir format:

> auth_ssl_require_client_cert = yes
> auth_ssl_username_from_cert = yes
> auth_mechanisms = plain

> ssl = required
> ssl_cert = </etc/pki/dovecot/certs/...
> ssl_key = </etc/pki/dovecot/private/...
> ssl_ca = </etc/pki/dovecot/certs/...
> ssl_verify_client_cert = yes

> mail_location = maildir:~
> userdb {
>   driver = static
>   args = uid=... gid=... home=/home/.../%Ld_realm/%Ln
> }
> passdb {
>   driver = static
>   args = password=...
> }

The client certs have CNs unique to the appliance and no client besides
that appliance is supposed to access the mailbox. Appliances take note
of the UIDs they've seen (and not yet deleted) and expect to be able to
re-access e-mails by the UIDs they know. (Needless to say, I keep my
hands away from the cache files so as not to cause the UIDs to be
reassigned.)

The dovecot version used is updated along with the CentOS 6 source RPMs
- which contain a 2.0.9 plus whatever patches CentOS backported - and
receives two additional patches from me, one from current dovecot
sources to improve closure of inactive SSL connections and one from me
that turns the IMAP CREATE command into a NOP; the appliance code
doesn't expect mailboxes to have folders besides INBOX.

-------

QA is looking into an upcoming new feature that requires appliances to
reread old e-mails, and got errors that UIDs the test appliances
expected to still be on the (test) server weren't. I verified, the UIDs
are indeed absent from the dovecot-uidlist files. The mails in question
arrived *sometime* over the course of the last 12 months and must've
vanished *sometime* since then ... ugh. No trace of what may have
happened to them in any logs, of course.

On the test server, I re-enabled logging of DELETE, UNDELETE and EXPUNGE
commands, in case the appliances delete mails themselves and forget to
update their own cache. Are there other IMAP commands that could remove
mails, and thus should be added to the list?

Would it be possible to have dovecot configured so that there's a log
entry whenever a client tries to retrieve a UID that dovecot doesn't
have in the maildir anymore?

-------

While looking into the above issue, I noticed that a large number of
mailboxes (test as well as production) have UIDs that *are* listed in
dovecot-uidlist, yet, no corresponding file is present anymore. Example:

> # grep -c : *_realm/$EXAMPLE_USER/dovecot-uidlist
> 58
> # grep : *_realm/$EXAMPLE_USER/dovecot-uidlist | sed -e '2,57d' -e 's/ .*//'
> 1
> 152
> # for PAIR in `grep : *_realm/$EXAMPLE_USER/dovecot-uidlist | sed -e 's/ .*:/:/'` ; do
> > IMAP_UID=`echo $PAIR | sed -e 's/:.*//'`; BASE=`echo $PAIR | sed -e 's/.*://'`
> > CNT=`find *_realm/$EXAMPLE_USER -type f -name "$BASE"'*' -ls | wc -l`
> > if [ $CNT -ne 1 ]; then echo "Found $CNT files for UID $IMAP_UID"; fi; done
> Found 0 files for UID 135
> Found 0 files for UID 136
> Found 0 files for UID 139
> Found 0 files for UID 142
> Found 0 files for UID 143
> Found 0 files for UID 144
> Found 0 files for UID 145
> Found 0 files for UID 150
> Found 0 files for UID 151

Is that normal behaviour? If not, how would I try to find out what
happens there?

Kind regards,
-- 
Jochen Bern
Systemingenieur

Fon:    +49 6151 9067-231
Fax:    +49 6151 9067-290
E-Mail: jochen.bern at binect.de

www.binect.de
www.facebook.de/binect

Binect GmbH

Robert-Koch-Straße 9, 64331 Weiterstadt, DE

Geschäftsführung: Christian Ladner, Dr. Frank Wermeyer, Nils Manegold
Unternehmenssitz: Weiterstadt
Register:         Amtsgericht Darmstadt, HRB 94685
Umsatzsteuer-ID:  DE 221 302 264

MAX 21-Unternehmensgruppe


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4278 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://dovecot.org/pipermail/dovecot/attachments/20161111/46b0503d/attachment.p7s>