Errors with missing links to files when using Single Instance Storage and zlib (Dovecot 2.3.4)

[Daniel Schütze]

I'm running Dovecot 2.3.4 with Single Instance Storage (SIS) and zlib 
and I am frequently seeing errors where files are missing in the 
attachments directory even though the zipped file in the hash directory 
is actually there.  This appears not to have been an issue in Dovecot 
2.3.1 Requested output of the server below.

The error looks like this in the log

Dec 18 08:58:58 dovecot01 dovecot: 
imap(username)<3692><RKnPGEh9fC3AqAAB>: Error: Mailbox INBOX/FOLDER: 
UID=41: 
read(attachments-connector(zlib(/usr/home/vmail/mail/username//mdbox/storage/m.411))) 
failed: 
read(/usr/home/vmail/attachments/91/08/91085773a0774909207852b8f9c98dbe22fe5a31-e88ba0276aae135cd2050000d09efc50[base64:19 
b/l]) failed: 
open(/usr/home/vmail/attachments/91/08/91085773a0774909207852b8f9c98dbe22fe5a31-e88ba0276aae135cd2050000d09efc50) 
failed: No such file or directory (FETCH BODY[])

I'm also seeing errors when replication fails

dovecot: dsync-local(username)<bzg3MqjhGFxmFwAAzRvpBw>: Error: 
dsync(servername): 
read(attachments-connector(zlib(/usr/home/vmail/mail/username//mdbox/storage/m.413))) 
failed: 
read(/usr/home/vmail/attachments/45/42/4542dcf385f6b4a4dc8752b96db1d7ad8a8023d5-ac735007f6bd135c93440100cd1be907[base64:19 
b/l]) failed: 
open(/usr/home/vmail/attachments/45/42/4542dcf385f6b4a4dc8752b96db1d7ad8a8023d5-ac735007f6bd135c93440100cd1be907) 
failed: No such file or directory (last sent=mail, last recv=mail (EOL))

It is not the case that every replication fail has a corresponding imap 
access fail.

I am running Dovecot 2.3.4 on FreeBSD11.2 and I did not see this error 
on Dovecot 2.3.1 FreeBSD 10.4.  Both installs had the same configuration 
for Dovecot.  Both installs were made from source code and not packages 
(as the need for the non standard fts and mysql options).

During my testing I had the 2.3.1 install receive mail and replicate to 
the 2.3.4 and that was working fine (except for tcp connection issues 
which I saw was fixed in the changelog).

Unfortunately having gone into testing with real account and mail being 
sent to the 2.3.4 install I'm seeing the above errors.

My clients are using Thunderbird on desktops and Roundcube for webmail.

My users report they can see the e-mails fine in their webmail 
(Roundcube) and only Thunderbird is giving issues.  I admit I don't know 
how this can be the case as both systems use the same account details.

In everycase so far there is a file in the hashes directory I have hand 
crafted the missing file (i.e. link) using ln and this restores access 
in Thunderbird.

Clearly this is an big issue for my install and I'd appreciate any help!

At the moment it looks like my course will be to try and script creation 
(recreation?) of the links from the error logs (I've seen people talking 
about this online) but that isn't really what I'm after.

DOVECOT CONFIG


# 2.3.4 (0ecbaf23d): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.4 (60b0f48d)
# OS: FreeBSD 11.2-RELEASE-p5 amd64
# Hostname: dovecot01.cwa.uk.com
dict {
   acl = mysql:/usr/local/etc/dovecot/dovecot-dict-sql.conf.ext
}
disable_plaintext_auth = no
doveadm_password = # hidden, use -P to show it
first_valid_uid = 145
last_valid_uid = 145
mail_access_groups = mail
mail_attachment_dir = /usr/home/vmail/attachments
mail_attachment_min_size = 64 k
mail_gid = vmail
mail_location = mdbox:~/mdbox:INDEX=/indexdisk/indexes/%n
mail_plugins = " fts fts_solr acl zlib notify replication"
mail_privileged_group = mail
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date index ihave duplicate mime foreverypart extracttext
namespace {
   list = yes
   location = mdbox:%%h/mdbox:INDEXPVT=~/mdbox/shared/%%n
   prefix = shared/%%n/
   separator = /
   subscriptions = yes
   type = shared
}
namespace inbox {
   inbox = yes
   location =
   mailbox Drafts {
     auto = subscribe
     special_use = \Drafts
   }
   mailbox Junk {
     auto = subscribe
     special_use = \Junk
   }
   mailbox Sent {
     auto = subscribe
     special_use = \Sent
   }
   mailbox "Sent Messages" {
     special_use = \Sent
   }
   mailbox Trash {
     auto = subscribe
     special_use = \Trash
   }
   prefix =
   type = private
}
passdb {
   args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
   driver = sql
}
plugin {
   acl = vfile
   acl_shared_dict = proxy::acl
   fts = solr
   fts_autoindex = yes
   fts_solr = url=http://localhost:8983/solr/dovecot/
   mail_replica = tcp:192.168.0.138:12345
   replication_dsync_parameters = -d -l 30 -U
   sieve = ~/.dovecot.sieve
   sieve_dir = ~/sieve
   sieve_duplicate_default_period = 1h
   sieve_duplicate_max_period = 1d
   sieve_global_dir = /usr/home/vmail/sieve/global/
   sieve_global_path = /usr/home/vmail/default.sieve
   sieve_vacation_default_period = 1d
   zlib_save = gz
   zlib_save_level = 6
}
protocols = imap sieve
service aggregator {
   fifo_listener replication-notify-fifo {
     user = vmail
   }
   unix_listener replication-notify {
     user = vmail
   }
}
service auth {
   unix_listener auth-userdb {
     mode = 0777
   }
}
service dict {
   unix_listener dict {
     mode = 0600
     user = vmail
   }
}
service doveadm {
   inet_listener {
     port = 12345
   }
}
service replicator {
   process_min_avail = 1
   unix_listener replicator-doveadm {
     mode = 0600
     user = vmail
   }
}
ssl = no
userdb {
   driver = prefetch
}
userdb {
   args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
   driver = sql
}
protocol lda {
   mail_plugins = " fts fts_solr acl zlib notify replication sieve"
   submission_host = 192.168.0.1:25
}
protocol imap {
   mail_plugins = " fts fts_solr acl zlib notify replication imap_acl 
imap_zlib"
}