[Dovecot] got too little data ??

[Benoit Branciard]

I just switched in a hurry one of our mailbox servers from UW-imapd to
Dovecot in an attempt to address performance and stability problems.

This server hosts ~5500 accounts, ~2000 are daily used, ~600
simultaneous IMAP connexions in rush hours, and lots of POP ones. Some
accounts may be accessed simultaneously with IMAP and POP.

The system is :
- Debian Sarge (x86 with amd64 kernel)
- MBOX files mailboxes
- indexes migrated out of filesystem quotas
- Sendmail MTA
- Procmail local delivery agent (not easily replaceable due to extensive
use of procmailrc filters)
- Both Procmail and Dovecot use [dotlock, fcntl] as mailbox locking scheme
- No other processes have access to mailboxes in normal use
- Dovecot 1.0.8 hand-compiled from tarball

The good new is that it seems to perform better than UW-imapd.

The bad new is that we get lots or errors like this in log:

	IMAP(username): FETCH for mailbox INBOX UID 23862 got too little data:
3186 vs 3206

Such "accidents" happen about 4-5 times a day, and does not correct
themselves, even with a mailbox expunge or reopen. However manual
deletion of the faulty mailbox index seems to clear the problem, until
another one occurs. On such errors some mail clients (Thunderbird) retry
operation in infinite loop, so log gets really cluttered.

In imap-fetch-body.c it is stated : "We shouldn't really ever get here.
One reason is if mail was deleted from NFS server while we were reading
it. Another is some temporary disk error."

Both reasons do not seem to apply in our case: mail partition is not
NFS-mounted, and disks (a RAID5 array) seem healthy. Furthermore,
affected accounts are not over quota.

So may we have found a bug ? Or may we have a locking problem ? Or an
index handling one ?

I tried to disable mbox_lazy_writes without success. I will try to
disable disk indexes totally, which should help since problems seem
index-related; but performance may suffer.

Any clues appreciated...

Dovecot -n says :

# 1.0.8: /usr/local/etc/dovecot.conf
protocols: imap imaps pop3 pop3s
listen: [::]
ssl_cert_file: /usr/local/etc/ssl/private/our-certificate.pem
ssl_key_file: /usr/local/etc/ssl/private/our-certificate.pem
disable_plaintext_auth: no
login_dir: /var/run/dovecot/login
login_executable(default): /usr/local/libexec/dovecot/imap-login
login_executable(imap): /usr/local/libexec/dovecot/imap-login
login_executable(pop3): /usr/local/libexec/dovecot/pop3-login
first_valid_uid: 200
first_valid_gid: 200
mail_location:
mbox:~/mail:INBOX=~/mail/INBOX:INDEX=/espace/dovecot/indexes/%u
fsync_disable: yes
mbox_lazy_writes: no
mail_executable(default): /usr/local/libexec/dovecot/imap
mail_executable(imap): /usr/local/libexec/dovecot/imap
mail_executable(pop3): /usr/local/libexec/dovecot/pop3
mail_plugins(default): quota imap_quota
mail_plugins(imap): quota imap_quota
mail_plugins(pop3): quota
mail_plugin_dir(default): /usr/local/lib/dovecot/imap
mail_plugin_dir(imap): /usr/local/lib/dovecot/imap
mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3
imap_client_workarounds(default): outlook-idle tb-extra-mailbox-sep
delay-newmail
imap_client_workarounds(imap): outlook-idle tb-extra-mailbox-sep
delay-newmail
imap_client_workarounds(pop3): outlook-idle
pop3_uidl_format(default):
pop3_uidl_format(imap):
pop3_uidl_format(pop3): %08Xv%08Xu
pop3_client_workarounds(default):
pop3_client_workarounds(imap):
pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh
auth default:
  passdb:
    driver: pam
    args: *
  userdb:
    driver: passwd
plugin:
  quota: fs

-- 
Ce message a ete verifie par MailScanner
pour des virus ou des polluriels et rien de
suspect n'a ete trouve.