Looks like a bug to me: Dovecot ignores Maildir/new timestamp

[Fredrik Roubert]

Hello!

I run Debian jessie and have an issue with the interaction between
Dovecot (2.2.13), Mutt (1.5.23) and the Android Gmail (5.8.105868218)
IMAP client, when using Maildir:

I use Dovecot LMTP for local delivery into Maildir and access these
files both through Dovecot IMAP from the Android Gmail IMAP client and
also directly through the filesystem by running Mutt locally on the mail
server.

When a new mail arrives, Dovecot LMTP will add a new file in Maildir/new
and also update the Dovecot index cache. If I then open Mutt and just
delete this new message, then Mutt will just delete this file from
Maildir/new and the Dovecot index cache will be out-of-sync.

(For testing, Mutt can be cut out of the equation entirely by instead
just running "rm Maildir/new/*" from the command line to mimic what Mutt
does when deleting a new message, but I mention the use of Mutt here to
show why this is a real use-case for me.)

If I after this open the Gmail app and pull down the message index to
refresh from the Dovecot IMAP server, it'll get the cached version of
the index, in which the new message is still present (but only the
headers, no body will be loaded and the app will show no body snippet).
Pulling down the message index again to force a second refresh will
cause the new/deleted message to disappear again.

This seems to be caused by Dovecot not looking at the Maildir/new
timestamp to determine whether the index cache is up-to-date, but
instead only looking at the Maildir/cur timestamp.

If I instead repeat all these testing steps from the beginning, sending
a new mail message, deleting it with "rm Maildir/new/*" but then also
running "touch Maildir/cur" before opening the Gmail app and doing the
refresh, then Dovecot does update the cache and does not send any ghost
messages to the IMAP client.

It does seem reasonable to me that Mutt doesn't touch Maildir/cur when
just deleting new messages and I suspect that it might be a bug in
Dovecot that it timestamp of Maildir/new isn't checked and the index
cache updated if that timestamp is newer than the last index update done
by the Dovecot LMTP service.

Does this analysis seem sound? Did I miss anything?

(Output of "doveconf -n" attached.)

Cheers // Fredrik Roubert

-- 
Forsterstrasse 64  |  +41 78 8170377
CH-8044 Zürich     |  https://roubert.name/fredrik/
-------------- next part --------------
# 2.2.13: /etc/dovecot/dovecot.conf
# OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.2 
auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@/
auth_username_format = %Ln
first_valid_uid = 1000
last_valid_uid = 65533
listen = 5.148.172.24, 2a02:418:1003:7::1
mail_location = maildir:~/Maildir
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave editheader vnd.dovecot.duplicate
mbox_read_locks = flock
mbox_write_locks = flock
namespace inbox {
  inbox = yes
  location = 
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix = 
}
passdb {
  args = scheme=SHA512-CRYPT /etc/dovecot/passwd-imap
  driver = passwd-file
}
passdb {
  args = scheme=SHA512-CRYPT /etc/dovecot/passwd-smtp
  driver = passwd-file
}
plugin {
  sieve = ~/.dovecot.sieve
  sieve_before = /var/lib/dovecot/sieve/spam.sieve
  sieve_dir = ~/.sieve
  sieve_extensions = +editheader +vnd.dovecot.duplicate
}
protocols = " imap lmtp sieve"
service auth {
  unix_listener /var/spool/postfix/private/dovecot-auth {
    group = postfix
    mode = 0666
    user = postfix
  }
}
service imap-login {
  inet_listener imap {
    port = 143
  }
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0666
    user = postfix
  }
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
}
ssl = required
ssl_cert = </etc/ssl/startssl/certs/roubert.net.crt
ssl_cipher_list = EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:+CAMELLIA256:+AES256:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!ECDSA:CAMELLIA256-SHA:AES256-SHA:CAMELLIA128-SHA:AES128-SHA
ssl_dh_parameters_length = 2048
ssl_key = </etc/ssl/startssl/private/roubert.net.key
ssl_prefer_server_ciphers = yes
ssl_protocols = !SSLv3 !SSLv2
userdb {
  args = home=/home/%n allow_all_users=yes
  driver = static
  result_failure = return-fail
  result_internalfail = return-fail
  result_success = continue
}
userdb {
  args = blocking=no
  driver = passwd
}
protocol lmtp {
  mail_plugins = " sieve"
  postmaster_address = postmaster at roubert.net
}
protocol imap {
  passdb {
    args = scheme=SHA512-CRYPT /etc/dovecot/passwd-smtp
    deny = yes
    driver = passwd-file
    name = 
  }
}