[Dovecot] IMAP IDLE goes to sleep

[Julian Cowley]

First of all, let me say that I'm really impressed with Dovecot.
It is great.

Problem description
===================

Now I've got a possible bug I'd like to report.  I've been learning
about the IMAP protocol recently, and have started playing with the
IDLE command.  The problem is that Dovecot doesn't always notify the
client when there is new mail available.  While notifications work
for the most part, it seems that Dovecot occasionally gets itself
into a rut where it fails to notice any changes to the mailbox (new
messages, flag changes, etc.).  The problem usually becomes apparent
when there is more than one email client accessing the same mailbox.
In fact, this may be a requirement -- I only tested it once, but I
couldn't reproduce the problem when I had only one connection open
to the mailbox (which is in maildir format in my case).

An example is included below.  Any feedback on how to debug this
problem further would be appreciated.

An aside
========

After I found out about the IDLE command, I looked for implementations.
I ran across this plug-in for Apple Mail (one of my favorite mail
clients):

http://www.rothwell.us/imapidle/index.html

The web page notes that Leopard's Apple Mail may support the IDLE
command natively, which is good news if true.  It would be really
cool if Dovecot's IDLE command can be made robust enough to make this
feature really come alive.

Example
=======

This transcript is copied from an SSL connection directly to dovecot
using openssl s_client.  At the same time a separate client (Horde/IMP)
has the same mailbox open:

A0017 IDLE
+ idling
* 7927 FETCH (FLAGS (\Seen \Recent))
* 7927 FETCH (FLAGS (\Deleted \Seen \Recent))
* OK Still here
* OK Still here
* OK Still here
* OK Still here
* 7940 EXISTS
* 22 RECENT
* OK Still here
* OK Still here
* OK Still here
* OK Still here
* OK Still here
* OK Still here
* OK Still here
* OK Still here
* OK Still here
* OK Still here
* 7941 EXISTS

[At some point after this, something happens (or doesn't happen) and
dovecot no longer notifies of new messages.  The other client is still
reading messages in the same mailbox and new mail is still arriving.]

* OK Still here
* OK Still here
* OK Still here
* OK Still here
* OK Still here

[The above message repeated another 30 times here.  Finally, IDLE
mode is terminated:]

* OK Still here
DONE
* 7941 EXPUNGE
A0017 OK Idle completed.

[Now a NOOP is issued to gather any more untagged responses, and
there's a deluge of responses, presumably all the stuff that happened
while dovecot was asleep.  Notice that the highest message is 7967
whereas the last one heard about was 7941.]

A0018 NOOP
* 7913 FETCH (FLAGS (\Deleted \Recent))
* 7914 FETCH (FLAGS (\Deleted))
* 7915 FETCH (FLAGS (\Deleted))
* 7916 FETCH (FLAGS (\Deleted))
* 7917 FETCH (FLAGS (\Deleted))
* 7918 FETCH (FLAGS (\Deleted))
* 7919 FETCH (FLAGS (\Deleted \Recent))
* 7920 FETCH (FLAGS (\Deleted \Recent))
* 7921 FETCH (FLAGS (\Deleted \Recent))
* 7922 FETCH (FLAGS (\Deleted \Seen))
* 7923 FETCH (FLAGS (\Deleted \Seen))
* 7925 FETCH (FLAGS (\Deleted \Seen))
* 7928 FETCH (FLAGS (\Deleted \Recent))
* 7967 EXISTS
* 30 RECENT
A0018 OK NOOP completed.

Configuration
=============

Here are the specifications of the dovecot used in the above test:

version         dovecot 1.0.1
os              linux
distro          centos 4.5
fs              ext3
arch            x86_64
kernel          2.6.9-55.ELsmp
configure       --without inotify --with pam_stack --with forcequota2
cc              gcc (GCC) 3.4.6 20060404 (Red Hat 3.4.6-8)

In addition, I'm using the deliver LDA as the Postfix mailbox_command,
and am using the dovecot-sieve plugin (which is used to set customized
IMAP keywords).

dovecot -n:

# 1.0.1: /etc/dovecot.conf
ssl_cert_file: /usr/lib/courier-imap/share/imapd.pem
ssl_key_file: /usr/lib/courier-imap/share/imapd.pem
login_dir: /var/run/dovecot/login
login_executable(default): /usr/libexec/dovecot/imap-login
login_executable(imap): /usr/libexec/dovecot/imap-login
login_executable(pop3): /usr/libexec/dovecot/pop3-login
first_valid_uid: 100
mail_location: maildir:~/Maildir
maildir_copy_with_hardlinks: yes
mail_executable(default): /usr/libexec/dovecot/imap
mail_executable(imap): /usr/libexec/dovecot/imap
mail_executable(pop3): /usr/libexec/dovecot/pop3
mail_plugin_dir(default): /usr/lib64/dovecot/imap
mail_plugin_dir(imap): /usr/lib64/dovecot/imap
mail_plugin_dir(pop3): /usr/lib64/dovecot/pop3
namespace:
   type: private
   separator: /
   inbox: yes
auth default:
   default_realm: lava.net
   verbose: yes
   passdb:
     driver: sql
     args: /etc/dovecot-sql.conf
   userdb:
     driver: sql
     args: /etc/dovecot-sql.conf

ps. Good thing there's no POP IDLE command!