Aki,

Please see below.

On Mon, Mar 11, 2019, at 12:58 PM, Aki Tuomi via dovecot wrote:

On 10.3.2019 10.14, Kostya Vasilyev via dovecot wrote:

...

Hello,

I'm experiencing slow flags changes over IMAP IDLE:

• If I start an IDLE session / command
• Change the flags of some messages via another email app
• Dovecot can take a minute or more to notify the IDLE connection about flags changes

If I use another email app to add or remove a message, Dovecot sends that (EXISTS / EXPUNGE) instantly and also flushes the (so far delayed) flags changes. So it does "know" already.

The system is Debian Testing, 64 bit, file system is ext4.

Dovecot is 2.3.4.1 (f79e8e7e4) - I believe from Debian repos (I had Dovecot's own repo added at some point, disabled now).

My mail is stored under ~/mail/.imap (not sure what this format is called), I mean not "single file mbox".

I have not changed any IDLE related config settings:

doveconf | grep -i idle default_idle_kill = 1 mins director_ping_idle_timeout = 30 secs imap_idle_notify_interval = 2 mins imapc_max_idle_time = 29 mins mailbox_idle_check_interval = 30 secs

What can I do to make Dovecot notify IDLE clients about flags changes - more quickly? Preferably near-instant?

Can you send doveconf -n?

doveconf -n # 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.4 () # OS: Linux 4.18.16-x86_64-linode118 x86_64 Debian buster/sid # Hostname: kman.mobi auth_default_realm = kman.mobi auth_mechanisms = plain login login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c %k mail_location = mbox:~/mail:INBOX=/var/mail/%n mail_privileged_group = mail namespace inbox { inbox = yes location = mailbox Drafts { auto = create special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { auto = create special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = create special_use = \Trash } prefix = } passdb { args = scheme=CRYPT username_format=%u /etc/dovecot/users driver = passwd-file } protocols = " imap" service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } } service imap-login { inet_listener imap { port = 0 } } service pop3-login { inet_listener pop3 { port = 0 } } ssl_cert = </etc/letsencrypt/live/kman.mobi/fullchain.pem ssl_cipher_list = kECDHE+CHACHA20:kECDHE+AESGCM:kECDHE+AES:!AESCCM:!aNULL ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it ssl_min_protocol = TLSv1.2 ssl_prefer_server_ciphers = yes userdb { args = username_format=%u /etc/dovecot/users driver = passwd-file } protocol imap { mail_max_userip_connections = 20 }

-- K

Timo,

On Tue, Mar 12, 2019, at 3:56 AM, Timo Sirainen via dovecot wrote:

On 10 Mar 2019, at 10.14, Kostya Vasilyev via dovecot <dovecot@dovecot.org> wrote:

...

My mail is stored under ~/mail/.imap (not sure what this format is called), I mean not "single file mbox".

I have not changed any IDLE related config settings:

doveconf | grep -i idle default_idle_kill = 1 mins director_ping_idle_timeout = 30 secs imap_idle_notify_interval = 2 mins imapc_max_idle_time = 29 mins mailbox_idle_check_interval = 30 secs

What can I do to make Dovecot notify IDLE clients about flags changes - more quickly? Preferably near-instant?

It should simply just work, assuming there aren't any weird inotify limits, but you should get errors logged about reaching those. You could see if it makes any difference to set mailbox_idle_check_interval=1s

mailbox_idle_check_interval = 1 secs

did not help

Here is an interesting case

• Let's say I have my IDLE connection.

• Connection "A" - an email app where I set or clear \Flagged on two messages.

The IDLE connection is silent (no unsolicited notifications about these flags).

• Connection "B" - an email app where I do a refresh - it does a SELECT (CONDSTORE) followed by FETCH UID FLAGS CHANGEDSINCE (because it sees that there are only flags changes).

---> the IDLE connection is notified about flags changes immediately right as Connection "B" is pulling its changes.

I tried a direct network connection (netcat) instead of Connection "B" and the actual trigger is SELECT (CONDSTORE).

• Start up IDLE connection

• Use Connection "A" to make flags changes

( IDLE connection is silent )

• Use a netcat connection to SELECT (CONDSTORE)

Dovecot flushes flags to the IDLE connection immediately

• Doing SELECT (without CONDSTORE) does not

Looks like some sort of bug in Dovecot related to CONDSTORE?

This can probably be reproduced with several direct network connections using netcat / openssl s_client - and CONDSTORE seems to be an important part of the scenario.

Ideas?

-- K

On 12 Mar 2019, at 10.21, Kostya Vasilyev via dovecot <dovecot@dovecot.org> wrote:

It makes no difference if the IDLE connection does SELECT or SELECT (CONDSTORE) prior to going IDLE.

But then as far as I know (?) - in Dovecot, once any connection uses CONDSTORE ever, even once, Dovecot creates data structures to track MODSEQ values, and those data structures are forever.

So are you saying that you can reproduce if you do for a completely new user:

doveadm exec imap -u testuser1 a select inbox b idle

And then run: echo foo | doveadm save -u testuser1 doveadm flags add -u testuser1 '\Seen' mailbox inbox 1

And the EXISTS shows up immediately after saving, but the flag change won't show up? It works fine with me.

Do you see any errors in "doveadm log errors"? Can you reproduce this if you try with some other mailbox format than mbox?

Timo,

On Wed, Mar 13, 2019, at 9:36 AM, Kostya Vasilyev wrote:

Timo,

On Wed, Mar 13, 2019, at 12:39 AM, Timo Sirainen wrote:

...

On 12 Mar 2019, at 10.21, Kostya Vasilyev via dovecot <dovecot@dovecot.org> wrote:

...

It makes no difference if the IDLE connection does SELECT or SELECT (CONDSTORE) prior to going IDLE.

But then as far as I know (?) - in Dovecot, once any connection uses CONDSTORE ever, even once, Dovecot creates data structures to track MODSEQ values, and those data structures are forever.

So are you saying that you can reproduce if you do for a completely new user:

doveadm exec imap -u testuser1 a select inbox b idle

And then run: echo foo | doveadm save -u testuser1 doveadm flags add -u testuser1 '\Seen' mailbox inbox 1

And the EXISTS shows up immediately after saving, but the flag change won't show up? It works fine with me.

Do you see any errors in "doveadm log errors"? Can you reproduce this if you try with some other mailbox format than mbox?

1 - Yes I was able to reproduce with a completely new user.

I didn't use your doveadm commands above, but created a new Linux user in group mail, set a password in /etc/dovecot/users, and sent a few emails to that user from another mail account / web mail. My SMTP server is Postfix.

Then I used my IDLE client code (which does SELECT CONDSTORE) and a separate netcat connection to make flags changes.

Same thing as before: 1) no flags changes flushed to IDLE 2) until the other connection (which changed flags) does a SELECT.

The only difference from before - it was enough for netcat to do plain SELECT (i.e. SELECT CONDSTORE was not necessary).

I also tried a netcat connection instead of "my IDLE client" and did plain SELECT before going IDLE (i.e. SELECT without CONDSTORE). Same thing: 1) delay 2) flags changes are flushed after the other connection does SELECT.

2 - I switched to Maildir last night - and the problem stopped for same old existing user.

So it looks triggered by a combination of mbox + SELECT (CONDSTORE)?

3 - No errors in doveadm log errors

4 - /var/log/mail.err has this

Mar 13 09:21:17 kman dovecot: auth: Error: Master requested auth for nonexistent client 29548 (created 0 msecs ago, handshake 0 msecs ago) Mar 13 09:21:18 kman dovecot: imap-login: Error: net_connect_unix(/var/run/dovecot/stats-writer) failed: No such file or directory

Probably means nothing - I tried to log in with IMAP before remembering to add my new user to /etc/dovecot/users

-- K

One more test with this new "test" user.

I'd stopped postfix and dovecot, deleted this user's mail in /var/mail/%u and ~%u/mail and started over. Still mbox format.

This time I used netcat for the IDLE connection, plain SELECT without CONDSTORE.

Flags changes were sent quickly.

Then on the IDLE connection I sent DONE and SELECT Inbox (CONDSTORE) - without logging out or closing the connection.

The issue started again.

I also learned that it's enough for the "other" (non-idle, flag changing) connection to do just "tag SELECT" - i.e. without mailbox name - which results in

• OK [CLOSED] Previous mailbox closed. tag BAD Error in IMAP command SELECT: Invalid arguments (0.001 + 0.000 secs).

but also does send the buffered up FLAGS changes on the IDLE connection.

Hope these are useful data points.

-- K