On Thu, Dec 14, 2023 at 07:17:16PM +0200, Timo Sirainen wrote:

...

I wonder if this debounce delay was added as a workaround for an internal Dovecot bug or a client bug and is not necessary anymore. It does not seem to be useful for well-behaving clients because if necessary the client may debounce notifications on its side and postpone interrupting IDLE with "DONE".

I can't really think of any situation where this would be necessary anymore. Not sure why it was necessary back in 2009 - maybe Maildir didn't have quite as many locks back then. Anyway now even if a change is partial at the time the mailbox-watch triggers, it runs full mailbox syncing which will lock the mailbox, so it waits for any previous changes to finish being written.

I have only tested with Maildir, maybe it was important for some other storage at the time it was introduced.

...

I would actually like to disable this delay completely on our server setup [6], as for chat it is common to receive multiple messages in a short period of time, e.g. when sending a message to a group and receiving multiple read receipts from currently active users.

We could of course make this a setting, but maybe it's not necessary. Would this behavior also work for you? :

• If the current IDLE hasn't notified anything yet, send the notification immediately without waiting
• Otherwise, send it max once per 0.5 seconds.

For IDLE this would work and be indistinguishable from no delay as Delta Chat core always interrupts IDLE with DONE as soon as it receives a response (e.g. EXISTS or RECENT, but not keepalive).

But as this delay code is a part of the storage (src/lib-storage/mailbox-watch.c) and knows nothing about IDLE, I am not sure how this could be implemented. There is also a plan to implement NOTIFY (RFC 5465) extension support in Delta Chat [1] where "NOTIFY SET" is executed only once and then there are incoming notifications for the rest of the IMAP session. NOTIFY uses the same callback and has the same 0.5 second delay as a result.

What could work is a change to callback immediately if there have been no callback call in the last 0.5 seconds and otherwise delay it. Then there will be no difference as long as two emails in a row do not come wihin half a second.

But I also wonder why not remove the delay completely then if there is no need for it? Debounce can always be implemented on the client side if necessary, if the client does not want to react to interruptions more frequently than once every half a second it can delay the reaction locally.

So if you're using the same connection to always break out of IDLE after a notification, there would be no dalays.

[1] https://github.com/deltachat/deltachat-core-rust/pull/4990

On Fri, Dec 15, 2023 at 11:37:10AM +0200, Timo Sirainen wrote:

On 15. Dec 2023, at 1.59, Alex alex@merlinux.eu wrote:

...

...

...

I would actually like to disable this delay completely on our server setup [6], as for chat it is common to receive multiple messages in a short period of time, e.g. when sending a message to a group and receiving multiple read receipts from currently active users.

We could of course make this a setting, but maybe it's not necessary. Would this behavior also work for you? :

• If the current IDLE hasn't notified anything yet, send the notification immediately without waiting
• Otherwise, send it max once per 0.5 seconds.

For IDLE this would work and be indistinguishable from no delay as Delta Chat core always interrupts IDLE with DONE as soon as it receives a response (e.g. EXISTS or RECENT, but not keepalive).

But as this delay code is a part of the storage (src/lib-storage/mailbox-watch.c) and knows nothing about IDLE, I am not sure how this could be implemented. There is also a plan to implement NOTIFY (RFC 5465) extension support in Delta Chat [1] where "NOTIFY SET" is executed only once and then there are incoming notifications for the rest of the IMAP session. NOTIFY uses the same callback and has the same 0.5 second delay as a result.

Yeah, it would need some additional API changes to make it work that way.

Maybe move this debounce out of storage and into IDLE implementation (imap/cmd-idle.c) then? Then other callers of mailbox_notify_changes will have no delay. I only see other calls to mailbox_notify_changes() in imap/imap-notify.c and plugins/virtual/virtual-storage.c. Virtual storage does not need debounce then just as the normal storage.

Then if IDLE implementation manages this debounce itself and sends the first notification immediately after each IDLE resetup, that would be perfect without the need for a configuration option.

As for NOTIFY, same logic cannot be implemented, but hopefully NOTIFY users are advanced enough to debounce themselves? My plan for using NOTIFY is to subscribe directly to receive message body, so in my case every notification should be immediate.

...

What could work is a change to callback immediately if there have been no callback call in the last 0.5 seconds and otherwise delay it. Then there will be no difference as long as two emails in a row do not come wihin half a second.

I thought it would be important in those cases also? But if not, that's the simplest fix, yes. And it could be shrunk even further from 0.5 seconds to maybe 0.1 seconds.

Yes, ideally first notification after IDLE should always be immediate. If debounce logic is moved to IDLE implementation this looks simple too.

...

But I also wonder why not remove the delay completely then if there is no need for it? Debounce can always be implemented on the client side if necessary, if the client does not want to react to interruptions more frequently than once every half a second it can delay the reaction locally.

There could be some performance loss if there is never any delay. For example some offline client could connect and start issuing a bunch of flag updates, while another IDLEing connection keeps trying to rapidly send notifications about each of those changes separately instead of bundling them all into a single mailbox sync.

So this IDLE connection is some UI client that will be slowed down by trying to refresh the view on every change? Either the client tries to update the state and redraw the view on every incoming update or it updates the state immediately and debounces view redraws on its own. In the first case delaying IDLE/NOTIFY updates would not help as the number of redraws is not reduced. In the second case there is no need to debounce on the server side as state updates are going to be fast anyway.

I have made a patch that actually deletes 500 ms delay from mailbox-watch.c rather than reducing it to 1 ms.

I then looked into adding debounce to src/imap/cmd-idle.c, but there I cannot be sure that mailbox pointer passed to idle_callback() will not be freed by the time debounce timer expires. If we want to have debounce in IDLE, it should be kept in mailbox-watch.c, but reset via special API from cmd-idle.c every time IDLE is started.

Then I thought about the issue again and am pretty sure debounce is not needed for IDLE. If an offline client gets online and synchronizes a lot of flags, it usually can do it in one command. If it does not, because of bad implementation or because flags are different for different messages, it will send multiple commands. In this case IDLE-ing client will receive multiple updates, but this is not bad for performance. If IDLE-ing client does not ignore these updates and actually processes them, worst case it will immediately get busy processing the first incoming update. But by the time it finishes processing the first update it will receive many updates and process them in batch, effectively debouncing on the client even if there is no special code doing this.

So unless there is an IMAP client that is known to benefits from debouncing, I suggest that no debouncing is added for IDLE.

Alex (1): storage: remove 500 ms debounce on IMAP IDLE notifications

src/lib-storage/mail-storage-private.h | 2 +- src/lib-storage/mailbox-watch.c | 21 ++++----------------- 2 files changed, 5 insertions(+), 18 deletions(-)

-- 2.43.0