On 14. Dec 2023, at 16.21, Alex <alex@merlinux.eu> wrote:
Hello all,
I have been investigating the reason for high end-to-end email delivery delay (>1 second to send a message to echo bot and receive a reply back) on a Postfix+Dovecot email server setup used to run the tests for Delta Chat core [1]. It appeared that most of the time was spent between the submission SMTP service accepting the mail for delivery and receiving an IDLE wakeup on the other side.
Looking into the Dovecot source code, I found that all notifications from the storage to callbacks set up by IDLE are delayed by a debounce constant NOTIFY_DELAY_MSECS [2]. It appears to have been added in 2009 [3], but even before that Dovecot was only delivering notifications when a second-resolution timer is changed by at least 1, so IDLE notifications probably were delayed by half a second on average even before that.
Commit message says "If the notification is done immediately, IDLE may not notice the change because it's not finished yet." This sounds like a workaround for some internal Dovecot bug.
...
I wonder if this debounce delay was added as a workaround for an internal Dovecot bug or a client bug and is not necessary anymore. It does not seem to be useful for well-behaving clients because if necessary the client may debounce notifications on its side and postpone interrupting IDLE with "DONE".
I can't really think of any situation where this would be necessary anymore. Not sure why it was necessary back in 2009 - maybe Maildir didn't have quite as many locks back then. Anyway now even if a change is partial at the time the mailbox-watch triggers, it runs full mailbox syncing which will lock the mailbox, so it waits for any previous changes to finish being written.
I would actually like to disable this delay completely on our server setup [6], as for chat it is common to receive multiple messages in a short period of time, e.g. when sending a message to a group and receiving multiple read receipts from currently active users.
We could of course make this a setting, but maybe it's not necessary. Would this behavior also work for you? :
- If the current IDLE hasn't notified anything yet, send the notification immediately without waiting
- Otherwise, send it max once per 0.5 seconds.
So if you're using the same connection to always break out of IDLE after a notification, there would be no dalays.