Hello all,
I have been investigating the reason for high end-to-end email delivery delay (>1 second to send a message to echo bot and receive a reply back) on a Postfix+Dovecot email server setup used to run the tests for Delta Chat core [1]. It appeared that most of the time was spent between the submission SMTP service accepting the mail for delivery and receiving an IDLE wakeup on the other side.
Looking into the Dovecot source code, I found that all notifications from the storage to callbacks set up by IDLE are delayed by a debounce constant NOTIFY_DELAY_MSECS [2]. It appears to have been added in 2009 [3], but even before that Dovecot was only delivering notifications when a second-resolution timer is changed by at least 1, so IDLE notifications probably were delayed by half a second on average even before that.
Commit message says "If the notification is done immediately, IDLE may not notice the change because it's not finished yet." This sounds like a workaround for some internal Dovecot bug.
I have nevertheless tried to compile Dovecot with this constant reduced from 500 ms to 1 ms [4] and it resulted in exceptionally fast delivery without any problems caused to the Delta Chat core online test suite which is using this Postfix+Dovecot server accounts. We are running all the tests against this server and it appears to work fine. Round trip time for sending a message to an echo bot and receiving a reply back is reduced from 1.1 s to 0.2 s, which is easily visible in a chat app [5].
I wonder if this debounce delay was added as a workaround for an internal Dovecot bug or a client bug and is not necessary anymore. It does not seem to be useful for well-behaving clients because if necessary the client may debounce notifications on its side and postpone interrupting IDLE with "DONE".
I would actually like to disable this delay completely on our server setup [6], as for chat it is common to receive multiple messages in a short period of time, e.g. when sending a message to a group and receiving multiple read receipts from currently active users.
[1] https://github.com/deltachat/deltachat-core-rust/ [2] https://github.com/dovecot/core/blob/93a53a9d590f0220de28600f36a969cc38c3148... [3] https://github.com/dovecot/core/commit/56fb5d09955b6097f77b341717fd9b70e9f13... [4] https://github.com/deltachat/chatmail/issues/72#issuecomment-1806645153 [5] https://delta.chat/assets/blog/nine-echo.mp4 [6] https://github.com/deltachat/chatmail
Best regards, Alex