My mail is stored under ~/mail/.imap (not sure what this format is called), I mean not "single file mbox".

I have not changed any IDLE related config settings:

doveconf | grep -i idle
default_idle_kill = 1 mins
director_ping_idle_timeout = 30 secs
imap_idle_notify_interval = 2 mins
imapc_max_idle_time = 29 mins
mailbox_idle_check_interval = 30 secs

What can I do to make Dovecot notify IDLE clients about flags changes - more quickly? Preferably near-instant?

It should simply just work, assuming there aren't any weird inotify limits, but you should get errors logged about reaching those. You could see if it makes any difference to set mailbox_idle_check_interval=1s