Dovecot doesn't try to enforce filesystem quota limits. I'm admittedly feeling utterly stupid in trying to tell an author what his programm is doing, but ...
It just handles the EDQUOT error from write(). ... I ktrace'd imap and there was no failing write(), only suspicious rpcs to the file server.
So the only thing that the fs quota plugin does is report the limits and current usage to clients via IMAP QUOTA commands. I thought the latter was the job of the imap-quota plugin?
I read the code and verified by attaching gdb to a running imap proces that there is the following code path leading to enforcement of NFS soft quotas:
v.copy -> quota_copy() -> quota_check() -> quota_try_alloc() -> quota_test_alloc() -> quota_tracsaction_set_limits() -> quota_get_resource() -> fs_quota_get_resource() -> do_quota().
I don't see an easy way to fix this. If one wants the IMAP GETQUOTA command to report the soft limits (which I find reasonable) but continue allowing allocation up to the hard limit (which I find mandatory), one must either teach the whole quota system about hard/soft limits or keep it from enforcing the (soft) limits if the fs is going to enforce them anyway.