Is rquota purposefully not present in the RPM at http://atrpms.net/dist/el5/dovecot/dovecot-1.2.8-0_103.el5.x86_64.rpm.html ?
On a test machine I decided to give dovecot 1.2 a go with that RPM and everything went great except I was getting the following error in the dovecot log (and quota was not working):
Dec 9 10:50:43 webmailtest2 dovecot: IMAP(auser) rip=127.0.0.1: quotactl(Q_GETQUOTA, nfsserver.geneseo.edu:/vol/email/users) failed: No such file or directory
I had seen some messages on the mailing list talking about making sure you build with HAVE_RQUOTA defined at compile time. I wasn't able to determine if the RPM on the aforementioned page was simply missing the rquota code, but upon building the SRPM from the same place (http://atrpms.net/dist/el5/dovecot/, SRPM itself is http://dl.atrpms.net/all/dovecot-1.2.8-0_103.src.rpm) I saw HAVE_RQUOTA fly by and the rquota source files get built and linked, and when using those binaries rquota started working.
I scanned the changelog for the binary RPM for something indicating that rquota support had been taken out but didn't see anything.
Apologies if this is better sent to the package maintainer; I'm hoping he checks this list or someone has already run into this and can set me (and anyone else who may run in to this problem) straight.
-David Warden