[Dovecot] Limiting SSL (external) access to certain users ?

I am looking for a way to limit outside access to the IMAP server to certain users only. Basic idea is to use two userdbs, one for SSL connections and one "internal" with most users in it. Is there a way to do this without running two dovecot configurations?

Thanks for hints, Jakob