Conditionally disabling auth policy

I've been digging into the auth policy stuff with weakforced lately. There are cases (IP ranges, so could be wrapped up in remote {} blocks) where it'd be nice to skip the auth policy (internal hosts that I can trust, but that are hitting the same servers as the outside world).

Is there any way to disable auth policy, possibly inside a remote{}?

auth_policy_server_url complains that it can't be used inside a remote block, so no dice there. Anything I'm missing?