I want to suggest you add a feature for validation by IP before validation by a provider (google for example).
Here is a real case... We have a private web resource, which should be protected by vouch-proxy. But we have web requests from the internal network and also external. I suppose it would be very useful to not disturb users from the internal network.
Unfortunately, it is not possible to realize using Nginx. (because 'if' is not allowed for auth_request)