Add Support for Managing the Envoy Proxy Service

[danehans]

As of v0.2.0, the Kube Infra Manager manages a Service of type LoadBalancer to perform L4 load-balancing across the managed Envoy proxy fleet. Some users may wish to use Envoy for internal-only proxying, configuring various load-balancer settings, etc. Envoy Gateway provides the EnvoyProxy API type for configuring the Envoy proxy infrastructure. This API should be expanded to support common Service configuration requirements.

[bmetzdorf]

Thanks @danehans. I'm specifically interested in the integration with 3rd party cloud providers, beginning with AWS and their NLB. Something like these annotations will be needed on the Service:

service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
service.beta.kubernetes.io/aws-load-balancer-type: "external"

Since Envoy Gateway will handle TLS and HTTP(S) itself, we just need TCP and UDP traffic forwarded by the NLB. And in order to not pay the node port double hop penalty, we could use NLB IP based targeting and use Proxy Protocol to retain client IP information (UDP/QUIC TBD).

[arkodg]

Thanks @danehans. I'm specifically interested in the integration with 3rd party cloud providers, beginning with AWS and their NLB. Something like these annotations will be needed on the Service:

service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
service.beta.kubernetes.io/aws-load-balancer-type: "external"

Since Envoy Gateway will handle TLS and HTTP(S) itself, we just need TCP and UDP traffic forwarded by the NLB. And in order to not pay the node port double hop penalty, we could use NLB IP based targeting and use Proxy Protocol to retain client IP information (UDP/QUIC TBD).

@bmetzdorf this should be addressed with #377

[danehans]

Currently, an ELB gets created for the Envoy service when running EG in AWS. We should probably set service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp to put the ELB in TCP mode and service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" to enable proxy protocol by default (when running in AWS). Or should we default to NLB with the annotations specified by @bmetzdorf? As I mentioned in #377, I prefer creating an API for managing the Envoy network endpoints instead of passing service annotations. @LukeShu @skriss @youngnick @AliceProxy @Xunzhuo thoughts?

[danehans]

I've created a design doc to help faciliate discussion for supporting this use case. @bmetzdorf @arkodg @youngnick @skriss @LukeShu @AliceProxy @Xunzhuo PTAL.

[arkodg]

I've created a design doc to help faciliate discussion for supporting this use case. @bmetzdorf @arkodg @youngnick @skriss @LukeShu @AliceProxy @Xunzhuo PTAL.

@danehans took a look at this review, was unable to leave comments in there due to limited permissions, so leaving them here - the doc proposes creating high level load balancer providers such as aws

• I worry such level of vendor specificity will trigger the project supporting and maintaining many
  such loadbalancer types/providers and a lot of time might get spent managing such driver code
• taking the aws case, there are many user inputted annotations such as service.beta.kubernetes.io/load-balancer-source-ranges that EG will not be able to compute beforehand

[danehans]

took a look at this review, was unable to leave comments in there due to limited permissions

Anyone should now be able to comment in the doc.

I worry such level of vendor specificity will trigger the project supporting and maintaining many
such loadbalancer types/providers and a lot of time might get spent managing such driver code

It does require more maintenance but provides a more consistent UX. Otherwise, users can freely pass annotations that may be invalid. We could validate the annotations but that can turn into a difficult matrix to support.

taking the aws case, there are many user inputted annotations such as service.beta.kubernetes.io/load-balancer-source-ranges that EG will not be able to compute beforehand

EG can perform some validation for this use case. For example, if a user specifies an invalid range, e.g. 1.2.3.4/34. In this instance, EG would not apply the config and surface an EnvoyProxy status condition that states the provided CIDR is invalid.

[danehans]

@bmetzdorf do you have any thoughts or preferences on the approach to resolving this issue?