• Release Signoff Checklist
• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
  • User Stories
    • Story 1 - Opening communication to NodePorts of other cluster
    • Story 2 - Blocking the egress for not allowed insecure ports
    • Story 3 - Containerized Passive FTP Server
  • Notes/Constraints/Caveats
  • Risks and Mitigations
• Design Details
  • Validations
  • Test Plan
    • Prerequisite testing updates
    • Unit tests
    • e2e tests
  • Graduation Criteria
    • Alpha
    • Beta
    • GA
  • Upgrade / Downgrade Strategy
• Production Readiness Review Questionnaire
  • Feature Enablement and Rollback
  • Monitoring Requirements
  • Dependencies
  • Scalability
  • Troubleshooting
• Implementation History
• Drawbacks
• Alternatives

Items marked with (R) are required prior to targeting to a milestone / release.

• (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
• (R) KEP approvers have approved the KEP status as implementable
• (R) Design details are appropriately documented
• (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
• (R) Graduation criteria is in place
• (R) Production readiness review completed
• Production readiness review approved
• "Implementation History" section is up-to-date for milestone
• User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
• Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Today the ports field in ingress and egress network policies is an array that needs a declaration of each single port to be contemplated. This KEP proposes to add a new field that allows a declaration of a port range, simplifying the creation of rules with multiple ports.

NetworkPolicy object is a complex object, that allows a developer to specify what's the traffic behavior expected of the application and allow/deny undesired traffic.

There are a number of user issues like kubernetes #67526 and kubernetes #93111 where users expose the need to create a policy that allow a range of ports but some specific port, or also cases that a user wants to create a policy that allows the egress to other cluster to the NodePort range (eg 32000-32768) and in this case, the rule should be created specifying each port separately, as:

spec:
  egress:
  - ports:
    - protocol: TCP
      port: 32000
    - protocol: TCP
      port: 32001
    - protocol: TCP
      port: 32002
    - protocol: TCP
      port: 32003
[...]
    - protocol: TCP
      port: 32768

So for the user:

• To allow a range of ports, each of them must be declared as an item from ports array
• To make an exception needs a declaration of all ports but the exception

Adding a new endPort field inside the ports will allow a simpler creation of NetworkPolicy to the user.

Add an endPort field in NetworkPolicyPort

• Support specific Exception field.
• Support endPort when the starting port is a named port.

In NetworkPolicy specification, inside NetworkPolicyPort specify a new endPort field composed of a numbered port that defines if this is a range and when it ends.

I have an application that communicates with NodePorts of a different cluster and I want to allow the egress of the traffic only the NodePort range (eg. 30000-32767) as I don't know which port is going to be allocated on the other side, but don't want to create a rule for each of them.

As a developer, I need to create an application that scrapes informations from multiple sources, being those sources databases running in random ports, web applications and other sources. But the security policy of my company asks me to block communication with well known ports, like 111 and 445, so I need to create a network policy that allows me to communicate with any port except those two and so I can be compliant with the company's policy.

As a Kubernetes User, I've received a demand from my boss to run our FTP server in an existing Kubernetes cluster, to support some of my legacy applications. This FTP Server must be acessible from inside the cluster and outside the cluster, but I still need to keep the basic security policies from my company, that demands the existence of a default deny rule for all workloads and allowing only specific ports.

Because this FTP Server runs in PASV mode, I need to open the Network Policy to ports 21 and also to the range 49152-65535 without allowing any other ports.

• The technology used by the CNI provider might not support port range in a trivial way as described in [#drawbacks]

CNIs will need to support the new field in their controllers. For this case we'll try to make broader communication with the main CNIs so they can be aware of the new field.

API changes to NetworkPolicy:

• Add a new field called EndPort inside NetworkPolicyPort as the following:

// NetworkPolicyPort describes a port to allow traffic on
type NetworkPolicyPort struct {
  // The protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this
  // field defaults to TCP.
  // +optional
  Protocol   *v1.Protocol `json:"protocol,omitempty" protobuf:"bytes,1,opt,name=protocol,casttype=k8s.io/api/core/v1.Protocol"`
  
  // The port on the given protocol. This can either be a numerical or named 
  // port on a pod. If this field is not provided, this matches all port names and
  // numbers, whether an endPort is defined or not.
  // +optional
  Port       *intstr.IntOrString `json:"port,omitempty" protobuf:"bytes,2,opt,name=port"`
  
  // EndPort defines the last port included in the port range.
  // Example:
  //  endPort: 12345
  // +optional
  EndPort    int32 `json:"port,omitempty" protobuf:"bytes,2,opt,name=endPort"`
}

The NetworkPolicyPort will need to be validated, with the following scenarios:

• If an EndPort is specified a Port must also be specified
• If Port is a string (named port) EndPort cannot be specified
• EndPort must be equal or bigger than Port

[X] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Unit tests:

• test API validation logic
• test API strategy to ensure disabled fields

E2E tests:

• Add e2e tests exercising only the API operations for port ranges. Data-path validation should be done by CNIs.

• pkg/apis/networking/validation/validation: 14/Jun/2022 - 92.5%
• pkg/registry/networking/networkpolicy/strategy: 14/Jun/2022 - 75.9%

• Feature:NetworkPolicyEndPort: https://storage.googleapis.com/k8s-triage/index.html?text=EndPort#eaa4b8cdb7b461dccfa9

The flakes shown here are not related to this feature, per the tests logs

• Add a feature gated new field to NetworkPolicy
• Communicate CNI providers about the new field
• Add validation tests in API

• EndPort has been supported for at least 1 minor release
• Three commonly used NetworkPolicy (or CNI providers) implement the new field, with generally positive feedback on its usage.
• Feature Gate is enabled by Default.

• At least four NetworkPolicy providers (or CNI providers) support the EndPort field
• EndPort has been enabled by default for at least 1 minor release

The following are the CNIs that implement this feature:

• Calico
• Antrea
• Openshift SDN
• Kuberouter

If upgraded no impact should happen as this is a new field.

If downgraded the CNI wont be able to look into the new field, as this does not exists and network policies using this field will stop working correctly and start working incorrectly. This is a fail-closed failure, so it is acceptable.

• Feature gate (also fill in values in kep.yaml)
  • Feature gate name: NetworkPolicyEndPort
  • Components depending on the feature gate: Kubernetes API Server

No

Yes. One caveat here is that NetworkPolicies created with EndPort field set when the feature was enabled will continue to have that field set when the feature is disabled unless user removes it from the object.

If the value is dropped with the FeatureGate disabled, the field can only be re-inserted if feature gate is enabled again.

Rolling back the Kubernetes API Server that does not have this field will make the field not be returned anymore on GET operations, so CNIs relying on the new field wont recognize it anymore.

If this happens, CNIs will recognize the policy as a single port instead of a port range, which may break users, which is inevitable but satisfies the fail-closed requirement.

Nothing.

Yes and they can be found here

Not probably, but still there's the risk of some bug that fails validation, or conversion function crashes.

The increase of 5xx http error count on Network Policies Endpoint

Yes, with unit tests. Manual tests were also executed as the following:

• Created a KinD cluster in v1.24 and Calico as a CNI
• Created a Network Policy with endPort field to allow a Pod egress to ports from 70 to 90
• Did a test against a target in port 80 - Worked
• Disabled the Feature Gate
• The Network Policy still worked fine
• Changed the Network Policy so the range is 70 to 79, and the Network Policy was changed fine
• Traffic started to be blocked, but could call port 78 as it is still within range
• Removed endPort field, and wasn't able to re-add it as Feature gate was disabled
• Re-enabled feature gate
• Re-added endPort field with value of 90
• Traffic started to flow/be accepted again

Per the manual tests, all worked as desired.

None

Operators can determine if NetworkPolicies are making use of EndPort creating an object specifying the range and validating if the traffic is allowed within the specified range.

Also Network Policy object now supports (as alpha) status/condition fields, so Network Policy providers can add a feedback to the user whether the policy was processed correctly or not. Providing this feedback is optional and depends on implementation by each NPP.

• Other
• Details: The API Field must be present when a NetworkPolicy is created with that field. The feature working correctly depends on the CNI implementation, so the operator can look into CNI metrics to check if the rules are being applied correctly, like Calico that provides metrics like felix_iptables_restore_errors that can be used to verify if the amount of restoring errors raised after the feature being applied. For NetworkPolicy Providers that doesn't support this feature, a new status field was added in Network Policy object allowing the providers to give feedback to users using conditions. Any NPP that does not support this feature should add a condition on the Network Policy object.

Operators can use metrics provided by the CNI to use as SLI, like felix_iptables_restore_errors from Calico to verify if the errors rate has raised.

• per-day percentage of API calls finishing with 5XX errors <= 1% is a reasonable SLO

• Are there any missing metrics that would be useful to have to improve observability of this feature? N/A

Yes, a CNI supporting the new feature

No

No

No

• API type(s): NetworkPolicyPorts
• Estimated increase in size: 2 bytes for each new EndPort value specified + the field name/number in its serialized format
• Estimated amount of new objects: N/A

N/A

It might get some increase of resource usage by the CNI while parsing the new field.

As this feature is mainly used by CNI providers, the reaction with API server and/or etcd being unavailable will be the same as before.

N/A

Remove EndPort field and check if the number of errors reduce, although this might lead to undesired Network Policy, blocking previously working rules.

• 2022-06-14 Propose GA graduation
• 2021-05-11 Propose Beta graduation and add more Performance Review data
• 2020-10-08 Initial KEP PR

• The technology used by the CNI provider might not support port range in a trivial way. As an example, OpenFlow did not supported to specify port range for a while as commented in kubernetes #67526. While this has changed in Open vSwitch v1.6, this still might be a caveat for other CNIs, like eBPF based CNIs will need to populate their maps in a different way.

For this cases, CNIs will have to iteract through the Port Range and populate their packet filtering tables with each port.

During the development of this KEP there was an alternative implementation of the NetworkPolicyPortRange field inside the NetworkPolicyPort as the following:

// NetworkPolicyPort describes a port or a range of ports to allow traffic on
type NetworkPolicyPort struct {
  // The protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this
  // field defaults to TCP.
  // +optional
  Protocol *api.Protocol
  
  // The port on the given protocol. This can either be a numerical or named 
  // port on a pod. If this field is not provided but a Range is 
  // provided, this field is ignored. Otherwise this matches all port names and
  // numbers.
  // +optional
  Port *intstr.IntOrString

  // A range of ports on a given protocol and the exceptions. If this field 
  // is not provided, this doesn't matches anything
  // +optional
  Range *NetworkPolicyPortRange
}

But the main design suggested in this Kep seems more clear, so this alternative has been discarded.

Also it has been proposed that the implementation contains an Except array and a new struct to be used in Ingress/Egress rules, but because it would bring much more complexity than desired the proposal has been dropped right now:

// NetworkPolicyPortRange describes the range of ports to be used in a 
// NetworkPolicyPort struct
type NetworkPolicyPortRange struct {
    // From defines the start of the port range
    From         uint16
    
    // To defines the end of the port range, being the end included within the
    // range
    To           uint16
    // Except defines all the exceptions in the port range
    +optional
    Except       []uint16