calico on Digi Hunch

Traffic Segmentation on Kubernetes Platform

Thu, 27 Jan 2022 13:54:00 -0400

When operating Kubernetes as a platform for multiple tenants, one of the concerns is controlling the network traffic. This is sometimes referred to as traffic segmentation. This initiative involves a broad range of technical topics from networking to containerization. By no means I am an expert on each of those topics. I have however developed some best practices in how to break down this challenge and hence bringing the thought into this post.

Tenant Isolation

Kubernetes has the concept of namespace to logically separate resources allocated for each tenant. Each tenant only operates within their given namespaces. The isolation of computing resources such as CPU and memory can be managed via ResourceQuota objects, and they are enforced at the kernel level, leaving networking isolation the main discussion in the topic of tenant isolation. If the platform hosts a lot of stateful workload then we also needs to address tenant isolation at the storage layer. In this post we focus on the network aspect of resource isolation, aka traffic segmentation.

Controlling network traffic can require a significant amount of efforts depending on the goal. That is why we need to first assess the multi-tenancy models:

Soft multi-tenancy: usually a platform is shared by multiple teams within the same organization. Tenants are incentivized to be good neighbours.
Hard multi-tenancy: usually a platform shared by multiple customers from different organizations. There is no trust between different tenants, or between tenant and platform operator.

Reality may sits somewhere in between, but we often have to come back to this model when making a technical decision, because it determines the degree of tenant isolation, or the amount of effort we are willing to put in on tenant isolation. At the tough end, is zero-trust network, which usually have the following requirement:

Requirement 1: All network connections are subject to enforcement (not just those that cross zone boundaries).
Requirement 2: Establishing the identity of a remote endpoint is always based on multiple criteria including strong cryptographic proofs of identity. In particular, network-level identifiers like IP address and port are not sufficient on their own as they can be spoofed by a hostile network.
Requirement 3: All expected and allowed network flows are explicitly allowed. Any connection not explicitly allowed is denied.
Requirement 4: Compromised workloads must not be able to circumvent policy enforcement.
Requirement 5: Many Zero Trust Networks also rely on encryption of network traffic to prevent disclosure of sensitive data to hostile entities snooping network traffic. This is not an absolute requirement if private data are not exchanged over the network, but to fit the criteria of a Zero Trust Network, encryption must be used on every network connection if it is required at all. A Zero Trust Network does not distinguish between trusted and untrusted network links or paths. Also note that even when not using encryption for data privacy, cryptographic proofs of authenticity are still used to establish identity.

As you can see there’s a lot of efforts involved in building a zero-trust network. The cost of building a zero-trust network is worth it only when we determines that the overall business requirement demands it.

Pod Networking

It is important to understand Pod networking before developing a traffic segmentation strategy. Pod networking has to do with the CNI driver used for the cluster. There are in general two categories:

Overlay network: Pods are placed on a VXLAN configuration. This is mostly seen in basic Kubenet mode or CNI drives such as Flannel. NAT is required for Pods to communicate across nodes, which might introduce performance issues when deployed at scale. Pods do not use IP address from the host network.
Regular network: In this mode Pods are on the same network as the nodes are. For example, Azure CNI assigns Pods with IP address from a given V-Net. The AWS-VPC CNI integrates VPC networking with Pods. Since Pods are on a corporate network, the traffic control must also consider measures at the whole network level.

The main benefit of the first approach, is that IP exhaustion is less likely due to the introduction of a VxLAN. The other benefit from a networking perspective is that the Pod networking is born separated from the corporate network. In the second approach, by assigning Pods with a corporate IP address (which brings the risk of IP exhaustion), Pods are also potentially exposed to all corporate traffic at layer 3. To tackle this additional risk, network security group should be used in the V-Net for Azure AKS, or security groups for Pods should be considered with AWS EKS. Although we will discuss Network Policy in the rest of this essay, Network Policy mostly addresses the traffic segmentation issue within a Kubernetes cluster. A Pod placed on the corporate network needs traffic segmentation strategies from the perspective of the whole network.

Another network-level traffic segmentation strategy is on the corporate firewall. For example, with AKS you can specify outbound type as user-defined routes (UDR) to direct all outbound traffic through a corporate firewall where traffic will be inspected. There are firewall products dedicated for managing highly dynamic pod traffic from Kubernetes. This strategy can be used in conjunction with network security groups.

Network Policy

Kubernetes’s default behaviour is to allow traffic between any two pods in the cluster network. This is undesirable. NetworkPolicy is the native Kubernetes construct for platform operators and application developer to control network traffic at layer 3/4. It uses namespace and pod selectors, and is defined based on allow rules, which is good for general use.

Further to the native Network Policy, you can adopt third party policies for advanced features. For example, Azure has Azure Network policy (works for Azure CNI only) and Calico Network policy (works for Calico CNI, Azure CNI or Kubenet). The third party network policies usually provides advanced features such as:

Deny rules
multiple types of endpoints in addition to Pods, for example, VMs, network interfaces which can be useful in network-level traffic control
ordering and priority of rules
Flexible matching rules

Calico network has a page that summarizes its features and how it extends the Kubernetes NetworkPolicy. Below is an example of a Calico’s network policy:

apiVersion: projectcalico.org/v3 
kind: NetworkPolicy 
metadata: 
 name: allow-tcp-6379 
 namespace: production 
spec: 
 selector: color == 'red'
 ingress: 
 - action: Allow 
 protocol: TCP 
 source: 
 selector: color == 'blue' 
 namespaceSelector: shape == 'circle'
 destination: 
 ports: 
 - 6379

It is as self-explanatory as Kubernetes Network Policy. No matter which kind of network policy, this approach takes effect at layer 3/4. The rules are eventually implemented in the kernel on the node (Iptables). The management of this layer is usually by the platform team and they need to have some application knowledge.

Authorization at Application Layer

Traffic above layer 4 is considered application layer traffic. At application layer, the decision to allow or deny a request is by definition an authorization decision. Another layer of protection can be placed at layer 4 is mTLS which ensures that each request to have an identity. The authorization can be built in the application, but it is also very common to offload these functions to the service mesh layer. For example, Istio has constructs such as PeerAuthentication, Request Authentication and Authorization Policy. We will those in more details in a few coming blog posts. Below is a simple example of Istio’s Authorization Policy:

apiVersion: security.istio.io/v1beta1 
kind: AuthorizationPolicy 
metadata: 
 name: "details-viewer" 
 namespace: default 
spec: 
 selector: 
 matchLabels: 
 app: details 
 action: ALLOW 
 rules: 
 - from: 
 - source: 
 principals: ["cluster.local/ns/default/sa/bookinfo-productpage"]
 to: 
 - operation: 
 methods: ["GET"]

The rule is also fairly self-explanatory. Compared to Network Policy, the point of enforcement of these Authorization policies are at the envoy proxy. The management of policies at this layer can be debatable if department boundaries are not clear, but it should in general be owned by personnels with good application knowledge.

Consistency between Policies

In-cluster traffic can be controlled with both Network Policy (Calico or Kubernetes) operating at layer 3-4, and Authorization Policy (Istio) at layer 4-7. This brings another challenge of maintaining consistency between the two types of policies. This is especially challenging when they are managed by different teams in a corporate and therefore many operators for soft multi-tenant platform choose not to implement Network Policy or only implements a baseline.

Some network solution providers builds a solution for this. For example, Calico has the capability to enforce network policy for Istio. This integration requires some configuration, but the enhanced GlobalNetworkPolicy supports HTTP methods, eliminating the need to define a separate Authorization Policy in Istio and worry about its consistency with NetworkPolicy. The platform build however, still needs to determine who owns this policy construct. Below is an example from Calico documentation:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
 name: customer
spec:
 selector: app == 'customer'
 ingress:
 - action: Allow
 http:
 methods: ["GET"]
 egress:
 - action: Allow

One of the benefits of using this integration is a unified policy language based on GlobalNetworkPolicy CRD. In the mean time, organization should also develop strategy to ensure that, once Calico is integrated with Istio, then there is no need to separately build authorization policies, which may come in conflict with Global network policy.

Summary

Controlling network traffic is difficult on Kubernetes platform. In this article I proposed a few angles to approach this issue for enterprise clients.

Kubernetes Networking Solutions Overview

Tue, 22 Jun 2021 12:14:26 -0400

Kubernetes networking involves a lot of details. We discuss some CNI plugins in this post.

The most basic mode is kubenet. We use –network-plugin=kubenet with kubelet process to use it. Kubenet is not a CNI plugin, but it works with bridge, lo and host-local (CNI-compliant implementations). We can directly specify MTU with –network-plugin-mtu. Kubenet is a basic network plugin, based on bridge plugin, with the addition of port mapping and traffic shaping. It does not offer cross-node networking itself. Today it is typically used with managed clusters by cloud providers, where the cloud provider set up routing rules themselves for inter-node communication.

When a cluster goes multi-node, the main challenge is communication between Pods across different nodes. Pods come and go. The size of cluster could increase or decrease as well. The network solutions come in two network types: overlay network based on encapsulation, or non-overlay networks, most likely using routing techniques. Common backends for for multi-host container networking solutions include VXLAN encapsulation, IPIP encapsulation, host-gw, IPSec. In addition, there are some backends that only used by certain plugins.

Common Backends

VXLAN: use in-kernel VXLAN to encapsulate the packets. VXLAN is a virtual networking capability in Linux which is also used in virtualization technology. VXLAN is an overlay technology requiring encapsulation of overlay network’s layer-2 frame into UDP packet at layer 4 of underlay network. When configured, the VxLAN backend creates a Flannel interface on every host. When a container on one node wishes to send traffic to a different node, the packet goes from the container to the bridge interface in the host’s network namespace. From there the bridge forwards it to the Flannel inteface because the kernel route table designates that this interface is the target for the non-local portion of the overlay network. The Flannel network drive wraps the packet in a UDP packet and sends it to the target host. Once it arrives at its destination, the process flows in reverse, with the Flannel driver on the destination host unwrapping the packet, sending it to the bridge interface, and from there the packet find its way into the overlay network and to the destination Pod.

host-gw: the host-gw is a non-overlay solution that maintains route tables on Linux Host to allow Pods to communicate across Nodes. It is only used in Flannel plugin. Suppose we have two hosts, each with two containers as connected below. Initially, container0 is not able to reach container2 because eth0 on node0 does not have an entry that matches container2’s IP address. The packet is there sent to default route, which isn’t destined to container2.

However, if we build rules to match container IP address, on the route table of each node. The issue would be solved. This is essentially how host-gw works. Specifically, on node 0, we add “ip route add 192.168.1.0/24 via 10.20.0.2 dev eth0”, on node 1, we add “ip route add 192.168.0.0/24 via 10.20.0.1 dev eth0). The host-gw in Flannel will manage rule addition to us. Note that the two hosts must have direct layer 2 connectivity. In other words, there must not be a router between the two nodes. Otherwise, the routing table on the router is out of reach. In fact, all nodes in a Flannel network must have layer 2 connectivity with each other. In other words, all nodes must be in a single LAN. Host-gw provides better performance than VxLAN.

IPSec uses in-kernel IPSec to encapsulate and encrypt the packets. IPsec is a group of protocols to ensure authentication and encryption per packet between devices. Since it secures traffic at layer 3 and now it has become a major backend technology for VPN. IPsec adds several headers and trailers to datagram containing authentication and encryption information. The two major protocols working in IPSec are AH (Authentication Header) and ESP (Encapsulating Security Payload). AH serves up authentication services only; ESP provides both authentication and encryption abilities. It also uses IKE protocol for key exchange.

IPSec works in two modes: transport and tunnelling mode.

Transport mode creates a secure tunnel between two devices end to end. The payload of each datagram is encrypted, but the original IP header is not. Intermediary routers are thus able to view the final destination of each datagram, unless a separate tunnelling protocol (e.g. GRE) is used.
Tunnel mode works between two endpoints, such as two routers, protecting all traffic that goes through the tunnel. The original IP header containing the final destination of the datagram is encrypted, in addition to the payload. To tell intermediary routers where to forward the datagrams, IPsec adds a new IP header. At each end of the tunnel, the routers decrypt the IP headers to deliver the datagram to their destinations. The intermediary routers does not know the final destination, or what transport protocol is used.

IPIP (IP over IP) tunnel is typically used to connect two internal IPv4 subnets through public IPv4 internete. It has the lowest overhead but can only transmit IPv4 unicast traffic.

CNI Plugins

Originally, the network functions were developed in-tree. Then the CNI specification came up to allow plugin development out-of-tree to implement cluster networking functions. The Container Network Interface seeks to completely decoupled network management from container runtime. Kubernetes picked CNI over CNM in 2016, as discussed in my virtualization discussion. CNI clearly defines the specification for following activities:

When a Pod comes up, give it a network interface
Assign IP to the network interface
When a Pod is deleted, delete the associated network interface

When we configure a Kubernetes cluster, we must specify –network-plugin switch, so that the cluster is operational. If we use CNI as network-plugin, we also need to install the plugin, optionally with the help of Rancher.

On the worker node, we use –network-plugin=cni with kubelet process to use CNI plugins. A plugin may consist one or more binaries. The binaries are located in /opt/cni/bin (or otherwise specified by –cni-bin-dir). The configurations are located in /etc/cni/net.d (or otherwise specified in –cni-conf-dir). Note that the configuration file may reference different plugin implementations for different network management purpose (e.g. interface creating, address allocation, etc). The container networking repo provided some reference implementations and some of them are used by other plugins. These reference implementations include:

Main (interface creating): bridge, ipvlan, loopback, ptp, macvlan, etc
IPAM (IP address management): host-local, dhcp, static
Meta (other plugins): portmap, bandwidth

So, a CNI plugin consists of a networking solution for backend, and binaries to cover the aspects outlined above. I discussed some common backends above. Below I will introduce some common plugins and backends only available to each plugin

Flannel

Flannel by CoreOS: supports a range of backends. The advantage of Flannel is it reduces the complexity of doing port mapping. This is a great post that covers the mechanism.

Flannel with overlay (e.g. VxLAN on UDP encapsulation)

It supports VXLAN, host-gw, IPSec, IPIP as well as the followings:

Amazon VPC: recommended with Amazon VPC. AWS VPC creates IP routes in an AWS route table. The number of records in this table is limited by 50 so you can’t have more than 50 machines in a cluster.

GCE: recommended with Google Compute Engine Network. Instead of using encapsulation, GCE also manipulates IP route to achieve maximum performance. Because of this, a separate flannel interface is not created.
UDP: debugging only for old kernels that don’t support VXLAN

Calico

Border Gateway Protocol (BGP) is a standardized exterior gateway protocol designed to exchange routing and reachability information among autonomous systems (AS) on the Internet.

Calico operates at layer 3. It prefers BGP without an overlay network for the highest speed and efficiency, but in scenarios where hosts cannot directly communicate with one another, it can utilize an overlay solution (e.g. VxLAN or IP-in-IP). Calico also supports network policies for protecting workloads and nodes from malicious activity or aberrant applications.

The Calico networking Pod contains a CNI container to keep track of Pod deployment, and register addresses and routes. It also contains a daemon that announces the IP and route information to the network via the Border Gateway Protocol (BGP). The BGP daemon build a map of the network that enables cross-host communication.

Calico requires a distributed and fault-tolerant key/value store, and deployments often choose etcd to deliver this component. Calico uses it to store metadata about routes, virtual interfaces, and entwork policy objects. Calico can either use a separate HA deployment of etcd, or the same etcd datastore with the Kubernetes cluster.

When we are unable to use BGP (e.g. with cloud provider, or in an environment where we have no permission to configure router peers. Calico’s IP-in-IP mode encapsulates packets before sending them to other nodes.

Once IP-in-IP is configured, Calico wraps inter-Pod packets in a new packet with headers that indicate the source of the packet is the host with the originating Pod, and the target of the packet is the host with the destination Pod. The Linux kernel performs this encapsulation, and then forwards the packet to the destination host where it is unwrapped and delivered to the destination Pod.

Canal

The followings is quoted from Rancher website:

Canal seeks to integrate the networking layer provided by Flannel with the networking policy capabilities of Calico. As the contributors worked through the details however, it became apparent that a full integration was not necessarily needed if work was done on both projects to ensure standardization and flexibility. As a result, the official project became somewhat defunct, but the intended ability to deploy the two technology together was achieved. For this reason, it’s still sometimes easiest to refer to the combination as “Canal” even if the project no longer exists.

Because Canal is a combination of Flannel and Calico, its benefits are also at the intersection of these two technologies. The networking layer is the simple overlay provided by Flannel that works across many different deployment environments without much additional configuration. The network policy capabilities layered on top supplement the base network with Calico’s powerful networking rule evaluation to provide additional security and control.

Weave Net

Weave Net by Weaveworks offers a different paradigm. Weave creates a mesh overlay network between each of the nodes in the cluster, allowing for flexible routing between participants. Applications use the network just as if the containers were all plugged into the same network switch, with no need to configure port mappings and links.

For more good references to determine networking options, check out these posts:

Calico blog
Rancher blog
Kubevious blog