service mesh on Digi Hunch

Kubernetes Platform as a Service and Red Hat OpenShift

Sun, 25 Jun 2023 11:10:15 -0400

The Three-layer model

Kubernetes is so complex that it becomes a buzz word itself. I categorize the related work into three layers: a cluster layer, a platform layer and an application layer, by their purposes. The three layers are illustrated as below:

Let’s examine each layer in this model and where the Kubernetes Platform as a Service fits in.

The Kubernetes Cluster Layer

At the bottom, the Kubernetes Cluster layer is the foundational layer. It focus on using self-hosted VMs or cloud resources to build a functional Kubernetes cluster and worker node groups. A functional cluster includes a highly available control plane, as well as scalable node groups that all communicate with the control plane. Cloud Service Providers like AWS and Azure provides managed Kubernetes service, which takes away the complexity (and flexibility as well) of managing control plane components such as etcd store and API server. The managed services also automatically provisions computing nodes and join them into the cluster. The cluster layer may also involve integration with of CNI and CSI, to ensure Pod-to-Pod communication and available storage classes.

Professionals working at this layer are infrastructure experts who understand networking, storage, as well as how to manage cloud resources or VMs, infrastructure as code. On a daily basis, they deal with VPCs/V-Nets, subnets, EBS/Azure Disk, File storage, EC2/Azure VMs, etc. When the team is doing a bad job at this layer, you might see symptoms like unresponsive cluster API, orphaned worker nodes, or kubectl failing to connect to cluster endpoint.

The tenants (applications) of the Kubernetes platform does not directly interact with this layer. If you decide to switch CSP vendor, this layer requires 100% re-engineering because the managed Kubernetes service by each CSP is different.

The Kubernetes Platform Layer

The Platform layer sits in the middle. When organization decides to adopt Kubernetes, they often underestimate the efforts required in this layer. This layer works on a functional cluster, without directly interacting with the underlying cloud resources. This layer involves any Kubernetes abstractions that do not creates tangible business value. Rather, this layer is an enabler. It allows the applications to deploy smoothly, evolve quickly, and more importantly, focus on the business.

Teams working on this layer needs to be Kubernetes experts. On a daily basis, they play with common CNCF toolings, such as Prometheus, ArgoCD, Istio, Cilium, Tekton, Open Policy Agent, etc. They are comfortable with Operators, Helm Charts, Ingress, etc. Inside of the Kubernetes cluster, they also manage the foundational services such as Event streaming (e.g. Kafka), PostgreSQL database (e.g. PostgreSQL), software-defined storage (e.g. Ceph), service mesh (e.g. Istio), Authentication (e.g. Keykloak) , etc. These services act as the infrastructure layer to the business workload. If the team is doing a bad job, you would see data loss with database, observability service not populating data, ingress does not process request, etc.

The tenants (application) share services in this layer. If you decide to switch CSP vendor. I estimate 80% of the work at this layer is portable, and 20% requires re-engineering. That is because each CSP offers different external resources, therefor the low level Kubernetes objects in this layer, such as storage classes, load balancers, supported CNIs are different. High level objects such as Kafka remains portable across platforms.

The Application Layer

The next layer at the top is application layer. Workloads in this layer are directly linked to the business value. The applications are very diverse. Most of the time, the release team is the main player at this layer. If the organization develops its own application, the software development team also work at this layer.

In terms of knowledge, the members of development team are experts in software engineering, and Software Development Life Cycle (SDLC), etc. On a daily basis, they deal with programming languages, product development, build and release. If they screw up their work, expect business errors, such as orders sent to wrong client, incorrect balance sheet, etc. This team has high visibility in the organization due to its direct link to business value.

This layer of work involves multiple tenants. Each tenant is isolated within their own namespace. When you switch CSP vendor, this layer should be readily portable with minimal effort.

It is also worth noting that, with solid platform and cluster layers, the team working at this layer do not write bespoke code for networking, observability, authentication and authorization, encryption and many other aspects not relevant to the core business. Once deployed, the application services are resilient, scale to demands, and cost efficient. This layer reaps the benefits of Kubernetes.

Kubernetes Platform as a Service

As the Kubernetes dust is still settling, a builder’s title may not always reflect which layer she or he focuses on. Today it is pretty common for infrastructure engineers to expand their role into the platform layer, or likewise, a software engineer to drill down to the platform layer.

The boundary between platform layer and cluster layer is clear. The cluster layer deals with underlying infrastructure, either in the cloud or on premise. They abstract away the complex infrastructure world from those working with the platform layer.

The boundary between platform layer and application layer is a little tricky to articulate. The application layer focuses on implementing the business logics. The platform layer takes care of the functions that are not part of business logic but essential to the business application. Take an HTTP request for example, application developer should not have to write code to terminate TLS (not part of business logic). They should only write the code to process the HTTP request (business logic). TLS termination is delegated to an Ingress, to be configured by platform builders.

The folks working at the Platform layer needs to interface with both sides. They provide Platform as a Service to the Application teams. However, their work appears mostly invisible in an organization. Their effort is oftentimes underestimated. There are several reasons for that. First, the platform layer does not directly create tangible business value. They are just someone else’s enabler. Second, their building blocks involve a lot of abstractions by Kubernetes API. Third, the idea of platform engineering is newly emerged. There hasn’t been a populous recognition of its value.

Red Hat OpenShift Container Platform

The platform team builds the platform with their choice of open-source tools. For clusters using OpenShift Kubernetes Engine, Red Hat introduces Open Shift container platform consisting of Red Hat’s opinionated (but validated) choice of toolings, for example:

OpenShift Service Mesh: Istio
OpenShift Streams: Apache Kafka
OpenShift GitOps: ArgoCD
OpenShift Container Platform Pipelines: Tekton
OpenShift Serverless: Knative
OpenShift Data Foundation: Ceph

Clients building their clusters with OpenShift Kubernetes Engine may build their own platform with the toolings in the OpenShift enterprise Kubernetes container platform. For more services, check out the documentation for OpenShift Container Platform. For customers with OpenShift Kubernetes Engine, their options to DIY platform are:

Entry-Level: Red Hat OpenShift Kubernetes Engine: Enterprise Kubernetes distribution on RHEL CoreOS
Mid-Level: Red Hat OpenShift Container Platform (RHOCP):
Plus-Level: Red Hat OpenShift Platform Plus: RHOCP + advanced cluster management, security, data management essentials, enterprise container registry

OpenShift runs the business model of Kubernetes PaaS.This is a unique business model that I do not find a matching competitor. Even if you choose to DIY your own platform, the Red Hat’s choices are still a great reference. The OpenShift enterprise Kubernetes container platform maps perfectly to the platform layer of the three-layer model, aiming to simplify the work in the platform layer.

Managed RedHat OpenShift

At first, the OpenShift container platform started as a value add-on to the Kubernetes Engine. Now it’s a separate product line in their business model. In the mean time, OpenShift partners with major CSPs, to develop the cloud service editions, including:

Red Hat OpenShift on AWS (ROSA)
Microsoft Azure Red Hat OpenShift (ARO)
Red Hat OpenShift Dedicated – on AWS and GCP
Red Hat OpenShift on IBM Cloud

These offerings are managed Kubernetes Platform as a Service in the cloud. Since RedHat is the only player in this model, we can refer to them as managed OpenShift services. In addition to an already-confusing world of Kubernetes platform portfolios, these offerings gives consumers even more options. On AWS for example, users have the following options:

Managed Platform: OpenShift Dedicated, managed by Red Hat
Managed Platform: Red Hat OpenShift Service on AWS (ROSA), managed by Red Hat and AWS
Self-built cluster: OpenShift Container Platform

This video discussed more details about these options, such as support model. It is also worth noting that these options tend to be much pricier than managed clusters such as EKS and AKS.

Since a Managed RedHat Platform makes it easy to deploy, let’s take ROSA as an example and create a cluster. To enable ROSA in AWS console, click on “Getting Started”. The next page ensures ROSA is enabled and checks other prerequisite such as meeting service quotas and creating ELB service-linked role, as show below:

Now, with an AWS account (and ROSA enabled), a RedHat account, and the rosa-cli utility, we can create a cluster with just a few commands. As a note, be wary of the cost and do not forget to delete the cluster afterwards.

Create a ROSA cluster

With the following set of commands, we can kick off cluster creation, using STS. We can bring our own VPC, so long as it meets certain prerequisites. I use the Terraform template in the vpc-base project, to create the underlying VPC. We’ll need the followings from this template:

The CIDR range of the VPC: as input with a default
The subnet Ids of the private subnet to place, printed in the output

The subnets are private subnets, because we want to provision the cluster with private node and private endpoint. When we use rosa CLI, we provide the CIDR and subnet IDs.

# start with AWS cli configured to the correct profile
rosa login # with redhat account and past token
rosa create account-roles --mode auto -y # this command creates the IAM roles ManagedOpenShift-*-Role, with RedHat account as trust entity
rosa verify permissions # optional
rosa verify quota # optional

export ROSA_CLUSTER_NAME="dhc" \
OPENSHIFT_VERSION=4.13.4 \
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) \
AWS_DEFAULT_REGION=us-east-1

rosa create cluster --sts --private \
 --cluster-name $ROSA_CLUSTER_NAME \
 --multi-az \
 --private-link \
 --region $AWS_DEFAULT_REGION \
 --version $OPENSHIFT_VERSION \
 --enable-autoscaling \
 --min-replicas 3 \
 --max-replicas 3 \
 --compute-machine-type m5.xlarge \
 --machine-cidr 147.206.0.0/16 \
 --subnet-ids subnet-052852a1fb4d7d2ad,subnet-06d8d40ae39d55c47,subnet-0f67ce08bc588012c

The CLI will pick up the correct VPC by CIDR, and prompt you to confirm creation of private cluster. After the command kicks off, it will wait for OIDC provider creation, and role creation. Then it uses a Terraform template to create the related resources including VPC. Use this command to check status:

rosa list clusters
rosa describe cluster -c dhc

When the second command displays the state of waiting (Waiting for OIDC configuration), we can create OIDC provider:

rosa create operator-roles -c $ROSA_CLUSTER_NAME --mode auto --yes
rosa create oidc-provider -c $ROSA_CLUSTER_NAME --mode auto --yes

Throughout the process, we can monitor the install log (terraform output) with:

rosa logs install -c dhc --watch

In the log, you might see errors with terminals connecting to the terraform backend, which doesn’t necessarily indicate a cluster creation error. Always check the cluster state until it reports success.

Kick the tires

Eventually the describe cluster command will show ready state. We can now create an admin user:

rosa create admin -c $ROSA_CLUSTER_NAME

The command above prints an oc command (OpenShift CLI, equivalent to kubectl) with password to log in. Let’s examine the cluster with oc. Because it is a private cluster, the endpoint is not available publicly. However, it is accessible from the Bastion host. Use the SSM Session Manager technique from my previous post to SSH to the Bastion Host, which should have oc installed. To install oc yourself, use HomeBrew on Mac. On Linux or Windows, log on to OpenShift console, go to Downloads on the left pannel and find it out under CLI tools.

The oc command may report insecure TLS on the login URL. Wait for a few minutes for the certificate to come off as safe. Once you run the oc command with password, it should return “Login successful” and then we can connect to the cluster:

$ oc get node # or kubectl get node
NAME STATUS ROLES AGE VERSION
ip-147-206-135-41.ec2.internal Ready,SchedulingDisabled infra,worker 3m5s v1.26.5+7d22122
ip-147-206-155-141.ec2.internal Ready control-plane,master 25m v1.26.5+7d22122
ip-147-206-156-81.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-164-21.ec2.internal Ready infra,worker 3m3s v1.26.5+7d22122
ip-147-206-179-90.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-191-118.ec2.internal Ready,SchedulingDisabled control-plane,master 26m v1.26.5+7d22122
ip-147-206-192-232.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-193-198.ec2.internal Ready infra,worker 3m20s v1.26.5+7d22122
ip-147-206-218-114.ec2.internal Ready control-plane,master 26m v1.26.5+7d22122

You can use oc the same way you’d use kubectl. Both works through SOCK5 proxy. In the meantime, log in to the RedHat console with your Red Hat credential, you can see the cluster in Ready state as well:

The rosa create admin command creates a htpasswd type (username-password) of identity provider (IdP) with a user named cluster-admin and a preset password. In real life however, we often configure third party IdP with OIDC integration. I’ll have to leave this to the next blog post.

We shall see the nodes as EC2 instances from AWS console as well. Note that there are three roles: control-plane, worker and infra. The infra nodes are for infrastructure services. These services (Ingress Controller, GitOps, Pipeliens) are the ones in the platform player as we discussed above.

There are many customizations you can make in this installation process and I’d have to defer to the ROSA documentation. To clean up, use the following ROSA command:

rosa remove cluster -c dhc

The output also gives you the command to delete operator roles and OIDC provider, for example:

rosa delete operator-roles -c 23o4u3j98tqmlbtjo612opb7a4bbim5f --mode auto --yes
rosa delete oidc-provider -c 23o4u3j98tqmlbtjo612opb7a4bbim5f --mode auto --yes

Then we can destroy the VPCs using terraform.

ROSA with HCP

Update Oct 2023:

The deployment above provisioned a few nodes for control plane, which add to the overall time to provision a cluster. In Aug 2023, there is a new option Hosted Control Plane (HCP) that came to allow users to provision a hosted control plane. This results in cost savings and shorter time to provision a cluster. Here is a table of comparison between the ROSA with HCP and ROSA classic.

Final words

In this post, I discussed the three-layer model and pointed out that platform layer isn’t as visible as the other two. I also experimented ROSA as a turn-key Kubernetes platform with its opinionated stack of services.

Some misinformed organizations even skip the entire platform layer in their estimate of effort. They build a cluster, ran a hello-world service and assumes they can start putting applications on the Kubernetes cluster. There are also customers who purchased the entire Managed OpenShift platform but only use it as a cluster. Yikes!

The concept of Kubernetes platform, or generally platform engineering is still spreading. The consulting team that I worked in full-time last year re-branded itself as platform engineering. Marketings are pushing it. Builders are doing it. We’ll keep an eye, on whether customers are buying it.

Istio Operation Gotchas

Sat, 19 Mar 2022 11:09:00 -0400

In this post I discuss a few aspects when putting istio in operation.

Installation

Istio installation can be confusing, due to architectural and guideline changes as well as renaming of operator CRDs since its release, and especially since 2020. This left lots of information outdated on the web, adding to Istio’s perceived complexity.

Currently, the recommended installation methods are istioctl or Helm. Using Istio Operator manifest (with out istioctl) is discouraged. As to the Helm chart installer, it was once deprecated (around early 2020), but was later re-introduced in the fall of 2021. This back-and-forth had caused some aversion. As of Nov 2021, their re-introduced Helm charts version dropped alpha tag. The Helm repo consists of separate Helm charts, for Istio CRD (base), control plane (istiod), each gateway and CNI respectively. A typical deployment therefore requires multiple Helm Releases (example here). Istio document still considers Helm support as alpha, so I assume the most reliable method to install Istio is istioctl.

The istioctl utility can be used with many options to customize Istio install. For example, we can supply a YAML declaration input (to -f switch) to customize installation behaviours. The YAML file declares a CRD. Two different CRDs have been used: IstioOperator and IstioControlPlane. According to this blog and this post, since Istio 1.5 in early 2020, we’re supposed to IstioOperator CRD exclusively. The IstioControlPlane CRD is left only for legacy support. As stated in the current documentation:

The istioctl command supports the full IstioOperator API via command-line options for individual settings or for passing a yaml file containing an IstioOperator custom resource (CR).

With IstioOperator CRD, we still have a number of options to tweak the install behaviours. Here is a summary of potential options:

use IstioOperator API via IstioOperator CRD (without using “values” or overlay fields)
specify an attribute value in argument, including a pre-built profile e.g. –set meshConfig.accessLogFile=/dev/stdout, –set profile=demo
use K8sObjectOverlay by using “k8s/overlays/patches” field in IstioOperatorCRD
use Helm API by using “values” field in IstioOperatorCRD

The key-value specified in –set switch overrides the same key-value supplied in the IstioOperator CRD. So option 2 overrides option 1. The value for profile can also be empty if you’d rather start from scratch. However too many –set switches makes the command wordy so we should build our own IstioOperator CRD

For option 4, the document hyper-links Helm API to a section from version istio 1.4, and I appears to exist only for legacy (pre-2020 Helm support) compatibility. Option 3 (K8sObjectOverlay) would be helpful when a field cannot be conveniently customized with option 1 and we have to patch the object like in Kustomization.

So the most practical approach is IstioOperator CRD for per-component customization, potentially with K8sObjectOverlay. No matter which option, istioctl compiles the installation manifest before applying it against Kubernetes API. This manifest can be previewed using “istioctl manifest” command, so that you can take a look before installation.

Thank you Isito for so much confusion just to land on a working installation method. Below is the content of az-istio-operator.yaml file that I use for my installation:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
 name: istio-install-customization
spec:
 profile: default
 hub: docker.io/istio
 tag: 1.13.1
 revision: 1-13-1
 namespace: istio-system
 meshConfig:
 accessLogFile: /dev/stdout
 outboundTrafficPolicy:
 mode: "REGISTRY_ONLY"
 components:
 pilot:
 k8s:
 hpaSpec:
 maxReplicas: 7
 minReplicas: 1
 nodeSelector:
 beta.kubernetes.io/os: linux
 ingressGateways:
 - name: istio-ingressgateway
 #namespace: istio-ingress
 enabled: true
 label:
 istio: ingressgateway
 k8s:
 hpaSpec:
 maxReplicas: 11
 minReplicas: 1
 serviceAnnotations:
 service.beta.kubernetes.io/azure-load-balancer-internal: "true"
 service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "my-lb-subnet"
 egressGateways:
 - name: istio-egressgateway
 enabled: true

I can preview the install, run the install and validate installation status:

$ istioctl manifest generate -f az-istio-operator.yaml
$ istioctl install -f az-istio-operator.yaml -y --verify
$ istioctl verify-install -f az-istio-operator.yaml
$ kubectl -n istio-system get IstioOperator installed-state-istio-install-customization-1-13-1 -o yaml | less

Note that in the IstioOperator declaration I marked the revision. This is helpful when I run multiple versions of control plane (e.g. during upgrade). I can check revisions of istiod with:

$ istioctl x revision list

We can delete installed istio components

$ istioctl x uninstall -f az-istio-operator.yaml
$ istioctl x uninstall --revision 1-11-5

In practice, it is helpful to use separate operators each for a different component (pilot, ingressGateways, egressGateways). This makes maintenance and upgrade easier.

Debugging

The documentation has a page for common problems that one needs to be familiar with. It covers not only problems, but also steps to troubleshoot each kind of problem (e.g. authorization policy).

Although istioctl is pretty confusing as an installation tool, it is a good utility for many troubleshooting activities. We should probably add its path to PATH environment variable and add the export command (e.g. export PATH=”$HOME/istio/bin:$PATH“) to .zshrc or .bashrc. Istioctl has a few useful subcommands, some of which are only available as experimental and therefore needs to be following an x. Some common commands are given below:

To analyze Istio problems:

$ istioctl analyze -n istio-system

To look at proxy configuration of an Envoy instance at different levels, use proxy-config sub-command or pc for shorthand:

$ istioctl proxy-config <clusters|listeners|routes|endpoints|bootstrap|log|secret|stats>
$ istioctl pc cluster deploy/istio-ingressgateway -n istio-system # see what Envoy cluster an ingress gateway knows about
$ istioctl proxy-config log deploy/httpbin --level "rbac:debug"
$ istioctl pc log <pod_name> -n <namespace> --level connection:debug
$ istioctl pc secret -n istio-system deploy/istio-ingressgateway # check certificates loaded to a gateway
$ istioctl proxy-config listeners deploy/istio-ingressgateway -n istio-system # query envoy listener configuration
$ istioctl pc routes deploy/istio-ingressgateway -n istio-system --name http.8080 # query envoy route configuration

The last two commands set logging level to the specified workload. If we want to set logging level at mesh level, we can use these commands:

$ istioctl admin log --level authorization:debug
$ istioctl admin log

To look at synchronization status of each envoy in the mesh, use proxy-status sub-command, or ps for shorthand:

$ istioctl ps # ensure data plane is in sync

If an item for a workload shows STALE instead of SYNCED, it means that the configuration has not been pushed from control plane to that instance of Envoy proxy. Check if the Istio configuration change is valid. If the system is newly installed and there is no ingress or egress gateway resources declared, the RDS column for ingress or egress may show “NOT SENT”. The far right column displays the version of istiod connected.

To describe applied istio config:

$ istioctl describe <pod|service>
$ istioctl describe po workload1 -n my-workload # detect misconfigurations on workload

To view dashboard:

$ istioctl dashboard <envoy|grafana|prometheus>

To check authorization policy on a Pod,

$ istioctl x authz check mypod -n workload

To validate istio configuration in a file:

$ istioctl validate -f resource_authorization_policy.yaml

Sometimes we need to turn on access logging just on the envoy proxy on the Gateway Pod. In that case, we will need to apply Istio’s Envoy filter object. This filter is applied to Pods labelled as gateway. It patches the existing filter chain with the additional defined in the manifest:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
 name: hcm-gw-access-log
 namespace: istio-system
spec:
 workloadSelector:
 labels:
 istio: ingressgateway
 configPatches:
 - applyTo: NETWORK_FILTER
 match:
 context: GATEWAY
 listener:
 filterChain:
 sni: demo.digihunch.com
 filter:
 name: "envoy.filters.network.http_connection_manager"
 patch:
 operation: MERGE
 value:
 typed_config:
 "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
 access_log:
 - name: envoy.access_loggers.file
 typed_config:
 "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
 path: /dev/stdout
 format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"

The object above enables access logging on the Gateway Pods only, without impacting other Envoy proxies.

Multi-tenancy

In large enterprises, Istio is usually installed by a platform/operation team. The entire platform is shared by multiple application teams. Istio should not be seen as a responsibility of a single party. It is necessary to break down the Istio CRDs and define the responsibility of each CRD. For example:

Gateways are placed in the istio-system namespace (or a dedicated istio-ingress namespace in some case). Platform team can decide whether a Gateway is considered a shared infrastructure, or each tenant (application team) uses their own Gateway. In the servers/hosts field of Gateway declaration, add namespace before host to suggest that the routing behaviour of that host must be defined in a certain namespace.
Virtual Services can be managed in two models as well. They can be managed individually and are all placed in each tenant namespace. Alternatively in a shared responsibility model, Virtual Service can be created in istio-system namespace, and the processing of each match can be delegated to a Virtual Service in each tenant namespace, using the Delegate feature of Virtual Service. Another field that can help with multi-tenancy is the “gateways” field, you can specify a value of mesh to indicate the virtual service is available to the entire mesh.
Destination Rules are usually placed in each Tenant’s workspace
Peer Authentication also depends on the configuration. If a mesh level configuration is enforced, it is easier for the Platform Team to manage it and this should be the setup for new clusters. If for historical reasons Peer Authentication is enforced per tenant namespace, it can be delegated to each application team. It can make communication troubleshooting more complex.
Control Plane and observability workloads such as Kiali are usually the responsibility of Platform team.
Request Authentication and Authorization Policy should be the responsibility of individual application team.

In a multi-tenancy management model, Istio related resource should be subject to admission control policy, as well as scrutiny by the security and platform teams. Google Anthos has a good page on the constraint templates for Istio resources.

As to restricting traffic between namespaces, apart from the measures from this previous post, we can also use Sidecar CRD to restrict outbound traffic. As its documentation states:

Sidecar describes the configuration of the sidecar proxy that mediates inbound and outbound communication to the workload instance it is attached to. By default, Istio will program all sidecar proxies in the mesh with the necessary configuration required to reach every workload instance in the mesh, as well as accept traffic on all the ports associated with the workload. The Sidecar configuration provides a way to fine tune the set of ports, protocols that the proxy will accept when forwarding traffic to and from the workload. In addition, it is possible to restrict the set of services that the proxy can reach when forwarding outbound traffic from workload instances.

The documentation page also includes two examples. The first is a sidecar at the mesh level that restricts outbound traffic to the same namespace that the sidecar is in, and the istio-system namespace:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
 name: default
 namespace: istio-config
spec:
 egress:
 - hosts:
 - "./*"
 - "istio-system/*"

The second example overrides the mesh level default above, and allows egress traffic to three specified namespaces:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
 name: default
 namespace: prod-us1
spec:
 egress:
 - hosts:
 - "prod-us1/*"
 - "prod-apis/*"
 - "istio-system/*"

As to inbound control, we can expose a Virtual Service to other namespaces by using the exportTo field to specify which other namespaces the Virtual Service should be exported to. If no namespaces are specified then the virtual service is exported to all namespaces by default.

Sometimes we want a virtual services to expose to both outside of the mesh via Ingress, and within the mesh, and we hope to use the same hostname. For this requirement, we can use the ServiceEntry CRD. ServiceEntry enables adding additional entries into Istio’s internal service registry.

Service Proxy – from Nginx to Envoy

Wed, 09 Mar 2022 21:36:00 -0400

Update (Nov 20, 2022): 1. Envoy’s configuration schema can be hard to get used to. It is lacking examples because the documentation is mostly generated. Use its examples directory to find real-life configuration examples. 2. the Envoy implementation in the example project has been reverted in favour of Nginx.

Envoy proxy is the underlying technology for Istio, as well as a number of other service mesh products, such as AppMesh (AWS), Consul (Hashicorp) and OpenServiceMesh (Azure). Most of the capabilities of Isito is ultimately provided by Envoy proxy. Envoy has a page outlining its differences with similar technologies. I decided to take a look into Envoy by replacing Nginx with it.

Rate limiting and Circuit Breaker

In most SDLC, it is application developers that create backend APIs or server applications. Most developers specializes in application features, and cannot fathom all the nuances with the TCP/IP network stack. Nginx allows them to push network concerns (non-business features) to a dedicated proxy to handle the dynamics in network connection. Nginx can be configured as both a reverse proxy (handling incoming connection on behalf of the process) and a forward proxy (handling outgoing connection on behalf of the process). This is the prototype of sidecar pattern, an important idea behind service mesh. For example, when a sudden increase in connection to the server-side application, the server process could be either unresponsive (refer to “the queuing knee“, and Little’s Law), or OOM killed. When such interruptions are not automatically recovered, a downtime is caused. This traditionally requires some congestion control strategy for TCP/IP queue but two features provided by a network proxy can help circumvent this situation: rate limiting, and circuit breaking. Rate limiting keeps more requests above threshold from entering the queue. Circuit breaker releases downstream pressure by cutting out existing in-queue request. Nginx added both over the years but Circuit breaker still remains a premium feature exclusive to Nginx Plus. Envoy on the other hand has them free when it was launched.

Envoy also supports other advanced traffic management such as traffic shaping, and mirroring. It is on top of those features that Istio introduces its own abstraction such as virtual service, destination rules to its users. In that sense, we can think of Istio as a configurator (control plane) for Envoy proxy (data plane), similar to Ansible to Nginx proxy instances.

Dynamic Configuration via API

I used Nginx previously with traditional environment and loved its flexibility. As the system grows, I started to feel the pain of management overhead. With one of the production system, there were 25 + instances of Nginx each running on a VM and I managed configuration files with Ansible. Ansible pushes out configuration files and triggers a reload from each Nginx instance. In the cloud-native era where Pods are ephemeral, this kind of overhead would snowball to an unmanageable level. Envoy was designed for cloud-native applications, with all these kinds of problems in mind. Envoy has dynamic configuration. The majority of the configurations can be pulled from xDS API, or file system. Updating configuration drains connections gracefully without runtime having to reload the file. The idea of centrally managing Nginx instances with Ansible, also evolved into the concept of control plane.

TLS origination

In the Orthweb project, I used Nginx to proxy TLS and HTTP traffic, and performed TLS termination on both ports. This is know as TLS offloading. The traffic between the proxy and the upstream service takes place in the clear, even though they do not travel across different network interfaces in most cases. For a true end-to-end encryption, it is helpful to also encrypt the traffic between proxy and upstream server. This requires the capability of securing TCP traffic to upstream server. With Nginx, the ability to secure HTTP traffic to upstream server is offered in open-source. The ability to secure TCP traffic is available in Nginx Plus, or with self-compiled binary. In Envoy, both are available using the UpstreamTlsContext configuration.

More pros

In another post, I also discussed Nginx as a LDAP proxy to front services such as Kibana and Nifi. It requires a proxy service (ldap-auth in this case), to defer auth to third party. Envoy has this capability using a filter with extension for external authorization. Istio also exposes this capability, an enabler for the configuration proposed in my previous post. Envoy also uses WebAssembly for its extensibility.

Another useful feature is protocol detection. It can use filters to detect protocol (TLS or regular TCP) and route traffic to predefined destination.

Performance wise, this benchmark from 2018 ran a comparison among the popular options where Envoy leads by a margin.

Observability (logging, metrics and tracing) are well supported in Envoy. User can configure format of logs that takes effect immediately. There are many metrics that works with Prometheus and they are expandable using filters. On the tracing side, Envoy supports integration with jaeger, zipkin and datadog.

Basics of Envoy

The configuration of Envoy is more involving. There is an Envoy course by Tetrate, as well as two blog entries for envoy 101: Envoy as gateway proxy and File-based dynamic configuration. Another good way to get started is the Sandboxes projects, which covers a number of different areas of configuration. The admin port (by default at port 9901. Istio’s default is 15000) provides helpful information. If we need to turn on debug on some features, we can do so with curl:

curl -X POST http://localhost:9901/logging?client=debug

Stats are exposed at the same port:

curl -X GET http://localhost:9901/stats

When packets are received at a listener, the is first processed by listener filters. Then, depending on filter match, one or more network filter chains will further process the packet, including further actions, as illustrated below:

Envoy supports dynamic configuration, which uses a set of discovery services (xDS) APIs. Some of the important xDS APIs include:

LDS (Listener Discovery Service) – allows you to add listeners dynamically while Envoy is running
RDS (Route Discovery Service) – allows you to dynamically update routes for HTTP connection managers
CDS (Cluster Discovery Service) – allows you to update cluster definitions dynamically
EDS (Endpoint Discovery Service) – allows you to add or remove endpoints dynamically
Secret DS

The relation can be illustrated in this diagram below:

This post from Tetrate has more examples.

Nginx to Envoy

For the advantages of Envoy, I decided to migrate from Nginx to Envoy on my Orthweb project. Using Envoy as service proxy is not where Envoy is mostly used (as sidecar), but it is how Envoy was originally used at Lyft to replace ELB in 2015.

The original Nginx configuration was referenced in this old blog post. The Envoy setup also covers both TCP (DICOM) and HTTP (HTTPS) traffic. For HTTP traffic, it also encrypts the traffic to upstream. Below is what it looks like:

admin:
 address:
 socket_address: { address: 0.0.0.0, port_value: 9901 }
static_resources:
 listeners:
 - name: https_listener
 address:
 socket_address:
 address: 0.0.0.0
 port_value: 443
 filter_chains:
 - filters:
 - name: envoy.filters.network.http_connection_manager
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
 codec_type: AUTO
 stat_prefix: ingress_http
 route_config:
 name: local_route
 virtual_hosts:
 - name: app
 domains:
 - "*"
 routes:
 - match:
 prefix: "/"
 route:
 cluster: service-https
 http_filters:
 - name: envoy.filters.http.router
 transport_socket:
 name: envoy.transport_sockets.tls
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
 common_tls_context:
 tls_certificates:
 - certificate_chain: {"filename": "/etc/ssl/certs/site.pem"}
 private_key: {"filename": "/etc/ssl/certs/site.pem"}
 - name: dicomtls_listener
 address:
 socket_address:
 address: 0.0.0.0
 port_value: 11112
 filter_chains:
 - filters:
 - name: envoy.filters.network.tcp_proxy
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
 stat_prefix: downstream_cx_total
 cluster: service-dicomtls
 transport_socket:
 name: envoy.transport_sockets.tls
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
 common_tls_context:
 tls_certificates:
 - certificate_chain: {"filename": "/etc/ssl/certs/site.pem"}
 private_key: {"filename": "/etc/ssl/certs/site.pem"}
 validation_context:
 allow_expired_certificate: true
 trusted_ca: {"filename": "/etc/ssl/certs/site.pem"}
 require_client_certificate: false
 clusters:
 - name: service-https
 type: STRICT_DNS
 lb_policy: ROUND_ROBIN
 load_assignment:
 cluster_name: service-https
 endpoints:
 - lb_endpoints:
 - endpoint:
 address:
 socket_address:
 address: orthanc-backend
 port_value: 8042
 transport_socket:
 name: envoy.transport_sockets.tls
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
 - name: service-dicomtls
 type: STRICT_DNS
 lb_policy: ROUND_ROBIN
 load_assignment:
 cluster_name: service-dicomtls
 endpoints:
 - lb_endpoints:
 - endpoint:
 address:
 socket_address:
 address: orthanc-backend
 port_value: 4242
layered_runtime:
 layers:
 - name: static_layer_0
 static_layer:
 envoy:
 resource_limits:
 listener:
 https_listener:
 connection_limit: 1000
 overload:
 global_downstream_max_connections: 5000

From Nginx to Envoy, to achieve nearly the same functionalities, it takes 100 lines of configuration instead of less than 30. The configuration also appears more abstract, which is one of the cons of Envoy to consider before the migration.

Update from Jan 2025 – To align with what most uses in Orthanc community, the project switched back to using Nginx. The envoy configuration is kept here.

Istio External Authorization via OIDC

Fri, 25 Feb 2022 01:29:00 -0400

Istio service mesh allows application developers to offload non-core features to infrastructure layer. We explored authentication and authorization with Istio in a basic lab. In this post we continue to explore its capabilities with OIDC integration. This capability is made available thanks to the CUSTOM action in authorization policy, supported since the release of 1.9. It enables any workload on Istio to integrate with an external IAM solution.

The rest of this post, provides the step-by-step instruction to configure OIDC integration, based on Istio’s External Authorization use case. My work is influenced by two blog posts from jetstack and elastisys on similar topic, with my own additions, simplifications and clarifications. In this lab I use my own DNS hostname demo1.digihunch.com and to follow along you need to bring your own hostname as well. The prerequisites of this lab is summarized as below:

An identity provider: we use Google in the lab but it can be anything with OIDC capabilities (e.g. Azure AD, Facebook). Read my old post for more details on OIDC.
A public IP address for Cluster’s Ingress service: Since we use Google as identity provider on the Internet, we need our service to be exposed on the Internet because the OIDC integration involves a two-way HTTP redirect.
Public DNS resolution to the service’s public IP: during authentication, the identity provider will call back with JWT. So we need our hostname demo1.digihunch.com resolved to the service’s public IP address. This is also required in other scenarios. For example, cert-manager needs to automatically configure X.509 certificate using Let’s Encrypt.
An application workload: this is the tenant workload without its own IAM capability. We use a dummy workload base on a hello-world image, for simplicity.

The files required in this lab is in istio-oidc repo under the envoy directory. It is also important to understand the Authentication Code flow in OIDC authorization code flow to make sense of the integration between OIDC provider and Istio on our platform. For example, the client application needs a one-time registration with identity provider. Let’s start with this configuration.

Configure Identity Provider

We need a Google account with a GCP project and log on to APIs & Services to register our application. At the console, from the left side bar, click on “OAuth consent screen” and create an App. Provide the followings:

Step 1. OAuth consent screen. Under App Information, provide App name and User support email.
Step 1. OAuth consent screen. Under App domain, provide Application home page
Step 1. OAuth consent screen. Under Authorized domains, provide an Authorized domain.
Step 3. Test users. Add the emails for Test users.

The summary page looks like this:

OAuth Consent Screen

Then on the sidebar, click on Credentials. We create new credentials for OAuth 2.0 Client ID. Select Web application and add “https://demo1.digihunch.com/oauth2/callback” as an Authorized redirect URI. The credential created includes a Client ID and a Client secret, which can be downloaded as a JSON file. Here is my screen:

API Credential

Note that the redirect URIs cannot be localhost. Because the browser will consume the redirect URI and it has no clue where localhost is. It must resolve to the public IP of the ingress.

We need to keep the Client ID and Client secret for later use. According to Google’s documentation, there is a discovery document from which we can understand the keys in their Open ID configuration. We need the value of two keys (issuer and jwks_url) to use later in the configuration.

Configure Infrastructure Services

My real-quicK-cluster project has a few different ways to quickly bring up a cluster. For this lab, I use Azure CLI command to create a simple, three-node cluster:

$ az aks create \
 -g AutomationTest \
 -n orthCluster \
 --node-count 3 \
 --enable-addons monitoring \
 --generate-ssh-keys \
 --vm-set-type VirtualMachineScaleSets \
 --network-plugin azure \
 --network-policy calico \
 --tags Owner=MyOwner

Then we configure kubeconfig credential to connect to the cluster, so that kubectl and helm CLI commands to connect to the newly created cluster.

$ az aks get-credentials --resource-group AutomationTest --name orthCluster

Then we will install the infrastructure services and workload. We can start with loading environment variables for use later:

OIDC_DISCOVERY=$(curl "https://accounts.google.com/.well-known/openid-configuration")
OIDC_ISSUER_URL=$(echo $OIDC_DISCOVERY | jq -r .issuer)
OIDC_JWKS_URI=$(echo $OIDC_DISCOVERY | jq -r .jwks_uri) 
COOKIE_SECRET=$(openssl rand -base64 32 | tr -- '+/' '-_')
WEB_HOST="demo1.digihunch.com"
CLIENT_ID="ThisIsTheClientIDFromGoogle"
CLIENT_SECRET="ThisIsTheClientSecretFromGoogle"

For WEB_HOST, CLIENT_ID, CLIENT_SECRET, you need to bring your own values. The COOKIE_SECRET value is randomly generated. The variables OIDC_ISSUER_URL and OIDC_JWKS_URI are parsed from Google OpenID configuration and they should remain static. We can validate the value with echo commands. Also, to use helm v3 later, we need to add the repositories we need. These steps can be skipped if they have been performed in the client environment:

$ echo $OIDC_ISSUER_URL
https://accounts.google.com
$ echo $OIDC_JWKS_URI
https://www.googleapis.com/oauth2/v3/certs
$ helm repo add jetstack https://charts.jetstack.io
$ helm repo add istio https://istio-release.storage.googleapis.com/charts
$ helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
$ helm repo update

Now, with Helm ready, we can install Cert Manager, Istio CRD, Control Plane, Gateways as well as OAuth2-Proxy using Helm. We run each of the following commands from the envoy directory, where the required files are stored:

$ helm install cert-manager jetstack/cert-manager \
 --namespace cert-manager \
 --create-namespace \
 --version v1.7.1 \
 --set installCRDs=true
$ helm install -n istio-system istio-base istio/base --create-namespace
$ helm -n istio-system install istiod istio/istiod -f istiod-values.yaml --wait
$ helm -n istio-system install istio-ingress istio/gateway -f ingress-gateway-values.yaml
$ kubectl -n istio-system get po
$ kubectl create ns oauth2-proxy && kubectl label ns oauth2-proxy istio-injection=enabled
$ helm install -n oauth2-proxy \
 --version 6.0.1 \
 --values oauth2-proxy-values.yaml \
 --set config.clientID=$CLIENT_ID \
 --set config.clientSecret=$CLIENT_SECRET \
 --set config.cookieSecret=$COOKIE_SECRET \
 --set extraArgs.oidc-issuer-url=$OIDC_ISSUER_URL \
 --set extraArgs.whitelist-domain=$WEB_HOST \
 oauth2-proxy oauth2-proxy/oauth2-proxy
$ kubectl -n oauth2-proxy get pods -l app=oauth2-proxy

Note that the istiod installation step uses a value file that includes extensionProviders. This is where we tell Istio to connect to external authorization provider:

meshConfig:
 accessLogFile: /dev/stdout
 extensionProviders:
 - name: oauth2-proxy
 envoyExtAuthzHttp:
 service: oauth2-proxy.oauth2-proxy.svc.cluster.local
 port: 4180
 includeRequestHeadersInCheck:
 - cookie
 headersToUpstreamOnAllow:
 - authorization
 headersToDownstreamOnDeny:
 - set-cookie

When we install istio ingress gateway, we need to expose port 80 as well as 443. Port 80 is used in ACME protocol for certificate configuration. We also installed oauth2-proxy with some configurations from oauth2-proxy-values.yaml as well as some argument set imperatively.

Once configuration is successful, we should be able to confirm that oauth2-proxy service is running. We should also be able to tell the External IP address of the Ingress:

$ kubectl -n oauth2-proxy get svc
$ kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Now we can go to our DNS configuration portal, to populate the DNS A-record for demo1.digihunch.com with this IP address:

Add DNS A-record

It may take several minutes to a couple of hours, for the A-record update/creation to take effect, depending on the TTL. Sit tight and query DNS with nslookup command, until it is set. In the next step, we will need successful DNS resolution for certificate configuration.

Install workload and configure Certificate

Now we install the workload, along with the necessary constructs such as Gateway CRD, Virtual Service, and certificates. We can use kubectl kustomize command to preview the changes, then apply the changes with -k:

$ kubectl kustomize demo
$ kubectl apply -k demo

When you have your own DNS hostname, modify the kustomization.yaml accordingly before applying. This command uses Kustomization to apply numerous YAML manifests in the demo directory, including the following activities:

Create a Namespace named demo
Create Deployment and Service
Set up PeerAuthentication for the mesh
Configure Ingress class, ClusterIssuer and Certificate using cert-manager
Configure Virtual Service and Gateway

This configures everything we need and we need to verify from several aspects. The demo Service is a ClusterIP service exposed within the cluster at port 80. Then we check the status of certificate:

$ kubectl -n istio-system get certificate
$ kubectl -n istio-system describe certificate demo

The column of READY should have a value of True. Check the logs from cert manager Pods if that’s not the case. Common reasons include:

The automatic ACME validation is just not ready
The automatic ACME validation is still waiting for DNS resolution
The Istio ingress gateway port 80 is not open for ACME validation
The let’s encrypt server applies rate limiting

Note that for the ACME server, we can use staging server’s URL or productions. The former provisions a certificate that may come off as insecure as the CA is not fully trusted by browser. The latter doesn’t have this issue, but the server applies rate limiting. For more details of how validation with ACME work, check out my previous post.

Once the certificate is applied, our website is open to any visitor with HTTPS access:

Then we will use Request Authentication and Authorization Policy to tighten up the access by requiring visiting user to log in with Google identity.

RequestAuthentication and AuthorizationPolicy

We configure a CUSTOM action in the AuthorizationPolicy, and specified the provider by name oauth2-proxy. The YAML manifests for RequestAuthentication and AuthorizationPolicy looks like this:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
 name: istio-ingressgateway
 namespace: istio-system
spec:
 jwtRules:
 - issuer: https://accounts.google.com
 jwksUri: https://www.googleapis.com/oauth2/v3/certs
 selector:
 matchLabels:
 app: istio-ingressgateway
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: istio-ingressgateway
 namespace: istio-system
spec:
 action: CUSTOM
 provider:
 name: oauth2-proxy
 rules:
 - {}
 selector:
 matchLabels:
 app: istio-ingressgateway

Note that we cannot apply Request Authentication and Authorization Policy before we confirm that the Certificate has been created. This is because when letsencrypt use HTTP resolver to provision certificate with ACME protocol, the request from letsencrypt should get to the validating URI without being asked to provide a Google identity. So do not perform this step yet before confirming the previous step:

$ kubectl apply -f oidc-auth.yaml

Now launch your browser in incognito mode and browse to our URL (https://demo1.digihunch.com) and you will be re-redirected to log on to Google (accounts.google.com). Once logged on, you will have access to the site.

If you run into errors, check the oauth2-proxy log from Pod stdout, which should give a reason of 4xx errors. If oauth2-proxy log indicates no activity, confirm if the request has been forwarded to the proxy. Check the service object of the proxy and make sure it is exposed to the correct port, as indicated in the meshConfig. To check mesh config, examine the configmap named istio in the namespace of istio-system. To check if Istio’s authorization is unable to speak with oatuh2-proxy, inspect the log of istiod Pod.

To clean up the lab, remove the app registration from Google, and then delete the cluster from Azure:

$ az aks delete -g AutomationTest -n orthCluster

In this lab we used a very open AuthorizationPolicy. We can polish it up with more conditions such as:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: istio-ingressgateway
 namespace: istio-system
spec:
 action: CUSTOM
 provider:
 name: oauth2-proxy
 rules:
 - when:
 - key: request.auth.audiences
 values:
 - $CLIENT_ID
 selector:
 matchLabels:
 app: istio-ingressgateway

Under the rules section, we can explore the claims from the JWT and make creative use of it to achieve our specific goals for IAM.

Bottom line

In this lab, we registered our app with Google and obtained client ID and secret for it to identify our application. We configured oauth2-proxy as our IAM application, carrying the credential validated by Google. We also configured Istio to delegate authorization to oauth2-proxy as external authorization provide, making up an entire OIDC integration.

The third party service (IdP) we integrate with is GCP, which natively supports OIDC protocol. However, if the third party IdP does not support OIDC natively, such as Active Directory via LDAP, then we would need one more proxy to sit between OAuth2 and the external IdP service. One good choice is the Dex connector. Another more complex yet powerful alternative connector is Key Cloak. Consider Dex or KeyCloak as an OIDC proxy, when:

The external IdP does not support OIDC; or
We choose not to use the OIDC capability of the external IdP because we want to managed OIDC centrally in our own proxy
The external identity provider does support OIDC, but the configuration is managed by a different team (e.g. IAM team) other than the platform team

AuthorizationPolicy is the key piece in this integration, and it is executed at the http filter in envoy sidecar proxy. It can be done with alternatives to OAuth2-Proxy such as the authservice project. A slightly different approach to implement OIDC integration is to use leverage extensibility of WebAssembly (a sandboxing technology to extend Envoy), which is suggested in this example for WasmPlugin. However, Wasm is still considered experimental.

Istio Lab – Authentication and Authorization

Sun, 13 Feb 2022 13:21:13 -0400

My previous blog discussed as service mesh what Istio can offer in terms of authentication and authorization capabilities. Istio can authenticate an incoming HTTP request, ensuring the JWT issued has not been tampered somewhere in the middle. The fields in the JWT allows for more flexibilities at the point of authorization. This combination allows Istio to integrate with identity providers that can issue JWT.

We’ve also discussed how JWT works, and pointed out that the two key element of request authentication is the JWT (payload signed with private key) itself as well as the JWK (carrying public key). In this post, we will test it in a lab. To start with this lab, we need a test cluster (e.g. Minikube) with Istio installed.

Preparation

I have not find a native Bash way to produce JWT. We will do that with python packages python_jwt and jwcrypto in Python3. Let’s install the modules and import them in Python environment.

$ python3 -m pip install python_jwt jwcrypto datetime
$ python3
>>> import python_jwt as jwt, jwcrypto.jwk as jwk, datetime

Now we’re in the Python3 shell with needed modules loaded. We can take the following steps to produce the JWT as well as the JWK:

RSAkey = jwk.JWK.generate(kty='RSA', size=2048)
private_key = RSAkey.export_private()
public_key = RSAkey.export_public()
raw_payload = {'iss':'digihunch.com','sub':'DIGIHUNCH','role':'reader','permission':'read'}

## use private key to generate jwt token. HTTP request will bear this token 
jwt_token = jwt.generate_jwt(raw_payload, jwk.JWK.from_json(private_key), 'RS256', datetime.timedelta(minutes=50))
print(jwt_token)

## The JWKS keeps public key and is referenced by Istio RequestAuthentication object
jwks='{"keys":['+public_key+']}'
print(jwks)

# Helpful command to print the key in PEM format:
## RSAkey.export_to_pem(private_key=False)
# Helpful command to verify JWT token:
## header, claims = jwt.verify_jwt(jwt_token, jwk.JWK.from_json(public_key), ['RS256'])

To complete this lab, we need the values of jwt_token and jwks.

Next in the preparation is a local cluster, istio with metallb installed, which is covered in a previous post. We should be able to get the ingress IP address:

$ export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ echo $INGRESS_HOST

We create our own namespace and use the httpbin application:

$ kubectl create ns web 
$ kubectl label namespace web istio-injection=enabled
$ kubectl apply -n web -f httpbin.yaml
$ kubectl apply -n web -f httpbin-gateway.yaml
$ curl -I -XGET $INGRESS_HOST/headers

The curl command above (without any token) should return HTTP 200 code, indicating that no authorization token is needed to connect to the service. Note that the Pod for httpbin has the label app=httpbin which will be used in the request authentication. Also note that traffic is served over a named port called http in the Service object for http, which will implicitly enable HTTP based conditions for authorization policies we will build later.

Request Authentication

Let’s create a request authentication object with the following manifest:

apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
 name: jwt-req-authn
 namespace: web
spec:
 selector:
 matchLabels:
 app: httpbin
 jwtRules:
 - issuer: "digihunch.com"
 jwks: |
 ## jwks output from previous step ##

Replace the last line with the jwks output from the preparation step and store it to jwt-req-authn.yaml. It should look like this:

jwt-req-authn.yaml

Then apply it to the web namespace:

$ kubectl -n web apply -f jwt-req-authn.yaml

Now the request authentication resource is applied to the httpbin workload. We first test it with a random authentication token and it should be denied of 401 error:

$ curl -I -XGET $INGRESS_HOST/headers --header "Authorization: Bearer randomstring"
HTTP/1.1 401 Unauthorized
www-authenticate: Bearer realm="http://192.168.64.16/headers", error="invalid_token"
content-length: 79
content-type: text/plain
date: Sun, 13 Feb 2022 16:13:32 GMT
server: istio-envoy
x-envoy-upstream-service-time: 31

Then we take the jwt_token from the preparation step and present it to the request authentication resource by sending an HTTP request with the appropriate authorization token. It should return an HTTP 200 code this time:

$ curl -I -XGET $INGRESS_HOST/headers --header "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDQ3NzE0NjUsImlhdCI6MTY0NDc2ODQ2NSwiaXNzIjoiZGlnaWh1bmNoLmNvbSIsImp0aSI6Im1faUhla2pNbWRmUmlsWEdCaTFBR3ciLCJuYmYiOjE2NDQ3Njg0NjUsInBlcm1pc3Npb24iOiJyZWFkIiwicm9sZSI6InJlYWRlciIsInN1YiI6IkRJR0lIVU5DSCJ9.sVppwmvDqKvSsVdB05a_mDHymZq7Okvnwu-caTywXQgsUvOA6HfaySp_WXMyTp1HQ4WcTqKE4frZm7QNtrZsPso4bdD_4mEDYTswTCWhblaPy236NJBEH3ilB2BVySBVQKsjyxd94F1KV24SFWiR6lUxk52wKKE3ipBwR79jPizhAu9xxrfJ2Lfi5ypNa_kjBdJi63KCt2Y0eW94Fjq3PZs4ZalHJyaXYSx5Gxyei5f7QdpEOBpvs13mSdi9RqkgVQOjE0V1uBRpMckMyZs-IijknJcSu4fkrjgfNmXsrm__-vlM9UjUl2Jlj0x8bRC8l20IZ6t1ml-GFkwj39JC0g"

HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 13 Feb 2022 16:13:54 GMT
content-type: application/json
content-length: 589
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 17

In the example above, jwtRules can be used with other keys such as jwksUri to reference the jwks by Uri. More fields in JWTRules can be found here.

Now we have tested three curl commands:

Without authentication token at all: http server (istio-envoy) returns 200 code.
With an invalid authentication token: http server (istio-envoy) returns 401 code for error.
with a valid authentication: http server (istio-envoy) returns 200 code.

So we have the capability to validate token, but it is not yet mandatory to present the token. We can change this behaviour by tweaking authorization policy.

Authorization Policy

According to Istio documentation, to restrict access to authenticated requests only, this should be accompanied by an authorization rule. We start with the following policy:

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
 name: auth-pol
 namespace: web
spec:
 selector:
 matchLabels:
 app: httpbin
 action: ALLOW
 rules:
 - from:
 - source:
 requestPrincipals: ["*"]

Applying the manifest above to namespace web to it applies to workload httpbin. Then we

$ curl -I -XGET $INGRESS_HOST/headers
HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
date: Sun, 13 Feb 2022 16:29:58 GMT
server: istio-envoy
x-envoy-upstream-service-time: 35

The requestPrincipals clause makes it mandatory to present a token. The RequestAuthentication validates the token.

We can beef up the authorization policies by adding claims to the conditions, for example:

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
 name: auth-pol
 namespace: web
spec:
 selector:
 matchLabels:
 app: httpbin
 action: ALLOW
 rules:
 - from:
 - source:
 requestPrincipals: ["*"]
 to:
 - operation:
 methods: ["GET"]
 when:
 - key: request.auth.claims[iss]
 values: ["digihunch.com"]
 - key: request.auth.claims[role]
 values: ["reader"]

Both the to and when conditions are for HTTP traffic only, and we must tell Istio to inspect the traffic as HTTP, which is done implicitly with named ports on the Service object. Refer to this common problem from Istio’s documentation. The request principals, if a none wildcard value is specified, will be a SPIFFE format identity, the same one used for peer authentication, as discussed in the previous post.

The when clause above contains two key-value pairs. The first looks for the value of a standard claim (iss), the second for a custom claim (role). In the preparation step, we created the claims with those claims in Python and they will match the condition here.

Should any of the conditions above not match, a 403 (Forbidden) error code will be returned by the workload’s istio-envoy proxy.

Verify mTLS connection

In some cases, we need to audit whether the TLS traffic between workloads actually take place in mTLS. As I was developing my korthweb project, I don’t find a straightforward way of validating TLS. We can validate that mTLS mode on a workload using the following istio CTL command:

$ istioctl x describe pod my-workload-pod -n [namespace]

The output shows Effective PeerAuthentication and Applied PeerAuthentication. This verifies configuration. But how can we ensure that traffic are indeed using mTLS? For most of mTLS traffic, we can use Kiali’s observability feature. In Graph, we need to ensure “Security” is checked in the display drop-down. The pad lock will indicate the traffic is mTLS. To get reliable results, we have to artificially create some live traffic between workloads (e.g. curl from one Pod to another) so Kiali can pick up the update.

Validate mTLS traffic in Kiali

Unfortunately, even artificial traffic does not make Kiali the most reliable way to detect mTLS. There are three other approach to verify TLS traffic.

The first approach is to let Envoy proxy emit TLS related traffic. We can apply the following annotation line to a Pod, to tell its Envoy proxy to emit measurements related to tls_inspector:

sidecar.istio.io/statsInclusionPrefixes: "tls_inspector,listener.0.0.0.0_15006"

The metrics will be exposed to Envoy’s admin port (15000 on istio-proxy) with the path /stats. For example:

$ kubectl -n orthweb exec orthanc-7f4c9b759-lxnrb -c istio-proxy -- curl localhost:15000/stats | grep tls_inspector
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 22194 0 22194 0 0 21.1M 0 --:--:-- --:--:-- --:--:-- 21.1M
tls_inspector.alpn_found: 13
tls_inspector.alpn_not_found: 0
tls_inspector.client_hello_too_large: 0
tls_inspector.connection_closed: 0
tls_inspector.read_error: 0
tls_inspector.sni_found: 13
tls_inspector.sni_not_found: 0
tls_inspector.tls_found: 13
tls_inspector.tls_not_found: 0

We still need some live traffic to bump up the measurement. Comparing between tls_found and tls_not_found is a good way to determine if a Pod is receiving both TLS and plaintext traffic in PERMISSIVE mode.

The second approach to tell mTLS is via the connection_security_policy metric label. It is set to mtutual_tls if the connection is in mTLS. We need the dashboard for Prometheus:

$ istioctl dashboard prometheus

In the prometheus dashboard, click on “Graph” at the top, then search for “istio_tcp_connections_closed_total” for “istio_tcp_connections_opened_total“, the result should include a metrics called connection_security_policy which is labelled as mtutual_tls, as illustrated below:

mTLS metric

For more about collecting and querying metrics from Prometheus, check out Istio’s documentation here and here.

The third approach is to utilize the AUDIT feature of Authorization Policy. When a rule in Authorization Policy has a source with namespace or notNamespace field, it requires the incoming connection to have an SPIFFE identity and use mTLS. We can set the Authorization Policy’s action to AUDIT and use RBAC access logging, the same way we would do to troubleshoot RBAC access issues. From the log we can check RBAC failures due to missing SPIFFE identity.

Istio Authentication and Authorization

Sat, 05 Feb 2022 23:04:00 -0400

Applications running on Kubernetes platform seeks to offload common non-business features to the platform. Istio helps Kubernetes bridge that gap. It can enforce mTLS communication, which is known as Peer Authentication. It can help with two other things with the use of JWT token: when a web request presents a JWT token, it can validate whether it is authentic. Then, it can use the claims in JWT token to drive authorization decision on whether the specific request is allowed or denied. Both will use Istio CRDs.

Peer Authentication

Istio can enforce mTLS for TCP traffic between Pods. According to its documentation, enforcing mTLS at mesh level is as simple as applying a Peer Authentication resource to the root-level namespace:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
 name: default
 namespace: istio-system
spec:
 mtls:
 mode: STRICT

The role of mTLS is so Pods can validates each other’s identity and then encrypt the TLS traffic in between. Each workload must first have an identity and Envoy proxy addressed this issue by adopting SPIFFE framework. It gives each workload an identity in the format of <TRUST_DOMAIN>/ns/<NAMESPACE>/sa/<SERVICE_ACCOUNT>. For example: spiffe://cluster.local/ns/myapp-dev/sa/default. It is also important to understand that only Pods with injected Envoy sidecar have SPIFFE workload identity and therefore is able to speak in mTLS. For new services, this is usually not an issue. For migrating workload without sidecar, a Pod without sidecar may connect with one in the mesh (with sidecar) if the mtls mode is PERMISSIVE in Peer Authentication. Otherwise, the connect is reset at layer 4 with the following error:

curl: (56) Recv failure: Connection reset by peer
command terminated with exit code 56

Therefore, it is advisable to start with PERMISSIVE mode for a precautionary migration of workload to mTLS. With mTLS all effective at the mesh level, there is no need to natively configure TLS between services.

The SPIFFE identity used in PeerAuthentication can also be used in Request Authorization as rule conditions. I will discuss request authentication before request authorization. To understand request authentication, let’s first warm up on JWT.

JSON Web Token

JSON Web Token (JWT, RFC 7519) is a format to carry JSON payload with optional signature and/or encryption. It can be thought of as a document (in JSON format) with signature for web servers to exchange information. The signature portion makes it friendly for document consumers to validate the authenticity. It is also URL-safe, and thereby adopted in web-browser SSO context, to pass identity of an authenticated user between and identity provider and a service provider.

JWT enables token-based authentication, a significant improvement from traditional session-based authentication. The traditional session-based authentication can be illustrated as below:

session-based authentication

This authentication model has major drawbacks. First, a mechanism to validate the authenticity of Cookie is missing. Second, the server has to keep the session information, making itself not stateless, unless a state store such as memcached is introduced.

token-based authentication

In token-based authentication such as using JWT, a token is issued. The authenticity of the token are validated before the server provides data, and it can be validated by any backend server.

The payload of JWT consists of claims, which are statements about an identity (such as name, role, email). There are custom claims as well as standard reserved claims, such as iss (issuer), sub (subject), aud (audience), iat (issued at time), exp (expiration time), and jti (JWT ID). When a program produces a JWT, it turns the raw payload into standardize payload by adding the required reserved claims and may sort the claims alphabetically.

The JWT consists of three parts with a period as delimiter:

The third part is a signature in the format of JWS (JSON Web Signature, RFC 7515) for the JWT consumer to validate its authenticity. The first and second parts, as you can tell, are the claims in the document. Their base64 encoding can be decoded with no effort and should therefore be considered exposed. Although JWT addresses the authenticity of information, it does not intend to address the confidentiality of the payload at HTTP layer. The payload should not carry sensitive information and should always be used with secure HTTPS port. To tackle this issue, there is JWE (JSON Web Encryption, RFC 7516) which is an implementation similar to JWT which also encrypts the payload.

Some IAM protocols are built on top of JWT. For example, the OpenID Connect specification also defines a set of standard claims that it uses while still allow custom claims.

Request Authentication

Istio can perform request authentication using its CRD. It is important to distinguish request authentication and user authentication. In user authentication, the identify provider typically looks up an identity store and compares password hash results to check whether the identity of the visiting user is authentic or not. This is outside of Istio’s capability but many off-the-shelf solution excels at it, such as Azure AD. Once the user’s identity is validated by identity provider, and a JWT is issued for downstream service providers to consume. Istio’s CRD can front the service provider and validate that the presented JWT is authentic. It authenticates the identity of a request (as truly issued by the trusted issuer without being tampered). This process does not involve checking user’s identity, even though user’s identity could be stored in the payload by the JWT issuer.

Istio uses the RequestAuthentication CRD to perform this function. The JWT issuer signs with its private key and stores the signature in the JWT. When it is presented to Istio, Istio’s RequestAuthentication CRD needs the public key of the issuer in order to validate the JWT. The public key usually comes in as a JWK (JSON Web Key, RFC7517), a format convertible to and from PEM format. The JWK can be provided either inline in the RequestAuthentication’s YAML manifest, or via a URI.

Below is an example of a basic RequestAuthentication declaration:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
 name: httpbin
 namespace: foo
spec:
 selector:
 matchLabels:
 app: httpbin
 jwtRules:
 - issuer: "issuer-foo"
 jwksUri: https://example.com/.well-known/jwks.json

In this example (from the documentation), the jwtRule requires that the issuer be issuer-foo, and the JWK (containing public key) is provided by a given URI address. Istio will pass the authentication once the signature in the presented JWT is verified with the JWK.

Authorization Policy

Istio’s Authorization Policy by itself can operate at both TCP or HTTP layers and is enforced at the envoy proxy. The result is an ALLOW or DENY decision, based on a set of conditions at both levels. If the traffic is HTTP then you should consider use some HTTP level information as it provides a lot more flexibility. Even when operating at HTTP layer, AuthorizationPolicy does not have to work in conjunction with RequestAuthentication. The rules can use path, methods, etc to drive an authorization decision, for example:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: authz-policy-orthanc
spec:
 selector:
 matchLabels:
 app: orth
 action: ALLOW
 rules:
 - from:
 - source:
 namespaces: ["istio-system","orthweb"]
 to:
 - operation:
 methods: ["GET","POST","PUT","HEAD","DELETE"]
 ports: ["8042"]
 - from:
 - source:
 namespaces: ["istio-system","orthweb"]
 to:
 - operation:
 ports: ["4242"]

The claims in the JWT payload can also be used to drive authorization decision, as exemplified in the Istio documentation, by using a when keyword in a rule and specifying the claim as a key:

 when:
 - key: request.auth.claims[iss]
 values: ["https://accounts.google.com"]

The when clause requires that the iss claim in the JWT must carry a specific value in order to ALLOW the HTTP request. While the claims in JWT is just an additional factor to drive authorization decision, using authenticated information to drive authorization decision makes the overall workflow more secure, and should therefore be used when applicable.

When using AuthorizationPolicy CRD, keep in mind:

Use correct selectors so it only applies to workloads labelled as such. Without selector the policy takes effect in the entire namespace which is a recipe for issues
When multiple policies (each with multiple rules) are applied to the same workload, be aware of the policy precedence. Troubleshooting may get tricky.
For an AuthorizationPolicy to use HTTP fields in rules, it first needs to identify the traffic as HTTP. To tell Authorization Policy to treat the traffic as HTTP, we need to understand Protocol Selection. In a nutshell, we can name the port as http or http-* in the Service manifest of the workload. Otherwise all HTTP-based rules will be missed.

For troubleshooting, we can check authorization policies effective on a Pod with:

$ istioctl x authz check orthanc-6c9679d8c7-2ttlf -n dev-orthweb

This returns the effective policies but does not necessarily indicate which rule is matched when a request is denied or allowed. To find out further information, you will need to follow Istio FAQ to set RBAC logging to debug, and then monitor the log in the istio-proxy sidecar.

Apart from HTTP fields, path, authenticated claims in JWT, Istio Authorization can also integrate with an Open Policy Agent (OPA) to drive actions, in advanced use cases.

Bottom line

While Istio itself does not perform user authentication, its support of JWT in RequestAuthentication allows a workload to integrate with external identity provider. This capability, along with creative use of claims in JWT, also empowers authorization capability.

Traffic Segmentation on Kubernetes Platform

Thu, 27 Jan 2022 13:54:00 -0400

When operating Kubernetes as a platform for multiple tenants, one of the concerns is controlling the network traffic. This is sometimes referred to as traffic segmentation. This initiative involves a broad range of technical topics from networking to containerization. By no means I am an expert on each of those topics. I have however developed some best practices in how to break down this challenge and hence bringing the thought into this post.

Tenant Isolation

Kubernetes has the concept of namespace to logically separate resources allocated for each tenant. Each tenant only operates within their given namespaces. The isolation of computing resources such as CPU and memory can be managed via ResourceQuota objects, and they are enforced at the kernel level, leaving networking isolation the main discussion in the topic of tenant isolation. If the platform hosts a lot of stateful workload then we also needs to address tenant isolation at the storage layer. In this post we focus on the network aspect of resource isolation, aka traffic segmentation.

Controlling network traffic can require a significant amount of efforts depending on the goal. That is why we need to first assess the multi-tenancy models:

Soft multi-tenancy: usually a platform is shared by multiple teams within the same organization. Tenants are incentivized to be good neighbours.
Hard multi-tenancy: usually a platform shared by multiple customers from different organizations. There is no trust between different tenants, or between tenant and platform operator.

Reality may sits somewhere in between, but we often have to come back to this model when making a technical decision, because it determines the degree of tenant isolation, or the amount of effort we are willing to put in on tenant isolation. At the tough end, is zero-trust network, which usually have the following requirement:

Requirement 1: All network connections are subject to enforcement (not just those that cross zone boundaries).
Requirement 2: Establishing the identity of a remote endpoint is always based on multiple criteria including strong cryptographic proofs of identity. In particular, network-level identifiers like IP address and port are not sufficient on their own as they can be spoofed by a hostile network.
Requirement 3: All expected and allowed network flows are explicitly allowed. Any connection not explicitly allowed is denied.
Requirement 4: Compromised workloads must not be able to circumvent policy enforcement.
Requirement 5: Many Zero Trust Networks also rely on encryption of network traffic to prevent disclosure of sensitive data to hostile entities snooping network traffic. This is not an absolute requirement if private data are not exchanged over the network, but to fit the criteria of a Zero Trust Network, encryption must be used on every network connection if it is required at all. A Zero Trust Network does not distinguish between trusted and untrusted network links or paths. Also note that even when not using encryption for data privacy, cryptographic proofs of authenticity are still used to establish identity.

As you can see there’s a lot of efforts involved in building a zero-trust network. The cost of building a zero-trust network is worth it only when we determines that the overall business requirement demands it.

Pod Networking

It is important to understand Pod networking before developing a traffic segmentation strategy. Pod networking has to do with the CNI driver used for the cluster. There are in general two categories:

Overlay network: Pods are placed on a VXLAN configuration. This is mostly seen in basic Kubenet mode or CNI drives such as Flannel. NAT is required for Pods to communicate across nodes, which might introduce performance issues when deployed at scale. Pods do not use IP address from the host network.
Regular network: In this mode Pods are on the same network as the nodes are. For example, Azure CNI assigns Pods with IP address from a given V-Net. The AWS-VPC CNI integrates VPC networking with Pods. Since Pods are on a corporate network, the traffic control must also consider measures at the whole network level.

The main benefit of the first approach, is that IP exhaustion is less likely due to the introduction of a VxLAN. The other benefit from a networking perspective is that the Pod networking is born separated from the corporate network. In the second approach, by assigning Pods with a corporate IP address (which brings the risk of IP exhaustion), Pods are also potentially exposed to all corporate traffic at layer 3. To tackle this additional risk, network security group should be used in the V-Net for Azure AKS, or security groups for Pods should be considered with AWS EKS. Although we will discuss Network Policy in the rest of this essay, Network Policy mostly addresses the traffic segmentation issue within a Kubernetes cluster. A Pod placed on the corporate network needs traffic segmentation strategies from the perspective of the whole network.

Another network-level traffic segmentation strategy is on the corporate firewall. For example, with AKS you can specify outbound type as user-defined routes (UDR) to direct all outbound traffic through a corporate firewall where traffic will be inspected. There are firewall products dedicated for managing highly dynamic pod traffic from Kubernetes. This strategy can be used in conjunction with network security groups.

Network Policy

Kubernetes’s default behaviour is to allow traffic between any two pods in the cluster network. This is undesirable. NetworkPolicy is the native Kubernetes construct for platform operators and application developer to control network traffic at layer 3/4. It uses namespace and pod selectors, and is defined based on allow rules, which is good for general use.

Further to the native Network Policy, you can adopt third party policies for advanced features. For example, Azure has Azure Network policy (works for Azure CNI only) and Calico Network policy (works for Calico CNI, Azure CNI or Kubenet). The third party network policies usually provides advanced features such as:

Deny rules
multiple types of endpoints in addition to Pods, for example, VMs, network interfaces which can be useful in network-level traffic control
ordering and priority of rules
Flexible matching rules

Calico network has a page that summarizes its features and how it extends the Kubernetes NetworkPolicy. Below is an example of a Calico’s network policy:

apiVersion: projectcalico.org/v3 
kind: NetworkPolicy 
metadata: 
 name: allow-tcp-6379 
 namespace: production 
spec: 
 selector: color == 'red'
 ingress: 
 - action: Allow 
 protocol: TCP 
 source: 
 selector: color == 'blue' 
 namespaceSelector: shape == 'circle'
 destination: 
 ports: 
 - 6379

It is as self-explanatory as Kubernetes Network Policy. No matter which kind of network policy, this approach takes effect at layer 3/4. The rules are eventually implemented in the kernel on the node (Iptables). The management of this layer is usually by the platform team and they need to have some application knowledge.

Authorization at Application Layer

Traffic above layer 4 is considered application layer traffic. At application layer, the decision to allow or deny a request is by definition an authorization decision. Another layer of protection can be placed at layer 4 is mTLS which ensures that each request to have an identity. The authorization can be built in the application, but it is also very common to offload these functions to the service mesh layer. For example, Istio has constructs such as PeerAuthentication, Request Authentication and Authorization Policy. We will those in more details in a few coming blog posts. Below is a simple example of Istio’s Authorization Policy:

apiVersion: security.istio.io/v1beta1 
kind: AuthorizationPolicy 
metadata: 
 name: "details-viewer" 
 namespace: default 
spec: 
 selector: 
 matchLabels: 
 app: details 
 action: ALLOW 
 rules: 
 - from: 
 - source: 
 principals: ["cluster.local/ns/default/sa/bookinfo-productpage"]
 to: 
 - operation: 
 methods: ["GET"]

The rule is also fairly self-explanatory. Compared to Network Policy, the point of enforcement of these Authorization policies are at the envoy proxy. The management of policies at this layer can be debatable if department boundaries are not clear, but it should in general be owned by personnels with good application knowledge.

Consistency between Policies

In-cluster traffic can be controlled with both Network Policy (Calico or Kubernetes) operating at layer 3-4, and Authorization Policy (Istio) at layer 4-7. This brings another challenge of maintaining consistency between the two types of policies. This is especially challenging when they are managed by different teams in a corporate and therefore many operators for soft multi-tenant platform choose not to implement Network Policy or only implements a baseline.

Some network solution providers builds a solution for this. For example, Calico has the capability to enforce network policy for Istio. This integration requires some configuration, but the enhanced GlobalNetworkPolicy supports HTTP methods, eliminating the need to define a separate Authorization Policy in Istio and worry about its consistency with NetworkPolicy. The platform build however, still needs to determine who owns this policy construct. Below is an example from Calico documentation:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
 name: customer
spec:
 selector: app == 'customer'
 ingress:
 - action: Allow
 http:
 methods: ["GET"]
 egress:
 - action: Allow

One of the benefits of using this integration is a unified policy language based on GlobalNetworkPolicy CRD. In the mean time, organization should also develop strategy to ensure that, once Calico is integrated with Istio, then there is no need to separately build authorization policies, which may come in conflict with Global network policy.

Summary

Controlling network traffic is difficult on Kubernetes platform. In this article I proposed a few angles to approach this issue for enterprise clients.

From Ingress to CRD: why my solution needs Istio Gateways on Kubernetes platforms

Wed, 29 Dec 2021 22:50:16 -0400

Update: also read my other article here on the different generations of ingress technologies.

In my Korthweb project I was researching for the best ingress mechanism for HTTP and TCP workload, both of which need to be secured. I started with Kubernetes Ingress but eventually decided to go with Istio Gateway. This blog post is about the justification. In this essay, I will make the distinction between Ingress and Gateway and explain why a Kubernetes platform needs the latter going forward.

The word ingress can be used either to indicate ingress resource (in conjunction with ingress controller) in the context of Kubernetes cluster, or more generally to indicate the technology (e.g. provided by service mesh) to direct north-south traffic, originated from the outside into the workload running within the cluster. I will use both meanings of the word ingress throughout the article.

Background

In my previous post, I discussed Kubernetes Service object and native Ingress. In short, a Service object addresses the problem of exposing a workload (Pod or Deployment), operating at layer 3-4. A ClusterIP type of service exposes workload within the cluster only, and therefore is only used by internal services that are consumed by workload in the same cluster. A NodePort type of service exposes workload to outside of the cluster, using the node’s IP address and port range available on the node, a great step forward but still quite inconvenient and subject to the limitation of using Node’s IP and Port. A LoadBalancer type of Service brings separate IP address and port for running a service. To back up this type of service, the platform has to provide a load balancer with its own IP address and port range to manage. The implementation is platform specific e.g. Metal LB for Minikube, Azure Load Balancer for AKS, NLB for AWS EKS. In this type of service, Node Ports still exists but are not exposed to the outside world. Instead, they are only exposed to the backend of Load Balancer, whose front-end port is exposed to the outside world. In both NodePort and LoadBalancer types of Service, the kube-proxy process plays a role on each node to direct traffic into NodePort to target Pods.

Kubernetes Ingress, on the other hand, targets issues above layer 4, for example, path-based routing and is therefore mostly used for HTTP and HTTPS traffic. It is implemented by an Ingress Controller and now there has been a diverse ecosystem of those ingress controllers. In many implementations (e.g. Azure Application Gateway Ingress Controller, AGIC), the ingress themselves usually come with a load balancer managed by themselves. This eliminates the need for a separate LoadBalancer type of Service just for L4 capability. As a result, Ingress (with both L4 and L7 capabilities) is usually deployed along with ClusterIP type Service. Here is an example. The declaration of the Ingress in this case usually contains numerous lines of annotations in order to communicate the specification to the implementation.

Requirement

Using Kubernetes Ingress along with Service of ClusterIP type seems perfect to address the majority of use cases. However, it has its blind spots. My Korthweb project deals with DICOM traffic (a protocol on top of TCP) over TLS as well as HTTP traffic, which can be broken down as:

The ability to proxy TCP traffic over an arbitrary TCP port
The ability to terminate TLS/SSL encryption for TCP traffic

The Ingress documentation of Kubernetes clearly states that: an Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

The stance of community-driven Nginx ingress controller can be found in the documentation, which suggests the use of Service object for arbitrary TCP port. There is some unofficial claims of workaround available but I’m not confident.

I also went through a number of other Ingress providers but to my disappointment, the only product that supports it seems to be Traefik:

Product	Requirement #1	Requirement #2
F5 driven Nginx Ingress Controller	Supported	There have also been requests for TLS termination for TCP traffic but the request has not been closed as of yet.
HA Proxy Ingress controller	Supported	There’s no mention of this in documentation.
Kong’s Kubernetes Controller (with TCPIngress CRD)	Supported	Documentation claims support through SNI-based routing. However, the example demonstrates it using self-signed certificate only. BYO cert not supported according to this GitHub issue.
Traefik Lab’s IngressRoute (with IngressRouteTCP CRD)	Supported	Documentation seems to suggest that it is supported by IngressRoute.

My general impression is that Req #2 isn’t very popular so the providers either don’t support it or delaying the implementation. Even if they do, the CRD used for TCP ingress is different than HTTP’s. For example with Traefik Lab, the CRD for TCP is IngressRouteTCP, and for HTTP it is IngressRoute.

Introduction to Gateway

In addition to functional shortages, according to this blog post, there are signs of fragmentation into different but strikingly similar CRDs and overloaded annotations. In Kubecon 2019, a group of contributors discussed the evolution of Ingress into Gateway. Below are the two key slides stolen from their presentation:

This sums up how the concept of Gateway is different from Ingress. Gateway is an instantiation of a given LB. It works along with Route to achieve the functions brought by an Ingress. Gateway as a resource type of its own makes management easier at L4-L6 by a separate team. Routing at L7 is offloaded to “Route” resource type.

This conceptual evolution gives rise to the Gateway API open source project managed by the SIG-NETWORK community. Gateway API is a collection of resources that model service networking in Kubernetes, including GatewayClass, Gateway, HTTPRoute, TCPRoute, Service etc. The aim of this initiative is to evolve Kubernetes service networking through expressive, extensive and role-oriented interfaces that are implemented by many vendors and have broad industry support.

The diagram above stolen from Gateway API website illustrate the management model for each resource.

Kubernetes Gateway Implementations

The Gateway API is still a fairly young initiative and all the reference implementations are either work-in-progress or in early stages. In the last section, we learned that in the journey from Ingress to Gateway, the standardization initiative comes a bit behind the implementation efforts. For many supporters, the natural strategy is to continue with existing proprietary ingress controller implementation, and retrofit their technology to the emerging Gateway API along the way.

Because of that, we can see many market players with multiple flavours of implementations, typically one that evolves from their original product offering, with higher adoption rate and maturity level, and one that conforms to Gateway API specification.

For example, Traefik Labs has its Gateway API implementation in experimental stage. Its own implementation consists of (standard) Kubernetes Ingress and Kubernetes IngressRoute (based on custom resource with support of advanced features such as TCP route). Another example is the HAProxy Ingress, a community driven ingress controller implementation for HAProxy. Starting from its version 0.13, it partially supports the Gateway API’s v1alpha1 specification. Here is the conformance statement. Contour as a CNCF project for ingress controller has support for Gateway API at alpha version. Kong’s Kubernetes Ingress Controller follows the same path.

The Gateway functionality is also provided as part of Service Mesh product offering, and we can see some reference implementation by service mesh providers. Istio has its own Gateway implementation but tries to adapt to Kubernetes Gateway API. Hashicorp Consul also claims to be building support for Kubernetes Gateway API.

Until the Gateway API project matures, it is not recommended to use Gateway implementations that conforms to it, unless you intend to be their Guinea Pig. For my project, I chose Istio Gateway.

Istio Gateway implementation

As discussed, when the Gateway standard is still in its infancy, I chose a non-standard implementation of Gateway, even though it may include some CRDs. Out of the many Gateway implementations, I choose Istio mainly for two reasons.

First, I need a Gateway as part of service mesh because service mesh provides many other features that are needed in the platform. One of the goals of Service Mesh is to provide commonly used features (traffic management, observability, security, extensibility) in a commodity layer on top of Kubernetes. Gateway is not the only thing I need out of this commodity layer. With one install of Istio, many common platform-level problems are also addressed (e.g. mTLS, traceability, etc). I’ve compared three major service mesh technologies as below:

Linkderd	Istio	Consul
– lightweight – uses linkerd2-proxy – the original service mesh – no Gateway implementation (deal breaker in my use case)	– feature rich – uses Envoy proxy – complex but getting better	– initially a service discovery and distributed key-value store – lacks observability features but getting better – uses Envoy proxy

It is worth-noting that there are some initiatives to standardize service mesh (e.g. SMI) with reference implementation such as OpenServiceMesh. The standard is too weak to be considered important for now compared with maturity and stability. Istio has strong community support. The risk with Istio to acknowledge, is that it is not following an open-governance model (unlike many other CNCF projects), which could potentially causes vendor lock-in.

Second, The Istio Gateway features the separation between Gateway and Virtual Service CRDs. The former defines entry point and the latter defines routing rules. This design separates entry points from routing rules, enabling the flexibility of reusing the same Virtual Service for different gateways.

Istio uses Envoy proxy to manage traffic between Pods. Unlike the kube-proxy pattern, the Envoy proxies are side-car containers centrally managed by Istio Control Plane, which also enables other features such as traceability, as illustrated in the diagram below (stolen from this post).

In addition to Virtual Service, Istio Gateway also has the concept of Destination Rules. Virtual Service defines how to route traffic to different destinations. Destination rule defines how to split traffic to different subset at the routing destination. The concepts are documented on this page. For HTTP traffic, the relevant entities can be represented in the diagram below:

Here is an example of using Gateway and Virtual Service resources (from Korthweb sample project). Also note that in the Istio literature, there is neither a CRD name called “Ingress”, nor a resource type named “Ingress Gateway”, although they may be loosely used to refer to “Gateway resources configured to manage ingress traffic”.

Istio Gateway Installation

There are more than one ways to install Istio. In the past, Istio Operator was used to install Istio. The Operator calls IstioOperator API. Today the use of Istio Operator is not recommended anymore but the IstioOperator API is used implicitly by Istioctl installer. The two recommended approaches to install Istio today is by Istioctl and Helm.

With istioctl, you can specify an option imperatively, or using an overlay file as did in this lab. With Helm, istio has multiple charts, and requires multiple steps:

install CRDs using the base chart
install istiod using the istiod chart. Here is an example of values provided to Helm installer.
install ingress gateway using the gateway chart. Here is an example of values provided to Helm installer for ingress gateway. Note that egress gateway also requires the same helm chart with different value definition.

This instruction contains how to manually install Istio gateways using Helm, including installation of multiple Helm charts. I attempted to create a single chart to consolidate the multiple charts required for istio. It was not successful because of an error when trying to reference the same gateway chart dependency for multiple times.

From Microservice to Service Mesh

Thu, 25 Nov 2021 00:31:19 -0400

We all know what microservice is now but how does service mesh assist with microservice development.

Microservice

Microservice as an architecture was firstly conceptualized in this article by Martin Fowler in 2014. It covers the pros (strong module boundaries, independent deployment, technology diversity) and cons (dealing with distributed system, eventual consistency, operational complexity). The reality is, many teams develops their product with the microservice architectural pattern.

The implementation of microservice architecture involves a lot of programming patterns, and tools. The creation of these patterns and tools are usually done in a separate dedicated project so developers can focus on business logics. When building software, developers only need to interact with libraries and frameworks. Libraries (e.g. log4j) provides dependencies, and developers needs to write code to call the libraries. On the other hand, frameworks (e.g. Spring, Flask) not only provides tools, but also implements a pattern. It addresses a set of common problems such as authentication, expose http service, logger and database connectivity. Once set up, the framework will call the code that developers write (unlike in libraries).

Spring – an example of Microservice framework

When it comes to microservice, a well-known appliction framwork is the Spring framework. It solves problems such as:

Application context and dependency injection (for Inversion of Control, or IOC)
Database access and transaction management
Expose rest APIs (using spring MVC)

There’s an entire ecosystem of projects under Spring framework. This framework is a huge system requiring a lot of configuration efforts. This is where Spring Boot helps.

Spring Boot makes it easy to create stand-alone, production-grade Spring based applications that you can just run. It features the “convention over configuration” paradigm to save programmers from boiler plate configuration. SpringBoot gives you a standalone application ready to run without complicated deployment steps.

Managing configuration in property files does not scale in the time of microservice. Spring Cloud provides configuration as a service (in line with everything else microservice framework). It doesn’t necessarily have to be hosted in the cloud. It is comparable to Apachee Zookeeper, Etcd (distributed key value store), Hashicorp Consul and Netflix OSS (Eureka, Ribbon, Hystrix). You can pull from Git repo. The mission of Spring Cloud is to eliminate boilerplate associated with distributed systems problems for Spring Boot applications.

Many developers use Spring Boot along with Spring Cloud to build microservices. This page contains a diagram for such architecture.

In this architecture, Spring cloud helps with service discovery, traffic routing, circuit-breaking, distributed tracing and monitoring. It can also act as API gateway (in place of Nginx).

API Gateway and API Management

Microservices relies on API. Let’s distinguish API gateway, and API management (this long post has some good information).

API Gateway is a microserivce pattern. The idea is a single point of entry for all clients. The API gateway either proxy an incoming request to the appropriate service, or it may fan out a request to multiple services. The other important aspect is the API gatway can expose a different API for reach client.

API Gateway Pattern

A variation of this pattern is the Backends for frontends pattern, where it defines a separate API gateway for each kind of client. The API Gateway may authenticate user and pass an Access Token containing information about the user to the services. It may use a circuit breaker to invoke services. To summarize, the key functions of an API Gateway in this pattern is:

Unified entry point for multiple API implementations
Protocol transformation
Request morphing
Client specific logics

This page from Azure has a good comparison between API Gateway pattern vs direct connection between client and microservice. Note that API Gateway can also refers to API Gateway product, which implements the functions above. For example

Spring Cloud Gateway
Solo.io Gloo
Netflix Zuul

API management acts as a proxy for an existing API implementations. Typical functions include:

AuthN and AuthZ
Service Discovery Ingreation
Load balancing (e.g. L7 path based routing)
Logging, tracing (track user), correlation
Response Caching
Retry policies, circuit breaker, QoS
Enforce policy
Track usage and monetization
metrics (duration)
rate limiting and throttling
Request morphing (header, query string and claims transformation)
IP whitelisting

API management are usually implemented as tightly controlled shared infrastructure owned by either a “platform team”, “integration team”, or other API infrastructure teams. Examples of API management product (including SaaS) are:

Google Cloud Apigee
Mulesoft
Kong

In real world, people often use API management produce and API gateway product interchangeably.

Service Mesh

Many dub Service Mesh the next generation of Microservice. So what is the relationship between microservice and service mesh. The Microservice architectural pattern creates the need for API gateway pattern. To address this pattern, the API Gateway products first emerged. Service mesh emerged later. Service mesh and API gateway have a common set of features.

In this presentation (a tale of two frameworks) from 2018 (early days of service mesh), two teams discussed microservice (spring cloud) and service mesh (istio) approaches. There is a slide about when to use which. Many teams since have moved to Service Mesh for feature richness. This is a case study from 2021.

There are a number of service mesh technologies, such as Consul, Isito and Linkerd. Here is a comparison chart. Even though all those projects are open-source, there is some competition already. Linkerd is the first to bring up the concept of service mesh in this blog. It also purportedly has better performance than Istio. However, it does not use Envoy proxy. Istio is good at marketing. It has higher adoption rate and is feature rich. However, Google did not donate Istio project to CNCF as many expected. Instead, it created its own governing body, the Open Usage Commons. The Istio is not an open-governance project, which potentially diverge from CNCF in the future [1]. Hashicorp Consul initially was built for service discovery and distributed key/value store. It supports Kubernetes and VM. However, it still lacks observability features.

Apart from the three major technologies, other players tries to push for standardization of service mesh. The most influential initiative is the SMI (service mesh interface), pushed by Microsoft. The idea is a separation of standard and implementation, so late players will have a chance. OpenServiceMesh is Microsoft’s reference implementation of SMI. The SMI is something to watch for but it remains pretty week thus far. Google’s platform has Anthos Service Mesh which is a commercial distribution based on Istio. AWS has its own AppMesh technology, also using Envoy proxy.

[1] Update from Apr 25, 2022 – Istio applied to become CNCF project.

Istio Lab – Ingress and Egress

Tue, 16 Nov 2021 21:28:00 -0400

Istio is a popular open-source service mesh implementation using Envoy proxy. One of the benefit of using Istio is the ingress and egress it brings to native Kubernetes platform. This article is a hands-on guide to test Istio ingress and egress gateways on Minikube. It was tested on my MacBook. All the information in this post can be found in several different pages from Istio and Minikube documentation. The purpose of this article is a single cheat sheet for myself to quickly set up Istio ingress and egress testing.

Install Istio on Minikube

We start with configuring Minikube on MacBook (or change the command accordingly on other platform), and install metal LB load balancer. The steps with Istio also works with a regular Kubernetes platform.

minikube start --memory=12288 --cpus=6 --kubernetes-version=v1.20.2 --nodes 3 --container-runtime=containerd --driver=hyperkit
minikube addons enable metallb

Then we need to give load balancer an IP address range before setting up the load balancer. Minikube adds an interface named bridge100 to MacOS host environment. The IP address of the host can be found via the following command:

minikube ssh "ping host.minikube.internal -c 1"

For example, my host IP address is 192.168.64.1, so I dedicate the IP range 192.168.64.64 to 192.168.64.80 to the MetalLB load balancer. Then I need to provide this range when configuring the load balancer:

minikube addons configure metallb
kubectl -n metallb-system get po

After the configuration, make sure the MetalLB pod is up. Then we can install istio to the Kubernetes cluster. The steps are completed using istioctl following the instruction, I created a quick and dirty script to put the steps together:

#! /bin/bash
curl -L https://istio.io/downloadIstio | sh -
export PATH=$(realpath istio*/bin):$PATH
ln -sfn $(ls -d -- istio-*) istio
if istioctl x precheck; then
 echo ready to install istio and lable namespace for istio-injection
 istioctl install --set profile=demo -y --verify --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY
 kubectl label namespace default istio-injection=enabled
else
 echo failed precheck
 exit 1
fi

The script above download the istio zip file, unzips it, installs Istio (in demo mode and set outboundTrafficPolicy to REGISTRY_ONLY), labels default namespace for proxy injection. To use istioctl without spelling out full path, we can run the following command again separately outside of script:

export PATH=$(realpath istio/bin):$PATH

To validate, check istioctl version command.

Install Sample application

For simplicity, the rest of this article uses httpbin to test Istio’s ingress and egress. For completeness, this section gives the steps to install bookinfo application as it is used in other Istio testing as well. It also help understand how to ingress to an application with Istio using an IP provided by the load balancer. Skip this section if just need to test with the simple httpbin application. The Bookinfo application comes within the istio 1.12 directory. In the following steps from the istio directory, we install some observability add-ons as well as the bookinfo application. Then we test access to the website.

kubectl apply -f samples/addons/prometheus.yaml
kubectl apply -f samples/addons/kiali.yaml
kubectl apply -f samples/addons/jaeger.yaml
kubectl apply -f samples/addons/grafana.yaml

kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

kubectl exec "$(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}')" -c ratings -- curl -sS productpage:9080/productpage | grep -o "<title>.*</title>"

export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT
curl -s "http://${GATEWAY_URL}/productpage" | grep -o "<title>.*</title>"

On my laptop, the value of $GATEWAY_URL is 192.168.64.64:80. Therefore I can browse to site http://192.168.64.64:80/productpage with local browser. We can also check kiali dashboard:

istioctl dashboard kiali

Then we are good to examine ingress and egress.

Prep Work

The following custom resources are related to traffic management in istio:

Virtual Service (vs)
Destination Rule (dr)
Service Entry (se)
Gateway (gw, Istio’s ingress or egress)

We will expand on the role of each in a separate article. To execute command from within a Pod, we also use sleep application which is essentially a Pod with a sleeper thread for the purpose allowing user to run troubleshooting command. The sleep application is provided in istio directory:

kubectl apply -f samples/sleep/sleep.yaml
kubectl get po -l app=sleep

Find out the Pod name to execute command. Then we use httpbin website to emulate an external service, and use the sleep Pod (suppose the name is sleep-557747455f-q99cz) to access the service:

kubectl exec -it sleep-557747455f-q99cz -c sleep -- curl http://httpbin.org/headers

As we previously set outboundTrafficPolicy.mode=REGISTRY_ONLY, this step is expected to output nothing. If this is not the case, it means the cluster may have ALLOW_ALL as outbount traffic policy, and you will need to follow this guide to set it to REGISTRY_ONLY. With REGISTRY_ONLY, in order to allow access, we need to register a service entry, by applying the document below:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
 name: httpbin
spec:
 hosts:
 - httpbin.org
 ports:
 - number: 80
 name: http
 protocol: HTTP
 resolution: DNS

Now we should be able to access httpbin service as it is registered:

kubectl get se httpbin -o yaml

Using the exec command above again, we should see output from the curl command.

Ingress Testing

We deploy an httpbin service in order to test ingress. The YAML declaration is prepared by Istio installation package:

kubectl apply -f samples/httpbin/httpbin.yaml

Then we deploy the ingress:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: httpbin-gateway
spec:
 selector:
 istio: ingressgateway
 servers:
 - port:
 number: 80
 name: http
 protocol: HTTP
 hosts:
 - "httpbin.digihunch.com"

The selector above indicates that the gateway deployed to a Pod labeled with istio: ingressgateway. Then we create a corresponding virtual service for this gateway:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: httpbin
spec:
 hosts:
 - "httpbin.digihunch.com"
 gateways:
 - httpbin-gateway
 http:
 - match:
 - uri:
 prefix: /status
 - uri:
 prefix: /delay
 route:
 - destination:
 port:
 number: 8000
 host: httpbin

Note that the hosts setup must match the configuration in gateway. The virtual service defines two services at URIs /status and /delay to expose, and we can confirm the creation of virtual service:

kubectl get vs httpbin -o yaml

We can then test with curl command from MacOS host to verify that the ingress is working. We need to use the IP address of the service (192.168.64.64 with port 80 as previously indicated):

curl -I -HHost:httpbin.digihunch.com http://192.168.64.64/status/200
curl -I -HHost:httpbin.digihunch.com http://192.168.64.64/delay/2

We use -HHost to emulate this request going to httpbin.digihunch.com (otherwise we’d have to fake DNS entry in /etc/hosts). Both command should give their return without error. The second command will give return after a delay of 2 seconds.

This validates the success of ingress setup.

Egress Testing

To visit an external service, there are four options:

Just access external service from Pods, when you have global.outboundTrafficPolicy.mode=ALLOW_ANY
Use ServiceEntry (as illustrated above in Prep Work)
Bypass envoy proxy (not recommended, because there is no role of service mesh)
Configure an Egress Gateway

In the Prep Work, we’ve seen how to use Service Entry. However, there is no control of how Pods access the external service. We are going to configure an Egress Gateway to manage the outgoing traffic flow. This allows you to manage route, and apply observability features to the outgoing traffic, something the Service Entry alone will not provide. The egress gateway have two typical use cases as explained on Istio website.

First, we want to confirm that the egress pod exists, with a quick command:

kubectl get po -n istio-system -l app=istio-egressgateway && kubectl get se

It should exist as we installed istio as demo version. We then also need a service entry, as we have already configured in Prep Work. At this point, the egress gateway resource has not been created.

Now, in a separate terminal, we monitor the log of egress pod

kubectl logs -f istio-egressgateway-7d5d69dcfd-kvwxv -n istio-system

With the -f (follow) session on, if we visit external service as we did during Prep Work, there is no new entry pop up at the tail of the log. This is because the external access is still not managed by egress. Now we need to add egress, with the following resources applied:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: istio-egressgateway
spec:
 selector:
 istio: egressgateway
 servers:
 - port:
 number: 80
 name: http
 protocol: HTTP
 hosts:
 - httpbin.org

Then we setup the virtual service with related rules:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: vs-for-egressgateway
spec:
 hosts:
 - "httpbin.org"
 gateways:
 - istio-egressgateway
 - mesh
 http:
 - match:
 - gateways:
 - mesh
 port: 80
 route:
 - destination:
 host: istio-egressgateway.istio-system.svc.cluster.local
 subset: httpbin
 port:
 number: 80
 weight: 100
 - match:
 - gateways:
 - istio-egressgateway
 port: 80
 route:
 - destination:
 host: httpbin.org
 port:
 number: 80
 weight: 100

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
 name: dr-for-egressgateway
spec:
 host: istio-egressgateway.istio-system.svc.cluster.local
 subsets:
 - name: httpbin

With all applied, we can repeat the access command using curl, and watch for the log tail.

kubectl exec -it sleep-557747455f-q99cz -c sleep -- curl http://httpbin.org/ip

The log should output a line saying curl command access, which indicates that this outbound access is managed by istio egress gateway. The log line looks like below and in the example the IP address 10.244.2.4 is the Pod IP address.

[2021-11-22T01:22:47.547Z] "GET /ip HTTP/2" 200 - via_upstream - "-" 0 45 818 816 "10.244.2.9" "curl/7.80.0-DEV" "9ec0a193-77f0-90a5-b132-21c3259307da" "httpbin.org" "3.216.167.140:80" outbound|80||httpbin.org 10.244.2.4:42372 10.244.2.4:8080 10.244.2.9:52710 - -

Looking at the virtual service declaration, you might noticed multiple routes in the virtual service. The first route (mesh) tells sidecars to route request to egress gateway. The second route (istio-egressgateway) tells egress gateway to route traffic to external service as destination.

In real life, there’s more to consider. For example, we usually want the host to allow on egress to be a wild card host such as *.api.google.com. While the hosts field in virtual service can be wild card, the route destination in virtual service has to be a list of fully qualified domains. The other situation is the destination website may issue an HTTP redirect to a different host, which also needs to be registered as a destination. These details can add up to administrative effort.

Egress example

An other real life scenario is the use of authorization policy. Authorization policy rules use SPIFEE identity of a Pod. If a Pod initiates egress connection to destination host in TLS, then the Pod’s SPIFFEE identity and its identity for external TLS connection are in conflict.

With Azure AKS

The steps above also works on other cloud platform such as AKS. This is an AKS cluster that I tested the steps. The Terraform code provisions a private cluster, as well as a Bastion Host. From the Bastion host, we can perform the same steps as above.

When Istio gets installed, it will created a Service object called istio-ingressgateway in the istio-system namespace. The service is of load balancer type and by default will create a public load balancer in Azure AKS.

The test steps will also work for private load balancers. To tell Azure to create private load balancer on a specific subnet, the Service object needs to include annotation. We have to customize Istio installation in order to do that, using the following overlay file (azoverlay.yaml):

apiVersion: install.istio.io/v1alpha2
kind: IstioControlPlane
metadata:
 namespace: istio-operator
 name: custom-istiocontrolplane
spec:
 profile: demo
 values:
 gateways:
 istio-ingressgateway:
 serviceAnnotations:
 service.beta.kubernetes.io/azure-load-balancer-internal: "true"
 service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "suitable-porpoise-lb-subnet"

This is inspired by this post. Then the command to install Istio becomes:

istioctl install -f azoverlay.yaml -y --verify --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY

This concludes the hand-on guide for Istio ingress and egress labs. I will need to write another post explaining the related concepts.