cloud native on Digi Hunch

GraphQL and gRPC

Fri, 07 Oct 2022 14:49:00 -0400

Big Picture

For inter-process communication at a high level, the two styles are asynchronous and synchronous styles:

Asynchronous event-driven style: involving an event broker as a middle man.
Synchronous request-response style: including several families of technologies:
- RPC (Remote Procedure Call):
  - CORBA (Common Object Request Broker Architecture)
  - Java RMI (Remote Method Invocation)
- SOAP (Simple Object Access Protocol)
- REST (Representational State Transfer)
- gRPC
- GraphQL
- Apache Thrift

RPCs, built on top of TCP/UDP, are usually complex to implement. SOAP improved it and can operate on HTTP. Many large companies today still used SOAP for message exchange. However, it has a limitation with complex format and specifications for XML messaging, giving rise to REST. REST is not a standard, but rather a loosely defined architectural style. Its payload can be in any format (XML, JSON, etc) specified in the header. As long as the API conforms to certain guidelines (criteria outlined in this article), we can consider the API RESTful. REST has been steadily replacing SOAP in the past few years. In this post, I start with REST, then dive into gRPC and GraphQL.

REST

The de facto method of building microservices using REST architectural style is use HTTP protocol with JSON format payload. JSON format is human readable, but not optimized for machine-to-machine communication. So there is some room for compression. To emulate a web request in REST, one can use curl, a common utility to emulate any HTTP client.

REST has its shortcomings. For example, the interface between REST client and server is not strongly typed. You can choose to use OpenAPI/Swagger specification to define types but it is still not tightly integrated. There is no enforcement on the format of the payload either. RESTful services are quite bulky, inefficient, and error-prone. gRPC and GraphQL emerged to address different challenges with REST. GraphQL operates on HTTP and we can view it as a layer on top of REST in a broad sense. gRPC on the other hand, operates on HTTP /2, and thereby inherits many advantages from it.

HTTP /2

HTTP/2 is the second major version of HTTP. It overcomes some issues with HTTP/1.1 on security, speed, etc. This post is a good rundown of the difference between HTTP /2 and HTTP /1.1, which account for many of the advantages of gRPC. A thorough discussion on the differences between the two HTTP versions is beyond what this post can cover. One of the important difference with HTTP/2, is that all communication between a client and server is performed over a single TCP connection that can carry any number of bidirectional flows of bytes. This makes gRPC a high-performance RPC framework. In HTTP/2, the key concepts to understand are:

Stream: a bidirectional flow of bytes within an established connection. A stream may carry one or more messages;
Frame: the smallest unit of communication in HTTP/2. Each frame contains a frame header, which at a minimum identifies the stream to which the frame belongs.
Message: a complete sequence of frames that map to a logical HTTP message that consists of one or more frames.

The request message is always triggered by the client. During the interaction, the client and server break down the message into frames, interleave them, and then reassemble them on the other side. In this way HTTP /2 multiplex the messages, and enables the following communication patterns:

Simple RPC: a single request and a single response in the communication;
Server Streaming RPC: a single request and message followed by multiple response messages;
Client streaming RPC: client sends multiple messages and the server replies with one response message;
Bi-directional RPC: client setups connection by sending header frames. Once connection is established, both client and server send messages simultaneously without waiting for the other to finish;

The streaming communication patterns fundamentally improves performance, enabling the duplex streaming capability for gRPC.

gRPC

At the protocol level, gRPC has the following advantages over REST:

well-defined service interface and schema
strongly typed data
duplex streaming (thanks to HTTP2)
built-in commodity features (e.g. authentication, encryption, resiliency, service discovery, etc)

I previously touched on gRPC protocol in the context of Envoy and etcd. Envoy makes use of gRPC for its control plane, where it fetches configuration from management server(s) and in filters, such as for rate limiting or authorization checks. Etcd store implements gRPC protocol for client utility to communicate with.

Since the typical use case of gRPC is internal communication, I have not been able to find a playground online. To get a taste of gRPC client-server interaction, just play with etcd on MacOS. gRPC is language neutral. To start developing, refer to the tutorial in different languages (e.g. Golang, Python). When developing a gRPC application, the first thing to do is define a service interface in IDL (interface definition language). gRPC uses protocol buffers as the IDL to define the service interface. Protocol buffers are a language-agnostic, platform-neutral, extensible mechanism to serializing structured data. Using that service interface definition, we can generate the server-side code known as a server skeleton. Also you can generate the client-side code, known as a client stub. The methods that you specify in the service interface definition can be remotely invoked by the client side as easily as making a local function invocation.

gRPC also has some disadvantages. Currently, the ecosystem is still small. When we have a service interface, we have to maintain the interface across versions.

GraphQL

In most of the use cases of gRPC and GraphQL, GraphQL works for external-facing services/APIs while internal services backing the APIs are implemented using gRPC. Honeypot created good documentary for GraphQL available here.

We use GraphQL for external facing services because it gives API client the ability to query. An SQL query allows client to filter requested data based on conditions, and define the interested columns in the data return. This capability is missing in the REST style guideline. One may choose to implement their RESTful service to support their client’s query requirement, GraphQL standardizes this capability with typed data, thereby prevents unnecessary network round trips, over- and under-fetching of data.

The official site is a good reference for learning. One needs to know concepts around queries and mutations, schema and types (scalars, variable, fragment, interfaces, unions) to understand how query works.

This page lists a number of publicly available services in GraphQL. For example, country information service is available here. On the web page you can put in a query like this:

query myCountry {
 countries(filter:{code:{in:"CA"}}) {
 code,
 name,
 capital
 }
}

The query (with the name of myCountry) above asks to return entries with countries as type. Then it filters the result by the condition that the country code must include “CA”. The return should include code, name and capital columns. The web page looks like this:

GraphQL is on HTTP, so I can emulate the call with curl and get the same result:

curl --request POST https://countries.trevorblades.com/ \
--header 'Content-Type: application/json' \
--data-raw '{
 "query" : "query myCountry { countries (filter: { code: { in: \"CA\" } }) {code name capital} }"
}'

The return prints the same result:

The document has a page on library options for different languages, such as Graphene and Ariadne for Python. You can use Ariadne and Flask to build GraphQL API as this post suggests.

The example on this page has a better example that highlights how GraphQL allows client to ask multiple questions at the same time, and how the response includes different pieces of answers. GraphQL backend often needs to connect to different downstream system. This requires a proxy service to connect to disparate data sources. Examples of such services with such capabilities include Apollo, AWS AppSync, or ApiGee.

Summary

For API protocols, we often compare among REST, gRPC and GraphQL. REST is loosely defined and widely adopted for the past few years. The use case of REST diverged into two areas: external facing API and internal service-to-service communication. gRPC and GraphQL are relatively new and they each are good for one of the use cases.

Microservice pattern based on gRPC and GraphQL (source: “gRPC up & running” by Indrasiri & Kuruppu)

The gRPC protocol on HTTP2 is often used in internal communication between microservices. GraphQL offers query capability and is therefore often used to face external client.

Kubernetes Storage on Azure 3 of 3 – Ceph by Rook

Fri, 26 Aug 2022 19:43:00 -0400

In the last two posts, I covered the native storage options on Azure Kubernetes Service, as well as Portworx as an example of a proprietary Software Defined Storage (SDS) solution. There are also a number of open-source alternative SDS solutions. Ceph has nearly a decade of history from prior to containerization, and is the most widely adopted storage platform. In this post, we continue to explore Ceph as an open-source storage solution on Azure Kubernetes.

Ceph by Rook

Ceph is an open-source SDS platform for distributed storage on a cluster and provides object, block and file storage. Installation of Ceph SDS can be complex, especially on Kubernetes platform. Rook is a graduated CNCF project to orchestrate storage platform. Rook by itself is not SDS and it supports:

Ceph: configure a Ceph cluster. Think of this as the equivalent of cephadm on Kubernetes platform.
NFS: configure an NFS server. Think of this as the equivalent of nfsd daemon on Kubernetes platform.
Cassandra: an operator to configure a Cassandra database cluster. It is now deprecated.

We play with Rook Ceph. I also refer to it as Ceph by Rook. The contribution of Rook project is it simplifies the installation as a matter of declaring custom resources using CRDs. Here are some high-level CRDs to know:

CephCluster: creates a Ceph storage cluster
CephBlockPool: represents a block pool
CephFilesystem: represents a file system
CephObjectStore: represents an object store
CephNFS: spins up a NFS Ganesha server to export NFS shares of a CephFilesystem or CephObjectStore.

As with typical Kubernetes resources in controller pattern, Ceph by Rook needs an operator along with custom resources. We can use YAML manifest for both of them, and the manifests are usually very tediously long. We can also use Helm to install both of them, by providing a value file. Now we will install Ceph on AKS.

Install Ceph Operator on AKS

The steps are influenced by two relevant posts (here and here). However, I’ve incorporated the cluster configuration in the Azure directory of the cloudkube project, a modular Terraform template to configure AKS cluster and facilitate storage configuration. The node group and instance sizes are selected to be just enough to run a ceph POC cluster with minimum cost. One of the node groups is tainted with storage-node, as if the following command were run:

kubectl taint nodes my-node-pool-node-name storage-node=true:NoSchedule

You will only need to taint the nodes with the command above if you choose not to use the cloudkube template. The taint ensures that only Pods with corresponding toleration and effect can be scheduled to those nodes.

We use Helm to install Rook Operator. We need a value file (e.g. rook-ceph-operator-values.yaml) with content as below:

# https://github.com/rook/rook/blob/master/Documentation/Helm-Charts/operator-chart.md
crds:
 enabled: true
csi:
 provisionerTolerations:
 - effect: NoSchedule
 key: storage-node
 operator: Exists
 pluginTolerations:
 - effect: NoSchedule
 key: storage-node
 operator: Exists
agent:
 # AKS: https://rook.github.io/docs/rook/v1.7/flexvolume.html#azure-aks
 flexVolumeDirPath: "/etc/kubernetes/volumeplugins"

Then we install the operator with Helm:

helm install rook-ceph-operator rook-ceph --namespace rook-ceph --create-namespace --version v1.9.6 --repo https://charts.rook.io/release/ --values rook-ceph-operator-values.yaml
kubectl -n rook-ceph get po -l app=rook-ceph-operator

After installing the operator, we check the Pod status to make sure it is running. Then we can install the actual Ceph Cluster in one of the two ways. We can declare a CephClusterCRD ourself, or we can use Helm again to declare the CRD. Helm Chart gives us a lot of useful default values and saves us from editing a large body of YAML manifest.

Install Ceph CR on AKS

We use Helm to install CephCluster CRD. We create a value file (e.g. rook-ceph-cluster-values.yaml) with content as below:

# https://github.com/rook/rook/blob/master/Documentation/Helm-Charts/ceph-cluster-chart.md
operatorNamespace: rook-ceph
toolbox:
 enabled: true
cephObjectStores: [] # by default a cephObjectStore will be created. Setting this to null disables it
#cephBlockPools: # by default a cephBlockPool will also be created with default values
#cephFileSystems: # by default a cephFileSystem will also be created with default values

cephClusterSpec:
 mon:
 count: 3
 volumeClaimTemplate:
 spec:
 storageClassName: managed-premium
 resources:
 requests:
 storage: 10Gi
 resources:
 limits:
 cpu: "500m"
 memory: "1Gi"
 requests:
 cpu: "100m"
 memory: "500Mi"
 dashboard:
 enabled: true
 storage:
 storageClassDeviceSets:
 - name: set1
 # The number of OSDs to create from this device set
 count: 3
 # IMPORTANT: If volumes specified by the storageClassName are not portable across nodes
 # this needs to be set to false. For example, if using the local storage provisioner
 # this should be false.
 portable: false
 # Since the OSDs could end up on any node, an effort needs to be made to spread the OSDs
 # across nodes as much as possible. Unfortunately the pod anti-affinity breaks down
 # as soon as you have more than one OSD per node. The topology spread constraints will
 # give us an even spread on K8s 1.18 or newer.
 placement:
 topologySpreadConstraints:
 - maxSkew: 1
 topologyKey: kubernetes.io/hostname
 whenUnsatisfiable: ScheduleAnyway
 labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - rook-ceph-osd
 tolerations:
 - key: storage-node
 operator: Exists
 preparePlacement:
 tolerations:
 - key: storage-node
 operator: Exists
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: agentpool
 operator: In
 values:
 - storagenp
 topologySpreadConstraints:
 - maxSkew: 1
 # IMPORTANT: If you don't have zone labels, change this to another key such as kubernetes.io/hostname
 topologyKey: topology.kubernetes.io/zone
 whenUnsatisfiable: DoNotSchedule
 labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - rook-ceph-osd-prepare
 resources:
 limits:
 cpu: "500m"
 memory: "4Gi"
 requests:
 cpu: "500m"
 memory: "2Gi"
 volumeClaimTemplates:
 - metadata:
 name: data
 spec:
 resources:
 requests:
 storage: 100Gi
 storageClassName: managed-premium
 volumeMode: Block
 accessModes:
 - ReadWriteOnce

During the cluster provisioning, there will be a number of preparing Pods. We want those Pods to run on nodes with label agentpool=storagenp. In real life, we need to orchestrate where to run each workload, by restricting the nodes to schedule certain types of workload.

Then we can install the cluster using Helm:

helm install rook-ceph-cluster rook-ceph-cluster --namespace rook-ceph --create-namespace --version v1.9.6 --repo https://charts.rook.io/release/ --values rook-ceph-cluster-values.yaml

After running the Helm install, it may take as long as 15 minutes for all resources to settle. Watch the Pod status in rook-ceph namespace. At the end, make sure that the cluster is created successfully:

kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get CephCluster
NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL
rook-ceph /var/lib/rook 3 15m Ready Cluster created successfully HEALTH_OK
kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get cephBlockPools
NAME PHASE
ceph-blockpool Ready
kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get cephFileSystems
NAME ACTIVEMDS AGE PHASE
ceph-filesystem 1 20m Ready

In my case it took 15 minutes before the cluster comes up as created successfully. You should notice that two storage classes were also created as a part of the install. It however did not create a storage class or CRD for object storage, because we explicitly disabled it in the Helm value file by setting cephObjectStores value to null.

Dashboard

We enabled dashboard. To configure the dashboard view properly, we would need an ingress. For a quick view here, we can play port forwarding tricks. First we fetch the admin password for use in the next step. Then expose the dashboard to the bastion host:

$ kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath='{.data.password}' | base64 -d
$ kubectl -n rook-ceph port-forward svc/rook-ceph-mgr-dashboard 8443:8443

Since I don’t have UI on the bastion host, I use the port forwarding trick again from my own MacBook. Start a new terminal and SSH to the bastion host with port-forwarding switch:

$ ssh -L 8443:localhost:8443 kubeadmin@20.116.132.8

The command above suppose the public IP of the bastion host is 20.116.132.8. Then from my MacBook I can browse to localhost:8443 (with Safari browser which gives me the option to bypass certificate error). At the web portal, provide username (admin) and password (as retrieved above):

Ceph console for Kubernetes

Apart from the dashboard, we can also use ceph admin tool from a toolbox pod, following this instruction. For monitoring, Ceph by Rook can expose metrics for Prometheus to scrape.

Performance

With default ceph configuration on AKS, I ran quick performance test using kube-str . The result is as follows:

	read_iops	write_iops	read_bw	write_bw
ceph-block	IOPS=464.507294 BW(KiB/s)=1874	IOPS=243.296143 BW(KiB/s)=989	IOPS=509.928162 BW(KiB/s)=65797	IOPS=248.530762 BW(KiB/s)=32338
ceph-filesystem	IOPS=438.701324 BW(KiB/s)=1770	IOPS=226.270660 BW(KiB/s)=920	IOPS=405.936340 BW(KiB/s)=52456	IOPS=208.869293 BW(KiB/s)=27229

The metrics reflects performance under default configuration. It should not be considered as the best performance that Ceph can deliver on Azure Kubernetes.

Summary

I discussed three storage options for Azure Kubernetes but the idea applies to other Kubernetes platform hosted on a CSP. The native storage has significant limitation. NFS has latency. Block storage does not address high availability at the storage layer. Portworx and LINSTOR fill that gap as a commercial solution. Ceph is based on Object storage.

Kubernetes Storage on Azure 2 of 3 – Portworx

Fri, 12 Aug 2022 15:23:00 -0400

In the previous post, we have discussed built-in storage classes on Azure Kubernetes. Further to that, we will examine some third-party software defined storage (SDS) options that are compatible with Azure Kubernetes Service in this post. Then we take Portworx on Azure as an example. Although, these options are specific to Azure, most of the players also have solutions for other managed Kubernetes platforms. Also, the methodology to study storage options remain the same regardless of cloud service provider.

In fact, I touched on software defined storage (SDS) in the context of general non-containerized workload in a separate post. In short, storage providers decouple the SDS appliance from the full storage solution in order to lower the cost and increase flexibility. To the storage consumer (e.g. a process running on Linux OS), SDS can present a block disk or file system. There are also SDS solution that can host your own object storage and we will discuss that later. In fact, SDS has gained significant popularity in recent years. For example, the report “Validating Software-Defined Storage Operating Models for the Enterprise” by archiectingit divided the evolution into four phases and cited that Gartner predicts the SDS revolution to reach 50% of the storage market by 2024, from 15% in 2020.

This post discusses SDS in the context of container storage. Then we will install Portworx on Azure Kubernetes.

SDS for Kubernetes

Many SDS appliances also developed the capability to present storage volumes to containerized workload. I put together a list of SDS products that works on Azure with their supported access modes:

Solution	Licence	Development and Support	Access Mode
Portworx	Closed source. Free Essential tier. Enterprise features on License.	Commercially supported by PureStorage	RWO, RWX
Ceph by Rook	Open source. Rook is a graduated CNCF project.	Developed and commercially supported by Red Hat, Canonical and SoftIron	RWO, ROX, RWX
OpenEBS	Open source. Sandbox CNCF project.	Developed and commercially supported by MayaData et al.	RWO
Longhorn	Open source. Incubating CNCF project.	Originally developed by Rancher, and commercially supported by SUSE	RWO
StorageOS	Closed source. License required.	Commercial support by Ondat.	RWO
LINBIT	Open-source with enterprise plans	Enterprise support

RWO is the most commonly supported mode. The report “Performance Benchmarking Cloud Native Storage Solutions for Kubernetes” makes a comparison of performance among some of the options in early 2021. Another potentially opinionated comparison list is by LINBIT.

Portworx is a leading player with commercial SDS solution and I will test its free Essential tier in the rest of this post. Ceph is one of the most mature leading open-source offering and I will test it in the next post.

For Portworx, we can use the terraform template cloudkube for Azure. The template assigns the kubelet’s managed identity as contributor of the node resource group. The template also creates a bastion host with direct SSH access to the nodes.

Portworx Operator on Azure

We use Portworx Operator to configure storage cluster. Portworx has an instruction for AKS but it is not tailored to specific identity model. For simplicity, use my cloudkube Terraform template to create the AKS cluster, and skip the “Prepare Your AKS Platform” page. Instead, follow the “Deploy Portworx using Azure managed identity on new AKS cluster” page starting at step 7.

At that step, we need to create a secret with the client ID of the managed identity for node agent. The terraform template outputs the BYO identity’s client ID. After cluster creation, we simply SSH to the bastion host and create the secret using the output.

To install Porworx using operator, we can follow a wizard in PX-central. If this is the first time, we need to create an account and log in to the portal. If this is not the first time and you have previously created a cluster, you need to detach that cluster by going to Profile from bottom left corner on the portal page.

Follow the guide in “Install Portworx on AKS using the Operator“. In the wizard, click on “Portworx Essentials” for free tier, or “Portworx Enterprise” for the 30-day trial. Then select operator with latest version. In the rest of the wizard steps, select options applicable to Azure environment. The last step will present two kubectl commands to install operator and install the CR. Run the command to install operator and verify result:

kubectl apply -f 'https://install.portworx.com/2.9?comp=pxoperator'
kubectl -n kube-system get deployment portworx-operator

Portworx Custom Resource

To install the CR, we need to customize the given manifest in order to use our managed identity. We can download the YAML manifest (portworx_essentials.yml) and modify it in text editor. As the page “Deploy Portworx using Azure managed identity on new AKS cluster” suggest at step 9: in the env section, remove the AZURE_CLIENT_SECRET and AZURE_TENANT_ID sections but keep the AZURE_CLIENT_ID section. My CRD declaration looks like this:

# SOURCE: https://install.portworx.com/?operator=true&mc=false&kbver=&oem=esse&user=myuserid&b=true&kd=type%3DPremium_LRS%2Csize%3D150&s=%22type%3DPremium_LRS%2Csize%3D150%22&c=my-very-long-px-cluster-id&aks=true&stork=true&csi=true&mon=true&tel=false&st=k8s&promop=true
kind: StorageCluster
apiVersion: core.libopenstorage.org/v1
metadata:
 name: my-very-long-px-cluster-id
 namespace: kube-system
 annotations:
 portworx.io/install-source: "https://install.portworx.com/?operator=true&mc=false&kbver=&oem=esse&user=myuserid&b=true&kd=type%3DPremium_LRS%2Csize%3D150&s=%22type%3DPremium_LRS%2Csize%3D150%22&c=my-very-long-px-cluster-id&aks=true&stork=true&csi=true&mon=true&tel=false&st=k8s&promop=true"
 portworx.io/is-aks: "true"
 portworx.io/misc-args: "--oem esse"
spec:
 image: portworx/oci-monitor:2.10.2
 imagePullPolicy: Always
 kvdb:
 internal: true
 cloudStorage:
 deviceSpecs:
 - type=Premium_LRS,size=150
 kvdbDeviceSpec: type=Premium_LRS,size=150
 secretsProvider: k8s
 stork:
 enabled: true
 args:
 webhook-controller: "true"
 autopilot:
 enabled: true
 monitoring:
 prometheus:
 enabled: true
 exportMetrics: true
 featureGates:
 CSI: "true"
 env:
 - name: AZURE_CLIENT_ID
 valueFrom:
 secretKeyRef:
 name: px-azure
 key: AZURE_CLIENT_ID
---
apiVersion: v1
kind: Secret
metadata:
 name: px-essential
 namespace: kube-system
data:
 px-essen-user-id: aaaabbbbccccddddmyverylongpxessenuserid
 px-osb-endpoint: ssssssyyyyyyyzzzzzzmyverylongpxosbendpoint

In this custom resource manifest, I specify to get the cluster to create disk from Azure, even though it has the capability of using an existing disk. Then apply the CRD manifest:

 kubectl apply -f portworx_essentials.yml

This will take up to 10 minutes to create several related resources. There are several check points to ensure the cluster is created successfully. Here are some useful commands:

kubectl -n kube-system get po # all Pods related to portworx should be ready and running, especially the portworx-api ones
kubectl -n kube-system get storagecluster # the status should report online
PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}') 
# get the name of one of the portworx Pod for storage cluster
kubectl -n kube-system exec -c portworx -it $PX_POD --tty --stdin -- /opt/pwx/bin/pxctl status 
# should report "Status: PX is operational" with valid license loaded
kubectl get sc # portworx related storage classes are available

The pre-built storage classes are not CSI based. However, CSI should be automatically enabled in recent operator versions. We should create our own CSI storage classes and PVCs using our own storage classes. Here are two examples:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: px-csi-database
provisioner: pxd.portworx.com
parameters:
 repl: "2"
 priority_io: "high"
 io_profile: "db"
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: px-csi-artifact
provisioner: pxd.portworx.com
parameters:
 repl: "2"
 priority_io: "medium"
 io_profile: "sequential"

That is a bare minimum Portworx installation. Check out their documentation for the full features. With this minimum install we can go to the section “Validate Persistent Storage” from the previous post to validate the persistent volume.

For troubleshooting purpose, pxctl is the utility and it is available on Portworx Pods.

Performance

We care not only the functionality, but also the performance. So I ran a quick performance test using kube-str, using all default configuration. The result is as follows:

	read_iops	write_iops	read_bw	write_bw
px-csi-database	IOPS=969.614136 BW(KiB/s)=3894	IOPS=729.698059 BW(KiB/s)=2935	IOPS=1172.772827 BW(KiB/s)=150639	IOPS=691.626526 BW(KiB/s)=89053
px-csi-artifact	IOPS=780.681946 BW(KiB/s)=3139	IOPS=682.522766 BW(KiB/s)=2746	IOPS=773.548584 BW(KiB/s)=99549	IOPS=659.015320 BW(KiB/s)=84890

Note that those numbers reflect performance under default configuration, and they should not be considered as the best performance that Portworx can deliver on Azure Kubernetes. Before moving to production, it is important to establish your own test parameters that best represents the container workload, and then iterate through different parameters for the storage class based on the requirement and performance output.

Kubernetes Storage on Azure 1 of 3 – built-in storage and NFS

Sun, 31 Jul 2022 15:22:00 -0400

In the previous post, we understand that to host stateful workload, we need to manage persistent storage to the Kubernetes platform. In this post, I will explore the different storage options. These options are specific to Azure Kubernetes service. However, the principals apply to any Kubernetes platform regardless of cloud vendor.

In another old post, I discussed in-tree and CSI storage classes, and from a developer’s perspective, how to mount volumes statically and dynamically once the storage class is available. Here in this post we are concerned with how to make storage classes available, from a platform specialist perspective.

Storage in Azure Kubernetes

As soon as the Azure Kubernetes cluster is launched, a number of built-in storage classes are available. Unlike third-party storage classes, they do not require kubelet identity to be contributor for node resource group. As discussed we shall use CSI based storage classes.

managed-csi
managed-csi-premium
azurefile-csi
azurefile-csi-premium

The main difference between them is the backing technology. However, form Kubernetes workload’s perspective, the Pods as storage consumers are concerned with the access mode instead of backing technology. Here are supported access modes:

ReadWriteOnce: read-write by a single node
ReadOnlyMany: read only by many nodes
ReadWriteMany: read-write by many nodes
ReadWriteOncePod: new in Kubernetes 1.22 to restrict volume access to a single Pod

The storage classes managed-csi and managed-csi-premium support ReadWriteOnce. The storage classes azurefile-csi and azurefile-csi-premium support ReadWriteMany.

Apart from these built-in options, Azure also suggests a few more options based on other types of Azure resources. For example, AKS can integrate with HPC cache and it for HPC. We can also self-manage a virtual machine configured as NFS server, and use the NFS subdir external provisioner to configure storage class. Despite of the overhead with managing a VM, you have more configurability. My previous client reports that they gain better performance than the built-in options.

Another alternative is Azure NetApp Files. However, being a full enterprise grade solution (similar to FSx ONTAP), Azure NetApp Files costs an arm and a leg. In this comparison, it cost 60 to 100 times as the cost by built-in options.

Another option is Azure Ultra Disk, which needs to be enabled at cluster level. You can provision performance target (DiskIOPSReadWrite and DiskMBpsReadWrite) in the storage class. Ultra Disk is a good middle ground between the pricey NetApp files and the less performant built-in options.

Terraform Template for AKS

To explore the storage options, I use my own terraform template to create an AKS cluster. The template is in the azure directory of the cloudkube repo. The template consists a few configurations with Azure Kubernetes to facilitate storage configuration. First, it configures an SSH key pair to use between the bastion host and the node. Users can SSH to Kubernetes nodes from bastion host as soon as terraform apply is completed. Second, the third party storage options installed after the cluster creation need their Pod to instruct Azure to create Azure disks. This requires that a Kubernetes node agent have the permission to provision resources in the node resource group. This is important to understand because there are a couple of managed identities at play (refer to this post) when building an AKS cluster.

In Azure Kubernetes, it is the managed identity of kubelet, that needs to have contributor permission over the resource group for the nodes (not the one for the AKS cluster itself). A managed identity is expressed by a client ID, an object ID (aka principal ID), and the identity ID. We can find them out with an AZ CLI command as below:

We can also tell that the kubelet managed identity represents node agent, by connecting to a node and looking at the argument (kubernetes.azure.com/kubelet-identity-client-id) of kubelet process:

In the template, I also chose to designate the same BYO identity for both the cluster and for kubelet (node agent), in order to minimize my requirement on permission. If I had left it with a system assigned identity for node agent, I would have to assign that identity as a contributor for the node resource group, either as a user, or via Terraform’s identity. Either way, it is beyond what a Contributor is allowed to do.

Benchmarking with kubestr

I used fio utility for storage benchmarking from virtual machines. However, fio utility is not for container. For fio testing on Kubernetes, I’d have to use a Docker image, and test with target volume attached. Fortunately, the Kasten team shared their initiative in the open source project Kubestr. The kubestr release is available as an executable on common platforms. It connects to the cluster the same way as kubectl and here is a demo. To begin with, download the utility to bastion host, and run it without any argument, which prints the storage classes and volume snapshot classes:

curl -L -o kubestr.tar.gz https://github.com/kastenhq/kubestr/releases/download/v0.4.31/kubestr_0.4.31_Linux_amd64.tar.gz
tar -xvf kubestr.tar.gz && rm kubestr.tar.gz && chmod +x kubestr

./kubestr # if kubectl is configured, this command will print out the details of storage classes and volume snapshot classes

In addition to outputting details, it is also very simple to perform storage benchmarking with kubestr. All we need to do is giving it the storage class name and it will run four tests by default with common global options (ioengine=libaio verify=0 direct=1 gtod_reduce=1). The four tests are:

JobName	block_size	filesize	iodepth	rw
read_iops	4k	2G	64	randread
write_iops	4k	2G	64	randwrite
read_bw	128k	2G	64	randread
write_bw	128k	2G	64	randwrite

During each test, it measures and reports IOPS and bandwidth (throughput). If your I/O profile falls out of the four jobs, you can even customize your test by supplying a fio config file. For example, you need a longer test duration, or you need a larger total size for the test. Before the test, kubestr automatically mount their test volumes using the storage class being tested.

./kubestr fio -s my-storage-class # benchmarking a storage class

For read_iops and write_iops, we mainly look at the IOPS. For read_bw and write_bw, we mainly look at the bandwidth. The iops and bw based on samples are reported as first line of result, followed by min, max and average.

Metrics

With kubestr I ran a performance test amongst the native storage classes with Azure Kubernetes Service, with results as below:

	read_iops	write_iops	read_bw	write_bw
managed-csi	IOPS=314.729797 BW(KiB/s)=1275	IOPS=297.071136 BW(KiB/s)=1204	IOPS=315.311188 BW(KiB/s)=40887	IOPS=261.048645 BW(KiB/s)=33941
managed-csi-premium	IOPS=493.395844 BW(KiB/s)=1990	IOPS=426.731812 BW(KiB/s)=1723	IOPS=455.950348 BW(KiB/s)=58894	IOPS=422.888855 BW(KiB/s)=54662
azurefile-csi	IOPS=259.333282 BW(KiB/s)=1053	IOPS=283.985779 BW(KiB/s)=1152	IOPS=240.447403 BW(KiB/s)=31298	IOPS=230.689804 BW(KiB/s)=30048
azurefile-csi-premium	IOPS=394.044739 BW(KiB/s)=1586	IOPS=371.181793 BW(KiB/s)=1494	IOPS=380.360535 BW(KiB/s)=49018	IOPS=491.313446 BW(KiB/s)=63310

These numbers are based on default test parameters (e.g. 100Gi PVC size). The numbers indicate that block storage generally performs better in default setting. So we should use managed disk instead of azure file unless multiple pods needs to access the same volume.

Validate Persistent Storage

We can use mysql as a quick and dirty test of storage classes. We can deploy the following manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: mysql-pvc
spec:
 storageClassName: managed-csi-premium # the storage class being tested
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 5Gi
---
apiVersion: v1
kind: Secret
metadata:
 name: mysql
data:
 password: eHl6  # base64 code of xyz
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: mysql
 labels:
 app: mysql
spec:
 replicas: 1
 selector:
 matchLabels:
 app: mysql
 template:
 metadata:
 labels:
 app: mysql
 spec:
 containers:
 - image: mysql:5.6
 name: mysql
 env:
 - name: MYSQL_ROOT_PASSWORD
 valueFrom:
 secretKeyRef:
 name: mysql
 key: password
 ports:
 - containerPort: 3306
 name: mysql
 volumeMounts:
 - name: mysql-persistent-storage
 mountPath: /var/lib/mysql
 volumes:
 - name: mysql-persistent-storage
 persistentVolumeClaim:
 claimName: mysql-pvc
---
apiVersion: v1
kind: Service
metadata:
 name: mysql-service
 labels:
 service: mysql
spec:
 selector:
 app: mysql
 ports:
 - name: tcp-mysql
 protocol: TCP
 port: 3306
 targetPort: 3306

Once the Pod has been created, then we use a throw-away Pod to connect to mysql service and build some data:

$ kubectl run mysql-cli --rm -i --tty --image imega/mysql-client -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ # mysql --host=mysql-service.default.svc.cluster.local --user=root --password=xyz
Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 9 Server version: 5.6.51 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]>

From the mysql shell, we create a test database with dummy data populated:

CREATE DATABASE `testdb`; 
USE testdb; 
CREATE TABLE IF NOT EXISTS tasks ( 
 task_id INT AUTO_INCREMENT PRIMARY KEY, 
 title VARCHAR(255) NOT NULL, 
 description TEXT, 
 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP 
) ENGINE=INNODB; 
INSERT INTO tasks (title, description) VALUES ('Job A','Morning Standup');
INSERT INTO tasks (title, description) VALUES ('Job B','Latte with two shots of espresso');
INSERT INTO tasks (title, description) VALUES ('Job C','Coding coding and coding');
INSERT INTO tasks (title, description) VALUES ('Job D','git commit');

Each SQL command should return with number of rows affected and then we can exit the MySQL shell and Pod shell. Once we exit out of the Pod shell, the Pod is deleted. We can re-connect to validate the data are still present:

$ kubectl run mysql-tester --rm -i --tty --image imega/mysql-client -- mysql --host=mysql-service.default.svc.cluster.local --user=root --password=xyz --database testdb --execute='SELECT * FROM tasks;'
If you don't see a command prompt, try pressing enter.
Got error: Access denied for user 'root'@'147.206.3.15' (using password: NO)
+---------+-------+----------------------------------+---------------------+
| task_id | title | description | created_at |
+---------+-------+----------------------------------+---------------------+
| 1 | Job A | Morning Standup | 2022-06-22 20:29:12 |
| 2 | Job B | Latte with two shots of espresso | 2022-06-22 20:29:12 |
| 3 | Job C | Coding coding and coding | 2022-06-22 20:29:12 |
| 4 | Job D | git commit | 2022-06-22 20:29:13 |
+---------+-------+----------------------------------+---------------------+
Session ended, resume using 'kubectl attach mysql-tester -c mysql-tester -i -t' command when the pod is running
pod "mysql-tester" deleted

The output validates the persistent storage of the data. When testing a different storage class, simply start over with a different storage class specified in the PVC part of the manifest.

Summary

This post focuses on storage options for persistent volumes on Azure Kubernetes service. I use my own Terraform template with some custom configuration. I also covered kubestr as benchmarking tool and ran it against the built-in storage classes.

In some situations, the built-in options do not suit your needs. For example, you might want your application to use persistent volume in a consistent way across multiple cloud vendors. We will have to resort to third-party software defined storage layer. In the next post, I will explore a couple of SDS-based options, namely Portworx and Ceph. The custom configurations in the Terraform template will be helpful when we configure Portworx.

Intro to Ceph storage

Thu, 21 Jul 2022 18:55:00 -0400

Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability. In this post, I will introduce Ceph and explain how it stands out from traditional enterprise storage technology.

Software defined storage

In the realm of enterprise storage, I discussed PowerScale (Isilon) from Dell EMC, and touched on ONTAP by NetApp as an alternative. These solutions usually include both enterprise grade hardware, and the software layer that manages those expensive hardware. As the competition with cloud storage arises, those vendors start to decouple the software layer from the hardware to sell them separately. As a result, clients have the options to use commodity hardware. On the other hand, the software layer is built to be more accommodative to different hardware options. Eventually, the software layer evolves into Software Defined Storage (SDS) with the purpose of supporting cheaper storage hardware.

This table shows the full solution offering and SDS offering from NetApp and Dell EMC:

	Full solution offering	SDS offering
NetApp	ONTAP	ONTAP Select
EMC	PowerScale	PowerFlex

It is not easy to make a proprietary SDS appliance support commodity hardware. For example, PowerFlex currently supports (and bundles with) DELL’s commodity hardware only. It is most likely an involuntary move. Then, why would these commercial providers even be motivated to support a broader range of hardware by moving to SDS? It is because they face fierce competition from open-source SDS technologies, which were born to support commodity hardware. In this family of technologies, Ceph is a rising star. This family also includes other technologies such as Gluster and HDFS.

Note that the performance of a storage based on SDS still has to do with the underlying hardware. Therefore, comparing Ceph storage with PowerScale is apple to orange, without identical storage hardware. Now that we decoupled SDS and hardware, let’s take a look at two important aspects of SDS: the distributed technology to manage hardware, and the interface it provides to storage clients.

Distributed storage

The reason to use an SDS layer to manage hardware in a distributed architecture is for better scalability and high availability. The soul of this SDS layer is the ability to manage distributed system. However, a distributed storage introduces problems of its own, such as coordinating consistency. Different storage technologies have their own way to tackle these problems. For example, with PowerScale, OneFS has its own Group Management Protocol. Ceph uses CRUSH for data distribution. GlusterFS uses DHT(Distributed Hash Table) Translator.

Storage architects usually do not need to know these technologies in detail. It is not the intention of this post to cover the details of any distributed technology in any of the storage options above. However, storage architects needs to know supported API very well.

Access API

The supported access API of a storage system determines its compatibility with client systems. One good example is NFS for file storage, which defines the protocol for file share without defining the underlying implementation. Most GNU/Linux distributions come with nfsd (NFS server) which exports directories on XFS or ext4 FS as a file share with NFS protocol. In order to transfer data over network, NFS uses RPC, a request-response protocol. With object storage, S3 is a widespread protocol. Below is a list of storage implementations and their supported access API:

Ceph supports librados, S3, Swift and FUSE
GlusterFS supports SMB, NFS, FUSE,
PowerScale supports NFS, SMB/CIFS, HDFS, Object, POSIX

CephFS is distributed file system built on top of Ceph RADOS. It is also a client-server architecture. A Ceph Client, via librados, interacts directly with OSDs to store and retrieve data. In order to interact with OSDs, the client app must invoke librados and connect to a Ceph Monitor. For compatibility, CephFS namespaces can be export over NFS protocol using NFS-Ganesha NFS server.

Ceph Architecture

Ceph is a high-performance, distributed storage platform. It provides object storage, block storage and distributed file system, all backed by a single, reliable storage cluster running on commodity server hardware. A Ceph Storage Cluster consists of Ceph Nodes on a network. A Ceph Storage cluster requires at least one Ceph monitor (ceph-mon), Ceph Manager (ceph-mgr) and Ceph OSDs (ceph-osd). For file system clients, it also requires Ceph Metadata Server (MDS, ceph-mds) to allow user to execute basic commands on POSIX file system (e.g. ls, find)

Under the hood, Ceph stores data as objects within logical storage pools. Using the CRUSH algorithm, Ceph calculates which placement group (PG) should contain the object, and which OSD should store the placement group. The CRUSH algorithm enables the Ceph Storage Cluster to scale, rebalance, and recover dynamically.

Ceph is based on RADOS (reliable autonomic distributed object store), a self-healing system that distributes and replicates data across nodes. It then layers CephFS (a distributed file system), block storage service (RADOS Block Device or RBD), and s3-compatible object storage (RADOS Gateway or RGW) on top of RADOS. For a better description, refer to this page. The chart above shows how Ceph interacts with different kinds of client. For CephFS, the client can interact with the file system via metadata daemon, as illustrated below. This diagram looks similar to the diagram for NFS.

In a RADOS cluster, each server runs some daemons (i.e. OSD, MON or MDS).

When an I/O request occurs, it needs to be mapped to the specific OSD that keeps the storage units. Here is an illustration of the mapping:

As typically observed in distributed system, there is quite some communication overhead to serve a file.

Ceph Cluster Installation

Installing a VM-based Ceph cluster is no trivial effort and there are several methods. The recommended method is Cephadm. Here is a good instruction, where you will notice a lot of steps on each nodes, such as configuring NTP, installing docker, configuring hostname, Linux user and SSH, etc. You may also check this video for how involving it is. Red Hat adopts Ceph project as a product and has an installation guide on its documentation.

Previously, there was a legacy tool ceph-ansible to help administrators with server configuration. It is similar to the way kubespray helps administrators configure Kubernetes cluster. However, the document suggests that ceph-ansible is not integrated with new orchestrator APIs and therefore is not a viable option anymore. Also I did not find a way to install a single-node ceph cluster just for a quick demo. It involves tweaking the CRUSH map configuration.

If we deploy Ceph on Kubernetes for Kubernetes workload, we use Rook, an orchestrator running on Kubernetes, to integrate storage to a cluster.

Cloud Native Storage

Moving to cloud native storage, instead of presenting storage to operating system, we need to configure storage classes for Pods to use persistent volumes dynamically, using storage provisioners. Ceph also shows good presence in cloud native storage ecosystem. In a self-managed Kubernetes cluster, Ceph gives us the capability to configure storage classes to access connected storage. In public cloud, Ceph allows us to configure storage classes connecting to disks attached to the Nodes, an alternative to the cloud vendor provided native storage classes with high availability across availability zones. This layer enables the organization to normalize how their application connects to persistent volumes, a capability particularly helpful in the multi-cloud strategy of the cluster.

Rook is a CNCF project to orchestrate storage system on Kubernetes. It automates storage administrative tasks such as deployment, bootstrapping, configuring, provisioning and monitoring, using declarative templates. It supports Ceph and a number of other storage backends such as Cassandra, NFS, MinIO.

Kick the tires on ArgoCD

Sun, 10 Jul 2022 00:10:00 -0400

Background

In January, I wrote about FluxCD, and adopted it in the Korthweb project. I like the simple design of FluxCD, and I am comfortable with commands without using a web UI. Half a year later, I am re-visiting this choice, with ArgoCD in mind.

After reading numerous recent posts that compare the two (such as this one from the new stack), I started to give more thoughts on ArgoCD, for a couple reasons. First, ArgoCD has more contributing companies and public references. Red Hat adopted Argo CD as the underlying technology for OpenShift GitOps (and Tekton for pipeline). Second, ArgoCD has a mature UI. In corporate collaboration, especially with those not well-versed with command line, a UI is extremely helpful. Bundled with the UI, Argo CD also uses its own RBAC independent of Kubernetes RBAC, making it a great choice for continuous deployment for enterprise applications.

User Interface

You can choose to install just the core components, without UI, SSO and multi-cluster features. To me, it does not make sense because those features are exactly the reason to choose Argo CD. Argo CD also has an eponymous CLI tool, similar to flux for Flux CD. Since Argo CD uses its own RBAC independent of Kubernetes RBAC, Argo CD will have its own credential. The default username is admin and password needs to be retrieved from Secrets. In the tutorial we use this user’s credential to interact with the target cluster. This is different from Flux where the client simply uses kubectl configuration.

In the “Getting Started” guide, we create a namespace “argocd” and install it with manifests. Alternatively, we can use CLI tool to install it. With Argo CD, we can consider hosting it on a different port than the workload’s port, such as 8443. This port serves as “GitOps management port” separated from the application port (e.g. 443) for business workload. Just like any application on Kubernetes, we usually need to configure an Ingress on the management port.

In the argocd namespace, we can see a few services. The application set controller acts as the main controller that reconciles between actual state and desired state. There is also a redis server, for storing data in Argo CD. Another service to note is the dex server, indicating that the SSO capability is provided by the Dex project.

Features

In FluxCD, they use a CRD Kustomization with kustomize.toolkit.fluxcd.io/v1beta2 as version. This name is confusing. Luckily, in ArgoCD, the CRDs are Application (with argoproj.io/v1alpha1 version) and ApplicationSet (with argoproj.io/v1alpha1 version). The controllers monitors CRs created with these CRDs. ArgoCD defines Application as a group of Kubernetes resources as defined by a manifest. Argo CD Application resource deploys resources from a single Git repository to a single destination cluster/namespace. On the other hand, ApplicationSet uses templated automation to create, modify, and manage multiple Argo CD applications at once. ApplicationSet controllers monitors ApplicationSet resources. We can define a template within the declaration of an ApplicationSet and it allows for parameter substitution.

Another key capability for a GitOps utility is the support of multiple templating tools. FluxCD supports Helm and Kustomize. So do Argo CD. In addition, ArgoCD also supports jsonnet (ksonnet is not supported anymore). Jsonnet is a templating tool with many useful operators. Kubectl can directly consume Jsonnet’s JSON output as if it were YAML. In FluxCD, there are different CRDs for Helm and Kustomize (HelmRelease and Kustomization). In ArgoCD, the Application CRD covers Helm, Kustomize, and Jsonnet by embedding them as attributes in the declaration. This makes it even simpler than Flux CD.

Secret Management

It’s a big no-no to put secret in a code repository. Therefore any GitOps solution must solve the problem with secret logistics. What baffles me is ArgoCD remains un-opinionated on this matter. There appears to be some context for this stance. As a result, ArgoCD only points to a few third-party secret management solutions.

Secret logistics is a tricky problem with GitOps workflow. One of the techniques is sealed secret. In the repo we store the secret as encrypted by public key. The controller keeps the private key which decrypts the secret at the time of deployment. A few engineers don’t find it feasible for operation, as explained in this post and this post.

An alternative is to use external secret operator. Such operators will be able to sync a secret from external secret store such as Hashicorp Vault, AWS secret manager, Azure Key Vault, etc. This is much neater because we only store reference to secret in the repository. However, when the cluster assesses external secret store, it still requires either a secret, or a permission.

Another technique that drawing attentions is using sops for secret and kustomize-sops to integrate with ArgoCD. Mozilla’s sops (SecretOPerationS) project releases a binary utility called sops to encrypt and decrypt YAML manifest. It can encrypt only the values in YAML manifest and not the attributes (keys). We use sops to encrypt YAML files in order to store them in Git repo. We also use sops to decrypt YAML right before we deploy it with kubectl. The key pair for encryption and decryption can be stored in Azure Key Vault, AWS KMS, GCP KMS, and even age and pgp. Here is a simple tutorial.

Example

In a few command we can configure a minimal example. I first use KinD or Minikube to create a cluster following the steps in real-quicK-cluster repo. Then we configure an application using YAML manifest.

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
$ kubectl port-forward svc/argocd-server -n argocd 8443:443

From MacOS, I can now browse to https://localhost:8443/ for ArgoCD UI, and login with admin user and the password as printed above. We can then use the UI to create an Application. Alternatively, we can configure an application using CLI utility:

$ brew install argocd
$ argocd login localhost:8443
$ argocd app create guestbook --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace default
$ argocd app sync guestbook

On the GUI we should see the application is sync’ed.

On the left panel, click on the gear icon for settings. Under Clusters, we can see the cluster URL, which we use as dest-server value above. We can add more clusters here.

Pick the right open-source project

My experience with ArgoCD and FluxCD raised an issue to reflect on. For the same problem, there are multiple choices of open-source project. What would be a good methodology to choose the right open-source technology? I would first look at the following factors:

History
Community size
License model
Enterprise support
Governing model

The history and community size give a good idea of the technical maturity level. The License model has legal implications as to where you can use the technology. License model can change even after the project has been launched and even become not open-source any more. One example is Elasticsearch license change in Jan 2021, from the very open APLv2 to under dual license of both Elastic License (by Elastic) and Server Side Public License (SSPL, introduced by MongoDB). Neither license is certified by OSI as compliant with open source definition. For enterprises, apart from licensing, I’d also consider if any organization provides commercial support for the open-source product.

What is often neglected is the governing model. The governing model determines how product decisions are made that will shape the future of the project. For example, CNCF is a vendor neutral foundation for cloud computing and if an organization contributes a project to CNCF, then the member projects have to align with CNCF’s governing practices. Google also introduced an OUC (Open Usage Common) governing model.

Another good way is to just follow Red Hat’s choice. Red Hat built its business model around open-source technologies. They have to pick open-source project to provide enterprise level support so presumably they have a rigour selection process that one can piggyback off. Products supported by Red Hat are composed of open source components often vetted by multiple upstream communities, and changes made to these components are pushed to their respective upstream projects, often before they land in a supported product from Red Hat. Here are more insights in this business model.

Had I gone with the Red Hat rule, I would have picked Argo CD in the first place because that is the base of OpenShift GitOps!

Verdict

ArgoCD and FluxCD are tools for GitOps. However, just using a GitOps tool does not guarantee that all workload are deployed continuously. First, these GitOps controllers cannot replace Operators, which has domain knowledge about how to orchestrate their workloads. Second, resources not directly managed by these operators (such as those installed by Helm) are not being monitored continuously. For continuous deployment over all workloads, I recommend use operator to install third party applications, and use Argo CD or Flux CD to manage the operators.

GitOps makes use of the controller pattern to manage deployment in a continuous matter. Controller works closely with CRDs. ArgoCD uses similar set of CRDs to manage continuous deployment. What sets it apart is the user friendly web UI, the IAM integration and multi-cluster capabilities. These capabilities make ArgoCD well adapted in enterprise IT eco-system.

Chaos Mesh – Cloud Native Chaos Engineering

Thu, 30 Jun 2022 21:49:00 -0400

In this post, we discuss the resilience test problem and why chaos mesh emerged. Then we go over a lab of chaos mesh with a few experiments.

The Problem

I used to support a server application installed in customer’s data centre. The server application receives data from client application, cleanse the data and put them on customer-managed storage. Unfortunately, the customer-managed storage system hang up from time to time. When this happens, the application does not get an I/O failure to begin with so it continues to serve client traffic with many threads pending for I/O completion. All the threads will fail eventually after a long, unresponsive pause.

What is really bad, is that the server application first acknowledges the client application of the recipient of their data, and then asynchronously archive them to the customer’s storage. In this case, the client applications think they securely send out the data upon receiving acknowledgement. However, the server application lost the data afterwards because it does not interact with storage system in a fail-safe fashion. As a result, I often find myself in the business of restoring client data from the cache directory on the servers, and grumbling about how the quality process allowed this issue slipped into production.

To be fair, this is not an easy catch in the quality assurance process. It is caused by a hardware issue, which is technically the customer’s own problem. However, it is the vendor’s responsibility to design fault-tolerant software. Further, if we put on the site reliability engineering goggle, we see vendor’s software and customer’s hardware as a whole, instead of two disparate silos. The problem is however, how we emulate this kind of fault in house?

Chaos Engineering

A software engineer could have unmounted the storage while the software is active during test. This approach is faulty because it has a couple of issues. First, unmounting the storage does not produce exactly the same symptom. Unmounting the storage in most cases gives a clear failure upfront and the application would have caught it. What we need is a delay in storage long enough to cause a time out. Second, we’d have to perform this activity with an operating system command with sufficient privileges. The software being tested usually do not have such privileges. We need to systematically inject failures of particular kind.

Chaos are interruptions along with the cascading effects, that happen in production but are impossible or extremely difficult to emulate in lower environments through automated quality process. To improve robustness, we need the capability to emulate various failure scenarios in a systematic manner. Chaos Engineering aims to improve such capability.

Chaos Engineering emerged as a discipline to improve software’s ability to tolerate failures while still delivery adequate quality of service. This is known as the resiliency of the software. Suppose the storage has 99.9999% SLA, or 31 seconds of downtime every year. Chaos Engineer requires the functional parts of our software do what they can to protect the data, during the 31 seconds of turbulence each year, instead of just falling apart. In regulated industries, this is critical. Resilience testing is difficult in quality assurance process, but we can’t avoid it.

By definition, Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. It is particularly useful in distributed system, where network faults are normal. The one-page chaos engineering website has a good summary of such intent. At Netflix, they built Chaos Monkey, a tool to randomly cause failures on certain computing instances in the cloud. It is one of the first Chaos Engineering tools and it is a big step forward. Chaos Monkey can perform one type of experiment (faulty server) and requires writing custom code.

Chaos Mesh

Moving to Kubernetes platform, where everything operates on network, it is even more important to have a suite of resilience testing tools. Thankfully, we have Chaos Mesh, a cloud-native Chaos Engineering platform that helps platform engineers and software engineers simulate faults in a variety of scenarios. There are other chaos engineering tools (such as Litmus, also a CNCF project) but I will just focus on Chaos Mesh in this post because it has longer development history.

With Chaos Mesh, we can run experiments to simulate certain types of faults, including faults on Pod, Network, File I/O, DNS, Time, JVM, Linux Kernel, HTTP. We can also emulate incidents on cloud platforms (Azure, AWS, GCP) and emulate stress. We can even orchestrate multiple experiments into a workflow. Experiments can start ad hoc or on schedule. In addition to Kubernetes platform, the Chaos Mesh team also provides a tool Chaosd to emulate faults on a physical nodes. In the rest of this post, we will install Chaos Mesh and perform four simple experiments using the bookinfo sample application from Istio’s installer.

Demo Platform

In this lab, we install chaos mesh on Minikube on my Mac environment. I have install the platform in a particular way so the experiments will all be successful. I use this script in my real-quick-cluster project to create minikube cluster. This script configures a 3-node minikube cluster, with containerd as CRI, and enables plugins metallb and metrics server. Metallb provides a load balancer for the cluster. Metrics server allows scaling based on metrics, a test scenario for stressor in Chaos Mesh.

I will also install Istio in this lab, because I use it for Ingress and service routing. I also use a sample application called bookinfo which comes with Istio code repository. It is highly recommended to have an ingress as it is used in production, although you may use your own choice of ingress and service routing tool such as ingress-nginx. To install istio, we run this simple script:

#! /bin/bash
curl -L https://istio.io/downloadIstio | sh -
export PATH=$(realpath istio*/bin):$PATH
if istioctl x precheck; then
 echo ready to install istio and label namespace for istio-injection
 istioctl install -f istio-operator.yaml -y --verify
 kubectl label namespace default istio-injection=enabled
else
 echo failed precheck
 exit 1
fi

Below is the content of istio-operator.yaml

apiVersion: install.istio.io/v1alpha2
kind: IstioOperator
metadata:
 name: istio-operator
spec:
 profile: default
 hub: docker.io/istio
 tag: 1.13.2
 namespace: istio-system
 meshConfig:
 accessLogFile: /dev/stdout
 components:
 pilot:
 k8s:
 hpaSpec:
 maxReplicas: 7
 minReplicas: 1
 nodeSelector:
 beta.kubernetes.io/os: linux
 overlays:
 - kind: Deployment
 name: istiod
 patches:
 - path: spec.template.metadata.labels.version
 value: 1.13.2
 ingressGateways:
 - name: istio-ingressgateway
 enabled: true
 label:
 istio: ingressgateway
 k8s:
 hpaSpec:
 maxReplicas: 5
 minReplicas: 1
 service:
 ports:
 - name: http
 port: 80
 targetPort: 20080
 protocol: TCP
 - name: http-dashboard
 port: 8080
 targetPort: 28080
 protocol: TCP
 overlays:
 - kind: Deployment
 name: istio-ingressgateway
 patches:
 - path: spec.template.metadata.labels.version
 value: 1.13.2
 - path: spec.template.spec.containers[name:istio-proxy].lifecycle
 value:
 preStop:
 exec:
 command: ["sh", "-c", "sleep 5"]

For more about istio installation, refer to my previous post. Next, we’ll create the namespaces, install bookinfo and install chaos mesh using Helm:

kubectl create ns chaos-testing
kubectl create ns bookinfo

kubectl -n bookinfo apply -f istio/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl -n bookinfo apply -f istio/samples/bookinfo/networking/bookinfo-gateway.yaml

helm install chaos-mesh chaos-mesh/chaos-mesh -n chaos-testing --version 2.2.0 --set dashboard.service.type=ClusterIP --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock

As soon as we install bookinfo, we should be able to access it at http://192.168.64.16/productpage, suppose the IP address is 192.168.64.16. We will simply use HTTP because certificate is not the point of this demo. Note that when we install Chaos Mesh we have to specify the runtime (containerd). Otherwise, we might run into issues when running experiments (e.g. “unable to flush ip sets” for network faults). Before fault injections, we’ll start a curl command and observe the output:

while true; do
 curl -I http://192.168.64.16/productpage --connect-timeout 4 --max-time 5
 sleep 5
done

We can watch the output of the command above as we inject each of the following faults later on.

Chaos Mesh UI

Since we installed chaos mesh as cluster IP service, we need to expose it using Istio Gateway and Virtual Service with the following manifest. If you have your own choice of Ingress, use a corresponding ingress resource.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: chaos-dashboard-gateway
 namespace: chaos-testing
spec:
 selector:
 istio: ingressgateway
 servers:
 - port:
 number: 8080
 name: http
 protocol: HTTP
 hosts:
 - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: chaos-dashboard-vs
 namespace: chaos-testing
spec:
 hosts:
 - "*"
 gateways:
 - chaos-dashboard-gateway
 http:
 - route:
 - destination:
 port:
 number: 2333
 host: chaos-dashboard

Once we configured Virtual Service and Gateways, and waited all Pods in chaos-testing namespace to come up, then we should be able to access the web portal of chaos mesh at 192.168.64.16:8080. Follow the on-screen instruction to fetch the token for access at Cluster level.

Chaos Mesh Web Portal

Now, we can create experiments, either using web portal, or using their CRDs. We will use CRDs in this demo.

Network Latency Injection

In this section we artificially introduce a network latency of 3 seconds for a period of 45 seconds. It takes effect as soon as we create the NetworkChaos CRD as below:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
 name: network-delay
spec:
 action: delay # the specific chaos action to inject
 mode: one # the mode to run chaos action; supported modes are one/all/fixed/fixed-percent/random-max-percent
 selector: # pods where to inject chaos actions
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage
 delay:
 latency: '3s'
 duration: '45s'

Watch for the curl output, which normally returns 200 code every 5 seconds, now fails for about 45 second window.

After the window of latency injection, the curl test returns normal. During the experiment, we can check the status of the fault injection by describing the NetworkChaos object. Alternatively, we can look at Experiments and Events in Chaos Mesh web portal. When completed, we can delete the NetworkChaos object. Alternatively, we can archive the experiment from web portal.

HTTP Failure Injection

The NetworkChaos API injects faults at Pod’s network layer. HTTPChaos API introduces HTTP error. We can configure an HTTPChaos object following the instruction:

apiVersion: chaos-mesh.org/v1alpha1
kind: HTTPChaos
metadata:
 name: test-http-chaos
spec:
 mode: all
 selector:
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage
 target: Request
 port: 9080
 #method: GET
 path: '*'
 abort: true
 duration: 2m

This object intercepts HTTP calls to the Pods as labelled and aborts the request. If we monitor the curl output at the same time, we can see that it starts to receive 503 error as soon as we injected the fault.

Once we remove the HTTPChaos object, the return code becomes 200 again.

Pod Failure Injection

We can emulate a fault that kills a Pod with the following manifest:

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
 name: pod-failure-example
spec:
 action: pod-kill
 mode: one
 duration: '30s'
 selector:
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage

In the mean time, watch for the Pods in the bookinfo namespace, we’ll notice the old Pod being terminated.

Because the Pod is managed by a replica set, the replica set brings up a new Pod to ensure the desired number of replicas.

Stressor

In this experiment, we add an HPA (Horizontal Pod Autoscaler) to the productpage deployment, emulate a memory stress to the Pod and watch it scale up. Before the testing, we need to first configure a resource request for memory for the deployment. To configure memory request, edit the deployment productpage-v1 by adding memory request:

Then we add an HPA as below:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
 name: productpage-deploy
 namespace: bookinfo
spec:
 maxReplicas: 4
 minReplicas: 1
 scaleTargetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: productpage-v1
 metrics:
 - type: Resource
 resource:
 name: memory
 target:
 type: Utilization
 averageUtilization: 70

Now if we describe the HPA, we shall see that HPA is active:

Then we create a StressChaos object as below:

apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
 name: memory-stress-example
spec:
 mode: one
 selector:
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage
 stressors:
 memory:
 workers: 2
 size: 128MiB

It will take some time for the HPA to detect the memory stress and scale up. We can tweak this with scaling policy. After scaling activity, the HPA status will report with the new desired number of replica:

We can also confirm that by checking the deployment size. If we want the size to reduce after we remove the stressor, we need to configure HPA accordingly with scale down triggers.

Chaos Mesh uses stress-ng utility to emulate memory or cpu stress on a Pod.

Summary

Chaos mesh is a cloud native resilience testing tool. It supports many fault experiments not only on Kubernetes, but also on virtual machines. I wish I came across this tool sooner in my professional life. Had I had chaosd in the past, I could have incorporate it in our quality assurance process on Linux platform. Without Chaos Mesh, we’d have to use different utilities for different types of faults (e.g. stress or stress-ng for cpu and memory stress). In the domain of platform engineering on Kubernetes, it helps not only software developers, but also platform engineers with correct configuration for workloads on Kubernetes. In this post we explored four types of faults. For more about all fault types and their attributes, go to their documentation for more details.

Knative Eventing Introduction

Fri, 29 Apr 2022 10:16:00 -0400

In the previous post, I mentioned that Knative Serving and Knative Eventing should be seen as two different projects. The former is supposed to be widely used as a serving layer for microservices, whereas the latter has a narrower customer base. There are a dozen companies who need to build Platform as a Service, and will benefit from event-driven architecture. They are the niche customer for Knative Eventing. In this post, we go through two demos for Eventing.

Event Driven Architecture

Before jumping into Knative eventing, let’s first understand the value of event-driven architecture. As covered in Knative presentation in 2020, a growing micro-service architecture involves lots of service-to-service communications that looks like a spider web in a diagram. Two services communicating with each other are tightly coupled: changing one requires a change of the other. Event-driven architecture evolves to decouple those interdependent services. Knative eventing tries to standardize the event data with Cloud Event and also proposes a framework to drive this architecture.

The documentation uses different words to express the same concepts. So let’s take a note first: source = producer which emits events, and sink = subscriber = consumer which is the destination of events. Knative Eventing supports two models of in even-driven architecture.

Brokers and Triggers provides an “event mesh”. Event producers delivers events to a Broker. Triggers then distribute them uniformly by consumers.

Broker is a “hub” for events. It is a central location to receive and send events for delivery. We can choose from the following broker types:

MT (multi-tenant) channel-based broker (which a channel implementation, such as Kaka channel or an in-memory channel)
GCP broker
Apache Kafka broker
RabbitMQ broker

Triggers represents a desire to subscribe to events from a specific broker. It can act as a filter

channel-subscriber model

Channels and Subscriptions provide a “event pipe” model which transforms and routes events between Channels using Subscriptions. This model makes more sense when events out of one system needs to be transformed and then routed to another process.

A channel can be either a generic channel object, or a custom channel implementation (such as Kafka channel or in-memory channel)

The subscription consists of a Subscription object, which specifies the Channel and the Sink (aka the Subscriber) to deliver events to. You can also specify some Sink-specific options, such as how to handle failures.

The key difference is Broker-Trigger model allows you to filter events, where as Channel-Subscription model allows you to transform events.

Note that a Knative Service can act as both a Source and a Sink for events, and for good reason. You may want to consume events from the Broker and send modified events back to the Broker, as you would in any pipeline use-case. Between these components, Knative Eventing uses CloudEvents to send information back and forth.

Knative Eventing Lab

We can create a lab cluster using Kind as per the instruction in real-quicK-cluster. In docker-desktop settings, I configured 6 CPU and 8GB memory as the lab involves a number of nodes. The creation takes a couple minutes. This Knative eventing lab has four paths one can take, based on different model and different implementations:

Lab A demonstrates the configuration of broker-trigger model using Kafka based broker.
Lab B demonstrates the configuration of channel-subscription model, using in-memory channel.

The two labs share some common steps. We first install Knative Eventing using the provided manifests:

$ kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.3.0/eventing-crds.yaml
$ kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.3.0/eventing-core.yaml
$ kubectl get pods -n knative-eventing

The Pods should come to READY fairly quickly. In the mean time, we can configure the consumer as well using the manifests below:

apiVersion: v1
kind: Namespace
metadata:
 name: event-example
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: hello-display
 namespace: event-example
spec:
 replicas: 1
 selector:
 matchLabels: &labels
 app: hello-display
 template:
 metadata:
 labels: *labels
 spec:
 containers:
 - name: event-display
 image: gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
---
kind: Service
apiVersion: v1
metadata:
 name: hello-display
 namespace: event-example
spec:
 selector:
 app: hello-display
 ports:
 - protocol: TCP
 port: 80
 targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: goodbye-display
 namespace: event-example
spec:
 replicas: 1
 selector:
 matchLabels: &labels
 app: goodbye-display
 template:
 metadata:
 labels: *labels
 spec:
 containers:
 - name: event-display
 image: gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
---
kind: Service
apiVersion: v1
metadata:
 name: goodbye-display
 namespace: event-example
spec:
 selector:
 app: goodbye-display
 ports:
 - protocol: TCP
 port: 80
 targetPort: 8080

The consumers (aka sinks) will be able to process events at the validation step at the end.

Lab A – Install Kafka and configure Kafka-based Broker

This lab is for broker-trigger model. We first configure a Kafka Broker, which requires a Kafka cluster. The document uses Strimzi operator to quickly install Kafka cluster. Instead, here we use Helm to install the Kafka cluster:

helm install my-cluster-kafka bitnami/kafka -n kafka --create-namespace --set volumePermissions.enabled=true --set replicaCount=3

At the end of installation, the Helm notes indicate that the bootstrap server is my-cluster-kafka.kafka.svc.cluster.local:9092. All Kafka pods should come ready in about 5 minutes. After cluster install, we need to install Kafka controller and the Broker-layer data plane which allow you to map a Kafka instance to Broker CRD later:

kubectl apply -f https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.3.0/eventing-kafka-controller.yaml
kubectl apply -f https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.3.0/eventing-kafka-broker.yaml
kubectl get pods -n knative-eventing
kubectl -n knative-eventing edit cm kafka-broker-config

As the command above suggests, once data plane is installed, we edit the default broker config by updating the bootstrap.servers with the bootstrap server address noted above. We also need to make sure the replication factor is no greater than number of partitions, which should not be a problem in our lab as the replica count was set to 3.

Now we can create a Broker in the same namespace with consumer:

apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
 name: default
 namespace: event-example
 annotations:
 eventing.knative.dev/broker.class: Kafka
spec:
 config:
 apiVersion: v1
 kind: ConfigMap
 name: kafka-broker-config
 namespace: knative-eventing

Once the manifest above is applied, we should see the status of broker (named default) to be READY. We can also validate broker creation with kn CLI command:

 $ kubectl -n event-example get broker default
NAME URL AGE READY REASON
default http://kafka-broker-ingress.knative-eventing.svc.cluster.local/event-example/default 19m True
$ kn broker list -n event-example
NAME URL AGE CONDITIONS READY REASON
default http://kafka-broker-ingress.knative-eventing.svc.cluster.local/event-example/default 19m 7 OK / 7 True

Then we create Triggers that connects both to the broker with filters, and to the consumers by referencing service name. The manifest to create triggers is provided below:

apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
 name: hello-display
 namespace: event-example
spec:
 broker: default
 filter:
 attributes:
 type: greeting
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: hello-display
---
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
 name: goodbye-display
 namespace: event-example
spec:
 broker: default
 filter:
 attributes:
 source: sendoff
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: goodbye-display

We can confirm the setups above by monitoring the resources (like we did for broker), or use knative CLI:

$ kn trigger list -n event-example
NAME BROKER SINK AGE CONDITIONS READY REASON
goodbye-display default service:goodbye-display 3m24s 6 OK / 6 True
hello-display default service:hello-display 3m24s 6 OK / 6 True

Now we can go to the validation step to send event and validate result.

Lab B – Install in-memory channel and configure Channel

This lab is for channel-subscription model. we first install an in-memory channel:

kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.3.0/in-memory-channel.yaml

Then we can map this channel to a Channel resource in Eventing. We can do so by applying the manifest below:

apiVersion: messaging.knative.dev/v1
kind: Channel
metadata:
 name: demo-channel-1
 namespace: event-example
---
apiVersion: messaging.knative.dev/v1
kind: Channel
metadata:
 name: demo-channel-2
 namespace: event-example

The creation of Channel resources can be verified as below

$ kn -n event-example channel list
NAME TYPE URL AGE READY REASON
demo-channel-1 InMemoryChannel http://demo-channel-1-kn-channel.event-example.svc.cluster.local 6m34s True
demo-channel-2 InMemoryChannel http://demo-channel-2-kn-channel.event-example.svc.cluster.local 6m34s True

Now we need to create Subscription which connects Channels to Consumers. Subscriptions can be created with the manifest below:

apiVersion: messaging.knative.dev/v1
kind: Subscription
metadata:
 name: demo-subscription-1
 namespace: event-example
spec:
 channel:
 apiVersion: messaging.knative.dev/v1
 kind: Channel
 name: demo-channel-1
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: hello-display
---
apiVersion: messaging.knative.dev/v1
kind: Subscription
metadata:
 name: demo-subscription-2
 namespace: event-example
spec:
 channel:
 apiVersion: messaging.knative.dev/v1
 kind: Channel
 name: demo-channel-2
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: goodbye-display

We can confirm subscription:

$ kn -n event-example subscription list
NAME CHANNEL SUBSCRIBER REPLY DEAD LETTER SINK READY REASON
demo-subscription-1 Channel:demo-channel-1 service:hello-display True
demo-subscription-2 Channel:demo-channel-2 service:goodbye-display True

Now we can go to validation step.

Validation

For validation, we need to first configure an event source. The source in our lab is a throw-away Pod with curl utility. I use my favourite nicolaka netshoot to fire off the events using. To launch the Pod and get to command shell:

$ kubectl run tmp-shell -n event-example --rm -i --tty --image nicolaka/netshoot -- /bin/bash
If you don't see a command prompt, try pressing enter.
bash-5.1#

We can POST message to the corresponding endpoint URL to fire events. The endpoint is different for each lab:

Lab	URL
A (Kafka-based broker)	http://kafka-broker-ingress.knative-eventing.svc.cluster.local/event-example/default
B (channel)	http://demo-channel-1-kn-channel.event-example.svc.cluster.local http://demo-channel-2-kn-channel.event-example.svc.cluster.local

With the correct URL, we can build a payload for POST method. Below is three curl commands with different payload POST to the same endpoint (taking lab A as an example):

# curl -v "http://broker-ingress.knative-eventing.svc.cluster.local/event-example/default" \
-X POST \
-H "Ce-Id: say-hello" \
-H "Ce-Specversion: 1.0" \
-H "Ce-Type: greeting" \
-H "Ce-Source: not-sendoff" \
-H "Content-Type: application/json" \
-d '{"msg":"Hello Knative!"}'

# curl -v "http://broker-ingress.knative-eventing.svc.cluster.local/event-example/default" \
-X POST \
-H "Ce-Id: say-goodbye" \
-H "Ce-Specversion: 1.0" \
-H "Ce-Type: not-greeting" \
-H "Ce-Source: sendoff" \
-H "Content-Type: application/json" \
-d '{"msg":"Goodbye Knative!"}'

# curl -v "http://broker-ingress.knative-eventing.svc.cluster.local/event-example/default" \
-X POST \
-H "Ce-Id: say-hello-goodbye" \
-H "Ce-Specversion: 1.0" \
-H "Ce-Type: greeting" \
-H "Ce-Source: sendoff" \
-H "Content-Type: application/json" \
-d '{"msg":"Hello Knative! Goodbye Knative!"}'

All the requests should receive 202 return code. As highlighted above, we specified type and source for each event fired to the broker. On the consumer side, we can simply check the pod log to verify the events received. Below is an example of events received as a result of the events sent above (take broker-trigger model as example):

$ kubectl -n event-example logs -l app=hello-display --tail=100
2022/03/09 17:05:59 Failed to read tracing config, using the no-op default: empty json tracing config
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: greeting
 source: not-sendoff
 id: say-hello
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:19.196589801Z
Data,
 {
 "msg": "Hello Knative!"
 }
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: greeting
 source: sendoff
 id: say-hello-goodbye
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:42.977803608Z
Data,
 {
 "msg": "Hello Knative! Goodbye Knative!"
 }
$ kubectl -n event-example logs -l app=goodbye-display --tail=100
2022/03/09 17:05:58 Failed to read tracing config, using the no-op default: empty json tracing config
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: not-greeting
 source: sendoff
 id: say-goodbye
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:27.430702588Z
Data,
 {
 "msg": "Goodbye Knative!"
 }
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: greeting
 source: sendoff
 id: say-hello-goodbye
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:42.977803608Z
Data,
 {
 "msg": "Hello Knative! Goodbye Knative!"
 }

This is the gist of Knative evening. Note that the format of HTTP payload when we fire off events needs to conform to CloudEvents format. The container image (event_display) used in the container is also created for the purpose of processing events received and display them in stdout.

Conclusion

Persistent storage allows for stateful applications on Kubernetes. It also enables architectural patterns that requires persistent storage. Knative eventing is a framework that enables event driven architecture. It supports many messaging channels (including NATs, a cloud-native messaging system) and broker types (MT channel based, GCP, RabbitMQ).

Knative Serving Introduction

Sun, 17 Apr 2022 11:33:00 -0400

Background

As per IBM‘s definition, Knative enables serverless workloads to run on Kubernetes clusters, and makes building and orchestrating containers with Kubernetes faster and easier. It has drawn a lot of attention recently. It released version 1.0 in November 2021, and was accepted as a CNCF incubating project in March 2022.

Glories aside, the value it delivers is the ability to go serverless on a Kubernetes platform. Originally, Knative was built with three components: build, serving and eventing. The build component was deprecated in favour of the Tekton project. Tekton is cloud native CI/CD pipeline. It connects to source code repo and build artifacts. It can also deploy applications with pipeline as code. Tekton’s documentation includes a quality tutorial here linked to interactive terminal. On the other hand, we’ll focus on Knative Serving in this post, and Knative Eventing in the next.

Knative can be seen as a serverless framework with a number of open-source technologies as building blocks, such as Istio for ingress gateway, Kafka or Google Pub/Sub as event-streaming engine, and Prometheus for observability to name a few. Despite of using the same CLI tool (named kn), Knative Serving and Eventing are considered separate capabilities using different groups of CRDs. They are installed separately, using their respective YAML files, or operator.

Serving vs Eventing ?

After reading the article “Did we market Knative wrong” from Ahmet Balkan, my impression is the two are not related. They could have been two separate projects without sharing a common name. As the author puts, these two shared some core logics. But beyond that, they don’t have anything in common.

Serving is lightweight and requires Istio. It provides the capability to scale from N to 0 when the system is idle, and from 0 to N when requests come in. It is the missing serving layer for running microservices on Kubernetes and Ahmet position it as the first thing that people installed after creating a cluster.

The target user of Eventing is much smaller. Eventing is for event-driven architecture and is more complex than Serving. Ahmet admits that they over-estimated how many people on the planet want to build a Heroku-like PaaS layer on top of Knative. There are a couple of dozen companies who would work through the complexity and build their own Kubernetes-based internal PaaS or even public-facing FaaS using Knative. They are the niche audience of Knative Eventing.

For most platform builders who just need to run micro-service, Knative Serving is all they need.

A Demo on Serving

The definition of term “serverless” is very loose. Instead, Knative documentation promotes the ability to scale to zero and refer to it as “some people call this Serverless”. This suggests that “the ability to scale to zero” should be the better term to describe this capability. To deploy it, we need the Service object with apiVersion serving.knative.dev/v1. Below is a guide for a quick hands-on, with some steps modified from the Knative Serving tutorial and installation guide, with optional steps (extensions, DNS) skipped.

Follow the first part of this post or this guide in real-quicK-cluster project, to install minikube with metal LB and configure istio (using istioctl as per instruction). Then Knative serving can be installed with two YAML manifests:

$ kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.2.0/serving-crds.yaml
$ kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.2.0/serving-core.yaml

Then we can install Knative istio controller to integrate them so we can see the knative gateways:

$ kubectl apply -f https://github.com/knative/net-istio/releases/download/knative-v1.2.0/net-istio.yaml
$ kubectl -n knative-serving get gateway

Now we can install the dummy service, either using kn CLI tool or using YAML manifest. The manifest is given below:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
 name: hello
 namespace: default
spec:
 template:
 metadata:
 name: hello-world
 annotations:
 autoscaling.knative.dev/class: "kpa.autoscaling.knative.dev"
 autoscaling.knative.dev/metric: "rps"
 autoscaling.knative.dev/target: "50"
 autoscaling.knative.dev/min-scale: "0"
 autoscaling.knative.dev/max-scale: "10"
 spec:
 containerConcurrency: 0
 containers:
 - env:
 - name: TARGET
 value: World
 image: gcr.io/knative-samples/helloworld-go
 name: user-container
 ports:
 - containerPort: 8080
 protocol: TCP

We can notice that in the annotations there are some settings to overwrite the default configuration. They are explained in the autoscaling section of the Knative serving documentation. Here we use request per second (rps) as metric, and have 50 as target. The scale range is between 3 and 10 replicas. Once the resource is created, we can validate it with one of the two commands:

$ kubectl get services.serving.knative.dev
$ kn service describe hello

The creation of knative service also created a number of Kubernetes resources, such as deployment and services:

$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello ExternalName <none> knative-local-gateway.istio-system.svc.cluster.local 80/TCP 52s
hello-world ClusterIP 10.110.127.231 <none> 80/TCP 108s
hello-world-private ClusterIP 10.111.27.73 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP 108s
kubernetes ClusterIP 10.96.0.1 <none>
$ kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
hello-world-deployment 1/1 1 1 71s

Before testing, let’s mock the DNS entry to resolve to the Istio ingress service IP like below in /etc/hosts:

192.168.64.16 hello.default.example.com

Now we can confirm that the service returns the expected result (after a brief pause):

$ curl hello.default.example.com
Hello World!

Wait for a short period without curl or connection from browser, the service should scale down to zero:

$ kubectl get deploy -l serving.knative.dev/service=hello -w
NAME READY UP-TO-DATE AVAILABLE AGE
hello-world-deployment 0/0 0 0 78s

From a separate terminal, emulate HTTP request with hey command, so the service receives more than 50 rps per pod:

$ hey -n 200 -c 20 -z 20s -m GET http://hello.default.example.com

We can watch the deployment get scaled up to 10. Once the hey command completes its task, the deployment will scale back down:

$ kubectl get deploy -l serving.knative.dev/service=hello -w
NAME READY UP-TO-DATE AVAILABLE AGE
hello-world-deployment 0/0 0 0 78s
hello-world-deployment 0/1 0 0 3m54s
hello-world-deployment 0/1 0 0 3m54s
hello-world-deployment 0/1 0 0 3m54s
hello-world-deployment 0/1 1 0 3m54s
hello-world-deployment 1/1 1 1 4m2s
hello-world-deployment 1/10 1 1 4m4s
hello-world-deployment 1/10 1 1 4m4s
hello-world-deployment 1/10 1 1 4m4s
hello-world-deployment 1/10 10 1 4m5s
hello-world-deployment 2/10 10 2 4m9s
hello-world-deployment 3/10 10 3 4m10s
hello-world-deployment 4/10 10 4 4m10s
hello-world-deployment 7/10 10 7 4m10s
hello-world-deployment 9/10 10 9 4m10s
hello-world-deployment 10/10 10 10 4m11s
hello-world-deployment 10/5 10 10 5m18s
hello-world-deployment 10/5 10 10 5m18s
hello-world-deployment 5/5 5 5 5m18s
hello-world-deployment 5/2 5 5 5m20s
hello-world-deployment 5/2 5 5 5m20s
hello-world-deployment 2/2 2 2 5m20s
hello-world-deployment 2/1 2 2 5m22s
hello-world-deployment 2/1 2 2 5m22s
hello-world-deployment 1/1 1 1 5m22s
hello-world-deployment 1/0 1 1 5m52s
hello-world-deployment 1/0 1 1 5m52s
hello-world-deployment 0/0 0 0 5m52s

Autoscaling in Knative serving is backed by KPA (Knative Pod Autoscaler, used in the example above) or HPA. Also note that the trigger to scale up and down is client connection (RPS). This is different than KEDA. KEDA also supports the capability to scale to zero. However, with KEDA, it is an event that scales a deployment up from zero. In Knative serving, it is the connection to the Service itself that wakes up the service.

Cost of scale-to-zero

As discussed, both KEDA and Knative serving supports the ability to scale to zero and wake up from zero. The trigger to wake up from zero is different. With KEDA, workload wakes up by an event. With Knative Serving, workload has to wake up upon receiving a connection to the service. In the design to solve this problem, Knative has four sub-components :

Activator – When a service is scaled to zero, its request are routed to the activator which waits for a Pod to become ready (wake up) and proxies the traffic while editing the underlying Endpoints to route the traffic directly to the Pod(s)
Autoscaler – KPA
Controller – The main component responsible for watching API objects for Knative CRDs (KService, Configuration, Route, Revision) and manage their lifecycle, create the underlying Kubernetes resources and garbage-collect old objects.
Webhook – a Kubernetes Admission Webhook acting both as validating admission controller and mutating admission controller.

The key player is activator, which receives requests when a service is scaled to zero. The wake-up process is summarized as follows:

After receiving request, activator will buffer (hold onto) the request
Then, it will look at request’s hostname to find which KService it is for.
Then, it will scale up the Kubernetes Deployment and wait for a Pod to become ready.
In the meanwhile, it updates Kubernetes Service to point to Pod IP addresses (so that activator gets out of the network path if the KService is awake).
Finally, activator proxies the request to the started pod

So the “plumbing” of the service connection managed by the Activator pod is the cost of scale to zero. This is also called Load Balancing, and the behaviour can be tweaked using two parameters: activator capacity and target burst capacity:

Target burst capacity: once the target deploy has one more more Pod, service requests may still route via activator or bypassing the activator. Target burst capacity determines at what point service request should start to bypass activator.
Activator capacity: determines how many requests can activator hold on to. Considering the service-to-service connection also routes via Service CRD, there can be a lot of connections via Activator so its capacity needs to be adjusted.

The additional overhead of managing and optimizing the “Load Balancer” is also part of operational cost for the ability to scale to zero.

In comparison, KEDA now has an HTTP add-on still at beta but allows connection-based wake-up. The design is a little different. As discussed in this previous post.

Conclusion

Knative consists of two disparate components: Serving and Eventing. Serving uses KPA to provide scaling based on service request, and the ability to scale to zero. It is often compared with KEDA, a single-purpose lightweight tool for autoscaling. Here is a blurb on their difference by KEDA. The takeaways is that KEDA is more focused on scalability, whereas Knative serving covers more aspects. For example, the Service object of Knative also supports Traffic Splitting. Knative serving can integrate with istio.

Kubernetes Operator

Thu, 07 Apr 2022 09:39:00 -0400

Kubernetes has a number of tools to automate the deployment of a single workload. In previous posts, we had covered Helm and Kustomize. What are left unresolved is how to maintain the status of workload after deployment is completed. In this post, I will give an introduction to Kubernetes Operator. Compared with Helm (templating approach) and Kustomize (patching approach), Kubernetes Operator follows the operator pattern. Operators are usually provided by the developer of the application.

Operator Pattern

In Kubernetes, we know that a controller takes care of routine tasks to ensure that desired state expressed by Kubernetes resource types matches the current state. One example is that the Deployment controller ensures the number of pods running matches the amount specified in the replica field. Controller is the key to ensure that resources can be managed by declarative manifests for Kubernetes resources.

Kubernetes makes use of controller pattern throughout its own design. One of its key component, Controller Manager, is a collection of many controllers. Each controller is in charge of a control loop, responsible for listening the object it manages. Another component, Kube-scheduler, is also a special type of Controller. The kube-scheduler monitors unscheduled Pod and health of nodes and determines the best Node to schedule the new Pod to. Then it writes the decision to etcd store for kubelet to execute.

This controller pattern is fairly successful in what it does and we can extend the use of it. Beyond the built-in resource types, we can create our own custom resource definitions (CRDs), and create controllers that watches for the manifest that declares custom resources (CRs). The controller ensures that the resource status matches their specifications. This is also known as reconciliation, which is implemented as a control loop. Operator pattern can be illustrated in the diagram below:

Operator Pattern

Technically, there is no difference between a controller and an operator. What makes an Operator (used to install workload) different than a native Kubernetes controller, are two things. First, an Operator usually needs CRDs because the built-in resource types are insufficient. Second, the operator reflects the domain knowledge to keep the target workload running. For example, stateful workloads such as database needs their operational steps executed in certain orders.

On Operator Pattern, CNCF published a whitepaper with a deeper review. This white paper is the best reference for a good understanding of the Operator Pattern.

Custom Resource Definition

The built-in controllers work with built-in objects (pre-defined APIs). Custom operators usually need their own APIs to function. To extend Kubernetes API, we define the schema of these APIs in the form of CRDs (custom resource definitions) using OpenAPIv3 standard. Then, we can declare Custom Resources (CRs) in compliance with the schema. The OpenAPIv3 schema in the CRD resource tells validating web hook (admission control) how to validate the schema when we send an CR in to API server.

When we work with third-party operators, they usually provide CRDs along with the operator implementation. For example, in my operator example project, we have a minimalist CRD WordPress with one property: sqlRootPassword and we can declare a CR as in this example. For a more realistic use case, we can take a look at Kiali CRD. In the next section, we’ll use it along with Kiali operator to install Kiali.

Operator Usage

Like Artifact Hub to Helm, OperatorHub is a public registry of most used Kubernetes Operators. In this section, we will take an example of using Operators. We will install Kiali as an add-on to Istio using Kiali CR and operator, which also depends on Prometheus to be installed using Prometheus Operator first. Note that the Kiali installation outlined in this section is not the the quick-start install manifests from Istio’s sample directory. For Kiali on production system we have to customize the installation.

Suppose we have installed Istio, we can then install Prometheus operator using Helm. The Prometheus operator will install Prometheus. Then we use Helm again to install Kiali operator. The Kiali operator will watch for creation of Kiali CRD, to deploy services:

$ helm install -f prometheus-values.yaml --namespace istio-system --repo https://prometheus-community.github.io/helm-charts --version 13.6.0 istio-prometheus prometheus --insecure-skip-tls-verify
$ helm install -f kiali-operator-values.yaml --namespace kiali-operator --repo https://kiali.org/helm-charts --version 1.45.0 kiali-op kiali-operator --create-namespace
$ kubectl apply -f kiali-cr.yaml

I include example content for each file in the commands above on Github gist (prometheus-values.yalm, kiali-operator-values.yaml and kiali-cr.yaml). For more options for installing Kiali, refer to their documentation.

I use this example to install Kiali and it includes two Operators, the Prometheus Operator and the Kiali Operator. The Prometheus Operator is one of the first ever written Kubernetes Operator. As soon as the operator is deployed, it starts to deploy the operator service. For the Kiali operator, we need to deploy Kiali CR after the Kiali Operator has been deployed. Both are valid patterns.

Operator Development

Operator is powerful. However, authoring an Operator is not a trivial effort. One usually start with a framework. A framework creates a body of boiler plate code that has the pattern implemented and allows developers to enrich the functions following the pattern. The white paper introduced three frameworks:

CNCF Operator Framework – aims at Operator Developers with an SDK, a scaffolding tool and a test harness. It currently supports three project types: Golang, Helm and Ansible. CNCF Operator framework consists of SDK and OLM.
Kopf (Kubernetes Operator Pythonic Framework) – an easy-to-use framework in Python that abstracts away most of the low-level Kubernetes API communications hassle.
kubebuilder – helps build a Manager similar to the native kube-controller-manager. For difference with OperatorSDK, read here.
Metacontroller: lightweight Kubernetes Controller as a Service

In CNCF Operator Framework, the Operator SDK supports development using Ansible, Helm and Golang. The author of this post makes a general comparison as follows:

Type	Best use case	Underlying technology	Amt of Effort
Helm	Stateless workload	Helm Charts	Med
Ansible	Stateless workload	Ansible Roles and Playbooks	Med
Golang	Stateful workload	Code developed in Golang	High

The aforementioned Kiali operator is an example of Operator developed in Ansible. The prometheus operator, is developed in Golang as the workload can be stateful depending on configuration. One needs to know how to develop operator in Golang in order to tackle the most complicated situations. This is requires some serious development effort. The documentation with a quick start section is available here. Even that is not very straightforward. RedHat, the maintainer of the CNCF Operator framework has a good blog post on how to develop an Operator in Golang.

The example requires some development knowledge to go through. On my MacOS (Intel) I have to configure the following prerequisites:

Install gcc, using command: xcode-select –install
Install the right version of golang. You can find the version here. The MacOS has a version of golang installed already so I had to install version 1.17 and link to it: brew install go@1.17 && brew link –force go@1.17
Install operator-sdk with home brew: brew install operator-sdk
When you run “operator-sdk version”, ensure the result shows a golang version that matches your installation.
If you need to push docker image, also connect to docker registry by running: docker login

Then we can create our working directory, initialize the repository and create boilerplate code (scaffolding) with these commands:

$ mkdir wordpress-operator && cd wordpress-operator
$ operator-sdk init --domain digihunch.com --repo github.com/digihunch/wordpress-operator
$ operator-sdk create api --group wordpress --version v1 --kind WordPress --resource --controller

With the repo initialized, we can go to the section “Defining the API” and “Implementing the Controller”. The blog post does not cover every code editing needed to bring up wordpress. You are supposed to go to the author’s repository to fit the changes into your own repo. The author’s repo has a few more controllers such as common.go and mysql.go.

At the end of the lab, you should be able to run the controller and bring up wordpress. I used my own repository for this lab and have made the code changes for this lap in a couple commits. To test locally with the code:

$ git clone git@github.com:digihunch/wordpress-operator.git
$ cd wordpress-operator
$ make install run

Then we can validate wordpress install from a new terminal as the instruction shows:

$ kubectl create -f config/samples/wordpress_v1_wordpress.yaml
$ minikube service wordpress --url

For Developers that requires more details, RedHat has an eBook for Kubernetes Operators, in supplement to the documentation. As DevOps professional, I’m mainly concerned with understanding how Operator works and using Operators correctly.

Too many Tools?

Now we seem to have too many choice of tools when it comes to deploying workload on Kubernetes. Kustomize and Helm can deploy simple workloads. Operator can deploy stateful workloads, as well as keep the workload status in check. Further, we have FluxCD and ArgoCD based on GitOps workflow.

When assessing a tool, we should think about the complexity of the workload deployed. If it is a single stateless workload, Kustomize or Helm should be sufficient. If it is not very simple but still stateless, we can consider using Helm charts developed by the community. For multiple workloads, we can build our own top-level chart to combine existing sub-charts created by the community.

Helm is essentially a package manager. It does not follow controller pattern and therefore will not monitor the current status of deployment. Helm has other limitations compared to Operator. For example, as a templating scheme, it reaches limitation when dealing with complex logic, even with the help of its helper functions. It is also hard to reason through the template code when we have to troubleshoot a deployment. Refer to this blog post for the author’s experience with Helm.

If we want our deployment to be fully declarative and continuous, then we will follow the Operator pattern by using a Kubernetes Operator. When we have many workloads of different levels of complexity, we can combine them with GitOps tool. Operator is one of the underlying technologies behind GitOps.

Workload profile	Just Installation	Installation and Maintain Status
Single stateless workload	Helm or Kustomize	Operator (using Ansible or Helm)
Single stateful workload	Helm or Kustomize	Operator (using Golang)
Multiple workloads	Helm (e.g. build parent chart)	GitOps in combination with Operator, Helm and Kustomize

The table above helps refine deployment requirement. It’s not a recommendation, but rather a model of analyzing deployment requirement.

Autoscaling on Kubernetes Platform

Mon, 28 Mar 2022 13:14:00 -0400

Introduction

The concept of autoscaling on Kubernetes platform dates from the era where virtualization first became widespread and the overhead of provisioning a new server became lightweight through the use of cloud-init. With public cloud, customers operate on usage-based billing. Autoscaling allows workload to scale down during idle times to reduce cost, and scale up during peak time to meet the demand of business traffic. Vertical autoscaling replaces a VM with one of higher capacity, which is usually interruptive. Horizontal autoscaling adds or removes VMs to adjust capacity, and works in conjunction with load balancing mechanism to assign load to a specific target in the group. Unless otherwise specified, we simply refer to horizontal autoscaling as autoscaling. Depending on what triggers autoscaling, it can be metrics based or event driven.

Metric-based autoscaling is based on VM (or Pod) metrics in the autoscaling group. The metrics are mostly about CPU usage, memory, IOPS, number of connections, etc. For example, when the average CPU utilization across all VMs in the last five minutes hits 70% threshold, then the scaler introduces a new VM into the autoscaling group. The trigger can factor in a variety of metrics. Advanced autoscaling APIs can also support lifecycle hooks, i.e. custom activities upon creation of new VMs during scale-up, or upon deletion of existing VMs during scale-down. Other aspects of custom behaviours include a cool-off period, i.e a no-activity window after the previous scaling activity. Since scaling activities are re-active.

Using real-time metrics as a trigger of scaling is not always a good idea. For example, a buggy order processing program may consume 100% of CPU due to an infinite loop, or 99% of memory due to memory leak. Metric-based scaling may fire off even though there is currently no order pending in the queue. Event-driven approach is more flexible. Event can fire from any type of source. For example, in Kubernetes, when scheduler fails to schedule a Pod due to constraints, it is an event This event can trigger scaling. In some case, a metric hitting a threshold fires an event. For example, scale up when size of order queue reaches 20. In this sense, metric-based autoscaling is a special case of event-driven autoscaling.

With Kubernetes, let’s examine node group autoscaling (aka cluster autoscaling) and workload autoscaling (Pod autoscaling).

Node Group Autoscaling

Cluster autoscaler is the mechanism to auto-scale node groups for Kubernetes. As per its documentation, any metric-based cluster/node group autoscalers are NOT compatible with CA. They are also not particularly suitable for Kubernetes in general. Take AKS for example, the events to trigger scale-up and scale-down are as below:

The cluster autoscaler component can watch for pods in your cluster that can’t be scheduled because of resource constraints. The cluster then automatically increases the number of nodes.
The cluster autoscaler decreases the number of nodes when there has been unused capacity for a period of time. Pods on a node to be removed by the cluster autoscaler are safely scheduled elsewhere in the cluster.

Both are in essence event driven. The behaviours can be fine-tuned with a number of parameters as below:

scan-interval
scale-down-delay-after-add
scale-down-delay-after-delete
scale-down-delay-after-failure
scale-down-unneeded-time
scale-down-unready-time
scale-down-utilization-threshold
max-graceful-termination-sec
balance-similar-node-groups
expander: random, most-pods, least-waste, priority
skip-nodes-with-local-storage
skip-nodes-with-system-pods
max-empty-bulk-delete
new-pod-scale-up-delay
max-total-unready-percentage
max-node-provision-time
ok-total-unready-count

The parameters above constitute the autoscaler profiler, and are effective if cluster autoscaler is enabled. For many implementations, cluster autoscaler can be enabled and disabled even after the cluster has been created, and the parameters can be changed. The overhead of provisioning a new node should not be overlooked, because that is usually the window that a Pod needs to wait to get scheduled. As stated in CA’s FAQ, the main purpose of CA is to get pending pods a place to run, instead of pre-emptively accommodating to increasing workload.

The delay in pod scheduling while adding a new node can be controlled to a certain degree with one of the two workarounds below:

With HPA or KEDA, set lower threshold so the workload level scaling acts more aggressive than the increase of demand. This buys some buffer time
Use a tool to puff up utilization, such as cluster overprovisioner, which deploys pods that
- request enough resources to reserve virtually all resources for a node
- consume no actual resources
- use a priority class that causes them to be evicted as soon as any other Pod needs it.

In practice the cluster autoscaler setup should be conservative and keep node size as stable as it can. For example, a 20 minutes idle-window (low utilization) on a node is not worth the overhead to remove a node and add it back in 20 minute later.

When the cluster do need to scale down by removing a node, one common symptom is failing to scale down because some Pods have nowhere else to schedule to. Here is a list of possible causes as troubleshooting tips.

If node scaling should be triggered sparsely, then pod scaling is by design very dynamic. Cloud native applications should assume that pod scaling occurs very frequently.

In late 2021, AWS released the open-source project Karpenter for cluster autoscaler. Karpenter addresses some challenges with native Cluster Autoscaler on EKS. Karpenter is gaining momentum and now adding support for other cloud service providers including Azure.

Workload Autoscaling

Stateless workload are controlled by a Deployment object, which is associated with a replicaSet object. For stateless workload we can use HorizontalPodAutoscaler, or HPA. There is a VerticalPodAutoscaler (VPA) which is much less common. HPA is metrics based with flexible options such as specifying an object, depending on what metrics are available via metrics API. There are two versions of HorizontalPodAutoscaler: autoscaling/v1 and autoscaling/v2. The latter supports scaling policies, such as adjusting downscale stabilization window, and limiting scale down rate. No matter which API version, the metric-based triggers in HPA are fairly limited.

We already know that metrics are not always the best indicator to trigger scaling. We need an option to trigger scaling based on the status of other components such as queue size. KEDA (Kubernetes Event Driven Autoscaling) is a great option to consider for horizontal workload scaling. KEDA works with HPA, and significantly enriches trigger options. Apart from metrics, KEDA can use a number of external mechanisms as triggers, for example:

RabbitMQ/Kafka/SQS: scale based on queue size
Azure Log Analytics: scale based on a kusto query result against Azure Log Analytics
AWS CloudWatch, Azure Monitor: scale based on metrics from Azure Monitor/AWS CloudWatch
Azure Pipelines: scale based on agent pool queues of Azure Pipeline
Elasticsearch: scale based on elasticsearch query result
Kubernetes Workload: scale based on the count of running pods of a specified workload
MSSQL, MySQL, Postgres, Cassandra: scale based on a query result
Prometheus: scale based on prometheus query result

KEDA is a single-purpose and lightweight component. With KEDA, we don’t need to explicitly define HPA. It allows us to select from a longer list of triggering mechanisms for our auto scaler. We shall not underestimate the work needed to select the most suitable trigger because having an incorrect trigger (e.g. bad metrics) is costly. Let’s take Java applications as an example. Java workload operates in a JVM inside of the container. JVM request the entire heap size from operating system. The garbage collection activities also consumes a good portion of CPU cycles. This pattern makes CPU and memory metrics inaccurate as an indicator for scaling activity. Because of this we need to find out what is the best scaler for Java application, based on understanding of how the entire solution stack works as a whole.

The other aspect that KEDA beats HPA is its ability to scale to 0. This can be helpful when a service is idle most of the time but cannot shut down.

KEDA lab

let’s use Kafka as an example to configure KEDA for a dummy workload. We create a mock cluster using Kind with a simple configuration file. Then, let’s start with the following dummy workload with replica count set to 1:

apiVersion: v1
kind: Namespace
metadata:
 name: workload
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: aks-helloworld-one
 namespace: workload
spec:
 replicas: 1
 selector:
 matchLabels:
 app: aks-helloworld-one
 template:
 metadata:
 labels:
 app: aks-helloworld-one
 spec:
 containers:
 - name: aks-helloworld-one
 image: neilpeterson/aks-helloworld:v1
 ports:
 - containerPort: 80
 env:
 - name: TITLE
 value: "Welcome to Azure Kubernetes Service (AKS)"
---
apiVersion: v1
kind: Service
metadata:
 name: aks-helloworld-one
 namespace: workload
spec:
 type: LoadBalancer
 ports:
 - port: 80
 selector:
 app: aks-helloworld-one

We need to install KEDA and Kafka using Helm:

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda -n keda --create-namespace
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install kafka bitnami/kafka -n kafka --create-namespace --set volumePermissions.enabled=true --set replicaCount=3

Watch for all Pods to come up. Also read the notes from Kafa installation and confirm the Kafka service address. Now, we will apply KEDA scaled object, defined as below:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: kafka-scaledobject
 namespace: workload
spec:
 scaleTargetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: aks-helloworld-one
 pollingInterval: 10
 cooldownPeriod: 30
 idleReplicaCount: 0
 minReplicaCount: 2
 maxReplicaCount: 5
 fallback:
 failureThreshold: 3
 replicas: 1
 advanced:
 restoreToOriginalReplicaCount: true
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 300
 policies:
 - type: Percent
 value: 100
 periodSeconds: 15
 triggers:
 - type: kafka
 metadata:
 bootstrapServers: kafka-0.kafka-headless.kafka.svc.cluster.local:9092
 consumerGroup: my-group # Make sure that this consumer group name is the same one as the one that is consuming topics
 topic: test
 # Optional
 lagThreshold: "5"
 offsetResetPolicy: latest

The field definitions are on KEDA deploy documentation. Items under Kafka trigger are on the trigger documentation. In this lab we set the idelReplicaCount to 0. It will scale up with average lag of all partitions reaching 5. In the next few steps, we’ll mock up some messages posted to the Kafka topic named “test” for consumer group my-group. We can watch deployment size grow along with the growth of lags.

To emulate Kafka client activity, we can spin up a Kafka test Pod:

kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:2.8.1-debian-10-r73 --namespace kafka --command -- sleep infinity
kubectl exec --tty -i kafka-client --namespace kafka -- bash

From within the Pod, we can leverage the client-side scripts located in /opt/bitnami/kafka/bin/. For example, to post message to a topic (e.g. named test):

kafka-console-producer.sh --topic test --broker-list kafka-0.kafka-headless.kafka.svc.cluster.local:9092,kafka-1.kafka-headless.kafka.svc.cluster.local:9092,kafka-2.kafka-headless.kafka.svc.cluster.local:9092

The Helm installer also gives the command with broker list. To consume messages from a topic (e.g. test) to a given consumer group:

kafka-console-consumer.sh --topic test --bootstrap-server kafka.kafka.svc.cluster.local:9092 --group my-group

We can have two command terminals, post test messages on one terminal and watch it consumed nearly immediately on the other terminal.

The Kafka trigger documentation suggests that the number of replicas will not exceed the number of partitions on a topic when a topic is specified. To make this lab work, we need to have set 5 partitions:

kafka-topics.sh --alter --bootstrap-server kafka.kafka.svc.cluster.local:9092 --topic test --partitions 5
kafka-topics.sh --describe --bootstrap-server kafka.kafka.svc.cluster.local:9092 --topic test

Once we confirm five partitions, we can spin up two command terminals, one to produce message and the other to consume messages. If working, we can stop the consumer and use the command below to watch for the lag for each partition.

kafka-consumer-groups.sh --bootstrap-server kafka.kafka.svc.cluster.local:9092 --describe --group my-group

Now we can artificially trigger scaling by increasing average lag. We keep posting messages on the producer (each carriage return posts a message), and we can check the lag after posting:

Growth of average lags

The size of deployment starts with 0 as defined in the scaled object. As the average exceeds 5, we can see deployment size growing.

Growth of deployment size

This lab is an oversimplified scenario to illustrate the idea of scaling. Kafka is a typical queue construct and other queue configuration such as RabbitMQ or AWS SQS works in very similar ways. Real life use case involves more aspects to consider, such as multiple topics, and authentication.

Connection triggered wake-up

KEDA uses Events to scale workload from zero to one (wake up). There is no way to scale (wake up) based on an incoming web request. In many cases, such as serverless configuration, we need to scale the deployment size from zero to N once the service receives incoming web request. This is not supported by KEDA. By definition KEDA uses Events to wake up.

There is an HTTP-add-on for KEDA still at beta but it is trying to address this problem. This page shows the design. Suppose a service has scaled down to zero, the followings will happen to wake it up:

The incoming request is routed to an interceptor behind the service
interceptor keeps track of number of pending HTTP request
The scaler periodically watches for the size of the pending queue on the interceptor
Based on the queue size, the scaler reports scaling metrics as appropriate to KEDA
As the queue size increases, the scaler instructs KEDA to scale up as appropriate

The periodical check activity is the key to make it work and also what makes it a pseudo-trigger.

Summary

Fine-tuning autoscaling is important to the performance of workload on Kubernetes. At node level, we briefed on cluster autoscaler and suggest that we only use it sparsely. At pod level, we introduced native HPA as well as KEDA, with an example. We also discussed KEDA has limitations and the HTTP-add-on. In the next post, we’ll explore Knative’s autoscaling capability.

Istio Operation Gotchas

Sat, 19 Mar 2022 11:09:00 -0400

In this post I discuss a few aspects when putting istio in operation.

Installation

Istio installation can be confusing, due to architectural and guideline changes as well as renaming of operator CRDs since its release, and especially since 2020. This left lots of information outdated on the web, adding to Istio’s perceived complexity.

Currently, the recommended installation methods are istioctl or Helm. Using Istio Operator manifest (with out istioctl) is discouraged. As to the Helm chart installer, it was once deprecated (around early 2020), but was later re-introduced in the fall of 2021. This back-and-forth had caused some aversion. As of Nov 2021, their re-introduced Helm charts version dropped alpha tag. The Helm repo consists of separate Helm charts, for Istio CRD (base), control plane (istiod), each gateway and CNI respectively. A typical deployment therefore requires multiple Helm Releases (example here). Istio document still considers Helm support as alpha, so I assume the most reliable method to install Istio is istioctl.

The istioctl utility can be used with many options to customize Istio install. For example, we can supply a YAML declaration input (to -f switch) to customize installation behaviours. The YAML file declares a CRD. Two different CRDs have been used: IstioOperator and IstioControlPlane. According to this blog and this post, since Istio 1.5 in early 2020, we’re supposed to IstioOperator CRD exclusively. The IstioControlPlane CRD is left only for legacy support. As stated in the current documentation:

The istioctl command supports the full IstioOperator API via command-line options for individual settings or for passing a yaml file containing an IstioOperator custom resource (CR).

With IstioOperator CRD, we still have a number of options to tweak the install behaviours. Here is a summary of potential options:

use IstioOperator API via IstioOperator CRD (without using “values” or overlay fields)
specify an attribute value in argument, including a pre-built profile e.g. –set meshConfig.accessLogFile=/dev/stdout, –set profile=demo
use K8sObjectOverlay by using “k8s/overlays/patches” field in IstioOperatorCRD
use Helm API by using “values” field in IstioOperatorCRD

The key-value specified in –set switch overrides the same key-value supplied in the IstioOperator CRD. So option 2 overrides option 1. The value for profile can also be empty if you’d rather start from scratch. However too many –set switches makes the command wordy so we should build our own IstioOperator CRD

For option 4, the document hyper-links Helm API to a section from version istio 1.4, and I appears to exist only for legacy (pre-2020 Helm support) compatibility. Option 3 (K8sObjectOverlay) would be helpful when a field cannot be conveniently customized with option 1 and we have to patch the object like in Kustomization.

So the most practical approach is IstioOperator CRD for per-component customization, potentially with K8sObjectOverlay. No matter which option, istioctl compiles the installation manifest before applying it against Kubernetes API. This manifest can be previewed using “istioctl manifest” command, so that you can take a look before installation.

Thank you Isito for so much confusion just to land on a working installation method. Below is the content of az-istio-operator.yaml file that I use for my installation:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
 name: istio-install-customization
spec:
 profile: default
 hub: docker.io/istio
 tag: 1.13.1
 revision: 1-13-1
 namespace: istio-system
 meshConfig:
 accessLogFile: /dev/stdout
 outboundTrafficPolicy:
 mode: "REGISTRY_ONLY"
 components:
 pilot:
 k8s:
 hpaSpec:
 maxReplicas: 7
 minReplicas: 1
 nodeSelector:
 beta.kubernetes.io/os: linux
 ingressGateways:
 - name: istio-ingressgateway
 #namespace: istio-ingress
 enabled: true
 label:
 istio: ingressgateway
 k8s:
 hpaSpec:
 maxReplicas: 11
 minReplicas: 1
 serviceAnnotations:
 service.beta.kubernetes.io/azure-load-balancer-internal: "true"
 service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "my-lb-subnet"
 egressGateways:
 - name: istio-egressgateway
 enabled: true

I can preview the install, run the install and validate installation status:

$ istioctl manifest generate -f az-istio-operator.yaml
$ istioctl install -f az-istio-operator.yaml -y --verify
$ istioctl verify-install -f az-istio-operator.yaml
$ kubectl -n istio-system get IstioOperator installed-state-istio-install-customization-1-13-1 -o yaml | less

Note that in the IstioOperator declaration I marked the revision. This is helpful when I run multiple versions of control plane (e.g. during upgrade). I can check revisions of istiod with:

$ istioctl x revision list

We can delete installed istio components

$ istioctl x uninstall -f az-istio-operator.yaml
$ istioctl x uninstall --revision 1-11-5

In practice, it is helpful to use separate operators each for a different component (pilot, ingressGateways, egressGateways). This makes maintenance and upgrade easier.

Debugging

The documentation has a page for common problems that one needs to be familiar with. It covers not only problems, but also steps to troubleshoot each kind of problem (e.g. authorization policy).

Although istioctl is pretty confusing as an installation tool, it is a good utility for many troubleshooting activities. We should probably add its path to PATH environment variable and add the export command (e.g. export PATH=”$HOME/istio/bin:$PATH“) to .zshrc or .bashrc. Istioctl has a few useful subcommands, some of which are only available as experimental and therefore needs to be following an x. Some common commands are given below:

To analyze Istio problems:

$ istioctl analyze -n istio-system

To look at proxy configuration of an Envoy instance at different levels, use proxy-config sub-command or pc for shorthand:

$ istioctl proxy-config <clusters|listeners|routes|endpoints|bootstrap|log|secret|stats>
$ istioctl pc cluster deploy/istio-ingressgateway -n istio-system # see what Envoy cluster an ingress gateway knows about
$ istioctl proxy-config log deploy/httpbin --level "rbac:debug"
$ istioctl pc log <pod_name> -n <namespace> --level connection:debug
$ istioctl pc secret -n istio-system deploy/istio-ingressgateway # check certificates loaded to a gateway
$ istioctl proxy-config listeners deploy/istio-ingressgateway -n istio-system # query envoy listener configuration
$ istioctl pc routes deploy/istio-ingressgateway -n istio-system --name http.8080 # query envoy route configuration

The last two commands set logging level to the specified workload. If we want to set logging level at mesh level, we can use these commands:

$ istioctl admin log --level authorization:debug
$ istioctl admin log

To look at synchronization status of each envoy in the mesh, use proxy-status sub-command, or ps for shorthand:

$ istioctl ps # ensure data plane is in sync

If an item for a workload shows STALE instead of SYNCED, it means that the configuration has not been pushed from control plane to that instance of Envoy proxy. Check if the Istio configuration change is valid. If the system is newly installed and there is no ingress or egress gateway resources declared, the RDS column for ingress or egress may show “NOT SENT”. The far right column displays the version of istiod connected.

To describe applied istio config:

$ istioctl describe <pod|service>
$ istioctl describe po workload1 -n my-workload # detect misconfigurations on workload

To view dashboard:

$ istioctl dashboard <envoy|grafana|prometheus>

To check authorization policy on a Pod,

$ istioctl x authz check mypod -n workload

To validate istio configuration in a file:

$ istioctl validate -f resource_authorization_policy.yaml

Sometimes we need to turn on access logging just on the envoy proxy on the Gateway Pod. In that case, we will need to apply Istio’s Envoy filter object. This filter is applied to Pods labelled as gateway. It patches the existing filter chain with the additional defined in the manifest:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
 name: hcm-gw-access-log
 namespace: istio-system
spec:
 workloadSelector:
 labels:
 istio: ingressgateway
 configPatches:
 - applyTo: NETWORK_FILTER
 match:
 context: GATEWAY
 listener:
 filterChain:
 sni: demo.digihunch.com
 filter:
 name: "envoy.filters.network.http_connection_manager"
 patch:
 operation: MERGE
 value:
 typed_config:
 "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
 access_log:
 - name: envoy.access_loggers.file
 typed_config:
 "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
 path: /dev/stdout
 format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"

The object above enables access logging on the Gateway Pods only, without impacting other Envoy proxies.

Multi-tenancy

In large enterprises, Istio is usually installed by a platform/operation team. The entire platform is shared by multiple application teams. Istio should not be seen as a responsibility of a single party. It is necessary to break down the Istio CRDs and define the responsibility of each CRD. For example:

Gateways are placed in the istio-system namespace (or a dedicated istio-ingress namespace in some case). Platform team can decide whether a Gateway is considered a shared infrastructure, or each tenant (application team) uses their own Gateway. In the servers/hosts field of Gateway declaration, add namespace before host to suggest that the routing behaviour of that host must be defined in a certain namespace.
Virtual Services can be managed in two models as well. They can be managed individually and are all placed in each tenant namespace. Alternatively in a shared responsibility model, Virtual Service can be created in istio-system namespace, and the processing of each match can be delegated to a Virtual Service in each tenant namespace, using the Delegate feature of Virtual Service. Another field that can help with multi-tenancy is the “gateways” field, you can specify a value of mesh to indicate the virtual service is available to the entire mesh.
Destination Rules are usually placed in each Tenant’s workspace
Peer Authentication also depends on the configuration. If a mesh level configuration is enforced, it is easier for the Platform Team to manage it and this should be the setup for new clusters. If for historical reasons Peer Authentication is enforced per tenant namespace, it can be delegated to each application team. It can make communication troubleshooting more complex.
Control Plane and observability workloads such as Kiali are usually the responsibility of Platform team.
Request Authentication and Authorization Policy should be the responsibility of individual application team.

In a multi-tenancy management model, Istio related resource should be subject to admission control policy, as well as scrutiny by the security and platform teams. Google Anthos has a good page on the constraint templates for Istio resources.

As to restricting traffic between namespaces, apart from the measures from this previous post, we can also use Sidecar CRD to restrict outbound traffic. As its documentation states:

Sidecar describes the configuration of the sidecar proxy that mediates inbound and outbound communication to the workload instance it is attached to. By default, Istio will program all sidecar proxies in the mesh with the necessary configuration required to reach every workload instance in the mesh, as well as accept traffic on all the ports associated with the workload. The Sidecar configuration provides a way to fine tune the set of ports, protocols that the proxy will accept when forwarding traffic to and from the workload. In addition, it is possible to restrict the set of services that the proxy can reach when forwarding outbound traffic from workload instances.

The documentation page also includes two examples. The first is a sidecar at the mesh level that restricts outbound traffic to the same namespace that the sidecar is in, and the istio-system namespace:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
 name: default
 namespace: istio-config
spec:
 egress:
 - hosts:
 - "./*"
 - "istio-system/*"

The second example overrides the mesh level default above, and allows egress traffic to three specified namespaces:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
 name: default
 namespace: prod-us1
spec:
 egress:
 - hosts:
 - "prod-us1/*"
 - "prod-apis/*"
 - "istio-system/*"

As to inbound control, we can expose a Virtual Service to other namespaces by using the exportTo field to specify which other namespaces the Virtual Service should be exported to. If no namespaces are specified then the virtual service is exported to all namespaces by default.

Sometimes we want a virtual services to expose to both outside of the mesh via Ingress, and within the mesh, and we hope to use the same hostname. For this requirement, we can use the ServiceEntry CRD. ServiceEntry enables adding additional entries into Istio’s internal service registry.

Service Proxy – from Nginx to Envoy

Wed, 09 Mar 2022 21:36:00 -0400

Update (Nov 20, 2022): 1. Envoy’s configuration schema can be hard to get used to. It is lacking examples because the documentation is mostly generated. Use its examples directory to find real-life configuration examples. 2. the Envoy implementation in the example project has been reverted in favour of Nginx.

Envoy proxy is the underlying technology for Istio, as well as a number of other service mesh products, such as AppMesh (AWS), Consul (Hashicorp) and OpenServiceMesh (Azure). Most of the capabilities of Isito is ultimately provided by Envoy proxy. Envoy has a page outlining its differences with similar technologies. I decided to take a look into Envoy by replacing Nginx with it.

Rate limiting and Circuit Breaker

In most SDLC, it is application developers that create backend APIs or server applications. Most developers specializes in application features, and cannot fathom all the nuances with the TCP/IP network stack. Nginx allows them to push network concerns (non-business features) to a dedicated proxy to handle the dynamics in network connection. Nginx can be configured as both a reverse proxy (handling incoming connection on behalf of the process) and a forward proxy (handling outgoing connection on behalf of the process). This is the prototype of sidecar pattern, an important idea behind service mesh. For example, when a sudden increase in connection to the server-side application, the server process could be either unresponsive (refer to “the queuing knee“, and Little’s Law), or OOM killed. When such interruptions are not automatically recovered, a downtime is caused. This traditionally requires some congestion control strategy for TCP/IP queue but two features provided by a network proxy can help circumvent this situation: rate limiting, and circuit breaking. Rate limiting keeps more requests above threshold from entering the queue. Circuit breaker releases downstream pressure by cutting out existing in-queue request. Nginx added both over the years but Circuit breaker still remains a premium feature exclusive to Nginx Plus. Envoy on the other hand has them free when it was launched.

Envoy also supports other advanced traffic management such as traffic shaping, and mirroring. It is on top of those features that Istio introduces its own abstraction such as virtual service, destination rules to its users. In that sense, we can think of Istio as a configurator (control plane) for Envoy proxy (data plane), similar to Ansible to Nginx proxy instances.

Dynamic Configuration via API

I used Nginx previously with traditional environment and loved its flexibility. As the system grows, I started to feel the pain of management overhead. With one of the production system, there were 25 + instances of Nginx each running on a VM and I managed configuration files with Ansible. Ansible pushes out configuration files and triggers a reload from each Nginx instance. In the cloud-native era where Pods are ephemeral, this kind of overhead would snowball to an unmanageable level. Envoy was designed for cloud-native applications, with all these kinds of problems in mind. Envoy has dynamic configuration. The majority of the configurations can be pulled from xDS API, or file system. Updating configuration drains connections gracefully without runtime having to reload the file. The idea of centrally managing Nginx instances with Ansible, also evolved into the concept of control plane.

TLS origination

In the Orthweb project, I used Nginx to proxy TLS and HTTP traffic, and performed TLS termination on both ports. This is know as TLS offloading. The traffic between the proxy and the upstream service takes place in the clear, even though they do not travel across different network interfaces in most cases. For a true end-to-end encryption, it is helpful to also encrypt the traffic between proxy and upstream server. This requires the capability of securing TCP traffic to upstream server. With Nginx, the ability to secure HTTP traffic to upstream server is offered in open-source. The ability to secure TCP traffic is available in Nginx Plus, or with self-compiled binary. In Envoy, both are available using the UpstreamTlsContext configuration.

More pros

In another post, I also discussed Nginx as a LDAP proxy to front services such as Kibana and Nifi. It requires a proxy service (ldap-auth in this case), to defer auth to third party. Envoy has this capability using a filter with extension for external authorization. Istio also exposes this capability, an enabler for the configuration proposed in my previous post. Envoy also uses WebAssembly for its extensibility.

Another useful feature is protocol detection. It can use filters to detect protocol (TLS or regular TCP) and route traffic to predefined destination.

Performance wise, this benchmark from 2018 ran a comparison among the popular options where Envoy leads by a margin.

Observability (logging, metrics and tracing) are well supported in Envoy. User can configure format of logs that takes effect immediately. There are many metrics that works with Prometheus and they are expandable using filters. On the tracing side, Envoy supports integration with jaeger, zipkin and datadog.

Basics of Envoy

The configuration of Envoy is more involving. There is an Envoy course by Tetrate, as well as two blog entries for envoy 101: Envoy as gateway proxy and File-based dynamic configuration. Another good way to get started is the Sandboxes projects, which covers a number of different areas of configuration. The admin port (by default at port 9901. Istio’s default is 15000) provides helpful information. If we need to turn on debug on some features, we can do so with curl:

curl -X POST http://localhost:9901/logging?client=debug

Stats are exposed at the same port:

curl -X GET http://localhost:9901/stats

When packets are received at a listener, the is first processed by listener filters. Then, depending on filter match, one or more network filter chains will further process the packet, including further actions, as illustrated below:

Envoy supports dynamic configuration, which uses a set of discovery services (xDS) APIs. Some of the important xDS APIs include:

LDS (Listener Discovery Service) – allows you to add listeners dynamically while Envoy is running
RDS (Route Discovery Service) – allows you to dynamically update routes for HTTP connection managers
CDS (Cluster Discovery Service) – allows you to update cluster definitions dynamically
EDS (Endpoint Discovery Service) – allows you to add or remove endpoints dynamically
Secret DS

The relation can be illustrated in this diagram below:

This post from Tetrate has more examples.

Nginx to Envoy

For the advantages of Envoy, I decided to migrate from Nginx to Envoy on my Orthweb project. Using Envoy as service proxy is not where Envoy is mostly used (as sidecar), but it is how Envoy was originally used at Lyft to replace ELB in 2015.

The original Nginx configuration was referenced in this old blog post. The Envoy setup also covers both TCP (DICOM) and HTTP (HTTPS) traffic. For HTTP traffic, it also encrypts the traffic to upstream. Below is what it looks like:

admin:
 address:
 socket_address: { address: 0.0.0.0, port_value: 9901 }
static_resources:
 listeners:
 - name: https_listener
 address:
 socket_address:
 address: 0.0.0.0
 port_value: 443
 filter_chains:
 - filters:
 - name: envoy.filters.network.http_connection_manager
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
 codec_type: AUTO
 stat_prefix: ingress_http
 route_config:
 name: local_route
 virtual_hosts:
 - name: app
 domains:
 - "*"
 routes:
 - match:
 prefix: "/"
 route:
 cluster: service-https
 http_filters:
 - name: envoy.filters.http.router
 transport_socket:
 name: envoy.transport_sockets.tls
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
 common_tls_context:
 tls_certificates:
 - certificate_chain: {"filename": "/etc/ssl/certs/site.pem"}
 private_key: {"filename": "/etc/ssl/certs/site.pem"}
 - name: dicomtls_listener
 address:
 socket_address:
 address: 0.0.0.0
 port_value: 11112
 filter_chains:
 - filters:
 - name: envoy.filters.network.tcp_proxy
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
 stat_prefix: downstream_cx_total
 cluster: service-dicomtls
 transport_socket:
 name: envoy.transport_sockets.tls
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
 common_tls_context:
 tls_certificates:
 - certificate_chain: {"filename": "/etc/ssl/certs/site.pem"}
 private_key: {"filename": "/etc/ssl/certs/site.pem"}
 validation_context:
 allow_expired_certificate: true
 trusted_ca: {"filename": "/etc/ssl/certs/site.pem"}
 require_client_certificate: false
 clusters:
 - name: service-https
 type: STRICT_DNS
 lb_policy: ROUND_ROBIN
 load_assignment:
 cluster_name: service-https
 endpoints:
 - lb_endpoints:
 - endpoint:
 address:
 socket_address:
 address: orthanc-backend
 port_value: 8042
 transport_socket:
 name: envoy.transport_sockets.tls
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
 - name: service-dicomtls
 type: STRICT_DNS
 lb_policy: ROUND_ROBIN
 load_assignment:
 cluster_name: service-dicomtls
 endpoints:
 - lb_endpoints:
 - endpoint:
 address:
 socket_address:
 address: orthanc-backend
 port_value: 4242
layered_runtime:
 layers:
 - name: static_layer_0
 static_layer:
 envoy:
 resource_limits:
 listener:
 https_listener:
 connection_limit: 1000
 overload:
 global_downstream_max_connections: 5000

From Nginx to Envoy, to achieve nearly the same functionalities, it takes 100 lines of configuration instead of less than 30. The configuration also appears more abstract, which is one of the cons of Envoy to consider before the migration.

Update from Jan 2025 – To align with what most uses in Orthanc community, the project switched back to using Nginx. The envoy configuration is kept here.

Istio External Authorization via OIDC

Fri, 25 Feb 2022 01:29:00 -0400

Istio service mesh allows application developers to offload non-core features to infrastructure layer. We explored authentication and authorization with Istio in a basic lab. In this post we continue to explore its capabilities with OIDC integration. This capability is made available thanks to the CUSTOM action in authorization policy, supported since the release of 1.9. It enables any workload on Istio to integrate with an external IAM solution.

The rest of this post, provides the step-by-step instruction to configure OIDC integration, based on Istio’s External Authorization use case. My work is influenced by two blog posts from jetstack and elastisys on similar topic, with my own additions, simplifications and clarifications. In this lab I use my own DNS hostname demo1.digihunch.com and to follow along you need to bring your own hostname as well. The prerequisites of this lab is summarized as below:

An identity provider: we use Google in the lab but it can be anything with OIDC capabilities (e.g. Azure AD, Facebook). Read my old post for more details on OIDC.
A public IP address for Cluster’s Ingress service: Since we use Google as identity provider on the Internet, we need our service to be exposed on the Internet because the OIDC integration involves a two-way HTTP redirect.
Public DNS resolution to the service’s public IP: during authentication, the identity provider will call back with JWT. So we need our hostname demo1.digihunch.com resolved to the service’s public IP address. This is also required in other scenarios. For example, cert-manager needs to automatically configure X.509 certificate using Let’s Encrypt.
An application workload: this is the tenant workload without its own IAM capability. We use a dummy workload base on a hello-world image, for simplicity.

The files required in this lab is in istio-oidc repo under the envoy directory. It is also important to understand the Authentication Code flow in OIDC authorization code flow to make sense of the integration between OIDC provider and Istio on our platform. For example, the client application needs a one-time registration with identity provider. Let’s start with this configuration.

Configure Identity Provider

We need a Google account with a GCP project and log on to APIs & Services to register our application. At the console, from the left side bar, click on “OAuth consent screen” and create an App. Provide the followings:

Step 1. OAuth consent screen. Under App Information, provide App name and User support email.
Step 1. OAuth consent screen. Under App domain, provide Application home page
Step 1. OAuth consent screen. Under Authorized domains, provide an Authorized domain.
Step 3. Test users. Add the emails for Test users.

The summary page looks like this:

OAuth Consent Screen

Then on the sidebar, click on Credentials. We create new credentials for OAuth 2.0 Client ID. Select Web application and add “https://demo1.digihunch.com/oauth2/callback” as an Authorized redirect URI. The credential created includes a Client ID and a Client secret, which can be downloaded as a JSON file. Here is my screen:

API Credential

Note that the redirect URIs cannot be localhost. Because the browser will consume the redirect URI and it has no clue where localhost is. It must resolve to the public IP of the ingress.

We need to keep the Client ID and Client secret for later use. According to Google’s documentation, there is a discovery document from which we can understand the keys in their Open ID configuration. We need the value of two keys (issuer and jwks_url) to use later in the configuration.

Configure Infrastructure Services

My real-quicK-cluster project has a few different ways to quickly bring up a cluster. For this lab, I use Azure CLI command to create a simple, three-node cluster:

$ az aks create \
 -g AutomationTest \
 -n orthCluster \
 --node-count 3 \
 --enable-addons monitoring \
 --generate-ssh-keys \
 --vm-set-type VirtualMachineScaleSets \
 --network-plugin azure \
 --network-policy calico \
 --tags Owner=MyOwner

Then we configure kubeconfig credential to connect to the cluster, so that kubectl and helm CLI commands to connect to the newly created cluster.

$ az aks get-credentials --resource-group AutomationTest --name orthCluster

Then we will install the infrastructure services and workload. We can start with loading environment variables for use later:

OIDC_DISCOVERY=$(curl "https://accounts.google.com/.well-known/openid-configuration")
OIDC_ISSUER_URL=$(echo $OIDC_DISCOVERY | jq -r .issuer)
OIDC_JWKS_URI=$(echo $OIDC_DISCOVERY | jq -r .jwks_uri) 
COOKIE_SECRET=$(openssl rand -base64 32 | tr -- '+/' '-_')
WEB_HOST="demo1.digihunch.com"
CLIENT_ID="ThisIsTheClientIDFromGoogle"
CLIENT_SECRET="ThisIsTheClientSecretFromGoogle"

For WEB_HOST, CLIENT_ID, CLIENT_SECRET, you need to bring your own values. The COOKIE_SECRET value is randomly generated. The variables OIDC_ISSUER_URL and OIDC_JWKS_URI are parsed from Google OpenID configuration and they should remain static. We can validate the value with echo commands. Also, to use helm v3 later, we need to add the repositories we need. These steps can be skipped if they have been performed in the client environment:

$ echo $OIDC_ISSUER_URL
https://accounts.google.com
$ echo $OIDC_JWKS_URI
https://www.googleapis.com/oauth2/v3/certs
$ helm repo add jetstack https://charts.jetstack.io
$ helm repo add istio https://istio-release.storage.googleapis.com/charts
$ helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
$ helm repo update

Now, with Helm ready, we can install Cert Manager, Istio CRD, Control Plane, Gateways as well as OAuth2-Proxy using Helm. We run each of the following commands from the envoy directory, where the required files are stored:

$ helm install cert-manager jetstack/cert-manager \
 --namespace cert-manager \
 --create-namespace \
 --version v1.7.1 \
 --set installCRDs=true
$ helm install -n istio-system istio-base istio/base --create-namespace
$ helm -n istio-system install istiod istio/istiod -f istiod-values.yaml --wait
$ helm -n istio-system install istio-ingress istio/gateway -f ingress-gateway-values.yaml
$ kubectl -n istio-system get po
$ kubectl create ns oauth2-proxy && kubectl label ns oauth2-proxy istio-injection=enabled
$ helm install -n oauth2-proxy \
 --version 6.0.1 \
 --values oauth2-proxy-values.yaml \
 --set config.clientID=$CLIENT_ID \
 --set config.clientSecret=$CLIENT_SECRET \
 --set config.cookieSecret=$COOKIE_SECRET \
 --set extraArgs.oidc-issuer-url=$OIDC_ISSUER_URL \
 --set extraArgs.whitelist-domain=$WEB_HOST \
 oauth2-proxy oauth2-proxy/oauth2-proxy
$ kubectl -n oauth2-proxy get pods -l app=oauth2-proxy

Note that the istiod installation step uses a value file that includes extensionProviders. This is where we tell Istio to connect to external authorization provider:

meshConfig:
 accessLogFile: /dev/stdout
 extensionProviders:
 - name: oauth2-proxy
 envoyExtAuthzHttp:
 service: oauth2-proxy.oauth2-proxy.svc.cluster.local
 port: 4180
 includeRequestHeadersInCheck:
 - cookie
 headersToUpstreamOnAllow:
 - authorization
 headersToDownstreamOnDeny:
 - set-cookie

When we install istio ingress gateway, we need to expose port 80 as well as 443. Port 80 is used in ACME protocol for certificate configuration. We also installed oauth2-proxy with some configurations from oauth2-proxy-values.yaml as well as some argument set imperatively.

Once configuration is successful, we should be able to confirm that oauth2-proxy service is running. We should also be able to tell the External IP address of the Ingress:

$ kubectl -n oauth2-proxy get svc
$ kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Now we can go to our DNS configuration portal, to populate the DNS A-record for demo1.digihunch.com with this IP address:

Add DNS A-record

It may take several minutes to a couple of hours, for the A-record update/creation to take effect, depending on the TTL. Sit tight and query DNS with nslookup command, until it is set. In the next step, we will need successful DNS resolution for certificate configuration.

Install workload and configure Certificate

Now we install the workload, along with the necessary constructs such as Gateway CRD, Virtual Service, and certificates. We can use kubectl kustomize command to preview the changes, then apply the changes with -k:

$ kubectl kustomize demo
$ kubectl apply -k demo

When you have your own DNS hostname, modify the kustomization.yaml accordingly before applying. This command uses Kustomization to apply numerous YAML manifests in the demo directory, including the following activities:

Create a Namespace named demo
Create Deployment and Service
Set up PeerAuthentication for the mesh
Configure Ingress class, ClusterIssuer and Certificate using cert-manager
Configure Virtual Service and Gateway

This configures everything we need and we need to verify from several aspects. The demo Service is a ClusterIP service exposed within the cluster at port 80. Then we check the status of certificate:

$ kubectl -n istio-system get certificate
$ kubectl -n istio-system describe certificate demo

The column of READY should have a value of True. Check the logs from cert manager Pods if that’s not the case. Common reasons include:

The automatic ACME validation is just not ready
The automatic ACME validation is still waiting for DNS resolution
The Istio ingress gateway port 80 is not open for ACME validation
The let’s encrypt server applies rate limiting

Note that for the ACME server, we can use staging server’s URL or productions. The former provisions a certificate that may come off as insecure as the CA is not fully trusted by browser. The latter doesn’t have this issue, but the server applies rate limiting. For more details of how validation with ACME work, check out my previous post.

Once the certificate is applied, our website is open to any visitor with HTTPS access:

Then we will use Request Authentication and Authorization Policy to tighten up the access by requiring visiting user to log in with Google identity.

RequestAuthentication and AuthorizationPolicy

We configure a CUSTOM action in the AuthorizationPolicy, and specified the provider by name oauth2-proxy. The YAML manifests for RequestAuthentication and AuthorizationPolicy looks like this:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
 name: istio-ingressgateway
 namespace: istio-system
spec:
 jwtRules:
 - issuer: https://accounts.google.com
 jwksUri: https://www.googleapis.com/oauth2/v3/certs
 selector:
 matchLabels:
 app: istio-ingressgateway
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: istio-ingressgateway
 namespace: istio-system
spec:
 action: CUSTOM
 provider:
 name: oauth2-proxy
 rules:
 - {}
 selector:
 matchLabels:
 app: istio-ingressgateway

Note that we cannot apply Request Authentication and Authorization Policy before we confirm that the Certificate has been created. This is because when letsencrypt use HTTP resolver to provision certificate with ACME protocol, the request from letsencrypt should get to the validating URI without being asked to provide a Google identity. So do not perform this step yet before confirming the previous step:

$ kubectl apply -f oidc-auth.yaml

Now launch your browser in incognito mode and browse to our URL (https://demo1.digihunch.com) and you will be re-redirected to log on to Google (accounts.google.com). Once logged on, you will have access to the site.

If you run into errors, check the oauth2-proxy log from Pod stdout, which should give a reason of 4xx errors. If oauth2-proxy log indicates no activity, confirm if the request has been forwarded to the proxy. Check the service object of the proxy and make sure it is exposed to the correct port, as indicated in the meshConfig. To check mesh config, examine the configmap named istio in the namespace of istio-system. To check if Istio’s authorization is unable to speak with oatuh2-proxy, inspect the log of istiod Pod.

To clean up the lab, remove the app registration from Google, and then delete the cluster from Azure:

$ az aks delete -g AutomationTest -n orthCluster

In this lab we used a very open AuthorizationPolicy. We can polish it up with more conditions such as:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: istio-ingressgateway
 namespace: istio-system
spec:
 action: CUSTOM
 provider:
 name: oauth2-proxy
 rules:
 - when:
 - key: request.auth.audiences
 values:
 - $CLIENT_ID
 selector:
 matchLabels:
 app: istio-ingressgateway

Under the rules section, we can explore the claims from the JWT and make creative use of it to achieve our specific goals for IAM.

Bottom line

In this lab, we registered our app with Google and obtained client ID and secret for it to identify our application. We configured oauth2-proxy as our IAM application, carrying the credential validated by Google. We also configured Istio to delegate authorization to oauth2-proxy as external authorization provide, making up an entire OIDC integration.

The third party service (IdP) we integrate with is GCP, which natively supports OIDC protocol. However, if the third party IdP does not support OIDC natively, such as Active Directory via LDAP, then we would need one more proxy to sit between OAuth2 and the external IdP service. One good choice is the Dex connector. Another more complex yet powerful alternative connector is Key Cloak. Consider Dex or KeyCloak as an OIDC proxy, when:

The external IdP does not support OIDC; or
We choose not to use the OIDC capability of the external IdP because we want to managed OIDC centrally in our own proxy
The external identity provider does support OIDC, but the configuration is managed by a different team (e.g. IAM team) other than the platform team

AuthorizationPolicy is the key piece in this integration, and it is executed at the http filter in envoy sidecar proxy. It can be done with alternatives to OAuth2-Proxy such as the authservice project. A slightly different approach to implement OIDC integration is to use leverage extensibility of WebAssembly (a sandboxing technology to extend Envoy), which is suggested in this example for WasmPlugin. However, Wasm is still considered experimental.

Istio Lab – Authentication and Authorization

Sun, 13 Feb 2022 13:21:13 -0400

My previous blog discussed as service mesh what Istio can offer in terms of authentication and authorization capabilities. Istio can authenticate an incoming HTTP request, ensuring the JWT issued has not been tampered somewhere in the middle. The fields in the JWT allows for more flexibilities at the point of authorization. This combination allows Istio to integrate with identity providers that can issue JWT.

We’ve also discussed how JWT works, and pointed out that the two key element of request authentication is the JWT (payload signed with private key) itself as well as the JWK (carrying public key). In this post, we will test it in a lab. To start with this lab, we need a test cluster (e.g. Minikube) with Istio installed.

Preparation

I have not find a native Bash way to produce JWT. We will do that with python packages python_jwt and jwcrypto in Python3. Let’s install the modules and import them in Python environment.

$ python3 -m pip install python_jwt jwcrypto datetime
$ python3
>>> import python_jwt as jwt, jwcrypto.jwk as jwk, datetime

Now we’re in the Python3 shell with needed modules loaded. We can take the following steps to produce the JWT as well as the JWK:

RSAkey = jwk.JWK.generate(kty='RSA', size=2048)
private_key = RSAkey.export_private()
public_key = RSAkey.export_public()
raw_payload = {'iss':'digihunch.com','sub':'DIGIHUNCH','role':'reader','permission':'read'}

## use private key to generate jwt token. HTTP request will bear this token 
jwt_token = jwt.generate_jwt(raw_payload, jwk.JWK.from_json(private_key), 'RS256', datetime.timedelta(minutes=50))
print(jwt_token)

## The JWKS keeps public key and is referenced by Istio RequestAuthentication object
jwks='{"keys":['+public_key+']}'
print(jwks)

# Helpful command to print the key in PEM format:
## RSAkey.export_to_pem(private_key=False)
# Helpful command to verify JWT token:
## header, claims = jwt.verify_jwt(jwt_token, jwk.JWK.from_json(public_key), ['RS256'])

To complete this lab, we need the values of jwt_token and jwks.

Next in the preparation is a local cluster, istio with metallb installed, which is covered in a previous post. We should be able to get the ingress IP address:

$ export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ echo $INGRESS_HOST

We create our own namespace and use the httpbin application:

$ kubectl create ns web 
$ kubectl label namespace web istio-injection=enabled
$ kubectl apply -n web -f httpbin.yaml
$ kubectl apply -n web -f httpbin-gateway.yaml
$ curl -I -XGET $INGRESS_HOST/headers

The curl command above (without any token) should return HTTP 200 code, indicating that no authorization token is needed to connect to the service. Note that the Pod for httpbin has the label app=httpbin which will be used in the request authentication. Also note that traffic is served over a named port called http in the Service object for http, which will implicitly enable HTTP based conditions for authorization policies we will build later.

Request Authentication

Let’s create a request authentication object with the following manifest:

apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
 name: jwt-req-authn
 namespace: web
spec:
 selector:
 matchLabels:
 app: httpbin
 jwtRules:
 - issuer: "digihunch.com"
 jwks: |
 ## jwks output from previous step ##

Replace the last line with the jwks output from the preparation step and store it to jwt-req-authn.yaml. It should look like this:

jwt-req-authn.yaml

Then apply it to the web namespace:

$ kubectl -n web apply -f jwt-req-authn.yaml

Now the request authentication resource is applied to the httpbin workload. We first test it with a random authentication token and it should be denied of 401 error:

$ curl -I -XGET $INGRESS_HOST/headers --header "Authorization: Bearer randomstring"
HTTP/1.1 401 Unauthorized
www-authenticate: Bearer realm="http://192.168.64.16/headers", error="invalid_token"
content-length: 79
content-type: text/plain
date: Sun, 13 Feb 2022 16:13:32 GMT
server: istio-envoy
x-envoy-upstream-service-time: 31

Then we take the jwt_token from the preparation step and present it to the request authentication resource by sending an HTTP request with the appropriate authorization token. It should return an HTTP 200 code this time:

$ curl -I -XGET $INGRESS_HOST/headers --header "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDQ3NzE0NjUsImlhdCI6MTY0NDc2ODQ2NSwiaXNzIjoiZGlnaWh1bmNoLmNvbSIsImp0aSI6Im1faUhla2pNbWRmUmlsWEdCaTFBR3ciLCJuYmYiOjE2NDQ3Njg0NjUsInBlcm1pc3Npb24iOiJyZWFkIiwicm9sZSI6InJlYWRlciIsInN1YiI6IkRJR0lIVU5DSCJ9.sVppwmvDqKvSsVdB05a_mDHymZq7Okvnwu-caTywXQgsUvOA6HfaySp_WXMyTp1HQ4WcTqKE4frZm7QNtrZsPso4bdD_4mEDYTswTCWhblaPy236NJBEH3ilB2BVySBVQKsjyxd94F1KV24SFWiR6lUxk52wKKE3ipBwR79jPizhAu9xxrfJ2Lfi5ypNa_kjBdJi63KCt2Y0eW94Fjq3PZs4ZalHJyaXYSx5Gxyei5f7QdpEOBpvs13mSdi9RqkgVQOjE0V1uBRpMckMyZs-IijknJcSu4fkrjgfNmXsrm__-vlM9UjUl2Jlj0x8bRC8l20IZ6t1ml-GFkwj39JC0g"

HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 13 Feb 2022 16:13:54 GMT
content-type: application/json
content-length: 589
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 17

In the example above, jwtRules can be used with other keys such as jwksUri to reference the jwks by Uri. More fields in JWTRules can be found here.

Now we have tested three curl commands:

Without authentication token at all: http server (istio-envoy) returns 200 code.
With an invalid authentication token: http server (istio-envoy) returns 401 code for error.
with a valid authentication: http server (istio-envoy) returns 200 code.

So we have the capability to validate token, but it is not yet mandatory to present the token. We can change this behaviour by tweaking authorization policy.

Authorization Policy

According to Istio documentation, to restrict access to authenticated requests only, this should be accompanied by an authorization rule. We start with the following policy:

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
 name: auth-pol
 namespace: web
spec:
 selector:
 matchLabels:
 app: httpbin
 action: ALLOW
 rules:
 - from:
 - source:
 requestPrincipals: ["*"]

Applying the manifest above to namespace web to it applies to workload httpbin. Then we

$ curl -I -XGET $INGRESS_HOST/headers
HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
date: Sun, 13 Feb 2022 16:29:58 GMT
server: istio-envoy
x-envoy-upstream-service-time: 35

The requestPrincipals clause makes it mandatory to present a token. The RequestAuthentication validates the token.

We can beef up the authorization policies by adding claims to the conditions, for example:

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
 name: auth-pol
 namespace: web
spec:
 selector:
 matchLabels:
 app: httpbin
 action: ALLOW
 rules:
 - from:
 - source:
 requestPrincipals: ["*"]
 to:
 - operation:
 methods: ["GET"]
 when:
 - key: request.auth.claims[iss]
 values: ["digihunch.com"]
 - key: request.auth.claims[role]
 values: ["reader"]

Both the to and when conditions are for HTTP traffic only, and we must tell Istio to inspect the traffic as HTTP, which is done implicitly with named ports on the Service object. Refer to this common problem from Istio’s documentation. The request principals, if a none wildcard value is specified, will be a SPIFFE format identity, the same one used for peer authentication, as discussed in the previous post.

The when clause above contains two key-value pairs. The first looks for the value of a standard claim (iss), the second for a custom claim (role). In the preparation step, we created the claims with those claims in Python and they will match the condition here.

Should any of the conditions above not match, a 403 (Forbidden) error code will be returned by the workload’s istio-envoy proxy.

Verify mTLS connection

In some cases, we need to audit whether the TLS traffic between workloads actually take place in mTLS. As I was developing my korthweb project, I don’t find a straightforward way of validating TLS. We can validate that mTLS mode on a workload using the following istio CTL command:

$ istioctl x describe pod my-workload-pod -n [namespace]

The output shows Effective PeerAuthentication and Applied PeerAuthentication. This verifies configuration. But how can we ensure that traffic are indeed using mTLS? For most of mTLS traffic, we can use Kiali’s observability feature. In Graph, we need to ensure “Security” is checked in the display drop-down. The pad lock will indicate the traffic is mTLS. To get reliable results, we have to artificially create some live traffic between workloads (e.g. curl from one Pod to another) so Kiali can pick up the update.

Validate mTLS traffic in Kiali

Unfortunately, even artificial traffic does not make Kiali the most reliable way to detect mTLS. There are three other approach to verify TLS traffic.

The first approach is to let Envoy proxy emit TLS related traffic. We can apply the following annotation line to a Pod, to tell its Envoy proxy to emit measurements related to tls_inspector:

sidecar.istio.io/statsInclusionPrefixes: "tls_inspector,listener.0.0.0.0_15006"

The metrics will be exposed to Envoy’s admin port (15000 on istio-proxy) with the path /stats. For example:

$ kubectl -n orthweb exec orthanc-7f4c9b759-lxnrb -c istio-proxy -- curl localhost:15000/stats | grep tls_inspector
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 22194 0 22194 0 0 21.1M 0 --:--:-- --:--:-- --:--:-- 21.1M
tls_inspector.alpn_found: 13
tls_inspector.alpn_not_found: 0
tls_inspector.client_hello_too_large: 0
tls_inspector.connection_closed: 0
tls_inspector.read_error: 0
tls_inspector.sni_found: 13
tls_inspector.sni_not_found: 0
tls_inspector.tls_found: 13
tls_inspector.tls_not_found: 0

We still need some live traffic to bump up the measurement. Comparing between tls_found and tls_not_found is a good way to determine if a Pod is receiving both TLS and plaintext traffic in PERMISSIVE mode.

The second approach to tell mTLS is via the connection_security_policy metric label. It is set to mtutual_tls if the connection is in mTLS. We need the dashboard for Prometheus:

$ istioctl dashboard prometheus

In the prometheus dashboard, click on “Graph” at the top, then search for “istio_tcp_connections_closed_total” for “istio_tcp_connections_opened_total“, the result should include a metrics called connection_security_policy which is labelled as mtutual_tls, as illustrated below:

mTLS metric

For more about collecting and querying metrics from Prometheus, check out Istio’s documentation here and here.

The third approach is to utilize the AUDIT feature of Authorization Policy. When a rule in Authorization Policy has a source with namespace or notNamespace field, it requires the incoming connection to have an SPIFFE identity and use mTLS. We can set the Authorization Policy’s action to AUDIT and use RBAC access logging, the same way we would do to troubleshoot RBAC access issues. From the log we can check RBAC failures due to missing SPIFFE identity.

Istio Authentication and Authorization

Sat, 05 Feb 2022 23:04:00 -0400

Applications running on Kubernetes platform seeks to offload common non-business features to the platform. Istio helps Kubernetes bridge that gap. It can enforce mTLS communication, which is known as Peer Authentication. It can help with two other things with the use of JWT token: when a web request presents a JWT token, it can validate whether it is authentic. Then, it can use the claims in JWT token to drive authorization decision on whether the specific request is allowed or denied. Both will use Istio CRDs.

Peer Authentication

Istio can enforce mTLS for TCP traffic between Pods. According to its documentation, enforcing mTLS at mesh level is as simple as applying a Peer Authentication resource to the root-level namespace:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
 name: default
 namespace: istio-system
spec:
 mtls:
 mode: STRICT

The role of mTLS is so Pods can validates each other’s identity and then encrypt the TLS traffic in between. Each workload must first have an identity and Envoy proxy addressed this issue by adopting SPIFFE framework. It gives each workload an identity in the format of <TRUST_DOMAIN>/ns/<NAMESPACE>/sa/<SERVICE_ACCOUNT>. For example: spiffe://cluster.local/ns/myapp-dev/sa/default. It is also important to understand that only Pods with injected Envoy sidecar have SPIFFE workload identity and therefore is able to speak in mTLS. For new services, this is usually not an issue. For migrating workload without sidecar, a Pod without sidecar may connect with one in the mesh (with sidecar) if the mtls mode is PERMISSIVE in Peer Authentication. Otherwise, the connect is reset at layer 4 with the following error:

curl: (56) Recv failure: Connection reset by peer
command terminated with exit code 56

Therefore, it is advisable to start with PERMISSIVE mode for a precautionary migration of workload to mTLS. With mTLS all effective at the mesh level, there is no need to natively configure TLS between services.

The SPIFFE identity used in PeerAuthentication can also be used in Request Authorization as rule conditions. I will discuss request authentication before request authorization. To understand request authentication, let’s first warm up on JWT.

JSON Web Token

JSON Web Token (JWT, RFC 7519) is a format to carry JSON payload with optional signature and/or encryption. It can be thought of as a document (in JSON format) with signature for web servers to exchange information. The signature portion makes it friendly for document consumers to validate the authenticity. It is also URL-safe, and thereby adopted in web-browser SSO context, to pass identity of an authenticated user between and identity provider and a service provider.

JWT enables token-based authentication, a significant improvement from traditional session-based authentication. The traditional session-based authentication can be illustrated as below:

session-based authentication

This authentication model has major drawbacks. First, a mechanism to validate the authenticity of Cookie is missing. Second, the server has to keep the session information, making itself not stateless, unless a state store such as memcached is introduced.

token-based authentication

In token-based authentication such as using JWT, a token is issued. The authenticity of the token are validated before the server provides data, and it can be validated by any backend server.

The payload of JWT consists of claims, which are statements about an identity (such as name, role, email). There are custom claims as well as standard reserved claims, such as iss (issuer), sub (subject), aud (audience), iat (issued at time), exp (expiration time), and jti (JWT ID). When a program produces a JWT, it turns the raw payload into standardize payload by adding the required reserved claims and may sort the claims alphabetically.

The JWT consists of three parts with a period as delimiter:

The third part is a signature in the format of JWS (JSON Web Signature, RFC 7515) for the JWT consumer to validate its authenticity. The first and second parts, as you can tell, are the claims in the document. Their base64 encoding can be decoded with no effort and should therefore be considered exposed. Although JWT addresses the authenticity of information, it does not intend to address the confidentiality of the payload at HTTP layer. The payload should not carry sensitive information and should always be used with secure HTTPS port. To tackle this issue, there is JWE (JSON Web Encryption, RFC 7516) which is an implementation similar to JWT which also encrypts the payload.

Some IAM protocols are built on top of JWT. For example, the OpenID Connect specification also defines a set of standard claims that it uses while still allow custom claims.

Request Authentication

Istio can perform request authentication using its CRD. It is important to distinguish request authentication and user authentication. In user authentication, the identify provider typically looks up an identity store and compares password hash results to check whether the identity of the visiting user is authentic or not. This is outside of Istio’s capability but many off-the-shelf solution excels at it, such as Azure AD. Once the user’s identity is validated by identity provider, and a JWT is issued for downstream service providers to consume. Istio’s CRD can front the service provider and validate that the presented JWT is authentic. It authenticates the identity of a request (as truly issued by the trusted issuer without being tampered). This process does not involve checking user’s identity, even though user’s identity could be stored in the payload by the JWT issuer.

Istio uses the RequestAuthentication CRD to perform this function. The JWT issuer signs with its private key and stores the signature in the JWT. When it is presented to Istio, Istio’s RequestAuthentication CRD needs the public key of the issuer in order to validate the JWT. The public key usually comes in as a JWK (JSON Web Key, RFC7517), a format convertible to and from PEM format. The JWK can be provided either inline in the RequestAuthentication’s YAML manifest, or via a URI.

Below is an example of a basic RequestAuthentication declaration:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
 name: httpbin
 namespace: foo
spec:
 selector:
 matchLabels:
 app: httpbin
 jwtRules:
 - issuer: "issuer-foo"
 jwksUri: https://example.com/.well-known/jwks.json

In this example (from the documentation), the jwtRule requires that the issuer be issuer-foo, and the JWK (containing public key) is provided by a given URI address. Istio will pass the authentication once the signature in the presented JWT is verified with the JWK.

Authorization Policy

Istio’s Authorization Policy by itself can operate at both TCP or HTTP layers and is enforced at the envoy proxy. The result is an ALLOW or DENY decision, based on a set of conditions at both levels. If the traffic is HTTP then you should consider use some HTTP level information as it provides a lot more flexibility. Even when operating at HTTP layer, AuthorizationPolicy does not have to work in conjunction with RequestAuthentication. The rules can use path, methods, etc to drive an authorization decision, for example:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: authz-policy-orthanc
spec:
 selector:
 matchLabels:
 app: orth
 action: ALLOW
 rules:
 - from:
 - source:
 namespaces: ["istio-system","orthweb"]
 to:
 - operation:
 methods: ["GET","POST","PUT","HEAD","DELETE"]
 ports: ["8042"]
 - from:
 - source:
 namespaces: ["istio-system","orthweb"]
 to:
 - operation:
 ports: ["4242"]

The claims in the JWT payload can also be used to drive authorization decision, as exemplified in the Istio documentation, by using a when keyword in a rule and specifying the claim as a key:

 when:
 - key: request.auth.claims[iss]
 values: ["https://accounts.google.com"]

The when clause requires that the iss claim in the JWT must carry a specific value in order to ALLOW the HTTP request. While the claims in JWT is just an additional factor to drive authorization decision, using authenticated information to drive authorization decision makes the overall workflow more secure, and should therefore be used when applicable.

When using AuthorizationPolicy CRD, keep in mind:

Use correct selectors so it only applies to workloads labelled as such. Without selector the policy takes effect in the entire namespace which is a recipe for issues
When multiple policies (each with multiple rules) are applied to the same workload, be aware of the policy precedence. Troubleshooting may get tricky.
For an AuthorizationPolicy to use HTTP fields in rules, it first needs to identify the traffic as HTTP. To tell Authorization Policy to treat the traffic as HTTP, we need to understand Protocol Selection. In a nutshell, we can name the port as http or http-* in the Service manifest of the workload. Otherwise all HTTP-based rules will be missed.

For troubleshooting, we can check authorization policies effective on a Pod with:

$ istioctl x authz check orthanc-6c9679d8c7-2ttlf -n dev-orthweb

This returns the effective policies but does not necessarily indicate which rule is matched when a request is denied or allowed. To find out further information, you will need to follow Istio FAQ to set RBAC logging to debug, and then monitor the log in the istio-proxy sidecar.

Apart from HTTP fields, path, authenticated claims in JWT, Istio Authorization can also integrate with an Open Policy Agent (OPA) to drive actions, in advanced use cases.

Bottom line

While Istio itself does not perform user authentication, its support of JWT in RequestAuthentication allows a workload to integrate with external identity provider. This capability, along with creative use of claims in JWT, also empowers authorization capability.

FluxCD: Continuous Deployment with GitOps

Sat, 15 Jan 2022 18:49:00 -0400

This post explains why I land on FluxCD GitOps for my project. Let’s star

Background

In the Korthweb project, I landed on Istio for the Ingress Gateway technology. I first attempted to expand the orthanc Helm Chart to bring Istio as dependency (sub-chart). One of the external chart for Istio gateway needs to be referenced multiple times (for ingress and egress). However, it cannot even be used as dependency (sub-chart) because of this issue. Istio didn’t re-introduce Helm Chart as a supported deployment until September 2021. So I’m not too confident about it.

This leads to a second thought of the “Big Helm Chart” approach to deploy all tiers in Korthweb. Helm is based on templating, and having two layers of charts brings complexity. I do need to bring a number of Helm charts together, but not necessarily by another Helm Chart.

The Kubernetes documentation also mentions Kustomize as an alternative to Helm. Below is a quick exploration of it.

Kustomize

A challenge with managing declarative object is to organize numerous manifest files to maintain consistency and readability. Kustomize is a standalone tool to customize Kubernetes objects through a Kustomization file. This page provides a good list of fields that can be used:

vars and replacements: copy fields from one source into any number of specified targets.
namePrefix, namespace, nameSuffix: customize namespace and names.
configMapGenerator, secretGenerator, and generatorOptions: create configuration entries from literal, files, or environment variables.
resources: indicates another kustomization directory (e.g. as base)
patches (also called overlays) add or override fields on resources.
patchesStrategicMerge: modifies values in known (loaded) resources
images: modifies the name, tags and/or digest for images, without creating patches.
openapi: use Kubernetes OpenAPI data to get merge key and patch strategy information about resource types.

The base -> overlay pattern ensures readability as well as portability. Typical use pattern is create one base kustomization, and several overlay kustomizations each representing an environment, such as dev, qa and production. Here is a good tutorial.

Helm and Kustomize are two approaches to deploy Kubernetes workload. Both tackle the challenge of managing many YAML declarations. Helm is template driven, and is commonly used by application developers as a means of packaging application releases while orchestrating the dependencies. Kustomize follows a base-overlay pattern, and is more commonly used by cluster operators for re-using manifests across multiple environments (e.g. dev, staging and prod).

In my use case, my deployment needs both. I need third party Helm chart to configure components such as PostgreSQL for HA. I also need Kustomize for the orthanc workload. With the need for both, there should be a higher level deployment technology that ingrate with both mechanisms.

GitOps

GitOps is originally brought up by Weaveworks in 2017. It is a methodology to deploy workload continuously, using a Git repository as source of truth. The point of GitOps is not about tooling, or specific platform, but rather to ensure the workload state matches the declaration in repository. For example, you can implement GitOps with Ansible for VM environment.

When it comes to managing Kubernetes workload, a GitOps tool must handle the challenge with managing many YAML declarations. Therefore most GitOps tools seek to support many well-adopted mechanisms such as Helm and Kustomize as discussed above, instead of simply taking an enormous amount of raw YAML declarations. The amount of deployment mechanisms supported, is a key indicator of how powerful a GitOps tool is.

The most notable tools are ArgoCD and FluxCD. Both are currently CNCF incubating projects. ArgoCD is powerful with many tools supported, such as Kustomize, Helm, Ksonnet, Jsonnet, etc. It also contains a user interface and aims to manage an entire deployment workflow. On the other hand, FluxCD has controllers mainly for Kustomize and Helm, with a jsonnet extension from third party. This post draws a comparison of them (along with JenkinsX) based on earlier versions (from mid 2020). There is even a standard (OpenGitOps) in CNCF landscape but still at Sandbox level.

For Korthweb project, I chose FluxCD. It is simple, and provides just enough types of controller for what I do.

FluxCD

The diagram below illustrates the components:

The configurations starts with a bootstrapping process, which creates directory in Git repository (if not exist), and installs flux-system components in the target Kubernetes cluster. The sync process starts as soon as bootstrapping is completed. The process in charge of syncing declarations to target cluster, confusingly, is also called Kustomization. Therefore there are two Kustomizations. According to the FAQ on FluxCD website:

There are two Kustomization types. the kustomization.kustomize.toolkit.fluxcd.io is a Kubernetes custom resource while kustomization.kustomize.config.k8s.io is the type used to configure a Kustomize overlay. The kustomization.kustomize.toolkit.fluxcd.io object refers to a kustomization.yaml file path inside a Git repository or Bucket source.

Inside of the Git repository, with a kustomization.kustomize.toolkit.fluxcd.io obejct, the flux-system points to Kustomization file (representing kustomization.kustomize.config.k8s.io object) at root level. The kustomization.yaml file organizes resources in the same directory. A kustomize directory may also reference other kustomize directory, forming a hierarchy.

Implementation

FluxCD has a command “flux check –pre” to check the prerequisite, such as kubectl. The code is stored in the GitOps directory of Korthweb repository.

To configure deployment, we need to first create a personal access token. For GitHub, here is the instruction. Export the token to environment variable, and launch bootstrapping:

$ export GITHUB_TOKEN=xxx_yyy55555XXXodr7ABBBB234CCccw
$ flux bootstrap github \
 --owner=digihunch \
 --repository=korthweb \
 --branch=main \
 --personal \
 --path=gitops/environment/dev

A deploy key is configured during the bootstrapping process. As soon as bootstrapping is completed, the sync (aka kustomization, or reconciliation) has started, which can be monitored using:

flux get kustomizations --watch

Running this command without –watch switch returns the overview of all kustomizations. Once reconciliation is completed, it should display something like:

NAME 	READY	MESSAGE 	REVISION 	SUSPENDED
application 	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False
dependency 	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False
flux-system 	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False
infrastructure	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False

FluxCD introduced a number of CRDS. For example, to check configured source repository, check gitrepositories CRD:

kubectl get gitrepositories -n flux-system

To check the detail of one kustomization (e.g. infrastructure), check kusomization CRD in flux-system namespace:

kubectl -n flux-system get kustomizations flux-system -o yaml | less

Other commonly used custom resources include:

helmcharts
helmreleases
helmrepositories

A Helm release can be created imperatively, for example:

$ flux create source helm istio \
 --interval=1h \
 --url=https://istio-release.storage.googleapis.com/charts
$ flux create helmrelease istio-base \
 --interval=1h \
 --release-name=istio-base \
 --target-namespace=istio-system \
 --create-target-namespace=true \
 --source=HelmRepository/istio \
 --chart=base \
 --chart-version="1.12.0" 


$ flux create helmrelease istiod \
 --interval=1h \
 --release-name=istiod \
 --target-namespace=istio-system \
 --source=HelmRepository/istio \
 --chart=istiod \
 --chart-version="1.12.0" \
 --values=istiod-values.yaml

However, in a GitOps approach, they should be stored as code (use –export to export declaration). For example, the infrastructure kustomization keeps HelmReleases for installing Istio and PostgreSQL. This kustomization is referenced by a Flux Kustomization from higher level.

You can also manually reconcile one of the kustomizations (or other CRDs) with flux reconcile command, for example:

flux reconcile kustomization dependency

Since Nov 2021, FluxCD (0.20) supports reconciliation based on server-side apply. This increases performance and help address issues such as applying large config map, which is the equivalent of adding –server-side flag to the kubectl apply command.

Limitation

Troubleshooting the FluxCD repo can be involving and counter-intuitive. I had to commit a lot of changes to the repo because it serves as source of truth. Even though the commits can be made to a branch, it still involves a lot of code pushes. Traditionally I commit a change after testing. In the GitOps workflow, I commit a change then to test.

More flexible troubleshooting options are still to be desired. For example, there is no way to run one (FluxCD’s) Kustomization object at a time (and disable the rest), unless you remove their YAML files from the repo.

The next limitation is the ordering of resources in Kustomization. Arguably this is a limitation from Kustomize, instead of FluxCD’s. For example, I need the following manifest to be executed:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: selfsigned-issuer
spec:
 selfSigned: {}

The resource is a CRD that needs to be first installed using Helm chart (cert-manager). However there’s no way to control the sequence (Helm release executed first before declaration using CRD). Although there’s some workaround, it is not convenient. Alternatively, we can separate the resource creation and CRD creation into separate kustomization objects with dependency relationship, as suggested in this example.

Summary

Deployment in Kubernetes can get complicated with a lot of manifests. Helm, Kustomize and the like are means to handle the complexity due to numerous manifests. GitOps tools such as FlexCD brings these tools under a single framework, and more importantly, implements the idea of using Git repository as source of truth for continuous deployment.

Kubernetes Admission Control

Fri, 07 Jan 2022 22:21:00 -0400

This post discusses admission control, and its implementation – the OPA Gatekeeper. I also discuss Azure Policy as a different Gatekeeper implementation.

Admission Webhooks

Admission controller intercepts requests to the Kubernetes API server after the request has been authenticated and authorized, and prior to persistence of the object into etcd store. There are many compiled-in controllers, which can be turned on and off on the node with the arguments of kube-apiserver process. For example, the ImagePolicyWebhook can be enabled with value ImagePolicyWebhook added to the –enable-admission-plugins switch. Its configuration can be provided via the –admission-control-config-file switch.

In addition to the compiled-in admission plugins (which must be configured for kube-apiserver process on the node), admission plugins can be developed as extensions and run as webhooks configured at runtime. This allows users to configure webhooks via API access, dynamically without having to restart kube-apiserver process on the Node, which is usually hard to do with managed Kubernetes platforms. They are therefore called Dynamic Admission Control.

You can define two types of admission webhooks in dynamic admission control: validating admission webhook, and mutating admission webhook. Their interaction with API server can be illustrated in the diagram below:

The mutating admission hook takes action to change the API request, whereas the validating admission hook accepts or denies the request.

A good example of mutating webhook is Istio’s sidecar injector. We can view the configuration with this command:

$ kubectl get MutatingWebhookConfiguration istio-sidecar-injector -o yaml | less

From the manifest returned, we can see that in this configuration the request is forwarded to istiod service on port 443, at path /inject for processing. We can also see some matching rules to find the target Pod creation API request.

Validating webhook can be display with the following call:

$ kubectl get ValidatingWebhookConfiguration

The output of validating webhook is a yes or no decision. We usually use validating webhook in conjunction with a policy engine to decide whether the request should be accepted or denied.

Open Policy Agent

Open Policy Agent (OPA) is an open-source general-purpose policy engine that applies policies written in Rego language to ingested JSON document and returns a result. It is usually integrated with system which requires a policy engine. For example, Kyverno is a policy engine designed for Kubernetes. Styra (one of the OPA contributors) develops policy engines to integrate with Istio’s authorization policy. They have online courses on OPA policy authoring and microservice authorization with their product.

OPA is build to be a general-purpose, unified way of solving policy and authorization problem. With microservice authorization, the activities includes decision making (determine action based on input, aka Policy Decision Point, PDP), and decision enforcement (issue 400 code or 200 code depending on decision, aka Policy Enforcement Point, PEP). OPA is introduced to decouple these two activities. OPA’s input is a JSON payload and it uses Policy in Rego language to come to decision.

The team that developers Open Policy Agent also created their controller (with OPA as the core component) to run validating web hook and mutating web hook. The original version is OPA-Kubernetes that uses kube-mgmt. This original version is also dubbed Gatekeeper v1.0. When OPA starts, the kube-mgmt sidecar container will load Kubernetes Namespace and Ingress objects into OPA. You can configure the sidecar to load any kind of Kubernetes object into OPA. The sidecar establishes watches on the Kubernetes API server so that OPA has access to an eventually consistent cache of Kubernetes objects. It has gone through a couple of major version changes as summarized in this section. As of today, when we deploy Gatekeeper we should use version 3.

Gatekeeper v3

Currently, Gatekeeper v3 is the most popular choice for Kubernetes Policy Controller. The diagram bellow illustrate how Gatekeeper integrates with Kubernetes API server.

Gatekeeper and Kubernetes

We can follow this guide to install Gatekeeper but the key step is as simple as to apply the correct version of manifest. Alternatively it can be installed using Helm. After the installation, we should see a Service named gatekeeper-webhook-service in the gatekeeper-system namespace. We can also inspect the newly created validationg web hook configuration

k get validatingwebhookconfiguration gatekeeper-validating-webhook-configuration -o yaml | less

The result indicates that the configuration forwards incoming manifests to the gatekeeper-webhook-service web service at the path /v1/admin for validation, and then at /v1/admitlabel for labelling. The configuration also stores rules as matching criteria.

We can smoke test Gatekeeper v3, with the basic example in its directory. Apply the template, constraint and then the manifests in resources. The pod creation will fail with an error like:

Error from server ([pod-must-have-gk] you must provide labels: {"gatekeeper"}): error when creating "resources/bad_pod_namespaceselector.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [pod-must-have-gk] you must provide labels: {"gatekeeper"}

The gatekeeper document also covers the details of using ConstraintTemplate and Constraints. However, Writing your own a policy in Rego still takes time and we want to piggyback on the community for commonly used policies. OPA‘s gatekeeper-library projects keeps a handful of those in its library directory. We can test the privileged container example:

$ cd gatekeeper-library/library/pod-security-policy/privileged-containers
$ kustomize build . | kubectl apply -f -
constrainttemplate.templates.gatekeeper.sh/k8spspprivilegedcontainer created
$ kubectl apply -f samples/psp-privileged-container/example_disallowed.yaml
pod/nginx-privileged-disallowed created
$ kubectl delete -f samples/psp-privileged-container/example_disallowed.yaml
pod "nginx-privileged-disallowed" deleted
$ kubectl apply -f samples/psp-privileged-container/constraint.yaml
k8spspprivilegedcontainer.constraints.gatekeeper.sh/psp-privileged-container created
$ kubectl apply -f samples/psp-privileged-container/example_disallowed.yaml
Error from server ([psp-privileged-container] Privileged container is not allowed: nginx, securityContext: {"privileged": true}): error when creating "samples/psp-privileged-container/example_disallowed.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [psp-privileged-container] Privileged container is not allowed: nginx, securityContext: {"privileged": true}

Currently the library directory contains two sub-directories, general and pod-scurity-policy. The latter is to regulate Pod creation, while the former includes more common usecases such as disable node port, enforce https, and enforce probes. This is the place I start with when building a policy.

The policy constraints take effect cluster wide. When we have multiple clusters, we would like a unified place to manage policies.

Azure Policy with AKS

We take Azure Policy with AKS as an example to illustrate how public cloud platform can simplify policy management. When building AKS cluster, an addon profile for Azure Policy can be installed. This allows Azure Policy to connect to the AKS cluster. Azure Policy contains many built-in policies definitions (as well as initiative definitions which are groups of related policies). We can simply search by Kubernetes keyword and look for the built-in policies. For example, there is a built-in policy definition “Kubernetes clusters should not allow container privilege escalation. The definitions (policy or initiative) can be assigned to a resource group with enforcement action set to denied, and with excluded namespaces, as shown in the screenshot below

The assignment can take as long as 10 minutes to push down to the cluster. Then we should be able to confirm by checking the constraint CRDs. We can see this This setup brings a centralized policy management system that can be easily hooked up to multiple clusters.

Other benefits of this architecture includes the ability to report compliance. As per CIS report for Azure AKS recommendation 4.3:

Azure Policy extends Gatekeeper v3, an admission controller webhook for Open Policy Agent (OPA), to apply at-scale enforcements and safeguards on your clusters in a centralized, consistent manner. It covers many basic resource types but does not cover any well-known CRDs. Azure Policy makes it possible to manage and report on the compliance state of your Kubernetes clusters from one place.

Checks with Azure Policy service for policy assignments to the cluster.
Deploys policy definitions into the cluster as constraint template and constraint custom resources.
Reports auditing and compliance details back to Azure Policy service.

As of February 2022, AWS EKS doesn’t seem to have the equivalent of this capability to integrate with a policy management. The only option would be to install Gatekeeper v3 yourself on the cluster, or host it separately.

Bottom line

Admission control should be a standard setup in Kubernetes deployment. When building gatekeeper system on your own, it can be set up separately on a different cluster. When Kubernetes is provided as a platform, it is very helpful for platform operator to manage their tenants. If the tenant is application development team, it also makes sense for them to develop their own policies for the developers in their team.

From Ingress to CRD: why my solution needs Istio Gateways on Kubernetes platforms

Wed, 29 Dec 2021 22:50:16 -0400

Update: also read my other article here on the different generations of ingress technologies.

In my Korthweb project I was researching for the best ingress mechanism for HTTP and TCP workload, both of which need to be secured. I started with Kubernetes Ingress but eventually decided to go with Istio Gateway. This blog post is about the justification. In this essay, I will make the distinction between Ingress and Gateway and explain why a Kubernetes platform needs the latter going forward.

The word ingress can be used either to indicate ingress resource (in conjunction with ingress controller) in the context of Kubernetes cluster, or more generally to indicate the technology (e.g. provided by service mesh) to direct north-south traffic, originated from the outside into the workload running within the cluster. I will use both meanings of the word ingress throughout the article.

Background

In my previous post, I discussed Kubernetes Service object and native Ingress. In short, a Service object addresses the problem of exposing a workload (Pod or Deployment), operating at layer 3-4. A ClusterIP type of service exposes workload within the cluster only, and therefore is only used by internal services that are consumed by workload in the same cluster. A NodePort type of service exposes workload to outside of the cluster, using the node’s IP address and port range available on the node, a great step forward but still quite inconvenient and subject to the limitation of using Node’s IP and Port. A LoadBalancer type of Service brings separate IP address and port for running a service. To back up this type of service, the platform has to provide a load balancer with its own IP address and port range to manage. The implementation is platform specific e.g. Metal LB for Minikube, Azure Load Balancer for AKS, NLB for AWS EKS. In this type of service, Node Ports still exists but are not exposed to the outside world. Instead, they are only exposed to the backend of Load Balancer, whose front-end port is exposed to the outside world. In both NodePort and LoadBalancer types of Service, the kube-proxy process plays a role on each node to direct traffic into NodePort to target Pods.

Kubernetes Ingress, on the other hand, targets issues above layer 4, for example, path-based routing and is therefore mostly used for HTTP and HTTPS traffic. It is implemented by an Ingress Controller and now there has been a diverse ecosystem of those ingress controllers. In many implementations (e.g. Azure Application Gateway Ingress Controller, AGIC), the ingress themselves usually come with a load balancer managed by themselves. This eliminates the need for a separate LoadBalancer type of Service just for L4 capability. As a result, Ingress (with both L4 and L7 capabilities) is usually deployed along with ClusterIP type Service. Here is an example. The declaration of the Ingress in this case usually contains numerous lines of annotations in order to communicate the specification to the implementation.

Requirement

Using Kubernetes Ingress along with Service of ClusterIP type seems perfect to address the majority of use cases. However, it has its blind spots. My Korthweb project deals with DICOM traffic (a protocol on top of TCP) over TLS as well as HTTP traffic, which can be broken down as:

The ability to proxy TCP traffic over an arbitrary TCP port
The ability to terminate TLS/SSL encryption for TCP traffic

The Ingress documentation of Kubernetes clearly states that: an Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

The stance of community-driven Nginx ingress controller can be found in the documentation, which suggests the use of Service object for arbitrary TCP port. There is some unofficial claims of workaround available but I’m not confident.

I also went through a number of other Ingress providers but to my disappointment, the only product that supports it seems to be Traefik:

Product	Requirement #1	Requirement #2
F5 driven Nginx Ingress Controller	Supported	There have also been requests for TLS termination for TCP traffic but the request has not been closed as of yet.
HA Proxy Ingress controller	Supported	There’s no mention of this in documentation.
Kong’s Kubernetes Controller (with TCPIngress CRD)	Supported	Documentation claims support through SNI-based routing. However, the example demonstrates it using self-signed certificate only. BYO cert not supported according to this GitHub issue.
Traefik Lab’s IngressRoute (with IngressRouteTCP CRD)	Supported	Documentation seems to suggest that it is supported by IngressRoute.

My general impression is that Req #2 isn’t very popular so the providers either don’t support it or delaying the implementation. Even if they do, the CRD used for TCP ingress is different than HTTP’s. For example with Traefik Lab, the CRD for TCP is IngressRouteTCP, and for HTTP it is IngressRoute.

Introduction to Gateway

In addition to functional shortages, according to this blog post, there are signs of fragmentation into different but strikingly similar CRDs and overloaded annotations. In Kubecon 2019, a group of contributors discussed the evolution of Ingress into Gateway. Below are the two key slides stolen from their presentation:

This sums up how the concept of Gateway is different from Ingress. Gateway is an instantiation of a given LB. It works along with Route to achieve the functions brought by an Ingress. Gateway as a resource type of its own makes management easier at L4-L6 by a separate team. Routing at L7 is offloaded to “Route” resource type.

This conceptual evolution gives rise to the Gateway API open source project managed by the SIG-NETWORK community. Gateway API is a collection of resources that model service networking in Kubernetes, including GatewayClass, Gateway, HTTPRoute, TCPRoute, Service etc. The aim of this initiative is to evolve Kubernetes service networking through expressive, extensive and role-oriented interfaces that are implemented by many vendors and have broad industry support.

The diagram above stolen from Gateway API website illustrate the management model for each resource.

Kubernetes Gateway Implementations

The Gateway API is still a fairly young initiative and all the reference implementations are either work-in-progress or in early stages. In the last section, we learned that in the journey from Ingress to Gateway, the standardization initiative comes a bit behind the implementation efforts. For many supporters, the natural strategy is to continue with existing proprietary ingress controller implementation, and retrofit their technology to the emerging Gateway API along the way.

Because of that, we can see many market players with multiple flavours of implementations, typically one that evolves from their original product offering, with higher adoption rate and maturity level, and one that conforms to Gateway API specification.

For example, Traefik Labs has its Gateway API implementation in experimental stage. Its own implementation consists of (standard) Kubernetes Ingress and Kubernetes IngressRoute (based on custom resource with support of advanced features such as TCP route). Another example is the HAProxy Ingress, a community driven ingress controller implementation for HAProxy. Starting from its version 0.13, it partially supports the Gateway API’s v1alpha1 specification. Here is the conformance statement. Contour as a CNCF project for ingress controller has support for Gateway API at alpha version. Kong’s Kubernetes Ingress Controller follows the same path.

The Gateway functionality is also provided as part of Service Mesh product offering, and we can see some reference implementation by service mesh providers. Istio has its own Gateway implementation but tries to adapt to Kubernetes Gateway API. Hashicorp Consul also claims to be building support for Kubernetes Gateway API.

Until the Gateway API project matures, it is not recommended to use Gateway implementations that conforms to it, unless you intend to be their Guinea Pig. For my project, I chose Istio Gateway.

Istio Gateway implementation

As discussed, when the Gateway standard is still in its infancy, I chose a non-standard implementation of Gateway, even though it may include some CRDs. Out of the many Gateway implementations, I choose Istio mainly for two reasons.

First, I need a Gateway as part of service mesh because service mesh provides many other features that are needed in the platform. One of the goals of Service Mesh is to provide commonly used features (traffic management, observability, security, extensibility) in a commodity layer on top of Kubernetes. Gateway is not the only thing I need out of this commodity layer. With one install of Istio, many common platform-level problems are also addressed (e.g. mTLS, traceability, etc). I’ve compared three major service mesh technologies as below:

Linkderd	Istio	Consul
– lightweight – uses linkerd2-proxy – the original service mesh – no Gateway implementation (deal breaker in my use case)	– feature rich – uses Envoy proxy – complex but getting better	– initially a service discovery and distributed key-value store – lacks observability features but getting better – uses Envoy proxy

It is worth-noting that there are some initiatives to standardize service mesh (e.g. SMI) with reference implementation such as OpenServiceMesh. The standard is too weak to be considered important for now compared with maturity and stability. Istio has strong community support. The risk with Istio to acknowledge, is that it is not following an open-governance model (unlike many other CNCF projects), which could potentially causes vendor lock-in.

Second, The Istio Gateway features the separation between Gateway and Virtual Service CRDs. The former defines entry point and the latter defines routing rules. This design separates entry points from routing rules, enabling the flexibility of reusing the same Virtual Service for different gateways.

Istio uses Envoy proxy to manage traffic between Pods. Unlike the kube-proxy pattern, the Envoy proxies are side-car containers centrally managed by Istio Control Plane, which also enables other features such as traceability, as illustrated in the diagram below (stolen from this post).

In addition to Virtual Service, Istio Gateway also has the concept of Destination Rules. Virtual Service defines how to route traffic to different destinations. Destination rule defines how to split traffic to different subset at the routing destination. The concepts are documented on this page. For HTTP traffic, the relevant entities can be represented in the diagram below:

Here is an example of using Gateway and Virtual Service resources (from Korthweb sample project). Also note that in the Istio literature, there is neither a CRD name called “Ingress”, nor a resource type named “Ingress Gateway”, although they may be loosely used to refer to “Gateway resources configured to manage ingress traffic”.

Istio Gateway Installation

There are more than one ways to install Istio. In the past, Istio Operator was used to install Istio. The Operator calls IstioOperator API. Today the use of Istio Operator is not recommended anymore but the IstioOperator API is used implicitly by Istioctl installer. The two recommended approaches to install Istio today is by Istioctl and Helm.

With istioctl, you can specify an option imperatively, or using an overlay file as did in this lab. With Helm, istio has multiple charts, and requires multiple steps:

install CRDs using the base chart
install istiod using the istiod chart. Here is an example of values provided to Helm installer.
install ingress gateway using the gateway chart. Here is an example of values provided to Helm installer for ingress gateway. Note that egress gateway also requires the same helm chart with different value definition.

This instruction contains how to manually install Istio gateways using Helm, including installation of multiple Helm charts. I attempted to create a single chart to consolidate the multiple charts required for istio. It was not successful because of an error when trying to reference the same gateway chart dependency for multiple times.

Logging and Monitoring in Kubernetes with PLG stack

Wed, 13 Oct 2021 21:29:00 -0400

We’ve checked out the the actors in PLG stack (Promtail, Loki, Node Exporter, Prometheus, Grafana) and whipped up a quick pipeline on MacOS. Now I’m going a little further to implement the same PLG stack (Prometheus Loki and Grafana) in a Kubernetes cluster. This setup is for demo only, therefore no persistent storage is enabled.

Test Workload

I host a deployment of Flog with three pods running on Minikube. Flog is an open-source emulating log generation behaviour of an application. On the Minikube cluster we start the deployment as below:

minikube start --driver=hyperkit --container-runtime=containerd --memory=12288 --cpus=2
kubectl create ns obsv
kubectl -n obsv create deployment flog --image=mingrammer/flog --replicas=3 -- flog -f rfc3164 -l -d 300ms
kubectl -n obsv get po

The Pods will come up in a heartbeat.

I will use helm to install the objects required for logging and metrics pipelines. There are multiple Helm charts for each components. Some high-level charts (usually with a name suffix of -stack) contain several other resource as sub-charts. They are created as one-stop-shop for multiple components but I found none of them serve my exact purpose. For example, both loki-stack and kube-prometheus-stack include Grafana. But I only need one instance of Grafana. Therefore I stick to the low-level charts.

Log Shipping

Add a helm repo, and install loki and promtail. Note that we need to specify correct loki address when installing Promtail.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --namespace obsv --install loki grafana/loki
helm upgrade --namespace obsv --install promtail grafana/promtail --set "config.lokiAddress=http://loki.obsv.svc.cluster.local:3100/loki/api/v1/push"

When installing Promtail, a DaemonSet is created on the Node. The default configuration applies appropriate configuration and tagging strategy for Kubernetes Pod and Node. So the only customization I specified is Loki address. We can then check logging with Loki. To do so, first expose port 3100 to host, and then use logcli (e.g. on MacOS) to query for logs:

kubectl -n obsv port-forward service/loki 3100:3100
logcli labels
logcli labels pod
logcli query '{pod="flog-775d5fc5c8-p4rlx"}'

Log lines should be pumped to Loki a minute after Loki comes up. The logcli labels command should display the tags. The logcli query command should return the log lines.

Metrics

I use Premetheus with node exporter. In its architecture, Prometheus contain the server, the pushgateway, and alertmanager. The helm chart for Prometheus contains all of those components. It also has a dependency repo for kube-state-metrics. To install:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm --namespace=obsv install prometheus prometheus-community/prometheus
kubectl -n obsv get svc

Once installed, a DaemonSet for Prometheus Node Exporter is created. The exporter is already configured by default for Kubernetes monitoring. The Prometheus server is also configured, on port 80 by default. it needs to be forwarded in order to access from Browser:

kubectl --namespace obsv port-forward service/prometheus-server 9100:80

To verify installation of Prometheus, browse to localhost:9100 to examine the metrics.

Visualization

Last but not least, I will configure Grafana. The repo has been added already so we’ll just install the chart:

helm upgrade --namespace obsv --install grafana grafana/grafana
kubectl get secret --namespace obsv grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

The second command retrieves the credential. To access the web portal, we need port forwarding again:

kubectl port-forward --namespace obsv service/grafana 3000:80

To verify installation, browse to http://localhost:3000 and log in as user admin with the password above. Then add two data sources:

Type: Prometheus, URL: http://prometheus-server.obsv.svc.cluster.local:80
Type: Loki, URL: http://loki.obsv.svc.cluster.local:3100

Then we can explore data using both data sources.

PLG stack

Summary

In the last two posts I reviewed the setups for PLG stack in Kubernetes, from a regular environment to k8s cluster. Fluentd, Prometheus are both CNCF projects. The PLG stack seems to be more adopted than EFK but both have their own advantages. Welcome to the PLG vs EFK debate.