Kubernetes API on Digi Hunch

EKS impression

Fri, 23 Dec 2022 18:18:19 -0400

I’ve worked on a few AKS projects previously. Since I joined AWS I wanted to put aside some time to check out EKS (Elastic Kubernetes Service). Here in this post, I put down my first impression on EKS, and also share my Terraform template in cloudkube project to create an EKS cluster.

Similar to AKS, EKS exposes API endpoint and the control plane components are hidden from AWS users. When creating EKS cluster it does not create the underlying VPC and subnets. Therefore, you have create an existing VPC and at least two subnets ahead of time, and specify them during EKS creation. Bear in mind that there is a list of requirement for the VPC and subnets.

In the cluster, the CNI that EKS officially supports is Amazon VPC CNI plugin. It is available as an add-on. Similar to Azure CNI, each Pod gets its own IP address. In addition, EKS supports other compatible CNI plugins such as Calico, Cilium, Weave Net and Antrea.

Computing Nodes in EKS

There are three modes to address computing capacity: self-managed nodes, EKS managed node groups and AWS Fargate. The documentation has a comparison table.

With self-managed nodes, users create EC2 instances separately and then register them to the control plane. The instances must use the same IAM role and AMI. You can use Auto Scaling groups of Bottlerocket (AWS-sponsored purpose-built Linux distro for container host) nodes. The self-managed node option is mostly for AWS outpost customers who bring in their own computing capacity from data centre.

If you provision computing capacity from AWS, it makes sense to assign EKS managed node groups when creating EKS cluster. We can turn on Cluster Autoscaler, a Kubernetes construct to manage the auto scaling of node groups. Sometimes we want to have more than one node groups. For example, to build a multi-architecture cluster, we need one node group with amd64 nodes and the other with arm64 nodes (e.g. instances with Graviton processor). In general, arm-based CPU delivers better performance with less power consumption and the industry is slowly moving towards more arm-based CPU architecture.

Fargate is what I call managed computing service for EKS. With Fargate you do not need to tweak Cluster Autoscaler to self-manage computing capacity. The Fargate documentation has a long list of considerations. For example, Pods must match a Fargate profile (here‘s an example) at the time that they’re scheduled to run on Fargate. So we need to build Fargate profile and Pod labelling properly. Also, Fargate does not support DaemonSet. Another big consideration is that Fargate does not support non-VPC CNI. In my opinion these are pretty significant limitations. Many workloads (system-level or application-level) would need Daemonset (e.g. kube-proxy, some CNI or CSI drivers, Dynatrace monitoring).

The pro of Fargate is the serverless computing model. The construct of a Fargate profile isn’t complicated. You just specify subnets, namespace and labels. However, the downside is the long list of considerations. Some teams may consider these restrictions too much. The other overhead is the need to manage Fargate profile to ensure all Pods are scheduled somewhere.

To me, using Fargate alone impairs portability of workload. The good thing is that Fargate and Managed Node Group are not mutually exclusive on a cluster. In most cases, we can go partially serverless, and reap the benefits of both of them.

Node AutoScaling

For workloads that don’t have a matching Fargate profile, we have to figure out node autoscaling ourselves. I touched on Cluster Autoscaler in “Autoscaling on Kubernetes Platform“. CA works on AWS as well and is triggered upon a Pod coming to unschedulable status in Scheduler. There is some limitations though. For example, CA interacts with Autoscaling Group (instead of EC2 instances directly). When it determines it’s time to scale up, it bumps up the desired capacity by one at a time in the Autoscaling group. The configurations in Autoscaling group may also be at play and CA do not have direct control. For example, the “scaling cooldown“. The pool of nodes is homogenous as per the pre-configured launch template and CA has no control. If a Pod requires a different type of node (e.g. ARM64 CPU, spot instance, etc), then we’d first have to create a node group with the desired node type. Moreover, in the worst cases, one-at-a-time scale-up does not meet the increase of demand driven by Pod increases, causing nuances such as racing conditions.

Because the Cluster Autoscaler doesn’t really deal with the nodes themselves, this kind of integration is clunky and slow. Nearly half of Kubernetes customers on AWS report that configuring cluster auto scaling using the Kubernetes Cluster Autoscaler is challenging and restrictive, according to this blog post. As a result, AWS launched an open-source cluster autoscaler project, Karpenter. Karpenter first only supported EKS but now the support includes other CSPs. For EKS, Karpenter directly interact with different types of EC2 instances.

Karpenter makes node scaling work in a more cloud-native manner. In the presence of unschedulable Pods, Karpenter bypasses the Kubernetes scheduler and works directly with the Cloud provider, to launch the minimal compute resources needed to fit those Pods and immediately binds the Pods to the newly provisioned Nodes without waiting for scheduler. As Pods are removed or rescheduled to other nodes, Karpenter looks for opportunities to terminate under-utilized nodes. Karpender defines a CR called Provisioner to specify node provisioning configuration, such as instance size, zone, CPU architecture, etc. It is a manifest that describes a node group so the node scaler is aware of all the available node types. You can have multiple Provisioners for different needs, just like node groups. The Provisioner CR can also set TTL for empty Nodes, such that once a Node has no pods other than DaemonSet, Karpenter will terminate the Node on TTL expiry.

Karpenter’s idea is similar to the idea of AutoPilot cluster in GKE. The new EKS workshop has an section on how to set up CA and Karpenter in practice.

Identity Management for EKS

For IAM, we need to be concerned with three aspects. The management traffic to the cloud service, the management traffic for Kubernetes cluster and business traffic.

Traffic type	AWS	Azure
I. Cloud Service Endpoint (Management Traffic for Cloud Service)	AWS IAM identity	Azure RBAC
II. Kubernetes API (Management Traffic for K8s Cluster)	IAM mapping or OIDC	Azure RBAC (implementation of OIDC)
III. Business traffic	Up to Kubernetes Ingress	Up to Kubernetes Ingress

For business traffic (type III), it is all up to the Ingress. I’ve written another article on managing ingress traffic on Kubernetes platforms. We interact with cloud service endpoint (type II) with either AWS CLI or Terraform, to create any object, including resources needed for a cluster. This is generally how we work with cloud service, not specific to Kubernetes. Usually the IAM identity assumes another IAM role, which empowers it with a lot of permissions.

For access to Kubernetes API (type III), EKS supports OIDC and IAM mapping. AWS documentation refers to this as “Cluster Authentication“. There is one special scenario where your identity for type II access inherits your identity for type I access. As the document puts:

When you create an Amazon EKS cluster, the AWS Identity and Access Management (IAM) entity user or role, such as a federated user that creates the cluster, is automatically granted system:masters permissions in the cluster’s role-based access control (RBAC) configuration in the Amazon EKS control plane. This IAM entity doesn’t appear in any visible configuration, so make sure to keep track of which IAM entity originally created the cluster.

This special scenario (I call it the “implicit master user“) allows us to perform critical activities on the cluster, such as creating IAM mapping, or OIDC configuration.

The above addressed how AWS resource access Kubernetes resource. On the other hand, to address how a Kubernetes resource access AWS resources, we need IRSA (IAM Roles for Service Account). We have a service account in Kubernetes and map it to an IAM role.

AppMesh

AppMesh is AWS’ Envoy based service-mesh offering supporting Kubernetes cluster, ECS service and even EC2 instance. AppMesh’s control plane is a managed AWS service, with a controller running on the Kubernetes cluster. To install AppMesh on the cluster:

On the EKS cluster, install AppMesh Controller using Helm
Associate the cluster with IAM OIDC provider
Create an IAM role for the appmesh-controller service account

After these steps, you can create a mesh using CloudFormation, Terraform, etc. The data plane (Envoy proxy) can run on Kubernetes (as sidecar). Traffic between control plane and data plane can go through private link (Interface VPC endpoint) for added security. Like Istio, AppMesh enables mTLS. For observability, you can export Envoy metrics with Prometheus. Coupled with XRay, AppMesh also supports distributed tracing.

AppMesh uses a different set of CRDs than Istio. Key CRDs are:

Mesh: represents an entire service mesh. At mesh level you can configure Egress filter (to allow or deny external traffic) and set IP version (v4 vs v6)
VirtualGateway: a CRD that represents an Ingress in to the Mesh. A virtual gateway allows resources that are outside of your mesh to communicate to resources that are inside of your mesh. A virtual gateway references Envoy proxy deployment by podSelector. It references GatewayRoutes by namespaceSelector, and optionally gatewayRouteSelector. You also specify listeners in the manifest to reference Envoy proxy Service (LoadBalancer Type).
GatewayRoute: A gateway route is attached to a virtual gateway and routes traffic to an existing virtual service. If a route matches a request, it can distribute traffic to a target virtual service. In the manifest, you specify a list of httpRoute, each with matching condition and action. In the action section you can specify virtualService as target.
VirtualService: an abstraction of a real service provided by a virtual node directly or indirectly by means of a virtual router. Dependent services call your virtual service by its virtualServiceName, and those requests are routed to the VirtualNode or VirtualRouter that is specified as the provider for the VirtualService.
VirtualRouter: Virtual routers handle traffic for virtual services. In a virtual router manifest, you can define Route to direct incoming requests to virtual nodes as target.
VirtualNode: A virtual node acts as a logical pointer to a particular task group (i.e. ECS service, Kubernetes deployment). It represent a Service in the AppMesh. In the manifest, you reference Pods by podSelector, specify listeners for any inbound traffic that your virtual node expects, and specify serviceDiscovery for your task group.

You can configure those Custom Resources using YAML manifests (and check the API reference a lot). Alternatively, you can configure them from AWS CLI or AWS console. The console will help you visualize what can be configured. For further details on how these CRs play together, there is a workshop for AppMesh.

EKS cluster using Terraform

Officially, there is an EKS blueprint project for provisioning EKS cluster in Terraform.

I also keep my own Terraform code in the AWS directory of cloudkube project. It works out to be a little more complex than my Terraform template to create Azure Kubernetes Cluster (Azure directory). Because I had to create Cognito resources with initial credential to allow users to connect to cluster without using the implicit master account.

Below is the diagram of the processes.

Create EKS cluster with Terraform module

The template configures kubectl access on a Bastion host, which assumed the same role that our IAM user uses to create the Kubernetes cluster. Therefore, the IAM role is the master identity.

Note that the IAM user (power-user) has very powerful permissions. Usually it is ideal to assign lots of permission to IAM Roles (temporary credential) instead of IAM user (long-term credential). So the role chaining would look like:

The IAM user that Terraform uses has no permission other than assuming a “PowerUser” role
The PowerUser role trusts the IAM user. It also has the permission to assume the “EKS-Manager” role
The EKS-Manager role trusts PowerUser’s role session.

However, the role chaining scenario above is not currently supported in Terraform.

I use a Bastion host because the cluster endpoint is on private subnet. The bastion host is on a public subnet. However, if we do not like public subnet and public IP, we can place the bastion host on a private subnet, and use SSM system manager agent with SSH tunnel plugin to have SSH access to private bastion host.

Summary

I first came across this article about EKS and its awfulness and then decided to check out EKS. I’m not sure all points are still valid but it’s generally real-life experiences. There are also many peripheral services, such as AMP (AWS Managed Prometheus), AMG (AWS Managed Grafana), ADOT (AWS Distro for Open Telemetry), AppMesh (Another Envoy-based Service Mesh, easier to manage than Istio but less Powerful), with a lot to explore.

Kubernetes Operator

Thu, 07 Apr 2022 09:39:00 -0400

Kubernetes has a number of tools to automate the deployment of a single workload. In previous posts, we had covered Helm and Kustomize. What are left unresolved is how to maintain the status of workload after deployment is completed. In this post, I will give an introduction to Kubernetes Operator. Compared with Helm (templating approach) and Kustomize (patching approach), Kubernetes Operator follows the operator pattern. Operators are usually provided by the developer of the application.

Operator Pattern

In Kubernetes, we know that a controller takes care of routine tasks to ensure that desired state expressed by Kubernetes resource types matches the current state. One example is that the Deployment controller ensures the number of pods running matches the amount specified in the replica field. Controller is the key to ensure that resources can be managed by declarative manifests for Kubernetes resources.

Kubernetes makes use of controller pattern throughout its own design. One of its key component, Controller Manager, is a collection of many controllers. Each controller is in charge of a control loop, responsible for listening the object it manages. Another component, Kube-scheduler, is also a special type of Controller. The kube-scheduler monitors unscheduled Pod and health of nodes and determines the best Node to schedule the new Pod to. Then it writes the decision to etcd store for kubelet to execute.

This controller pattern is fairly successful in what it does and we can extend the use of it. Beyond the built-in resource types, we can create our own custom resource definitions (CRDs), and create controllers that watches for the manifest that declares custom resources (CRs). The controller ensures that the resource status matches their specifications. This is also known as reconciliation, which is implemented as a control loop. Operator pattern can be illustrated in the diagram below:

Operator Pattern

Technically, there is no difference between a controller and an operator. What makes an Operator (used to install workload) different than a native Kubernetes controller, are two things. First, an Operator usually needs CRDs because the built-in resource types are insufficient. Second, the operator reflects the domain knowledge to keep the target workload running. For example, stateful workloads such as database needs their operational steps executed in certain orders.

On Operator Pattern, CNCF published a whitepaper with a deeper review. This white paper is the best reference for a good understanding of the Operator Pattern.

Custom Resource Definition

The built-in controllers work with built-in objects (pre-defined APIs). Custom operators usually need their own APIs to function. To extend Kubernetes API, we define the schema of these APIs in the form of CRDs (custom resource definitions) using OpenAPIv3 standard. Then, we can declare Custom Resources (CRs) in compliance with the schema. The OpenAPIv3 schema in the CRD resource tells validating web hook (admission control) how to validate the schema when we send an CR in to API server.

When we work with third-party operators, they usually provide CRDs along with the operator implementation. For example, in my operator example project, we have a minimalist CRD WordPress with one property: sqlRootPassword and we can declare a CR as in this example. For a more realistic use case, we can take a look at Kiali CRD. In the next section, we’ll use it along with Kiali operator to install Kiali.

Operator Usage

Like Artifact Hub to Helm, OperatorHub is a public registry of most used Kubernetes Operators. In this section, we will take an example of using Operators. We will install Kiali as an add-on to Istio using Kiali CR and operator, which also depends on Prometheus to be installed using Prometheus Operator first. Note that the Kiali installation outlined in this section is not the the quick-start install manifests from Istio’s sample directory. For Kiali on production system we have to customize the installation.

Suppose we have installed Istio, we can then install Prometheus operator using Helm. The Prometheus operator will install Prometheus. Then we use Helm again to install Kiali operator. The Kiali operator will watch for creation of Kiali CRD, to deploy services:

$ helm install -f prometheus-values.yaml --namespace istio-system --repo https://prometheus-community.github.io/helm-charts --version 13.6.0 istio-prometheus prometheus --insecure-skip-tls-verify
$ helm install -f kiali-operator-values.yaml --namespace kiali-operator --repo https://kiali.org/helm-charts --version 1.45.0 kiali-op kiali-operator --create-namespace
$ kubectl apply -f kiali-cr.yaml

I include example content for each file in the commands above on Github gist (prometheus-values.yalm, kiali-operator-values.yaml and kiali-cr.yaml). For more options for installing Kiali, refer to their documentation.

I use this example to install Kiali and it includes two Operators, the Prometheus Operator and the Kiali Operator. The Prometheus Operator is one of the first ever written Kubernetes Operator. As soon as the operator is deployed, it starts to deploy the operator service. For the Kiali operator, we need to deploy Kiali CR after the Kiali Operator has been deployed. Both are valid patterns.

Operator Development

Operator is powerful. However, authoring an Operator is not a trivial effort. One usually start with a framework. A framework creates a body of boiler plate code that has the pattern implemented and allows developers to enrich the functions following the pattern. The white paper introduced three frameworks:

CNCF Operator Framework – aims at Operator Developers with an SDK, a scaffolding tool and a test harness. It currently supports three project types: Golang, Helm and Ansible. CNCF Operator framework consists of SDK and OLM.
Kopf (Kubernetes Operator Pythonic Framework) – an easy-to-use framework in Python that abstracts away most of the low-level Kubernetes API communications hassle.
kubebuilder – helps build a Manager similar to the native kube-controller-manager. For difference with OperatorSDK, read here.
Metacontroller: lightweight Kubernetes Controller as a Service

In CNCF Operator Framework, the Operator SDK supports development using Ansible, Helm and Golang. The author of this post makes a general comparison as follows:

Type	Best use case	Underlying technology	Amt of Effort
Helm	Stateless workload	Helm Charts	Med
Ansible	Stateless workload	Ansible Roles and Playbooks	Med
Golang	Stateful workload	Code developed in Golang	High

The aforementioned Kiali operator is an example of Operator developed in Ansible. The prometheus operator, is developed in Golang as the workload can be stateful depending on configuration. One needs to know how to develop operator in Golang in order to tackle the most complicated situations. This is requires some serious development effort. The documentation with a quick start section is available here. Even that is not very straightforward. RedHat, the maintainer of the CNCF Operator framework has a good blog post on how to develop an Operator in Golang.

The example requires some development knowledge to go through. On my MacOS (Intel) I have to configure the following prerequisites:

Install gcc, using command: xcode-select –install
Install the right version of golang. You can find the version here. The MacOS has a version of golang installed already so I had to install version 1.17 and link to it: brew install go@1.17 && brew link –force go@1.17
Install operator-sdk with home brew: brew install operator-sdk
When you run “operator-sdk version”, ensure the result shows a golang version that matches your installation.
If you need to push docker image, also connect to docker registry by running: docker login

Then we can create our working directory, initialize the repository and create boilerplate code (scaffolding) with these commands:

$ mkdir wordpress-operator && cd wordpress-operator
$ operator-sdk init --domain digihunch.com --repo github.com/digihunch/wordpress-operator
$ operator-sdk create api --group wordpress --version v1 --kind WordPress --resource --controller

With the repo initialized, we can go to the section “Defining the API” and “Implementing the Controller”. The blog post does not cover every code editing needed to bring up wordpress. You are supposed to go to the author’s repository to fit the changes into your own repo. The author’s repo has a few more controllers such as common.go and mysql.go.

At the end of the lab, you should be able to run the controller and bring up wordpress. I used my own repository for this lab and have made the code changes for this lap in a couple commits. To test locally with the code:

$ git clone git@github.com:digihunch/wordpress-operator.git
$ cd wordpress-operator
$ make install run

Then we can validate wordpress install from a new terminal as the instruction shows:

$ kubectl create -f config/samples/wordpress_v1_wordpress.yaml
$ minikube service wordpress --url

For Developers that requires more details, RedHat has an eBook for Kubernetes Operators, in supplement to the documentation. As DevOps professional, I’m mainly concerned with understanding how Operator works and using Operators correctly.

Too many Tools?

Now we seem to have too many choice of tools when it comes to deploying workload on Kubernetes. Kustomize and Helm can deploy simple workloads. Operator can deploy stateful workloads, as well as keep the workload status in check. Further, we have FluxCD and ArgoCD based on GitOps workflow.

When assessing a tool, we should think about the complexity of the workload deployed. If it is a single stateless workload, Kustomize or Helm should be sufficient. If it is not very simple but still stateless, we can consider using Helm charts developed by the community. For multiple workloads, we can build our own top-level chart to combine existing sub-charts created by the community.

Helm is essentially a package manager. It does not follow controller pattern and therefore will not monitor the current status of deployment. Helm has other limitations compared to Operator. For example, as a templating scheme, it reaches limitation when dealing with complex logic, even with the help of its helper functions. It is also hard to reason through the template code when we have to troubleshoot a deployment. Refer to this blog post for the author’s experience with Helm.

If we want our deployment to be fully declarative and continuous, then we will follow the Operator pattern by using a Kubernetes Operator. When we have many workloads of different levels of complexity, we can combine them with GitOps tool. Operator is one of the underlying technologies behind GitOps.

Workload profile	Just Installation	Installation and Maintain Status
Single stateless workload	Helm or Kustomize	Operator (using Ansible or Helm)
Single stateful workload	Helm or Kustomize	Operator (using Golang)
Multiple workloads	Helm (e.g. build parent chart)	GitOps in combination with Operator, Helm and Kustomize

The table above helps refine deployment requirement. It’s not a recommendation, but rather a model of analyzing deployment requirement.

Local multi-node cluster – Minikube, MicroK8s and KinD

Tue, 14 Sep 2021 11:18:00 -0400

In this post we compare Minikube, MicroK8s and KinD as different approaches to build multi-node cluster locally.

Is Docker desktop bad?

In the previous post about docker desktop as a single-node Kubernetes cluster setup, I touched on the deprecation of docker-shim. Now that CRI beats OCI as the standard for container runtime, the docker runtime will no longer be supported by Kubernetes. Also deprecated is docker-shim, the temporary interface that had make Docker runtime work in Kubernetes. This was announced in December 2020, and is coming through in Kubernetes 1.23, expected Oct 2021. However, docker desktop still uses docker runtime in it’s single-node Kubernetes cluster. This essentially renders itself a non-compliant Kubernetes environment.

Docker desktop still has great value for application developers. If your role is development, spending a lot of time coding business logics and need an easy-to-use container runtime on your laptop, Docker desktop is a good choice. The recent moves by the company seems to suggest that this is the business they are targeting now. On the other hand, if your roles are deployment, automation, orchestration, cloud native etc and you are looking for a playground, most likely you do need a runtime compliant to Kubernetes CRI. Docker desktop is not a CNCF-certified project anymore, and it is not your choice.

Alternatives

There are a number of alternatives, the most well-known ones are Minikube, MicroK8s, KinD and K3s with K3d. This presentation from CNCF in 2020 covers a lot of details about these technologies. I’ll try to add my opinion.

K3s is Rancher Lab’s lightweight Kubernetes distribution that supports multi-node cluster as well as different runtimes (e.g. containerd). It is not straightforward to setup, and k3d is an command-line wrapper to make it easy to install K3s cluster. K3s was accepted as a CNCF project but only at Sandbox maturity level, so it is not my choice.

The other three: Minikue, MicroK8s and KinD are all certified CNCF project. I will further discuss how to choose among them. These projects are technologies that takes different approach to address the challenges with deploying multiple nodes in local environment (e.g. my laptop).

The challenge with running a Kubernetes cluster with multiple nodes locally is how to manage these nodes. They are separate virtual resources that need to be isolated from computing perspective, and connected as a cluster. This is typically the use case of a Type II hypervisor, or alternatively, it can also be implemented with container technology. This layer of technology (referred to as drivers) makes a big difference.

Minikube

Minikube supports multiple drivers. Depending on your platform (Windows, Linux, or MacOS), the preferred driver is different. Refer to the document here for preferred driver, and this blog post for more instructions. In addition to the documents, here some notes from my personal experience:

On MacOS, Minikube lists Docker as preferred driver. I disagree with that. If you have no other reason to install Docker, then I would recommend hyperkit as the the preferred driver. Hyperkit can be installed with a simple Homebrew command. For two reasons I do not recommend Docker as the driver of Minikube. First, it requires a separate installation of Docker Desktop, which includes a built-in instance of hyperkit on its own. This isn’t neat. Second, I often need Metal LB add-on with Minikube for testing Kubernetes Ingress. With Minikube on Docker, the Ingress ports are not exposed to MacOS’s. Therefore you cannot directly visit websites spun up on Minikube. This is a known issue for a while due to limitation on docker bridge with Mac. Some reported an ugly workaround with docker-mac-net-connect but I never got it to work.
On Windows native environment, the preferred driver is hyper-V. The Minikube cli command have to run from Windows PowerShell.
On WSL2, Minikube doesn’t play well, regardless of driver. The hyperkit driver won’t work (it is designed for MacOS only). The kvm2 driver would require a KVM2 hypervisor. However, WSL2 itself is a VM on top of hypervisor, as explained here. If KVM2 driver works it would require nested virtualization so I doubt it will ever be supported. As for Docker on WSL2 as driver, Minikube has it as an experimental feature, and requires configuring cgroup to allow setting memory. I am not confident with it.

To me, Minikube is the tool for MacOS (I have Intel processor). On MacOS, we first need to install minikube and hyperkit with home brew.

We can then start a kubernetes cluster, with minikube in a single command. I noticed a process on my MacBook called dnscrypt-proxy that conflicts with hyperkit DNS server when starting minikube. I had to remove dnscrypt-proxy (part of Cisco Umbrella Roaming Client) in order to get minikube to work, as this thread suggests. You can find out by running:

sudo lsof -i :53

If dnscrypt-proxy is running, find out the application by PID and remove the application. Otherwise there will be issues. Check out this section on the document. The commands that I use to start multi-node cluster is:

minikube start --driver=hyperkit --container-runtime=containerd --memory=12288 --cpus=2 --disk-size=150g --nodes 3
kubectl get po -A
kubectl describe node minikube|grep Runtime

Node administration is simple. To enable dashboard, simply run “minikube dashboard”. To SSH to a node, simply do “minikube ssh -n <node_name>”. In order to stop the node and delete cluster, run “minikube stop && minikube delete”.

There are some addons in minikube, for example, efk, gvisor, istio, metrics-server. To list add-ons, and enable metrics-server, for example, run:

minikube addons list
minikube addons enable metrics-server

When creating cluster, instead of specifying the cluster imperatively, the configuration (e.g. driver, container runtime, cpu, memory, number of nodes, etc) can be stored as a profile with -p switch. Like other Minikube configuration information, Minikube profiles are stored in ~/.minikube under the profile directory.

Minikube also has a page that benchmarks the performance of these technologies, where it presents itself as the most performant.

Minikube, KinD, k3d and microK8s performance

MicroK8s

MicroK8s is developed by Canonical. It can use either Multipass or LXD container as driver. Multipass can configure Ubuntu VMs using cloud-init. It supports multiple hypervisor backends as well but hyperkit is the default on MacOS, Hyper-V on Windows, and KVM on Linux.

MicroK8s supports multi-node configuration across multiple machines. That is, nodes can span across multiple physical machines. This is more powerful than Minikube where multiple nodes are on the same physical machine. It brings MicroK8s additional use cases such as edge and IoT devices.

With that capability comes the extra step to configure a MicroK8s cluster. You will need to manually join a node to a cluster because the new node is potentially located on a different machine, and you execute the command from the new machine. On the other hand, with Minikube you simply specify the number of nodes desired in a command or profile.

Snap is the native package manager to install MicroK8s, making GNU Linux (e.g. Ubuntu) the native platform. It also supports MacOS and Windows. MicroK8s does not rely on Docker (unlike KinD and Minikube with Docker as driver), and uses containerd as runtime.

MicroK8s comes with its own packaged version of kubectl, and you use that with “microk8s kubectl” command, which is not convenient. You can configure your host kubectl to point to the MicroK8s cluster, as an extra step.

Compared to the other two technologies, MicroK8s is more powerful in the sense that the cluster is build on nodes across multiple machines. However, it takes more step to configure even for a multi-node, single-machine environment. Refer to this post for the steps.

KinD

KinD is similar to Minikube with Docker as driver. It is more restricted than Minikube considering Docker is the only driver it supports. This makes it a requirement to have Docker installed locally.

Although KinD uses Docker to run nodes, it does not use Docker as its container runtime. Therefore it remains as compliant environment.

Another advantage of KinD is it supports Docker on WSL2 very well. Simply install KinD on WSL2 and start Docker. This blog post contains the steps required to install KinD vs Minikube on WSL2. There is a comparison table in the conclusion section that highlights the fact that it is much easier to install KinD with WSL2 than to install Minikube.

However, there are currently some known limitations with Docker desktop for Windows (including on WSL2). One is the absence of docker0 bridge. This means on Windows you cannot route traffic to the containers.

For cluster specification, KinD can configure a cluster declaratively using YAML file for example, the kind-config.yaml contains the following snippet:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
 - role: control-plane
 - role: worker
 - role: worker
 - role: worker
networking:
 disableDefaultCNI: true

We can bring up a cluster with a command:

kind create cluster --config=kind-config.yaml

The command will also configure the kubectl context so we can check node with kubectl command. The file is in my real-quicK-cluster repo.

Conclusion

After reviewing the technologies that back up multi-node kubernetes cluster for my role, I find that Minikube with hyperkit is my favourite for MacOS. On WSL2, I prefer to use KinD. Since I do not use Windows native environment or Ubuntu on my laptop, I cannot make recommendations. However I would start with Minikube (with hypverv or kvm2 as driver).

Update July 2022: When the test workload involves persistent storage, KinD is a better choice. When the test workload involves load balancer. Minikube is a better choice.

As to storage provisioner, Minikube with storage-provisioner addon uses k8s.io/minikube-hostpath. KinD uses rancher.io/local-path. When I have to test workload with persistent storage (e.g. PostgreSQL with Crunchy pgo), I realized Minikube have permission issues with persistent volume, as discussed here as an issue with multiple nodes. The issue has been open since Aug 2021.

For Load Balancer, Minikube has metallb as an addon and I can configure it within a bash script conveniently. With KinD, I’d have to configure that in a few steps with both kubectl and Docker CLI commands and I was not able to connect to the load balancer by IP even after following the steps. So I tend to just use Minikube to test workload requiring load balancer and service mesh.

I find myself switch between Minikube and KinD on my MacBook depending on the test workload.

Creating X.509 TLS certificate for workload on Kubernetes

Sun, 29 Aug 2021 23:19:00 -0400

In deployment automation, I often had to create self-signed X.509 certificate for testing TLS traffic into Kubernetes. Sometimes self-signed, sometimes signed by a CA. This post summarized the approaches I’ve taken.

Create self-signed certificate with OpenSSL

Traditionally, this is done in three OpenSSL commands:

openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=Health Certificate Authority'
openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=*.orthweb.com'
openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt

I have an older post to cover the basics of cryptography in TLS certificate and PKI. In the three commands above, the first produces a private key and self-signed certificate for a CA. The second creates a private key and a CSR for the web site. The third one uses the CA’s signing private key to sign the CSR from the website. The output is the certificate for the website.

Workloads running in Kubernetes typically consume certificates stored in Kubernetes Secret. The cons of this approach is that it usually requires an extra step to import the certificate files into Kubernetes Secret. For example:

kubectl create -n orthweb secret generic orthweb-cred --from-file=tls.key=server.key --from-file=tls.crt=server.crt --from-file=ca.crt=ca.crt

Note, people use the term self-signed certificate loosely. It sometimes means literally a certificate that is self-signed, like the one generated above. Sometimes, I had to self-sign a CA, then use the CA to sign one certificate for the server and one for the client. Because the application being tested requires that the client and server’s certificates both under the same CA. This would involve a few more commands, for example:

# Self-sign a CA
openssl req -x509 -sha256 -newkey rsa:4096 -days 365 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=issuer.digihunch.com/emailAddress=info@www.digihunch.com -keyout /tmp/ca.key -out /tmp/ca.crt

# Generate a CSR for server
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=server.digihunch.com/emailAddress=orthweb@www.digihunch.com -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:server2.digihunch.com -keyout /tmp/server.key -out /tmp/server.csr

# Use the self-signed CA to issue a certificate to the server
openssl x509 -req -sha256 -days 3650 -in /tmp/server.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/server.crt -extfile <(echo subjectAltName=DNS:orthweb.digihunch.com,DNS:server2.digihunch.com)

# Generate a CSR for clietn
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=client.digihunch.com/emailAddress=client@www.digihunch.com -keyout /tmp/client.key -out /tmp/client.csr

# Use the self-signed CA to issue a certificate to the client
openssl x509 -req -sha256 -days 365 -in /tmp/client.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/client.crt

In the example above, it is important to note that even though the server’s CSR contains subject alternative name (SAN), I still have to specify the SAN again when signing the certificate for the server.

Similar to OpenSSL there are other toolkits such as CFSSL that supports specifying configuration files. However, the steps in Shell command are generally not always easy to automate.

Create self-signed certificate with Helm

Moving to the context of workload deployment in Kubernetes, running openSSL command isn’t always a viable option. For example, generating a certificate in the middle of deployment using a Helm Chart.

In Helm, template functions is for this purpose. In my Korthweb project I used genSignedCert to create self-signed certificate and then store the key, certificate and CA certificate as Kubernetes Secret:

{{- $dbtlscert := genSignedCert .Values.dbtls.certCommonName nil (list .Values.dbtls.certCommonName) 365 $ca }}
apiVersion: v1
kind: Secret
metadata:
 name: {{ .Values.dbtls.certCommonName | quote }}
 namespace: {{ $.Release.Namespace | quote }}
type: kubernetes.io/tls
data:
 tls.crt: {{ $dbtlscert.Cert | b64enc | quote }}
 tls.key: {{ $dbtlscert.Key | b64enc | quote }}
 ca.crt: {{ $ca.Cert | b64enc | quote }}
{{- end }}

The cons of this approach is that the syntax is not straightforward. As indicated in Helm documentation: Helm Chart templates are written in the Go template language, with the addition of 50 or so add-on template functions from the Sprig library and a few other specialized functions. While we talk about the “Helm template language” as if it is Helm-specific, it is actually a combination of the Go template language, some extra functions, and a variety of wrappers to expose certain objects to the templates. Many resources on Go templates may be helpful as you learn about templating.

Create self-signed certificate with Cert-Manager

The Cert Manager project is very popular to produce X.509 certificates directly in Kubernetes secret. We can install cert manager using Helm:

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.0.3 --set installCRDs=true
kubectl get pods -n cert-manager
kubectl get crd | grep cert-manager.io

Alternatively, FluxCD’s documentation on Kustomization dependency uses Cert Manager as an example. It is a good way of installing cert-manager if you have GitOps pattern.

Creating self-signed certificate for website is fairly simple. It starts with bootstrapping a CA issuer. Take the manifest below as an example. When creating the first certificate, make sure to specify isCA=true, so it stores the signing private key along with its own certificate in the ca-secret. Then use the newly created CA as issuer to create the X.509 certificate for the website.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: selfsigned-issuer
spec:
 selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: my-ca
 namespace: orthweb
spec:
 isCA: true
 commonName: my-ca
 secretName: ca-secret
 privateKey:
 algorithm: ECDSA
 size: 256
 issuerRef:
 name: selfsigned-issuer
 kind: ClusterIssuer
 group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: my-ca-issuer
 namespace: orthweb
spec:
 ca:
 secretName: ca-secret
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: orthweb-cert
 namespace: orthweb
spec:
 commonName: orthweb.com
 secretName: orthweb-secret
 duration: 2160h
 renewBefore: 72h
 subject:
 organizations:
 - digihunch
 dnsNames:
 - web.orthweb.com
 - dcm.orthweb.com
 privateKey:
 algorithm: ECDSA
 size: 256
 issuerRef:
 name: my-ca-issuer
 kind: Issuer
 group: cert-manager.io

The site certificate is directly stored in Kubernetes Secret as specified in the secretName field. To fetch the certificate text, we need to decode the secret entry, for example:

kubectl -n orthweb get secret orthweb-secret -o jsonpath='{.data.ca\.crt}' | base64 -d

Note that the example above uses ECDSA algorithm with size 256 for private key and certificate. It requires that the TLS client to support ECDSA algorithm as well. For more supportability, you can use RSA algorithm (2048 or 4096 size).

In addition to creating self-signed certificate, Cert Manager supports a number of other issuer types. For example, the support of ACME issuer type enables integration with Let’s Encrypt.

Cert Manager can secure Kubernetes Ingress resources with a sub-component called ingress-shim. It is configured via annotation on the Ingress resource.

Cert Manager

Create CA-signed certificate manually

For a certificate signed by a CA, there are may paid options, from manual, to self-help, to automated. The classic manual way is using OpenSSL, generating key, CSR. The CA takes CSR to sign a X.509 certificate returned to the website administration.

Many CA websites charges for a fee and makes it easy. For example, this site currently uses certificate from SSLs.com. Apart from the fee-for-cert option, there is a website named “SSL for free“, a CA with free option for 90-day single-domain, non-wildcard certificate and we can request it simply on their website, with proof of domain ownership.

The other popular free option is Let’s Encrypt, which also employs ACME protocol. The protocol requires ACME challenges to be satisfied in order to proof domain ownership. There are a few types of challenges:

HTTP-01 challenge
DNS-01 challenge
TLS-SNI-01 challenge
TLS-ALPN-01 challenge

I have used the HTTP-01 and DNS-01 challenges. The DNS-01 challenge requires adding TXT records to DNS configuration. The HTTP-01 challenge requires adding a DNS A-record to resolve to the server, then two URIs with pre-defined value.

When I first set up this site I used certbot (the client program for letsencrypt) to create certificate every 90 days from the wordpress server, following this guide, including solving DNS-01 challenges.

Create CA-signed certificate automatically with cert manager and letsencrypt

With Kubernetes, cert-manager has the ability to integrate with let’s encrypt for full automation. Here is a good blog post on this. Domain verification is still required but it can be done automatically. We first need to register an A record that resolves host name to the Ingress IP to enable this automation. The domain ownership validation may use the ACME protocol. This should also work on private networks with private DNS and ACME protocol using a private boulder server.

Take domain name demo1.digihunch.com for example, if ingress exposes a public IP address which the domain name resolves to, then we can configure certificate with the following manifest:

kind: IngressClass
metadata:
 name: istio
spec:
 controller: istio.io/ingress-controller
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt
spec:
 acme:
 privateKeySecretRef:
 name: letsencrypt
 server: https://acme-staging-v02.api.letsencrypt.org/directory
 solvers:
 - http01:
 ingress:
 class: istio
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: demo
spec:
 dnsNames:
 - demo1.digihunch.com
 issuerRef:
 kind: ClusterIssuer
 name: letsencrypt
 secretName: demo-tls

This example uses Istio as ingress controller but the method works regardless of the controller technology behind Ingress. In the ClusterIssuer object, we’re telling it to use the staging server from letsencrypt. We also specify http01 as challenge type, and that the ingress type is istio. In the Certificate object, we provided dnsName and specified ClusterIssuser. We also tell it to store the credentials to a secret named demo-tls.

When we apply the resources above, the ClusterIssuer connects to letsencrypt server via ACME protocol. Since the DNS name already resolves to the Public IP that the ingress is hosting, the ClusterIssuer configures the required Ingress, Services and Pods accordingly so the token to satisfy the challenge is presented at the designated URI. Instead of a staging server, we can also use production ACME server for production deployment. Note that the production ACME endpoint has a stricter rate limit.

When the ACME validation is in progress, it is important to ensure that port 80 is open and there is no other mechanism (such as routing rule, authorization requirement, mandatory redirect to 443) that blocks access from letsencrypt server.

Bottom line

Cert Manager is deployed in Kubernetes, supporting a variety of issuer types. As a Kubernetes-native tool, it is a no-brainer for Kubernetes workload for X.509 certificate. Compared with using template function in Helm, it is not dependent on template function and the syntax is consistent (YAML). Compared with OpenSSL or other binary tools, it is easy to integrate with the platform.

Single-node Kubernetes cluster – docker desktop

Sun, 22 Aug 2021 00:24:00 -0400

While there are many tools to set up single-node Kubernetes cluster (e.g. minikube, MicroK8s, kind, or k3s with the k3d wrapper), docker-desktop has a significant advantage: it comes with Docker installation, on MacOS, or on Windows. It is installed simply by enabling the option “Enable Kubernetes”. It can be blown away and reset in a heartbeat (with the button “Reset Kubernetes Cluster”). For its versatility, docker-desktop is a great development environment.

However, there are always nuances, which motivates me to write this blog. I wanted to note down what is on earth different about Docker-desktop, because the instructions for applications might differ slightly between single-node cluster on MacOS/Windows and the “real” multi-node cluster.

I will start with with a deep dive into the docker-desktop architecture, then we’ll go through the steps to install some common applications with Kubernetes.

Docker-Desktop on MacOS

There are a number of open-source and proprietary projects involved to bring docker-desktop to implementation. Let’s begin with the following five:

Hypervisor Framework: Apple’s APIs on MacOS that allows you to interact with virtualization technologies in user space.
bhyve: A type-2 hypervisor initially written for FreeBSD (and was contributed to FreeBSD in May 2011).
xhyve: A port of bhyve project to MacOS with integration via Apple’s Hypervisor Framework. The Hypervisor Framework allows xhyve to run entirely in userspace. It is sometimes loosely referred to as xhyve/bhyve hypervisor, and is optimized for lightweight virtual machines and container deployment.
HyperKit is an open-source toolkit on macOS based on xhyve. HyperKit is lightweight and therefore allows you to embed hypervisor capabilities in your application. The hypervisor component in HyperKit is based on xhyve/bhyve. HyperKit is designed to be interfaced with higher-level components such as the VPNKit and DataKit. Docker-desktop and MiniKube are built on HyperKit.
LinuxKit is a toolkit for building custom minimal, immutable and purpose-build Linux distributions. It supports several well-known hypervisor platforms, such as HyperKit, Hyper-V, qemu and VMware. LinuxKit started as an internal project in Docker Inc and is now managed as a Moby Project.

Hyperkit is installed as part of docker desktop. The process can be found with ps command:

ps -Af | grep hyperkit

Or in the activity monitor:

Docker related processes

Docker-deskop is essentially a LinuxKit virtual machine (as defined here). It runs containerd process inside of the virtual machine. This is an older article about this architecture. If Kubernetes is enabled, the virtual machine is also installed with kubelet, the agent process running on each Kubernetes node.

Since MacOS is not the direct host of the containers, there is no way to map MacOS file system to container’s as you can with a Docker/Kubernetes host. Prior to Docker 20.10, there used to be a trick to indirectly access host volume from MacOS terminal. It has stopped working according to this issue but workarounds are provided here. This post proposes some good alternatives to access the file system of LinuxKit VM. For example, use netcat:

 user@LinVM # nc -U ~/Library/Containers/com.docker.docker/Data/debug-shell.sock

/ # cat /etc/os-release
cat /etc/os-release
PRETTY_NAME="Docker Desktop"
/ # cat /etc/kubernetes/current-version
cat /etc/kubernetes/current-version
kubeadm version: &version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"archive", BuildDate:"2021-06-18T05:24:26Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

Typing in this terminal session feels clunky. According to this thread, we can connect to the LinuxKit VM with tty and sane auto completion, using the command below:

stty -echo -icanon && nc -U ~/Library/Containers/com.docker.docker/Data/debug-shell.sock && stty sane

There are some other alternatives, using privileged Docker containers:

docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh

The following command uses a smaller image:

docker run -it --rm --privileged --pid=host justincormack/nsenter1

As with Kubernetes, to access the file system on the node is via a privileged container, you can follow the tips from Azure, identify node name, and debug against the node using a special container:

kubectl debug node/docker-desktop -it --image=mcr.microsoft.com/aks/fundamental/base-ubuntu:v0.0.11

Note that the root directory on the host is mounted to container’s file system as /host. This mapping renders a lot of symbolic link as dangled, even though they are actually not on the host file system.

Docker-Desktop on Windows

Docker works with Linux kernel. There have been a couple of efforts to run Linux virtual machine on Windows. For example, Hyper-V backend, and Windows Subsystem Linux (WSL) backend.

Traditionally, Docker on Windows was implemented with Hyper-V as the hypervisor. A LinuxKit distro is running on the Hypver-V VM, provider Linux kernel capabilities. Docker refers to containers running in this architecture as “Windows Containers”, which is a misnomer in my opinion.

The first release of WSL provides a Linux-compatible kernel interface and runs a GNU user space on top of the interface. Neither the Linux kernel code, or a hypervisor is involved. The user space contains GNU Bash shell, command language, command-line tools and interpreters. The absence of Linux kernel in WSL, makes it useless for Docker setup. At that time The Hypver-V backend was still the only option to host docker container during the first version of WSL. This post has a diagram of Docker on Windows with Hyper-V backend.

WSL2 comes with a real Linux Kernel (also on top of Hyper-V), making WSL2 a better alternative than the legacy Hyper-V as the backend of Docker on Windows. It can be turned on as the screenshot shows above. The rest of this post assumes WSL2 as backend. In this setup, we run a Bootstrapping distro independent of the WSL2 Linux distro, although both inside of the lightweight Linux Utility VM. Below is the diagram:

The innovative component is the lightweight Linux Utility VM. It is called a VM, but very different from the traditional sense of VM such as VirtualBox or VMware. Traditional VM is isolated from host OS, slow to boot and has large memory footprint. The lightweight Utility VM on the other hand, is integrated with host OS, super fast to boot (i.e. ~1 second), and comes with small memory footprint. It is not turned on until needed. The VM runs both a WSL2 Linux Kernel and GNU/Linux usermode (known as “distribution”, for example, Ubuntu). When an end-user say WSL2, s/he most likely refers to the distribution. Similarly, the so called “docker-desktop on Windows with WSL2 backend”, is also managed as two WSL2 distros: the bootstrapping distro (docker-desktop) and the data store distro (docker-desktop-data).

The detailed components in the Bootstrapping distro is in the second diagram in this post, which has a detailed discussion. With this architecture, you don’t even need the WSL2 Linux Distro for Docker desktop to function. You can even run docker CLI command from Windows PowerShell without any Linux distro (although this is implemented only for backward compatibility and not recommended anymore) . In the following session, we first list out the WSL2 distros. Notice that the docker-desktop distro is not the default. We then connect to the distro using -d switch. Last, we run docker info from windows user space.

PS C:\WINDOWS\system32> wsl -l -v
 NAME STATE VERSION
* Ubuntu Running 2
 docker-desktop Running 2
 docker-desktop-data Running 2
PS C:\WINDOWS\system32> wsl -d docker-desktop
WINLAPTOP:/mnt/host/c/WINDOWS/system32# cd ~
WINLAPTOP:~# printenv|grep DIST
WSL_DISTRO_NAME=docker-desktop
WINLAPTOP:~# exit
PS C:\WINDOWS\system32> docker info
Client:
 Context: default
 Debug Mode: false
 Plugins:
 buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
 compose: Docker Compose (Docker Inc., v2.0.0-beta.6)
 scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 93
 Running: 80
 Paused: 0
 Stopped: 13
 Images: 28
 Server Version: 20.10.7
 Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
 userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
 seccomp
 Profile: default
 Kernel Version: 5.10.16.3-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 12.32GiB
 Name: docker-desktop
 ID: WHDE:PJF3:HVFC:AZJA:EDKH:VUZR:RRUJ:HHXX:TDV5:4UJG:XY4E:PK4F
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
 127.0.0.0/8
 Live Restore Enabled: false

The wsl -d command as illustrated above is a good way to connect to the docker-desktop distro. The alternative to get to the distro is via privileged container (Docker) or helper pod (Kubernetes), which is the same as in Docker-desktop on MacOS. Refer to the section above for specific steps.

Application Install on docker desktop

The followings are my notes to install commonly used applications in Docker Desktop with Kubernetes. They work on both MacOS or WSL2, requiring Kubernetes enabled.

Metric server

The metric server is provided in the official repository. Releases are publish here, which provides the installation step as follows:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.5.0/components.yaml

However, there is some issues when deploying it on MacOS, the deployment will fail due to certificates not matching the hostname. To fix the issue, it is recommended to download the yaml file (components.yaml), and edit the file by adding –kubelet-insecure-tls to the args section of the container named metrics-server. This is sufficient to fix the issue. Some people are not comfortable with port 443 being insecure TLS, and would rather change the port to 4443. This is completely unnecessary but if that’s the case, make sure the named port for https is also updated to 4443.

Once metric server has been installed, the following two commands should return meaningful results:

kubectl top no
kubectl top po

This should also allow application (Pods) to query for cluster resource usage. When working with single-node cluster on MacOS or WSL2, multiple Pods might come up with a single command and the memory can be easily over-subscribed. The two commands above allows you to check and make adjustment on the node configuration (in Docker preference).

Dashboard

Similar to metric server, the dashboard is kept in the official repo, in the the path of aio/deploy/recommended.yaml. Here is the published release, where the instruction says:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.1/aio/deploy/recommended.yaml

However, this is not directly applicable either because the login page requires token or kubeconfig. We need to be able to bypass that. To do so, download the yaml file (recommended.yaml), and add parameter –enable-skip-login to the args section for the container named kubernetes-dashboard.

To display the login page properly, we need to start the proxy using this command:

kubectl proxy

The login page will then be available at this URL. The URL reflects the namespace and service name. On the login page, the skip button will be available.

Rancher

The official installation guide of Rancher 2.5x recommends RKE Kubernetes. If you prefer not to run a separate cluster on MacOS, you can install it on docker desktop (with Kubernetes enabled). The installation steps require Helm 3 and are completed in three helm commands.

Install Nginx Ingress controller using Helm, following the three commands here. Alternatively, you can apply the rendered template as posted here. The controller will later be used by the ingress that Rancher’s chart creates.
Follow the steps on this page to install Rancher, even though the page does not say it applies to docker desktop. If you do not have TLS certificate, the Rancher helm chart can generate one for you, using cert-manager.
The installation exposes rancher application on port 443 of the MacBook, and the cert is issued to “rancher.my.org” by default. To access it, add “127.0.0.1 rancher.my.org” to the host file.

To back out from the steps above, just uninstall with helm. For example:

helm uninstall rancher -n cattle-system
helm uninstall cert-manager -n cert-manager
helm uninstall ingress-nginx

Jenkins

Similar to Rancher, Jenkins instruction assumes minikube cluster instead of docker desktop.

helm repo add jenkinsci https://charts.jenkins.io
helm repo update
kubectl create namespace jenkins
helm install jenkins -n jenkins -f https://raw.githubusercontent.com/jenkinsci/helm-charts/main/charts/jenkins/values.yaml jenkinsci/jenkins
kubectl --namespace jenkins port-forward svc/jenkins 8080:8080

To find out the default password for admin user:

kubectl -n jenkins get secrets jenkins -o jsonpath={.data.jenkins-admin-password} | base64 -D

To uninstall Jenkins:

helm uninstall jenkins -n jenkins

Container Runtime

Although I seem to be a proponent of docker desktop thus far, this post would be incomplete to not discuss what is missing with docker desktop. One key difference between docker desktop and minikube is the container runtime being used. Docker desktop uses docker as the runtime, and it does not support other runtime as of now. Minikube allows user to choose runtime, including containerd.

Because of this difference, Kubernetes nodes with Docker as runtime and with containerd as runtime place pod log files in different locations. To find out the runtime, use the following command:

kubectl describe node <node_name> | grep Runtime

If the runtime is docker, the stdout of container is placed in /var/lib/docker/containers/<sha>/. If the runtime is containerd, the stdout log of pods are stored in /var/log/containers/. This is important to know when you configure log shipping and needs to get stdout from node. The log path used in containerd is the standard path in compliance with Container Runtime Interface (CRI) so you should develop log shipping solution based on that.

Docker’s refusal to comply to CRI also caused Kubernetes to stop supporting it as container runtime as of Dec 2020. For more background, refer to this and this. Here is also an article with great diagrams on the removal of docker-shim.

Bottom line

Docker-desktop is a great tool for a quick single-node Kubernetes environment. As of docker 20.10, docker-desktop still uses docker as runtime. This limits its use case to development only.

If you need a CRI compliant environment, docker-desktop is not a good choice. We will discuss alternatives in the next post.

Helm – Configuration Management for Kubernetes Resources

Mon, 26 Jul 2021 19:28:22 -0400

Developer ships application in Docker container, so it can eventually hosted in Kubernetes cluster. However, there are still some installation steps, before the application can operate online in production. In this post, we use the container image of Orthanc application as a starting point. We first build services in Kubernetes to go through these steps. Then, to automate the steps, we build a helm chart. The code is kept in Korthweb project, in which the manual directory has the files requirement for manual deployment, and the helm directory is the helm chart.

Manual Deployment

The manual deployment steps include different kinds of activities, such as creating X.509 certificates, apply config map, create Kubernetes deployment using the YAML declarations, and use helm to install dependency. The steps need to take place in a particular sequence. Some step requires pulling information from secrets created in the previous step. This is why the deployment is not portable.

In order to automate the steps, one might think of wrapper script, which is very limited. A configuration management tool is needed in this scenario. Two common options are Kustomize, and Helm. Kustomize is a native tool which can be run by kubectl. It is also driven by declarative statement in YAML, which is simple to grasp. However, in lack of a templating mechanism, Kustomize may require wordy statements. Helm, on the other hand, comes with a templating mechanism which greatly increase reusability, making it more suitable for complex steps required in installation.

Helm Repo and Chart

Helm is known as package manager for applications running on Kubernetes. Helm defines an application as a collection of related Kubernetes resources, and it manages application deployment through a templated approach. An installation workbook is called a chart. Charts are kept in repositories. There are some well-known repositories, such as Bitnami, Helm stable. You need to add a repostory before using the Helm Charts in it. To add a repo, run:

helm repo add bitnami https://charts.bitnami.com/bitnami

You can host your own repo (public or private) as well. To search for charts across repositories, the best place is artifact hub, which indexes charts from a lot of public repositories. To search for charts from the repositories added, run:

helm search repo postgres

Template is the soul of Helm chart. A Helm chart consists of a directory of files following specific pattern so Helm can understand how to deploy the application. For example, the chart name is the name of the working directory. Under the directory, the values.yaml and chart.yaml defines variables and constants, both serving as template inputs. The template directory is the most important part of the directory where the installation logics are defined. Helm runs the entire directory hierarchy (except for paths specified in .helmignore file) through a Go template rendering engine. The template result spec out the detailed steps.

A great example of using Helm chart to simplify installation is the wordpress chart by Bitnami. You can install all the required components in a single command:

helm install my-release bitnami/wordpress

The helm chart in Korthweb project is also an evolving helm chart I created for installing Orthanc application.

Helm V3 (released in late 2019) includes an important architectural change – the removal of tiller. This means Helm can operate on the client-side – a significant simplification. Helm graduated from CNCF project in 2020. There are also a few changes in V3, as outlined here, including the consolidation of requirements.yaml into Charts.yaml.

Template and Function

As discussed, templating is the key towards reusability and flexibility in configuration management. We’ve worked with Jinja2 template engine in Ansible and Python. Here in Helm, we use Go templates with some enhancement. The syntax is mostly based on Go template, which is somewhat similar to Jinja2. Helm also added all functions from the Sprig library, making it more powerful and flexible than Jinja2. Helm chart developer should be very familiar with these functions, as well as the best practices. For example, the cryptographic and security functions in Sprig library gives us the ability to create self-signed X509 certificates during installation.

Since template introduces another layer of abstraction, to help troubleshooting we should be able to preview rendered template with the template command:

helm template orthanc | less

The command above simply renders template without attempting to execute the chart. To go one step further, you can dry-run the installation with:

helm install orthweb ./orthanc --debug --dry-run | less

Although Jinja2 (using {% … %} to express template control) and Go (using {{ … }} to express template control) have different syntaxes, one aspect that is similar between them, is chomping whitespace with minus sign (-). This is pretty common in templating language. The documentation of both Jinja2 and Helm have a section on whitespace control. Not paying attention to this nuance may cause pesky errors.

Dependency

The Orthanc application relies on Postgres database, which itself is deployed by a separate helm chart. This can be specified in Chart.yaml (Helm V3), like this:

dependencies:
 - condition: postgresql-ha.enabled
 name: postgresql-ha
 repository: https://charts.bitnami.com/bitnami
 version: 7.8.x

The values of variables of the dependency chart can be specified in values.yaml of the root chart. They can also be imperatively specified as a parameter of helm install command.

The section above also requires the dependency chart to be downloaded into the charts sub-directory. This can be done with:

helm dependency update

Then you will notice a file with tgz extension in the charts sub-directory. Note that when you change the version of the dependency package in Chart.yaml, then you will need to run the command again. Alternatively, this command can be automatically executed before helm install if you specify the switch –dependency-update with helm install.

The main chart (e.g. wordpress) is referred to as parent chart, and the charts it depends on are referred to as sub-chart (e.g. mariadb, memcached). When it comes to managing property values, values from parent chart can override those from sub-chart, as explained here. On the other hand, values from sub-chart can override those from parent chart in two formats: export format (keyword exports) and child-parent format (keyword import-values). This is something to be careful and we can use the aforementioned template command to display the rendered values.

Hooks

Helm does a great job in figuring out the dependency relationship between kubernetes objects defined in the chart, and create them in order. So typically you do not need hooks for objects in the chart. However, in certain circumstances, such as cleaning up after uninstallation, we may need hooks. Here is a list of available hooks.

It is worth-noting that hook is not tied to an action. Instead it is tied to a kubernetes resource. The resource could be a job, a config map, etc. The resource is tied to a hook simply by resource annotation.

Moving to GUI

Helm is a command-line tool. For a team with varying levels of familiarity with command-line, GUI-based tool is a better option. For that, some enterprises adopt Rancher, a comprehensive Kubernetes cluster management platform. Rancher manages many aspects of Kubernetes cluster through web portal. One aspect is the support of helm chart. Rancher can be install on a cluster of its own. For demo, it can also be installed on docker desktop, a single-node Kubernetes cluster by Docker. In both cases, Nginx ingress controller needs to be configured.

Certified Kubernetes Administrator (CKA) Exam

Fri, 30 Apr 2021 09:50:00 -0400

The Certified Kubernetes Administrator (CKA) exam is a hands-on session where you need to follow the instructions to configure the system in a bash terminal on the web browser. In my experience, some shortcut keys (such as Alt+F) do not work, which slows me down a little bit. For each question, you need to switch kubectl context as instructed in the question. Some questions share the same context so it is very easy to omit this step. You can verify response with your own command but will not be told whether you scored in each question. During the CKA exam I tried to spin up a terminal session from within Vim editor and the terminal ran out of buffer. I had to reboot the machine with the help of proctor, and my completed work are saved.

In general this is an exam I enjoy preparing and writing because it is very hands on. Result is out a day after, and I passed at 96%. I heard about tight timelines but I managed to finish 15 minutes before the end, most likely owing to my dexterity with Linux commands. With that I’m happy to share my notes in preparation for the CKA exam.

For taking CKA exam, we should be familiar with the diagram above.

Tips for Troubleshooting

The CKA exam is hands-on and therefore requires quite a bit of troubleshooting. Here are my notes.

Check Node status to start with
Check core services on each node:
- sudo systemctl status kubelet
- sudo systemctl status docker
- sudo journalctl -u kubelet
- sudo journalctl -u docker
Check component logs (on hosting VM)
- /var/log/kube-apiserver.log
- /var/log/kube-scheduler.log
- /var/log/kube-controller-manager.log
If cluster is built by kubeadm, then some of those services are running in Pods within kube-system namespace. Check those pods:
- run interactive shell: > kubectl exec podname –stdin –tty — /bin/sh
- there is an image for lots of useful network tool called nicolaka/netshoot

Store pod names to variable. e.g. > POD_NAME=$(kubectl get pods -l run=nginx -o jsonpath=”{.items[0].metadata.name}”)
With kubectl, you may
- alias it to k for faster typing
- –dry-run: to run imperative command without creating object
- –record: record the command that was used to make a change
- -o: set output format, wide, yaml, or jsonpath=”expression”. For example, to get pod name: > kubectl get pods -l run=nginx -o jsonpath=”{.items[0].metadata.name}”
- –sort-by: use JSONPath expression
- –selector: filter results by label

Build K8s cluster using kubeadm

The CKA exam requires you to know how to build cluster with kubeadm. This involves installing four components (docker-ce, kubeadm, kubectl and kubelet), as outlined below:

step	command
1. Install docker-ce	> curl -fsSL https://download.docker.com/linux/ubuntu/gpg \| sudo apt-key add - > sudo add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" > sudo apt-get update > sudo apt-get install -y docker-ce=18.06.1~ce~3-0~ubuntu > sudo apt-mark hold docker-ce > sudo systemctl status docker
2. Install kubeadm, kubelet and kubectl	> curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg \| sudo apt-key add - cat << EOF \| sudo tee /etc/apt/sources.list.d/kubernetes.list deb https://apt.kubernetes.io/ kubernetes-xenial main EOF > sudo apt-get update > sudo apt-get install -y kubelet kubeadm kubectl > sudo apt-mark hold kubelet kubeadm kubectl
3. Form a K8s cluster	On master node: > sudo kubeadm init --pod-network-cidr=10.244.0.0/16 This command prints out a command for worker nodes to join.	On worker node: sudo the command generated on master
4. Configure kubectl	On master node: > mkdir -p $HOME/.kube > sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config > sudo chown $(id -u):$(id -g) $HOME/.kube/config	Optionally on worker node: > mkdir -p $HOME/.kube then scp $HOME/.kube/config from control plane node
5. Set up cluster networking	> echo "net.bridge.bridge-nf-call-iptables=1" \| sudo tee -a /etc/sysctl.conf > sudo sysctl -p Then from any environment with kubectl, bring up the system pods for cluster networking > kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Add new node to KubeAdm cluster

This is fairly simple with the help of kubeadm. The node to join cluster must be able to communicate with master node. Create a token and print join command from master node:

> kubeadm token create --print-join-command

Then from the node to join, run this command as sudo. You will see that it performs the TLS bootstrap for you. Once completed, the standard output will say this node has joined the cluster. You can confirm with command:

> kubectl get nodes

Sometimes one needs to migrate pods to the newly joined node. This can be done by draining the existing nodes.

Note you can also use kubespray to build K8s cluster as previously discussed, and here is a my IaC project to launch AWS instances and build a K8s cluster with kubespray on top of it. For my learning, I often create a GKE (Google Kubernetes Engine) cluster from GCP’s cloudshell. There is a guide on how to start a cluster but it comes down to three commands:

$ gcloud config set compute/zone us-east1-b
$ gcloud container clusters create tcluster --num-nodes=3
$ gcloud container clusters get-credentials tcluster

The third command above is to configure kubectl on the cloudshell. Follow this guide if you need to SSH to node.

Upgrade KubeAdm cluster

This involves upgrade three components (kubeadm, kubectl and kubelet) on two types of node: master node and worker node. They steps vary slightly for two nodes. But drain and uncordon is needed for both types of nodes. Pick a node and follow the steps below:

Step	Command
1. drain the node from kubectl client (e.g. master node)	> sudo kubectl drain nodename --ignore-daemonsets
2. Determine kubeadm target version	> apt-mark showhold > sudo apt-mark unhold kubeadm kubectl kubelet > apt list --installed \| grep kube > apt-cache show kubeadm \| less > sudo apt-get install -y kubeadm=1.20.2-00
3. update kubeadm	On master node: > sudo kubeadm upgrade plan v1.20.2 > sudo kubeadm upgrade apply v1.20.2	On worker node: > sudo kubeadm upgrade node
4. On the node to update, determine target version for kubectl and kubelet, then install	> apt-cache show kubectl \| less > apt-cache show kubelet \| less > sudo apt-get install -y kubectl=1.20.2-00 kubelet=1.20.2-00
5. Restart kubelet	> sudo systemctl daemon-reload > sudo systemctl restart kubelet
6. Uncordon	> kubectl uncordon nodename

Backup and restore Etcd

Etcd is a distributed key-value store. It uses Raft protocol for distributed consensus. Etcd is the third distributed system I touch on. The previous two are: Cassandra (using Paxos protocol for distributed consensus) and ZooKeeper (using ZAB protocol). Here is a good article that summarizes the protocols. As for the exam we only need to use etcd with the client tool.

The etcd itself can run on a cluster of servers, each running etcd as a systemd service as etcd/etcd (user/group). It can be deployed in two ways:

stacked etcd: an instance of etcd lives with kube-api-server on the same control plane node
external etcd: in a dedicated cluster of etcd

Alternatively, etcd can run as a pod, most likely in kube-system namespace.

The etcd service listens on port 2379 for client communication and on port 2380 for server (peer-to-peer) communication. When the systemd service was initialized there are a few key environment variables (e.g. cert locations, ETCD_DATA_DIR) privoded as configuration. To see them, run:

> cat /etc/systemd/system/etcd.service | grep Env

These environment variables (prefixed with ETCD_) are for the service only. They can provide current configuration information for us to use later. When it’s running as a pod, check out the directory for static pod for the yaml declaration (e.g. /etc/Kubernetes/manifests/etcd.yaml), where these parameters are passed in as environment variable.

The etcdctl utility is a command line client for etcd. The default API version is 3 so no need any more to set ETCDCTL_API=3 before each command. The utility needs three arguments three arguments (–cacert, –cert, and –key) but we can pass the information via environment variables:

> export ETCDCTL_CACERT= /home/cloud_user/etcd-certs/etcd-ca.pem
> export ETCDCTL_CERT= /home/cloud_user/etcd-certs/etcd-server.crt
> export ETCDCTL_KEY= /home/cloud_user/etcd-certs/etcd-server.key
> export ETCDCTL_ENDPOINTS=https://etcd1:2379

The environment variable names are uppercase of the argument name with prefix ETCDCTL_. Only global options of arguments can be supplied via environment variables. They remain effective throughout the rest of activities. Also note that the CACERT is needed only when client-cert-auth is true. Now to backup, we can simply run:

> etcdctl snapshot save /home/cloud_user/etcd_backup.db

To restore from a file, you want to remove existing etcd data directory first. The directory can be found in ETCD_DATA_DIR variable. Suppose it is /var/lib/etcd, you need root permission to write to it, then correct ownership before starting the service:

> sudo systemctl stop etcd && sudo mv /var/lib/etcd/ /tmp/
> sudo etcdctl snapshot restore /home/cloud_user/etcd_backup.db --data-dir /var/lib/etcd
> sudo chown -R etcd:etcd /var/lib/etcd && sudo systemctl start etcd

To verify the restore result, simply run:

> etcdctl get cluster.name

Object Management

In the CKA exam, we need to interact with many types of built-in Kubernetes objects.

RBAC objects:
- A Role defines permissions within namespace.
- A ClusterRole defines cluster-wide permissions.
- Both Roles and ClusterRoles are K8s objects that defines a set of permissions
- RoleBinding and ClusterRoleBinding are objects that connect Roles and ClusterRoles to users.
Service Account: an account used by container processes within Pods to authenticate the K8s API. If your Pods need to communicate with the K8s API, you can use service accounts to control their access.

Inspect resource usage either with a K8s Metrics Server, or by command:

> kubectl top pod --sort-by <JSONPATH> --selector <selector>

Here is a good guide to install metrics server and dashboard (e.g. on docker-desktop).

Pods and Containers

ConfigMaps: store data in key-value map.
Secrets: same as ConfigMaps but for sensitive data only
Two ways to pass ConfigMap and Secret data to your container:
- As environment variables in container operating system
- As files presented on mounted volumes in container file system.
Container Resource management:
- Resource requests: K8s scheduler will use resource requests to avoid scheduling pods on nodes that do not have enough available resources. 1 CPU unit = 1/1000 of one core
- Resource limits: allow you to limit the amount of resources your containers can use. The container runtime is responsible for enforcement. The enforcement behaviour is different. For example, some terminates container that attempts to use more resource than the limit.
Probes
- Liveness Probe: automatically determine whether or not a container application is in a healthy state. By default K8s does not consider a container to be down until the container process stops. Liveness Probe allow you to customize this detection mechanism and make it more sophisticated.
- Startup Probes: similar to liveness probes. However, while liveness probes run constantly on a schedule, startup probes run at container startup and stop running once they succeed. Startup probes are used to determine when the application has successfully started up. It is especially useful for legacy applications that can have long startup times.
- Readiness Probes: determine when a container is ready to accept requests. When you have a service backed by multiple container endpoints, user traffic will not be sent to a particular pod until its containers have all passed the readiness checks defined by their readinesse probes. Use readiness probes to prevent user traffic from being sent to pods that are still in the process of starting up.

Restart policy for self-healing pods
- (default) Always: container will always be restarted if they stop, even if they completed successfully (returned 0).
- OnFailure: container will be restarted if the container process exists with an error code, or the container is determined to be unhealthy by a liveness probe.
- Never: let it be
Multi-container pods:
- containers share the same networking namespace and can communicate with one another on any port, even if the port is not exposed to the cluster
- Container can use volumes to share data in a Pod. Example: a legacy application is hard-coded to write log output to a file on disk. You use a sidecar container to read the log file from shared volume and prints it to the console so the log output will appear in the container log.
Init containers: containers that run once during the startup process of a pod. A pod can have any number of init containers, and they will each run once into completion, before the next init container starts. You may use init containers to perform a variety of startup tasks, they can contain and use software and setup scripts that are not needed by your main containers. They are often useful in keeping your main containers lighter and more secure by offloading startup tasks to a separate container. Use case include:
- cause a pod to wait for another K8s resource to be created before finishing startup
- perform sensitive startup steps securely outside of app containers
- populate data into a shared volume at startup
- communicate with another service at startup
Scheduling: Scheduler (a component in control plane) assigns Pods to a suitable Node so kubelets can run them. The factor taken into account:
- resource request vs available node resources
- various configurations that affect scheduling using node labels
Pod allocation
- nodeSelector is an attribute of Pod to allow you to limit which Node(s) the Pod can be scheduled on. The selector is based on label.
- nodeName is an attribute of Pod that allows you to bypass scheduling and assign Pod to a specific Node by name.
DaemonSet: automatically runs a copy of a Pod on each node. When a new node is added to the clsuter, DaemonSet will run a new copy of the Pod on it. DaemonSets also respect normal scheduling rules around node labels, taints and tolerations. If a pod would not normally be scheduled on a node, a DaemonSet will not create a copy of the Pod on that node.
Static Pod: A Pod that is managed directly by the kubelet on a node, not by the K8s API server. They can run even if there is not K8s API server present. Kubelet automatically creates static Pods from YAML manifest files located in the manifest path on the node.
Mirror Pod: Kubelet will create a mirror Pod for each static Pod. Mirror Pods allow you to see the status of the static Pod via the K8s API, but you cannot change or manage them via the API.
Taints: applied to nodes to repel a set of pods. A taint specifies key-value and effect. Effect can be NoSchedule or NoEffect. The former prevents pods without matching tolerations to schedule to the tainted node. The latter also evicts pre-existing pods with no matching toleration.
Tolerations: applied to pods so they can be scheduled to nodes with matching taints. A toleration consists of key-value pair, effect and operation. The operation can be Equal or Exists. To determine whether a toleration matches a taint. The keys and the effects must be the same. In addition:
- the operator is Exists (and thus no value should be specified in the toleration); or
- the operator is Equal, and all the values match those of the taints;

Now we have three ways to influence the scheduling behaviour. The first, is simply by specifying nodeSelector on the Pod, with the required the node label. The second, as just discussed, is to use Taints and Tolerations. The third way, is similar to the first, using nodeAffinity attributes on Pods. nodeAffinity is more powerful and flexible than nodeSelector by supporting more complex scheduling rules (e.g. matching rules).

Use Node Affinity when your scheduling rule is based on direct condition, i.e. schedule a Pod to this Node when XXX. In this case, you have well-known labels on nodes, and specify nodeAffinity on Pods.
Use Taints and Tolerations when your scheduling rule is based on inverse statement, i.e. do not schedule a Pod to this Node unless XXX. In this case, you put a taint “MyCondition:NoSchedule” on a Node, so that no Pod will ever get scheduled to this Node. The only exception is when a Pod has the Toleration “MyCondition:NoSchedule”.

Deployments

Deployment is an object that defines a desired state for a ReplicaSet (a set of replica Pods). The Deployment Controller seeks to maintain the desired state by creating, deleting, and replacing Pods with new configurations.
With Deployments, you can horizontally scale an application up and down by changing the number of replicas. You can perform rolling updates and rollback.

Networking

The K8s network model defines how Pods communicate with each other, regardless of which Node they are running on.
Each Pod has its own unique IP address within the cluster. Any Pod can reach any other Pod using that Pod’s IP address. This creates a virtual network that allows Pods to easily communicate with each other.
One type of K8s network plugin is CNI plugin. It has many flavours such as Calico. Each plugin has its own unique installation process. Kubenetes nodes will remain NotReady until a network plugin is installed.
The K8s virtual network uses a DNS (e.g. a Kubeadm cluster uses CoreDNS pod in kube-system namespace) to allow Pods to locate other Pods and Services using domain names. The Pod DNS name follows this format: pod-ip-address.namespace.pod.cluster.local
A K8s NetworkPolicy is an object that allows you to control the flow of network communication to and from Pods so you can isolate traffic. NetworkPolicy can apply to Ingress (using from selector), Egress (using to selector) or both.
NetworkPolicy has an attribute podSelector to determine to which Pods in the namespace the NetworkPolicy applies, by selecting Pods by with Pod labels.
By default, Pods are considered non-isolated and completed open to all communication. If any NetworkPolidy selects a Pod, the Pod is considered isolated and will only be open to traffic allowed by NetworkPolicies.
A variety of selector can be used: podSelector, namespaceSelector, ipBlockSelector and port.

Services

Services provide a way to expose an application running as a set of pods, so clients can access applications in an abstract way without needing to be aware of the application pods. In this model, client make requests to a Service, which routes traffic to its pods in a load-balanced fashion
Endpoints are the backend entities to which Services route traffic. If there are multiple Pods behind a service, each Pod will have an endpoint associated with the service.
Each service has a type that determines how and where service will expose your application.
- ClusterIP: expose application inside the cluster network
- NodePort: expose application outside the cluster network
- LoadBalancer: expose application outside thecluster network, but use an extermal cloud load balancer from cloud platform.
Services are assigned with DNS names. The FQDN follows this format: service.namespace.svc.cluster-domain.example, which is used by pods across namespaces
Pods within the same namespace can reference service simply by service name.
To manage external access to service, you can also use Ingress object. Ingress object is capable of providing more functionality than a simple NodePort Service, such as SSL termination, advanced load balancing, or name-based virtual hosting. You must install one or more Ingerss controller (many different implementations) to back up the ingress objects.
Ingress defines a set of routing rules. Each rule has a set of paths, each with a backend. Requests matching a path will be routed to its associated backend.
If a Service uses a named port, an ingress can also use the port’s name (instead of port number) to choose to which port of a service it will route.

Storage

Volumes allow you to store data outside the container file system, while allowing the container to access the data at runtime. When Pod is gone, volumes do not persist.
Persistent Volumes are a slightly more advanced form of Volume. They allow you to treat storage as an abstract resource and consume it in Pods. PV can be provisioned separately by storage administrator, and they persist regardless of pod lifecycle. PV needs to be claimed by pods. PV uses a set of attributes to describe the underlying storage resource.
Both volumes and PVs each have a volume type: NFS, Cloud (AWS, Azure, GCP), ConfigMaps and Secrets, Simple Directory on node
Two volume types to distinguish:
- hostPath: store data in a specified directory on K8s node
- emptyDir: store data in dynamically created location on the node. The directory exists only as long as the Pod exists on the node. The directory and the data are deleted as Pod is removed. This type is useful for simply sharing data between containers in the same pod.
Both volumes and PVs are specified under Pod, and individual containers must include volumeMounts object to map volume name to local mountPath
Storage Class object allow K8s admins to specify the types of storage services they offer on their platform. A key property is allowVolumeExpansion. This allows PVC to resize. At storage class level, there are two reclaim policies: Retain and Delete. The default is Delete.
PV has an attribute named persistentVolumeReclaimPolicy. This is reclaim policy at PV level. If the attribute is not defined, it is inherited from storage class. The persistentVolumeReclaimPolicy has three options. When PVC is deleted:
- Retain: keeps all data but requires admin to manually reclaim the volume (i.e. delete PV, clean up data, delete storage asset)
- Delete (cloud storage only): deletes both PV and the underlying storage resource automatically
- Recycle: scrub (rm -rf /vol/) all data in the underlying storage resource, and allow the volume to be reused.
PVC represents a user’s request for storage resources. It defines a set of attributes similiar to those of a PV. When a PVC is created, it will look for a PV that is able to meet the requested criteria. If it finds one, it will automatically be bound to the PV. PVC can be mounted to a Pod’s containers just like any other volume

In general the CKA exam experience is quite positive and rewarding. In future posts I will shift focus on Kubernetes not only for the CKA exam, but also for keeping track of my learning.

Good luck with your CKA exam.

Basic Resource Object in Kubernetes 1 of 2

Sat, 16 Jan 2021 22:13:00 -0400

For someone from a system administration background, it would be amazing to discover that Kubernetes provides a solution to every pain point in the traditional software deployment landscape. On the contrary, it also brings about a lot of complexity due to the types of resource objects introduced.

Pod

A Pod is a shared execution environment for one or more containers. The containers running in a Pod share resources such as memory, volumes, network namespace (e.g. IP address, port range, hostname, routing table), UTS namespace (e.g. hostname) and IPC namespace (Unix domain sockets). Every Pod has its own IP address that is routable on the Pod network. All Pods connect to the same flat network called the Pod network.

A pod most commonly only contains a single container, which is considered a good practice, unless there is good reasons to put two containers in a single pod (sharing resource). One such good reason is to co-schedule tightly-coupled workloads (such as logging, sharing volume, etc). Within the Pod, the containers communicate with each other via localhost interface of the Pod. In service mesh model, there is also a proxy container in each application Pod. The proxy container handles all network traffic entering and leaving the Pod. Also, within the Pod, to avoid competing for resources, individual containers can have their own cgroup limits, which actively police resource usage.

Pods are mortal (composable). They come and go (with dynamic IPs), so application should not store state in Pods. Deploying a Pod is an atomic (all or nothing) operation. When a Pod is scheduled to a node, it enters the pending state while the container runtime on the node downloads images and starts any containers. Once’s everything is ready, the Pod enters the running state.

We typically deploy Pods via higher-level controllers such as Deployments (to offer scalability and rolling updates), DaemonSets (to run one instance of a service on every node in the cluster), StatefulSets (for stateful application components), and CronJobs (for short-lived tasks that need to run at set times just like a Linux cronjob).

Deployments

Deployment manages multiple replicas of the same Pod (via ReplicaSets). To follow best practice, you interact with Deployments instead of ReplicaSets, and use YAML file (declarative model). You can perform rolling update or rollback.

ReplicaSets

ReplicaSets provide self-healing and scaling capabilities to Pods. If a Pod fails, it will be replaced. If load increases, then the ReplicaSets creates new Pod. This is all implemented with a background reconciliation loop that is constantly checking whether the right number of Pod replicas are present on the cluster. If not, Kubernetes declares a red-alert condition, orders the control plan to bring up more replicas. The best practice however, is that you should not manage ReplicaSets directly. Instead, you should perform all actions against the Deployment object and leave the Deployment to manage ReplicaSets.

Service

Pods themselves are mortal (IP churn) so it’s a bad idea to talk directly to individual Pods. Service object provides stable and reliable networking for a set of dynamic Pods. Service gets its own stable IP address, stable port and stable DNS name. It can also load-balance request across the Pods.

Services are loosely coupled with Pods via labels and label selectors. You specify label selector for Service and labels on Pods when creating them. All the labels in label selector are used to select target Pods. Service acts as front-end, consisting of stable IP, DNS name and port, with Pods acting as backend, consisting of constantly changing Pods. Labels are simple yet extremely powerful. During blue-green update, you may use version label as a technique to control what backend pool is used behind Service object. For example, start with version=1, deploy version 2, remove version from label selector, and eventually add version=2 back to label selector, before phasing out the old Deployment.

Services learn Pod status via Endpoint object, more details to follow.

There are several types of Service, the default being ClusterIP. A ClusterIP Service has a stable IP address and port that is only accessible from inside the cluster. The ClusterIP gets registered against the name of the Service on the cluster’s internal DNS service (implemented via coreDNS with Control plane Pods). This means that the ClusterIP only works within the cluster, not outside. The other type of Service is called a NodePort, which is built on top of ClusterIP, but also enables access from outside of the cluster. The Service object has a reliable NodePort mapped to every node in the cluster. The NodePort value is the same on every cluster. Traffic from outside of the cluster can hit any node in the cluster on the NodePort and get through the the Pods.

Other types of Services include LoadBalancer and ExternalName. LoadBalancer Services integrate with load-balancers from cloud provider. They build on top of NodePort Services and allow clients on the internet to reach your Pods via the load balancer of cloud vendor. ExternalName Services route traffic to systems outside of your K8s cluster.

For service discovery within the cluster, Kubelet program every container with the knowledge of the internal DNS (/etc/resolv.conf). The internal DNS service watches constantly the API server for new Services and automatically register them in the DNS. The other means of service discovery is through environment variables. However, in this method the Pods have no way of learning about new Services added to the cluster after the Pod itself is created.

Endpoints

Endpoints object is a dynamic list of all the healthy Pods on the cluster that match the Service’s label selector. Each Service gets its own Endpoints objects for an up-to-date list of matching Pods. Kubernetes is constantly evaluating the Service’s label selector against the currently list of healthy Pods on the cluster. Any new Pods that match the selector get added to the Endpoints object, and any Pods that disappear get removed.

When sending traffic to Pods, via a Service, an application will query the cluster’s internal DNS for the IP address of a Service, then sends the traffic to this stable IP address. Service then forwards it on to a Pod. Kubernetes-native application however, has the ability to query the Endpoints API directly, bypassing the DNS lookup and use of the Service’s IP.

It requires a thorough understanding of Services, Endpoints and the service discovery mechanism to perform effective troubleshooting in Kubernetes.

The aforementioned internal DNS service (we usually call it the “cluster DNS”) is implemented in the kube-system Namespace as a set of Pods managed by a Deployment called coredns. These Pods are fronted by a Service called kube-dns. The cluster DNS is constantly looking for new Services and automatically register their details (metadata.name). We might need to check the logs for each of the coredns Pods during troubleshooting.

The kubelet process on every node is watching the API Server for new Endpoints objects, when it sees them, it creates local networking rules that redirect ClusterIP traffic to Pod IPs, using IPVS technology on Linux to manage these rules.

DaemonSet

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

Some typical uses of a DaemonSet are: cluster storage daemon on every node, logs collection daemon on every node, a node monitoring daemon on every node.

Horizontal Pod Autoscaler

The Horizontal Pod Autoscaler automatically scales the number of Pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics). Note that Horizontal Pod Autoscaling does not apply to objects that can’t be scaled, for example, DaemonSets.

The Horizontal Pod Autoscaler is implemented as a Kubernetes API resource and a controller. The resource determines the behaviour of the controller. The controller periodically adjusts the number of replicas in a replication controller or deployment to match the observed average CPU utilization to the target specified by user.

There are more details about HPA here and here.

StatefulSets are designed for stateful application, which creates and saves valuable data. The three properties that form the state of a Pod are:

Pod names (<StatefulSetName>-<Integer>)
DNS hostnames
volume bindings

They are sometimes referred to as the Pods sticky ID. StatefulSets ensures that these are all predictable and persistent. For example, failed Pods managed by a StatefulSet will be replaced by new Pods with the exact same Pod name, the exact same DNS hostname, and the exact same volumes, even if the replacement Pod is started on a different cluster Node.

Note that StatefulSets create one Pod at a time, and always wait for previous Pods to be running and ready before creating the next. Scaling operations are also governed by the same ordered startup rules. This is different from Deployments that use a ReplicaSet controller to start all Pods at the same time, causing potential race conditions. The way StatefulSet controllers do their own self-healing and scaling is architecturally different to Deployments which use a separate ReplicaSet controller for these operations. The reason it is a game changer to know the order in which Pods will be scaled down, as well as that Pods will not be terminated in parallel, is because clustered apps that store data are usually at high risk of losing data if multiple replicas go down at the same time.

Deleting a StatefulSet does not terminate Pods in order. So you may want to scale a StatefulSet to 0 replicas before deleting it. You might also set 10 seconds grace period before terminating to allow applications a chance to flush local buffers and safely commit any writes still in flight.

In Kubernetes, Volumes are decoupled from Pods via PersistentVolumes and PersistentVolumeClaims. So volumes have separate lifecycles to Pods and can survive Pod failures and termination operations. When a StatefulSet Pod is created, any volumes it needs are created at the same time and named in a way to connect them to the right Pod. Any time a StatefulSet Pod fails or is terminated, the associated volumes are unaffected. This allows replacement Pods to attach to the same storage as the Pods they’re replacing, even if the replacement Pod is scheduled to a different cluster Node. Similarly, if a StatefulSet Pod is detected as part of a scale-down operation, subsequent scale-up operations will attach new Pods to the existing volumes that match their names.

Since each StatefulSet Pod needs its own unique storage, hence its own PVC, this can be done by volumeClaimTemplate, which dynamically creates a PVC each time a new Pod replica is dynamically created. This eliminates the hassle to have to pre-create a unique PVC for every potential StatefulSet Pod.

Namespaces

Namespaces allows you to partition resource objects. For example, you may create a Namespace called prod and dev. Object names must be unique within Namespaces but not across Namespaces.