Cluster on Digi Hunch

Kubernetes Platform as a Service and Red Hat OpenShift

Sun, 25 Jun 2023 11:10:15 -0400

The Three-layer model

Kubernetes is so complex that it becomes a buzz word itself. I categorize the related work into three layers: a cluster layer, a platform layer and an application layer, by their purposes. The three layers are illustrated as below:

Let’s examine each layer in this model and where the Kubernetes Platform as a Service fits in.

The Kubernetes Cluster Layer

At the bottom, the Kubernetes Cluster layer is the foundational layer. It focus on using self-hosted VMs or cloud resources to build a functional Kubernetes cluster and worker node groups. A functional cluster includes a highly available control plane, as well as scalable node groups that all communicate with the control plane. Cloud Service Providers like AWS and Azure provides managed Kubernetes service, which takes away the complexity (and flexibility as well) of managing control plane components such as etcd store and API server. The managed services also automatically provisions computing nodes and join them into the cluster. The cluster layer may also involve integration with of CNI and CSI, to ensure Pod-to-Pod communication and available storage classes.

Professionals working at this layer are infrastructure experts who understand networking, storage, as well as how to manage cloud resources or VMs, infrastructure as code. On a daily basis, they deal with VPCs/V-Nets, subnets, EBS/Azure Disk, File storage, EC2/Azure VMs, etc. When the team is doing a bad job at this layer, you might see symptoms like unresponsive cluster API, orphaned worker nodes, or kubectl failing to connect to cluster endpoint.

The tenants (applications) of the Kubernetes platform does not directly interact with this layer. If you decide to switch CSP vendor, this layer requires 100% re-engineering because the managed Kubernetes service by each CSP is different.

The Kubernetes Platform Layer

The Platform layer sits in the middle. When organization decides to adopt Kubernetes, they often underestimate the efforts required in this layer. This layer works on a functional cluster, without directly interacting with the underlying cloud resources. This layer involves any Kubernetes abstractions that do not creates tangible business value. Rather, this layer is an enabler. It allows the applications to deploy smoothly, evolve quickly, and more importantly, focus on the business.

Teams working on this layer needs to be Kubernetes experts. On a daily basis, they play with common CNCF toolings, such as Prometheus, ArgoCD, Istio, Cilium, Tekton, Open Policy Agent, etc. They are comfortable with Operators, Helm Charts, Ingress, etc. Inside of the Kubernetes cluster, they also manage the foundational services such as Event streaming (e.g. Kafka), PostgreSQL database (e.g. PostgreSQL), software-defined storage (e.g. Ceph), service mesh (e.g. Istio), Authentication (e.g. Keykloak) , etc. These services act as the infrastructure layer to the business workload. If the team is doing a bad job, you would see data loss with database, observability service not populating data, ingress does not process request, etc.

The tenants (application) share services in this layer. If you decide to switch CSP vendor. I estimate 80% of the work at this layer is portable, and 20% requires re-engineering. That is because each CSP offers different external resources, therefor the low level Kubernetes objects in this layer, such as storage classes, load balancers, supported CNIs are different. High level objects such as Kafka remains portable across platforms.

The Application Layer

The next layer at the top is application layer. Workloads in this layer are directly linked to the business value. The applications are very diverse. Most of the time, the release team is the main player at this layer. If the organization develops its own application, the software development team also work at this layer.

In terms of knowledge, the members of development team are experts in software engineering, and Software Development Life Cycle (SDLC), etc. On a daily basis, they deal with programming languages, product development, build and release. If they screw up their work, expect business errors, such as orders sent to wrong client, incorrect balance sheet, etc. This team has high visibility in the organization due to its direct link to business value.

This layer of work involves multiple tenants. Each tenant is isolated within their own namespace. When you switch CSP vendor, this layer should be readily portable with minimal effort.

It is also worth noting that, with solid platform and cluster layers, the team working at this layer do not write bespoke code for networking, observability, authentication and authorization, encryption and many other aspects not relevant to the core business. Once deployed, the application services are resilient, scale to demands, and cost efficient. This layer reaps the benefits of Kubernetes.

Kubernetes Platform as a Service

As the Kubernetes dust is still settling, a builder’s title may not always reflect which layer she or he focuses on. Today it is pretty common for infrastructure engineers to expand their role into the platform layer, or likewise, a software engineer to drill down to the platform layer.

The boundary between platform layer and cluster layer is clear. The cluster layer deals with underlying infrastructure, either in the cloud or on premise. They abstract away the complex infrastructure world from those working with the platform layer.

The boundary between platform layer and application layer is a little tricky to articulate. The application layer focuses on implementing the business logics. The platform layer takes care of the functions that are not part of business logic but essential to the business application. Take an HTTP request for example, application developer should not have to write code to terminate TLS (not part of business logic). They should only write the code to process the HTTP request (business logic). TLS termination is delegated to an Ingress, to be configured by platform builders.

The folks working at the Platform layer needs to interface with both sides. They provide Platform as a Service to the Application teams. However, their work appears mostly invisible in an organization. Their effort is oftentimes underestimated. There are several reasons for that. First, the platform layer does not directly create tangible business value. They are just someone else’s enabler. Second, their building blocks involve a lot of abstractions by Kubernetes API. Third, the idea of platform engineering is newly emerged. There hasn’t been a populous recognition of its value.

Red Hat OpenShift Container Platform

The platform team builds the platform with their choice of open-source tools. For clusters using OpenShift Kubernetes Engine, Red Hat introduces Open Shift container platform consisting of Red Hat’s opinionated (but validated) choice of toolings, for example:

OpenShift Service Mesh: Istio
OpenShift Streams: Apache Kafka
OpenShift GitOps: ArgoCD
OpenShift Container Platform Pipelines: Tekton
OpenShift Serverless: Knative
OpenShift Data Foundation: Ceph

Clients building their clusters with OpenShift Kubernetes Engine may build their own platform with the toolings in the OpenShift enterprise Kubernetes container platform. For more services, check out the documentation for OpenShift Container Platform. For customers with OpenShift Kubernetes Engine, their options to DIY platform are:

Entry-Level: Red Hat OpenShift Kubernetes Engine: Enterprise Kubernetes distribution on RHEL CoreOS
Mid-Level: Red Hat OpenShift Container Platform (RHOCP):
Plus-Level: Red Hat OpenShift Platform Plus: RHOCP + advanced cluster management, security, data management essentials, enterprise container registry

OpenShift runs the business model of Kubernetes PaaS.This is a unique business model that I do not find a matching competitor. Even if you choose to DIY your own platform, the Red Hat’s choices are still a great reference. The OpenShift enterprise Kubernetes container platform maps perfectly to the platform layer of the three-layer model, aiming to simplify the work in the platform layer.

Managed RedHat OpenShift

At first, the OpenShift container platform started as a value add-on to the Kubernetes Engine. Now it’s a separate product line in their business model. In the mean time, OpenShift partners with major CSPs, to develop the cloud service editions, including:

Red Hat OpenShift on AWS (ROSA)
Microsoft Azure Red Hat OpenShift (ARO)
Red Hat OpenShift Dedicated – on AWS and GCP
Red Hat OpenShift on IBM Cloud

These offerings are managed Kubernetes Platform as a Service in the cloud. Since RedHat is the only player in this model, we can refer to them as managed OpenShift services. In addition to an already-confusing world of Kubernetes platform portfolios, these offerings gives consumers even more options. On AWS for example, users have the following options:

Managed Platform: OpenShift Dedicated, managed by Red Hat
Managed Platform: Red Hat OpenShift Service on AWS (ROSA), managed by Red Hat and AWS
Self-built cluster: OpenShift Container Platform

This video discussed more details about these options, such as support model. It is also worth noting that these options tend to be much pricier than managed clusters such as EKS and AKS.

Since a Managed RedHat Platform makes it easy to deploy, let’s take ROSA as an example and create a cluster. To enable ROSA in AWS console, click on “Getting Started”. The next page ensures ROSA is enabled and checks other prerequisite such as meeting service quotas and creating ELB service-linked role, as show below:

Now, with an AWS account (and ROSA enabled), a RedHat account, and the rosa-cli utility, we can create a cluster with just a few commands. As a note, be wary of the cost and do not forget to delete the cluster afterwards.

Create a ROSA cluster

With the following set of commands, we can kick off cluster creation, using STS. We can bring our own VPC, so long as it meets certain prerequisites. I use the Terraform template in the vpc-base project, to create the underlying VPC. We’ll need the followings from this template:

The CIDR range of the VPC: as input with a default
The subnet Ids of the private subnet to place, printed in the output

The subnets are private subnets, because we want to provision the cluster with private node and private endpoint. When we use rosa CLI, we provide the CIDR and subnet IDs.

# start with AWS cli configured to the correct profile
rosa login # with redhat account and past token
rosa create account-roles --mode auto -y # this command creates the IAM roles ManagedOpenShift-*-Role, with RedHat account as trust entity
rosa verify permissions # optional
rosa verify quota # optional

export ROSA_CLUSTER_NAME="dhc" \
OPENSHIFT_VERSION=4.13.4 \
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) \
AWS_DEFAULT_REGION=us-east-1

rosa create cluster --sts --private \
 --cluster-name $ROSA_CLUSTER_NAME \
 --multi-az \
 --private-link \
 --region $AWS_DEFAULT_REGION \
 --version $OPENSHIFT_VERSION \
 --enable-autoscaling \
 --min-replicas 3 \
 --max-replicas 3 \
 --compute-machine-type m5.xlarge \
 --machine-cidr 147.206.0.0/16 \
 --subnet-ids subnet-052852a1fb4d7d2ad,subnet-06d8d40ae39d55c47,subnet-0f67ce08bc588012c

The CLI will pick up the correct VPC by CIDR, and prompt you to confirm creation of private cluster. After the command kicks off, it will wait for OIDC provider creation, and role creation. Then it uses a Terraform template to create the related resources including VPC. Use this command to check status:

rosa list clusters
rosa describe cluster -c dhc

When the second command displays the state of waiting (Waiting for OIDC configuration), we can create OIDC provider:

rosa create operator-roles -c $ROSA_CLUSTER_NAME --mode auto --yes
rosa create oidc-provider -c $ROSA_CLUSTER_NAME --mode auto --yes

Throughout the process, we can monitor the install log (terraform output) with:

rosa logs install -c dhc --watch

In the log, you might see errors with terminals connecting to the terraform backend, which doesn’t necessarily indicate a cluster creation error. Always check the cluster state until it reports success.

Kick the tires

Eventually the describe cluster command will show ready state. We can now create an admin user:

rosa create admin -c $ROSA_CLUSTER_NAME

The command above prints an oc command (OpenShift CLI, equivalent to kubectl) with password to log in. Let’s examine the cluster with oc. Because it is a private cluster, the endpoint is not available publicly. However, it is accessible from the Bastion host. Use the SSM Session Manager technique from my previous post to SSH to the Bastion Host, which should have oc installed. To install oc yourself, use HomeBrew on Mac. On Linux or Windows, log on to OpenShift console, go to Downloads on the left pannel and find it out under CLI tools.

The oc command may report insecure TLS on the login URL. Wait for a few minutes for the certificate to come off as safe. Once you run the oc command with password, it should return “Login successful” and then we can connect to the cluster:

$ oc get node # or kubectl get node
NAME STATUS ROLES AGE VERSION
ip-147-206-135-41.ec2.internal Ready,SchedulingDisabled infra,worker 3m5s v1.26.5+7d22122
ip-147-206-155-141.ec2.internal Ready control-plane,master 25m v1.26.5+7d22122
ip-147-206-156-81.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-164-21.ec2.internal Ready infra,worker 3m3s v1.26.5+7d22122
ip-147-206-179-90.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-191-118.ec2.internal Ready,SchedulingDisabled control-plane,master 26m v1.26.5+7d22122
ip-147-206-192-232.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-193-198.ec2.internal Ready infra,worker 3m20s v1.26.5+7d22122
ip-147-206-218-114.ec2.internal Ready control-plane,master 26m v1.26.5+7d22122

You can use oc the same way you’d use kubectl. Both works through SOCK5 proxy. In the meantime, log in to the RedHat console with your Red Hat credential, you can see the cluster in Ready state as well:

The rosa create admin command creates a htpasswd type (username-password) of identity provider (IdP) with a user named cluster-admin and a preset password. In real life however, we often configure third party IdP with OIDC integration. I’ll have to leave this to the next blog post.

We shall see the nodes as EC2 instances from AWS console as well. Note that there are three roles: control-plane, worker and infra. The infra nodes are for infrastructure services. These services (Ingress Controller, GitOps, Pipeliens) are the ones in the platform player as we discussed above.

There are many customizations you can make in this installation process and I’d have to defer to the ROSA documentation. To clean up, use the following ROSA command:

rosa remove cluster -c dhc

The output also gives you the command to delete operator roles and OIDC provider, for example:

rosa delete operator-roles -c 23o4u3j98tqmlbtjo612opb7a4bbim5f --mode auto --yes
rosa delete oidc-provider -c 23o4u3j98tqmlbtjo612opb7a4bbim5f --mode auto --yes

Then we can destroy the VPCs using terraform.

ROSA with HCP

Update Oct 2023:

The deployment above provisioned a few nodes for control plane, which add to the overall time to provision a cluster. In Aug 2023, there is a new option Hosted Control Plane (HCP) that came to allow users to provision a hosted control plane. This results in cost savings and shorter time to provision a cluster. Here is a table of comparison between the ROSA with HCP and ROSA classic.

Final words

In this post, I discussed the three-layer model and pointed out that platform layer isn’t as visible as the other two. I also experimented ROSA as a turn-key Kubernetes platform with its opinionated stack of services.

Some misinformed organizations even skip the entire platform layer in their estimate of effort. They build a cluster, ran a hello-world service and assumes they can start putting applications on the Kubernetes cluster. There are also customers who purchased the entire Managed OpenShift platform but only use it as a cluster. Yikes!

The concept of Kubernetes platform, or generally platform engineering is still spreading. The consulting team that I worked in full-time last year re-branded itself as platform engineering. Marketings are pushing it. Builders are doing it. We’ll keep an eye, on whether customers are buying it.

EKS impression

Fri, 23 Dec 2022 18:18:19 -0400

I’ve worked on a few AKS projects previously. Since I joined AWS I wanted to put aside some time to check out EKS (Elastic Kubernetes Service). Here in this post, I put down my first impression on EKS, and also share my Terraform template in cloudkube project to create an EKS cluster.

Similar to AKS, EKS exposes API endpoint and the control plane components are hidden from AWS users. When creating EKS cluster it does not create the underlying VPC and subnets. Therefore, you have create an existing VPC and at least two subnets ahead of time, and specify them during EKS creation. Bear in mind that there is a list of requirement for the VPC and subnets.

In the cluster, the CNI that EKS officially supports is Amazon VPC CNI plugin. It is available as an add-on. Similar to Azure CNI, each Pod gets its own IP address. In addition, EKS supports other compatible CNI plugins such as Calico, Cilium, Weave Net and Antrea.

Computing Nodes in EKS

There are three modes to address computing capacity: self-managed nodes, EKS managed node groups and AWS Fargate. The documentation has a comparison table.

With self-managed nodes, users create EC2 instances separately and then register them to the control plane. The instances must use the same IAM role and AMI. You can use Auto Scaling groups of Bottlerocket (AWS-sponsored purpose-built Linux distro for container host) nodes. The self-managed node option is mostly for AWS outpost customers who bring in their own computing capacity from data centre.

If you provision computing capacity from AWS, it makes sense to assign EKS managed node groups when creating EKS cluster. We can turn on Cluster Autoscaler, a Kubernetes construct to manage the auto scaling of node groups. Sometimes we want to have more than one node groups. For example, to build a multi-architecture cluster, we need one node group with amd64 nodes and the other with arm64 nodes (e.g. instances with Graviton processor). In general, arm-based CPU delivers better performance with less power consumption and the industry is slowly moving towards more arm-based CPU architecture.

Fargate is what I call managed computing service for EKS. With Fargate you do not need to tweak Cluster Autoscaler to self-manage computing capacity. The Fargate documentation has a long list of considerations. For example, Pods must match a Fargate profile (here‘s an example) at the time that they’re scheduled to run on Fargate. So we need to build Fargate profile and Pod labelling properly. Also, Fargate does not support DaemonSet. Another big consideration is that Fargate does not support non-VPC CNI. In my opinion these are pretty significant limitations. Many workloads (system-level or application-level) would need Daemonset (e.g. kube-proxy, some CNI or CSI drivers, Dynatrace monitoring).

The pro of Fargate is the serverless computing model. The construct of a Fargate profile isn’t complicated. You just specify subnets, namespace and labels. However, the downside is the long list of considerations. Some teams may consider these restrictions too much. The other overhead is the need to manage Fargate profile to ensure all Pods are scheduled somewhere.

To me, using Fargate alone impairs portability of workload. The good thing is that Fargate and Managed Node Group are not mutually exclusive on a cluster. In most cases, we can go partially serverless, and reap the benefits of both of them.

Node AutoScaling

For workloads that don’t have a matching Fargate profile, we have to figure out node autoscaling ourselves. I touched on Cluster Autoscaler in “Autoscaling on Kubernetes Platform“. CA works on AWS as well and is triggered upon a Pod coming to unschedulable status in Scheduler. There is some limitations though. For example, CA interacts with Autoscaling Group (instead of EC2 instances directly). When it determines it’s time to scale up, it bumps up the desired capacity by one at a time in the Autoscaling group. The configurations in Autoscaling group may also be at play and CA do not have direct control. For example, the “scaling cooldown“. The pool of nodes is homogenous as per the pre-configured launch template and CA has no control. If a Pod requires a different type of node (e.g. ARM64 CPU, spot instance, etc), then we’d first have to create a node group with the desired node type. Moreover, in the worst cases, one-at-a-time scale-up does not meet the increase of demand driven by Pod increases, causing nuances such as racing conditions.

Because the Cluster Autoscaler doesn’t really deal with the nodes themselves, this kind of integration is clunky and slow. Nearly half of Kubernetes customers on AWS report that configuring cluster auto scaling using the Kubernetes Cluster Autoscaler is challenging and restrictive, according to this blog post. As a result, AWS launched an open-source cluster autoscaler project, Karpenter. Karpenter first only supported EKS but now the support includes other CSPs. For EKS, Karpenter directly interact with different types of EC2 instances.

Karpenter makes node scaling work in a more cloud-native manner. In the presence of unschedulable Pods, Karpenter bypasses the Kubernetes scheduler and works directly with the Cloud provider, to launch the minimal compute resources needed to fit those Pods and immediately binds the Pods to the newly provisioned Nodes without waiting for scheduler. As Pods are removed or rescheduled to other nodes, Karpenter looks for opportunities to terminate under-utilized nodes. Karpender defines a CR called Provisioner to specify node provisioning configuration, such as instance size, zone, CPU architecture, etc. It is a manifest that describes a node group so the node scaler is aware of all the available node types. You can have multiple Provisioners for different needs, just like node groups. The Provisioner CR can also set TTL for empty Nodes, such that once a Node has no pods other than DaemonSet, Karpenter will terminate the Node on TTL expiry.

Karpenter’s idea is similar to the idea of AutoPilot cluster in GKE. The new EKS workshop has an section on how to set up CA and Karpenter in practice.

Identity Management for EKS

For IAM, we need to be concerned with three aspects. The management traffic to the cloud service, the management traffic for Kubernetes cluster and business traffic.

Traffic type	AWS	Azure
I. Cloud Service Endpoint (Management Traffic for Cloud Service)	AWS IAM identity	Azure RBAC
II. Kubernetes API (Management Traffic for K8s Cluster)	IAM mapping or OIDC	Azure RBAC (implementation of OIDC)
III. Business traffic	Up to Kubernetes Ingress	Up to Kubernetes Ingress

For business traffic (type III), it is all up to the Ingress. I’ve written another article on managing ingress traffic on Kubernetes platforms. We interact with cloud service endpoint (type II) with either AWS CLI or Terraform, to create any object, including resources needed for a cluster. This is generally how we work with cloud service, not specific to Kubernetes. Usually the IAM identity assumes another IAM role, which empowers it with a lot of permissions.

For access to Kubernetes API (type III), EKS supports OIDC and IAM mapping. AWS documentation refers to this as “Cluster Authentication“. There is one special scenario where your identity for type II access inherits your identity for type I access. As the document puts:

When you create an Amazon EKS cluster, the AWS Identity and Access Management (IAM) entity user or role, such as a federated user that creates the cluster, is automatically granted system:masters permissions in the cluster’s role-based access control (RBAC) configuration in the Amazon EKS control plane. This IAM entity doesn’t appear in any visible configuration, so make sure to keep track of which IAM entity originally created the cluster.

This special scenario (I call it the “implicit master user“) allows us to perform critical activities on the cluster, such as creating IAM mapping, or OIDC configuration.

The above addressed how AWS resource access Kubernetes resource. On the other hand, to address how a Kubernetes resource access AWS resources, we need IRSA (IAM Roles for Service Account). We have a service account in Kubernetes and map it to an IAM role.

AppMesh

AppMesh is AWS’ Envoy based service-mesh offering supporting Kubernetes cluster, ECS service and even EC2 instance. AppMesh’s control plane is a managed AWS service, with a controller running on the Kubernetes cluster. To install AppMesh on the cluster:

On the EKS cluster, install AppMesh Controller using Helm
Associate the cluster with IAM OIDC provider
Create an IAM role for the appmesh-controller service account

After these steps, you can create a mesh using CloudFormation, Terraform, etc. The data plane (Envoy proxy) can run on Kubernetes (as sidecar). Traffic between control plane and data plane can go through private link (Interface VPC endpoint) for added security. Like Istio, AppMesh enables mTLS. For observability, you can export Envoy metrics with Prometheus. Coupled with XRay, AppMesh also supports distributed tracing.

AppMesh uses a different set of CRDs than Istio. Key CRDs are:

Mesh: represents an entire service mesh. At mesh level you can configure Egress filter (to allow or deny external traffic) and set IP version (v4 vs v6)
VirtualGateway: a CRD that represents an Ingress in to the Mesh. A virtual gateway allows resources that are outside of your mesh to communicate to resources that are inside of your mesh. A virtual gateway references Envoy proxy deployment by podSelector. It references GatewayRoutes by namespaceSelector, and optionally gatewayRouteSelector. You also specify listeners in the manifest to reference Envoy proxy Service (LoadBalancer Type).
GatewayRoute: A gateway route is attached to a virtual gateway and routes traffic to an existing virtual service. If a route matches a request, it can distribute traffic to a target virtual service. In the manifest, you specify a list of httpRoute, each with matching condition and action. In the action section you can specify virtualService as target.
VirtualService: an abstraction of a real service provided by a virtual node directly or indirectly by means of a virtual router. Dependent services call your virtual service by its virtualServiceName, and those requests are routed to the VirtualNode or VirtualRouter that is specified as the provider for the VirtualService.
VirtualRouter: Virtual routers handle traffic for virtual services. In a virtual router manifest, you can define Route to direct incoming requests to virtual nodes as target.
VirtualNode: A virtual node acts as a logical pointer to a particular task group (i.e. ECS service, Kubernetes deployment). It represent a Service in the AppMesh. In the manifest, you reference Pods by podSelector, specify listeners for any inbound traffic that your virtual node expects, and specify serviceDiscovery for your task group.

You can configure those Custom Resources using YAML manifests (and check the API reference a lot). Alternatively, you can configure them from AWS CLI or AWS console. The console will help you visualize what can be configured. For further details on how these CRs play together, there is a workshop for AppMesh.

EKS cluster using Terraform

Officially, there is an EKS blueprint project for provisioning EKS cluster in Terraform.

I also keep my own Terraform code in the AWS directory of cloudkube project. It works out to be a little more complex than my Terraform template to create Azure Kubernetes Cluster (Azure directory). Because I had to create Cognito resources with initial credential to allow users to connect to cluster without using the implicit master account.

Below is the diagram of the processes.

Create EKS cluster with Terraform module

The template configures kubectl access on a Bastion host, which assumed the same role that our IAM user uses to create the Kubernetes cluster. Therefore, the IAM role is the master identity.

Note that the IAM user (power-user) has very powerful permissions. Usually it is ideal to assign lots of permission to IAM Roles (temporary credential) instead of IAM user (long-term credential). So the role chaining would look like:

The IAM user that Terraform uses has no permission other than assuming a “PowerUser” role
The PowerUser role trusts the IAM user. It also has the permission to assume the “EKS-Manager” role
The EKS-Manager role trusts PowerUser’s role session.

However, the role chaining scenario above is not currently supported in Terraform.

I use a Bastion host because the cluster endpoint is on private subnet. The bastion host is on a public subnet. However, if we do not like public subnet and public IP, we can place the bastion host on a private subnet, and use SSM system manager agent with SSH tunnel plugin to have SSH access to private bastion host.

Summary

I first came across this article about EKS and its awfulness and then decided to check out EKS. I’m not sure all points are still valid but it’s generally real-life experiences. There are also many peripheral services, such as AMP (AWS Managed Prometheus), AMG (AWS Managed Grafana), ADOT (AWS Distro for Open Telemetry), AppMesh (Another Envoy-based Service Mesh, easier to manage than Istio but less Powerful), with a lot to explore.

Build and Manage Kubernetes Clusters

Fri, 23 Sep 2022 11:50:00 -0400

There are numerous options to build a Kubernetes cluster. If your company has a multi-cloud strategy, most likely you will have to deal with cluster creation on multiple cloud platform or on virtual machines on premise.

Most likely, the chosen cloud platform already make it simple for us. However, it is still important to understand what it really takes to build a Kubernetes cluster. In general, we need to figure out these tasks:

Decide where to host the computing infrastructure (i.e. Node) : on premise or public cloud;
Choose a Kubernetes release: either the vanilla release or one of the third-party distributions;
Install Kubernetes to the computing environment, and integrate it with the cloud platform;
Determine required add-ons (e.g. Istio or Linkerd for Service Mesh, dashboard utility, etc);
Deploy application workload to Kubernetes platform;

A public cloud platform provider usually can assist you with task 1 through 3, and partially 4, depending on the provider. If your Kubernetes resides on private cloud or on-prem environment, you can use a Platform solution such as VMware Tanzu or Openshift, which usually covers task 1, 3 and 4. There is no standard about what task these platform solution must address. Therefore it is important to have this list of tasks in mind in order to make a good comparison. I will discuss each of the tasks in this post.

Hosting environment

Nodes are the building blocks of a Kubernetes cluster. We need master nodes as well as worker nodes. In addition, a working cluster also requires storage, and networking infrastructure.

Public cloud platforms typically provides control plane as a service, obviating administrator’s effort to provision master nodes. For example, the control plane of Azure AKS has two levels of uptime commitment: a free tier of 99.5% SLO and a paid tier with an SLA of 99.95% (using AZs) and 99.9% (without using AZs). This uptime commitment applies to control plane only and do not apply to worker nodes. The management of etcd store is also a responsibility of the cloud provider, which frees up the cluster administrator from managing etcd store. However, they cannot access etcd store either. This is not very convenient because as the size of the cluster grows it is a common requirement to connect to etcd store for troubleshooting purpose.

The deployment APIs for public cloud allow the cluster administrator to define the instance size, count and availability zone for the worker nodes. They also automatically register the worker nodes to control plane so that the cluster administrators do not have to do so by themselves. As to storage, the public cloud usually provide some default storage classes based on their storage as service. For networking device, the cluster provision process automatically configures the cloud API so the cluster can manage cloud resources such as network load balancer.

With private cloud or data centre, we usually use virtual machines, or bare-metal servers. Cluster administrators will need to make their own control plane with master nodes. and install worker nodes and register them to the master nodes. The Kubernetes Installation section below will discuss this.

Kubernetes release

If you have to install Kubernetes, you have to think about the Kubernetes release being used. You can use the binary from official Github repository. For example, the release note of version 1.24.3 points to the change log file for download links to server binaries, node binaries. This is the vanilla Kubernetes release.

Apart from the vanilla release, many developers build their own distributions, based off forks of the Kubernetes project. CNCF has a page to keep track of certified Kubernetes distributions. Some of the distributions are open source and can be used for on-prem infrastructure. Here is a list of top players:

Distribution Name	Repo	Description
EKS Distro	Link	Used in EKS managed service or EKS Anywhere for on-prem infrastructure
AKS Engine	Link	Used in Azure Stack for on-prem infrastructure.
Google Kubernetes Engine	N/A	Used in GKE managed service only.
OpenShift Kubernetes Engine	Link	Community distribution (OKD, or OpenShift Kubernetes Distribution) is the open-source upstream.
Rancher Kubernetes Engine (RKE)	Link	still using Docker as container runtime. Supported CNI include: Canal, Flannel, Calico and Weave
K3s	Link	Lightweight distro without small resource requirement. Great for Edge, IoT, ARM etc
RKE2	Link	Originally named RKE government. Supports deployment via Cluster API. Supports containerd as container runtime. Supported CNI include: Cillium, Calico, Canal and Multus. Lightweight
VMware Tanzu	Link	VMWare Tanzu Grid and VMWare Tanzu Community

Above is just a very incomplete list of Kubernetes distributions. There are many more distributions that are not on this list, such as CoreOS Tectonic, Docker Kubernetes, Heptio, Mesosphere, Mirantis, Platform9, Stackube, Telekube. For full details of how each distribution is different, you will need to go over their documents.

With the selected distribution, we still need to deploy the binaries to the nodes. We can do this with a cluster management platform, or standalone installers. Cluster management platform can also help us with baseline configuration (e.g. IAM integration, CNI plugin), in addition to the binary installation.

Cluster Management Platform

These platforms are also sometimes referred to as container management platform.

For example, OpenShift container platform is a self-managed platform based on OpenShift Kubernetes Engine and can run on a variety of hosting environment, public cloud, or private cloud. The installation steps varies depending on the hosting environment. When running on public cloud such as AWS (aka ROSA), the public cloud only provides computing nodes and associated infrastructure. Many corporate with multi-cluster strategy use this option on public cloud to keep their Kubernetes cluster fleet consistent across cloud vendors.

The Openshift container platform also packages some useful open-source add-ons with corporate support, for example:

Red Hat’s strategy is to pick the most renowned open-source project in each domain and add enterprise support to it. However, for management portal, Red Hat developed its own Advanced Cluster Management tool for Kubernetes, and open-sourced it in 2020 in the upstream project Open Cluster Management.

Similar to OpenShift, VMware Tanzu also attempts to cover the domains, with a smaller product portfolio:

Service Mesh: compatible with Istio
Mission Control: management portal
Observability

Google Anthos is also a container platform. Their product line include, but not limited to:

Anthos Config Management (ACM)
Anthos Service Mesh (ASM, an Istio distribution)

SUSE, the developer of RKE, RKE2, and K3s) offers Rancher as multi-cluster management platform. Apart from the engines, SUSE also offers Lonhorn as a storage solution. However, they do not have offerings for service mesh or GitOps. So there is no doubt that Red Hat OpenShift has the most complete portfolio for Kubernetes.

There are also companies that only offers management platforms without their own Kubernetes distribution. For example:

Product capabilities in this category vary a lot and you should refer to their specific documentation to understand. You will probably see a stack chart from each of the platform provider (e.g. SUSE Enterprise Container, OpenShift, Tanzu, Anthos, Rafay) with all technology integrations.

Cluster Installation Tools

As we saw in the installation steps for OpenShift, they are highly dependent on platform. With public cloud, the provisioning process also applies only to a specific platform. Since Kubernetes Installation process is tedious, some tools emerged to help, for example: kubespray, kubeadm, kops and Cluster API. These are governed by SIG cluster lifecycle special interest group.

Here are some traditional options to install a Kubernetes clusters:

kube-up: the first tool to build cluster from 2015. It has been deprecated.
Kubeadm: a tool built to provide best-practice “fast paths” for creating Kubernetes clusters that are minimum viable, and secure. Kubeadm’s scope is limited to the local node filesystem and the Kubernetes API, and it is intended to be a composable building block of higher level tools. It is first released in Sep 2016. The high level configuration steps goes through initialization (kubeadm init), control plane (kubeadm join control plane), and node (kubeadm join node). Kubeadm does not integrate with cloud providers and it does not install addons (auth, monitoring, CNI, storage class)
Kubespray: runs on bare metal or VMs using Ansible for provisioning and orchestration. The first release was in Oct 2015. Since v2.3 (Oct 2017) kubespray started to use kubeadm internally. In addition to kubeadm, kubespray configures CNI, storage class, other CRI. It supports cloud providers and air-gap environment. However it does not support infrastructure management.

The options above are official options. You may use kubeadm and kubespray to quickly (i.e. in an hour) spin up clusters for education purposes. However, with their limitations, it typically requires a lot of efforts to build a production-grade cluster with the needed addons and integrations.

Apart from the official options, there are also unofficial tools such as kubicorn, which was first introduced in 2018 as a cluster management framework with modular support for cloud providers. However it appears to be short-lived.

In the next two sections, we introduce kops and cluster API, two most recent projects to install cluster.

Kops

The kops utility directly perform the provisioning and orchestration via API to the cloud deployment engine. Kops, with first release in Oct 2016, is tightly integrated with the unique features of the cloud providers (e.g. AWS: ASG, ELB, EBS, KMS, S3, IAM). However, kops is only CLI without controller-style reconciliation. It does not support baremetal or vsphere. It also bundles addons with fixed version.

When picking a tool to install cluster, we need to strike a balance between how much simplification the tool brings, and how many different platform the installer can work with. Kops appears to be such a good compromise. It works with a number of cloud platforms using different set of APIs, although most are in alpha and beta stages today. Here is how to install cluster on AWS.

Both kops and Cluster API have good momentum but they work differently. Cluster API was first released in Mar 2019, and is currently less mature than kops. However, it is declarative and may reflect the direction of where cluster lifecycle management is heading.

Cluster API

Cluster API focuses on following areas:

Manage cluster lifecycle declaratively
Infrastructure abstraction (e.g. computing, storage, networking, security, etc)
Utilizing existing tools (e.g. kubeadm, cloud-init)
Modular and pluggable: to be adaptable to different infrastructure providers.

It involves a number of CRs as illustrated in its diagram. We should be clear on the providers for Bootstrap, Infrastructure and Control Plane.

The biggest benefit is the controller pattern to manage the entire lifecycle of a cluster. This allows managing clusters with GitOps, and rolling upgrade of the cluster. It also allows for declarative node scaling, self healing and multi-cluster management.

The client utility for is clusterctl, and with that along with the manifest, we can create a cluster in a few commands. A lot of workflows are still in development but we can take a look at its quick start guide to get a taste of how it works. The installation steps vary a lot based on the environment and the cluster. Also it introduces the separation of management cluster and workload cluster.

Workload cluster is the target cluster being created, as per the manifests.
Management cluster is where you keep track of the workload cluster being managed. You can manage multiple workload clusters from a single management cluster. Note that this management cluster will store credentials about workload clusters, and may become a single point of failure.

Although Cluster API reflects a great initiative to standardize the provisioning of Kubernetes cluster, whether it will succeed has to do with the level of complexity. In the next section, we will get a taste of how it looks to deploy a Kubernetes cluster in a lab.

Management cluster vs workload cluster

In the lab, I use my MacBook to create a management cluster with KinD. Then we configure a workload cluster in AWS from the management cluster.

Cluster API Lab

Note that the steps here are based on the quick start guide on Cluster API document. Also, there is a bug with the AWS provider so the end of the lab will report a warning. The main purpose of this lab is to demonstrate how Cluster API is supposed to work, even though it still has yet to mature.

To start, I install clusterctl (the cluster API client utility), clusterawsadm (the utility specific for AWS) on MacBook, then start a simple KinD cluster.

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.2.0/clusterctl-darwin-amd64 -o clusterctl
chmod +x ./clusterctl
sudo mv ./clusterctl /usr/local/bin/clusterctl
clusterctl version

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/download/v1.4.1/clusterawsadm-darwin-amd64 -o clusterawsadm
chmod +x clusterawsadm
sudo mv clusterawsadm /usr/local/bin
clusterawsadm version

kind create cluster

So far, I installed the required utility and a KinD cluster on MacBook. Then I use clusterawsadm to create InstanceProfile, ManagedPolicy and IAM Roles required for cluster creation. The AWS region and access are configured as environment variables:

export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID=AKIAXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=J8ByduiofpwuisDjDoijOISDs
clusterawsadm bootstrap iam create-cloudformation-stack

This runs a CloudFormation stack to create the permission related resources:

Then I initialize the management cluster with the clusterctl utility, specifying AWS as a provider. I also need to assign the environment variable AWS_B64ENCODED_CREDENTIALS with proper value:

export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)
clusterctl init --infrastructure aws

Now I use clusterctl to generate the manifest for the workload cluster. In environment variables, I specify cluster and node sizes, SSH key name, control plane machine type and node machine type:

export AWS_SSH_KEY_NAME=cskey
export AWS_CONTROL_PLANE_MACHINE_TYPE=t3.large
export AWS_NODE_MACHINE_TYPE=t3.large
clusterctl generate cluster myekscluster --kubernetes-version 1.24.3 --control-plane-machine-count=3 --worker-machine-count=3 > capi-quickstart.yaml

kubectl apply -f capi-quickstart.yaml

At the end I tell the management cluster to create a workload cluster as per the manifest, by simply declaring the CRs. It will take some time for the cluster to create, and there are a number of ways to monitor the progress. You can monitor the log on the controller pods in their respect namespaces. You can also check the cluster status with:

kubectl get kubeadmcontrolplane
clusterctl describe cluster myekscluster

Currently there is a bug and the commands at the end will report as below:

Hopefully the bug will be fixed shortly. To delete the cluster, simply delete the resources in the manifest with kubectl delete -f capi-quickstart.yaml

Summary

There are numerous ways to build a Kubernetes cluster. Before deciding on the approach, I recommend having a full understanding of the hosting environment. This is because installation approach and hosting environment are still tightly coupled. This is the status quo and is not going to change in the near future. Both kops and cluster API reflects initiative to decouple the two but both are still in early stage and already facing growing complexity. Cluster API manages complexity with CRDs to abstract system resources and infrastructure, as illustrated here:

CRDs and providers to abstract system resources and infrastructure

The diagram is from the “Cluster API and declarative Kubernetes Management” white paper. Here is a stream with more about the same topic.

Kubernetes Storage on Azure 3 of 3 – Ceph by Rook

Fri, 26 Aug 2022 19:43:00 -0400

In the last two posts, I covered the native storage options on Azure Kubernetes Service, as well as Portworx as an example of a proprietary Software Defined Storage (SDS) solution. There are also a number of open-source alternative SDS solutions. Ceph has nearly a decade of history from prior to containerization, and is the most widely adopted storage platform. In this post, we continue to explore Ceph as an open-source storage solution on Azure Kubernetes.

Ceph by Rook

Ceph is an open-source SDS platform for distributed storage on a cluster and provides object, block and file storage. Installation of Ceph SDS can be complex, especially on Kubernetes platform. Rook is a graduated CNCF project to orchestrate storage platform. Rook by itself is not SDS and it supports:

Ceph: configure a Ceph cluster. Think of this as the equivalent of cephadm on Kubernetes platform.
NFS: configure an NFS server. Think of this as the equivalent of nfsd daemon on Kubernetes platform.
Cassandra: an operator to configure a Cassandra database cluster. It is now deprecated.

We play with Rook Ceph. I also refer to it as Ceph by Rook. The contribution of Rook project is it simplifies the installation as a matter of declaring custom resources using CRDs. Here are some high-level CRDs to know:

CephCluster: creates a Ceph storage cluster
CephBlockPool: represents a block pool
CephFilesystem: represents a file system
CephObjectStore: represents an object store
CephNFS: spins up a NFS Ganesha server to export NFS shares of a CephFilesystem or CephObjectStore.

As with typical Kubernetes resources in controller pattern, Ceph by Rook needs an operator along with custom resources. We can use YAML manifest for both of them, and the manifests are usually very tediously long. We can also use Helm to install both of them, by providing a value file. Now we will install Ceph on AKS.

Install Ceph Operator on AKS

The steps are influenced by two relevant posts (here and here). However, I’ve incorporated the cluster configuration in the Azure directory of the cloudkube project, a modular Terraform template to configure AKS cluster and facilitate storage configuration. The node group and instance sizes are selected to be just enough to run a ceph POC cluster with minimum cost. One of the node groups is tainted with storage-node, as if the following command were run:

kubectl taint nodes my-node-pool-node-name storage-node=true:NoSchedule

You will only need to taint the nodes with the command above if you choose not to use the cloudkube template. The taint ensures that only Pods with corresponding toleration and effect can be scheduled to those nodes.

We use Helm to install Rook Operator. We need a value file (e.g. rook-ceph-operator-values.yaml) with content as below:

# https://github.com/rook/rook/blob/master/Documentation/Helm-Charts/operator-chart.md
crds:
 enabled: true
csi:
 provisionerTolerations:
 - effect: NoSchedule
 key: storage-node
 operator: Exists
 pluginTolerations:
 - effect: NoSchedule
 key: storage-node
 operator: Exists
agent:
 # AKS: https://rook.github.io/docs/rook/v1.7/flexvolume.html#azure-aks
 flexVolumeDirPath: "/etc/kubernetes/volumeplugins"

Then we install the operator with Helm:

helm install rook-ceph-operator rook-ceph --namespace rook-ceph --create-namespace --version v1.9.6 --repo https://charts.rook.io/release/ --values rook-ceph-operator-values.yaml
kubectl -n rook-ceph get po -l app=rook-ceph-operator

After installing the operator, we check the Pod status to make sure it is running. Then we can install the actual Ceph Cluster in one of the two ways. We can declare a CephClusterCRD ourself, or we can use Helm again to declare the CRD. Helm Chart gives us a lot of useful default values and saves us from editing a large body of YAML manifest.

Install Ceph CR on AKS

We use Helm to install CephCluster CRD. We create a value file (e.g. rook-ceph-cluster-values.yaml) with content as below:

# https://github.com/rook/rook/blob/master/Documentation/Helm-Charts/ceph-cluster-chart.md
operatorNamespace: rook-ceph
toolbox:
 enabled: true
cephObjectStores: [] # by default a cephObjectStore will be created. Setting this to null disables it
#cephBlockPools: # by default a cephBlockPool will also be created with default values
#cephFileSystems: # by default a cephFileSystem will also be created with default values

cephClusterSpec:
 mon:
 count: 3
 volumeClaimTemplate:
 spec:
 storageClassName: managed-premium
 resources:
 requests:
 storage: 10Gi
 resources:
 limits:
 cpu: "500m"
 memory: "1Gi"
 requests:
 cpu: "100m"
 memory: "500Mi"
 dashboard:
 enabled: true
 storage:
 storageClassDeviceSets:
 - name: set1
 # The number of OSDs to create from this device set
 count: 3
 # IMPORTANT: If volumes specified by the storageClassName are not portable across nodes
 # this needs to be set to false. For example, if using the local storage provisioner
 # this should be false.
 portable: false
 # Since the OSDs could end up on any node, an effort needs to be made to spread the OSDs
 # across nodes as much as possible. Unfortunately the pod anti-affinity breaks down
 # as soon as you have more than one OSD per node. The topology spread constraints will
 # give us an even spread on K8s 1.18 or newer.
 placement:
 topologySpreadConstraints:
 - maxSkew: 1
 topologyKey: kubernetes.io/hostname
 whenUnsatisfiable: ScheduleAnyway
 labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - rook-ceph-osd
 tolerations:
 - key: storage-node
 operator: Exists
 preparePlacement:
 tolerations:
 - key: storage-node
 operator: Exists
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: agentpool
 operator: In
 values:
 - storagenp
 topologySpreadConstraints:
 - maxSkew: 1
 # IMPORTANT: If you don't have zone labels, change this to another key such as kubernetes.io/hostname
 topologyKey: topology.kubernetes.io/zone
 whenUnsatisfiable: DoNotSchedule
 labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - rook-ceph-osd-prepare
 resources:
 limits:
 cpu: "500m"
 memory: "4Gi"
 requests:
 cpu: "500m"
 memory: "2Gi"
 volumeClaimTemplates:
 - metadata:
 name: data
 spec:
 resources:
 requests:
 storage: 100Gi
 storageClassName: managed-premium
 volumeMode: Block
 accessModes:
 - ReadWriteOnce

During the cluster provisioning, there will be a number of preparing Pods. We want those Pods to run on nodes with label agentpool=storagenp. In real life, we need to orchestrate where to run each workload, by restricting the nodes to schedule certain types of workload.

Then we can install the cluster using Helm:

helm install rook-ceph-cluster rook-ceph-cluster --namespace rook-ceph --create-namespace --version v1.9.6 --repo https://charts.rook.io/release/ --values rook-ceph-cluster-values.yaml

After running the Helm install, it may take as long as 15 minutes for all resources to settle. Watch the Pod status in rook-ceph namespace. At the end, make sure that the cluster is created successfully:

kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get CephCluster
NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL
rook-ceph /var/lib/rook 3 15m Ready Cluster created successfully HEALTH_OK
kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get cephBlockPools
NAME PHASE
ceph-blockpool Ready
kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get cephFileSystems
NAME ACTIVEMDS AGE PHASE
ceph-filesystem 1 20m Ready

In my case it took 15 minutes before the cluster comes up as created successfully. You should notice that two storage classes were also created as a part of the install. It however did not create a storage class or CRD for object storage, because we explicitly disabled it in the Helm value file by setting cephObjectStores value to null.

Dashboard

We enabled dashboard. To configure the dashboard view properly, we would need an ingress. For a quick view here, we can play port forwarding tricks. First we fetch the admin password for use in the next step. Then expose the dashboard to the bastion host:

$ kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath='{.data.password}' | base64 -d
$ kubectl -n rook-ceph port-forward svc/rook-ceph-mgr-dashboard 8443:8443

Since I don’t have UI on the bastion host, I use the port forwarding trick again from my own MacBook. Start a new terminal and SSH to the bastion host with port-forwarding switch:

$ ssh -L 8443:localhost:8443 kubeadmin@20.116.132.8

The command above suppose the public IP of the bastion host is 20.116.132.8. Then from my MacBook I can browse to localhost:8443 (with Safari browser which gives me the option to bypass certificate error). At the web portal, provide username (admin) and password (as retrieved above):

Ceph console for Kubernetes

Apart from the dashboard, we can also use ceph admin tool from a toolbox pod, following this instruction. For monitoring, Ceph by Rook can expose metrics for Prometheus to scrape.

Performance

With default ceph configuration on AKS, I ran quick performance test using kube-str . The result is as follows:

	read_iops	write_iops	read_bw	write_bw
ceph-block	IOPS=464.507294 BW(KiB/s)=1874	IOPS=243.296143 BW(KiB/s)=989	IOPS=509.928162 BW(KiB/s)=65797	IOPS=248.530762 BW(KiB/s)=32338
ceph-filesystem	IOPS=438.701324 BW(KiB/s)=1770	IOPS=226.270660 BW(KiB/s)=920	IOPS=405.936340 BW(KiB/s)=52456	IOPS=208.869293 BW(KiB/s)=27229

The metrics reflects performance under default configuration. It should not be considered as the best performance that Ceph can deliver on Azure Kubernetes.

Summary

I discussed three storage options for Azure Kubernetes but the idea applies to other Kubernetes platform hosted on a CSP. The native storage has significant limitation. NFS has latency. Block storage does not address high availability at the storage layer. Portworx and LINSTOR fill that gap as a commercial solution. Ceph is based on Object storage.

Etcd – the key-value store for Kubernetes

Tue, 14 Jun 2022 00:10:00 -0400

Etcd in Kubernetes

In Kubernetes architecture, etcd is the data store. It stores the desired state of Kubernetes object. API server is the only client that connects to etcd (via gRPC protocol). Cluster builder specifies the endpoint of etcd as a parameter to the kube-api-server process. Other Kubernetes components, whether in the control plane or from the nodes, connect to API server. API server translates their request into etcd query, and then translates etcd query result into what its clients ask for. For this reason, communication with etcd accounts for a lot of network traffic in a Kubernetes cluster.

The etcd store is a CNCF project for “a distributed, reliable key-value store for critical data in a distributed system”, developed by CoreOS team. So it is essentially a distributed key-value store for any distributed application. If an application runs on Kubernetes, it can leverage etcd store, by keeping their configurations in ConfigMap and Secret objects. One key feature is to watch for specific keys or directories for changes, and react to the changes. Voila! This is the underlying mechanism for controller!

A Kubernetes cluster may have stacked etcd deployment or connect to an external etcd store.

stacked etcd architecture

external etcd architecture

In managed Kubernetes services such as EKS in AWS and AKS in Azure, users usually do not directly access etcd store. However, it is still a very important component to understand. Its use case includes:

Configuration sharing
Service discovery
Consistency
Watching mechanism
Expiry and extension of key

The consistency use case is based on Raft protocol for distributed consensus.

Raft protocol

I am not an expert in distributed consensus protocols and nor do I intent to cover it in depth. At a high level, I have heard of three of them so far:

Etcd uses Raft protocol
Zookeeper uses ZAB protocol
Cassandra uses paxos protocol

Here is a good intro to the three protocols. Instead of getting into the fine details, I would like to discuss why we need such a consensus protocol (or consensus mechanism) in distributed systems, which are also decentralized systems.

Centralized, Decentralized, Distributed systems

The reason a distributed system needs consensus protocol, is that a distributed system lacks a single source of truth as centralized systems do. Different parts of the distributed system may receive different signals but they must come to agreement of a single plan to act. Lamport studies this with an analogy of Byzantine Generals problem, and first proposed Paxos protocol. Paxos has been an important foundation to modern distributed systems. In Paxos, consensus is achieved in two phases, which creates the problem of livelocks. Raft is an alternative to Paxos, and is widely adopted today. Here is a link to an animated illustration for Raft protocol. The Raft protocol is also used in Redis. It has three roles: Leader, Candidate, and follower. ZAB protocol is similar to Raft, where it needs to select a leader.

Etcd Lab

In troubleshooting, if we suspect that the response from API server is inconsistent with etcd store, we want to directly connect to it.

Managed Kubernetes services do not expose their etcd store. We can use KinD or Minikube. There are two types of jump box to access etcd store: using etcd Pod, or SSH to a Node. To connect to etcd, we also need the X509 key, certificate and CA’s certificate, in addition to the endpoint, usually an IP with port 2389. When I connect to Pod shell, I find the command shell not easy to use. They might miss basic command such as ls, or do not support auto completion.

Take KinD for example, we first create a secret, then we can connect to the node with docker CLI command:

kubectl create ns myns
kubectl -n myns create secret generic mysecret --from-literal key1=value1
kubectl -n myns get secret mysecret -o jsonpath='{.data.key1}' | base64 -d
docker exec -it control /bin/bash

From the node,

apt update && apt install etcd-client
etcdctl version
nc -vz localhost 2379
cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd

export ETCDCTL_API=3
export ETCDCTL_CERT=/etc/kubernetes/pki/apiserver-etcd-client.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/apiserver-etcd-client.key
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_ENDPOINTS='https://127.0.0.1:2379'
etcdctl member list write out=table

Now we can see the secret object directly with etcd store:

etcdctl get /registry/secrets/myns/mysecret

With get query, when using –prefix, we can use –keys-only switch to list keys without values:

etcdctl get --prefix /registry/api --keys-only
etcdctl get --prefix /registry/namespace -wjson

We can write key-value with put command:

etcdctl put myloc 0
etcdctl get myloc -wjson

In Kubernetes, all the key names start with / which makes the key looks like a POSIX path. Every Kubernetes object is stored in etcd with a unique key following a self-explanatory naming pattern. To display the path, we can also use debug log that records the call to API server:

kubectl get ns myns -v9

Look for curl command such as:

I0523 22:51:43.517728 32347 round_trippers.go:466] curl -v -XGET -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.23.6 (darwin/amd64) kubernetes/ad33385" 'https://127.0.0.1:64081/api/v1/namespaces/myns'

From there we can see the etcd query as the URI is namespaces/myns, which we use in etcdctl query path:

etcdctl get /registry/namespaces/myns

Every type of Kubernetes object has a storage.go file in their implementation that defines how api server should write object. Here is an example for Pod object.

Etcd also supports watch command to watch for changes. For example:

etcdctl watch --prefix /registry/namespace # watch output k create ns newns

Now we create a namespace with kubectl:

kubectl create ns myns

The output from etcdctl will reflect the change. The communication between etcdctl and etcd is gRPC protocol. The output is based on stream, as we can see from the watch result.

Etcd Maintenance

Like any distributed store, etcd needs maintenance and operation work. For example, we can check endpoint status with endpoint command:

etcdctl endpoint status

We can also backup and restore etcd store with etcdctl command:

etcdctl snapshot save /tmp/backup.db

This was an question in CKA exam. In real life, when the workload scales up, the etcd store may come across many pitfalls, such as degraded performance, unresponsiveness, some etcd member going down, network partition on etcd store causing split brain. It is important to ensure efficient communication between API server and etcd store. The etcdctl provides defrag and compact commands for common maintenance activities.

Kubernetes Networking Solutions Overview

Tue, 22 Jun 2021 12:14:26 -0400

Kubernetes networking involves a lot of details. We discuss some CNI plugins in this post.

The most basic mode is kubenet. We use –network-plugin=kubenet with kubelet process to use it. Kubenet is not a CNI plugin, but it works with bridge, lo and host-local (CNI-compliant implementations). We can directly specify MTU with –network-plugin-mtu. Kubenet is a basic network plugin, based on bridge plugin, with the addition of port mapping and traffic shaping. It does not offer cross-node networking itself. Today it is typically used with managed clusters by cloud providers, where the cloud provider set up routing rules themselves for inter-node communication.

When a cluster goes multi-node, the main challenge is communication between Pods across different nodes. Pods come and go. The size of cluster could increase or decrease as well. The network solutions come in two network types: overlay network based on encapsulation, or non-overlay networks, most likely using routing techniques. Common backends for for multi-host container networking solutions include VXLAN encapsulation, IPIP encapsulation, host-gw, IPSec. In addition, there are some backends that only used by certain plugins.

Common Backends

VXLAN: use in-kernel VXLAN to encapsulate the packets. VXLAN is a virtual networking capability in Linux which is also used in virtualization technology. VXLAN is an overlay technology requiring encapsulation of overlay network’s layer-2 frame into UDP packet at layer 4 of underlay network. When configured, the VxLAN backend creates a Flannel interface on every host. When a container on one node wishes to send traffic to a different node, the packet goes from the container to the bridge interface in the host’s network namespace. From there the bridge forwards it to the Flannel inteface because the kernel route table designates that this interface is the target for the non-local portion of the overlay network. The Flannel network drive wraps the packet in a UDP packet and sends it to the target host. Once it arrives at its destination, the process flows in reverse, with the Flannel driver on the destination host unwrapping the packet, sending it to the bridge interface, and from there the packet find its way into the overlay network and to the destination Pod.

host-gw: the host-gw is a non-overlay solution that maintains route tables on Linux Host to allow Pods to communicate across Nodes. It is only used in Flannel plugin. Suppose we have two hosts, each with two containers as connected below. Initially, container0 is not able to reach container2 because eth0 on node0 does not have an entry that matches container2’s IP address. The packet is there sent to default route, which isn’t destined to container2.

However, if we build rules to match container IP address, on the route table of each node. The issue would be solved. This is essentially how host-gw works. Specifically, on node 0, we add “ip route add 192.168.1.0/24 via 10.20.0.2 dev eth0”, on node 1, we add “ip route add 192.168.0.0/24 via 10.20.0.1 dev eth0). The host-gw in Flannel will manage rule addition to us. Note that the two hosts must have direct layer 2 connectivity. In other words, there must not be a router between the two nodes. Otherwise, the routing table on the router is out of reach. In fact, all nodes in a Flannel network must have layer 2 connectivity with each other. In other words, all nodes must be in a single LAN. Host-gw provides better performance than VxLAN.

IPSec uses in-kernel IPSec to encapsulate and encrypt the packets. IPsec is a group of protocols to ensure authentication and encryption per packet between devices. Since it secures traffic at layer 3 and now it has become a major backend technology for VPN. IPsec adds several headers and trailers to datagram containing authentication and encryption information. The two major protocols working in IPSec are AH (Authentication Header) and ESP (Encapsulating Security Payload). AH serves up authentication services only; ESP provides both authentication and encryption abilities. It also uses IKE protocol for key exchange.

IPSec works in two modes: transport and tunnelling mode.

Transport mode creates a secure tunnel between two devices end to end. The payload of each datagram is encrypted, but the original IP header is not. Intermediary routers are thus able to view the final destination of each datagram, unless a separate tunnelling protocol (e.g. GRE) is used.
Tunnel mode works between two endpoints, such as two routers, protecting all traffic that goes through the tunnel. The original IP header containing the final destination of the datagram is encrypted, in addition to the payload. To tell intermediary routers where to forward the datagrams, IPsec adds a new IP header. At each end of the tunnel, the routers decrypt the IP headers to deliver the datagram to their destinations. The intermediary routers does not know the final destination, or what transport protocol is used.

IPIP (IP over IP) tunnel is typically used to connect two internal IPv4 subnets through public IPv4 internete. It has the lowest overhead but can only transmit IPv4 unicast traffic.

CNI Plugins

Originally, the network functions were developed in-tree. Then the CNI specification came up to allow plugin development out-of-tree to implement cluster networking functions. The Container Network Interface seeks to completely decoupled network management from container runtime. Kubernetes picked CNI over CNM in 2016, as discussed in my virtualization discussion. CNI clearly defines the specification for following activities:

When a Pod comes up, give it a network interface
Assign IP to the network interface
When a Pod is deleted, delete the associated network interface

When we configure a Kubernetes cluster, we must specify –network-plugin switch, so that the cluster is operational. If we use CNI as network-plugin, we also need to install the plugin, optionally with the help of Rancher.

On the worker node, we use –network-plugin=cni with kubelet process to use CNI plugins. A plugin may consist one or more binaries. The binaries are located in /opt/cni/bin (or otherwise specified by –cni-bin-dir). The configurations are located in /etc/cni/net.d (or otherwise specified in –cni-conf-dir). Note that the configuration file may reference different plugin implementations for different network management purpose (e.g. interface creating, address allocation, etc). The container networking repo provided some reference implementations and some of them are used by other plugins. These reference implementations include:

Main (interface creating): bridge, ipvlan, loopback, ptp, macvlan, etc
IPAM (IP address management): host-local, dhcp, static
Meta (other plugins): portmap, bandwidth

So, a CNI plugin consists of a networking solution for backend, and binaries to cover the aspects outlined above. I discussed some common backends above. Below I will introduce some common plugins and backends only available to each plugin

Flannel

Flannel by CoreOS: supports a range of backends. The advantage of Flannel is it reduces the complexity of doing port mapping. This is a great post that covers the mechanism.

Flannel with overlay (e.g. VxLAN on UDP encapsulation)

It supports VXLAN, host-gw, IPSec, IPIP as well as the followings:

Amazon VPC: recommended with Amazon VPC. AWS VPC creates IP routes in an AWS route table. The number of records in this table is limited by 50 so you can’t have more than 50 machines in a cluster.

GCE: recommended with Google Compute Engine Network. Instead of using encapsulation, GCE also manipulates IP route to achieve maximum performance. Because of this, a separate flannel interface is not created.
UDP: debugging only for old kernels that don’t support VXLAN

Calico

Border Gateway Protocol (BGP) is a standardized exterior gateway protocol designed to exchange routing and reachability information among autonomous systems (AS) on the Internet.

Calico operates at layer 3. It prefers BGP without an overlay network for the highest speed and efficiency, but in scenarios where hosts cannot directly communicate with one another, it can utilize an overlay solution (e.g. VxLAN or IP-in-IP). Calico also supports network policies for protecting workloads and nodes from malicious activity or aberrant applications.

The Calico networking Pod contains a CNI container to keep track of Pod deployment, and register addresses and routes. It also contains a daemon that announces the IP and route information to the network via the Border Gateway Protocol (BGP). The BGP daemon build a map of the network that enables cross-host communication.

Calico requires a distributed and fault-tolerant key/value store, and deployments often choose etcd to deliver this component. Calico uses it to store metadata about routes, virtual interfaces, and entwork policy objects. Calico can either use a separate HA deployment of etcd, or the same etcd datastore with the Kubernetes cluster.

When we are unable to use BGP (e.g. with cloud provider, or in an environment where we have no permission to configure router peers. Calico’s IP-in-IP mode encapsulates packets before sending them to other nodes.

Once IP-in-IP is configured, Calico wraps inter-Pod packets in a new packet with headers that indicate the source of the packet is the host with the originating Pod, and the target of the packet is the host with the destination Pod. The Linux kernel performs this encapsulation, and then forwards the packet to the destination host where it is unwrapped and delivered to the destination Pod.

Canal

The followings is quoted from Rancher website:

Canal seeks to integrate the networking layer provided by Flannel with the networking policy capabilities of Calico. As the contributors worked through the details however, it became apparent that a full integration was not necessarily needed if work was done on both projects to ensure standardization and flexibility. As a result, the official project became somewhat defunct, but the intended ability to deploy the two technology together was achieved. For this reason, it’s still sometimes easiest to refer to the combination as “Canal” even if the project no longer exists.

Because Canal is a combination of Flannel and Calico, its benefits are also at the intersection of these two technologies. The networking layer is the simple overlay provided by Flannel that works across many different deployment environments without much additional configuration. The network policy capabilities layered on top supplement the base network with Calico’s powerful networking rule evaluation to provide additional security and control.

Weave Net

Weave Net by Weaveworks offers a different paradigm. Weave creates a mesh overlay network between each of the nodes in the cluster, allowing for flexible routing between participants. Applications use the network just as if the containers were all plugged into the same network switch, with no need to configure port mappings and links.

For more good references to determine networking options, check out these posts:

Calico blog
Rancher blog
Kubevious blog

Certified Kubernetes Administrator (CKA) Exam

Fri, 30 Apr 2021 09:50:00 -0400

The Certified Kubernetes Administrator (CKA) exam is a hands-on session where you need to follow the instructions to configure the system in a bash terminal on the web browser. In my experience, some shortcut keys (such as Alt+F) do not work, which slows me down a little bit. For each question, you need to switch kubectl context as instructed in the question. Some questions share the same context so it is very easy to omit this step. You can verify response with your own command but will not be told whether you scored in each question. During the CKA exam I tried to spin up a terminal session from within Vim editor and the terminal ran out of buffer. I had to reboot the machine with the help of proctor, and my completed work are saved.

In general this is an exam I enjoy preparing and writing because it is very hands on. Result is out a day after, and I passed at 96%. I heard about tight timelines but I managed to finish 15 minutes before the end, most likely owing to my dexterity with Linux commands. With that I’m happy to share my notes in preparation for the CKA exam.

For taking CKA exam, we should be familiar with the diagram above.

Tips for Troubleshooting

The CKA exam is hands-on and therefore requires quite a bit of troubleshooting. Here are my notes.

Check Node status to start with
Check core services on each node:
- sudo systemctl status kubelet
- sudo systemctl status docker
- sudo journalctl -u kubelet
- sudo journalctl -u docker
Check component logs (on hosting VM)
- /var/log/kube-apiserver.log
- /var/log/kube-scheduler.log
- /var/log/kube-controller-manager.log
If cluster is built by kubeadm, then some of those services are running in Pods within kube-system namespace. Check those pods:
- run interactive shell: > kubectl exec podname –stdin –tty — /bin/sh
- there is an image for lots of useful network tool called nicolaka/netshoot

Store pod names to variable. e.g. > POD_NAME=$(kubectl get pods -l run=nginx -o jsonpath=”{.items[0].metadata.name}”)
With kubectl, you may
- alias it to k for faster typing
- –dry-run: to run imperative command without creating object
- –record: record the command that was used to make a change
- -o: set output format, wide, yaml, or jsonpath=”expression”. For example, to get pod name: > kubectl get pods -l run=nginx -o jsonpath=”{.items[0].metadata.name}”
- –sort-by: use JSONPath expression
- –selector: filter results by label

Build K8s cluster using kubeadm

The CKA exam requires you to know how to build cluster with kubeadm. This involves installing four components (docker-ce, kubeadm, kubectl and kubelet), as outlined below:

step	command
1. Install docker-ce	> curl -fsSL https://download.docker.com/linux/ubuntu/gpg \| sudo apt-key add - > sudo add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" > sudo apt-get update > sudo apt-get install -y docker-ce=18.06.1~ce~3-0~ubuntu > sudo apt-mark hold docker-ce > sudo systemctl status docker
2. Install kubeadm, kubelet and kubectl	> curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg \| sudo apt-key add - cat << EOF \| sudo tee /etc/apt/sources.list.d/kubernetes.list deb https://apt.kubernetes.io/ kubernetes-xenial main EOF > sudo apt-get update > sudo apt-get install -y kubelet kubeadm kubectl > sudo apt-mark hold kubelet kubeadm kubectl
3. Form a K8s cluster	On master node: > sudo kubeadm init --pod-network-cidr=10.244.0.0/16 This command prints out a command for worker nodes to join.	On worker node: sudo the command generated on master
4. Configure kubectl	On master node: > mkdir -p $HOME/.kube > sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config > sudo chown $(id -u):$(id -g) $HOME/.kube/config	Optionally on worker node: > mkdir -p $HOME/.kube then scp $HOME/.kube/config from control plane node
5. Set up cluster networking	> echo "net.bridge.bridge-nf-call-iptables=1" \| sudo tee -a /etc/sysctl.conf > sudo sysctl -p Then from any environment with kubectl, bring up the system pods for cluster networking > kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Add new node to KubeAdm cluster

This is fairly simple with the help of kubeadm. The node to join cluster must be able to communicate with master node. Create a token and print join command from master node:

> kubeadm token create --print-join-command

Then from the node to join, run this command as sudo. You will see that it performs the TLS bootstrap for you. Once completed, the standard output will say this node has joined the cluster. You can confirm with command:

> kubectl get nodes

Sometimes one needs to migrate pods to the newly joined node. This can be done by draining the existing nodes.

Note you can also use kubespray to build K8s cluster as previously discussed, and here is a my IaC project to launch AWS instances and build a K8s cluster with kubespray on top of it. For my learning, I often create a GKE (Google Kubernetes Engine) cluster from GCP’s cloudshell. There is a guide on how to start a cluster but it comes down to three commands:

$ gcloud config set compute/zone us-east1-b
$ gcloud container clusters create tcluster --num-nodes=3
$ gcloud container clusters get-credentials tcluster

The third command above is to configure kubectl on the cloudshell. Follow this guide if you need to SSH to node.

Upgrade KubeAdm cluster

This involves upgrade three components (kubeadm, kubectl and kubelet) on two types of node: master node and worker node. They steps vary slightly for two nodes. But drain and uncordon is needed for both types of nodes. Pick a node and follow the steps below:

Step	Command
1. drain the node from kubectl client (e.g. master node)	> sudo kubectl drain nodename --ignore-daemonsets
2. Determine kubeadm target version	> apt-mark showhold > sudo apt-mark unhold kubeadm kubectl kubelet > apt list --installed \| grep kube > apt-cache show kubeadm \| less > sudo apt-get install -y kubeadm=1.20.2-00
3. update kubeadm	On master node: > sudo kubeadm upgrade plan v1.20.2 > sudo kubeadm upgrade apply v1.20.2	On worker node: > sudo kubeadm upgrade node
4. On the node to update, determine target version for kubectl and kubelet, then install	> apt-cache show kubectl \| less > apt-cache show kubelet \| less > sudo apt-get install -y kubectl=1.20.2-00 kubelet=1.20.2-00
5. Restart kubelet	> sudo systemctl daemon-reload > sudo systemctl restart kubelet
6. Uncordon	> kubectl uncordon nodename

Backup and restore Etcd

Etcd is a distributed key-value store. It uses Raft protocol for distributed consensus. Etcd is the third distributed system I touch on. The previous two are: Cassandra (using Paxos protocol for distributed consensus) and ZooKeeper (using ZAB protocol). Here is a good article that summarizes the protocols. As for the exam we only need to use etcd with the client tool.

The etcd itself can run on a cluster of servers, each running etcd as a systemd service as etcd/etcd (user/group). It can be deployed in two ways:

stacked etcd: an instance of etcd lives with kube-api-server on the same control plane node
external etcd: in a dedicated cluster of etcd

Alternatively, etcd can run as a pod, most likely in kube-system namespace.

The etcd service listens on port 2379 for client communication and on port 2380 for server (peer-to-peer) communication. When the systemd service was initialized there are a few key environment variables (e.g. cert locations, ETCD_DATA_DIR) privoded as configuration. To see them, run:

> cat /etc/systemd/system/etcd.service | grep Env

These environment variables (prefixed with ETCD_) are for the service only. They can provide current configuration information for us to use later. When it’s running as a pod, check out the directory for static pod for the yaml declaration (e.g. /etc/Kubernetes/manifests/etcd.yaml), where these parameters are passed in as environment variable.

The etcdctl utility is a command line client for etcd. The default API version is 3 so no need any more to set ETCDCTL_API=3 before each command. The utility needs three arguments three arguments (–cacert, –cert, and –key) but we can pass the information via environment variables:

> export ETCDCTL_CACERT= /home/cloud_user/etcd-certs/etcd-ca.pem
> export ETCDCTL_CERT= /home/cloud_user/etcd-certs/etcd-server.crt
> export ETCDCTL_KEY= /home/cloud_user/etcd-certs/etcd-server.key
> export ETCDCTL_ENDPOINTS=https://etcd1:2379

The environment variable names are uppercase of the argument name with prefix ETCDCTL_. Only global options of arguments can be supplied via environment variables. They remain effective throughout the rest of activities. Also note that the CACERT is needed only when client-cert-auth is true. Now to backup, we can simply run:

> etcdctl snapshot save /home/cloud_user/etcd_backup.db

To restore from a file, you want to remove existing etcd data directory first. The directory can be found in ETCD_DATA_DIR variable. Suppose it is /var/lib/etcd, you need root permission to write to it, then correct ownership before starting the service:

> sudo systemctl stop etcd && sudo mv /var/lib/etcd/ /tmp/
> sudo etcdctl snapshot restore /home/cloud_user/etcd_backup.db --data-dir /var/lib/etcd
> sudo chown -R etcd:etcd /var/lib/etcd && sudo systemctl start etcd

To verify the restore result, simply run:

> etcdctl get cluster.name

Object Management

In the CKA exam, we need to interact with many types of built-in Kubernetes objects.

RBAC objects:
- A Role defines permissions within namespace.
- A ClusterRole defines cluster-wide permissions.
- Both Roles and ClusterRoles are K8s objects that defines a set of permissions
- RoleBinding and ClusterRoleBinding are objects that connect Roles and ClusterRoles to users.
Service Account: an account used by container processes within Pods to authenticate the K8s API. If your Pods need to communicate with the K8s API, you can use service accounts to control their access.

Inspect resource usage either with a K8s Metrics Server, or by command:

> kubectl top pod --sort-by <JSONPATH> --selector <selector>

Here is a good guide to install metrics server and dashboard (e.g. on docker-desktop).

Pods and Containers

ConfigMaps: store data in key-value map.
Secrets: same as ConfigMaps but for sensitive data only
Two ways to pass ConfigMap and Secret data to your container:
- As environment variables in container operating system
- As files presented on mounted volumes in container file system.
Container Resource management:
- Resource requests: K8s scheduler will use resource requests to avoid scheduling pods on nodes that do not have enough available resources. 1 CPU unit = 1/1000 of one core
- Resource limits: allow you to limit the amount of resources your containers can use. The container runtime is responsible for enforcement. The enforcement behaviour is different. For example, some terminates container that attempts to use more resource than the limit.
Probes
- Liveness Probe: automatically determine whether or not a container application is in a healthy state. By default K8s does not consider a container to be down until the container process stops. Liveness Probe allow you to customize this detection mechanism and make it more sophisticated.
- Startup Probes: similar to liveness probes. However, while liveness probes run constantly on a schedule, startup probes run at container startup and stop running once they succeed. Startup probes are used to determine when the application has successfully started up. It is especially useful for legacy applications that can have long startup times.
- Readiness Probes: determine when a container is ready to accept requests. When you have a service backed by multiple container endpoints, user traffic will not be sent to a particular pod until its containers have all passed the readiness checks defined by their readinesse probes. Use readiness probes to prevent user traffic from being sent to pods that are still in the process of starting up.

Restart policy for self-healing pods
- (default) Always: container will always be restarted if they stop, even if they completed successfully (returned 0).
- OnFailure: container will be restarted if the container process exists with an error code, or the container is determined to be unhealthy by a liveness probe.
- Never: let it be
Multi-container pods:
- containers share the same networking namespace and can communicate with one another on any port, even if the port is not exposed to the cluster
- Container can use volumes to share data in a Pod. Example: a legacy application is hard-coded to write log output to a file on disk. You use a sidecar container to read the log file from shared volume and prints it to the console so the log output will appear in the container log.
Init containers: containers that run once during the startup process of a pod. A pod can have any number of init containers, and they will each run once into completion, before the next init container starts. You may use init containers to perform a variety of startup tasks, they can contain and use software and setup scripts that are not needed by your main containers. They are often useful in keeping your main containers lighter and more secure by offloading startup tasks to a separate container. Use case include:
- cause a pod to wait for another K8s resource to be created before finishing startup
- perform sensitive startup steps securely outside of app containers
- populate data into a shared volume at startup
- communicate with another service at startup
Scheduling: Scheduler (a component in control plane) assigns Pods to a suitable Node so kubelets can run them. The factor taken into account:
- resource request vs available node resources
- various configurations that affect scheduling using node labels
Pod allocation
- nodeSelector is an attribute of Pod to allow you to limit which Node(s) the Pod can be scheduled on. The selector is based on label.
- nodeName is an attribute of Pod that allows you to bypass scheduling and assign Pod to a specific Node by name.
DaemonSet: automatically runs a copy of a Pod on each node. When a new node is added to the clsuter, DaemonSet will run a new copy of the Pod on it. DaemonSets also respect normal scheduling rules around node labels, taints and tolerations. If a pod would not normally be scheduled on a node, a DaemonSet will not create a copy of the Pod on that node.
Static Pod: A Pod that is managed directly by the kubelet on a node, not by the K8s API server. They can run even if there is not K8s API server present. Kubelet automatically creates static Pods from YAML manifest files located in the manifest path on the node.
Mirror Pod: Kubelet will create a mirror Pod for each static Pod. Mirror Pods allow you to see the status of the static Pod via the K8s API, but you cannot change or manage them via the API.
Taints: applied to nodes to repel a set of pods. A taint specifies key-value and effect. Effect can be NoSchedule or NoEffect. The former prevents pods without matching tolerations to schedule to the tainted node. The latter also evicts pre-existing pods with no matching toleration.
Tolerations: applied to pods so they can be scheduled to nodes with matching taints. A toleration consists of key-value pair, effect and operation. The operation can be Equal or Exists. To determine whether a toleration matches a taint. The keys and the effects must be the same. In addition:
- the operator is Exists (and thus no value should be specified in the toleration); or
- the operator is Equal, and all the values match those of the taints;

Now we have three ways to influence the scheduling behaviour. The first, is simply by specifying nodeSelector on the Pod, with the required the node label. The second, as just discussed, is to use Taints and Tolerations. The third way, is similar to the first, using nodeAffinity attributes on Pods. nodeAffinity is more powerful and flexible than nodeSelector by supporting more complex scheduling rules (e.g. matching rules).

Use Node Affinity when your scheduling rule is based on direct condition, i.e. schedule a Pod to this Node when XXX. In this case, you have well-known labels on nodes, and specify nodeAffinity on Pods.
Use Taints and Tolerations when your scheduling rule is based on inverse statement, i.e. do not schedule a Pod to this Node unless XXX. In this case, you put a taint “MyCondition:NoSchedule” on a Node, so that no Pod will ever get scheduled to this Node. The only exception is when a Pod has the Toleration “MyCondition:NoSchedule”.

Deployments

Deployment is an object that defines a desired state for a ReplicaSet (a set of replica Pods). The Deployment Controller seeks to maintain the desired state by creating, deleting, and replacing Pods with new configurations.
With Deployments, you can horizontally scale an application up and down by changing the number of replicas. You can perform rolling updates and rollback.

Networking

The K8s network model defines how Pods communicate with each other, regardless of which Node they are running on.
Each Pod has its own unique IP address within the cluster. Any Pod can reach any other Pod using that Pod’s IP address. This creates a virtual network that allows Pods to easily communicate with each other.
One type of K8s network plugin is CNI plugin. It has many flavours such as Calico. Each plugin has its own unique installation process. Kubenetes nodes will remain NotReady until a network plugin is installed.
The K8s virtual network uses a DNS (e.g. a Kubeadm cluster uses CoreDNS pod in kube-system namespace) to allow Pods to locate other Pods and Services using domain names. The Pod DNS name follows this format: pod-ip-address.namespace.pod.cluster.local
A K8s NetworkPolicy is an object that allows you to control the flow of network communication to and from Pods so you can isolate traffic. NetworkPolicy can apply to Ingress (using from selector), Egress (using to selector) or both.
NetworkPolicy has an attribute podSelector to determine to which Pods in the namespace the NetworkPolicy applies, by selecting Pods by with Pod labels.
By default, Pods are considered non-isolated and completed open to all communication. If any NetworkPolidy selects a Pod, the Pod is considered isolated and will only be open to traffic allowed by NetworkPolicies.
A variety of selector can be used: podSelector, namespaceSelector, ipBlockSelector and port.

Services

Services provide a way to expose an application running as a set of pods, so clients can access applications in an abstract way without needing to be aware of the application pods. In this model, client make requests to a Service, which routes traffic to its pods in a load-balanced fashion
Endpoints are the backend entities to which Services route traffic. If there are multiple Pods behind a service, each Pod will have an endpoint associated with the service.
Each service has a type that determines how and where service will expose your application.
- ClusterIP: expose application inside the cluster network
- NodePort: expose application outside the cluster network
- LoadBalancer: expose application outside thecluster network, but use an extermal cloud load balancer from cloud platform.
Services are assigned with DNS names. The FQDN follows this format: service.namespace.svc.cluster-domain.example, which is used by pods across namespaces
Pods within the same namespace can reference service simply by service name.
To manage external access to service, you can also use Ingress object. Ingress object is capable of providing more functionality than a simple NodePort Service, such as SSL termination, advanced load balancing, or name-based virtual hosting. You must install one or more Ingerss controller (many different implementations) to back up the ingress objects.
Ingress defines a set of routing rules. Each rule has a set of paths, each with a backend. Requests matching a path will be routed to its associated backend.
If a Service uses a named port, an ingress can also use the port’s name (instead of port number) to choose to which port of a service it will route.

Storage

Volumes allow you to store data outside the container file system, while allowing the container to access the data at runtime. When Pod is gone, volumes do not persist.
Persistent Volumes are a slightly more advanced form of Volume. They allow you to treat storage as an abstract resource and consume it in Pods. PV can be provisioned separately by storage administrator, and they persist regardless of pod lifecycle. PV needs to be claimed by pods. PV uses a set of attributes to describe the underlying storage resource.
Both volumes and PVs each have a volume type: NFS, Cloud (AWS, Azure, GCP), ConfigMaps and Secrets, Simple Directory on node
Two volume types to distinguish:
- hostPath: store data in a specified directory on K8s node
- emptyDir: store data in dynamically created location on the node. The directory exists only as long as the Pod exists on the node. The directory and the data are deleted as Pod is removed. This type is useful for simply sharing data between containers in the same pod.
Both volumes and PVs are specified under Pod, and individual containers must include volumeMounts object to map volume name to local mountPath
Storage Class object allow K8s admins to specify the types of storage services they offer on their platform. A key property is allowVolumeExpansion. This allows PVC to resize. At storage class level, there are two reclaim policies: Retain and Delete. The default is Delete.
PV has an attribute named persistentVolumeReclaimPolicy. This is reclaim policy at PV level. If the attribute is not defined, it is inherited from storage class. The persistentVolumeReclaimPolicy has three options. When PVC is deleted:
- Retain: keeps all data but requires admin to manually reclaim the volume (i.e. delete PV, clean up data, delete storage asset)
- Delete (cloud storage only): deletes both PV and the underlying storage resource automatically
- Recycle: scrub (rm -rf /vol/) all data in the underlying storage resource, and allow the volume to be reused.
PVC represents a user’s request for storage resources. It defines a set of attributes similiar to those of a PV. When a PVC is created, it will look for a PV that is able to meet the requested criteria. If it finds one, it will automatically be bound to the PV. PVC can be mounted to a Pod’s containers just like any other volume

In general the CKA exam experience is quite positive and rewarding. In future posts I will shift focus on Kubernetes not only for the CKA exam, but also for keeping track of my learning.

Good luck with your CKA exam.

High Performance Computing

Fri, 11 Dec 2020 23:42:00 -0400

Overview

High Performance Computing (HPC) has recently been commoditized with the advent of commodity server hardware (x86 server), virtualization technology and cloud delivery model. It is common in specialized industries where intensive computing tasks are required, for example:

HCL (healthcare and life science): drug discovery, computer aided diagnosis (CAD), genome engineering;
CAD, CAE, CAM (computer aided design, engineering, and manufacturing): 3D modeling, computational fluid dynamics (CFD), finite element analysis (FEA), structural mechanical design, etc
Finance: portfolio management, automated trading, risk analysis
Geoscience and geo-engineering: oil and gas exploration, geographic data, weather forecasting;
Scientific computation

Computing performance is measured in FLOPS (floating point operations per second) and is usually delivered in a cluster to aggregate the computing power from a number of networked nodes. This is referred to as an HPC cluster.

Hardware stack

An HPC cluster features the following components:

Head node (aka master node or login node): a gateway and coordinator; head node may be broken into several nodes
Compute node (worker node): the executor of jobs; the compute node can either be homogenous or heterogeneous, for different purposes. the number of compute nodes can be quite large

There are four common form factors for server: tower, rack-mount, blade, mainframe. Traditionally, the nodes are rack-mount 1U “pizza box” servers. Bladed systems started to replace due to the increased node density, thanks to the shared/redundant power and cooling management. In the past, the HPC cluster is operated in data centres, which is an expensive operation item. In the last decades, many organizations extends their compute workload to the cloud, forming a hybrid model.

HPC typically has specialized storage system because HPC applications notoriously create large amounts of data. NFS traditionally does not scale well as number of node increases. Some proprietary storage system such as Isilon provides good performance via NFS protocol. There are also open-source parallel file system such as Lustre and HDFS.

HPC networking handles three types of traffic:

computation traffic between compute nodes (if the compute nodes interact with each other)
file system traffic: for compute nodes to read and write on file system (e.g. NFS)
administrative traffic: fairly light compared to the two above

For that, many HPC runs two networks, a private (backend) network and a public (frontend) network. Backend network must be high speed and low latency, typically in the form of 10Gig Ethernet, or InfiniBand.

Software stack

On the software layer, the core functionality is Message Passing Interface (MPI), a specification for the developers and users of message passing libraries. MPI constitutes a standardized and portable message-passing system which consists of a library and a protocol to support parallel computing. MPI enables passing information between various nodes of a HPC cluster or between particular clusters, and has different implementations that provide the libraries to run HPC applications in a distributed manner across different physical nodes.

In the operation, user submits a job through head node in order to request the resource. User needs to specify the resources for the job (e.g. how many CPU cores, how much memory, etc). The head node runs a scheduler to allocate computing resource based on pre-defined policies, based on priority of jobs, availability of resources, distribution of load, etc. Depending on the nature of the computing jobs, the nodes participating in the task may or may not communicate with one another. If they do need to talk to each other, the program must support it. Such program can be called a cluster program, and the MPI (message passing interface) library greatly facilitates the development of such program. The sub-jobs communicating with each other also creates a considerable amount of network traffic within the cluster.

Cluster software ties all nodes in the cluster together. It turns raw hardware into a functioning cluster by provisioning (installing and configuring) the head nodes. Compute nodes can usually be added or removed dynamically therefore the head nodes should be able to provision compute nodes, and administer cluster, leaving the programming as the job for the user to complete. As mentioned, in parallel programming, the most important HPC tool is MPI (Message Passing Interface), which allows programs to talk to one another over cluster networks. There are both open (e.g. Open MPI) and commercial MPI (e.g. Microsoft MPI) versions. Cluster software should also provide compilers, debuggers, and profilers in addition to MPI.

There are cluster software in both Linux and Windows operating systems: Rocks Clusters, Oscar (Open Source Clusters Application Resources), Red Hat HPC solution, Microsoft HPC pack and AWS Parallel Cluster.

Implementation

Here is an example of setting up HPC cluster with CentOS. Despite of the well documented steps, note that the author of the document refers to HPC cluster simply as cluster, which is ambiguous. There are three basic motivators for creating a cluster: high performance computing (HPC), network traffic load balancing, and service resilience in the form of high availability (HA). The author should be specific in the document about the HPC cluster. If RDMA (Infiniband) network is involved, a configuration guide is provided in RedHat literature.

Here is an example of deploying HPC cluster in AWS. Here is the guide to deploy HPC pack in Microsoft technologies.

HPC and Big Data

HPC and Big Data are two distinctive computing paradigmes. Although there is some signs of convergence and blurred boundaries, it is still a long way before one can treat HPC and Big Data interchangeably. This paper does a phenomenal job in comparing the two paradigms.

The fundamental difference lies in the respective problems they intend to address. HPC focuses on the large computational loads, whereas Big Data targets applications that need to handle very large and complex data sets (usually in the order of multi-terabytes or exabytes). Many scientific data analytics applications are becoming I/O bound in modern systems, such as seismic algorithms, Big Data applications are thus very demanding in terms of storage, to accommodate such a masive amount of data, while HPC is usualy thought more in inters of sheer computational needs. The open-source projects in Big Data also aims to run on conventional hardware to make it easier and less expensive to scale. This is not the main focus of HPC.

So, you can run Big Data (e.g. Hadoop) analytics jobs on HPC gear. On the other hand, you can’t run HPC jobs on commodity hardware as commonly seen in the Big Data stack. Both HPC and Hadoop analytics use parallel processing of data. In a Hadoop/analytics environment, data is stored on commodity hardware and distributed across multiple nodes of hardware. In HPC, where the size of data file is much greater, data storage in centralized. Also, because of the sheer volume of its files, HPC also requires more expensive networking communications such as Infiniband, because the size of the file it processes require high throughput and low latency.

In BigData job, each query in Hadoop reads data from disk and runs as a separate MapReduce job. Spark enables in-memory iterative processing (through the RDD abstraction), allowing the user to query repeatedly on a dataset without having to perform intermediate disk operations. RDD are exposed in the Spark API where each dataset is represented as a read-only object, and transformations are invoked using methods on these objects. For an example project, check out this post.

The underlying software stacks for HPC and Big Data are fundamentally different, mainly due to the differences represent in their target class of applications, as outlined in the diagram below:

software stack difference between HPC and Big Data

As to which one is for me, the over-simplified advice is: if you can avoid HPC and just use Hadoop for your analytics, do it. It is cheaper, easier, and more cloud friendly. However, bear in mind that an all-Hadoop shop is not possible for many industries such as life sciences, weather, pharmaceutical, and academic applications.

DataStax Python Driver

Sat, 27 Jun 2020 14:20:34 -0400

For someone with relational database background, analyzing data in Cassandra isn’t intuitive. There are two reasons. First, Cassandra data table is hardly updated or deleted in avoidance of tombstones. Insertion is the only action on the table resulting in multiple versions of each record all stored in the same table, thus a much longer table than its relational counterpart. Second, Cassandra schema is designed around how end-user will query the database, rather than a modelling of entity-relations. There are less fields, but some field may contain large data chunk, such as an entire XML document being stored in a column.

Data engineers with Cassandra may need to run full table scan, and extract values from wide columns of XML document by drilling down the XML tree structure, in order to produce a data frame (two-dimensional mutable, possibly heterogeneous tabular data structure with labeled rows and columns).

I’ve came across this task in the past and the duration of a full table scan on Cassandra table is in the order of hours, which is beyond what the built-in cqlsh tool can handle. I had to use Python to iterate through 200 million rows. Datastax Provides Cassandra client driver as a Python3 package, known as DataStax Python Driver. It allows us to build a simple Python3 script to complete a full table scan. The driver can be installed with pip3:

pip3 install cassandra-driver

With the driver installed, we can start to pull data from Cassandra table into Python client class. here is a basic example of how to print the rows into a file:

#! /usr/bin/python3
from cassandra.query import SimpleStatement
from cassandra.cluster import Cluster
from cassandra import ConsistencyLevel
import datetime

if __name__ == "__main__":
 cluster = Cluster(['cass_host'],port=9042,protocol_version=3)
 try:
 print (datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")+" start")
 session = cluster.connect('myownkeyspace', wait_for_all_pools=True)
 query = "SELECT * FROM mytable"
 statement = SimpleStatement(query, fetch_size=50, consistency_level=ConsistencyLevel.ONE)
 csv_file = open('result.csv','w',8192)
 csv_file.write("header")
 for tbrow in session.execute(statement,timeout=2.0):
 csv_file.write(tbrow.user_id+"\n")
 except Exception as ex:
 print(ex)
 except KeyboardInterrupt:
 print("Task Interrupted by SIGINT.")
 finally:
 cluster.shutdown()
 csv_file.close()
 print (datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")+" finish")

Note that the fetch_size can be set to larger number, but it may increase the chance of server read timeout (code=1200) in the middle of execution.

The processing logic can be implemented in the loop while each record in the table is being pulled out. The logic is repeated for every row so it will have a significant impact on the overall execution time.

Some the data needs to be ported into pandas data frame for further engineering, instead of being printed out to file. The following snippet will do the trick:

#! /usr/bin/python3
from cassandra.query import SimpleStatement
from cassandra.cluster import Cluster
from cassandra import ConsistencyLevel
import datetime
import pandas as pd

def pandas_factory(colnames,rows):
 res = []
 res.append(pd.DataFrame(rows, columns=colnames))
 return res

if __name__ == "__main__":
 cluster = Cluster(['cass_host'],port=9042,protocol_version=3)
 try:
 print (datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")+" start")
 session = cluster.connect('myownkeyspace', wait_for_all_pools=True)
 session.row_factory = pandas_factory
 query = "SELECT * FROM mytable"

 statement = SimpleStatement(query, consistency_level=ConsistencyLevel.ONE,fetch_size=50)
 df=pd.DataFrame()
 for tbrow in session.execute(statement):
 df=df.append(tbrow.user_id,ignore_index=True)
 except Exception as ex:
 print(ex)
 except KeyboardInterrupt:
 print("Task Interrupted by SIGINT.")
 finally:
 cluster.shutdown()
 print (datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")+" finish")

In my test environment with 180 million rows in the table, the execution of the first script takes 37 minutes (of course there’s a lot of factors at play). I experimented several approaches to improve the speed, such as tuning the buffering options for file write. However, It turns out that the speed bottleneck of the script is not even file IO, but rather pulling data out of Cassandra.

The output can be stored as CSV file, which can be lately loaded to relational database for analysis. PostgreSQL would be a good open-source choice because it is both transactional and analytical.