Nodes on Digi Hunch

Kubernetes Platform as a Service and Red Hat OpenShift

Sun, 25 Jun 2023 11:10:15 -0400

The Three-layer model

Kubernetes is so complex that it becomes a buzz word itself. I categorize the related work into three layers: a cluster layer, a platform layer and an application layer, by their purposes. The three layers are illustrated as below:

Let’s examine each layer in this model and where the Kubernetes Platform as a Service fits in.

The Kubernetes Cluster Layer

At the bottom, the Kubernetes Cluster layer is the foundational layer. It focus on using self-hosted VMs or cloud resources to build a functional Kubernetes cluster and worker node groups. A functional cluster includes a highly available control plane, as well as scalable node groups that all communicate with the control plane. Cloud Service Providers like AWS and Azure provides managed Kubernetes service, which takes away the complexity (and flexibility as well) of managing control plane components such as etcd store and API server. The managed services also automatically provisions computing nodes and join them into the cluster. The cluster layer may also involve integration with of CNI and CSI, to ensure Pod-to-Pod communication and available storage classes.

Professionals working at this layer are infrastructure experts who understand networking, storage, as well as how to manage cloud resources or VMs, infrastructure as code. On a daily basis, they deal with VPCs/V-Nets, subnets, EBS/Azure Disk, File storage, EC2/Azure VMs, etc. When the team is doing a bad job at this layer, you might see symptoms like unresponsive cluster API, orphaned worker nodes, or kubectl failing to connect to cluster endpoint.

The tenants (applications) of the Kubernetes platform does not directly interact with this layer. If you decide to switch CSP vendor, this layer requires 100% re-engineering because the managed Kubernetes service by each CSP is different.

The Kubernetes Platform Layer

The Platform layer sits in the middle. When organization decides to adopt Kubernetes, they often underestimate the efforts required in this layer. This layer works on a functional cluster, without directly interacting with the underlying cloud resources. This layer involves any Kubernetes abstractions that do not creates tangible business value. Rather, this layer is an enabler. It allows the applications to deploy smoothly, evolve quickly, and more importantly, focus on the business.

Teams working on this layer needs to be Kubernetes experts. On a daily basis, they play with common CNCF toolings, such as Prometheus, ArgoCD, Istio, Cilium, Tekton, Open Policy Agent, etc. They are comfortable with Operators, Helm Charts, Ingress, etc. Inside of the Kubernetes cluster, they also manage the foundational services such as Event streaming (e.g. Kafka), PostgreSQL database (e.g. PostgreSQL), software-defined storage (e.g. Ceph), service mesh (e.g. Istio), Authentication (e.g. Keykloak) , etc. These services act as the infrastructure layer to the business workload. If the team is doing a bad job, you would see data loss with database, observability service not populating data, ingress does not process request, etc.

The tenants (application) share services in this layer. If you decide to switch CSP vendor. I estimate 80% of the work at this layer is portable, and 20% requires re-engineering. That is because each CSP offers different external resources, therefor the low level Kubernetes objects in this layer, such as storage classes, load balancers, supported CNIs are different. High level objects such as Kafka remains portable across platforms.

The Application Layer

The next layer at the top is application layer. Workloads in this layer are directly linked to the business value. The applications are very diverse. Most of the time, the release team is the main player at this layer. If the organization develops its own application, the software development team also work at this layer.

In terms of knowledge, the members of development team are experts in software engineering, and Software Development Life Cycle (SDLC), etc. On a daily basis, they deal with programming languages, product development, build and release. If they screw up their work, expect business errors, such as orders sent to wrong client, incorrect balance sheet, etc. This team has high visibility in the organization due to its direct link to business value.

This layer of work involves multiple tenants. Each tenant is isolated within their own namespace. When you switch CSP vendor, this layer should be readily portable with minimal effort.

It is also worth noting that, with solid platform and cluster layers, the team working at this layer do not write bespoke code for networking, observability, authentication and authorization, encryption and many other aspects not relevant to the core business. Once deployed, the application services are resilient, scale to demands, and cost efficient. This layer reaps the benefits of Kubernetes.

Kubernetes Platform as a Service

As the Kubernetes dust is still settling, a builder’s title may not always reflect which layer she or he focuses on. Today it is pretty common for infrastructure engineers to expand their role into the platform layer, or likewise, a software engineer to drill down to the platform layer.

The boundary between platform layer and cluster layer is clear. The cluster layer deals with underlying infrastructure, either in the cloud or on premise. They abstract away the complex infrastructure world from those working with the platform layer.

The boundary between platform layer and application layer is a little tricky to articulate. The application layer focuses on implementing the business logics. The platform layer takes care of the functions that are not part of business logic but essential to the business application. Take an HTTP request for example, application developer should not have to write code to terminate TLS (not part of business logic). They should only write the code to process the HTTP request (business logic). TLS termination is delegated to an Ingress, to be configured by platform builders.

The folks working at the Platform layer needs to interface with both sides. They provide Platform as a Service to the Application teams. However, their work appears mostly invisible in an organization. Their effort is oftentimes underestimated. There are several reasons for that. First, the platform layer does not directly create tangible business value. They are just someone else’s enabler. Second, their building blocks involve a lot of abstractions by Kubernetes API. Third, the idea of platform engineering is newly emerged. There hasn’t been a populous recognition of its value.

Red Hat OpenShift Container Platform

The platform team builds the platform with their choice of open-source tools. For clusters using OpenShift Kubernetes Engine, Red Hat introduces Open Shift container platform consisting of Red Hat’s opinionated (but validated) choice of toolings, for example:

OpenShift Service Mesh: Istio
OpenShift Streams: Apache Kafka
OpenShift GitOps: ArgoCD
OpenShift Container Platform Pipelines: Tekton
OpenShift Serverless: Knative
OpenShift Data Foundation: Ceph

Clients building their clusters with OpenShift Kubernetes Engine may build their own platform with the toolings in the OpenShift enterprise Kubernetes container platform. For more services, check out the documentation for OpenShift Container Platform. For customers with OpenShift Kubernetes Engine, their options to DIY platform are:

Entry-Level: Red Hat OpenShift Kubernetes Engine: Enterprise Kubernetes distribution on RHEL CoreOS
Mid-Level: Red Hat OpenShift Container Platform (RHOCP):
Plus-Level: Red Hat OpenShift Platform Plus: RHOCP + advanced cluster management, security, data management essentials, enterprise container registry

OpenShift runs the business model of Kubernetes PaaS.This is a unique business model that I do not find a matching competitor. Even if you choose to DIY your own platform, the Red Hat’s choices are still a great reference. The OpenShift enterprise Kubernetes container platform maps perfectly to the platform layer of the three-layer model, aiming to simplify the work in the platform layer.

Managed RedHat OpenShift

At first, the OpenShift container platform started as a value add-on to the Kubernetes Engine. Now it’s a separate product line in their business model. In the mean time, OpenShift partners with major CSPs, to develop the cloud service editions, including:

Red Hat OpenShift on AWS (ROSA)
Microsoft Azure Red Hat OpenShift (ARO)
Red Hat OpenShift Dedicated – on AWS and GCP
Red Hat OpenShift on IBM Cloud

These offerings are managed Kubernetes Platform as a Service in the cloud. Since RedHat is the only player in this model, we can refer to them as managed OpenShift services. In addition to an already-confusing world of Kubernetes platform portfolios, these offerings gives consumers even more options. On AWS for example, users have the following options:

Managed Platform: OpenShift Dedicated, managed by Red Hat
Managed Platform: Red Hat OpenShift Service on AWS (ROSA), managed by Red Hat and AWS
Self-built cluster: OpenShift Container Platform

This video discussed more details about these options, such as support model. It is also worth noting that these options tend to be much pricier than managed clusters such as EKS and AKS.

Since a Managed RedHat Platform makes it easy to deploy, let’s take ROSA as an example and create a cluster. To enable ROSA in AWS console, click on “Getting Started”. The next page ensures ROSA is enabled and checks other prerequisite such as meeting service quotas and creating ELB service-linked role, as show below:

Now, with an AWS account (and ROSA enabled), a RedHat account, and the rosa-cli utility, we can create a cluster with just a few commands. As a note, be wary of the cost and do not forget to delete the cluster afterwards.

Create a ROSA cluster

With the following set of commands, we can kick off cluster creation, using STS. We can bring our own VPC, so long as it meets certain prerequisites. I use the Terraform template in the vpc-base project, to create the underlying VPC. We’ll need the followings from this template:

The CIDR range of the VPC: as input with a default
The subnet Ids of the private subnet to place, printed in the output

The subnets are private subnets, because we want to provision the cluster with private node and private endpoint. When we use rosa CLI, we provide the CIDR and subnet IDs.

# start with AWS cli configured to the correct profile
rosa login # with redhat account and past token
rosa create account-roles --mode auto -y # this command creates the IAM roles ManagedOpenShift-*-Role, with RedHat account as trust entity
rosa verify permissions # optional
rosa verify quota # optional

export ROSA_CLUSTER_NAME="dhc" \
OPENSHIFT_VERSION=4.13.4 \
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) \
AWS_DEFAULT_REGION=us-east-1

rosa create cluster --sts --private \
 --cluster-name $ROSA_CLUSTER_NAME \
 --multi-az \
 --private-link \
 --region $AWS_DEFAULT_REGION \
 --version $OPENSHIFT_VERSION \
 --enable-autoscaling \
 --min-replicas 3 \
 --max-replicas 3 \
 --compute-machine-type m5.xlarge \
 --machine-cidr 147.206.0.0/16 \
 --subnet-ids subnet-052852a1fb4d7d2ad,subnet-06d8d40ae39d55c47,subnet-0f67ce08bc588012c

The CLI will pick up the correct VPC by CIDR, and prompt you to confirm creation of private cluster. After the command kicks off, it will wait for OIDC provider creation, and role creation. Then it uses a Terraform template to create the related resources including VPC. Use this command to check status:

rosa list clusters
rosa describe cluster -c dhc

When the second command displays the state of waiting (Waiting for OIDC configuration), we can create OIDC provider:

rosa create operator-roles -c $ROSA_CLUSTER_NAME --mode auto --yes
rosa create oidc-provider -c $ROSA_CLUSTER_NAME --mode auto --yes

Throughout the process, we can monitor the install log (terraform output) with:

rosa logs install -c dhc --watch

In the log, you might see errors with terminals connecting to the terraform backend, which doesn’t necessarily indicate a cluster creation error. Always check the cluster state until it reports success.

Kick the tires

Eventually the describe cluster command will show ready state. We can now create an admin user:

rosa create admin -c $ROSA_CLUSTER_NAME

The command above prints an oc command (OpenShift CLI, equivalent to kubectl) with password to log in. Let’s examine the cluster with oc. Because it is a private cluster, the endpoint is not available publicly. However, it is accessible from the Bastion host. Use the SSM Session Manager technique from my previous post to SSH to the Bastion Host, which should have oc installed. To install oc yourself, use HomeBrew on Mac. On Linux or Windows, log on to OpenShift console, go to Downloads on the left pannel and find it out under CLI tools.

The oc command may report insecure TLS on the login URL. Wait for a few minutes for the certificate to come off as safe. Once you run the oc command with password, it should return “Login successful” and then we can connect to the cluster:

$ oc get node # or kubectl get node
NAME STATUS ROLES AGE VERSION
ip-147-206-135-41.ec2.internal Ready,SchedulingDisabled infra,worker 3m5s v1.26.5+7d22122
ip-147-206-155-141.ec2.internal Ready control-plane,master 25m v1.26.5+7d22122
ip-147-206-156-81.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-164-21.ec2.internal Ready infra,worker 3m3s v1.26.5+7d22122
ip-147-206-179-90.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-191-118.ec2.internal Ready,SchedulingDisabled control-plane,master 26m v1.26.5+7d22122
ip-147-206-192-232.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-193-198.ec2.internal Ready infra,worker 3m20s v1.26.5+7d22122
ip-147-206-218-114.ec2.internal Ready control-plane,master 26m v1.26.5+7d22122

You can use oc the same way you’d use kubectl. Both works through SOCK5 proxy. In the meantime, log in to the RedHat console with your Red Hat credential, you can see the cluster in Ready state as well:

The rosa create admin command creates a htpasswd type (username-password) of identity provider (IdP) with a user named cluster-admin and a preset password. In real life however, we often configure third party IdP with OIDC integration. I’ll have to leave this to the next blog post.

We shall see the nodes as EC2 instances from AWS console as well. Note that there are three roles: control-plane, worker and infra. The infra nodes are for infrastructure services. These services (Ingress Controller, GitOps, Pipeliens) are the ones in the platform player as we discussed above.

There are many customizations you can make in this installation process and I’d have to defer to the ROSA documentation. To clean up, use the following ROSA command:

rosa remove cluster -c dhc

The output also gives you the command to delete operator roles and OIDC provider, for example:

rosa delete operator-roles -c 23o4u3j98tqmlbtjo612opb7a4bbim5f --mode auto --yes
rosa delete oidc-provider -c 23o4u3j98tqmlbtjo612opb7a4bbim5f --mode auto --yes

Then we can destroy the VPCs using terraform.

ROSA with HCP

Update Oct 2023:

The deployment above provisioned a few nodes for control plane, which add to the overall time to provision a cluster. In Aug 2023, there is a new option Hosted Control Plane (HCP) that came to allow users to provision a hosted control plane. This results in cost savings and shorter time to provision a cluster. Here is a table of comparison between the ROSA with HCP and ROSA classic.

Final words

In this post, I discussed the three-layer model and pointed out that platform layer isn’t as visible as the other two. I also experimented ROSA as a turn-key Kubernetes platform with its opinionated stack of services.

Some misinformed organizations even skip the entire platform layer in their estimate of effort. They build a cluster, ran a hello-world service and assumes they can start putting applications on the Kubernetes cluster. There are also customers who purchased the entire Managed OpenShift platform but only use it as a cluster. Yikes!

The concept of Kubernetes platform, or generally platform engineering is still spreading. The consulting team that I worked in full-time last year re-branded itself as platform engineering. Marketings are pushing it. Builders are doing it. We’ll keep an eye, on whether customers are buying it.

Connect kubectl to private Kubernetes cluster in EKS and AKS

Sat, 10 Jun 2023 19:31:00 -0400

Managed Kubernetes services give user a cluster endpoint and a number of worker nodes, with the choice. For each access, users have the choice of making them publicly available, or keeping them on private networking. In my opinion, any deployment beyond personal hobbies, should use Kubernetes private cluster, with both cluster endpoint and worker nodes on private subnet. There is no reason to expose computing nodes or Kubernetes management traffic publicly.

For worker nodes, it is fairly easy to put VMs on private network, but many companies still have the cluster endpoint exposed publicly. There are usually two reasons. First, their CI/CD agent is hosted somewhere else on the Internet (instead of on private network with private connectivity to Kubernetes cluster) and need to access Kubernetes cluster endpoint. Second, when the cluster needs to connect with third-party identity provider as OIDC provider, a two-way communication is needed.

There is a classic pattern of using a public bastion host (jump box), with a bastion host on the public subnet, routable to the private endpoint of managed Kubernetes service. Clients then connect to the bastion host via port 22 on a public IP address. The authentication is based on SSH key pair, or worse, password. The port forwarding (aka SSH tunnelling) capability enables all the magics.

Exposing a jump box in the public subnet with RSA key authentication is still not favourable. In this post, I’ll examine some secure patterns to connect to private endpoint with improved security posture.

AWS options

There are two problems. First, how to establish connectivity to the Bastion host in a private subnet. Second, how to use the Bastion host to proxy traffic to the cluster endpoint also in private subnet. To the first problem, there are two potential solutions: SSM Session Manager, and EC2 Instance Connect (EIC) with EIC endpoint (EICE).

SSM Session Manager was introduce in 2018. It runs an agent on the EC2, which initiates a connection to the SSM endpoint on the AWS side. This connection enables not only Session Manager, but also other Systems Managers (SSM) services such as Fleet Manager, Patch Manager and State Manager. The problem that session manager originally addresses is server management.

AWS launched EC2 Instance Connect (EIC) in 2019, and EIC Endpoint (EICE) in 2023. EIC addresses the problem with managing SSH key pairs at scale. It dynamically generates an SSH key pair for server access, based on IAM permission. However, it still requires an instance to have its SSH port publicly accessible. With EICE, it is no longer a requirement. In the diagram, EICE is placed in a private subnet, allowing EICE service to reach private instances at their SSH port.

Here is a comparison of the two:

	EC2 Instance Connect (EIC) with EIC Endpoint	SSM Session Manager
Location of Bastion host	Private Subnet.	Private Subnet
Need Ingress Port	Yes. Port 22 must open to the endpoint.	No. SSM agent initiate outbound connection from the instance
Traffic Path	AWS CLI → AWS EIC ES → EICE→EC2 Inst	AWS CLI → AWS SSM ES → SSM ← EC2 Inst
Authentication	AWS IAM and ephemeral SSH key when using AWS CLI directly AWS IAM and long-term SSH key when using SSH proxy command	AWS IAM and long-term SSH key when using AWS CLI directly or SSH proxy command
Work with OpenSSH	Yes	Yes
Cost	There is no additional cost for using EIC.	No additional cost, unless private SSM Endpoint.

Let’s take a look at each option.

EC2 Instance Connect

To use EIC, pick an AMI that has it pre-installed and ensure instance profile has correct policy, as the document states here. AWC CLI will make use of local OpenSSL client. So make sure there connection at port 22 is open. To make it work with EC2 instance on a private subnet, create an EC2 Instance Connect Endpoint on the VPC, and ensure that the security group of EC2 allows port 22 from the Endpoint. Run this command:

$ aws ec2-instance-connect ssh --instance-id i-00ea30a6e02db33fe

The command above simply generates a key pair internally, add the public key to the server side, and connect with SSH from the client side. The command takes you to an SSH session. Checking ps -ef | grep ssh on the client machine, you can see the full parameter of SSH, including the location of the ephemeral private key.

However, if you use AWS CLI open-tunnel as proxy command to ssh, then you’d still have to use the key pair used to create the EC2 instance. As suggested at the bottom of this blog post, the command is:

$ ssh ec2-user@[INSTANCE] -i [SSH-KEY] -o ProxyCommand='aws ec2-instance-connect open-tunnel --instance-id %h'

This is a bummer, because with native SSH tool you do not get the primary benefit of EIC – ephemeral key pair.

SSM Session Manager

Now let’s look at SSM session manager. Similarly, it needs an agent installed and IAM role configured. You can connect to from web console but more importantly, from AWS CLI:

$ aws ssm start-session --target i-0531b19bec8ad022d

This command takes you to an SSH session with user ssm-user, without starting an OpenSSH client process locally. User do not have to manage key pair. There is also a document about using this command as proxy command, which uses an SSM document. I have one of the SSH config entry as:

host i-* mi-*
 ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
 User ec2-user
 IdentityFile ~/.ssh/id_rsa

This allows me to directly SSH to client using OpenSSL client (e.g. ssh i-0531b19bec8ad022d) by Instance ID. With this, I also need to specify my own OS user and matching private key.

I know I will use the OpenSSH client a lot from pipelines because it is very powerful. In both options, I have to live with managing key pairs myself. With SSM session manager’s proxy command, the instance does not need port 22 to open, which is a great advantage, in terms of security and operation. SSM Session Manager is a winner.

SOCKS5 proxy for kubectl

Either SSM Session Manager or EIC with EICE enables an SSH tunnel with key encryption between client (a local computer or a pipeline agent). On top of the SSH tunnel, we can build a SOCKS5 proxy. Kubernetes document has a good page on how to do this. I managed to get this to work with a few gotchas.

To put this in practice, I first created a VPC stack with a bastion host using terraform template from my vpc-base project. The terraform output will give the next set of commands to run to create a private cluster, using a manifest rendered from the file private-cluster.yaml.tmpl:

# cd aws_vpc
# terraform init
# terraform plan
# terraform apply
# ... run the given command ...
# envsubst < private-cluster.yaml.tmpl | tee | eksctl create cluster -f -

# Run this from a remote host without access to cluster endpoint.
# Run terraform apply and terraform output contains the variables needed for the next steps

# the command below may take 15 minutes to create a private cluster

eksctl create cluster -f private-cluster.yaml 

aws eks update-kubeconfig --name private-cluster

At this point, the kubeconfig file has been updated, but kubectl (from Internet or on-prem) is unable to connect to cluster endpoint (on private network). In order to

BASTION_SECURITY_GROUP_ID=$(terraform output -raw bastion_sg_id)
CLUSTER_SECURITY_GROUP_ID=$(aws eks describe-cluster --name private-cluster --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output text)

# In Cluster Endpoint's security group, open up port 443 to Bastion host
aws ec2 authorize-security-group-ingress --group-id $CLUSTER_SECURITY_GROUP_ID --source-group $BASTION_SECURITY_GROUP_ID --protocol tcp --port 443

# Test with connecting to Bastion host with ssh i-0750643179667a5b6, assuming .ssh/config file is configured as above. From the bastion host, you can test:
# curl -k https://EC5405EE1846F19F9F61ED28FB12A6A9.sk1.us-west-2.eks.amazonaws.com/api 
# if you get an HTTP response, even an error code 403, the bastion host has TCP connectivity to cluster endpoint

# then we can start an SSH session as a SOCKS5 proxy on the remote host
ssh -D 1080 -q -N i-0750643179667a5b6

# add > /dev/null 2>&1 & to push it to background, or use ctrl+z after running the command
# to validate that the SOCKS5 proxy is working, you can run the same curl command with a proxy parameter:
# curl -k https://EC5405EE1846F19F9F61ED28FB12A6A9.sk1.us-west-2.eks.amazonaws.com/api --proxy socks5://localhost:1080

# you can instruct kubectl to use the SOCKS5 proxy with the following environment variable
export HTTPS_PROXY=socks5://localhost:1080

kubectl get node

# alternatively, add "proxy-url: socks5://localhost:1080" below server attribute in ~/.kube/config file.

There are some pitfalls to watch for. On the remote host both ssh command and kubectl command implicitly uses AWS CLI. Therefore, make sure the profile and IAM role are correctly configured. For example, if SSM agent requires one IAM role, and kubectl is created with another IAM role, then make sure AWS CLI assumes the correct IAM role using environment variables, and use “aws sts get-caller-identity” to validate the IAM identity being used.

What about AKS in Azure

I touched on this in my Azure notes in 2021 and did a research again. Unfortunately, options are still fairly limited.

The first option is to use a managed service called “Azure Bastion”, which requires public IP and a dedicated subnet with the exact name of AzureBastionSubnet, as well as some additional requirement. I’m not impressed with these requirement because it is meant to be a managed service. The other option, is essentially to DIY a JumpBox. The idea is the same: put the jumpbox in a public subnet, which is routable to private subnets. When you need to connect to private VMs, get to the jumpbox first.

Apart from having to put the bastion VM on a public subnet, the pattern that we discussed above involving SOCKS5 proxy still works. Exposing a bastion host isn’t ideal but it still reduces attack surface significantly, comparing to exposing the cluster endpoints of all Kubernetes API servers.

Summary

Many immature Kubernetes configurations exposes private endpoint publicly. Having cluster endpoint in private subnet greatly improves security posture. In my opinion, there are very few situations where cluster endpoint must exposed publicly. Having private endpoint should be mandatory for all Kubernetes cluster. In the next post, I also cover how to create a ROSA cluster with private endpoint.

Optimize CPU and Memory for Kubernetes Pod

Fri, 13 Jan 2023 11:47:00 -0400

When optimizing workload performance, it is important to understand how on earth operating system allocates CPU and memory to processes. This helps understand how to set resource limit Kubernetes Pod in an optimal way.

CPU resource assignment

The OS distributes CPU resource to processes by the unit of time share of CPU time. Most of the time, many processes with CPU instructions (machine code) are waiting in the Job queue, for their share of CPU time in order to execute their instructions. As soon as CPU becomes idle, the CPU scheduler selects a process from the ready queue to run next:

Ideally, OS should schedule CPU in a way that it should not waste any CPU cycle. It should also minimizes waiting time and response time of processes. At a high level, there are two types of CPU scheduling:

Preemptive: OS allocate CPU resources to a process for only a limited period of time and then takes those resources back. It could interrupt a running process to execute a higher priority process.
Non-preemptive: New processes are executed only after the current executing process has completed its execution.

Here is more information about preemptive and non-preemptive scheduling.

CPU is compressible resource in Linux

In the Linux world, all scheduling is preemptive. We also call it kernel preemption. As the wikipedia entry states: the scheduler is permitted to forcibly perform a context switch (on behalf of a runnable and higher-priority process) on a driver or other part of the kernel during its execution, rather than co-operatively waiting for the driver or kernel function (such as a system call) to complete its execution and return control of the processor to the scheduler when done.

The Linux scheduler implements a number of scheduling policies, which determine when and for how long a thread runs on a particular CPU core. The scheduling policies in RHEL include real time policies such as SCHED_FIFO and SCHED_RR where processes have a sched_priority value in the range of 1 (low) to 99 (high); and normal policies such as SCHED_OTHER, SCHED_BATCH and SCHED_IDLE, where sched_priority (specified as 0) is not used in scheduling decisions.

It is important to understand preemptive CPU scheduling on Linux. When OS allocate CPU resource to a process for one time slot, it is not committed to the same process for the next time slot. The OS reserves the ability to revoke the next CPU use and re-assign it for processes of higher priority.

Because of this, we regard CPU as a compressible resource. The compressible characteristic impacts how we optimize CPU utilization for a process, including setting CPU request and limit for Kubernetes workload.

Memory is non-compressible resource

A few years ago, I discussed how to calculate memory usage. A process requests memory from OS using memory allocation functions (the malloc family), and return memory to OS using free functions. The design of Linux OS knows that processes have a tendency to request more memory than they use, which causes under-utilization. In combat against under-utilization, the Linux OS supports memory overcommitment (on by default), allowing processes to request more memory than what is available. The processes have access to virtual memory space and the OS may swap some pages out to disks. The overcommitment mechanism also prevents processes from crashing due to insufficient memory assignment. The kernel can also OOM kill a process when the entire system is in a crisis.

Memory is non-compressible resource. When OS assigns memory pages to a process, the process has to right to keep those pages, until the OS takes them away. Unlike assigning CPU cycles, the assignment of memory pages to processes does not have an expiry time. This is the non-compressible characteristic of memory assignment.

CPU limit and requests for Kubernetes workload

It was considered best practice to set request and limit for memory and CPU. However, knowing CPU is compressible resource and memory isn’t, we should re-consider this practice. In short, for CPU, we should set request only, without setting limit. For memory, we should set limit to exactly the same as request.

A process has different level of demands for CPU at different times. Depending on the activity in the process, the level of demand can even be spiky. If there is a lot of iowait, it may not need a lot of CPU. But when there are lots of computing-bound activities, the program is CPU-thirsty as it is programmed to to more. The last thing we want is to throttle the CPU use for a process in such legit situations. When throttling happens, the process does not get sufficient time share of CPU time. At the platform level, we can’t control when the Pod (process) gets busy. The best thing it can do, is trying to fit more CPU time shares to this process when it becomes CPU thirsty. When we apply a limit of CPU in workload setting, we are potentially throttling the CPU use for a process at the times it needs more CPU time shares, which is counter-productive.

We should still configure CPU request, so that kube-scheduler factors it in when scheduling multiple Pods to a Node. The CPU request alone ensures the number of Pods are not excessive. This is the only thing we can do about controlling CPU assignment for Pods. We should also monitor CPU throttling.

Memory limit and request for Kubernetes workload

Memory is not compressible, therefore we should set both limit and request to the same value. We set memory request so that kube-scheduler has an idea assigning Pods. We set the limit so that no single Pod takes more memory than its fair share. Unlike CPU, once a Pod takes more memory than its fair share, the platform will have to be aggressive to reclaim it back, which may impacts the running of the Pod (process). In contrast, CPU scheduler never guarantees the assignment of CPU time share to a Pod beyond the end of the current CPU cycle.

When we’re setting memory limit and request with different values, we’re sending a confusing signal. We’re inviting Pods to use more memory than they requested. This increases the chance of memory shortage at the node level, and hence the need to OOM kill a Pod.

Horizontal autoscaling and Cluster Autoscaling

The native HPA is metrics-based. As I previously discussed, neither CPU nor memory metrics are good indicators of time to scale. A process or a Pod may have a temporary high demand of CPU purely due to how programmers write the code. Even if we followed the best practices as above, I would still not regard CPU and memory metrics as a reliable indicator to drive auto scaling. If a service is a potential point of congestion, we should use a queue in front and the queue size is almost always a much better indicator of the timing to scale.

As to cluster autoscaler, on it FAQ, it says flat out that you should NOT use a CPU usage based scaling mechanism. I guess this is for a similar reason (compressibility). As discussed, when a Pod is pending for schedule for too long, it emits and event that drives the cluster autoscaler.

Summary

When I first worked on Kubernetes workload I did not give this much thought and proposed the use of CPU limit. As of January 2023 I still find static code analysis tools that requires CPU limit for Pods in the check (e.g. CKV_K8S_11 on Checkov), which leads me to investigate the issue further, and noticed more voices advocating the correct use of resource limit (such as this post) in 2022. For existing deployments, it is worth a review the resource limit configuration.

EKS impression

Fri, 23 Dec 2022 18:18:19 -0400

I’ve worked on a few AKS projects previously. Since I joined AWS I wanted to put aside some time to check out EKS (Elastic Kubernetes Service). Here in this post, I put down my first impression on EKS, and also share my Terraform template in cloudkube project to create an EKS cluster.

Similar to AKS, EKS exposes API endpoint and the control plane components are hidden from AWS users. When creating EKS cluster it does not create the underlying VPC and subnets. Therefore, you have create an existing VPC and at least two subnets ahead of time, and specify them during EKS creation. Bear in mind that there is a list of requirement for the VPC and subnets.

In the cluster, the CNI that EKS officially supports is Amazon VPC CNI plugin. It is available as an add-on. Similar to Azure CNI, each Pod gets its own IP address. In addition, EKS supports other compatible CNI plugins such as Calico, Cilium, Weave Net and Antrea.

Computing Nodes in EKS

There are three modes to address computing capacity: self-managed nodes, EKS managed node groups and AWS Fargate. The documentation has a comparison table.

With self-managed nodes, users create EC2 instances separately and then register them to the control plane. The instances must use the same IAM role and AMI. You can use Auto Scaling groups of Bottlerocket (AWS-sponsored purpose-built Linux distro for container host) nodes. The self-managed node option is mostly for AWS outpost customers who bring in their own computing capacity from data centre.

If you provision computing capacity from AWS, it makes sense to assign EKS managed node groups when creating EKS cluster. We can turn on Cluster Autoscaler, a Kubernetes construct to manage the auto scaling of node groups. Sometimes we want to have more than one node groups. For example, to build a multi-architecture cluster, we need one node group with amd64 nodes and the other with arm64 nodes (e.g. instances with Graviton processor). In general, arm-based CPU delivers better performance with less power consumption and the industry is slowly moving towards more arm-based CPU architecture.

Fargate is what I call managed computing service for EKS. With Fargate you do not need to tweak Cluster Autoscaler to self-manage computing capacity. The Fargate documentation has a long list of considerations. For example, Pods must match a Fargate profile (here‘s an example) at the time that they’re scheduled to run on Fargate. So we need to build Fargate profile and Pod labelling properly. Also, Fargate does not support DaemonSet. Another big consideration is that Fargate does not support non-VPC CNI. In my opinion these are pretty significant limitations. Many workloads (system-level or application-level) would need Daemonset (e.g. kube-proxy, some CNI or CSI drivers, Dynatrace monitoring).

The pro of Fargate is the serverless computing model. The construct of a Fargate profile isn’t complicated. You just specify subnets, namespace and labels. However, the downside is the long list of considerations. Some teams may consider these restrictions too much. The other overhead is the need to manage Fargate profile to ensure all Pods are scheduled somewhere.

To me, using Fargate alone impairs portability of workload. The good thing is that Fargate and Managed Node Group are not mutually exclusive on a cluster. In most cases, we can go partially serverless, and reap the benefits of both of them.

Node AutoScaling

For workloads that don’t have a matching Fargate profile, we have to figure out node autoscaling ourselves. I touched on Cluster Autoscaler in “Autoscaling on Kubernetes Platform“. CA works on AWS as well and is triggered upon a Pod coming to unschedulable status in Scheduler. There is some limitations though. For example, CA interacts with Autoscaling Group (instead of EC2 instances directly). When it determines it’s time to scale up, it bumps up the desired capacity by one at a time in the Autoscaling group. The configurations in Autoscaling group may also be at play and CA do not have direct control. For example, the “scaling cooldown“. The pool of nodes is homogenous as per the pre-configured launch template and CA has no control. If a Pod requires a different type of node (e.g. ARM64 CPU, spot instance, etc), then we’d first have to create a node group with the desired node type. Moreover, in the worst cases, one-at-a-time scale-up does not meet the increase of demand driven by Pod increases, causing nuances such as racing conditions.

Because the Cluster Autoscaler doesn’t really deal with the nodes themselves, this kind of integration is clunky and slow. Nearly half of Kubernetes customers on AWS report that configuring cluster auto scaling using the Kubernetes Cluster Autoscaler is challenging and restrictive, according to this blog post. As a result, AWS launched an open-source cluster autoscaler project, Karpenter. Karpenter first only supported EKS but now the support includes other CSPs. For EKS, Karpenter directly interact with different types of EC2 instances.

Karpenter makes node scaling work in a more cloud-native manner. In the presence of unschedulable Pods, Karpenter bypasses the Kubernetes scheduler and works directly with the Cloud provider, to launch the minimal compute resources needed to fit those Pods and immediately binds the Pods to the newly provisioned Nodes without waiting for scheduler. As Pods are removed or rescheduled to other nodes, Karpenter looks for opportunities to terminate under-utilized nodes. Karpender defines a CR called Provisioner to specify node provisioning configuration, such as instance size, zone, CPU architecture, etc. It is a manifest that describes a node group so the node scaler is aware of all the available node types. You can have multiple Provisioners for different needs, just like node groups. The Provisioner CR can also set TTL for empty Nodes, such that once a Node has no pods other than DaemonSet, Karpenter will terminate the Node on TTL expiry.

Karpenter’s idea is similar to the idea of AutoPilot cluster in GKE. The new EKS workshop has an section on how to set up CA and Karpenter in practice.

Identity Management for EKS

For IAM, we need to be concerned with three aspects. The management traffic to the cloud service, the management traffic for Kubernetes cluster and business traffic.

Traffic type	AWS	Azure
I. Cloud Service Endpoint (Management Traffic for Cloud Service)	AWS IAM identity	Azure RBAC
II. Kubernetes API (Management Traffic for K8s Cluster)	IAM mapping or OIDC	Azure RBAC (implementation of OIDC)
III. Business traffic	Up to Kubernetes Ingress	Up to Kubernetes Ingress

For business traffic (type III), it is all up to the Ingress. I’ve written another article on managing ingress traffic on Kubernetes platforms. We interact with cloud service endpoint (type II) with either AWS CLI or Terraform, to create any object, including resources needed for a cluster. This is generally how we work with cloud service, not specific to Kubernetes. Usually the IAM identity assumes another IAM role, which empowers it with a lot of permissions.

For access to Kubernetes API (type III), EKS supports OIDC and IAM mapping. AWS documentation refers to this as “Cluster Authentication“. There is one special scenario where your identity for type II access inherits your identity for type I access. As the document puts:

When you create an Amazon EKS cluster, the AWS Identity and Access Management (IAM) entity user or role, such as a federated user that creates the cluster, is automatically granted system:masters permissions in the cluster’s role-based access control (RBAC) configuration in the Amazon EKS control plane. This IAM entity doesn’t appear in any visible configuration, so make sure to keep track of which IAM entity originally created the cluster.

This special scenario (I call it the “implicit master user“) allows us to perform critical activities on the cluster, such as creating IAM mapping, or OIDC configuration.

The above addressed how AWS resource access Kubernetes resource. On the other hand, to address how a Kubernetes resource access AWS resources, we need IRSA (IAM Roles for Service Account). We have a service account in Kubernetes and map it to an IAM role.

AppMesh

AppMesh is AWS’ Envoy based service-mesh offering supporting Kubernetes cluster, ECS service and even EC2 instance. AppMesh’s control plane is a managed AWS service, with a controller running on the Kubernetes cluster. To install AppMesh on the cluster:

On the EKS cluster, install AppMesh Controller using Helm
Associate the cluster with IAM OIDC provider
Create an IAM role for the appmesh-controller service account

After these steps, you can create a mesh using CloudFormation, Terraform, etc. The data plane (Envoy proxy) can run on Kubernetes (as sidecar). Traffic between control plane and data plane can go through private link (Interface VPC endpoint) for added security. Like Istio, AppMesh enables mTLS. For observability, you can export Envoy metrics with Prometheus. Coupled with XRay, AppMesh also supports distributed tracing.

AppMesh uses a different set of CRDs than Istio. Key CRDs are:

Mesh: represents an entire service mesh. At mesh level you can configure Egress filter (to allow or deny external traffic) and set IP version (v4 vs v6)
VirtualGateway: a CRD that represents an Ingress in to the Mesh. A virtual gateway allows resources that are outside of your mesh to communicate to resources that are inside of your mesh. A virtual gateway references Envoy proxy deployment by podSelector. It references GatewayRoutes by namespaceSelector, and optionally gatewayRouteSelector. You also specify listeners in the manifest to reference Envoy proxy Service (LoadBalancer Type).
GatewayRoute: A gateway route is attached to a virtual gateway and routes traffic to an existing virtual service. If a route matches a request, it can distribute traffic to a target virtual service. In the manifest, you specify a list of httpRoute, each with matching condition and action. In the action section you can specify virtualService as target.
VirtualService: an abstraction of a real service provided by a virtual node directly or indirectly by means of a virtual router. Dependent services call your virtual service by its virtualServiceName, and those requests are routed to the VirtualNode or VirtualRouter that is specified as the provider for the VirtualService.
VirtualRouter: Virtual routers handle traffic for virtual services. In a virtual router manifest, you can define Route to direct incoming requests to virtual nodes as target.
VirtualNode: A virtual node acts as a logical pointer to a particular task group (i.e. ECS service, Kubernetes deployment). It represent a Service in the AppMesh. In the manifest, you reference Pods by podSelector, specify listeners for any inbound traffic that your virtual node expects, and specify serviceDiscovery for your task group.

You can configure those Custom Resources using YAML manifests (and check the API reference a lot). Alternatively, you can configure them from AWS CLI or AWS console. The console will help you visualize what can be configured. For further details on how these CRs play together, there is a workshop for AppMesh.

EKS cluster using Terraform

Officially, there is an EKS blueprint project for provisioning EKS cluster in Terraform.

I also keep my own Terraform code in the AWS directory of cloudkube project. It works out to be a little more complex than my Terraform template to create Azure Kubernetes Cluster (Azure directory). Because I had to create Cognito resources with initial credential to allow users to connect to cluster without using the implicit master account.

Below is the diagram of the processes.

Create EKS cluster with Terraform module

The template configures kubectl access on a Bastion host, which assumed the same role that our IAM user uses to create the Kubernetes cluster. Therefore, the IAM role is the master identity.

Note that the IAM user (power-user) has very powerful permissions. Usually it is ideal to assign lots of permission to IAM Roles (temporary credential) instead of IAM user (long-term credential). So the role chaining would look like:

The IAM user that Terraform uses has no permission other than assuming a “PowerUser” role
The PowerUser role trusts the IAM user. It also has the permission to assume the “EKS-Manager” role
The EKS-Manager role trusts PowerUser’s role session.

However, the role chaining scenario above is not currently supported in Terraform.

I use a Bastion host because the cluster endpoint is on private subnet. The bastion host is on a public subnet. However, if we do not like public subnet and public IP, we can place the bastion host on a private subnet, and use SSM system manager agent with SSH tunnel plugin to have SSH access to private bastion host.

Summary

I first came across this article about EKS and its awfulness and then decided to check out EKS. I’m not sure all points are still valid but it’s generally real-life experiences. There are also many peripheral services, such as AMP (AWS Managed Prometheus), AMG (AWS Managed Grafana), ADOT (AWS Distro for Open Telemetry), AppMesh (Another Envoy-based Service Mesh, easier to manage than Istio but less Powerful), with a lot to explore.

Build and Manage Kubernetes Clusters

Fri, 23 Sep 2022 11:50:00 -0400

There are numerous options to build a Kubernetes cluster. If your company has a multi-cloud strategy, most likely you will have to deal with cluster creation on multiple cloud platform or on virtual machines on premise.

Most likely, the chosen cloud platform already make it simple for us. However, it is still important to understand what it really takes to build a Kubernetes cluster. In general, we need to figure out these tasks:

Decide where to host the computing infrastructure (i.e. Node) : on premise or public cloud;
Choose a Kubernetes release: either the vanilla release or one of the third-party distributions;
Install Kubernetes to the computing environment, and integrate it with the cloud platform;
Determine required add-ons (e.g. Istio or Linkerd for Service Mesh, dashboard utility, etc);
Deploy application workload to Kubernetes platform;

A public cloud platform provider usually can assist you with task 1 through 3, and partially 4, depending on the provider. If your Kubernetes resides on private cloud or on-prem environment, you can use a Platform solution such as VMware Tanzu or Openshift, which usually covers task 1, 3 and 4. There is no standard about what task these platform solution must address. Therefore it is important to have this list of tasks in mind in order to make a good comparison. I will discuss each of the tasks in this post.

Hosting environment

Nodes are the building blocks of a Kubernetes cluster. We need master nodes as well as worker nodes. In addition, a working cluster also requires storage, and networking infrastructure.

Public cloud platforms typically provides control plane as a service, obviating administrator’s effort to provision master nodes. For example, the control plane of Azure AKS has two levels of uptime commitment: a free tier of 99.5% SLO and a paid tier with an SLA of 99.95% (using AZs) and 99.9% (without using AZs). This uptime commitment applies to control plane only and do not apply to worker nodes. The management of etcd store is also a responsibility of the cloud provider, which frees up the cluster administrator from managing etcd store. However, they cannot access etcd store either. This is not very convenient because as the size of the cluster grows it is a common requirement to connect to etcd store for troubleshooting purpose.

The deployment APIs for public cloud allow the cluster administrator to define the instance size, count and availability zone for the worker nodes. They also automatically register the worker nodes to control plane so that the cluster administrators do not have to do so by themselves. As to storage, the public cloud usually provide some default storage classes based on their storage as service. For networking device, the cluster provision process automatically configures the cloud API so the cluster can manage cloud resources such as network load balancer.

With private cloud or data centre, we usually use virtual machines, or bare-metal servers. Cluster administrators will need to make their own control plane with master nodes. and install worker nodes and register them to the master nodes. The Kubernetes Installation section below will discuss this.

Kubernetes release

If you have to install Kubernetes, you have to think about the Kubernetes release being used. You can use the binary from official Github repository. For example, the release note of version 1.24.3 points to the change log file for download links to server binaries, node binaries. This is the vanilla Kubernetes release.

Apart from the vanilla release, many developers build their own distributions, based off forks of the Kubernetes project. CNCF has a page to keep track of certified Kubernetes distributions. Some of the distributions are open source and can be used for on-prem infrastructure. Here is a list of top players:

Distribution Name	Repo	Description
EKS Distro	Link	Used in EKS managed service or EKS Anywhere for on-prem infrastructure
AKS Engine	Link	Used in Azure Stack for on-prem infrastructure.
Google Kubernetes Engine	N/A	Used in GKE managed service only.
OpenShift Kubernetes Engine	Link	Community distribution (OKD, or OpenShift Kubernetes Distribution) is the open-source upstream.
Rancher Kubernetes Engine (RKE)	Link	still using Docker as container runtime. Supported CNI include: Canal, Flannel, Calico and Weave
K3s	Link	Lightweight distro without small resource requirement. Great for Edge, IoT, ARM etc
RKE2	Link	Originally named RKE government. Supports deployment via Cluster API. Supports containerd as container runtime. Supported CNI include: Cillium, Calico, Canal and Multus. Lightweight
VMware Tanzu	Link	VMWare Tanzu Grid and VMWare Tanzu Community

Above is just a very incomplete list of Kubernetes distributions. There are many more distributions that are not on this list, such as CoreOS Tectonic, Docker Kubernetes, Heptio, Mesosphere, Mirantis, Platform9, Stackube, Telekube. For full details of how each distribution is different, you will need to go over their documents.

With the selected distribution, we still need to deploy the binaries to the nodes. We can do this with a cluster management platform, or standalone installers. Cluster management platform can also help us with baseline configuration (e.g. IAM integration, CNI plugin), in addition to the binary installation.

Cluster Management Platform

These platforms are also sometimes referred to as container management platform.

For example, OpenShift container platform is a self-managed platform based on OpenShift Kubernetes Engine and can run on a variety of hosting environment, public cloud, or private cloud. The installation steps varies depending on the hosting environment. When running on public cloud such as AWS (aka ROSA), the public cloud only provides computing nodes and associated infrastructure. Many corporate with multi-cluster strategy use this option on public cloud to keep their Kubernetes cluster fleet consistent across cloud vendors.

The Openshift container platform also packages some useful open-source add-ons with corporate support, for example:

Red Hat’s strategy is to pick the most renowned open-source project in each domain and add enterprise support to it. However, for management portal, Red Hat developed its own Advanced Cluster Management tool for Kubernetes, and open-sourced it in 2020 in the upstream project Open Cluster Management.

Similar to OpenShift, VMware Tanzu also attempts to cover the domains, with a smaller product portfolio:

Service Mesh: compatible with Istio
Mission Control: management portal
Observability

Google Anthos is also a container platform. Their product line include, but not limited to:

Anthos Config Management (ACM)
Anthos Service Mesh (ASM, an Istio distribution)

SUSE, the developer of RKE, RKE2, and K3s) offers Rancher as multi-cluster management platform. Apart from the engines, SUSE also offers Lonhorn as a storage solution. However, they do not have offerings for service mesh or GitOps. So there is no doubt that Red Hat OpenShift has the most complete portfolio for Kubernetes.

There are also companies that only offers management platforms without their own Kubernetes distribution. For example:

Product capabilities in this category vary a lot and you should refer to their specific documentation to understand. You will probably see a stack chart from each of the platform provider (e.g. SUSE Enterprise Container, OpenShift, Tanzu, Anthos, Rafay) with all technology integrations.

Cluster Installation Tools

As we saw in the installation steps for OpenShift, they are highly dependent on platform. With public cloud, the provisioning process also applies only to a specific platform. Since Kubernetes Installation process is tedious, some tools emerged to help, for example: kubespray, kubeadm, kops and Cluster API. These are governed by SIG cluster lifecycle special interest group.

Here are some traditional options to install a Kubernetes clusters:

kube-up: the first tool to build cluster from 2015. It has been deprecated.
Kubeadm: a tool built to provide best-practice “fast paths” for creating Kubernetes clusters that are minimum viable, and secure. Kubeadm’s scope is limited to the local node filesystem and the Kubernetes API, and it is intended to be a composable building block of higher level tools. It is first released in Sep 2016. The high level configuration steps goes through initialization (kubeadm init), control plane (kubeadm join control plane), and node (kubeadm join node). Kubeadm does not integrate with cloud providers and it does not install addons (auth, monitoring, CNI, storage class)
Kubespray: runs on bare metal or VMs using Ansible for provisioning and orchestration. The first release was in Oct 2015. Since v2.3 (Oct 2017) kubespray started to use kubeadm internally. In addition to kubeadm, kubespray configures CNI, storage class, other CRI. It supports cloud providers and air-gap environment. However it does not support infrastructure management.

The options above are official options. You may use kubeadm and kubespray to quickly (i.e. in an hour) spin up clusters for education purposes. However, with their limitations, it typically requires a lot of efforts to build a production-grade cluster with the needed addons and integrations.

Apart from the official options, there are also unofficial tools such as kubicorn, which was first introduced in 2018 as a cluster management framework with modular support for cloud providers. However it appears to be short-lived.

In the next two sections, we introduce kops and cluster API, two most recent projects to install cluster.

Kops

The kops utility directly perform the provisioning and orchestration via API to the cloud deployment engine. Kops, with first release in Oct 2016, is tightly integrated with the unique features of the cloud providers (e.g. AWS: ASG, ELB, EBS, KMS, S3, IAM). However, kops is only CLI without controller-style reconciliation. It does not support baremetal or vsphere. It also bundles addons with fixed version.

When picking a tool to install cluster, we need to strike a balance between how much simplification the tool brings, and how many different platform the installer can work with. Kops appears to be such a good compromise. It works with a number of cloud platforms using different set of APIs, although most are in alpha and beta stages today. Here is how to install cluster on AWS.

Both kops and Cluster API have good momentum but they work differently. Cluster API was first released in Mar 2019, and is currently less mature than kops. However, it is declarative and may reflect the direction of where cluster lifecycle management is heading.

Cluster API

Cluster API focuses on following areas:

Manage cluster lifecycle declaratively
Infrastructure abstraction (e.g. computing, storage, networking, security, etc)
Utilizing existing tools (e.g. kubeadm, cloud-init)
Modular and pluggable: to be adaptable to different infrastructure providers.

It involves a number of CRs as illustrated in its diagram. We should be clear on the providers for Bootstrap, Infrastructure and Control Plane.

The biggest benefit is the controller pattern to manage the entire lifecycle of a cluster. This allows managing clusters with GitOps, and rolling upgrade of the cluster. It also allows for declarative node scaling, self healing and multi-cluster management.

The client utility for is clusterctl, and with that along with the manifest, we can create a cluster in a few commands. A lot of workflows are still in development but we can take a look at its quick start guide to get a taste of how it works. The installation steps vary a lot based on the environment and the cluster. Also it introduces the separation of management cluster and workload cluster.

Workload cluster is the target cluster being created, as per the manifests.
Management cluster is where you keep track of the workload cluster being managed. You can manage multiple workload clusters from a single management cluster. Note that this management cluster will store credentials about workload clusters, and may become a single point of failure.

Although Cluster API reflects a great initiative to standardize the provisioning of Kubernetes cluster, whether it will succeed has to do with the level of complexity. In the next section, we will get a taste of how it looks to deploy a Kubernetes cluster in a lab.

Management cluster vs workload cluster

In the lab, I use my MacBook to create a management cluster with KinD. Then we configure a workload cluster in AWS from the management cluster.

Cluster API Lab

Note that the steps here are based on the quick start guide on Cluster API document. Also, there is a bug with the AWS provider so the end of the lab will report a warning. The main purpose of this lab is to demonstrate how Cluster API is supposed to work, even though it still has yet to mature.

To start, I install clusterctl (the cluster API client utility), clusterawsadm (the utility specific for AWS) on MacBook, then start a simple KinD cluster.

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.2.0/clusterctl-darwin-amd64 -o clusterctl
chmod +x ./clusterctl
sudo mv ./clusterctl /usr/local/bin/clusterctl
clusterctl version

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/download/v1.4.1/clusterawsadm-darwin-amd64 -o clusterawsadm
chmod +x clusterawsadm
sudo mv clusterawsadm /usr/local/bin
clusterawsadm version

kind create cluster

So far, I installed the required utility and a KinD cluster on MacBook. Then I use clusterawsadm to create InstanceProfile, ManagedPolicy and IAM Roles required for cluster creation. The AWS region and access are configured as environment variables:

export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID=AKIAXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=J8ByduiofpwuisDjDoijOISDs
clusterawsadm bootstrap iam create-cloudformation-stack

This runs a CloudFormation stack to create the permission related resources:

Then I initialize the management cluster with the clusterctl utility, specifying AWS as a provider. I also need to assign the environment variable AWS_B64ENCODED_CREDENTIALS with proper value:

export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)
clusterctl init --infrastructure aws

Now I use clusterctl to generate the manifest for the workload cluster. In environment variables, I specify cluster and node sizes, SSH key name, control plane machine type and node machine type:

export AWS_SSH_KEY_NAME=cskey
export AWS_CONTROL_PLANE_MACHINE_TYPE=t3.large
export AWS_NODE_MACHINE_TYPE=t3.large
clusterctl generate cluster myekscluster --kubernetes-version 1.24.3 --control-plane-machine-count=3 --worker-machine-count=3 > capi-quickstart.yaml

kubectl apply -f capi-quickstart.yaml

At the end I tell the management cluster to create a workload cluster as per the manifest, by simply declaring the CRs. It will take some time for the cluster to create, and there are a number of ways to monitor the progress. You can monitor the log on the controller pods in their respect namespaces. You can also check the cluster status with:

kubectl get kubeadmcontrolplane
clusterctl describe cluster myekscluster

Currently there is a bug and the commands at the end will report as below:

Hopefully the bug will be fixed shortly. To delete the cluster, simply delete the resources in the manifest with kubectl delete -f capi-quickstart.yaml

Summary

There are numerous ways to build a Kubernetes cluster. Before deciding on the approach, I recommend having a full understanding of the hosting environment. This is because installation approach and hosting environment are still tightly coupled. This is the status quo and is not going to change in the near future. Both kops and cluster API reflects initiative to decouple the two but both are still in early stage and already facing growing complexity. Cluster API manages complexity with CRDs to abstract system resources and infrastructure, as illustrated here:

CRDs and providers to abstract system resources and infrastructure

The diagram is from the “Cluster API and declarative Kubernetes Management” white paper. Here is a stream with more about the same topic.

MinIO for S3-compatible Object Storage

Fri, 09 Sep 2022 09:00:00 -0400

I reviewed some storage technologies on Kubernetes but they are all for block and file storage. In this post, I will discuss the current available options for container workload to use object storage. I will also touch on MinIO as an object storage solution.

Object storage

Block and file system are more native to operating system because they present themselves to the OS as a block device or file system attached to the OS. In other words, application processes running on the OS will be able to access the storage by address expressed as a POSIX-compatible path. On the contrary, object storage is a REST API service, operating at the application layer in the TCP/IP stack. Therefore, we can think of object storage as “storage as a web service”.

Object storage can be made very cheap. However, the application protocol may vary depending on the object storage provider. Amazon S3 is a forerunner in object storage market and its protocol has emerged as the de-facto standard for object storage. When building an application and if there is one object storage protocol to support, it should be S3. For non-S3 object storage services, we can front them with an S3 interface, if the provider itself does not have one. For example Ceph storage has its Gateway S3 API.

Container Object Storage Interface

If we use S3 as the universal object storage protocol, does that also address object storage access with container workload on Kubernetes? Absolutely. Nonetheless, for a number of reasons using REST API from containers are not the best option. From platform’s perspective, it is the platform that should define how to access object storage, instead of leaving it with an application-layer protocol.

When a pattern (for storage, or networking, etc) turns out very common, the platform layer should incorporate it as an infrastructure service, manage it with its own standard, and provide it to application so that developer can focus on business features. With that vision, the community brought up the Container Object Storage Interface (COSI) initiative. It is currently in very early stage, but the idea is to commoditize object storage in Kubernetes platform with a unified interface. For more background about this initiative, refer to the post “Beyond block and file – COSI enables object storage in Kubernetes“.

COSI is the ultimate cloud native solution but it is still in pre-alpha phase as of mid 2022. Unfortunately, it is not a recommended solution to any real-life project in 2022, and we are stuck with the unified API approach until COSI matures.. The unified API approach is by no means cloud native, but has come to maturity for adoption. S3 Rest API is our friend, regardless of whether the client process is in a container or not.

Update: on Sept 2, 2022, Kubernetes introduced COSI as alpha feature.

MinIO Introduction

In order to use S3 protocol without using Amazon S3 storage, we can use MinIO to build our own object storage service serve client via a S3-compatible REST API interface. The main developer of the MinIO project is MinIO Inc, a startup from 2014. Having learned the lessons from GlusterFS, the founders and developers make MinIO very simple. MinIO operates in two modes: gateway mode (soon to be legacy) and server mode.

In the Gateway mode, MinIO as a gateway between client and destination storage, and does not persist data to itself. In the past, the destination storage can be Azure Blob and Google Cloud Storage (GCS) and HDFS as backend. However, these supports are deprecated now. The current release (July 2022) only supports S3 and NAS backend. According to MinIO’s blog post from February 2022, the entire MinIO Gateway feature will be removed in August, leaving server mode the only option for MinIO.

In the Server mode, the MinIO service will persist data to itself in a file system (or volume). You can specify that file system (or volume) as you launch the server. As one of the quick-start guides shows, we can host MinIO server using a single executable. For administrative tasks, MinIO has a web console and a client utility called mc.

MinIO Deployment Options

For storage service, there are a number of deployment options:

SNSD (single-node, single-drive): single MinIO server with a single storage volume or folder.
SNMD (signle-node, multi-drive): single MinIO server with four or more storage volumes.
MNMD (multi-node, multi-drive, aka distributed): multiple MinIO servers with at least four drives across all servers. This should be considered for production grade configuration.

The deployment options above describes the node and volume topology. No matter which topology option, there are also a number of ways to host the MinIO service process: on Linux OS, Windows OS, MacOS, Docker Container, and on Kubernetes platform.

In addition, MinIO Inc ships the software under different business models. For example, there are fully managed applications in Azure Marketplace, AWS Marketplace, and GCP Marketplace all hosted on virtual machines with extra charges. Clients not willing to pay can host MinIO storage all on their own, either on virtual machines, or on managed Kubernetes environment provided by each cloud provider.

MinIO Hosting solutions

MinIO lists these hosting solutions under multi-cloud products. These hosting solutions (or “products” in MinIO’s term) vary in terms of where peripheral services and data tiers are hosted. Here is the list of the supported platforms:

To illustrate how these solutions are different, I put some details on a few options together for an incomplete comparison below:

	Kubernetes	EKS	AKS	GKE
Hot Storage	Direct PV (NVMe)	EKS EBS CSI	Azure CSI	GKE Standard SSD
Warm Storage	Direct PV (HDD)	S3 IA	Azure BlobStore	GCS
Cold Storage	Public Cloud storage	Glacier	Azure Cool Blob	GCS for Data Archiving
Encryption	HashiCorp Vault	KMS	Azure Key Vault	Cloud Key Management
Observability	Elastic Stack and Grafana	Managed ElasticSearch Prometheus	Azure Monitor	Stack Driver
Identity Provider	KeyCloak	LDAP, SSO	Azure Active Directory	GCP Cloud Identity
LB and Cert Mgmt	Nginx, Let’s Entrypt	AWS Cert Mgr, ELB	Azure Load Balancer, JetStack, Let’s Encrypt	GCP Cloud LB and Managed Cert

Note that all of these hosting solutions are based on some flavour of Kubernetes. The hot tier is usually based on storage options available to the platform. MinIO service access this hot tier via Kubernetes persistent volume. The warm and cold tiers are backed by different object storage service. Between MinIO and storage client, it always use the same S3 compatible Rest API.

MinIO also has tiering capability. While the hot storage destination has to be either a file system or Kubernetes persistent volume, remote tiers can be S3 , Azure Blob, or GCS. MinIO supports encryption at rest (SSE-KMS, SSE-S3, SSE-C) and in transit (TLS) for security, as well as many other useful features such as object replication, versioning, locking, events, Prometheus metrics, lifecycle management etc.

Connect to MinIO server with S3 client

To validate that the client is compatible, we use MinIO’s client utility (mc) to connect to an AWS S3 bucket. Then we use AWS CLI to connect to a MinIO server, similar to this instruction. To do so, we first install client and server utilities:

brew install minio/stable/minio
brew install minio/stable/mc
minio --version
mc --version

Then, we start MinIO server and store an object using AWS CLI’s S3 tool. In our working directory, we create a new directory called minio_data and launch MinIO server with it:

mkdir minio_data
minio server minio_data --console-address :9090

Once the server is up, the screen should display the details, including the portal URL and the default username and password will be used as Access Key ID and Secret Key:

Note that the MinIO service does NOT have TLS enabled by default, on the console or API service. At this point, we can browse to the console web page using the given credential. Then, we can configure AWS CLI with a new profile just to act as a client to communicate with the MinIO server:

$ aws configure --profile minio-cli
AWS Access Key ID [None]: minioadmin
AWS Secret Access Key [None]: minioadmin
Default region name [None]: us-east-1
Default output format [None]: json
$ aws configure set default.s3.signature_version s3v4 --profile minio-cli

At this point, the AWS CLI is configured to communicate with MinIO server. Then, we can create bucket, list object in the bucket, copy an object to the bucket, etc

$ aws --endpoint-url http://127.0.0.1:9000 s3 ls --profile minio-cli # list all bucket, should return empty
$ aws --endpoint-url http://127.0.0.1:9000 s3 mb s3://hehebucket --profile minio-cli # create new bucket
make_bucket: hehebucket
$ aws --endpoint-url http://127.0.0.1:9000 s3 cp README.md s3://hehebucket --profile minio-cli # copy a file to bucket as a new object
upload: ./README.md to s3://hehebucket/README.md
$ aws --endpoint-url http://127.0.0.1:9000 s3 ls s3://hehebucket --profile minio-cli # list objects in the bucket
2022-07-09 00:30:23 631 README.md

The created bucket and object are also visible in MinIO web console, under “Bucket”:

The steps above validate that AWS CLI can talk to MinIO server. Because of that, MinIO server can emulate an S3 service in any development environment so users do not always have to use S3 from AWS. This makes sense for both cost and security reasons for the organization.

Connect to S3 with MinIO client

In this lab, we create an S3 bucket and use mc utility to store an object to it. In order to consistently create S3 bucket and associated permissions, I use the CloudFormation template in this repo. The output of the CloudFormation stack returns the Access Key ID and Secret Key required for the client to access the bucket. Once we cloned the repo, let’s enter the obj-store-helper directory, and run aws cli command to launch the CloudFormation template, assuming it has been configured:

BUCKET_NAME=c0sas2dsadigihunch
S3_STACK_NAME=$BUCKET_NAME-stack
aws cloudformation create-stack --template-body file://aws-s3-stack.yaml --stack-name $S3_STACK_NAME --parameters ParameterKey=S3BucketName,ParameterValue=$BUCKET_NAME --capabilities CAPABILITY_IAM

# to delete stack after test, run: aws cloudformation delete-stack --stack-name $S3_STACK_NAME

In the AWS console, we should see the configuration information as below:

Supposed the bucket name is vna-tst-c0sas2dsadigihunch as shown above, this allows us to configure the client utility MC as below:

mc alias set awss3 https://s3.amazonaws.com # Fill in access key ID and Secret key at the prompt
mc ls awss3/vna-tst-c0sas2dsadigihunch # list objects in the bucket, should return empty
mc cp README.md awss3/vna-tst-c0sas2dsadigihunch/README.md # upload and object to bucket
mc ls awss3/vna-tst-c0sas2dsadigihunch # list objects in the bucket, the uploaded object should be there
mc rm awss3/vna-tst-c0sas2dsadigihunch/README.md # delete the object
mc alias remove awss3 # remove awss3 alias

Once we emptied the bucket, we can delete the CloudFormation stack. This test only needs client utility mc to verify that MinIO client is able to talk to AWS S3 server.

Erasure Coding

For scalable production use, we should deploy MinIO in distributed mode. When MinIO is configured in distributed deployment (MNMD, or multi-node, multi-drive), it implicitly enables an important feature called erasure coding. This erasure coding feature further unlocks a number of other MinIO features:

Erasure coding is MinIO’s data redundancy and availability feature that allows MinIO deployments to automatically reconstruct objects on-the-fly despite the loss of multiple drives or nodes in the cluster. Erasure coding provides object-level handling with less overhead than adjacent technologies such as RAID. The key concept is Erasure Set, a set of drives in a MinIO deployment that supports Erasure Coding. MinIO evenly distributes object data and parity blocks among the drives in the Erasure Set.

Two important variables are M and N: for a given erasure set of size M, MinIO splits objects into N parity blocks, and M-N data blocks. MinIO uses the EC:N notation to refer to the number of parity blocks (N) in the deployment. To determine optimal erasure set size for the cluster, use MinIO’s Erasure Coding Calculator tool.

To help client to specify per-object parity with Erasure Coding, MinIO uses storage classes. Note that the storage class concept in MinIO is distinct from AWS S3 storage class or Kubernetes storage class. In MinIO, a storage class defines parity settings per object. The STANDARD storage class (default) defines EC:N based on M, which can be overridden. In addition, there is REDUCED_REDUNDANCY storage class, whose parity must be less than or equal to that of STANDARD storage class.

The erasure coded backend also protects the storage against Bit Rot with HighwayHash algorithm.

More Features

Authentication and authorization between MinIO client and MinIO server have a number of options. MinIO client may use the built-in standalone identity management in MinIO server. This is the default mode. In addition, one may delegate IAM to external service. To Active Directory via LDAP, or any Identity provider that supports OIDC (JWT with Authorization Code Flow).

As to Object Lifecycle Management (OLM), MinIO allows you to define a remote tier storage for each local target (bucket). The remote tier can be Amazon S3, Google Cloud Storage or Azure Blob storage. We can use mc utility to administer the remote tier and OLM. Configuration steps (e.g. Azure Blob, AWS S3) usually include:

Configure required permissions on the MinIO bucket, create user account for OLM activities.
Configure the Remote Storage Tier
Create and Apply an ILM Transition Rule. The rule can be expressed in a json document.
Validate the creation of ILM transition rule
Validate the effect of transition rule.

As for encryption, MinIO can support encryption at rest. It can also work with etcd store to store encrypted IAM assets if KMS is configured.

Conclusion

Even though we watch for the progress of COSI initiative, we still use Rest API to access object storage from container, which is no different than from a virtual machine. If we develop an application, then we should make it support S3 protocol, a de-facto standard protocol for object storage. As for the storage backend, if we want to be vendor neutral, the feature-rich MinIO is the best bet. We can use MinIO to build our own Object storage as a service compatible with S3. We can also lifecycle our object to remote object storage tier backed by Azure, GCP or S3. In this post we validated the S3 compatibility, and discussed some advanced MinIO features.

Kubernetes Storage on Azure 3 of 3 – Ceph by Rook

Fri, 26 Aug 2022 19:43:00 -0400

In the last two posts, I covered the native storage options on Azure Kubernetes Service, as well as Portworx as an example of a proprietary Software Defined Storage (SDS) solution. There are also a number of open-source alternative SDS solutions. Ceph has nearly a decade of history from prior to containerization, and is the most widely adopted storage platform. In this post, we continue to explore Ceph as an open-source storage solution on Azure Kubernetes.

Ceph by Rook

Ceph is an open-source SDS platform for distributed storage on a cluster and provides object, block and file storage. Installation of Ceph SDS can be complex, especially on Kubernetes platform. Rook is a graduated CNCF project to orchestrate storage platform. Rook by itself is not SDS and it supports:

Ceph: configure a Ceph cluster. Think of this as the equivalent of cephadm on Kubernetes platform.
NFS: configure an NFS server. Think of this as the equivalent of nfsd daemon on Kubernetes platform.
Cassandra: an operator to configure a Cassandra database cluster. It is now deprecated.

We play with Rook Ceph. I also refer to it as Ceph by Rook. The contribution of Rook project is it simplifies the installation as a matter of declaring custom resources using CRDs. Here are some high-level CRDs to know:

CephCluster: creates a Ceph storage cluster
CephBlockPool: represents a block pool
CephFilesystem: represents a file system
CephObjectStore: represents an object store
CephNFS: spins up a NFS Ganesha server to export NFS shares of a CephFilesystem or CephObjectStore.

As with typical Kubernetes resources in controller pattern, Ceph by Rook needs an operator along with custom resources. We can use YAML manifest for both of them, and the manifests are usually very tediously long. We can also use Helm to install both of them, by providing a value file. Now we will install Ceph on AKS.

Install Ceph Operator on AKS

The steps are influenced by two relevant posts (here and here). However, I’ve incorporated the cluster configuration in the Azure directory of the cloudkube project, a modular Terraform template to configure AKS cluster and facilitate storage configuration. The node group and instance sizes are selected to be just enough to run a ceph POC cluster with minimum cost. One of the node groups is tainted with storage-node, as if the following command were run:

kubectl taint nodes my-node-pool-node-name storage-node=true:NoSchedule

You will only need to taint the nodes with the command above if you choose not to use the cloudkube template. The taint ensures that only Pods with corresponding toleration and effect can be scheduled to those nodes.

We use Helm to install Rook Operator. We need a value file (e.g. rook-ceph-operator-values.yaml) with content as below:

# https://github.com/rook/rook/blob/master/Documentation/Helm-Charts/operator-chart.md
crds:
 enabled: true
csi:
 provisionerTolerations:
 - effect: NoSchedule
 key: storage-node
 operator: Exists
 pluginTolerations:
 - effect: NoSchedule
 key: storage-node
 operator: Exists
agent:
 # AKS: https://rook.github.io/docs/rook/v1.7/flexvolume.html#azure-aks
 flexVolumeDirPath: "/etc/kubernetes/volumeplugins"

Then we install the operator with Helm:

helm install rook-ceph-operator rook-ceph --namespace rook-ceph --create-namespace --version v1.9.6 --repo https://charts.rook.io/release/ --values rook-ceph-operator-values.yaml
kubectl -n rook-ceph get po -l app=rook-ceph-operator

After installing the operator, we check the Pod status to make sure it is running. Then we can install the actual Ceph Cluster in one of the two ways. We can declare a CephClusterCRD ourself, or we can use Helm again to declare the CRD. Helm Chart gives us a lot of useful default values and saves us from editing a large body of YAML manifest.

Install Ceph CR on AKS

We use Helm to install CephCluster CRD. We create a value file (e.g. rook-ceph-cluster-values.yaml) with content as below:

# https://github.com/rook/rook/blob/master/Documentation/Helm-Charts/ceph-cluster-chart.md
operatorNamespace: rook-ceph
toolbox:
 enabled: true
cephObjectStores: [] # by default a cephObjectStore will be created. Setting this to null disables it
#cephBlockPools: # by default a cephBlockPool will also be created with default values
#cephFileSystems: # by default a cephFileSystem will also be created with default values

cephClusterSpec:
 mon:
 count: 3
 volumeClaimTemplate:
 spec:
 storageClassName: managed-premium
 resources:
 requests:
 storage: 10Gi
 resources:
 limits:
 cpu: "500m"
 memory: "1Gi"
 requests:
 cpu: "100m"
 memory: "500Mi"
 dashboard:
 enabled: true
 storage:
 storageClassDeviceSets:
 - name: set1
 # The number of OSDs to create from this device set
 count: 3
 # IMPORTANT: If volumes specified by the storageClassName are not portable across nodes
 # this needs to be set to false. For example, if using the local storage provisioner
 # this should be false.
 portable: false
 # Since the OSDs could end up on any node, an effort needs to be made to spread the OSDs
 # across nodes as much as possible. Unfortunately the pod anti-affinity breaks down
 # as soon as you have more than one OSD per node. The topology spread constraints will
 # give us an even spread on K8s 1.18 or newer.
 placement:
 topologySpreadConstraints:
 - maxSkew: 1
 topologyKey: kubernetes.io/hostname
 whenUnsatisfiable: ScheduleAnyway
 labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - rook-ceph-osd
 tolerations:
 - key: storage-node
 operator: Exists
 preparePlacement:
 tolerations:
 - key: storage-node
 operator: Exists
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: agentpool
 operator: In
 values:
 - storagenp
 topologySpreadConstraints:
 - maxSkew: 1
 # IMPORTANT: If you don't have zone labels, change this to another key such as kubernetes.io/hostname
 topologyKey: topology.kubernetes.io/zone
 whenUnsatisfiable: DoNotSchedule
 labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - rook-ceph-osd-prepare
 resources:
 limits:
 cpu: "500m"
 memory: "4Gi"
 requests:
 cpu: "500m"
 memory: "2Gi"
 volumeClaimTemplates:
 - metadata:
 name: data
 spec:
 resources:
 requests:
 storage: 100Gi
 storageClassName: managed-premium
 volumeMode: Block
 accessModes:
 - ReadWriteOnce

During the cluster provisioning, there will be a number of preparing Pods. We want those Pods to run on nodes with label agentpool=storagenp. In real life, we need to orchestrate where to run each workload, by restricting the nodes to schedule certain types of workload.

Then we can install the cluster using Helm:

helm install rook-ceph-cluster rook-ceph-cluster --namespace rook-ceph --create-namespace --version v1.9.6 --repo https://charts.rook.io/release/ --values rook-ceph-cluster-values.yaml

After running the Helm install, it may take as long as 15 minutes for all resources to settle. Watch the Pod status in rook-ceph namespace. At the end, make sure that the cluster is created successfully:

kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get CephCluster
NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL
rook-ceph /var/lib/rook 3 15m Ready Cluster created successfully HEALTH_OK
kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get cephBlockPools
NAME PHASE
ceph-blockpool Ready
kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get cephFileSystems
NAME ACTIVEMDS AGE PHASE
ceph-filesystem 1 20m Ready

In my case it took 15 minutes before the cluster comes up as created successfully. You should notice that two storage classes were also created as a part of the install. It however did not create a storage class or CRD for object storage, because we explicitly disabled it in the Helm value file by setting cephObjectStores value to null.

Dashboard

We enabled dashboard. To configure the dashboard view properly, we would need an ingress. For a quick view here, we can play port forwarding tricks. First we fetch the admin password for use in the next step. Then expose the dashboard to the bastion host:

$ kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath='{.data.password}' | base64 -d
$ kubectl -n rook-ceph port-forward svc/rook-ceph-mgr-dashboard 8443:8443

Since I don’t have UI on the bastion host, I use the port forwarding trick again from my own MacBook. Start a new terminal and SSH to the bastion host with port-forwarding switch:

$ ssh -L 8443:localhost:8443 kubeadmin@20.116.132.8

The command above suppose the public IP of the bastion host is 20.116.132.8. Then from my MacBook I can browse to localhost:8443 (with Safari browser which gives me the option to bypass certificate error). At the web portal, provide username (admin) and password (as retrieved above):

Ceph console for Kubernetes

Apart from the dashboard, we can also use ceph admin tool from a toolbox pod, following this instruction. For monitoring, Ceph by Rook can expose metrics for Prometheus to scrape.

Performance

With default ceph configuration on AKS, I ran quick performance test using kube-str . The result is as follows:

	read_iops	write_iops	read_bw	write_bw
ceph-block	IOPS=464.507294 BW(KiB/s)=1874	IOPS=243.296143 BW(KiB/s)=989	IOPS=509.928162 BW(KiB/s)=65797	IOPS=248.530762 BW(KiB/s)=32338
ceph-filesystem	IOPS=438.701324 BW(KiB/s)=1770	IOPS=226.270660 BW(KiB/s)=920	IOPS=405.936340 BW(KiB/s)=52456	IOPS=208.869293 BW(KiB/s)=27229

The metrics reflects performance under default configuration. It should not be considered as the best performance that Ceph can deliver on Azure Kubernetes.

Summary

I discussed three storage options for Azure Kubernetes but the idea applies to other Kubernetes platform hosted on a CSP. The native storage has significant limitation. NFS has latency. Block storage does not address high availability at the storage layer. Portworx and LINSTOR fill that gap as a commercial solution. Ceph is based on Object storage.

Etcd – the key-value store for Kubernetes

Tue, 14 Jun 2022 00:10:00 -0400

Etcd in Kubernetes

In Kubernetes architecture, etcd is the data store. It stores the desired state of Kubernetes object. API server is the only client that connects to etcd (via gRPC protocol). Cluster builder specifies the endpoint of etcd as a parameter to the kube-api-server process. Other Kubernetes components, whether in the control plane or from the nodes, connect to API server. API server translates their request into etcd query, and then translates etcd query result into what its clients ask for. For this reason, communication with etcd accounts for a lot of network traffic in a Kubernetes cluster.

The etcd store is a CNCF project for “a distributed, reliable key-value store for critical data in a distributed system”, developed by CoreOS team. So it is essentially a distributed key-value store for any distributed application. If an application runs on Kubernetes, it can leverage etcd store, by keeping their configurations in ConfigMap and Secret objects. One key feature is to watch for specific keys or directories for changes, and react to the changes. Voila! This is the underlying mechanism for controller!

A Kubernetes cluster may have stacked etcd deployment or connect to an external etcd store.

stacked etcd architecture

external etcd architecture

In managed Kubernetes services such as EKS in AWS and AKS in Azure, users usually do not directly access etcd store. However, it is still a very important component to understand. Its use case includes:

Configuration sharing
Service discovery
Consistency
Watching mechanism
Expiry and extension of key

The consistency use case is based on Raft protocol for distributed consensus.

Raft protocol

I am not an expert in distributed consensus protocols and nor do I intent to cover it in depth. At a high level, I have heard of three of them so far:

Etcd uses Raft protocol
Zookeeper uses ZAB protocol
Cassandra uses paxos protocol

Here is a good intro to the three protocols. Instead of getting into the fine details, I would like to discuss why we need such a consensus protocol (or consensus mechanism) in distributed systems, which are also decentralized systems.

Centralized, Decentralized, Distributed systems

The reason a distributed system needs consensus protocol, is that a distributed system lacks a single source of truth as centralized systems do. Different parts of the distributed system may receive different signals but they must come to agreement of a single plan to act. Lamport studies this with an analogy of Byzantine Generals problem, and first proposed Paxos protocol. Paxos has been an important foundation to modern distributed systems. In Paxos, consensus is achieved in two phases, which creates the problem of livelocks. Raft is an alternative to Paxos, and is widely adopted today. Here is a link to an animated illustration for Raft protocol. The Raft protocol is also used in Redis. It has three roles: Leader, Candidate, and follower. ZAB protocol is similar to Raft, where it needs to select a leader.

Etcd Lab

In troubleshooting, if we suspect that the response from API server is inconsistent with etcd store, we want to directly connect to it.

Managed Kubernetes services do not expose their etcd store. We can use KinD or Minikube. There are two types of jump box to access etcd store: using etcd Pod, or SSH to a Node. To connect to etcd, we also need the X509 key, certificate and CA’s certificate, in addition to the endpoint, usually an IP with port 2389. When I connect to Pod shell, I find the command shell not easy to use. They might miss basic command such as ls, or do not support auto completion.

Take KinD for example, we first create a secret, then we can connect to the node with docker CLI command:

kubectl create ns myns
kubectl -n myns create secret generic mysecret --from-literal key1=value1
kubectl -n myns get secret mysecret -o jsonpath='{.data.key1}' | base64 -d
docker exec -it control /bin/bash

From the node,

apt update && apt install etcd-client
etcdctl version
nc -vz localhost 2379
cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd

export ETCDCTL_API=3
export ETCDCTL_CERT=/etc/kubernetes/pki/apiserver-etcd-client.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/apiserver-etcd-client.key
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_ENDPOINTS='https://127.0.0.1:2379'
etcdctl member list write out=table

Now we can see the secret object directly with etcd store:

etcdctl get /registry/secrets/myns/mysecret

With get query, when using –prefix, we can use –keys-only switch to list keys without values:

etcdctl get --prefix /registry/api --keys-only
etcdctl get --prefix /registry/namespace -wjson

We can write key-value with put command:

etcdctl put myloc 0
etcdctl get myloc -wjson

In Kubernetes, all the key names start with / which makes the key looks like a POSIX path. Every Kubernetes object is stored in etcd with a unique key following a self-explanatory naming pattern. To display the path, we can also use debug log that records the call to API server:

kubectl get ns myns -v9

Look for curl command such as:

I0523 22:51:43.517728 32347 round_trippers.go:466] curl -v -XGET -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.23.6 (darwin/amd64) kubernetes/ad33385" 'https://127.0.0.1:64081/api/v1/namespaces/myns'

From there we can see the etcd query as the URI is namespaces/myns, which we use in etcdctl query path:

etcdctl get /registry/namespaces/myns

Every type of Kubernetes object has a storage.go file in their implementation that defines how api server should write object. Here is an example for Pod object.

Etcd also supports watch command to watch for changes. For example:

etcdctl watch --prefix /registry/namespace # watch output k create ns newns

Now we create a namespace with kubectl:

kubectl create ns myns

The output from etcdctl will reflect the change. The communication between etcdctl and etcd is gRPC protocol. The output is based on stream, as we can see from the watch result.

Etcd Maintenance

Like any distributed store, etcd needs maintenance and operation work. For example, we can check endpoint status with endpoint command:

etcdctl endpoint status

We can also backup and restore etcd store with etcdctl command:

etcdctl snapshot save /tmp/backup.db

This was an question in CKA exam. In real life, when the workload scales up, the etcd store may come across many pitfalls, such as degraded performance, unresponsiveness, some etcd member going down, network partition on etcd store causing split brain. It is important to ensure efficient communication between API server and etcd store. The etcdctl provides defrag and compact commands for common maintenance activities.

Logging and Monitoring in Kubernetes with PLG stack

Wed, 13 Oct 2021 21:29:00 -0400

We’ve checked out the the actors in PLG stack (Promtail, Loki, Node Exporter, Prometheus, Grafana) and whipped up a quick pipeline on MacOS. Now I’m going a little further to implement the same PLG stack (Prometheus Loki and Grafana) in a Kubernetes cluster. This setup is for demo only, therefore no persistent storage is enabled.

Test Workload

I host a deployment of Flog with three pods running on Minikube. Flog is an open-source emulating log generation behaviour of an application. On the Minikube cluster we start the deployment as below:

minikube start --driver=hyperkit --container-runtime=containerd --memory=12288 --cpus=2
kubectl create ns obsv
kubectl -n obsv create deployment flog --image=mingrammer/flog --replicas=3 -- flog -f rfc3164 -l -d 300ms
kubectl -n obsv get po

The Pods will come up in a heartbeat.

I will use helm to install the objects required for logging and metrics pipelines. There are multiple Helm charts for each components. Some high-level charts (usually with a name suffix of -stack) contain several other resource as sub-charts. They are created as one-stop-shop for multiple components but I found none of them serve my exact purpose. For example, both loki-stack and kube-prometheus-stack include Grafana. But I only need one instance of Grafana. Therefore I stick to the low-level charts.

Log Shipping

Add a helm repo, and install loki and promtail. Note that we need to specify correct loki address when installing Promtail.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --namespace obsv --install loki grafana/loki
helm upgrade --namespace obsv --install promtail grafana/promtail --set "config.lokiAddress=http://loki.obsv.svc.cluster.local:3100/loki/api/v1/push"

When installing Promtail, a DaemonSet is created on the Node. The default configuration applies appropriate configuration and tagging strategy for Kubernetes Pod and Node. So the only customization I specified is Loki address. We can then check logging with Loki. To do so, first expose port 3100 to host, and then use logcli (e.g. on MacOS) to query for logs:

kubectl -n obsv port-forward service/loki 3100:3100
logcli labels
logcli labels pod
logcli query '{pod="flog-775d5fc5c8-p4rlx"}'

Log lines should be pumped to Loki a minute after Loki comes up. The logcli labels command should display the tags. The logcli query command should return the log lines.

Metrics

I use Premetheus with node exporter. In its architecture, Prometheus contain the server, the pushgateway, and alertmanager. The helm chart for Prometheus contains all of those components. It also has a dependency repo for kube-state-metrics. To install:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm --namespace=obsv install prometheus prometheus-community/prometheus
kubectl -n obsv get svc

Once installed, a DaemonSet for Prometheus Node Exporter is created. The exporter is already configured by default for Kubernetes monitoring. The Prometheus server is also configured, on port 80 by default. it needs to be forwarded in order to access from Browser:

kubectl --namespace obsv port-forward service/prometheus-server 9100:80

To verify installation of Prometheus, browse to localhost:9100 to examine the metrics.

Visualization

Last but not least, I will configure Grafana. The repo has been added already so we’ll just install the chart:

helm upgrade --namespace obsv --install grafana grafana/grafana
kubectl get secret --namespace obsv grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

The second command retrieves the credential. To access the web portal, we need port forwarding again:

kubectl port-forward --namespace obsv service/grafana 3000:80

To verify installation, browse to http://localhost:3000 and log in as user admin with the password above. Then add two data sources:

Type: Prometheus, URL: http://prometheus-server.obsv.svc.cluster.local:80
Type: Loki, URL: http://loki.obsv.svc.cluster.local:3100

Then we can explore data using both data sources.

PLG stack

Summary

In the last two posts I reviewed the setups for PLG stack in Kubernetes, from a regular environment to k8s cluster. Fluentd, Prometheus are both CNCF projects. The PLG stack seems to be more adopted than EFK but both have their own advantages. Welcome to the PLG vs EFK debate.

Certified Kubernetes Administrator (CKA) Exam

Fri, 30 Apr 2021 09:50:00 -0400

The Certified Kubernetes Administrator (CKA) exam is a hands-on session where you need to follow the instructions to configure the system in a bash terminal on the web browser. In my experience, some shortcut keys (such as Alt+F) do not work, which slows me down a little bit. For each question, you need to switch kubectl context as instructed in the question. Some questions share the same context so it is very easy to omit this step. You can verify response with your own command but will not be told whether you scored in each question. During the CKA exam I tried to spin up a terminal session from within Vim editor and the terminal ran out of buffer. I had to reboot the machine with the help of proctor, and my completed work are saved.

In general this is an exam I enjoy preparing and writing because it is very hands on. Result is out a day after, and I passed at 96%. I heard about tight timelines but I managed to finish 15 minutes before the end, most likely owing to my dexterity with Linux commands. With that I’m happy to share my notes in preparation for the CKA exam.

For taking CKA exam, we should be familiar with the diagram above.

Tips for Troubleshooting

The CKA exam is hands-on and therefore requires quite a bit of troubleshooting. Here are my notes.

Check Node status to start with
Check core services on each node:
- sudo systemctl status kubelet
- sudo systemctl status docker
- sudo journalctl -u kubelet
- sudo journalctl -u docker
Check component logs (on hosting VM)
- /var/log/kube-apiserver.log
- /var/log/kube-scheduler.log
- /var/log/kube-controller-manager.log
If cluster is built by kubeadm, then some of those services are running in Pods within kube-system namespace. Check those pods:
- run interactive shell: > kubectl exec podname –stdin –tty — /bin/sh
- there is an image for lots of useful network tool called nicolaka/netshoot

Store pod names to variable. e.g. > POD_NAME=$(kubectl get pods -l run=nginx -o jsonpath=”{.items[0].metadata.name}”)
With kubectl, you may
- alias it to k for faster typing
- –dry-run: to run imperative command without creating object
- –record: record the command that was used to make a change
- -o: set output format, wide, yaml, or jsonpath=”expression”. For example, to get pod name: > kubectl get pods -l run=nginx -o jsonpath=”{.items[0].metadata.name}”
- –sort-by: use JSONPath expression
- –selector: filter results by label

Build K8s cluster using kubeadm

The CKA exam requires you to know how to build cluster with kubeadm. This involves installing four components (docker-ce, kubeadm, kubectl and kubelet), as outlined below:

step	command
1. Install docker-ce	> curl -fsSL https://download.docker.com/linux/ubuntu/gpg \| sudo apt-key add - > sudo add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" > sudo apt-get update > sudo apt-get install -y docker-ce=18.06.1~ce~3-0~ubuntu > sudo apt-mark hold docker-ce > sudo systemctl status docker
2. Install kubeadm, kubelet and kubectl	> curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg \| sudo apt-key add - cat << EOF \| sudo tee /etc/apt/sources.list.d/kubernetes.list deb https://apt.kubernetes.io/ kubernetes-xenial main EOF > sudo apt-get update > sudo apt-get install -y kubelet kubeadm kubectl > sudo apt-mark hold kubelet kubeadm kubectl
3. Form a K8s cluster	On master node: > sudo kubeadm init --pod-network-cidr=10.244.0.0/16 This command prints out a command for worker nodes to join.	On worker node: sudo the command generated on master
4. Configure kubectl	On master node: > mkdir -p $HOME/.kube > sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config > sudo chown $(id -u):$(id -g) $HOME/.kube/config	Optionally on worker node: > mkdir -p $HOME/.kube then scp $HOME/.kube/config from control plane node
5. Set up cluster networking	> echo "net.bridge.bridge-nf-call-iptables=1" \| sudo tee -a /etc/sysctl.conf > sudo sysctl -p Then from any environment with kubectl, bring up the system pods for cluster networking > kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Add new node to KubeAdm cluster

This is fairly simple with the help of kubeadm. The node to join cluster must be able to communicate with master node. Create a token and print join command from master node:

> kubeadm token create --print-join-command

Then from the node to join, run this command as sudo. You will see that it performs the TLS bootstrap for you. Once completed, the standard output will say this node has joined the cluster. You can confirm with command:

> kubectl get nodes

Sometimes one needs to migrate pods to the newly joined node. This can be done by draining the existing nodes.

Note you can also use kubespray to build K8s cluster as previously discussed, and here is a my IaC project to launch AWS instances and build a K8s cluster with kubespray on top of it. For my learning, I often create a GKE (Google Kubernetes Engine) cluster from GCP’s cloudshell. There is a guide on how to start a cluster but it comes down to three commands:

$ gcloud config set compute/zone us-east1-b
$ gcloud container clusters create tcluster --num-nodes=3
$ gcloud container clusters get-credentials tcluster

The third command above is to configure kubectl on the cloudshell. Follow this guide if you need to SSH to node.

Upgrade KubeAdm cluster

This involves upgrade three components (kubeadm, kubectl and kubelet) on two types of node: master node and worker node. They steps vary slightly for two nodes. But drain and uncordon is needed for both types of nodes. Pick a node and follow the steps below:

Step	Command
1. drain the node from kubectl client (e.g. master node)	> sudo kubectl drain nodename --ignore-daemonsets
2. Determine kubeadm target version	> apt-mark showhold > sudo apt-mark unhold kubeadm kubectl kubelet > apt list --installed \| grep kube > apt-cache show kubeadm \| less > sudo apt-get install -y kubeadm=1.20.2-00
3. update kubeadm	On master node: > sudo kubeadm upgrade plan v1.20.2 > sudo kubeadm upgrade apply v1.20.2	On worker node: > sudo kubeadm upgrade node
4. On the node to update, determine target version for kubectl and kubelet, then install	> apt-cache show kubectl \| less > apt-cache show kubelet \| less > sudo apt-get install -y kubectl=1.20.2-00 kubelet=1.20.2-00
5. Restart kubelet	> sudo systemctl daemon-reload > sudo systemctl restart kubelet
6. Uncordon	> kubectl uncordon nodename

Backup and restore Etcd

Etcd is a distributed key-value store. It uses Raft protocol for distributed consensus. Etcd is the third distributed system I touch on. The previous two are: Cassandra (using Paxos protocol for distributed consensus) and ZooKeeper (using ZAB protocol). Here is a good article that summarizes the protocols. As for the exam we only need to use etcd with the client tool.

The etcd itself can run on a cluster of servers, each running etcd as a systemd service as etcd/etcd (user/group). It can be deployed in two ways:

stacked etcd: an instance of etcd lives with kube-api-server on the same control plane node
external etcd: in a dedicated cluster of etcd

Alternatively, etcd can run as a pod, most likely in kube-system namespace.

The etcd service listens on port 2379 for client communication and on port 2380 for server (peer-to-peer) communication. When the systemd service was initialized there are a few key environment variables (e.g. cert locations, ETCD_DATA_DIR) privoded as configuration. To see them, run:

> cat /etc/systemd/system/etcd.service | grep Env

These environment variables (prefixed with ETCD_) are for the service only. They can provide current configuration information for us to use later. When it’s running as a pod, check out the directory for static pod for the yaml declaration (e.g. /etc/Kubernetes/manifests/etcd.yaml), where these parameters are passed in as environment variable.

The etcdctl utility is a command line client for etcd. The default API version is 3 so no need any more to set ETCDCTL_API=3 before each command. The utility needs three arguments three arguments (–cacert, –cert, and –key) but we can pass the information via environment variables:

> export ETCDCTL_CACERT= /home/cloud_user/etcd-certs/etcd-ca.pem
> export ETCDCTL_CERT= /home/cloud_user/etcd-certs/etcd-server.crt
> export ETCDCTL_KEY= /home/cloud_user/etcd-certs/etcd-server.key
> export ETCDCTL_ENDPOINTS=https://etcd1:2379

The environment variable names are uppercase of the argument name with prefix ETCDCTL_. Only global options of arguments can be supplied via environment variables. They remain effective throughout the rest of activities. Also note that the CACERT is needed only when client-cert-auth is true. Now to backup, we can simply run:

> etcdctl snapshot save /home/cloud_user/etcd_backup.db

To restore from a file, you want to remove existing etcd data directory first. The directory can be found in ETCD_DATA_DIR variable. Suppose it is /var/lib/etcd, you need root permission to write to it, then correct ownership before starting the service:

> sudo systemctl stop etcd && sudo mv /var/lib/etcd/ /tmp/
> sudo etcdctl snapshot restore /home/cloud_user/etcd_backup.db --data-dir /var/lib/etcd
> sudo chown -R etcd:etcd /var/lib/etcd && sudo systemctl start etcd

To verify the restore result, simply run:

> etcdctl get cluster.name

Object Management

In the CKA exam, we need to interact with many types of built-in Kubernetes objects.

RBAC objects:
- A Role defines permissions within namespace.
- A ClusterRole defines cluster-wide permissions.
- Both Roles and ClusterRoles are K8s objects that defines a set of permissions
- RoleBinding and ClusterRoleBinding are objects that connect Roles and ClusterRoles to users.
Service Account: an account used by container processes within Pods to authenticate the K8s API. If your Pods need to communicate with the K8s API, you can use service accounts to control their access.

Inspect resource usage either with a K8s Metrics Server, or by command:

> kubectl top pod --sort-by <JSONPATH> --selector <selector>

Here is a good guide to install metrics server and dashboard (e.g. on docker-desktop).

Pods and Containers

ConfigMaps: store data in key-value map.
Secrets: same as ConfigMaps but for sensitive data only
Two ways to pass ConfigMap and Secret data to your container:
- As environment variables in container operating system
- As files presented on mounted volumes in container file system.
Container Resource management:
- Resource requests: K8s scheduler will use resource requests to avoid scheduling pods on nodes that do not have enough available resources. 1 CPU unit = 1/1000 of one core
- Resource limits: allow you to limit the amount of resources your containers can use. The container runtime is responsible for enforcement. The enforcement behaviour is different. For example, some terminates container that attempts to use more resource than the limit.
Probes
- Liveness Probe: automatically determine whether or not a container application is in a healthy state. By default K8s does not consider a container to be down until the container process stops. Liveness Probe allow you to customize this detection mechanism and make it more sophisticated.
- Startup Probes: similar to liveness probes. However, while liveness probes run constantly on a schedule, startup probes run at container startup and stop running once they succeed. Startup probes are used to determine when the application has successfully started up. It is especially useful for legacy applications that can have long startup times.
- Readiness Probes: determine when a container is ready to accept requests. When you have a service backed by multiple container endpoints, user traffic will not be sent to a particular pod until its containers have all passed the readiness checks defined by their readinesse probes. Use readiness probes to prevent user traffic from being sent to pods that are still in the process of starting up.

Restart policy for self-healing pods
- (default) Always: container will always be restarted if they stop, even if they completed successfully (returned 0).
- OnFailure: container will be restarted if the container process exists with an error code, or the container is determined to be unhealthy by a liveness probe.
- Never: let it be
Multi-container pods:
- containers share the same networking namespace and can communicate with one another on any port, even if the port is not exposed to the cluster
- Container can use volumes to share data in a Pod. Example: a legacy application is hard-coded to write log output to a file on disk. You use a sidecar container to read the log file from shared volume and prints it to the console so the log output will appear in the container log.
Init containers: containers that run once during the startup process of a pod. A pod can have any number of init containers, and they will each run once into completion, before the next init container starts. You may use init containers to perform a variety of startup tasks, they can contain and use software and setup scripts that are not needed by your main containers. They are often useful in keeping your main containers lighter and more secure by offloading startup tasks to a separate container. Use case include:
- cause a pod to wait for another K8s resource to be created before finishing startup
- perform sensitive startup steps securely outside of app containers
- populate data into a shared volume at startup
- communicate with another service at startup
Scheduling: Scheduler (a component in control plane) assigns Pods to a suitable Node so kubelets can run them. The factor taken into account:
- resource request vs available node resources
- various configurations that affect scheduling using node labels
Pod allocation
- nodeSelector is an attribute of Pod to allow you to limit which Node(s) the Pod can be scheduled on. The selector is based on label.
- nodeName is an attribute of Pod that allows you to bypass scheduling and assign Pod to a specific Node by name.
DaemonSet: automatically runs a copy of a Pod on each node. When a new node is added to the clsuter, DaemonSet will run a new copy of the Pod on it. DaemonSets also respect normal scheduling rules around node labels, taints and tolerations. If a pod would not normally be scheduled on a node, a DaemonSet will not create a copy of the Pod on that node.
Static Pod: A Pod that is managed directly by the kubelet on a node, not by the K8s API server. They can run even if there is not K8s API server present. Kubelet automatically creates static Pods from YAML manifest files located in the manifest path on the node.
Mirror Pod: Kubelet will create a mirror Pod for each static Pod. Mirror Pods allow you to see the status of the static Pod via the K8s API, but you cannot change or manage them via the API.
Taints: applied to nodes to repel a set of pods. A taint specifies key-value and effect. Effect can be NoSchedule or NoEffect. The former prevents pods without matching tolerations to schedule to the tainted node. The latter also evicts pre-existing pods with no matching toleration.
Tolerations: applied to pods so they can be scheduled to nodes with matching taints. A toleration consists of key-value pair, effect and operation. The operation can be Equal or Exists. To determine whether a toleration matches a taint. The keys and the effects must be the same. In addition:
- the operator is Exists (and thus no value should be specified in the toleration); or
- the operator is Equal, and all the values match those of the taints;

Now we have three ways to influence the scheduling behaviour. The first, is simply by specifying nodeSelector on the Pod, with the required the node label. The second, as just discussed, is to use Taints and Tolerations. The third way, is similar to the first, using nodeAffinity attributes on Pods. nodeAffinity is more powerful and flexible than nodeSelector by supporting more complex scheduling rules (e.g. matching rules).

Use Node Affinity when your scheduling rule is based on direct condition, i.e. schedule a Pod to this Node when XXX. In this case, you have well-known labels on nodes, and specify nodeAffinity on Pods.
Use Taints and Tolerations when your scheduling rule is based on inverse statement, i.e. do not schedule a Pod to this Node unless XXX. In this case, you put a taint “MyCondition:NoSchedule” on a Node, so that no Pod will ever get scheduled to this Node. The only exception is when a Pod has the Toleration “MyCondition:NoSchedule”.

Deployments

Deployment is an object that defines a desired state for a ReplicaSet (a set of replica Pods). The Deployment Controller seeks to maintain the desired state by creating, deleting, and replacing Pods with new configurations.
With Deployments, you can horizontally scale an application up and down by changing the number of replicas. You can perform rolling updates and rollback.

Networking

The K8s network model defines how Pods communicate with each other, regardless of which Node they are running on.
Each Pod has its own unique IP address within the cluster. Any Pod can reach any other Pod using that Pod’s IP address. This creates a virtual network that allows Pods to easily communicate with each other.
One type of K8s network plugin is CNI plugin. It has many flavours such as Calico. Each plugin has its own unique installation process. Kubenetes nodes will remain NotReady until a network plugin is installed.
The K8s virtual network uses a DNS (e.g. a Kubeadm cluster uses CoreDNS pod in kube-system namespace) to allow Pods to locate other Pods and Services using domain names. The Pod DNS name follows this format: pod-ip-address.namespace.pod.cluster.local
A K8s NetworkPolicy is an object that allows you to control the flow of network communication to and from Pods so you can isolate traffic. NetworkPolicy can apply to Ingress (using from selector), Egress (using to selector) or both.
NetworkPolicy has an attribute podSelector to determine to which Pods in the namespace the NetworkPolicy applies, by selecting Pods by with Pod labels.
By default, Pods are considered non-isolated and completed open to all communication. If any NetworkPolidy selects a Pod, the Pod is considered isolated and will only be open to traffic allowed by NetworkPolicies.
A variety of selector can be used: podSelector, namespaceSelector, ipBlockSelector and port.

Services

Services provide a way to expose an application running as a set of pods, so clients can access applications in an abstract way without needing to be aware of the application pods. In this model, client make requests to a Service, which routes traffic to its pods in a load-balanced fashion
Endpoints are the backend entities to which Services route traffic. If there are multiple Pods behind a service, each Pod will have an endpoint associated with the service.
Each service has a type that determines how and where service will expose your application.
- ClusterIP: expose application inside the cluster network
- NodePort: expose application outside the cluster network
- LoadBalancer: expose application outside thecluster network, but use an extermal cloud load balancer from cloud platform.
Services are assigned with DNS names. The FQDN follows this format: service.namespace.svc.cluster-domain.example, which is used by pods across namespaces
Pods within the same namespace can reference service simply by service name.
To manage external access to service, you can also use Ingress object. Ingress object is capable of providing more functionality than a simple NodePort Service, such as SSL termination, advanced load balancing, or name-based virtual hosting. You must install one or more Ingerss controller (many different implementations) to back up the ingress objects.
Ingress defines a set of routing rules. Each rule has a set of paths, each with a backend. Requests matching a path will be routed to its associated backend.
If a Service uses a named port, an ingress can also use the port’s name (instead of port number) to choose to which port of a service it will route.

Storage

Volumes allow you to store data outside the container file system, while allowing the container to access the data at runtime. When Pod is gone, volumes do not persist.
Persistent Volumes are a slightly more advanced form of Volume. They allow you to treat storage as an abstract resource and consume it in Pods. PV can be provisioned separately by storage administrator, and they persist regardless of pod lifecycle. PV needs to be claimed by pods. PV uses a set of attributes to describe the underlying storage resource.
Both volumes and PVs each have a volume type: NFS, Cloud (AWS, Azure, GCP), ConfigMaps and Secrets, Simple Directory on node
Two volume types to distinguish:
- hostPath: store data in a specified directory on K8s node
- emptyDir: store data in dynamically created location on the node. The directory exists only as long as the Pod exists on the node. The directory and the data are deleted as Pod is removed. This type is useful for simply sharing data between containers in the same pod.
Both volumes and PVs are specified under Pod, and individual containers must include volumeMounts object to map volume name to local mountPath
Storage Class object allow K8s admins to specify the types of storage services they offer on their platform. A key property is allowVolumeExpansion. This allows PVC to resize. At storage class level, there are two reclaim policies: Retain and Delete. The default is Delete.
PV has an attribute named persistentVolumeReclaimPolicy. This is reclaim policy at PV level. If the attribute is not defined, it is inherited from storage class. The persistentVolumeReclaimPolicy has three options. When PVC is deleted:
- Retain: keeps all data but requires admin to manually reclaim the volume (i.e. delete PV, clean up data, delete storage asset)
- Delete (cloud storage only): deletes both PV and the underlying storage resource automatically
- Recycle: scrub (rm -rf /vol/) all data in the underlying storage resource, and allow the volume to be reused.
PVC represents a user’s request for storage resources. It defines a set of attributes similiar to those of a PV. When a PVC is created, it will look for a PV that is able to meet the requested criteria. If it finds one, it will automatically be bound to the PV. PVC can be mounted to a Pod’s containers just like any other volume

In general the CKA exam experience is quite positive and rewarding. In future posts I will shift focus on Kubernetes not only for the CKA exam, but also for keeping track of my learning.

Good luck with your CKA exam.

Blockchain and DeFi

Sat, 23 Jan 2021 11:30:00 -0400

Background

This is a reading note for book “How to DeFi“.

When Tom hand in $20 to Jack in exchange of goods. Both Tom and Jack has to agree that the greenback with $20 sign actually is worth the value of the goods. Since the paper money is signed by some big shot from the central banker, which they both trust, they can therefore reach consensus on the value of that paper. The central bank acts as a centralized body of trust. Now you’d ask what if the central banker cheats on us? As long as we centralize our trust to a single body, we have to worry about the centralized trust deteriorate. This is what decentralized finance aspires to address.

The other background problem is the centralized payment and clearance system. When you need to send money from one country to another, there are not only high fees involved, but also days holding for clearance. This is another opportunity for decentralized finance.

Decentralized Finance (DeFi)

The technologies in DeFi falls under three categories based on the level of decentralization:

Centralized: custodial, uses centralized price feeds, centrally determined interest rates
Semi-Decentralized: non-custodial, decentralized price feeds
Completely Decentralized: every component is decentralized.

Most DeFi dapps are sitting in the semi-decentralized category. There is no DeFi protocol that is completely decentralized yet. DeFi involves protocols that covers financial sectors such as Lending & Borrowing, Exchanges, Derivatives, Lottery, Payments, Insurance, etc. This all sounds futurism, but there are a few protocols already at play.

Ethereum

The majority of the DeFi Dapps are built on the Ethereum blockchain, a global, open-source platform for decentralized applications. You can think of it as a world computer that cannot be shutdown. Developers can also deploy smart contracts to the Ethereum network, where it will run 24/7. Smart Contract is a programmable contract that allows two counterparties to set conditions of a transaction without needing to trust another third party for the execution. Whenever a certain condition is fulfilled, the smart contract will carry out the operation as programmed, and the process is transparent to all involved parties, bypassing the need for a trusted third party intermediary.

Ether is the native currency of the Ethereum blockchain so Ether is similar to Bitcoin. Ether is also used to pay for the fee that allows smart contracts and Dapps to run on the Ethereum network. Ether is also evolving to become its own unique reserve currency and store of value.

Dapps (decentralized applications) are interfaces that interact with the blockchain through the use of smart contracts. On Ethereum, all transactions and smart contract executions require a small fee to be paid. The fee is called Gas. In technical terms, Gas refers to the unit of measure on the amount of computational effort required to execute an operation or a smart contract. Gas fees are paied entirely in ETH. The price of gas can fluctuate from time to time depending on the network demand.

Ethereum can also be used for two other functions: creating DAO (Decentralized Autonomous Organization), or issuing other cryptocurrencies. A DAO is a fully autonomous organization which is not governed by a single person but is instead governed through code. This code is based on smart contracts and enables DAOs to replace how traditional organizations are typically run. As it runs on code, it would be protected from human intervention and will operate transparently. Governance decisions or rulings would be decided via DAO token voting.

There are currently two popular protocols for tokens on the Ethereum Network: ERC-20 and ERC-721

A wallet is a user-friendly interface to the blockchain network. It manages your private keys, which are basically keys to the lock on your cryptocurrencies’ vault. Wallets allow you to receive, store and send cryptocurrencies.

Custodial wallets are wallets where third-parties keep and maintain control over your cryptocurrencies on your behalf. By using a custodial wallet, you trust an external party to store your coins safely. However, by trusting a third party with your cryptocurrencies, you open yourself up to the risk of the custodian losing your cryptocurrencies through mismanagement or hacks (Mt. Gox)

Non-custodial wallets are wallets where you take full control and ownership of your cryptocurrencies. By using a non-custodial wallet, you trust no external party and only yourself to ensure safe storage. However, you pass the burden of security to yourself and you have to be fully equipped to store your private keys safely. If you lose your private keys, you will lose access to your cryptocurrencies too. Example of non-custodial wallet: Argent

Stablecoins

Stablecoins are pegged to other stable assets such as the USD. The top 5 cryptocurrency stablecoins as of Feb 2020 are:

Tether (USDT)
USD Coin (USDC)
Paxos Standard (PAX)
True USD(TUSD)
Dai (DAI)

Not all stablecoins are the same as they employ different mechanisms to keep their peg against USD. There are two types of pegs: fiat-collateralized (e.g. USDT)and crypto-collateralized (e.g. DAI). Most stablecoins are the former.

USDT pegs itself to $1 by maintaining reserves of $1 per Tether token minted. While Tether is the largest and most widely used USD stablecoin, Thether reserves are kept in financial institutions and users will have to trust Tether as an entity to actually have the reserved amounts that they claim. Tether is therefore a centralized, fiat-collateralized stablecoin.

Dai (DAI) on the other hand, is collateralized using cryptocurrencies such as Ethereum (ETH). Its value is pegged to $1 through protocols voted on by a decentralized autonomous organization and smart contracts. At any given time, the collateral to generate DAI can be easily validated by users. DAI is a decentralized, crypto-collateralized stablecoin.

DAI has a smaller market capitalization but is increasing tremendously. DAI is the native stablecoin used most widely in the DeFi ecosystem. It is the preferred USD stablecoin used in DeFi trading, lending and more. DAI operates on Maker, a smart-contract platform that runs on the Ethereum blockchain and has three tokens:

Sai – aka Single Collateral Dai, backed only by Ether(ETH) as collateral. It is legacy Dai, and will be phased out.
Dai – aka multi-collateral Dai. Currently backed by Ether (ETH) and Basic Attention Token (BAT) as collaterals with plans to add other assets
Maker (MKR): is Maker’s governance token and users can use it to vote for improvements on the Maker platform via the Maker Improvement Proposals. Maker is a type of DAO.

MKR holders have voting rights proportional to the amount of MKR tokens they own in the DAO and can vote on parameters governing the Maker Protocol. The parameters that MKR holder vote on are vital in keeping the ecosystem healthy, which in turn helps ensure that Dai remains pegged to $1.

The amount of Dai that can be minted is dependent on the collateral ratio (150% worth of ETH or BAT to mint Dai). There is a stability fee and Dai Saving Rate (DSR).

Lending and Borrowing

Compound Fiannce is an Ethereum-based open-source money market protocol where anyone can supply or borrow cryptocurrencies frictionlessly. Many tokens (BAT, ETH, USDC, DAI, and more) can be supplied or used as collateral on the Compound platform.

Compound operates as a liquidity pool that is built on the Ethereum blockchain. Suppliers supply asset to the pool and earn interest, while borrowers take a loan from the pool and pay interest on their debt. In essense, Compound bridges the gaps between the lenders who wish to accrue interest from idle funds and borrowers who wish to borrow funds for productive or investment use. Suppliers and Borrowers interact directly with the protocol for interest rate.

Anyone with a supported cryptocurrency wallet such as Argent can start using Compound immediately. To earn interest, you have to supply assets to the protocol. Once you have deposited your asset into Compound, you will immediately begin to earn interest on the assets you have put in. Upon deposit, you will receive corresponding amounts of cTokens. If you supply DAI, you will receive cDAI. If you supply Ether, you will receive cETH. Interest is not immediately distributed to you, but rather accrues on the cTokens which you now hold and are redeemable for the underlying asset and interest it represents. cTokens represents your balance in the protocol. cTokens become convertible into an increasing amount of the underlying asset it represents over time.

If you want to borrow, you have to first supply assets into the system as collateral for your loan. Borrowed assets are sent directly to your Ethereum wallet and from there you can use them. Do not that borrowing incurs a small fee of 0.025% to avoid spams and misuse of the Compound protocol

DEX (Decentralized Exchanges)

Uniswap Exchange is a decentralized token exchange protocol built on Ethereum that allows direct swapping of tokens without the need to use a centralized exchange. On Uniswap, you can simply swap your tokens directly from your wallet without having to go through centralized exchange.

dYdX is a decentralized exchange protocol for lending, borrowing and margein/leveraged trading. It supports ETH, USDC, and DAI. You can enter either short or long positions with leverages up to 5x.

Other Use Cases

Derivatives: A derivative is a contract whose value is derived from another underlying asset such as stocks, commodities, currencies, indexes, bonds, or interest rates. There are several types of derivatives such as futures, options and swaps, each serving a different purpose. In DeFi, the biggest derivative protocol is Synthetix

Fund Management: In DeFi, fund management is conducted in a manner where it removes the investment manager and lets you choose the asset management strategy that best suits your financial need. TokenSets is a platform that allows crypto users to buy Strategy Enabled Tokens

Payments: Lighting Network, Request Network, xDai and Sablier

Insurance: Nexus Mutual is a decentralized insurance protocol built on Ethereum that currently offers cover on any smart contract on the Ethereum blockchain.

Dashboard: a dashboard is a simple platform that aggregates all your DeFi activities in one place.

High Performance Computing

Fri, 11 Dec 2020 23:42:00 -0400

Overview

High Performance Computing (HPC) has recently been commoditized with the advent of commodity server hardware (x86 server), virtualization technology and cloud delivery model. It is common in specialized industries where intensive computing tasks are required, for example:

HCL (healthcare and life science): drug discovery, computer aided diagnosis (CAD), genome engineering;
CAD, CAE, CAM (computer aided design, engineering, and manufacturing): 3D modeling, computational fluid dynamics (CFD), finite element analysis (FEA), structural mechanical design, etc
Finance: portfolio management, automated trading, risk analysis
Geoscience and geo-engineering: oil and gas exploration, geographic data, weather forecasting;
Scientific computation

Computing performance is measured in FLOPS (floating point operations per second) and is usually delivered in a cluster to aggregate the computing power from a number of networked nodes. This is referred to as an HPC cluster.

Hardware stack

An HPC cluster features the following components:

Head node (aka master node or login node): a gateway and coordinator; head node may be broken into several nodes
Compute node (worker node): the executor of jobs; the compute node can either be homogenous or heterogeneous, for different purposes. the number of compute nodes can be quite large

There are four common form factors for server: tower, rack-mount, blade, mainframe. Traditionally, the nodes are rack-mount 1U “pizza box” servers. Bladed systems started to replace due to the increased node density, thanks to the shared/redundant power and cooling management. In the past, the HPC cluster is operated in data centres, which is an expensive operation item. In the last decades, many organizations extends their compute workload to the cloud, forming a hybrid model.

HPC typically has specialized storage system because HPC applications notoriously create large amounts of data. NFS traditionally does not scale well as number of node increases. Some proprietary storage system such as Isilon provides good performance via NFS protocol. There are also open-source parallel file system such as Lustre and HDFS.

HPC networking handles three types of traffic:

computation traffic between compute nodes (if the compute nodes interact with each other)
file system traffic: for compute nodes to read and write on file system (e.g. NFS)
administrative traffic: fairly light compared to the two above

For that, many HPC runs two networks, a private (backend) network and a public (frontend) network. Backend network must be high speed and low latency, typically in the form of 10Gig Ethernet, or InfiniBand.

Software stack

On the software layer, the core functionality is Message Passing Interface (MPI), a specification for the developers and users of message passing libraries. MPI constitutes a standardized and portable message-passing system which consists of a library and a protocol to support parallel computing. MPI enables passing information between various nodes of a HPC cluster or between particular clusters, and has different implementations that provide the libraries to run HPC applications in a distributed manner across different physical nodes.

In the operation, user submits a job through head node in order to request the resource. User needs to specify the resources for the job (e.g. how many CPU cores, how much memory, etc). The head node runs a scheduler to allocate computing resource based on pre-defined policies, based on priority of jobs, availability of resources, distribution of load, etc. Depending on the nature of the computing jobs, the nodes participating in the task may or may not communicate with one another. If they do need to talk to each other, the program must support it. Such program can be called a cluster program, and the MPI (message passing interface) library greatly facilitates the development of such program. The sub-jobs communicating with each other also creates a considerable amount of network traffic within the cluster.

Cluster software ties all nodes in the cluster together. It turns raw hardware into a functioning cluster by provisioning (installing and configuring) the head nodes. Compute nodes can usually be added or removed dynamically therefore the head nodes should be able to provision compute nodes, and administer cluster, leaving the programming as the job for the user to complete. As mentioned, in parallel programming, the most important HPC tool is MPI (Message Passing Interface), which allows programs to talk to one another over cluster networks. There are both open (e.g. Open MPI) and commercial MPI (e.g. Microsoft MPI) versions. Cluster software should also provide compilers, debuggers, and profilers in addition to MPI.

There are cluster software in both Linux and Windows operating systems: Rocks Clusters, Oscar (Open Source Clusters Application Resources), Red Hat HPC solution, Microsoft HPC pack and AWS Parallel Cluster.

Implementation

Here is an example of setting up HPC cluster with CentOS. Despite of the well documented steps, note that the author of the document refers to HPC cluster simply as cluster, which is ambiguous. There are three basic motivators for creating a cluster: high performance computing (HPC), network traffic load balancing, and service resilience in the form of high availability (HA). The author should be specific in the document about the HPC cluster. If RDMA (Infiniband) network is involved, a configuration guide is provided in RedHat literature.

Here is an example of deploying HPC cluster in AWS. Here is the guide to deploy HPC pack in Microsoft technologies.

HPC and Big Data

HPC and Big Data are two distinctive computing paradigmes. Although there is some signs of convergence and blurred boundaries, it is still a long way before one can treat HPC and Big Data interchangeably. This paper does a phenomenal job in comparing the two paradigms.

The fundamental difference lies in the respective problems they intend to address. HPC focuses on the large computational loads, whereas Big Data targets applications that need to handle very large and complex data sets (usually in the order of multi-terabytes or exabytes). Many scientific data analytics applications are becoming I/O bound in modern systems, such as seismic algorithms, Big Data applications are thus very demanding in terms of storage, to accommodate such a masive amount of data, while HPC is usualy thought more in inters of sheer computational needs. The open-source projects in Big Data also aims to run on conventional hardware to make it easier and less expensive to scale. This is not the main focus of HPC.

So, you can run Big Data (e.g. Hadoop) analytics jobs on HPC gear. On the other hand, you can’t run HPC jobs on commodity hardware as commonly seen in the Big Data stack. Both HPC and Hadoop analytics use parallel processing of data. In a Hadoop/analytics environment, data is stored on commodity hardware and distributed across multiple nodes of hardware. In HPC, where the size of data file is much greater, data storage in centralized. Also, because of the sheer volume of its files, HPC also requires more expensive networking communications such as Infiniband, because the size of the file it processes require high throughput and low latency.

In BigData job, each query in Hadoop reads data from disk and runs as a separate MapReduce job. Spark enables in-memory iterative processing (through the RDD abstraction), allowing the user to query repeatedly on a dataset without having to perform intermediate disk operations. RDD are exposed in the Spark API where each dataset is represented as a read-only object, and transformations are invoked using methods on these objects. For an example project, check out this post.

The underlying software stacks for HPC and Big Data are fundamentally different, mainly due to the differences represent in their target class of applications, as outlined in the diagram below:

software stack difference between HPC and Big Data

As to which one is for me, the over-simplified advice is: if you can avoid HPC and just use Hadoop for your analytics, do it. It is cheaper, easier, and more cloud friendly. However, bear in mind that an all-Hadoop shop is not possible for many industries such as life sciences, weather, pharmaceutical, and academic applications.

Zookeeper Summary

Wed, 26 Aug 2020 23:10:00 -0400

Distributed systems

Distributed system involves independent computing entities linked together by network. The components communicate and coordinate with each other to achieve a common goal. In early days, designers and developers often had made some assumptions (aka. fallacies) of distributed computing:

The network is reliable
Latency is zero
Bandwidth is infinite
Network is secure
Topology doesn’t change: in reality, components to a network get removed/added over time. the system should tolerate such changes.
There is one administrator: for distributed systems to function, they interact with external system beyond administrative control.
Transport cost is zero: cost is involved everywhere, in the form of CPU cycles spent, to actual dollars paid to service provider.
Network is homogenous

These fallacies make coordinating distributed computing entities a huge challenge and Zookeeper is introduced to address these challenges. Zookeeper implements common tasks for distributed coordination, such as:

Configuration Management (propagate configuration changes to all worker nodes dynamically)
Naming service
Distributed synchronization (locks and barriers)
Cluster membership operations (e.g. detection of node leave/join)

ZooKeeper is a centralized coordination service for the distributed application. ZooKeeper itself is distributed as well. It runs on its own cluster of servers called a ZooKeeper ensemble, separate from application’s cluster. Distributed consensus, group management, presence protocols, and leader election are implemented by the service so that the application developers do not need to reinvent the wheel by implementing them on their own.

Developers will have to use APIs through ZooKeeper’s client library, which has language bindings for almost all popular programming languages. The client library is responsible for the interactions of an application with the ZooKeeper service. For testing with API access one can alternatively use its Java-based command-line shell (zkCli.sh)

$ zkCli.sh -server zknode:2181

How Zookeeper works

Data Model

ZooKeeper allows distributed process to coordinate with each other through a shared hierarchical namespace of data registers (znodes). The hierarchy start with root node which has child znode(s). Each znode can have their children, as well as store its own data (hence the name data register). The data in a znode is stored in byte format for a maximum of 1MB (ZooKeeper by design is just a coordinator service of host application, so its own data set size is fairly small).

Zookeeper data model

Znodes have two types (set at time of creation)

persistent znode: for storing persistent data, such as configuration. The znodes and their data will exist even if the creator client dies.
ephemeral znode: deleted by ZooKeeper service when the creating client’s session ends (due to disconnection or explicit termination). It can also be explicitly deleted by creator client through delete API call. They cannot have children. Their visibility is controlled by ACL policy

ZooKeeper can assign an incremental sequence number as part of znode name during its creation. This makes a sequential node. Both persistent znode and ephemeral znode can be either sequential or not.

In typical client-server architecture, server is passively open and do not initiate communication to client. Client pulls information from server. This is however an anti-pattern for large scale distributed system. ZooKeeper implements a Watch mechanism where clients can get notifications from ZooKeeper service, instead of having to poll for events. Clients can register with the ZooKeeper service (by setting a watch on znode) for any changes associated with a znode. A watch will only trigger notification once, and needs to be re-registered (by client) for trigger the next notification. A watch is triggered upon:

Any changes to the data of a znode;
any changes to the children of a znode;
Creation of deletion of a znode

ZooKeeper guarantees that notifications are delivered in the order of event occurrence. When a client disconnects from ZooKeeper server, it doesn’t receive any watches until the connection is re-established.

API Operations

The ZooKeeper operations are:

Operation	Description
create	Creates a znode in the specified path
delete	Deletes a znodes from the specified path. Not allowed if the znode has children. version number required
exists	Check if a znode at the specified path exists, and get version number; support watch
getChildren	Get a list of children of a znode; support watch
getData	get the data associated with a znode; support watch
setData	writes data into the data field of a znode. Version number required.
getACL	get the ACL of a znode
setACL	set the ACL in a znode
sync	synchronizes a client’s view of a znode

The write operations (setData, create, delete) are atomic, durable and eventually consistent. Every znode has a stat structure including cZxid, mZxid an dpZxid that keeps track of the ID of the transactions that created, last modified this znode, or pertains to adding or removing its children.

Production znode ensemble with more than one node is running in quorum mode. Updates to ZooKeeper tree by clients must be persistently stored in this quorum of nodes for a transaction to be completed successfully. Odd number of node is recommended to avoid split-brain where network partition causes two subsets of servers in the ensemble function independently, and different clients get different results for the same requests, depending upon the server they are connected to.

All ZooKeeper nodes are listed in the configuration for client application to randomly pick from and try to connect and establish a session. The session is associated with every operation the client executes in a ZooKeeper service. The session also has a timeout period specified by the application client during session establishment. If the connection remains idle for more than the timeout period, the server expires the session. Appropriate session timeout should be set based on network condition. Sessions are kept alive by client sending heartbeat to ZooKeeper service. Application developer needs to handle connection-loss scenarios properly.

Leader Election and Atomic Broadcast

ZooKeeper ensemble contains a leader nodes, follower nodes and observer nodes.

The leader node is elected by the cluster. It handles all write requests.
The follower nodes are leader candidates that are not elected. They are backup to the leader nodes. They handle read request, and receive the updates proposed by the leader, and through a majority consensus mechanism, a consistent state is maintained across the ensemble.
The observer nodes are ineligible as leader candidates. They have otherwise the same function as followers.

The service relies on the replication mechanism to ensure that all updates are persistent in all servers that constitute the ensemble. This is the core mechanism in ZooKeeper, implemented as a special atomic messaging protocol called ZooKeeper Atomic Broadcast (ZAB). ZAB (a variant of Paxos algorithm) ensures the election of new leader in the event of old leader crash, and ensures integrity of data. It defines three states (looking, following and leading) of a node, and goes through four phases (election, discovery, sync, broadcast) in its operation.

All read requests (exists, getData, getChildren) are process locally by the ZooKeeper node where the client is connected to. This makes read operation fast. All write requests (create, delete, and setData) are forwarded to the leader in the ensemble, which carries out the client request as a transaction. A transaction is identified by zxid and is idempotent. Transaction also satisfies the property of isolation (no transaction is interfered with by any other transaction). Only after a majority of the followers acknowledge that they have persisted the change does the leader commit the update.

Transaction processing involves two steps in ZooKeeper: leader election and atomic broadcast. This resembles a two-phase commit protocol (which also includes a leader election and an atomic broadcast)

ZooKeeper use local storage to persist transactions. The transactions are logged to transaction logs, in sync’ed write, requiring a dedicated block device separated from boot device of server. The local storage also keep point-in-time copies (snapshots) of the ZooKeeper tree.

ZooKeeper Recipes

The ZooKeeper recipes defines high-level implementation (construct) of some common distributed coordination mechanism:

Barrier: any thread/process must stop at this point and cannot proceed until all other threads/processes reach this barrier.
Queue: allow FIFO in distributed system
Lock: Fully distributed locks that are globally synchronous, meaning at any snapshot in time no two clients think they hold the same lock.
Leader Election: designate a single process as the organizer of some task distributed among several nodes.
Group membership: node may join or leave a group, which needs to be made available to clients. An alternative to ZooKeeper to manage group membership is gossip protocol.
Service discovery: help client to determine IP and port for a service that are hosted by multiple servers.
Two-phase commit: a mechanism for atomic commitment in two steps: first a commit request phase involving a voting by participants; and second, either a commit action, or an abort action, based on the voting result.

Zookeeper Administration

The official documentation includes all we need to know about administration. In addition, we need to configure log4j for proper logging. As best practices, we also should turn off swapping on ZooKeeper. We should clean up the data directory periodically if auto purge is not enabled. For optimal performance, ZooKeeper transaction log should be configured in a dedicated device.

For monitoring, ZooKeeper responds to a small sets of four-letter commands issued through telnet or nc to server’s client port. This allows the admin to check health of server or diagnose any problems. This requires the following property in zoo keeper config:

4lw.commands.whitelist=stat, ruok, conf, isro, wchc

The value can be set to asterick to allow all four-letter keyword. Once enabled, we can check server status

$ echo ruok | nc localhost 2181
imok

More four-letter commands are listed here. Apart from the four-letter commands, ZooKeeper can also be managed through Java Management Extensions (JMX).

Conclusion

Apache ZooKeeper is a coordination service for distributed application. It has become the solution for high availability for many other projects. Some of Apache’s well known open-source distributed services include:

Apache Hadoop (an umbrella of projects including many components for BigData processing such as Hadoop Common, Hadoop Distributed File System (HDFS), Hadoop YARN (yet another resource negotiator) and Hadoop MapReduce)
Apache HBase: non-relational database on top of HDFS
Apache Hive: data warehouse with SQL-like interface
Apache Kafka: stream processing
Apache Nifi: automated data flow processing.

Some of them, such as Nifi, has an embedded implementation of ZooKeeper ensemble if there isn’t a separate ensemble. There is some limitation with embedded Zookeeper ensemble. First, we cannot start ZooKeeper without starting Nifi service on the same server. Second, we need to orchestrate the configuration so that the ZooKeeper ensemble does not grow too large. We need to keep in mind that the ZooKeeper ensemble is a separate cluster of its own, and the it is not recommended to have more than 7 nodes on ZooKeeper.

Kafka high-level Overview

Tue, 21 Jul 2020 23:19:00 -0400

Zookeeper

General definition of distributed system: a software system that is composed of independent computing entities linked together by a computer network whose components communicate and coordinate with each other to achieve a common computational goal. Implementing coordination among components of a distributed system is hard. For example, designated master node becomes single point of failure; cluster needs to detect availability of new nodes as it joins cluster.

Zookeeper is designed to simplify cluster coordination. Zookeeper implements key aspects in cluster coordination, such as distributed consensus, group management, presence protocols and leader election. In order to coordinate a cluster, zookeeper itself also runs in its own cluster, called ensemble. Zookeeper exposes a simple but powerful interface of primitives. Applications can be designed on these primitives implemented through ZooKeeper APIs to solve the problems of distributed synchronization, cluster configuration management, group membership, etc.

Zookeeper Ensemble

Clients can connect to a Zookeeper service by connecting to any member of the ensemble. The members of the ensemble are aware of each other’s state. As long as a majority of the nodes are available, the service will be available. Zookeeper cli (zkCli.sh) can be used to connect to Zookeeper server. they can be downloaded from here.

Zookeeper is integrated with many other services apart from Kafka, such as Nifi and Hadoop.

Kafka

Kafka is a messaging system that is horizontally scalable, fault tolerant. It can also serve as queue storage system and stream processing system. It is distributed and use Zookeeper for cluster coordination. Each node is called a broker.

Topics in Kafka (think of table in database) is a category or feed name to which messages (records) are published. Topic is broken up into ordered commit logs called partitions. Each partition has an ID. Each message in a partition is assigned an offset. Topics that are created in Kafka are distributed across brokers based on the partition, replication, and other factors. Each partition is replicated across several brokers depending on replication factor. For each partition, Kafka elect one replica as the leader of partition.

Writes to a partition is generally sequential. Reading messages can either be from the beginning, or rewind or skip to any port in partition given an offset value. Data in a topic is retained for a configurable period of time.

A message is a unit of data in Kafka, in the format of key-value pair. A key is used to control the message that is to be written to partitions. Messages with the same keys are always written to the same partition (hash map)

A producer publishes new message to a topic. Producers do not care which partition the message is written to and will balance messages over every partition of a topic evenly. Directing messages to a partition is done using the message key and a partitioner, this will generate a hash of the key and map it to a partition.

A consumer is subscribed to one or more topics and read messages sequentially. The consumer keeps track of messages it has consumed by keeping track on the offset of the message. The offset is a bit of metadata (an integer value that continually increases) that kafka adds to each message. Each partition has a unique offset which is stored with the offset of the last consumed message. A consumer can stop and start without losing its current state.

A Kafka broker is designed to operate as part of a cluster. One broker in the cluster also function as the cluster’s controller, which is responsible for administrative operations such as: assigning partitions to brokers; monitoring for broker failures in cluster. A particular partition is owned by a broker and that broker is called the leader of the partition.

All consumers and producers operating on that partition must connect to the leader.

Kafka cluster may replicate across cluster using MirrorMaker.

Reference: Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale