helm on Digi Hunch

MinIO for S3-compatible Object Storage

Fri, 09 Sep 2022 09:00:00 -0400

I reviewed some storage technologies on Kubernetes but they are all for block and file storage. In this post, I will discuss the current available options for container workload to use object storage. I will also touch on MinIO as an object storage solution.

Object storage

Block and file system are more native to operating system because they present themselves to the OS as a block device or file system attached to the OS. In other words, application processes running on the OS will be able to access the storage by address expressed as a POSIX-compatible path. On the contrary, object storage is a REST API service, operating at the application layer in the TCP/IP stack. Therefore, we can think of object storage as “storage as a web service”.

Object storage can be made very cheap. However, the application protocol may vary depending on the object storage provider. Amazon S3 is a forerunner in object storage market and its protocol has emerged as the de-facto standard for object storage. When building an application and if there is one object storage protocol to support, it should be S3. For non-S3 object storage services, we can front them with an S3 interface, if the provider itself does not have one. For example Ceph storage has its Gateway S3 API.

Container Object Storage Interface

If we use S3 as the universal object storage protocol, does that also address object storage access with container workload on Kubernetes? Absolutely. Nonetheless, for a number of reasons using REST API from containers are not the best option. From platform’s perspective, it is the platform that should define how to access object storage, instead of leaving it with an application-layer protocol.

When a pattern (for storage, or networking, etc) turns out very common, the platform layer should incorporate it as an infrastructure service, manage it with its own standard, and provide it to application so that developer can focus on business features. With that vision, the community brought up the Container Object Storage Interface (COSI) initiative. It is currently in very early stage, but the idea is to commoditize object storage in Kubernetes platform with a unified interface. For more background about this initiative, refer to the post “Beyond block and file – COSI enables object storage in Kubernetes“.

COSI is the ultimate cloud native solution but it is still in pre-alpha phase as of mid 2022. Unfortunately, it is not a recommended solution to any real-life project in 2022, and we are stuck with the unified API approach until COSI matures.. The unified API approach is by no means cloud native, but has come to maturity for adoption. S3 Rest API is our friend, regardless of whether the client process is in a container or not.

Update: on Sept 2, 2022, Kubernetes introduced COSI as alpha feature.

MinIO Introduction

In order to use S3 protocol without using Amazon S3 storage, we can use MinIO to build our own object storage service serve client via a S3-compatible REST API interface. The main developer of the MinIO project is MinIO Inc, a startup from 2014. Having learned the lessons from GlusterFS, the founders and developers make MinIO very simple. MinIO operates in two modes: gateway mode (soon to be legacy) and server mode.

In the Gateway mode, MinIO as a gateway between client and destination storage, and does not persist data to itself. In the past, the destination storage can be Azure Blob and Google Cloud Storage (GCS) and HDFS as backend. However, these supports are deprecated now. The current release (July 2022) only supports S3 and NAS backend. According to MinIO’s blog post from February 2022, the entire MinIO Gateway feature will be removed in August, leaving server mode the only option for MinIO.

In the Server mode, the MinIO service will persist data to itself in a file system (or volume). You can specify that file system (or volume) as you launch the server. As one of the quick-start guides shows, we can host MinIO server using a single executable. For administrative tasks, MinIO has a web console and a client utility called mc.

MinIO Deployment Options

For storage service, there are a number of deployment options:

SNSD (single-node, single-drive): single MinIO server with a single storage volume or folder.
SNMD (signle-node, multi-drive): single MinIO server with four or more storage volumes.
MNMD (multi-node, multi-drive, aka distributed): multiple MinIO servers with at least four drives across all servers. This should be considered for production grade configuration.

The deployment options above describes the node and volume topology. No matter which topology option, there are also a number of ways to host the MinIO service process: on Linux OS, Windows OS, MacOS, Docker Container, and on Kubernetes platform.

In addition, MinIO Inc ships the software under different business models. For example, there are fully managed applications in Azure Marketplace, AWS Marketplace, and GCP Marketplace all hosted on virtual machines with extra charges. Clients not willing to pay can host MinIO storage all on their own, either on virtual machines, or on managed Kubernetes environment provided by each cloud provider.

MinIO Hosting solutions

MinIO lists these hosting solutions under multi-cloud products. These hosting solutions (or “products” in MinIO’s term) vary in terms of where peripheral services and data tiers are hosted. Here is the list of the supported platforms:

To illustrate how these solutions are different, I put some details on a few options together for an incomplete comparison below:

	Kubernetes	EKS	AKS	GKE
Hot Storage	Direct PV (NVMe)	EKS EBS CSI	Azure CSI	GKE Standard SSD
Warm Storage	Direct PV (HDD)	S3 IA	Azure BlobStore	GCS
Cold Storage	Public Cloud storage	Glacier	Azure Cool Blob	GCS for Data Archiving
Encryption	HashiCorp Vault	KMS	Azure Key Vault	Cloud Key Management
Observability	Elastic Stack and Grafana	Managed ElasticSearch Prometheus	Azure Monitor	Stack Driver
Identity Provider	KeyCloak	LDAP, SSO	Azure Active Directory	GCP Cloud Identity
LB and Cert Mgmt	Nginx, Let’s Entrypt	AWS Cert Mgr, ELB	Azure Load Balancer, JetStack, Let’s Encrypt	GCP Cloud LB and Managed Cert

Note that all of these hosting solutions are based on some flavour of Kubernetes. The hot tier is usually based on storage options available to the platform. MinIO service access this hot tier via Kubernetes persistent volume. The warm and cold tiers are backed by different object storage service. Between MinIO and storage client, it always use the same S3 compatible Rest API.

MinIO also has tiering capability. While the hot storage destination has to be either a file system or Kubernetes persistent volume, remote tiers can be S3 , Azure Blob, or GCS. MinIO supports encryption at rest (SSE-KMS, SSE-S3, SSE-C) and in transit (TLS) for security, as well as many other useful features such as object replication, versioning, locking, events, Prometheus metrics, lifecycle management etc.

Connect to MinIO server with S3 client

To validate that the client is compatible, we use MinIO’s client utility (mc) to connect to an AWS S3 bucket. Then we use AWS CLI to connect to a MinIO server, similar to this instruction. To do so, we first install client and server utilities:

brew install minio/stable/minio
brew install minio/stable/mc
minio --version
mc --version

Then, we start MinIO server and store an object using AWS CLI’s S3 tool. In our working directory, we create a new directory called minio_data and launch MinIO server with it:

mkdir minio_data
minio server minio_data --console-address :9090

Once the server is up, the screen should display the details, including the portal URL and the default username and password will be used as Access Key ID and Secret Key:

Note that the MinIO service does NOT have TLS enabled by default, on the console or API service. At this point, we can browse to the console web page using the given credential. Then, we can configure AWS CLI with a new profile just to act as a client to communicate with the MinIO server:

$ aws configure --profile minio-cli
AWS Access Key ID [None]: minioadmin
AWS Secret Access Key [None]: minioadmin
Default region name [None]: us-east-1
Default output format [None]: json
$ aws configure set default.s3.signature_version s3v4 --profile minio-cli

At this point, the AWS CLI is configured to communicate with MinIO server. Then, we can create bucket, list object in the bucket, copy an object to the bucket, etc

$ aws --endpoint-url http://127.0.0.1:9000 s3 ls --profile minio-cli # list all bucket, should return empty
$ aws --endpoint-url http://127.0.0.1:9000 s3 mb s3://hehebucket --profile minio-cli # create new bucket
make_bucket: hehebucket
$ aws --endpoint-url http://127.0.0.1:9000 s3 cp README.md s3://hehebucket --profile minio-cli # copy a file to bucket as a new object
upload: ./README.md to s3://hehebucket/README.md
$ aws --endpoint-url http://127.0.0.1:9000 s3 ls s3://hehebucket --profile minio-cli # list objects in the bucket
2022-07-09 00:30:23 631 README.md

The created bucket and object are also visible in MinIO web console, under “Bucket”:

The steps above validate that AWS CLI can talk to MinIO server. Because of that, MinIO server can emulate an S3 service in any development environment so users do not always have to use S3 from AWS. This makes sense for both cost and security reasons for the organization.

Connect to S3 with MinIO client

In this lab, we create an S3 bucket and use mc utility to store an object to it. In order to consistently create S3 bucket and associated permissions, I use the CloudFormation template in this repo. The output of the CloudFormation stack returns the Access Key ID and Secret Key required for the client to access the bucket. Once we cloned the repo, let’s enter the obj-store-helper directory, and run aws cli command to launch the CloudFormation template, assuming it has been configured:

BUCKET_NAME=c0sas2dsadigihunch
S3_STACK_NAME=$BUCKET_NAME-stack
aws cloudformation create-stack --template-body file://aws-s3-stack.yaml --stack-name $S3_STACK_NAME --parameters ParameterKey=S3BucketName,ParameterValue=$BUCKET_NAME --capabilities CAPABILITY_IAM

# to delete stack after test, run: aws cloudformation delete-stack --stack-name $S3_STACK_NAME

In the AWS console, we should see the configuration information as below:

Supposed the bucket name is vna-tst-c0sas2dsadigihunch as shown above, this allows us to configure the client utility MC as below:

mc alias set awss3 https://s3.amazonaws.com # Fill in access key ID and Secret key at the prompt
mc ls awss3/vna-tst-c0sas2dsadigihunch # list objects in the bucket, should return empty
mc cp README.md awss3/vna-tst-c0sas2dsadigihunch/README.md # upload and object to bucket
mc ls awss3/vna-tst-c0sas2dsadigihunch # list objects in the bucket, the uploaded object should be there
mc rm awss3/vna-tst-c0sas2dsadigihunch/README.md # delete the object
mc alias remove awss3 # remove awss3 alias

Once we emptied the bucket, we can delete the CloudFormation stack. This test only needs client utility mc to verify that MinIO client is able to talk to AWS S3 server.

Erasure Coding

For scalable production use, we should deploy MinIO in distributed mode. When MinIO is configured in distributed deployment (MNMD, or multi-node, multi-drive), it implicitly enables an important feature called erasure coding. This erasure coding feature further unlocks a number of other MinIO features:

Erasure coding is MinIO’s data redundancy and availability feature that allows MinIO deployments to automatically reconstruct objects on-the-fly despite the loss of multiple drives or nodes in the cluster. Erasure coding provides object-level handling with less overhead than adjacent technologies such as RAID. The key concept is Erasure Set, a set of drives in a MinIO deployment that supports Erasure Coding. MinIO evenly distributes object data and parity blocks among the drives in the Erasure Set.

Two important variables are M and N: for a given erasure set of size M, MinIO splits objects into N parity blocks, and M-N data blocks. MinIO uses the EC:N notation to refer to the number of parity blocks (N) in the deployment. To determine optimal erasure set size for the cluster, use MinIO’s Erasure Coding Calculator tool.

To help client to specify per-object parity with Erasure Coding, MinIO uses storage classes. Note that the storage class concept in MinIO is distinct from AWS S3 storage class or Kubernetes storage class. In MinIO, a storage class defines parity settings per object. The STANDARD storage class (default) defines EC:N based on M, which can be overridden. In addition, there is REDUCED_REDUNDANCY storage class, whose parity must be less than or equal to that of STANDARD storage class.

The erasure coded backend also protects the storage against Bit Rot with HighwayHash algorithm.

More Features

Authentication and authorization between MinIO client and MinIO server have a number of options. MinIO client may use the built-in standalone identity management in MinIO server. This is the default mode. In addition, one may delegate IAM to external service. To Active Directory via LDAP, or any Identity provider that supports OIDC (JWT with Authorization Code Flow).

As to Object Lifecycle Management (OLM), MinIO allows you to define a remote tier storage for each local target (bucket). The remote tier can be Amazon S3, Google Cloud Storage or Azure Blob storage. We can use mc utility to administer the remote tier and OLM. Configuration steps (e.g. Azure Blob, AWS S3) usually include:

Configure required permissions on the MinIO bucket, create user account for OLM activities.
Configure the Remote Storage Tier
Create and Apply an ILM Transition Rule. The rule can be expressed in a json document.
Validate the creation of ILM transition rule
Validate the effect of transition rule.

As for encryption, MinIO can support encryption at rest. It can also work with etcd store to store encrypted IAM assets if KMS is configured.

Conclusion

Even though we watch for the progress of COSI initiative, we still use Rest API to access object storage from container, which is no different than from a virtual machine. If we develop an application, then we should make it support S3 protocol, a de-facto standard protocol for object storage. As for the storage backend, if we want to be vendor neutral, the feature-rich MinIO is the best bet. We can use MinIO to build our own Object storage as a service compatible with S3. We can also lifecycle our object to remote object storage tier backed by Azure, GCP or S3. In this post we validated the S3 compatibility, and discussed some advanced MinIO features.

Kubernetes Storage on Azure 3 of 3 – Ceph by Rook

Fri, 26 Aug 2022 19:43:00 -0400

In the last two posts, I covered the native storage options on Azure Kubernetes Service, as well as Portworx as an example of a proprietary Software Defined Storage (SDS) solution. There are also a number of open-source alternative SDS solutions. Ceph has nearly a decade of history from prior to containerization, and is the most widely adopted storage platform. In this post, we continue to explore Ceph as an open-source storage solution on Azure Kubernetes.

Ceph by Rook

Ceph is an open-source SDS platform for distributed storage on a cluster and provides object, block and file storage. Installation of Ceph SDS can be complex, especially on Kubernetes platform. Rook is a graduated CNCF project to orchestrate storage platform. Rook by itself is not SDS and it supports:

Ceph: configure a Ceph cluster. Think of this as the equivalent of cephadm on Kubernetes platform.
NFS: configure an NFS server. Think of this as the equivalent of nfsd daemon on Kubernetes platform.
Cassandra: an operator to configure a Cassandra database cluster. It is now deprecated.

We play with Rook Ceph. I also refer to it as Ceph by Rook. The contribution of Rook project is it simplifies the installation as a matter of declaring custom resources using CRDs. Here are some high-level CRDs to know:

CephCluster: creates a Ceph storage cluster
CephBlockPool: represents a block pool
CephFilesystem: represents a file system
CephObjectStore: represents an object store
CephNFS: spins up a NFS Ganesha server to export NFS shares of a CephFilesystem or CephObjectStore.

As with typical Kubernetes resources in controller pattern, Ceph by Rook needs an operator along with custom resources. We can use YAML manifest for both of them, and the manifests are usually very tediously long. We can also use Helm to install both of them, by providing a value file. Now we will install Ceph on AKS.

Install Ceph Operator on AKS

The steps are influenced by two relevant posts (here and here). However, I’ve incorporated the cluster configuration in the Azure directory of the cloudkube project, a modular Terraform template to configure AKS cluster and facilitate storage configuration. The node group and instance sizes are selected to be just enough to run a ceph POC cluster with minimum cost. One of the node groups is tainted with storage-node, as if the following command were run:

kubectl taint nodes my-node-pool-node-name storage-node=true:NoSchedule

You will only need to taint the nodes with the command above if you choose not to use the cloudkube template. The taint ensures that only Pods with corresponding toleration and effect can be scheduled to those nodes.

We use Helm to install Rook Operator. We need a value file (e.g. rook-ceph-operator-values.yaml) with content as below:

# https://github.com/rook/rook/blob/master/Documentation/Helm-Charts/operator-chart.md
crds:
 enabled: true
csi:
 provisionerTolerations:
 - effect: NoSchedule
 key: storage-node
 operator: Exists
 pluginTolerations:
 - effect: NoSchedule
 key: storage-node
 operator: Exists
agent:
 # AKS: https://rook.github.io/docs/rook/v1.7/flexvolume.html#azure-aks
 flexVolumeDirPath: "/etc/kubernetes/volumeplugins"

Then we install the operator with Helm:

helm install rook-ceph-operator rook-ceph --namespace rook-ceph --create-namespace --version v1.9.6 --repo https://charts.rook.io/release/ --values rook-ceph-operator-values.yaml
kubectl -n rook-ceph get po -l app=rook-ceph-operator

After installing the operator, we check the Pod status to make sure it is running. Then we can install the actual Ceph Cluster in one of the two ways. We can declare a CephClusterCRD ourself, or we can use Helm again to declare the CRD. Helm Chart gives us a lot of useful default values and saves us from editing a large body of YAML manifest.

Install Ceph CR on AKS

We use Helm to install CephCluster CRD. We create a value file (e.g. rook-ceph-cluster-values.yaml) with content as below:

# https://github.com/rook/rook/blob/master/Documentation/Helm-Charts/ceph-cluster-chart.md
operatorNamespace: rook-ceph
toolbox:
 enabled: true
cephObjectStores: [] # by default a cephObjectStore will be created. Setting this to null disables it
#cephBlockPools: # by default a cephBlockPool will also be created with default values
#cephFileSystems: # by default a cephFileSystem will also be created with default values

cephClusterSpec:
 mon:
 count: 3
 volumeClaimTemplate:
 spec:
 storageClassName: managed-premium
 resources:
 requests:
 storage: 10Gi
 resources:
 limits:
 cpu: "500m"
 memory: "1Gi"
 requests:
 cpu: "100m"
 memory: "500Mi"
 dashboard:
 enabled: true
 storage:
 storageClassDeviceSets:
 - name: set1
 # The number of OSDs to create from this device set
 count: 3
 # IMPORTANT: If volumes specified by the storageClassName are not portable across nodes
 # this needs to be set to false. For example, if using the local storage provisioner
 # this should be false.
 portable: false
 # Since the OSDs could end up on any node, an effort needs to be made to spread the OSDs
 # across nodes as much as possible. Unfortunately the pod anti-affinity breaks down
 # as soon as you have more than one OSD per node. The topology spread constraints will
 # give us an even spread on K8s 1.18 or newer.
 placement:
 topologySpreadConstraints:
 - maxSkew: 1
 topologyKey: kubernetes.io/hostname
 whenUnsatisfiable: ScheduleAnyway
 labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - rook-ceph-osd
 tolerations:
 - key: storage-node
 operator: Exists
 preparePlacement:
 tolerations:
 - key: storage-node
 operator: Exists
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: agentpool
 operator: In
 values:
 - storagenp
 topologySpreadConstraints:
 - maxSkew: 1
 # IMPORTANT: If you don't have zone labels, change this to another key such as kubernetes.io/hostname
 topologyKey: topology.kubernetes.io/zone
 whenUnsatisfiable: DoNotSchedule
 labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - rook-ceph-osd-prepare
 resources:
 limits:
 cpu: "500m"
 memory: "4Gi"
 requests:
 cpu: "500m"
 memory: "2Gi"
 volumeClaimTemplates:
 - metadata:
 name: data
 spec:
 resources:
 requests:
 storage: 100Gi
 storageClassName: managed-premium
 volumeMode: Block
 accessModes:
 - ReadWriteOnce

During the cluster provisioning, there will be a number of preparing Pods. We want those Pods to run on nodes with label agentpool=storagenp. In real life, we need to orchestrate where to run each workload, by restricting the nodes to schedule certain types of workload.

Then we can install the cluster using Helm:

helm install rook-ceph-cluster rook-ceph-cluster --namespace rook-ceph --create-namespace --version v1.9.6 --repo https://charts.rook.io/release/ --values rook-ceph-cluster-values.yaml

After running the Helm install, it may take as long as 15 minutes for all resources to settle. Watch the Pod status in rook-ceph namespace. At the end, make sure that the cluster is created successfully:

kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get CephCluster
NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL
rook-ceph /var/lib/rook 3 15m Ready Cluster created successfully HEALTH_OK
kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get cephBlockPools
NAME PHASE
ceph-blockpool Ready
kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get cephFileSystems
NAME ACTIVEMDS AGE PHASE
ceph-filesystem 1 20m Ready

In my case it took 15 minutes before the cluster comes up as created successfully. You should notice that two storage classes were also created as a part of the install. It however did not create a storage class or CRD for object storage, because we explicitly disabled it in the Helm value file by setting cephObjectStores value to null.

Dashboard

We enabled dashboard. To configure the dashboard view properly, we would need an ingress. For a quick view here, we can play port forwarding tricks. First we fetch the admin password for use in the next step. Then expose the dashboard to the bastion host:

$ kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath='{.data.password}' | base64 -d
$ kubectl -n rook-ceph port-forward svc/rook-ceph-mgr-dashboard 8443:8443

Since I don’t have UI on the bastion host, I use the port forwarding trick again from my own MacBook. Start a new terminal and SSH to the bastion host with port-forwarding switch:

$ ssh -L 8443:localhost:8443 kubeadmin@20.116.132.8

The command above suppose the public IP of the bastion host is 20.116.132.8. Then from my MacBook I can browse to localhost:8443 (with Safari browser which gives me the option to bypass certificate error). At the web portal, provide username (admin) and password (as retrieved above):

Ceph console for Kubernetes

Apart from the dashboard, we can also use ceph admin tool from a toolbox pod, following this instruction. For monitoring, Ceph by Rook can expose metrics for Prometheus to scrape.

Performance

With default ceph configuration on AKS, I ran quick performance test using kube-str . The result is as follows:

	read_iops	write_iops	read_bw	write_bw
ceph-block	IOPS=464.507294 BW(KiB/s)=1874	IOPS=243.296143 BW(KiB/s)=989	IOPS=509.928162 BW(KiB/s)=65797	IOPS=248.530762 BW(KiB/s)=32338
ceph-filesystem	IOPS=438.701324 BW(KiB/s)=1770	IOPS=226.270660 BW(KiB/s)=920	IOPS=405.936340 BW(KiB/s)=52456	IOPS=208.869293 BW(KiB/s)=27229

The metrics reflects performance under default configuration. It should not be considered as the best performance that Ceph can deliver on Azure Kubernetes.

Summary

I discussed three storage options for Azure Kubernetes but the idea applies to other Kubernetes platform hosted on a CSP. The native storage has significant limitation. NFS has latency. Block storage does not address high availability at the storage layer. Portworx and LINSTOR fill that gap as a commercial solution. Ceph is based on Object storage.

Kubernetes Storage on Azure 2 of 3 – Portworx

Fri, 12 Aug 2022 15:23:00 -0400

In the previous post, we have discussed built-in storage classes on Azure Kubernetes. Further to that, we will examine some third-party software defined storage (SDS) options that are compatible with Azure Kubernetes Service in this post. Then we take Portworx on Azure as an example. Although, these options are specific to Azure, most of the players also have solutions for other managed Kubernetes platforms. Also, the methodology to study storage options remain the same regardless of cloud service provider.

In fact, I touched on software defined storage (SDS) in the context of general non-containerized workload in a separate post. In short, storage providers decouple the SDS appliance from the full storage solution in order to lower the cost and increase flexibility. To the storage consumer (e.g. a process running on Linux OS), SDS can present a block disk or file system. There are also SDS solution that can host your own object storage and we will discuss that later. In fact, SDS has gained significant popularity in recent years. For example, the report “Validating Software-Defined Storage Operating Models for the Enterprise” by archiectingit divided the evolution into four phases and cited that Gartner predicts the SDS revolution to reach 50% of the storage market by 2024, from 15% in 2020.

This post discusses SDS in the context of container storage. Then we will install Portworx on Azure Kubernetes.

SDS for Kubernetes

Many SDS appliances also developed the capability to present storage volumes to containerized workload. I put together a list of SDS products that works on Azure with their supported access modes:

Solution	Licence	Development and Support	Access Mode
Portworx	Closed source. Free Essential tier. Enterprise features on License.	Commercially supported by PureStorage	RWO, RWX
Ceph by Rook	Open source. Rook is a graduated CNCF project.	Developed and commercially supported by Red Hat, Canonical and SoftIron	RWO, ROX, RWX
OpenEBS	Open source. Sandbox CNCF project.	Developed and commercially supported by MayaData et al.	RWO
Longhorn	Open source. Incubating CNCF project.	Originally developed by Rancher, and commercially supported by SUSE	RWO
StorageOS	Closed source. License required.	Commercial support by Ondat.	RWO
LINBIT	Open-source with enterprise plans	Enterprise support

RWO is the most commonly supported mode. The report “Performance Benchmarking Cloud Native Storage Solutions for Kubernetes” makes a comparison of performance among some of the options in early 2021. Another potentially opinionated comparison list is by LINBIT.

Portworx is a leading player with commercial SDS solution and I will test its free Essential tier in the rest of this post. Ceph is one of the most mature leading open-source offering and I will test it in the next post.

For Portworx, we can use the terraform template cloudkube for Azure. The template assigns the kubelet’s managed identity as contributor of the node resource group. The template also creates a bastion host with direct SSH access to the nodes.

Portworx Operator on Azure

We use Portworx Operator to configure storage cluster. Portworx has an instruction for AKS but it is not tailored to specific identity model. For simplicity, use my cloudkube Terraform template to create the AKS cluster, and skip the “Prepare Your AKS Platform” page. Instead, follow the “Deploy Portworx using Azure managed identity on new AKS cluster” page starting at step 7.

At that step, we need to create a secret with the client ID of the managed identity for node agent. The terraform template outputs the BYO identity’s client ID. After cluster creation, we simply SSH to the bastion host and create the secret using the output.

To install Porworx using operator, we can follow a wizard in PX-central. If this is the first time, we need to create an account and log in to the portal. If this is not the first time and you have previously created a cluster, you need to detach that cluster by going to Profile from bottom left corner on the portal page.

Follow the guide in “Install Portworx on AKS using the Operator“. In the wizard, click on “Portworx Essentials” for free tier, or “Portworx Enterprise” for the 30-day trial. Then select operator with latest version. In the rest of the wizard steps, select options applicable to Azure environment. The last step will present two kubectl commands to install operator and install the CR. Run the command to install operator and verify result:

kubectl apply -f 'https://install.portworx.com/2.9?comp=pxoperator'
kubectl -n kube-system get deployment portworx-operator

Portworx Custom Resource

To install the CR, we need to customize the given manifest in order to use our managed identity. We can download the YAML manifest (portworx_essentials.yml) and modify it in text editor. As the page “Deploy Portworx using Azure managed identity on new AKS cluster” suggest at step 9: in the env section, remove the AZURE_CLIENT_SECRET and AZURE_TENANT_ID sections but keep the AZURE_CLIENT_ID section. My CRD declaration looks like this:

# SOURCE: https://install.portworx.com/?operator=true&mc=false&kbver=&oem=esse&user=myuserid&b=true&kd=type%3DPremium_LRS%2Csize%3D150&s=%22type%3DPremium_LRS%2Csize%3D150%22&c=my-very-long-px-cluster-id&aks=true&stork=true&csi=true&mon=true&tel=false&st=k8s&promop=true
kind: StorageCluster
apiVersion: core.libopenstorage.org/v1
metadata:
 name: my-very-long-px-cluster-id
 namespace: kube-system
 annotations:
 portworx.io/install-source: "https://install.portworx.com/?operator=true&mc=false&kbver=&oem=esse&user=myuserid&b=true&kd=type%3DPremium_LRS%2Csize%3D150&s=%22type%3DPremium_LRS%2Csize%3D150%22&c=my-very-long-px-cluster-id&aks=true&stork=true&csi=true&mon=true&tel=false&st=k8s&promop=true"
 portworx.io/is-aks: "true"
 portworx.io/misc-args: "--oem esse"
spec:
 image: portworx/oci-monitor:2.10.2
 imagePullPolicy: Always
 kvdb:
 internal: true
 cloudStorage:
 deviceSpecs:
 - type=Premium_LRS,size=150
 kvdbDeviceSpec: type=Premium_LRS,size=150
 secretsProvider: k8s
 stork:
 enabled: true
 args:
 webhook-controller: "true"
 autopilot:
 enabled: true
 monitoring:
 prometheus:
 enabled: true
 exportMetrics: true
 featureGates:
 CSI: "true"
 env:
 - name: AZURE_CLIENT_ID
 valueFrom:
 secretKeyRef:
 name: px-azure
 key: AZURE_CLIENT_ID
---
apiVersion: v1
kind: Secret
metadata:
 name: px-essential
 namespace: kube-system
data:
 px-essen-user-id: aaaabbbbccccddddmyverylongpxessenuserid
 px-osb-endpoint: ssssssyyyyyyyzzzzzzmyverylongpxosbendpoint

In this custom resource manifest, I specify to get the cluster to create disk from Azure, even though it has the capability of using an existing disk. Then apply the CRD manifest:

 kubectl apply -f portworx_essentials.yml

This will take up to 10 minutes to create several related resources. There are several check points to ensure the cluster is created successfully. Here are some useful commands:

kubectl -n kube-system get po # all Pods related to portworx should be ready and running, especially the portworx-api ones
kubectl -n kube-system get storagecluster # the status should report online
PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}') 
# get the name of one of the portworx Pod for storage cluster
kubectl -n kube-system exec -c portworx -it $PX_POD --tty --stdin -- /opt/pwx/bin/pxctl status 
# should report "Status: PX is operational" with valid license loaded
kubectl get sc # portworx related storage classes are available

The pre-built storage classes are not CSI based. However, CSI should be automatically enabled in recent operator versions. We should create our own CSI storage classes and PVCs using our own storage classes. Here are two examples:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: px-csi-database
provisioner: pxd.portworx.com
parameters:
 repl: "2"
 priority_io: "high"
 io_profile: "db"
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: px-csi-artifact
provisioner: pxd.portworx.com
parameters:
 repl: "2"
 priority_io: "medium"
 io_profile: "sequential"

That is a bare minimum Portworx installation. Check out their documentation for the full features. With this minimum install we can go to the section “Validate Persistent Storage” from the previous post to validate the persistent volume.

For troubleshooting purpose, pxctl is the utility and it is available on Portworx Pods.

Performance

We care not only the functionality, but also the performance. So I ran a quick performance test using kube-str, using all default configuration. The result is as follows:

	read_iops	write_iops	read_bw	write_bw
px-csi-database	IOPS=969.614136 BW(KiB/s)=3894	IOPS=729.698059 BW(KiB/s)=2935	IOPS=1172.772827 BW(KiB/s)=150639	IOPS=691.626526 BW(KiB/s)=89053
px-csi-artifact	IOPS=780.681946 BW(KiB/s)=3139	IOPS=682.522766 BW(KiB/s)=2746	IOPS=773.548584 BW(KiB/s)=99549	IOPS=659.015320 BW(KiB/s)=84890

Note that those numbers reflect performance under default configuration, and they should not be considered as the best performance that Portworx can deliver on Azure Kubernetes. Before moving to production, it is important to establish your own test parameters that best represents the container workload, and then iterate through different parameters for the storage class based on the requirement and performance output.

Kubernetes Storage on Azure 1 of 3 – built-in storage and NFS

Sun, 31 Jul 2022 15:22:00 -0400

In the previous post, we understand that to host stateful workload, we need to manage persistent storage to the Kubernetes platform. In this post, I will explore the different storage options. These options are specific to Azure Kubernetes service. However, the principals apply to any Kubernetes platform regardless of cloud vendor.

In another old post, I discussed in-tree and CSI storage classes, and from a developer’s perspective, how to mount volumes statically and dynamically once the storage class is available. Here in this post we are concerned with how to make storage classes available, from a platform specialist perspective.

Storage in Azure Kubernetes

As soon as the Azure Kubernetes cluster is launched, a number of built-in storage classes are available. Unlike third-party storage classes, they do not require kubelet identity to be contributor for node resource group. As discussed we shall use CSI based storage classes.

managed-csi
managed-csi-premium
azurefile-csi
azurefile-csi-premium

The main difference between them is the backing technology. However, form Kubernetes workload’s perspective, the Pods as storage consumers are concerned with the access mode instead of backing technology. Here are supported access modes:

ReadWriteOnce: read-write by a single node
ReadOnlyMany: read only by many nodes
ReadWriteMany: read-write by many nodes
ReadWriteOncePod: new in Kubernetes 1.22 to restrict volume access to a single Pod

The storage classes managed-csi and managed-csi-premium support ReadWriteOnce. The storage classes azurefile-csi and azurefile-csi-premium support ReadWriteMany.

Apart from these built-in options, Azure also suggests a few more options based on other types of Azure resources. For example, AKS can integrate with HPC cache and it for HPC. We can also self-manage a virtual machine configured as NFS server, and use the NFS subdir external provisioner to configure storage class. Despite of the overhead with managing a VM, you have more configurability. My previous client reports that they gain better performance than the built-in options.

Another alternative is Azure NetApp Files. However, being a full enterprise grade solution (similar to FSx ONTAP), Azure NetApp Files costs an arm and a leg. In this comparison, it cost 60 to 100 times as the cost by built-in options.

Another option is Azure Ultra Disk, which needs to be enabled at cluster level. You can provision performance target (DiskIOPSReadWrite and DiskMBpsReadWrite) in the storage class. Ultra Disk is a good middle ground between the pricey NetApp files and the less performant built-in options.

Terraform Template for AKS

To explore the storage options, I use my own terraform template to create an AKS cluster. The template is in the azure directory of the cloudkube repo. The template consists a few configurations with Azure Kubernetes to facilitate storage configuration. First, it configures an SSH key pair to use between the bastion host and the node. Users can SSH to Kubernetes nodes from bastion host as soon as terraform apply is completed. Second, the third party storage options installed after the cluster creation need their Pod to instruct Azure to create Azure disks. This requires that a Kubernetes node agent have the permission to provision resources in the node resource group. This is important to understand because there are a couple of managed identities at play (refer to this post) when building an AKS cluster.

In Azure Kubernetes, it is the managed identity of kubelet, that needs to have contributor permission over the resource group for the nodes (not the one for the AKS cluster itself). A managed identity is expressed by a client ID, an object ID (aka principal ID), and the identity ID. We can find them out with an AZ CLI command as below:

We can also tell that the kubelet managed identity represents node agent, by connecting to a node and looking at the argument (kubernetes.azure.com/kubelet-identity-client-id) of kubelet process:

In the template, I also chose to designate the same BYO identity for both the cluster and for kubelet (node agent), in order to minimize my requirement on permission. If I had left it with a system assigned identity for node agent, I would have to assign that identity as a contributor for the node resource group, either as a user, or via Terraform’s identity. Either way, it is beyond what a Contributor is allowed to do.

Benchmarking with kubestr

I used fio utility for storage benchmarking from virtual machines. However, fio utility is not for container. For fio testing on Kubernetes, I’d have to use a Docker image, and test with target volume attached. Fortunately, the Kasten team shared their initiative in the open source project Kubestr. The kubestr release is available as an executable on common platforms. It connects to the cluster the same way as kubectl and here is a demo. To begin with, download the utility to bastion host, and run it without any argument, which prints the storage classes and volume snapshot classes:

curl -L -o kubestr.tar.gz https://github.com/kastenhq/kubestr/releases/download/v0.4.31/kubestr_0.4.31_Linux_amd64.tar.gz
tar -xvf kubestr.tar.gz && rm kubestr.tar.gz && chmod +x kubestr

./kubestr # if kubectl is configured, this command will print out the details of storage classes and volume snapshot classes

In addition to outputting details, it is also very simple to perform storage benchmarking with kubestr. All we need to do is giving it the storage class name and it will run four tests by default with common global options (ioengine=libaio verify=0 direct=1 gtod_reduce=1). The four tests are:

JobName	block_size	filesize	iodepth	rw
read_iops	4k	2G	64	randread
write_iops	4k	2G	64	randwrite
read_bw	128k	2G	64	randread
write_bw	128k	2G	64	randwrite

During each test, it measures and reports IOPS and bandwidth (throughput). If your I/O profile falls out of the four jobs, you can even customize your test by supplying a fio config file. For example, you need a longer test duration, or you need a larger total size for the test. Before the test, kubestr automatically mount their test volumes using the storage class being tested.

./kubestr fio -s my-storage-class # benchmarking a storage class

For read_iops and write_iops, we mainly look at the IOPS. For read_bw and write_bw, we mainly look at the bandwidth. The iops and bw based on samples are reported as first line of result, followed by min, max and average.

Metrics

With kubestr I ran a performance test amongst the native storage classes with Azure Kubernetes Service, with results as below:

	read_iops	write_iops	read_bw	write_bw
managed-csi	IOPS=314.729797 BW(KiB/s)=1275	IOPS=297.071136 BW(KiB/s)=1204	IOPS=315.311188 BW(KiB/s)=40887	IOPS=261.048645 BW(KiB/s)=33941
managed-csi-premium	IOPS=493.395844 BW(KiB/s)=1990	IOPS=426.731812 BW(KiB/s)=1723	IOPS=455.950348 BW(KiB/s)=58894	IOPS=422.888855 BW(KiB/s)=54662
azurefile-csi	IOPS=259.333282 BW(KiB/s)=1053	IOPS=283.985779 BW(KiB/s)=1152	IOPS=240.447403 BW(KiB/s)=31298	IOPS=230.689804 BW(KiB/s)=30048
azurefile-csi-premium	IOPS=394.044739 BW(KiB/s)=1586	IOPS=371.181793 BW(KiB/s)=1494	IOPS=380.360535 BW(KiB/s)=49018	IOPS=491.313446 BW(KiB/s)=63310

These numbers are based on default test parameters (e.g. 100Gi PVC size). The numbers indicate that block storage generally performs better in default setting. So we should use managed disk instead of azure file unless multiple pods needs to access the same volume.

Validate Persistent Storage

We can use mysql as a quick and dirty test of storage classes. We can deploy the following manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: mysql-pvc
spec:
 storageClassName: managed-csi-premium # the storage class being tested
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 5Gi
---
apiVersion: v1
kind: Secret
metadata:
 name: mysql
data:
 password: eHl6  # base64 code of xyz
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: mysql
 labels:
 app: mysql
spec:
 replicas: 1
 selector:
 matchLabels:
 app: mysql
 template:
 metadata:
 labels:
 app: mysql
 spec:
 containers:
 - image: mysql:5.6
 name: mysql
 env:
 - name: MYSQL_ROOT_PASSWORD
 valueFrom:
 secretKeyRef:
 name: mysql
 key: password
 ports:
 - containerPort: 3306
 name: mysql
 volumeMounts:
 - name: mysql-persistent-storage
 mountPath: /var/lib/mysql
 volumes:
 - name: mysql-persistent-storage
 persistentVolumeClaim:
 claimName: mysql-pvc
---
apiVersion: v1
kind: Service
metadata:
 name: mysql-service
 labels:
 service: mysql
spec:
 selector:
 app: mysql
 ports:
 - name: tcp-mysql
 protocol: TCP
 port: 3306
 targetPort: 3306

Once the Pod has been created, then we use a throw-away Pod to connect to mysql service and build some data:

$ kubectl run mysql-cli --rm -i --tty --image imega/mysql-client -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ # mysql --host=mysql-service.default.svc.cluster.local --user=root --password=xyz
Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 9 Server version: 5.6.51 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]>

From the mysql shell, we create a test database with dummy data populated:

CREATE DATABASE `testdb`; 
USE testdb; 
CREATE TABLE IF NOT EXISTS tasks ( 
 task_id INT AUTO_INCREMENT PRIMARY KEY, 
 title VARCHAR(255) NOT NULL, 
 description TEXT, 
 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP 
) ENGINE=INNODB; 
INSERT INTO tasks (title, description) VALUES ('Job A','Morning Standup');
INSERT INTO tasks (title, description) VALUES ('Job B','Latte with two shots of espresso');
INSERT INTO tasks (title, description) VALUES ('Job C','Coding coding and coding');
INSERT INTO tasks (title, description) VALUES ('Job D','git commit');

Each SQL command should return with number of rows affected and then we can exit the MySQL shell and Pod shell. Once we exit out of the Pod shell, the Pod is deleted. We can re-connect to validate the data are still present:

$ kubectl run mysql-tester --rm -i --tty --image imega/mysql-client -- mysql --host=mysql-service.default.svc.cluster.local --user=root --password=xyz --database testdb --execute='SELECT * FROM tasks;'
If you don't see a command prompt, try pressing enter.
Got error: Access denied for user 'root'@'147.206.3.15' (using password: NO)
+---------+-------+----------------------------------+---------------------+
| task_id | title | description | created_at |
+---------+-------+----------------------------------+---------------------+
| 1 | Job A | Morning Standup | 2022-06-22 20:29:12 |
| 2 | Job B | Latte with two shots of espresso | 2022-06-22 20:29:12 |
| 3 | Job C | Coding coding and coding | 2022-06-22 20:29:12 |
| 4 | Job D | git commit | 2022-06-22 20:29:13 |
+---------+-------+----------------------------------+---------------------+
Session ended, resume using 'kubectl attach mysql-tester -c mysql-tester -i -t' command when the pod is running
pod "mysql-tester" deleted

The output validates the persistent storage of the data. When testing a different storage class, simply start over with a different storage class specified in the PVC part of the manifest.

Summary

This post focuses on storage options for persistent volumes on Azure Kubernetes service. I use my own Terraform template with some custom configuration. I also covered kubestr as benchmarking tool and ran it against the built-in storage classes.

In some situations, the built-in options do not suit your needs. For example, you might want your application to use persistent volume in a consistent way across multiple cloud vendors. We will have to resort to third-party software defined storage layer. In the next post, I will explore a couple of SDS-based options, namely Portworx and Ceph. The custom configurations in the Terraform template will be helpful when we configure Portworx.

Kick the tires on ArgoCD

Sun, 10 Jul 2022 00:10:00 -0400

Background

In January, I wrote about FluxCD, and adopted it in the Korthweb project. I like the simple design of FluxCD, and I am comfortable with commands without using a web UI. Half a year later, I am re-visiting this choice, with ArgoCD in mind.

After reading numerous recent posts that compare the two (such as this one from the new stack), I started to give more thoughts on ArgoCD, for a couple reasons. First, ArgoCD has more contributing companies and public references. Red Hat adopted Argo CD as the underlying technology for OpenShift GitOps (and Tekton for pipeline). Second, ArgoCD has a mature UI. In corporate collaboration, especially with those not well-versed with command line, a UI is extremely helpful. Bundled with the UI, Argo CD also uses its own RBAC independent of Kubernetes RBAC, making it a great choice for continuous deployment for enterprise applications.

User Interface

You can choose to install just the core components, without UI, SSO and multi-cluster features. To me, it does not make sense because those features are exactly the reason to choose Argo CD. Argo CD also has an eponymous CLI tool, similar to flux for Flux CD. Since Argo CD uses its own RBAC independent of Kubernetes RBAC, Argo CD will have its own credential. The default username is admin and password needs to be retrieved from Secrets. In the tutorial we use this user’s credential to interact with the target cluster. This is different from Flux where the client simply uses kubectl configuration.

In the “Getting Started” guide, we create a namespace “argocd” and install it with manifests. Alternatively, we can use CLI tool to install it. With Argo CD, we can consider hosting it on a different port than the workload’s port, such as 8443. This port serves as “GitOps management port” separated from the application port (e.g. 443) for business workload. Just like any application on Kubernetes, we usually need to configure an Ingress on the management port.

In the argocd namespace, we can see a few services. The application set controller acts as the main controller that reconciles between actual state and desired state. There is also a redis server, for storing data in Argo CD. Another service to note is the dex server, indicating that the SSO capability is provided by the Dex project.

Features

In FluxCD, they use a CRD Kustomization with kustomize.toolkit.fluxcd.io/v1beta2 as version. This name is confusing. Luckily, in ArgoCD, the CRDs are Application (with argoproj.io/v1alpha1 version) and ApplicationSet (with argoproj.io/v1alpha1 version). The controllers monitors CRs created with these CRDs. ArgoCD defines Application as a group of Kubernetes resources as defined by a manifest. Argo CD Application resource deploys resources from a single Git repository to a single destination cluster/namespace. On the other hand, ApplicationSet uses templated automation to create, modify, and manage multiple Argo CD applications at once. ApplicationSet controllers monitors ApplicationSet resources. We can define a template within the declaration of an ApplicationSet and it allows for parameter substitution.

Another key capability for a GitOps utility is the support of multiple templating tools. FluxCD supports Helm and Kustomize. So do Argo CD. In addition, ArgoCD also supports jsonnet (ksonnet is not supported anymore). Jsonnet is a templating tool with many useful operators. Kubectl can directly consume Jsonnet’s JSON output as if it were YAML. In FluxCD, there are different CRDs for Helm and Kustomize (HelmRelease and Kustomization). In ArgoCD, the Application CRD covers Helm, Kustomize, and Jsonnet by embedding them as attributes in the declaration. This makes it even simpler than Flux CD.

Secret Management

It’s a big no-no to put secret in a code repository. Therefore any GitOps solution must solve the problem with secret logistics. What baffles me is ArgoCD remains un-opinionated on this matter. There appears to be some context for this stance. As a result, ArgoCD only points to a few third-party secret management solutions.

Secret logistics is a tricky problem with GitOps workflow. One of the techniques is sealed secret. In the repo we store the secret as encrypted by public key. The controller keeps the private key which decrypts the secret at the time of deployment. A few engineers don’t find it feasible for operation, as explained in this post and this post.

An alternative is to use external secret operator. Such operators will be able to sync a secret from external secret store such as Hashicorp Vault, AWS secret manager, Azure Key Vault, etc. This is much neater because we only store reference to secret in the repository. However, when the cluster assesses external secret store, it still requires either a secret, or a permission.

Another technique that drawing attentions is using sops for secret and kustomize-sops to integrate with ArgoCD. Mozilla’s sops (SecretOPerationS) project releases a binary utility called sops to encrypt and decrypt YAML manifest. It can encrypt only the values in YAML manifest and not the attributes (keys). We use sops to encrypt YAML files in order to store them in Git repo. We also use sops to decrypt YAML right before we deploy it with kubectl. The key pair for encryption and decryption can be stored in Azure Key Vault, AWS KMS, GCP KMS, and even age and pgp. Here is a simple tutorial.

Example

In a few command we can configure a minimal example. I first use KinD or Minikube to create a cluster following the steps in real-quicK-cluster repo. Then we configure an application using YAML manifest.

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
$ kubectl port-forward svc/argocd-server -n argocd 8443:443

From MacOS, I can now browse to https://localhost:8443/ for ArgoCD UI, and login with admin user and the password as printed above. We can then use the UI to create an Application. Alternatively, we can configure an application using CLI utility:

$ brew install argocd
$ argocd login localhost:8443
$ argocd app create guestbook --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace default
$ argocd app sync guestbook

On the GUI we should see the application is sync’ed.

On the left panel, click on the gear icon for settings. Under Clusters, we can see the cluster URL, which we use as dest-server value above. We can add more clusters here.

Pick the right open-source project

My experience with ArgoCD and FluxCD raised an issue to reflect on. For the same problem, there are multiple choices of open-source project. What would be a good methodology to choose the right open-source technology? I would first look at the following factors:

History
Community size
License model
Enterprise support
Governing model

The history and community size give a good idea of the technical maturity level. The License model has legal implications as to where you can use the technology. License model can change even after the project has been launched and even become not open-source any more. One example is Elasticsearch license change in Jan 2021, from the very open APLv2 to under dual license of both Elastic License (by Elastic) and Server Side Public License (SSPL, introduced by MongoDB). Neither license is certified by OSI as compliant with open source definition. For enterprises, apart from licensing, I’d also consider if any organization provides commercial support for the open-source product.

What is often neglected is the governing model. The governing model determines how product decisions are made that will shape the future of the project. For example, CNCF is a vendor neutral foundation for cloud computing and if an organization contributes a project to CNCF, then the member projects have to align with CNCF’s governing practices. Google also introduced an OUC (Open Usage Common) governing model.

Another good way is to just follow Red Hat’s choice. Red Hat built its business model around open-source technologies. They have to pick open-source project to provide enterprise level support so presumably they have a rigour selection process that one can piggyback off. Products supported by Red Hat are composed of open source components often vetted by multiple upstream communities, and changes made to these components are pushed to their respective upstream projects, often before they land in a supported product from Red Hat. Here are more insights in this business model.

Had I gone with the Red Hat rule, I would have picked Argo CD in the first place because that is the base of OpenShift GitOps!

Verdict

ArgoCD and FluxCD are tools for GitOps. However, just using a GitOps tool does not guarantee that all workload are deployed continuously. First, these GitOps controllers cannot replace Operators, which has domain knowledge about how to orchestrate their workloads. Second, resources not directly managed by these operators (such as those installed by Helm) are not being monitored continuously. For continuous deployment over all workloads, I recommend use operator to install third party applications, and use Argo CD or Flux CD to manage the operators.

GitOps makes use of the controller pattern to manage deployment in a continuous matter. Controller works closely with CRDs. ArgoCD uses similar set of CRDs to manage continuous deployment. What sets it apart is the user friendly web UI, the IAM integration and multi-cluster capabilities. These capabilities make ArgoCD well adapted in enterprise IT eco-system.

Hosting database on Kubernetes

Sun, 29 May 2022 11:01:00 -0400

Background

“We want to host Postgres database on Kubernetes. Can you help us?”. The client appears assertive and reluctant to resort to managed services. So I did some homework and went through this tutorial. My thought: it’s doable, but don’t do it unless operating database as a service is your main business.

I believed that was the client’s best interest, until I came across this the blog post A Case for Databases on Kubernetes from a Former Skeptic. The author explained his journey from being a skeptic, to grudging acceptance, and eventually to an evangelist on running database on Kubernetes. The same voice came from the author of the upcoming book Managing Cloud Native Data on Kubernetes, who also advocates hosting database on Kubernetes. While the points in the chapters are valid, the book also includes a good amount of technical details which might lead reader to believe the opposite view.

Just a few years back, Kubernetes was not mature to host database. This is changing in 2022. Nowadays, for clients with their own Kubernetes platform, technological maturity is no longer the main reason that keeps them from hosting database on Kubernetes, it is the operational cost. The operational cost has to do with whether the client has in-house expertise in database and Kubernetes. If they do, the hard path makes economical sense.

In this post, we discuss what we need to be aware of in order to host database on Kubernetes.

Benefit with Kubernetes

The first few versions of Kubernetes only supported stateless workload (reference documentary). That is what Kubernetes was born to solve. Built-in objects such as replicaSet, deployment, horizontalPodAutoscaler are abstractions of operations particular to stateless workload. Pods for stateless workload are ephemeral: they crash and get replaced at any time. Because they don’t carry persistent data themselves, they are expendable.

Kubernetes’ orchestration capability are driven by controllers. As discussed, the controller pattern is adopted in all controller implementations. They are the engines of the platform that works tirelessly in a control loop to ensure desired states matches their declared states. This is a key feature of Kubernetes as container platform. Let’s examine a web service that requires 5 instances behind load balancer. With traditional hosting model on Linux servers, you’d have it installed on all five VMs. If the process on one of the VMs dies, the VM has to be removed from the load balancer’s target pool.

One may wrap the process with process monitor and control utility such as supervisord, and re-install the application using automation utility (e.g. Ansible). However, each server is unaware of the status of its peer. Without a central “Control Plane”, there is no coordination between the activities of each VMs. Kubernetes controller solved all these operational problems.

Control Loop

Kubernetes comes with a set of build-in controllers that run inside the kube-controller-manager. Here is a good page about how controllers work. Controller is what is missing in many automation tools other that Kubernetes. Even though Red Hat now brands Ansible as Automation Controller, it does not involve a control loop or controller pattern. If there’s one thing that sets Kubernetes apart from other hosting platforms and automation platforms, it is the implementation of controller pattern.

Stateful workload

Does the controller pattern also benefit stateful workload? Yes. How to orchestrate Pods for stateful workload is usually more tricky. CRD can define a custom object type for controller to consume. In this case, an operator is an implementation of the controller pattern. This pattern is also known as the operator pattern. In a replicaSet, Pod names have extensions of randomly generated numbers. A statefulSet names its the Pods by sequential numbers.

For Postgres database, Bitnami built a good Helm Chart to install the database automatically. However, it does not have a control loop. If someone changes the workload after initial installation, the change is not monitored or controlled by any controller. This is a disadvantage of Helm chart as compared with operators. For PostgreSQL, there are a number of operators, the most notable being PGO (Postgres Operator) from Crunchy Data.

To install an instance of PostgreSQL database, we need to install the operator, and then declare a Custom Resource using the PostgresCluster CRD. The operator will set up the cluster according to the declaration made in the PostgresCluster CR. I used the quick start guide to bring Postgres up real quick on an Azure Kubernetes cluster. The operator (v5) supports common cloud Kubernetes platforms (GKE, EKS, AKS), VMware Tanzu, Openshift, Rancher, Kubernetes. It does not explicitly indicate whether PGO supports Minikube or kind.

So far, I’ve discussed the pros of running PostgreSQL on Kubernetes using Postgres Operator. We can describe the database deployment in a CR and the controller (operator) will monitor the resource incessantly to ensure the actual state matches the state defined in the CR. Not only is it doable to host database in Kubernetes, it makes our lives even easier.

Persistent storage

Database is not only a stateful workload, it also has special requirement on storage. It needs to persist data, support ACID transaction, and make optimal use of disks. When we operate everything on premise, we use fibre cable with a SAN as the storage media for database file. The operating system allows the database process to interact with blocks on the storage volume via device mapper.

In Kubernetes, we need to give Pods persistent volumes. There are a few APIs: Storage Class, Volume Storage Class, Persistent Volume and Persistent Volume Claims. Storage Class represents how Pod can connect to a storage. Pods will need PVCs in order to read and write on PVs. However, since Pods are ephemeral – a Pod may crash any time, even if it is in the middle of writing to a PV, during an ACID transaction. The scheduler may reschedule the crashed Pod to a different node. Then it will need to pick up the PV from where it left off, on the new Node.

Take Azure Kubernetes Service for example, a few storage classes are available by default, backed by Azure managed disk (managed-csi) or Azure file storage (azurefile-csi):

	StorageClass	Azure storage service
in-tree	default	Managed Disk using Azure StandardSSD
	managed-premium	Managed Disk using Azure Premium Storage
	azurefile	Azure File Share using Azure Standard Storage
	azurefile-premium	Azure File Share using Azure Premium Storage
csi	managed-csi	Managed Disk using Azure StandardSSD
	managed-csi-premium	Managed Disk using Azure Premium Storage
	azurefile-csi	Azure File Share using Azure Standard Storage
	azurefile-csi-premium	Azure File Share using Azure Premium Storage

If we use storage class based on Azure disks to create a PV, only one Pod can use the PV. If we use storage class based on Azure files to create a PV, then the storage is mounted as NFS (Linux) or SMB (Windows) share. File storage is not a valid use case for database workload and it can significantly degrade database performance. When I tried to use a file-storage based CSI with PGO, the Pod reports an error and will not start properly. We should use Azure disk based CSI storage classes. That leaves us with two options: managed-csi and managed-csi-premium.

High Availability

Even with these to options left, we still have to investigate how database Pods interact with persistent volume for high availability, in order to determine whether any of the options are suitable. The two storage classes differ by disk performance but both have its own limitation with multi-AZ support on Azure managed disks. When the cluster operates across zones, the Kubernetes scheduler may reschedule a Pod crashed in one zone to a Node in a different availability zone (a different data centre). Even though the managed disks, when attached to VMs, can be configured as zone-redundant, when they are used as Kubernetes volume, they are NOT zone-redundant. So the node in a different zone will not be able to attach PV to the new Pod.

There are SDS (software-defined storage) solution such as Portworx that solves the limitation of Azure disk for cross-region storage volume. The SDS layer brings managed disks from multiple availability zones into a pool. This storage pool acts as a highly available, cross-zone storage tier presented to AKS as persistent volumes. We can install Portworx as the SDS layer using Portworx operator. To do so, we first have to configure grant the cluster the permission to provision resources in Azure, because the Portworx operator will use node’s identity (kubelet identity) to provision Azure resources on behalf of the nodes. Portworx will provision Azure disks and acts as the intermediary layer.

Apart from cross-zone high availability enabled by PX-Store, Portworx can also help with cross-region replication of persistent volumes. The PX-DR component can perform asynchronous replication across Azure regions. The destination region needs to have its own cluster because a single AKS cluster cannot span across regions.

Storage Class

Once we have portworx installed, the following storage classes are available by default:

px-db
px-db-cloud-snapshot
px-db-cloud-snapshot-encrypted
px-db-encrypted
px-db-local-snapshot
px-db-local-snapshot-encrypted
px-replicated
px-replicated-encrypted

The steps for installing porworx on AKS are documented here. This blog post has more details in the installation process on a different platform. We can also built CSI based storage classes with different IO priority and replication factors.

In summary, Kubernetes operator pattern makes it easier to manage stateful workload. However, database performance depends largely on storage. To host database on Kubernetes, one will have to also manage the storage volumes on their own. There has not been a study on the impact to performance by moving database to Kubernetes platform. However, I only expect a degraded performance due to the layers introduced.

Example

In this section we configure a (minimally viable) PostgreSQL cluster using Crunchy Data pgo to demonstrate the idea. The steps are based on its tutorial but it works on a local KinD cluster. As discussed in a previous post, I use KinD for testing workload requiring persistent storage because Minikube has this open issue with permissions on PVs with multiple nodes.

To prepare the cluster, we can use kind-config.yaml file from my real-quicK-cluster repo:

kind create cluster --config=kind-config.yaml
# to delete cluster after testing: kind delete cluster --name kind

We use Helm to install the operator. Since the Helm chart is not hosted in a public repo, we’d have to download the directory of the Helm Chart.

git clone https://github.com/CrunchyData/postgres-operator-examples
cd postgres-operator-examples
helm install -n postgres-operator --create-namespace crunchy-pgo helm/install
kubectl -n postgres-operator get po --watch
kubectl explain postgresclusters

Now we can create a YAML file for the Custom Resource and let’s call it test-cluster.yaml with the following content:

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
 name: hippo
 namespace: postgres-operator
spec:
 backups:
 pgbackrest:
 image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi8-2.38-1
 repos:
 - name: repo1
 volume:
 volumeClaimSpec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 1Gi
 storageClassName: "standard"
 image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.3-0
 instances:
 - dataVolumeClaimSpec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 1Gi
 storageClassName: "standard"
 name: instance1
 replicas: 3
 minAvailable: 2
 postgresVersion: 14

In the manifest, we specified a cluster, using storageclass named “standard”, with 3 replicas and requiring 2 available. We assume a storage class named “standard” already exists and optimized for database workload. In the manifest, we also configured a backup job. We can apply the manifest and watch for the Pods to come up in a few minutes.

kubectl apply -f test-cluster.yaml
kubectl -n postgres-operator get po --watch
kubectl -n postgres-operator describe postgresclusters hippo

The Pods in the postgres-operator namespace should report something like this:

NAME READY STATUS RESTARTS AGE
hippo-backup-mwpm-ps8wk 0/1 Completed 0 21s
hippo-instance1-6mls-0 4/4 Running 0 3m35s
hippo-instance1-hjp6-0 4/4 Running 0 3m35s
hippo-instance1-k4qf-0 4/4 Running 0 3m35s
hippo-repo-host-0 2/2 Running 0 3m35s
pgo-548d5f48bc-9w4z4 1/1 Running 0 8m41s
pgo-upgrade-566b9cc98f-d7gkr 1/1 Running 0 8m41s

Three Pods for PostgreSQL are all up. The first backup run has completed already. We can connect to the cluster using psql following the quick start guide. We can also configure an application. A good example application that uses PostgreSQL database is KeyCloak. We briefly mentioned it in OIDC discussion. Currently the keycloak example on Crunchy pgo’s quick start guide is outdated. Instead, use the following content as keycloak.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: keycloak
 namespace: postgres-operator
 labels:
 app.kubernetes.io/name: keycloak
spec:
 selector:
 matchLabels:
 app.kubernetes.io/name: keycloak
 template:
 metadata:
 labels:
 app.kubernetes.io/name: keycloak
 spec:
 containers:
 - image: quay.io/keycloak/keycloak:latest
 name: keycloak
 args: ["start-dev"]
 env:
 - name: DB_VENDOR
 value: "postgres"
 - name: DB_ADDR
 valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: host } }
 - name: DB_PORT
 valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: port } }
 - name: DB_DATABASE
 valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: dbname } }
 - name: DB_USER
 valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: user } }
 - name: DB_PASSWORD
 valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: password } }
 - name: KEYCLOAK_USER
 value: "admin"
 - name: KEYCLOAK_PASSWORD
 value: "admin"
 - name: PROXY_ADDRESS_FORWARDING
 value: "true"
 ports:
 - name: http
 containerPort: 8080
 - name: https
 containerPort: 8443
 readinessProbe:
 httpGet:
 path: /realms/master # https://stackoverflow.com/questions/70577004/keycloak-could-not-find-resource-for-full-path
 port: 8080
 initialDelaySeconds: 30
 restartPolicy: Always

Once we apply keycloak.yaml, in a minute we should see and be able to port-forward web traffic:

$ kubectl apply -f keycloak.yaml
$ kubectl -n postgres-operator get po -l app.kubernetes.io/name=keycloak
NAME READY STATUS RESTARTS AGE
keycloak-7995d78d7c-zjp4d 1/1 Running 0 4m29s
$ kubectl port-forward deploy/keycloak -n postgres-operator 8080:8080

After using the port-forward command, we can browse to web portal on my MacBook by http://localhost:8080 and configure an initial password, as shown here:

In real life system we would need a proper Ingress. After testing, delete the cluster with kind command and specify the cluster name (kind).

Operation Cost

Operation cost is an important consideration. Troubleshooting on Kubernetes platform is in general more complicated than just on a Unix system. Hosting database on Kubernetes requires skills not only on the Kubernetes platform, but also on database. There used to be database administrator positions where someone has to maintain the upgrade, the storage, the replication, the multi-tenancy and the performance optimization of database. With a database hosted on Kubernetes, the database administrator will have to perform all these activities on a containerized platform. This is not an easy undertaking, and in many occasions warrants a full-time position on its own. Therefore, don’t host your database on Kubernetes, unless that is your main business. It is not the technology that shots down this option. It is the operation cost, such as complexity of configuration, and staff skillset, that makes this option not worth it.

Kubernetes Operator

Thu, 07 Apr 2022 09:39:00 -0400

Kubernetes has a number of tools to automate the deployment of a single workload. In previous posts, we had covered Helm and Kustomize. What are left unresolved is how to maintain the status of workload after deployment is completed. In this post, I will give an introduction to Kubernetes Operator. Compared with Helm (templating approach) and Kustomize (patching approach), Kubernetes Operator follows the operator pattern. Operators are usually provided by the developer of the application.

Operator Pattern

In Kubernetes, we know that a controller takes care of routine tasks to ensure that desired state expressed by Kubernetes resource types matches the current state. One example is that the Deployment controller ensures the number of pods running matches the amount specified in the replica field. Controller is the key to ensure that resources can be managed by declarative manifests for Kubernetes resources.

Kubernetes makes use of controller pattern throughout its own design. One of its key component, Controller Manager, is a collection of many controllers. Each controller is in charge of a control loop, responsible for listening the object it manages. Another component, Kube-scheduler, is also a special type of Controller. The kube-scheduler monitors unscheduled Pod and health of nodes and determines the best Node to schedule the new Pod to. Then it writes the decision to etcd store for kubelet to execute.

This controller pattern is fairly successful in what it does and we can extend the use of it. Beyond the built-in resource types, we can create our own custom resource definitions (CRDs), and create controllers that watches for the manifest that declares custom resources (CRs). The controller ensures that the resource status matches their specifications. This is also known as reconciliation, which is implemented as a control loop. Operator pattern can be illustrated in the diagram below:

Operator Pattern

Technically, there is no difference between a controller and an operator. What makes an Operator (used to install workload) different than a native Kubernetes controller, are two things. First, an Operator usually needs CRDs because the built-in resource types are insufficient. Second, the operator reflects the domain knowledge to keep the target workload running. For example, stateful workloads such as database needs their operational steps executed in certain orders.

On Operator Pattern, CNCF published a whitepaper with a deeper review. This white paper is the best reference for a good understanding of the Operator Pattern.

Custom Resource Definition

The built-in controllers work with built-in objects (pre-defined APIs). Custom operators usually need their own APIs to function. To extend Kubernetes API, we define the schema of these APIs in the form of CRDs (custom resource definitions) using OpenAPIv3 standard. Then, we can declare Custom Resources (CRs) in compliance with the schema. The OpenAPIv3 schema in the CRD resource tells validating web hook (admission control) how to validate the schema when we send an CR in to API server.

When we work with third-party operators, they usually provide CRDs along with the operator implementation. For example, in my operator example project, we have a minimalist CRD WordPress with one property: sqlRootPassword and we can declare a CR as in this example. For a more realistic use case, we can take a look at Kiali CRD. In the next section, we’ll use it along with Kiali operator to install Kiali.

Operator Usage

Like Artifact Hub to Helm, OperatorHub is a public registry of most used Kubernetes Operators. In this section, we will take an example of using Operators. We will install Kiali as an add-on to Istio using Kiali CR and operator, which also depends on Prometheus to be installed using Prometheus Operator first. Note that the Kiali installation outlined in this section is not the the quick-start install manifests from Istio’s sample directory. For Kiali on production system we have to customize the installation.

Suppose we have installed Istio, we can then install Prometheus operator using Helm. The Prometheus operator will install Prometheus. Then we use Helm again to install Kiali operator. The Kiali operator will watch for creation of Kiali CRD, to deploy services:

$ helm install -f prometheus-values.yaml --namespace istio-system --repo https://prometheus-community.github.io/helm-charts --version 13.6.0 istio-prometheus prometheus --insecure-skip-tls-verify
$ helm install -f kiali-operator-values.yaml --namespace kiali-operator --repo https://kiali.org/helm-charts --version 1.45.0 kiali-op kiali-operator --create-namespace
$ kubectl apply -f kiali-cr.yaml

I include example content for each file in the commands above on Github gist (prometheus-values.yalm, kiali-operator-values.yaml and kiali-cr.yaml). For more options for installing Kiali, refer to their documentation.

I use this example to install Kiali and it includes two Operators, the Prometheus Operator and the Kiali Operator. The Prometheus Operator is one of the first ever written Kubernetes Operator. As soon as the operator is deployed, it starts to deploy the operator service. For the Kiali operator, we need to deploy Kiali CR after the Kiali Operator has been deployed. Both are valid patterns.

Operator Development

Operator is powerful. However, authoring an Operator is not a trivial effort. One usually start with a framework. A framework creates a body of boiler plate code that has the pattern implemented and allows developers to enrich the functions following the pattern. The white paper introduced three frameworks:

CNCF Operator Framework – aims at Operator Developers with an SDK, a scaffolding tool and a test harness. It currently supports three project types: Golang, Helm and Ansible. CNCF Operator framework consists of SDK and OLM.
Kopf (Kubernetes Operator Pythonic Framework) – an easy-to-use framework in Python that abstracts away most of the low-level Kubernetes API communications hassle.
kubebuilder – helps build a Manager similar to the native kube-controller-manager. For difference with OperatorSDK, read here.
Metacontroller: lightweight Kubernetes Controller as a Service

In CNCF Operator Framework, the Operator SDK supports development using Ansible, Helm and Golang. The author of this post makes a general comparison as follows:

Type	Best use case	Underlying technology	Amt of Effort
Helm	Stateless workload	Helm Charts	Med
Ansible	Stateless workload	Ansible Roles and Playbooks	Med
Golang	Stateful workload	Code developed in Golang	High

The aforementioned Kiali operator is an example of Operator developed in Ansible. The prometheus operator, is developed in Golang as the workload can be stateful depending on configuration. One needs to know how to develop operator in Golang in order to tackle the most complicated situations. This is requires some serious development effort. The documentation with a quick start section is available here. Even that is not very straightforward. RedHat, the maintainer of the CNCF Operator framework has a good blog post on how to develop an Operator in Golang.

The example requires some development knowledge to go through. On my MacOS (Intel) I have to configure the following prerequisites:

Install gcc, using command: xcode-select –install
Install the right version of golang. You can find the version here. The MacOS has a version of golang installed already so I had to install version 1.17 and link to it: brew install go@1.17 && brew link –force go@1.17
Install operator-sdk with home brew: brew install operator-sdk
When you run “operator-sdk version”, ensure the result shows a golang version that matches your installation.
If you need to push docker image, also connect to docker registry by running: docker login

Then we can create our working directory, initialize the repository and create boilerplate code (scaffolding) with these commands:

$ mkdir wordpress-operator && cd wordpress-operator
$ operator-sdk init --domain digihunch.com --repo github.com/digihunch/wordpress-operator
$ operator-sdk create api --group wordpress --version v1 --kind WordPress --resource --controller

With the repo initialized, we can go to the section “Defining the API” and “Implementing the Controller”. The blog post does not cover every code editing needed to bring up wordpress. You are supposed to go to the author’s repository to fit the changes into your own repo. The author’s repo has a few more controllers such as common.go and mysql.go.

At the end of the lab, you should be able to run the controller and bring up wordpress. I used my own repository for this lab and have made the code changes for this lap in a couple commits. To test locally with the code:

$ git clone git@github.com:digihunch/wordpress-operator.git
$ cd wordpress-operator
$ make install run

Then we can validate wordpress install from a new terminal as the instruction shows:

$ kubectl create -f config/samples/wordpress_v1_wordpress.yaml
$ minikube service wordpress --url

For Developers that requires more details, RedHat has an eBook for Kubernetes Operators, in supplement to the documentation. As DevOps professional, I’m mainly concerned with understanding how Operator works and using Operators correctly.

Too many Tools?

Now we seem to have too many choice of tools when it comes to deploying workload on Kubernetes. Kustomize and Helm can deploy simple workloads. Operator can deploy stateful workloads, as well as keep the workload status in check. Further, we have FluxCD and ArgoCD based on GitOps workflow.

When assessing a tool, we should think about the complexity of the workload deployed. If it is a single stateless workload, Kustomize or Helm should be sufficient. If it is not very simple but still stateless, we can consider using Helm charts developed by the community. For multiple workloads, we can build our own top-level chart to combine existing sub-charts created by the community.

Helm is essentially a package manager. It does not follow controller pattern and therefore will not monitor the current status of deployment. Helm has other limitations compared to Operator. For example, as a templating scheme, it reaches limitation when dealing with complex logic, even with the help of its helper functions. It is also hard to reason through the template code when we have to troubleshoot a deployment. Refer to this blog post for the author’s experience with Helm.

If we want our deployment to be fully declarative and continuous, then we will follow the Operator pattern by using a Kubernetes Operator. When we have many workloads of different levels of complexity, we can combine them with GitOps tool. Operator is one of the underlying technologies behind GitOps.

Workload profile	Just Installation	Installation and Maintain Status
Single stateless workload	Helm or Kustomize	Operator (using Ansible or Helm)
Single stateful workload	Helm or Kustomize	Operator (using Golang)
Multiple workloads	Helm (e.g. build parent chart)	GitOps in combination with Operator, Helm and Kustomize

The table above helps refine deployment requirement. It’s not a recommendation, but rather a model of analyzing deployment requirement.

Istio Operation Gotchas

Sat, 19 Mar 2022 11:09:00 -0400

In this post I discuss a few aspects when putting istio in operation.

Installation

Istio installation can be confusing, due to architectural and guideline changes as well as renaming of operator CRDs since its release, and especially since 2020. This left lots of information outdated on the web, adding to Istio’s perceived complexity.

Currently, the recommended installation methods are istioctl or Helm. Using Istio Operator manifest (with out istioctl) is discouraged. As to the Helm chart installer, it was once deprecated (around early 2020), but was later re-introduced in the fall of 2021. This back-and-forth had caused some aversion. As of Nov 2021, their re-introduced Helm charts version dropped alpha tag. The Helm repo consists of separate Helm charts, for Istio CRD (base), control plane (istiod), each gateway and CNI respectively. A typical deployment therefore requires multiple Helm Releases (example here). Istio document still considers Helm support as alpha, so I assume the most reliable method to install Istio is istioctl.

The istioctl utility can be used with many options to customize Istio install. For example, we can supply a YAML declaration input (to -f switch) to customize installation behaviours. The YAML file declares a CRD. Two different CRDs have been used: IstioOperator and IstioControlPlane. According to this blog and this post, since Istio 1.5 in early 2020, we’re supposed to IstioOperator CRD exclusively. The IstioControlPlane CRD is left only for legacy support. As stated in the current documentation:

The istioctl command supports the full IstioOperator API via command-line options for individual settings or for passing a yaml file containing an IstioOperator custom resource (CR).

With IstioOperator CRD, we still have a number of options to tweak the install behaviours. Here is a summary of potential options:

use IstioOperator API via IstioOperator CRD (without using “values” or overlay fields)
specify an attribute value in argument, including a pre-built profile e.g. –set meshConfig.accessLogFile=/dev/stdout, –set profile=demo
use K8sObjectOverlay by using “k8s/overlays/patches” field in IstioOperatorCRD
use Helm API by using “values” field in IstioOperatorCRD

The key-value specified in –set switch overrides the same key-value supplied in the IstioOperator CRD. So option 2 overrides option 1. The value for profile can also be empty if you’d rather start from scratch. However too many –set switches makes the command wordy so we should build our own IstioOperator CRD

For option 4, the document hyper-links Helm API to a section from version istio 1.4, and I appears to exist only for legacy (pre-2020 Helm support) compatibility. Option 3 (K8sObjectOverlay) would be helpful when a field cannot be conveniently customized with option 1 and we have to patch the object like in Kustomization.

So the most practical approach is IstioOperator CRD for per-component customization, potentially with K8sObjectOverlay. No matter which option, istioctl compiles the installation manifest before applying it against Kubernetes API. This manifest can be previewed using “istioctl manifest” command, so that you can take a look before installation.

Thank you Isito for so much confusion just to land on a working installation method. Below is the content of az-istio-operator.yaml file that I use for my installation:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
 name: istio-install-customization
spec:
 profile: default
 hub: docker.io/istio
 tag: 1.13.1
 revision: 1-13-1
 namespace: istio-system
 meshConfig:
 accessLogFile: /dev/stdout
 outboundTrafficPolicy:
 mode: "REGISTRY_ONLY"
 components:
 pilot:
 k8s:
 hpaSpec:
 maxReplicas: 7
 minReplicas: 1
 nodeSelector:
 beta.kubernetes.io/os: linux
 ingressGateways:
 - name: istio-ingressgateway
 #namespace: istio-ingress
 enabled: true
 label:
 istio: ingressgateway
 k8s:
 hpaSpec:
 maxReplicas: 11
 minReplicas: 1
 serviceAnnotations:
 service.beta.kubernetes.io/azure-load-balancer-internal: "true"
 service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "my-lb-subnet"
 egressGateways:
 - name: istio-egressgateway
 enabled: true

I can preview the install, run the install and validate installation status:

$ istioctl manifest generate -f az-istio-operator.yaml
$ istioctl install -f az-istio-operator.yaml -y --verify
$ istioctl verify-install -f az-istio-operator.yaml
$ kubectl -n istio-system get IstioOperator installed-state-istio-install-customization-1-13-1 -o yaml | less

Note that in the IstioOperator declaration I marked the revision. This is helpful when I run multiple versions of control plane (e.g. during upgrade). I can check revisions of istiod with:

$ istioctl x revision list

We can delete installed istio components

$ istioctl x uninstall -f az-istio-operator.yaml
$ istioctl x uninstall --revision 1-11-5

In practice, it is helpful to use separate operators each for a different component (pilot, ingressGateways, egressGateways). This makes maintenance and upgrade easier.

Debugging

The documentation has a page for common problems that one needs to be familiar with. It covers not only problems, but also steps to troubleshoot each kind of problem (e.g. authorization policy).

Although istioctl is pretty confusing as an installation tool, it is a good utility for many troubleshooting activities. We should probably add its path to PATH environment variable and add the export command (e.g. export PATH=”$HOME/istio/bin:$PATH“) to .zshrc or .bashrc. Istioctl has a few useful subcommands, some of which are only available as experimental and therefore needs to be following an x. Some common commands are given below:

To analyze Istio problems:

$ istioctl analyze -n istio-system

To look at proxy configuration of an Envoy instance at different levels, use proxy-config sub-command or pc for shorthand:

$ istioctl proxy-config <clusters|listeners|routes|endpoints|bootstrap|log|secret|stats>
$ istioctl pc cluster deploy/istio-ingressgateway -n istio-system # see what Envoy cluster an ingress gateway knows about
$ istioctl proxy-config log deploy/httpbin --level "rbac:debug"
$ istioctl pc log <pod_name> -n <namespace> --level connection:debug
$ istioctl pc secret -n istio-system deploy/istio-ingressgateway # check certificates loaded to a gateway
$ istioctl proxy-config listeners deploy/istio-ingressgateway -n istio-system # query envoy listener configuration
$ istioctl pc routes deploy/istio-ingressgateway -n istio-system --name http.8080 # query envoy route configuration

The last two commands set logging level to the specified workload. If we want to set logging level at mesh level, we can use these commands:

$ istioctl admin log --level authorization:debug
$ istioctl admin log

To look at synchronization status of each envoy in the mesh, use proxy-status sub-command, or ps for shorthand:

$ istioctl ps # ensure data plane is in sync

If an item for a workload shows STALE instead of SYNCED, it means that the configuration has not been pushed from control plane to that instance of Envoy proxy. Check if the Istio configuration change is valid. If the system is newly installed and there is no ingress or egress gateway resources declared, the RDS column for ingress or egress may show “NOT SENT”. The far right column displays the version of istiod connected.

To describe applied istio config:

$ istioctl describe <pod|service>
$ istioctl describe po workload1 -n my-workload # detect misconfigurations on workload

To view dashboard:

$ istioctl dashboard <envoy|grafana|prometheus>

To check authorization policy on a Pod,

$ istioctl x authz check mypod -n workload

To validate istio configuration in a file:

$ istioctl validate -f resource_authorization_policy.yaml

Sometimes we need to turn on access logging just on the envoy proxy on the Gateway Pod. In that case, we will need to apply Istio’s Envoy filter object. This filter is applied to Pods labelled as gateway. It patches the existing filter chain with the additional defined in the manifest:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
 name: hcm-gw-access-log
 namespace: istio-system
spec:
 workloadSelector:
 labels:
 istio: ingressgateway
 configPatches:
 - applyTo: NETWORK_FILTER
 match:
 context: GATEWAY
 listener:
 filterChain:
 sni: demo.digihunch.com
 filter:
 name: "envoy.filters.network.http_connection_manager"
 patch:
 operation: MERGE
 value:
 typed_config:
 "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
 access_log:
 - name: envoy.access_loggers.file
 typed_config:
 "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
 path: /dev/stdout
 format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"

The object above enables access logging on the Gateway Pods only, without impacting other Envoy proxies.

Multi-tenancy

In large enterprises, Istio is usually installed by a platform/operation team. The entire platform is shared by multiple application teams. Istio should not be seen as a responsibility of a single party. It is necessary to break down the Istio CRDs and define the responsibility of each CRD. For example:

Gateways are placed in the istio-system namespace (or a dedicated istio-ingress namespace in some case). Platform team can decide whether a Gateway is considered a shared infrastructure, or each tenant (application team) uses their own Gateway. In the servers/hosts field of Gateway declaration, add namespace before host to suggest that the routing behaviour of that host must be defined in a certain namespace.
Virtual Services can be managed in two models as well. They can be managed individually and are all placed in each tenant namespace. Alternatively in a shared responsibility model, Virtual Service can be created in istio-system namespace, and the processing of each match can be delegated to a Virtual Service in each tenant namespace, using the Delegate feature of Virtual Service. Another field that can help with multi-tenancy is the “gateways” field, you can specify a value of mesh to indicate the virtual service is available to the entire mesh.
Destination Rules are usually placed in each Tenant’s workspace
Peer Authentication also depends on the configuration. If a mesh level configuration is enforced, it is easier for the Platform Team to manage it and this should be the setup for new clusters. If for historical reasons Peer Authentication is enforced per tenant namespace, it can be delegated to each application team. It can make communication troubleshooting more complex.
Control Plane and observability workloads such as Kiali are usually the responsibility of Platform team.
Request Authentication and Authorization Policy should be the responsibility of individual application team.

In a multi-tenancy management model, Istio related resource should be subject to admission control policy, as well as scrutiny by the security and platform teams. Google Anthos has a good page on the constraint templates for Istio resources.

As to restricting traffic between namespaces, apart from the measures from this previous post, we can also use Sidecar CRD to restrict outbound traffic. As its documentation states:

Sidecar describes the configuration of the sidecar proxy that mediates inbound and outbound communication to the workload instance it is attached to. By default, Istio will program all sidecar proxies in the mesh with the necessary configuration required to reach every workload instance in the mesh, as well as accept traffic on all the ports associated with the workload. The Sidecar configuration provides a way to fine tune the set of ports, protocols that the proxy will accept when forwarding traffic to and from the workload. In addition, it is possible to restrict the set of services that the proxy can reach when forwarding outbound traffic from workload instances.

The documentation page also includes two examples. The first is a sidecar at the mesh level that restricts outbound traffic to the same namespace that the sidecar is in, and the istio-system namespace:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
 name: default
 namespace: istio-config
spec:
 egress:
 - hosts:
 - "./*"
 - "istio-system/*"

The second example overrides the mesh level default above, and allows egress traffic to three specified namespaces:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
 name: default
 namespace: prod-us1
spec:
 egress:
 - hosts:
 - "prod-us1/*"
 - "prod-apis/*"
 - "istio-system/*"

As to inbound control, we can expose a Virtual Service to other namespaces by using the exportTo field to specify which other namespaces the Virtual Service should be exported to. If no namespaces are specified then the virtual service is exported to all namespaces by default.

Sometimes we want a virtual services to expose to both outside of the mesh via Ingress, and within the mesh, and we hope to use the same hostname. For this requirement, we can use the ServiceEntry CRD. ServiceEntry enables adding additional entries into Istio’s internal service registry.

FluxCD: Continuous Deployment with GitOps

Sat, 15 Jan 2022 18:49:00 -0400

This post explains why I land on FluxCD GitOps for my project. Let’s star

Background

In the Korthweb project, I landed on Istio for the Ingress Gateway technology. I first attempted to expand the orthanc Helm Chart to bring Istio as dependency (sub-chart). One of the external chart for Istio gateway needs to be referenced multiple times (for ingress and egress). However, it cannot even be used as dependency (sub-chart) because of this issue. Istio didn’t re-introduce Helm Chart as a supported deployment until September 2021. So I’m not too confident about it.

This leads to a second thought of the “Big Helm Chart” approach to deploy all tiers in Korthweb. Helm is based on templating, and having two layers of charts brings complexity. I do need to bring a number of Helm charts together, but not necessarily by another Helm Chart.

The Kubernetes documentation also mentions Kustomize as an alternative to Helm. Below is a quick exploration of it.

Kustomize

A challenge with managing declarative object is to organize numerous manifest files to maintain consistency and readability. Kustomize is a standalone tool to customize Kubernetes objects through a Kustomization file. This page provides a good list of fields that can be used:

vars and replacements: copy fields from one source into any number of specified targets.
namePrefix, namespace, nameSuffix: customize namespace and names.
configMapGenerator, secretGenerator, and generatorOptions: create configuration entries from literal, files, or environment variables.
resources: indicates another kustomization directory (e.g. as base)
patches (also called overlays) add or override fields on resources.
patchesStrategicMerge: modifies values in known (loaded) resources
images: modifies the name, tags and/or digest for images, without creating patches.
openapi: use Kubernetes OpenAPI data to get merge key and patch strategy information about resource types.

The base -> overlay pattern ensures readability as well as portability. Typical use pattern is create one base kustomization, and several overlay kustomizations each representing an environment, such as dev, qa and production. Here is a good tutorial.

Helm and Kustomize are two approaches to deploy Kubernetes workload. Both tackle the challenge of managing many YAML declarations. Helm is template driven, and is commonly used by application developers as a means of packaging application releases while orchestrating the dependencies. Kustomize follows a base-overlay pattern, and is more commonly used by cluster operators for re-using manifests across multiple environments (e.g. dev, staging and prod).

In my use case, my deployment needs both. I need third party Helm chart to configure components such as PostgreSQL for HA. I also need Kustomize for the orthanc workload. With the need for both, there should be a higher level deployment technology that ingrate with both mechanisms.

GitOps

GitOps is originally brought up by Weaveworks in 2017. It is a methodology to deploy workload continuously, using a Git repository as source of truth. The point of GitOps is not about tooling, or specific platform, but rather to ensure the workload state matches the declaration in repository. For example, you can implement GitOps with Ansible for VM environment.

When it comes to managing Kubernetes workload, a GitOps tool must handle the challenge with managing many YAML declarations. Therefore most GitOps tools seek to support many well-adopted mechanisms such as Helm and Kustomize as discussed above, instead of simply taking an enormous amount of raw YAML declarations. The amount of deployment mechanisms supported, is a key indicator of how powerful a GitOps tool is.

The most notable tools are ArgoCD and FluxCD. Both are currently CNCF incubating projects. ArgoCD is powerful with many tools supported, such as Kustomize, Helm, Ksonnet, Jsonnet, etc. It also contains a user interface and aims to manage an entire deployment workflow. On the other hand, FluxCD has controllers mainly for Kustomize and Helm, with a jsonnet extension from third party. This post draws a comparison of them (along with JenkinsX) based on earlier versions (from mid 2020). There is even a standard (OpenGitOps) in CNCF landscape but still at Sandbox level.

For Korthweb project, I chose FluxCD. It is simple, and provides just enough types of controller for what I do.

FluxCD

The diagram below illustrates the components:

The configurations starts with a bootstrapping process, which creates directory in Git repository (if not exist), and installs flux-system components in the target Kubernetes cluster. The sync process starts as soon as bootstrapping is completed. The process in charge of syncing declarations to target cluster, confusingly, is also called Kustomization. Therefore there are two Kustomizations. According to the FAQ on FluxCD website:

There are two Kustomization types. the kustomization.kustomize.toolkit.fluxcd.io is a Kubernetes custom resource while kustomization.kustomize.config.k8s.io is the type used to configure a Kustomize overlay. The kustomization.kustomize.toolkit.fluxcd.io object refers to a kustomization.yaml file path inside a Git repository or Bucket source.

Inside of the Git repository, with a kustomization.kustomize.toolkit.fluxcd.io obejct, the flux-system points to Kustomization file (representing kustomization.kustomize.config.k8s.io object) at root level. The kustomization.yaml file organizes resources in the same directory. A kustomize directory may also reference other kustomize directory, forming a hierarchy.

Implementation

FluxCD has a command “flux check –pre” to check the prerequisite, such as kubectl. The code is stored in the GitOps directory of Korthweb repository.

To configure deployment, we need to first create a personal access token. For GitHub, here is the instruction. Export the token to environment variable, and launch bootstrapping:

$ export GITHUB_TOKEN=xxx_yyy55555XXXodr7ABBBB234CCccw
$ flux bootstrap github \
 --owner=digihunch \
 --repository=korthweb \
 --branch=main \
 --personal \
 --path=gitops/environment/dev

A deploy key is configured during the bootstrapping process. As soon as bootstrapping is completed, the sync (aka kustomization, or reconciliation) has started, which can be monitored using:

flux get kustomizations --watch

Running this command without –watch switch returns the overview of all kustomizations. Once reconciliation is completed, it should display something like:

NAME 	READY	MESSAGE 	REVISION 	SUSPENDED
application 	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False
dependency 	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False
flux-system 	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False
infrastructure	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False

FluxCD introduced a number of CRDS. For example, to check configured source repository, check gitrepositories CRD:

kubectl get gitrepositories -n flux-system

To check the detail of one kustomization (e.g. infrastructure), check kusomization CRD in flux-system namespace:

kubectl -n flux-system get kustomizations flux-system -o yaml | less

Other commonly used custom resources include:

helmcharts
helmreleases
helmrepositories

A Helm release can be created imperatively, for example:

$ flux create source helm istio \
 --interval=1h \
 --url=https://istio-release.storage.googleapis.com/charts
$ flux create helmrelease istio-base \
 --interval=1h \
 --release-name=istio-base \
 --target-namespace=istio-system \
 --create-target-namespace=true \
 --source=HelmRepository/istio \
 --chart=base \
 --chart-version="1.12.0" 


$ flux create helmrelease istiod \
 --interval=1h \
 --release-name=istiod \
 --target-namespace=istio-system \
 --source=HelmRepository/istio \
 --chart=istiod \
 --chart-version="1.12.0" \
 --values=istiod-values.yaml

However, in a GitOps approach, they should be stored as code (use –export to export declaration). For example, the infrastructure kustomization keeps HelmReleases for installing Istio and PostgreSQL. This kustomization is referenced by a Flux Kustomization from higher level.

You can also manually reconcile one of the kustomizations (or other CRDs) with flux reconcile command, for example:

flux reconcile kustomization dependency

Since Nov 2021, FluxCD (0.20) supports reconciliation based on server-side apply. This increases performance and help address issues such as applying large config map, which is the equivalent of adding –server-side flag to the kubectl apply command.

Limitation

Troubleshooting the FluxCD repo can be involving and counter-intuitive. I had to commit a lot of changes to the repo because it serves as source of truth. Even though the commits can be made to a branch, it still involves a lot of code pushes. Traditionally I commit a change after testing. In the GitOps workflow, I commit a change then to test.

More flexible troubleshooting options are still to be desired. For example, there is no way to run one (FluxCD’s) Kustomization object at a time (and disable the rest), unless you remove their YAML files from the repo.

The next limitation is the ordering of resources in Kustomization. Arguably this is a limitation from Kustomize, instead of FluxCD’s. For example, I need the following manifest to be executed:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: selfsigned-issuer
spec:
 selfSigned: {}

The resource is a CRD that needs to be first installed using Helm chart (cert-manager). However there’s no way to control the sequence (Helm release executed first before declaration using CRD). Although there’s some workaround, it is not convenient. Alternatively, we can separate the resource creation and CRD creation into separate kustomization objects with dependency relationship, as suggested in this example.

Summary

Deployment in Kubernetes can get complicated with a lot of manifests. Helm, Kustomize and the like are means to handle the complexity due to numerous manifests. GitOps tools such as FlexCD brings these tools under a single framework, and more importantly, implements the idea of using Git repository as source of truth for continuous deployment.

From Ingress to CRD: why my solution needs Istio Gateways on Kubernetes platforms

Wed, 29 Dec 2021 22:50:16 -0400

Update: also read my other article here on the different generations of ingress technologies.

In my Korthweb project I was researching for the best ingress mechanism for HTTP and TCP workload, both of which need to be secured. I started with Kubernetes Ingress but eventually decided to go with Istio Gateway. This blog post is about the justification. In this essay, I will make the distinction between Ingress and Gateway and explain why a Kubernetes platform needs the latter going forward.

The word ingress can be used either to indicate ingress resource (in conjunction with ingress controller) in the context of Kubernetes cluster, or more generally to indicate the technology (e.g. provided by service mesh) to direct north-south traffic, originated from the outside into the workload running within the cluster. I will use both meanings of the word ingress throughout the article.

Background

In my previous post, I discussed Kubernetes Service object and native Ingress. In short, a Service object addresses the problem of exposing a workload (Pod or Deployment), operating at layer 3-4. A ClusterIP type of service exposes workload within the cluster only, and therefore is only used by internal services that are consumed by workload in the same cluster. A NodePort type of service exposes workload to outside of the cluster, using the node’s IP address and port range available on the node, a great step forward but still quite inconvenient and subject to the limitation of using Node’s IP and Port. A LoadBalancer type of Service brings separate IP address and port for running a service. To back up this type of service, the platform has to provide a load balancer with its own IP address and port range to manage. The implementation is platform specific e.g. Metal LB for Minikube, Azure Load Balancer for AKS, NLB for AWS EKS. In this type of service, Node Ports still exists but are not exposed to the outside world. Instead, they are only exposed to the backend of Load Balancer, whose front-end port is exposed to the outside world. In both NodePort and LoadBalancer types of Service, the kube-proxy process plays a role on each node to direct traffic into NodePort to target Pods.

Kubernetes Ingress, on the other hand, targets issues above layer 4, for example, path-based routing and is therefore mostly used for HTTP and HTTPS traffic. It is implemented by an Ingress Controller and now there has been a diverse ecosystem of those ingress controllers. In many implementations (e.g. Azure Application Gateway Ingress Controller, AGIC), the ingress themselves usually come with a load balancer managed by themselves. This eliminates the need for a separate LoadBalancer type of Service just for L4 capability. As a result, Ingress (with both L4 and L7 capabilities) is usually deployed along with ClusterIP type Service. Here is an example. The declaration of the Ingress in this case usually contains numerous lines of annotations in order to communicate the specification to the implementation.

Requirement

Using Kubernetes Ingress along with Service of ClusterIP type seems perfect to address the majority of use cases. However, it has its blind spots. My Korthweb project deals with DICOM traffic (a protocol on top of TCP) over TLS as well as HTTP traffic, which can be broken down as:

The ability to proxy TCP traffic over an arbitrary TCP port
The ability to terminate TLS/SSL encryption for TCP traffic

The Ingress documentation of Kubernetes clearly states that: an Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

The stance of community-driven Nginx ingress controller can be found in the documentation, which suggests the use of Service object for arbitrary TCP port. There is some unofficial claims of workaround available but I’m not confident.

I also went through a number of other Ingress providers but to my disappointment, the only product that supports it seems to be Traefik:

Product	Requirement #1	Requirement #2
F5 driven Nginx Ingress Controller	Supported	There have also been requests for TLS termination for TCP traffic but the request has not been closed as of yet.
HA Proxy Ingress controller	Supported	There’s no mention of this in documentation.
Kong’s Kubernetes Controller (with TCPIngress CRD)	Supported	Documentation claims support through SNI-based routing. However, the example demonstrates it using self-signed certificate only. BYO cert not supported according to this GitHub issue.
Traefik Lab’s IngressRoute (with IngressRouteTCP CRD)	Supported	Documentation seems to suggest that it is supported by IngressRoute.

My general impression is that Req #2 isn’t very popular so the providers either don’t support it or delaying the implementation. Even if they do, the CRD used for TCP ingress is different than HTTP’s. For example with Traefik Lab, the CRD for TCP is IngressRouteTCP, and for HTTP it is IngressRoute.

Introduction to Gateway

In addition to functional shortages, according to this blog post, there are signs of fragmentation into different but strikingly similar CRDs and overloaded annotations. In Kubecon 2019, a group of contributors discussed the evolution of Ingress into Gateway. Below are the two key slides stolen from their presentation:

This sums up how the concept of Gateway is different from Ingress. Gateway is an instantiation of a given LB. It works along with Route to achieve the functions brought by an Ingress. Gateway as a resource type of its own makes management easier at L4-L6 by a separate team. Routing at L7 is offloaded to “Route” resource type.

This conceptual evolution gives rise to the Gateway API open source project managed by the SIG-NETWORK community. Gateway API is a collection of resources that model service networking in Kubernetes, including GatewayClass, Gateway, HTTPRoute, TCPRoute, Service etc. The aim of this initiative is to evolve Kubernetes service networking through expressive, extensive and role-oriented interfaces that are implemented by many vendors and have broad industry support.

The diagram above stolen from Gateway API website illustrate the management model for each resource.

Kubernetes Gateway Implementations

The Gateway API is still a fairly young initiative and all the reference implementations are either work-in-progress or in early stages. In the last section, we learned that in the journey from Ingress to Gateway, the standardization initiative comes a bit behind the implementation efforts. For many supporters, the natural strategy is to continue with existing proprietary ingress controller implementation, and retrofit their technology to the emerging Gateway API along the way.

Because of that, we can see many market players with multiple flavours of implementations, typically one that evolves from their original product offering, with higher adoption rate and maturity level, and one that conforms to Gateway API specification.

For example, Traefik Labs has its Gateway API implementation in experimental stage. Its own implementation consists of (standard) Kubernetes Ingress and Kubernetes IngressRoute (based on custom resource with support of advanced features such as TCP route). Another example is the HAProxy Ingress, a community driven ingress controller implementation for HAProxy. Starting from its version 0.13, it partially supports the Gateway API’s v1alpha1 specification. Here is the conformance statement. Contour as a CNCF project for ingress controller has support for Gateway API at alpha version. Kong’s Kubernetes Ingress Controller follows the same path.

The Gateway functionality is also provided as part of Service Mesh product offering, and we can see some reference implementation by service mesh providers. Istio has its own Gateway implementation but tries to adapt to Kubernetes Gateway API. Hashicorp Consul also claims to be building support for Kubernetes Gateway API.

Until the Gateway API project matures, it is not recommended to use Gateway implementations that conforms to it, unless you intend to be their Guinea Pig. For my project, I chose Istio Gateway.

Istio Gateway implementation

As discussed, when the Gateway standard is still in its infancy, I chose a non-standard implementation of Gateway, even though it may include some CRDs. Out of the many Gateway implementations, I choose Istio mainly for two reasons.

First, I need a Gateway as part of service mesh because service mesh provides many other features that are needed in the platform. One of the goals of Service Mesh is to provide commonly used features (traffic management, observability, security, extensibility) in a commodity layer on top of Kubernetes. Gateway is not the only thing I need out of this commodity layer. With one install of Istio, many common platform-level problems are also addressed (e.g. mTLS, traceability, etc). I’ve compared three major service mesh technologies as below:

Linkderd	Istio	Consul
– lightweight – uses linkerd2-proxy – the original service mesh – no Gateway implementation (deal breaker in my use case)	– feature rich – uses Envoy proxy – complex but getting better	– initially a service discovery and distributed key-value store – lacks observability features but getting better – uses Envoy proxy

It is worth-noting that there are some initiatives to standardize service mesh (e.g. SMI) with reference implementation such as OpenServiceMesh. The standard is too weak to be considered important for now compared with maturity and stability. Istio has strong community support. The risk with Istio to acknowledge, is that it is not following an open-governance model (unlike many other CNCF projects), which could potentially causes vendor lock-in.

Second, The Istio Gateway features the separation between Gateway and Virtual Service CRDs. The former defines entry point and the latter defines routing rules. This design separates entry points from routing rules, enabling the flexibility of reusing the same Virtual Service for different gateways.

Istio uses Envoy proxy to manage traffic between Pods. Unlike the kube-proxy pattern, the Envoy proxies are side-car containers centrally managed by Istio Control Plane, which also enables other features such as traceability, as illustrated in the diagram below (stolen from this post).

In addition to Virtual Service, Istio Gateway also has the concept of Destination Rules. Virtual Service defines how to route traffic to different destinations. Destination rule defines how to split traffic to different subset at the routing destination. The concepts are documented on this page. For HTTP traffic, the relevant entities can be represented in the diagram below:

Here is an example of using Gateway and Virtual Service resources (from Korthweb sample project). Also note that in the Istio literature, there is neither a CRD name called “Ingress”, nor a resource type named “Ingress Gateway”, although they may be loosely used to refer to “Gateway resources configured to manage ingress traffic”.

Istio Gateway Installation

There are more than one ways to install Istio. In the past, Istio Operator was used to install Istio. The Operator calls IstioOperator API. Today the use of Istio Operator is not recommended anymore but the IstioOperator API is used implicitly by Istioctl installer. The two recommended approaches to install Istio today is by Istioctl and Helm.

With istioctl, you can specify an option imperatively, or using an overlay file as did in this lab. With Helm, istio has multiple charts, and requires multiple steps:

install CRDs using the base chart
install istiod using the istiod chart. Here is an example of values provided to Helm installer.
install ingress gateway using the gateway chart. Here is an example of values provided to Helm installer for ingress gateway. Note that egress gateway also requires the same helm chart with different value definition.

This instruction contains how to manually install Istio gateways using Helm, including installation of multiple Helm charts. I attempted to create a single chart to consolidate the multiple charts required for istio. It was not successful because of an error when trying to reference the same gateway chart dependency for multiple times.

Logging and Monitoring in Kubernetes with PLG stack

Wed, 13 Oct 2021 21:29:00 -0400

We’ve checked out the the actors in PLG stack (Promtail, Loki, Node Exporter, Prometheus, Grafana) and whipped up a quick pipeline on MacOS. Now I’m going a little further to implement the same PLG stack (Prometheus Loki and Grafana) in a Kubernetes cluster. This setup is for demo only, therefore no persistent storage is enabled.

Test Workload

I host a deployment of Flog with three pods running on Minikube. Flog is an open-source emulating log generation behaviour of an application. On the Minikube cluster we start the deployment as below:

minikube start --driver=hyperkit --container-runtime=containerd --memory=12288 --cpus=2
kubectl create ns obsv
kubectl -n obsv create deployment flog --image=mingrammer/flog --replicas=3 -- flog -f rfc3164 -l -d 300ms
kubectl -n obsv get po

The Pods will come up in a heartbeat.

I will use helm to install the objects required for logging and metrics pipelines. There are multiple Helm charts for each components. Some high-level charts (usually with a name suffix of -stack) contain several other resource as sub-charts. They are created as one-stop-shop for multiple components but I found none of them serve my exact purpose. For example, both loki-stack and kube-prometheus-stack include Grafana. But I only need one instance of Grafana. Therefore I stick to the low-level charts.

Log Shipping

Add a helm repo, and install loki and promtail. Note that we need to specify correct loki address when installing Promtail.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --namespace obsv --install loki grafana/loki
helm upgrade --namespace obsv --install promtail grafana/promtail --set "config.lokiAddress=http://loki.obsv.svc.cluster.local:3100/loki/api/v1/push"

When installing Promtail, a DaemonSet is created on the Node. The default configuration applies appropriate configuration and tagging strategy for Kubernetes Pod and Node. So the only customization I specified is Loki address. We can then check logging with Loki. To do so, first expose port 3100 to host, and then use logcli (e.g. on MacOS) to query for logs:

kubectl -n obsv port-forward service/loki 3100:3100
logcli labels
logcli labels pod
logcli query '{pod="flog-775d5fc5c8-p4rlx"}'

Log lines should be pumped to Loki a minute after Loki comes up. The logcli labels command should display the tags. The logcli query command should return the log lines.

Metrics

I use Premetheus with node exporter. In its architecture, Prometheus contain the server, the pushgateway, and alertmanager. The helm chart for Prometheus contains all of those components. It also has a dependency repo for kube-state-metrics. To install:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm --namespace=obsv install prometheus prometheus-community/prometheus
kubectl -n obsv get svc

Once installed, a DaemonSet for Prometheus Node Exporter is created. The exporter is already configured by default for Kubernetes monitoring. The Prometheus server is also configured, on port 80 by default. it needs to be forwarded in order to access from Browser:

kubectl --namespace obsv port-forward service/prometheus-server 9100:80

To verify installation of Prometheus, browse to localhost:9100 to examine the metrics.

Visualization

Last but not least, I will configure Grafana. The repo has been added already so we’ll just install the chart:

helm upgrade --namespace obsv --install grafana grafana/grafana
kubectl get secret --namespace obsv grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

The second command retrieves the credential. To access the web portal, we need port forwarding again:

kubectl port-forward --namespace obsv service/grafana 3000:80

To verify installation, browse to http://localhost:3000 and log in as user admin with the password above. Then add two data sources:

Type: Prometheus, URL: http://prometheus-server.obsv.svc.cluster.local:80
Type: Loki, URL: http://loki.obsv.svc.cluster.local:3100

Then we can explore data using both data sources.

PLG stack

Summary

In the last two posts I reviewed the setups for PLG stack in Kubernetes, from a regular environment to k8s cluster. Fluentd, Prometheus are both CNCF projects. The PLG stack seems to be more adopted than EFK but both have their own advantages. Welcome to the PLG vs EFK debate.

Local multi-node cluster – Minikube, MicroK8s and KinD

Tue, 14 Sep 2021 11:18:00 -0400

In this post we compare Minikube, MicroK8s and KinD as different approaches to build multi-node cluster locally.

Is Docker desktop bad?

In the previous post about docker desktop as a single-node Kubernetes cluster setup, I touched on the deprecation of docker-shim. Now that CRI beats OCI as the standard for container runtime, the docker runtime will no longer be supported by Kubernetes. Also deprecated is docker-shim, the temporary interface that had make Docker runtime work in Kubernetes. This was announced in December 2020, and is coming through in Kubernetes 1.23, expected Oct 2021. However, docker desktop still uses docker runtime in it’s single-node Kubernetes cluster. This essentially renders itself a non-compliant Kubernetes environment.

Docker desktop still has great value for application developers. If your role is development, spending a lot of time coding business logics and need an easy-to-use container runtime on your laptop, Docker desktop is a good choice. The recent moves by the company seems to suggest that this is the business they are targeting now. On the other hand, if your roles are deployment, automation, orchestration, cloud native etc and you are looking for a playground, most likely you do need a runtime compliant to Kubernetes CRI. Docker desktop is not a CNCF-certified project anymore, and it is not your choice.

Alternatives

There are a number of alternatives, the most well-known ones are Minikube, MicroK8s, KinD and K3s with K3d. This presentation from CNCF in 2020 covers a lot of details about these technologies. I’ll try to add my opinion.

K3s is Rancher Lab’s lightweight Kubernetes distribution that supports multi-node cluster as well as different runtimes (e.g. containerd). It is not straightforward to setup, and k3d is an command-line wrapper to make it easy to install K3s cluster. K3s was accepted as a CNCF project but only at Sandbox maturity level, so it is not my choice.

The other three: Minikue, MicroK8s and KinD are all certified CNCF project. I will further discuss how to choose among them. These projects are technologies that takes different approach to address the challenges with deploying multiple nodes in local environment (e.g. my laptop).

The challenge with running a Kubernetes cluster with multiple nodes locally is how to manage these nodes. They are separate virtual resources that need to be isolated from computing perspective, and connected as a cluster. This is typically the use case of a Type II hypervisor, or alternatively, it can also be implemented with container technology. This layer of technology (referred to as drivers) makes a big difference.

Minikube

Minikube supports multiple drivers. Depending on your platform (Windows, Linux, or MacOS), the preferred driver is different. Refer to the document here for preferred driver, and this blog post for more instructions. In addition to the documents, here some notes from my personal experience:

On MacOS, Minikube lists Docker as preferred driver. I disagree with that. If you have no other reason to install Docker, then I would recommend hyperkit as the the preferred driver. Hyperkit can be installed with a simple Homebrew command. For two reasons I do not recommend Docker as the driver of Minikube. First, it requires a separate installation of Docker Desktop, which includes a built-in instance of hyperkit on its own. This isn’t neat. Second, I often need Metal LB add-on with Minikube for testing Kubernetes Ingress. With Minikube on Docker, the Ingress ports are not exposed to MacOS’s. Therefore you cannot directly visit websites spun up on Minikube. This is a known issue for a while due to limitation on docker bridge with Mac. Some reported an ugly workaround with docker-mac-net-connect but I never got it to work.
On Windows native environment, the preferred driver is hyper-V. The Minikube cli command have to run from Windows PowerShell.
On WSL2, Minikube doesn’t play well, regardless of driver. The hyperkit driver won’t work (it is designed for MacOS only). The kvm2 driver would require a KVM2 hypervisor. However, WSL2 itself is a VM on top of hypervisor, as explained here. If KVM2 driver works it would require nested virtualization so I doubt it will ever be supported. As for Docker on WSL2 as driver, Minikube has it as an experimental feature, and requires configuring cgroup to allow setting memory. I am not confident with it.

To me, Minikube is the tool for MacOS (I have Intel processor). On MacOS, we first need to install minikube and hyperkit with home brew.

We can then start a kubernetes cluster, with minikube in a single command. I noticed a process on my MacBook called dnscrypt-proxy that conflicts with hyperkit DNS server when starting minikube. I had to remove dnscrypt-proxy (part of Cisco Umbrella Roaming Client) in order to get minikube to work, as this thread suggests. You can find out by running:

sudo lsof -i :53

If dnscrypt-proxy is running, find out the application by PID and remove the application. Otherwise there will be issues. Check out this section on the document. The commands that I use to start multi-node cluster is:

minikube start --driver=hyperkit --container-runtime=containerd --memory=12288 --cpus=2 --disk-size=150g --nodes 3
kubectl get po -A
kubectl describe node minikube|grep Runtime

Node administration is simple. To enable dashboard, simply run “minikube dashboard”. To SSH to a node, simply do “minikube ssh -n <node_name>”. In order to stop the node and delete cluster, run “minikube stop && minikube delete”.

There are some addons in minikube, for example, efk, gvisor, istio, metrics-server. To list add-ons, and enable metrics-server, for example, run:

minikube addons list
minikube addons enable metrics-server

When creating cluster, instead of specifying the cluster imperatively, the configuration (e.g. driver, container runtime, cpu, memory, number of nodes, etc) can be stored as a profile with -p switch. Like other Minikube configuration information, Minikube profiles are stored in ~/.minikube under the profile directory.

Minikube also has a page that benchmarks the performance of these technologies, where it presents itself as the most performant.

Minikube, KinD, k3d and microK8s performance

MicroK8s

MicroK8s is developed by Canonical. It can use either Multipass or LXD container as driver. Multipass can configure Ubuntu VMs using cloud-init. It supports multiple hypervisor backends as well but hyperkit is the default on MacOS, Hyper-V on Windows, and KVM on Linux.

MicroK8s supports multi-node configuration across multiple machines. That is, nodes can span across multiple physical machines. This is more powerful than Minikube where multiple nodes are on the same physical machine. It brings MicroK8s additional use cases such as edge and IoT devices.

With that capability comes the extra step to configure a MicroK8s cluster. You will need to manually join a node to a cluster because the new node is potentially located on a different machine, and you execute the command from the new machine. On the other hand, with Minikube you simply specify the number of nodes desired in a command or profile.

Snap is the native package manager to install MicroK8s, making GNU Linux (e.g. Ubuntu) the native platform. It also supports MacOS and Windows. MicroK8s does not rely on Docker (unlike KinD and Minikube with Docker as driver), and uses containerd as runtime.

MicroK8s comes with its own packaged version of kubectl, and you use that with “microk8s kubectl” command, which is not convenient. You can configure your host kubectl to point to the MicroK8s cluster, as an extra step.

Compared to the other two technologies, MicroK8s is more powerful in the sense that the cluster is build on nodes across multiple machines. However, it takes more step to configure even for a multi-node, single-machine environment. Refer to this post for the steps.

KinD

KinD is similar to Minikube with Docker as driver. It is more restricted than Minikube considering Docker is the only driver it supports. This makes it a requirement to have Docker installed locally.

Although KinD uses Docker to run nodes, it does not use Docker as its container runtime. Therefore it remains as compliant environment.

Another advantage of KinD is it supports Docker on WSL2 very well. Simply install KinD on WSL2 and start Docker. This blog post contains the steps required to install KinD vs Minikube on WSL2. There is a comparison table in the conclusion section that highlights the fact that it is much easier to install KinD with WSL2 than to install Minikube.

However, there are currently some known limitations with Docker desktop for Windows (including on WSL2). One is the absence of docker0 bridge. This means on Windows you cannot route traffic to the containers.

For cluster specification, KinD can configure a cluster declaratively using YAML file for example, the kind-config.yaml contains the following snippet:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
 - role: control-plane
 - role: worker
 - role: worker
 - role: worker
networking:
 disableDefaultCNI: true

We can bring up a cluster with a command:

kind create cluster --config=kind-config.yaml

The command will also configure the kubectl context so we can check node with kubectl command. The file is in my real-quicK-cluster repo.

Conclusion

After reviewing the technologies that back up multi-node kubernetes cluster for my role, I find that Minikube with hyperkit is my favourite for MacOS. On WSL2, I prefer to use KinD. Since I do not use Windows native environment or Ubuntu on my laptop, I cannot make recommendations. However I would start with Minikube (with hypverv or kvm2 as driver).

Update July 2022: When the test workload involves persistent storage, KinD is a better choice. When the test workload involves load balancer. Minikube is a better choice.

As to storage provisioner, Minikube with storage-provisioner addon uses k8s.io/minikube-hostpath. KinD uses rancher.io/local-path. When I have to test workload with persistent storage (e.g. PostgreSQL with Crunchy pgo), I realized Minikube have permission issues with persistent volume, as discussed here as an issue with multiple nodes. The issue has been open since Aug 2021.

For Load Balancer, Minikube has metallb as an addon and I can configure it within a bash script conveniently. With KinD, I’d have to configure that in a few steps with both kubectl and Docker CLI commands and I was not able to connect to the load balancer by IP even after following the steps. So I tend to just use Minikube to test workload requiring load balancer and service mesh.

I find myself switch between Minikube and KinD on my MacBook depending on the test workload.

Log Shipping in Kubernetes with EFK stack

Sat, 04 Sep 2021 21:50:00 -0400

I first worked on log shipping with ELK stack three years ago. In the context of Kubernetes cluster, log shipping has similar challenges. In this post I will discuss the set up of log shipping with Kubernetes cluster using EFK stack

Logging Architecture

As discussed, if the Kubernetes cluster has a runtime in compliant with CRI (e.g. containerd), then the stdout and stderr of the Pod is stored on the node, in the location /var/log/containers/.

When creating log shipping solution, it is important to use a compliant cluster (e.g. minikube) to ensure what you develop will work across environments.

The Kubernetes document has a section on logging architecture which is a good start point. It outlines several different patterns. Logging at the node level is turned on by default and does not require special configuration, as explained in the section above. The EFK pattern is close to the diagram under using a node logging agent for cluster-level logging.

Node logging agent

The diagram above is stolen from Kubernetes documentation. In EFK stack, the agent is a daemonset running fluentd Pod.

EFK stack

While we can use ELK (Elasticsearch, Logstash, Kibana) stack for log shipping, EFK (Elasticsearch, Fluentd, Kibana) is generally recommended in Kubernetes cluster. We compare the two in the following table:

	ELK	EFK
Development language	All in Java. Logstash and Filebeat both require JVM. Managed as an open-source project by Elastic company	Fluentd in Ruby and does not require JVM to run. Fluentd is a CNCF project built to integrate with Kubernetes.
Typical pattern	Filebeat acts as a lightweight collector to monitor the source log. Logstash as aggregator to receive from filebeat, and push to Elasticsearch	The fluentd Pod can be configured to serve as forwarder and aggregator based on configuration. fluentd-forwarder is deployed as daemonset on node, and ship the result to fluentd-aggregator, which may run in a separate cluster. The fluentd-aggregator pushes processed results to Elasticsearch.
Metrics	use metric beat for data collection	scrape metrics from prometheus server
Event Routing	tag-based	if-then statement

ELK and EFK stacks comparison

The main problems in log shipping are:

column mapping : identify column patterns in each log line and map them to appropriate column in Elasticsearch.
multi-line processing: identify when a logging entry spread across multiple lines and process accordingly.

Suppose we want to congregate the logs from stdout and stderr of PostgreSQL pods. The raw output in /var/log/container on the node, looks like this:

2021-08-28T15:27:46.75370563Z stdout F server stopped
2021-08-28T15:27:46.757173069Z stderr F postgresql-repmgr 15:27:46.75 INFO ==> Starting PostgreSQL in background...
2021-08-28T15:27:46.883126928Z stderr F postgresql-repmgr 15:27:46.88 INFO ==> Registering Primary...
2021-08-28T15:27:47.017164653Z stderr F postgresql-repmgr 15:27:47.01 INFO ==> Loading custom scripts...
2021-08-28T15:27:47.023334611Z stderr F postgresql-repmgr 15:27:47.02 INFO ==> Loading user's custom files from /docker-entrypoint-initdb.d ...
2021-08-28T15:27:47.026169813Z stderr F postgresql-repmgr 15:27:47.02 INFO ==> Starting PostgreSQL in background...
2021-08-28T15:27:47.343607487Z stderr F postgresql-repmgr 15:27:47.34 INFO ==> Stopping PostgreSQL...
2021-08-28T15:27:47.448111425Z stdout F waiting for server to shut down.... done
2021-08-28T15:27:47.448172479Z stdout F server stopped
2021-08-28T15:27:47.453722807Z stderr F postgresql-repmgr 15:27:47.45 INFO ==> ** PostgreSQL with Replication Manager setup finished! **
2021-08-28T15:27:47.453829953Z stdout F
2021-08-28T15:27:47.503516746Z stderr F postgresql-repmgr 15:27:47.50 INFO ==> Starting PostgreSQL in background...
2021-08-28T15:27:47.532558987Z stdout F waiting for server to start....2021-08-28 15:27:47.532 GMT [273] LOG: pgaudit extension initialized
2021-08-28T15:27:47.533307459Z stdout F 2021-08-28 15:27:47.533 GMT [273] LOG: listening on IPv4 address "0.0.0.0", port 5432
2021-08-28T15:27:47.533466407Z stdout F 2021-08-28 15:27:47.533 GMT [273] LOG: listening on IPv6 address "::", port 5432
2021-08-28T15:27:47.537987947Z stdout F 2021-08-28 15:27:47.537 GMT [273] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-08-28T15:27:47.547956465Z stdout F 2021-08-28 15:27:47.547 GMT [273] LOG: redirecting log output to logging collector process
2021-08-28T15:27:47.548005463Z stdout F 2021-08-28 15:27:47.547 GMT [273] HINT: Future log output will appear in directory "/opt/bitnami/postgresql/logs".
2021-08-28T15:27:47.551741571Z stdout F 2021-08-28 15:27:47.551 GMT [275] LOG: database system was shut down at 2021-08-28 15:27:47 GMT
2021-08-28T15:27:47.558012894Z stdout F 2021-08-28 15:27:47.557 GMT [273] LOG: database system is ready to accept connections
2021-08-28T15:27:47.618577092Z stdout F done
2021-08-28T15:27:47.618708978Z stdout F server started
2021-08-28T15:27:47.630065958Z stderr F postgresql-repmgr 15:27:47.62 INFO ==> ** Starting repmgrd **
2021-08-28T15:27:47.638116348Z stderr F [2021-08-28 15:27:47] [NOTICE] repmgrd (repmgrd 5.2.1) starting up
2021-08-28T15:27:47.65317627Z stderr F INFO: set_repmgrd_pid(): provided pidfile is /opt/bitnami/repmgr/tmp/repmgr.pid
2021-08-28T15:27:47.653232015Z stderr F [2021-08-28 15:27:47] [NOTICE] starting monitoring of node "orthweb-postgresql-ha-postgresql-0" (ID: 1000)
2021-08-28T15:27:47.681683703Z stderr F [2021-08-28 15:27:47] [NOTICE] monitoring cluster primary "orthweb-postgresql-ha-postgresql-0" (ID: 1000)
2021-08-28T15:28:11.742865958Z stderr F [2021-08-28 15:28:11] [NOTICE] new standby "orthweb-postgresql-ha-postgresql-1" (ID: 1001) has connected

From this snippet of log, we can see each line in stdout or stderr is appended with a timestamp. There are multi-line log entries but each is still appended with a timestamp. This is just how kubernetes keeps the log file for Pod stdout and stderr. To handle that, we need to first take out the real log line, and then process multi-line.

We will go over the installation of EFK stack and the mechanism to address the two challenges above.

Install Elasticsearch and Kibana

To install Elasticsearch, we use the helm chart provided by the official repository:

helm repo add elastic https://helm.elastic.co

If we run multiple pods on the same hosts, then we need some customized values in order to get the installation to work. The values.yaml file looks like this:

---

antiAffinity: "soft"

esJavaOpts: "-Xmx128m -Xms128m"

# Allocate smaller chunks of memory per pod.
resources:
 requests:
 cpu: "100m"
 memory: "512M"
 limits:
 cpu: "1000m"
 memory: "512M"

Then we can “preview” what values are used for installation, with helm’s template command:

helm template elasticsearch elastic/elasticsearch -f values.yaml

The effect of the antiAffinity property allows multiple Elasticsearch Pod to be scheduled on the same node. This is not required in production with multiple nodes. To install Elasticsearch and Kibana, run:

helm install elasticsearch elastic/elasticsearch -f values.yaml
helm install kibana elastic/kibana

The Kibana service is exposed on port 5601 of the cluster. To access the port on cluster, we need to run port-forward command as below:

kubectl port-forward deployment/kibana-kibana 5601

Install Fluentd

There are different ways to configure Fluentd. For example, in the forwarder-aggregator pattern, a forwarder Pod is a Daemonset on each Kubernetes node. The forwarder pushes to the aggregator, at port 24224. This page has some configuration details. This pattern is similar to filebeat -> logstash pattern in ELK stack.

In our case, we use a simplified pattern, with a fluentd daemonset acting as collector and then forward to Elasticsearch. Similarly, in ELK stack we can use filebeat to push to Elasticsearch without Logstash. The only reason is the entire architecture isn’t as complicated as requiring an aggregator.

We need to create configmap as below:

kind: ConfigMap
apiVersion: v1
metadata:
 name: fluentd-cm
 namespace: default
 labels:
 app.kubernetes.io/component: forwarder
 app.kubernetes.io/instance: fluentd
 app.kubernetes.io/managed-by: Helm
 app.kubernetes.io/name: fluentd
 helm.sh/chart: fluentd-1.3.0
 annotations:
 meta.helm.sh/release-name: fluentd
 meta.helm.sh/release-namespace: default
data:
 fluentd.conf: |
 # Ignore fluentd own events
 <match fluent.**>
 @type null
 </match>

 # HTTP input for the liveness and readiness probes
 <source>
 @type http
 port 9880
 </source>

 # Throw the healthcheck to the standard output instead of forwarding it
 <match fluentd.healthcheck>
 @type null
 </match>

 # Get the logs from the containers running in the node
 <source>
 @type tail
 read_from_head true
 tag kubernetes.*
 path /var/log/containers/orthweb-postgresql-ha-postgresql-**.log
 pos_file /opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos
 <parse>
 @type regexp
 expression ^(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.[^Z]*Z)\s(?<stream>[^\s]+)\s(?<character>[^\s])\s(?<message>.*)$
 </parse>
 </source>
 # enrich with kubernetes metadata
 <filter kubernetes.**>
 @type kubernetes_metadata
 @id filter_kube_metadata
 kubernetes_url "#{ENV['FLUENT_FILTER_KUBERNETES_URL'] || 'https://' + ENV.fetch('KUBERNETES_SERVICE_HOST') + ':' + ENV.fetch('KUBERNETES_SERVICE_PORT') + '/api'}"
 verify_ssl "#{ENV['KUBERNETES_VERIFY_SSL'] || true}"
 ca_file "#{ENV['KUBERNETES_CA_FILE']}"
 skip_labels "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_LABELS'] || 'false'}"
 skip_container_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_CONTAINER_METADATA'] || 'false'}"
 skip_master_url "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_MASTER_URL'] || 'false'}"
 skip_namespace_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_NAMESPACE_METADATA'] || 'false'}"
 </filter>


 <match kubernetes.var.log.containers.orthweb-postgresql-ha-postgresql-**.log>
 @type elasticsearch
 include_tag_key true
 host "elasticsearch-master.default.svc.cluster.local"
 port "9200"
 index_name "postgresql-logs"
 <buffer>
 @type file
 path /opt/bitnami/fluentd/logs/buffers/orthanc-logs.buffer
 flush_thread_count 2
 flush_interval 5s
 </buffer>
 </match>

Then we can create the resource, with helm chart pointing to the config map:

helm install fluentd bitnami/fluentd --set aggregator.enabled=false --set forwarder.configMap=fluentd-cm

We can validate the index creation on Elasticsearch:

kubectl port-forward service/elasticsearch-master 9200
curl -XGET localhost:9200/_cat/indices

From Kibana, we can forward the port as instructed above, and browse to the UI. Once logged on to Kibana, we need to add index pattern first before being able to view the content of index.

How about Fluent Bit

Fluentd has an even more lightweight brother Fluent Bit, also a CNCF project, designed by the same team, for environments with tighter resource restrictions. The technical differences are outlined on this page outlines the technical differences. In terms of use case, Fluentd is a log aggregator and Fluent Bit is simply a forwarder. In terms of ecosystem, Fluentd has a stronger ecosystem whereas Fluent Bit is more seen in IoT devices. Read this post for more comparison.

Summary

EFK stack (Elasticsearch, Fluentd and Kibana) on Kubernetes is a natural choice for ELK users. Fluentd is a CNCF project created for integration with Kubernetes. It is good alternative to enterprise solution such as Splunk. There are lots of plugins supported and articles on configuration. While developing a solution I had to spend time reading the input plugin documentations.

Creating X.509 TLS certificate for workload on Kubernetes

Sun, 29 Aug 2021 23:19:00 -0400

In deployment automation, I often had to create self-signed X.509 certificate for testing TLS traffic into Kubernetes. Sometimes self-signed, sometimes signed by a CA. This post summarized the approaches I’ve taken.

Create self-signed certificate with OpenSSL

Traditionally, this is done in three OpenSSL commands:

openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=Health Certificate Authority'
openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=*.orthweb.com'
openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt

I have an older post to cover the basics of cryptography in TLS certificate and PKI. In the three commands above, the first produces a private key and self-signed certificate for a CA. The second creates a private key and a CSR for the web site. The third one uses the CA’s signing private key to sign the CSR from the website. The output is the certificate for the website.

Workloads running in Kubernetes typically consume certificates stored in Kubernetes Secret. The cons of this approach is that it usually requires an extra step to import the certificate files into Kubernetes Secret. For example:

kubectl create -n orthweb secret generic orthweb-cred --from-file=tls.key=server.key --from-file=tls.crt=server.crt --from-file=ca.crt=ca.crt

Note, people use the term self-signed certificate loosely. It sometimes means literally a certificate that is self-signed, like the one generated above. Sometimes, I had to self-sign a CA, then use the CA to sign one certificate for the server and one for the client. Because the application being tested requires that the client and server’s certificates both under the same CA. This would involve a few more commands, for example:

# Self-sign a CA
openssl req -x509 -sha256 -newkey rsa:4096 -days 365 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=issuer.digihunch.com/emailAddress=info@www.digihunch.com -keyout /tmp/ca.key -out /tmp/ca.crt

# Generate a CSR for server
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=server.digihunch.com/emailAddress=orthweb@www.digihunch.com -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:server2.digihunch.com -keyout /tmp/server.key -out /tmp/server.csr

# Use the self-signed CA to issue a certificate to the server
openssl x509 -req -sha256 -days 3650 -in /tmp/server.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/server.crt -extfile <(echo subjectAltName=DNS:orthweb.digihunch.com,DNS:server2.digihunch.com)

# Generate a CSR for clietn
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=client.digihunch.com/emailAddress=client@www.digihunch.com -keyout /tmp/client.key -out /tmp/client.csr

# Use the self-signed CA to issue a certificate to the client
openssl x509 -req -sha256 -days 365 -in /tmp/client.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/client.crt

In the example above, it is important to note that even though the server’s CSR contains subject alternative name (SAN), I still have to specify the SAN again when signing the certificate for the server.

Similar to OpenSSL there are other toolkits such as CFSSL that supports specifying configuration files. However, the steps in Shell command are generally not always easy to automate.

Create self-signed certificate with Helm

Moving to the context of workload deployment in Kubernetes, running openSSL command isn’t always a viable option. For example, generating a certificate in the middle of deployment using a Helm Chart.

In Helm, template functions is for this purpose. In my Korthweb project I used genSignedCert to create self-signed certificate and then store the key, certificate and CA certificate as Kubernetes Secret:

{{- $dbtlscert := genSignedCert .Values.dbtls.certCommonName nil (list .Values.dbtls.certCommonName) 365 $ca }}
apiVersion: v1
kind: Secret
metadata:
 name: {{ .Values.dbtls.certCommonName | quote }}
 namespace: {{ $.Release.Namespace | quote }}
type: kubernetes.io/tls
data:
 tls.crt: {{ $dbtlscert.Cert | b64enc | quote }}
 tls.key: {{ $dbtlscert.Key | b64enc | quote }}
 ca.crt: {{ $ca.Cert | b64enc | quote }}
{{- end }}

The cons of this approach is that the syntax is not straightforward. As indicated in Helm documentation: Helm Chart templates are written in the Go template language, with the addition of 50 or so add-on template functions from the Sprig library and a few other specialized functions. While we talk about the “Helm template language” as if it is Helm-specific, it is actually a combination of the Go template language, some extra functions, and a variety of wrappers to expose certain objects to the templates. Many resources on Go templates may be helpful as you learn about templating.

Create self-signed certificate with Cert-Manager

The Cert Manager project is very popular to produce X.509 certificates directly in Kubernetes secret. We can install cert manager using Helm:

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.0.3 --set installCRDs=true
kubectl get pods -n cert-manager
kubectl get crd | grep cert-manager.io

Alternatively, FluxCD’s documentation on Kustomization dependency uses Cert Manager as an example. It is a good way of installing cert-manager if you have GitOps pattern.

Creating self-signed certificate for website is fairly simple. It starts with bootstrapping a CA issuer. Take the manifest below as an example. When creating the first certificate, make sure to specify isCA=true, so it stores the signing private key along with its own certificate in the ca-secret. Then use the newly created CA as issuer to create the X.509 certificate for the website.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: selfsigned-issuer
spec:
 selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: my-ca
 namespace: orthweb
spec:
 isCA: true
 commonName: my-ca
 secretName: ca-secret
 privateKey:
 algorithm: ECDSA
 size: 256
 issuerRef:
 name: selfsigned-issuer
 kind: ClusterIssuer
 group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: my-ca-issuer
 namespace: orthweb
spec:
 ca:
 secretName: ca-secret
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: orthweb-cert
 namespace: orthweb
spec:
 commonName: orthweb.com
 secretName: orthweb-secret
 duration: 2160h
 renewBefore: 72h
 subject:
 organizations:
 - digihunch
 dnsNames:
 - web.orthweb.com
 - dcm.orthweb.com
 privateKey:
 algorithm: ECDSA
 size: 256
 issuerRef:
 name: my-ca-issuer
 kind: Issuer
 group: cert-manager.io

The site certificate is directly stored in Kubernetes Secret as specified in the secretName field. To fetch the certificate text, we need to decode the secret entry, for example:

kubectl -n orthweb get secret orthweb-secret -o jsonpath='{.data.ca\.crt}' | base64 -d

Note that the example above uses ECDSA algorithm with size 256 for private key and certificate. It requires that the TLS client to support ECDSA algorithm as well. For more supportability, you can use RSA algorithm (2048 or 4096 size).

In addition to creating self-signed certificate, Cert Manager supports a number of other issuer types. For example, the support of ACME issuer type enables integration with Let’s Encrypt.

Cert Manager can secure Kubernetes Ingress resources with a sub-component called ingress-shim. It is configured via annotation on the Ingress resource.

Cert Manager

Create CA-signed certificate manually

For a certificate signed by a CA, there are may paid options, from manual, to self-help, to automated. The classic manual way is using OpenSSL, generating key, CSR. The CA takes CSR to sign a X.509 certificate returned to the website administration.

Many CA websites charges for a fee and makes it easy. For example, this site currently uses certificate from SSLs.com. Apart from the fee-for-cert option, there is a website named “SSL for free“, a CA with free option for 90-day single-domain, non-wildcard certificate and we can request it simply on their website, with proof of domain ownership.

The other popular free option is Let’s Encrypt, which also employs ACME protocol. The protocol requires ACME challenges to be satisfied in order to proof domain ownership. There are a few types of challenges:

HTTP-01 challenge
DNS-01 challenge
TLS-SNI-01 challenge
TLS-ALPN-01 challenge

I have used the HTTP-01 and DNS-01 challenges. The DNS-01 challenge requires adding TXT records to DNS configuration. The HTTP-01 challenge requires adding a DNS A-record to resolve to the server, then two URIs with pre-defined value.

When I first set up this site I used certbot (the client program for letsencrypt) to create certificate every 90 days from the wordpress server, following this guide, including solving DNS-01 challenges.

Create CA-signed certificate automatically with cert manager and letsencrypt

With Kubernetes, cert-manager has the ability to integrate with let’s encrypt for full automation. Here is a good blog post on this. Domain verification is still required but it can be done automatically. We first need to register an A record that resolves host name to the Ingress IP to enable this automation. The domain ownership validation may use the ACME protocol. This should also work on private networks with private DNS and ACME protocol using a private boulder server.

Take domain name demo1.digihunch.com for example, if ingress exposes a public IP address which the domain name resolves to, then we can configure certificate with the following manifest:

kind: IngressClass
metadata:
 name: istio
spec:
 controller: istio.io/ingress-controller
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt
spec:
 acme:
 privateKeySecretRef:
 name: letsencrypt
 server: https://acme-staging-v02.api.letsencrypt.org/directory
 solvers:
 - http01:
 ingress:
 class: istio
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: demo
spec:
 dnsNames:
 - demo1.digihunch.com
 issuerRef:
 kind: ClusterIssuer
 name: letsencrypt
 secretName: demo-tls

This example uses Istio as ingress controller but the method works regardless of the controller technology behind Ingress. In the ClusterIssuer object, we’re telling it to use the staging server from letsencrypt. We also specify http01 as challenge type, and that the ingress type is istio. In the Certificate object, we provided dnsName and specified ClusterIssuser. We also tell it to store the credentials to a secret named demo-tls.

When we apply the resources above, the ClusterIssuer connects to letsencrypt server via ACME protocol. Since the DNS name already resolves to the Public IP that the ingress is hosting, the ClusterIssuer configures the required Ingress, Services and Pods accordingly so the token to satisfy the challenge is presented at the designated URI. Instead of a staging server, we can also use production ACME server for production deployment. Note that the production ACME endpoint has a stricter rate limit.

When the ACME validation is in progress, it is important to ensure that port 80 is open and there is no other mechanism (such as routing rule, authorization requirement, mandatory redirect to 443) that blocks access from letsencrypt server.

Bottom line

Cert Manager is deployed in Kubernetes, supporting a variety of issuer types. As a Kubernetes-native tool, it is a no-brainer for Kubernetes workload for X.509 certificate. Compared with using template function in Helm, it is not dependent on template function and the syntax is consistent (YAML). Compared with OpenSSL or other binary tools, it is easy to integrate with the platform.

Helm – Configuration Management for Kubernetes Resources

Mon, 26 Jul 2021 19:28:22 -0400

Developer ships application in Docker container, so it can eventually hosted in Kubernetes cluster. However, there are still some installation steps, before the application can operate online in production. In this post, we use the container image of Orthanc application as a starting point. We first build services in Kubernetes to go through these steps. Then, to automate the steps, we build a helm chart. The code is kept in Korthweb project, in which the manual directory has the files requirement for manual deployment, and the helm directory is the helm chart.

Manual Deployment

The manual deployment steps include different kinds of activities, such as creating X.509 certificates, apply config map, create Kubernetes deployment using the YAML declarations, and use helm to install dependency. The steps need to take place in a particular sequence. Some step requires pulling information from secrets created in the previous step. This is why the deployment is not portable.

In order to automate the steps, one might think of wrapper script, which is very limited. A configuration management tool is needed in this scenario. Two common options are Kustomize, and Helm. Kustomize is a native tool which can be run by kubectl. It is also driven by declarative statement in YAML, which is simple to grasp. However, in lack of a templating mechanism, Kustomize may require wordy statements. Helm, on the other hand, comes with a templating mechanism which greatly increase reusability, making it more suitable for complex steps required in installation.

Helm Repo and Chart

Helm is known as package manager for applications running on Kubernetes. Helm defines an application as a collection of related Kubernetes resources, and it manages application deployment through a templated approach. An installation workbook is called a chart. Charts are kept in repositories. There are some well-known repositories, such as Bitnami, Helm stable. You need to add a repostory before using the Helm Charts in it. To add a repo, run:

helm repo add bitnami https://charts.bitnami.com/bitnami

You can host your own repo (public or private) as well. To search for charts across repositories, the best place is artifact hub, which indexes charts from a lot of public repositories. To search for charts from the repositories added, run:

helm search repo postgres

Template is the soul of Helm chart. A Helm chart consists of a directory of files following specific pattern so Helm can understand how to deploy the application. For example, the chart name is the name of the working directory. Under the directory, the values.yaml and chart.yaml defines variables and constants, both serving as template inputs. The template directory is the most important part of the directory where the installation logics are defined. Helm runs the entire directory hierarchy (except for paths specified in .helmignore file) through a Go template rendering engine. The template result spec out the detailed steps.

A great example of using Helm chart to simplify installation is the wordpress chart by Bitnami. You can install all the required components in a single command:

helm install my-release bitnami/wordpress

The helm chart in Korthweb project is also an evolving helm chart I created for installing Orthanc application.

Helm V3 (released in late 2019) includes an important architectural change – the removal of tiller. This means Helm can operate on the client-side – a significant simplification. Helm graduated from CNCF project in 2020. There are also a few changes in V3, as outlined here, including the consolidation of requirements.yaml into Charts.yaml.

Template and Function

As discussed, templating is the key towards reusability and flexibility in configuration management. We’ve worked with Jinja2 template engine in Ansible and Python. Here in Helm, we use Go templates with some enhancement. The syntax is mostly based on Go template, which is somewhat similar to Jinja2. Helm also added all functions from the Sprig library, making it more powerful and flexible than Jinja2. Helm chart developer should be very familiar with these functions, as well as the best practices. For example, the cryptographic and security functions in Sprig library gives us the ability to create self-signed X509 certificates during installation.

Since template introduces another layer of abstraction, to help troubleshooting we should be able to preview rendered template with the template command:

helm template orthanc | less

The command above simply renders template without attempting to execute the chart. To go one step further, you can dry-run the installation with:

helm install orthweb ./orthanc --debug --dry-run | less

Although Jinja2 (using {% … %} to express template control) and Go (using {{ … }} to express template control) have different syntaxes, one aspect that is similar between them, is chomping whitespace with minus sign (-). This is pretty common in templating language. The documentation of both Jinja2 and Helm have a section on whitespace control. Not paying attention to this nuance may cause pesky errors.

Dependency

The Orthanc application relies on Postgres database, which itself is deployed by a separate helm chart. This can be specified in Chart.yaml (Helm V3), like this:

dependencies:
 - condition: postgresql-ha.enabled
 name: postgresql-ha
 repository: https://charts.bitnami.com/bitnami
 version: 7.8.x

The values of variables of the dependency chart can be specified in values.yaml of the root chart. They can also be imperatively specified as a parameter of helm install command.

The section above also requires the dependency chart to be downloaded into the charts sub-directory. This can be done with:

helm dependency update

Then you will notice a file with tgz extension in the charts sub-directory. Note that when you change the version of the dependency package in Chart.yaml, then you will need to run the command again. Alternatively, this command can be automatically executed before helm install if you specify the switch –dependency-update with helm install.

The main chart (e.g. wordpress) is referred to as parent chart, and the charts it depends on are referred to as sub-chart (e.g. mariadb, memcached). When it comes to managing property values, values from parent chart can override those from sub-chart, as explained here. On the other hand, values from sub-chart can override those from parent chart in two formats: export format (keyword exports) and child-parent format (keyword import-values). This is something to be careful and we can use the aforementioned template command to display the rendered values.

Hooks

Helm does a great job in figuring out the dependency relationship between kubernetes objects defined in the chart, and create them in order. So typically you do not need hooks for objects in the chart. However, in certain circumstances, such as cleaning up after uninstallation, we may need hooks. Here is a list of available hooks.

It is worth-noting that hook is not tied to an action. Instead it is tied to a kubernetes resource. The resource could be a job, a config map, etc. The resource is tied to a hook simply by resource annotation.

Moving to GUI

Helm is a command-line tool. For a team with varying levels of familiarity with command-line, GUI-based tool is a better option. For that, some enterprises adopt Rancher, a comprehensive Kubernetes cluster management platform. Rancher manages many aspects of Kubernetes cluster through web portal. One aspect is the support of helm chart. Rancher can be install on a cluster of its own. For demo, it can also be installed on docker desktop, a single-node Kubernetes cluster by Docker. In both cases, Nginx ingress controller needs to be configured.