continuous deployment on Digi Hunch

Kick the tires on ArgoCD

Sun, 10 Jul 2022 00:10:00 -0400

Background

In January, I wrote about FluxCD, and adopted it in the Korthweb project. I like the simple design of FluxCD, and I am comfortable with commands without using a web UI. Half a year later, I am re-visiting this choice, with ArgoCD in mind.

After reading numerous recent posts that compare the two (such as this one from the new stack), I started to give more thoughts on ArgoCD, for a couple reasons. First, ArgoCD has more contributing companies and public references. Red Hat adopted Argo CD as the underlying technology for OpenShift GitOps (and Tekton for pipeline). Second, ArgoCD has a mature UI. In corporate collaboration, especially with those not well-versed with command line, a UI is extremely helpful. Bundled with the UI, Argo CD also uses its own RBAC independent of Kubernetes RBAC, making it a great choice for continuous deployment for enterprise applications.

User Interface

You can choose to install just the core components, without UI, SSO and multi-cluster features. To me, it does not make sense because those features are exactly the reason to choose Argo CD. Argo CD also has an eponymous CLI tool, similar to flux for Flux CD. Since Argo CD uses its own RBAC independent of Kubernetes RBAC, Argo CD will have its own credential. The default username is admin and password needs to be retrieved from Secrets. In the tutorial we use this user’s credential to interact with the target cluster. This is different from Flux where the client simply uses kubectl configuration.

In the “Getting Started” guide, we create a namespace “argocd” and install it with manifests. Alternatively, we can use CLI tool to install it. With Argo CD, we can consider hosting it on a different port than the workload’s port, such as 8443. This port serves as “GitOps management port” separated from the application port (e.g. 443) for business workload. Just like any application on Kubernetes, we usually need to configure an Ingress on the management port.

In the argocd namespace, we can see a few services. The application set controller acts as the main controller that reconciles between actual state and desired state. There is also a redis server, for storing data in Argo CD. Another service to note is the dex server, indicating that the SSO capability is provided by the Dex project.

Features

In FluxCD, they use a CRD Kustomization with kustomize.toolkit.fluxcd.io/v1beta2 as version. This name is confusing. Luckily, in ArgoCD, the CRDs are Application (with argoproj.io/v1alpha1 version) and ApplicationSet (with argoproj.io/v1alpha1 version). The controllers monitors CRs created with these CRDs. ArgoCD defines Application as a group of Kubernetes resources as defined by a manifest. Argo CD Application resource deploys resources from a single Git repository to a single destination cluster/namespace. On the other hand, ApplicationSet uses templated automation to create, modify, and manage multiple Argo CD applications at once. ApplicationSet controllers monitors ApplicationSet resources. We can define a template within the declaration of an ApplicationSet and it allows for parameter substitution.

Another key capability for a GitOps utility is the support of multiple templating tools. FluxCD supports Helm and Kustomize. So do Argo CD. In addition, ArgoCD also supports jsonnet (ksonnet is not supported anymore). Jsonnet is a templating tool with many useful operators. Kubectl can directly consume Jsonnet’s JSON output as if it were YAML. In FluxCD, there are different CRDs for Helm and Kustomize (HelmRelease and Kustomization). In ArgoCD, the Application CRD covers Helm, Kustomize, and Jsonnet by embedding them as attributes in the declaration. This makes it even simpler than Flux CD.

Secret Management

It’s a big no-no to put secret in a code repository. Therefore any GitOps solution must solve the problem with secret logistics. What baffles me is ArgoCD remains un-opinionated on this matter. There appears to be some context for this stance. As a result, ArgoCD only points to a few third-party secret management solutions.

Secret logistics is a tricky problem with GitOps workflow. One of the techniques is sealed secret. In the repo we store the secret as encrypted by public key. The controller keeps the private key which decrypts the secret at the time of deployment. A few engineers don’t find it feasible for operation, as explained in this post and this post.

An alternative is to use external secret operator. Such operators will be able to sync a secret from external secret store such as Hashicorp Vault, AWS secret manager, Azure Key Vault, etc. This is much neater because we only store reference to secret in the repository. However, when the cluster assesses external secret store, it still requires either a secret, or a permission.

Another technique that drawing attentions is using sops for secret and kustomize-sops to integrate with ArgoCD. Mozilla’s sops (SecretOPerationS) project releases a binary utility called sops to encrypt and decrypt YAML manifest. It can encrypt only the values in YAML manifest and not the attributes (keys). We use sops to encrypt YAML files in order to store them in Git repo. We also use sops to decrypt YAML right before we deploy it with kubectl. The key pair for encryption and decryption can be stored in Azure Key Vault, AWS KMS, GCP KMS, and even age and pgp. Here is a simple tutorial.

Example

In a few command we can configure a minimal example. I first use KinD or Minikube to create a cluster following the steps in real-quicK-cluster repo. Then we configure an application using YAML manifest.

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
$ kubectl port-forward svc/argocd-server -n argocd 8443:443

From MacOS, I can now browse to https://localhost:8443/ for ArgoCD UI, and login with admin user and the password as printed above. We can then use the UI to create an Application. Alternatively, we can configure an application using CLI utility:

$ brew install argocd
$ argocd login localhost:8443
$ argocd app create guestbook --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace default
$ argocd app sync guestbook

On the GUI we should see the application is sync’ed.

On the left panel, click on the gear icon for settings. Under Clusters, we can see the cluster URL, which we use as dest-server value above. We can add more clusters here.

Pick the right open-source project

My experience with ArgoCD and FluxCD raised an issue to reflect on. For the same problem, there are multiple choices of open-source project. What would be a good methodology to choose the right open-source technology? I would first look at the following factors:

History
Community size
License model
Enterprise support
Governing model

The history and community size give a good idea of the technical maturity level. The License model has legal implications as to where you can use the technology. License model can change even after the project has been launched and even become not open-source any more. One example is Elasticsearch license change in Jan 2021, from the very open APLv2 to under dual license of both Elastic License (by Elastic) and Server Side Public License (SSPL, introduced by MongoDB). Neither license is certified by OSI as compliant with open source definition. For enterprises, apart from licensing, I’d also consider if any organization provides commercial support for the open-source product.

What is often neglected is the governing model. The governing model determines how product decisions are made that will shape the future of the project. For example, CNCF is a vendor neutral foundation for cloud computing and if an organization contributes a project to CNCF, then the member projects have to align with CNCF’s governing practices. Google also introduced an OUC (Open Usage Common) governing model.

Another good way is to just follow Red Hat’s choice. Red Hat built its business model around open-source technologies. They have to pick open-source project to provide enterprise level support so presumably they have a rigour selection process that one can piggyback off. Products supported by Red Hat are composed of open source components often vetted by multiple upstream communities, and changes made to these components are pushed to their respective upstream projects, often before they land in a supported product from Red Hat. Here are more insights in this business model.

Had I gone with the Red Hat rule, I would have picked Argo CD in the first place because that is the base of OpenShift GitOps!

Verdict

ArgoCD and FluxCD are tools for GitOps. However, just using a GitOps tool does not guarantee that all workload are deployed continuously. First, these GitOps controllers cannot replace Operators, which has domain knowledge about how to orchestrate their workloads. Second, resources not directly managed by these operators (such as those installed by Helm) are not being monitored continuously. For continuous deployment over all workloads, I recommend use operator to install third party applications, and use Argo CD or Flux CD to manage the operators.

GitOps makes use of the controller pattern to manage deployment in a continuous matter. Controller works closely with CRDs. ArgoCD uses similar set of CRDs to manage continuous deployment. What sets it apart is the user friendly web UI, the IAM integration and multi-cluster capabilities. These capabilities make ArgoCD well adapted in enterprise IT eco-system.

Chaos Mesh – Cloud Native Chaos Engineering

Thu, 30 Jun 2022 21:49:00 -0400

In this post, we discuss the resilience test problem and why chaos mesh emerged. Then we go over a lab of chaos mesh with a few experiments.

The Problem

I used to support a server application installed in customer’s data centre. The server application receives data from client application, cleanse the data and put them on customer-managed storage. Unfortunately, the customer-managed storage system hang up from time to time. When this happens, the application does not get an I/O failure to begin with so it continues to serve client traffic with many threads pending for I/O completion. All the threads will fail eventually after a long, unresponsive pause.

What is really bad, is that the server application first acknowledges the client application of the recipient of their data, and then asynchronously archive them to the customer’s storage. In this case, the client applications think they securely send out the data upon receiving acknowledgement. However, the server application lost the data afterwards because it does not interact with storage system in a fail-safe fashion. As a result, I often find myself in the business of restoring client data from the cache directory on the servers, and grumbling about how the quality process allowed this issue slipped into production.

To be fair, this is not an easy catch in the quality assurance process. It is caused by a hardware issue, which is technically the customer’s own problem. However, it is the vendor’s responsibility to design fault-tolerant software. Further, if we put on the site reliability engineering goggle, we see vendor’s software and customer’s hardware as a whole, instead of two disparate silos. The problem is however, how we emulate this kind of fault in house?

Chaos Engineering

A software engineer could have unmounted the storage while the software is active during test. This approach is faulty because it has a couple of issues. First, unmounting the storage does not produce exactly the same symptom. Unmounting the storage in most cases gives a clear failure upfront and the application would have caught it. What we need is a delay in storage long enough to cause a time out. Second, we’d have to perform this activity with an operating system command with sufficient privileges. The software being tested usually do not have such privileges. We need to systematically inject failures of particular kind.

Chaos are interruptions along with the cascading effects, that happen in production but are impossible or extremely difficult to emulate in lower environments through automated quality process. To improve robustness, we need the capability to emulate various failure scenarios in a systematic manner. Chaos Engineering aims to improve such capability.

Chaos Engineering emerged as a discipline to improve software’s ability to tolerate failures while still delivery adequate quality of service. This is known as the resiliency of the software. Suppose the storage has 99.9999% SLA, or 31 seconds of downtime every year. Chaos Engineer requires the functional parts of our software do what they can to protect the data, during the 31 seconds of turbulence each year, instead of just falling apart. In regulated industries, this is critical. Resilience testing is difficult in quality assurance process, but we can’t avoid it.

By definition, Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. It is particularly useful in distributed system, where network faults are normal. The one-page chaos engineering website has a good summary of such intent. At Netflix, they built Chaos Monkey, a tool to randomly cause failures on certain computing instances in the cloud. It is one of the first Chaos Engineering tools and it is a big step forward. Chaos Monkey can perform one type of experiment (faulty server) and requires writing custom code.

Chaos Mesh

Moving to Kubernetes platform, where everything operates on network, it is even more important to have a suite of resilience testing tools. Thankfully, we have Chaos Mesh, a cloud-native Chaos Engineering platform that helps platform engineers and software engineers simulate faults in a variety of scenarios. There are other chaos engineering tools (such as Litmus, also a CNCF project) but I will just focus on Chaos Mesh in this post because it has longer development history.

With Chaos Mesh, we can run experiments to simulate certain types of faults, including faults on Pod, Network, File I/O, DNS, Time, JVM, Linux Kernel, HTTP. We can also emulate incidents on cloud platforms (Azure, AWS, GCP) and emulate stress. We can even orchestrate multiple experiments into a workflow. Experiments can start ad hoc or on schedule. In addition to Kubernetes platform, the Chaos Mesh team also provides a tool Chaosd to emulate faults on a physical nodes. In the rest of this post, we will install Chaos Mesh and perform four simple experiments using the bookinfo sample application from Istio’s installer.

Demo Platform

In this lab, we install chaos mesh on Minikube on my Mac environment. I have install the platform in a particular way so the experiments will all be successful. I use this script in my real-quick-cluster project to create minikube cluster. This script configures a 3-node minikube cluster, with containerd as CRI, and enables plugins metallb and metrics server. Metallb provides a load balancer for the cluster. Metrics server allows scaling based on metrics, a test scenario for stressor in Chaos Mesh.

I will also install Istio in this lab, because I use it for Ingress and service routing. I also use a sample application called bookinfo which comes with Istio code repository. It is highly recommended to have an ingress as it is used in production, although you may use your own choice of ingress and service routing tool such as ingress-nginx. To install istio, we run this simple script:

#! /bin/bash
curl -L https://istio.io/downloadIstio | sh -
export PATH=$(realpath istio*/bin):$PATH
if istioctl x precheck; then
 echo ready to install istio and label namespace for istio-injection
 istioctl install -f istio-operator.yaml -y --verify
 kubectl label namespace default istio-injection=enabled
else
 echo failed precheck
 exit 1
fi

Below is the content of istio-operator.yaml

apiVersion: install.istio.io/v1alpha2
kind: IstioOperator
metadata:
 name: istio-operator
spec:
 profile: default
 hub: docker.io/istio
 tag: 1.13.2
 namespace: istio-system
 meshConfig:
 accessLogFile: /dev/stdout
 components:
 pilot:
 k8s:
 hpaSpec:
 maxReplicas: 7
 minReplicas: 1
 nodeSelector:
 beta.kubernetes.io/os: linux
 overlays:
 - kind: Deployment
 name: istiod
 patches:
 - path: spec.template.metadata.labels.version
 value: 1.13.2
 ingressGateways:
 - name: istio-ingressgateway
 enabled: true
 label:
 istio: ingressgateway
 k8s:
 hpaSpec:
 maxReplicas: 5
 minReplicas: 1
 service:
 ports:
 - name: http
 port: 80
 targetPort: 20080
 protocol: TCP
 - name: http-dashboard
 port: 8080
 targetPort: 28080
 protocol: TCP
 overlays:
 - kind: Deployment
 name: istio-ingressgateway
 patches:
 - path: spec.template.metadata.labels.version
 value: 1.13.2
 - path: spec.template.spec.containers[name:istio-proxy].lifecycle
 value:
 preStop:
 exec:
 command: ["sh", "-c", "sleep 5"]

For more about istio installation, refer to my previous post. Next, we’ll create the namespaces, install bookinfo and install chaos mesh using Helm:

kubectl create ns chaos-testing
kubectl create ns bookinfo

kubectl -n bookinfo apply -f istio/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl -n bookinfo apply -f istio/samples/bookinfo/networking/bookinfo-gateway.yaml

helm install chaos-mesh chaos-mesh/chaos-mesh -n chaos-testing --version 2.2.0 --set dashboard.service.type=ClusterIP --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock

As soon as we install bookinfo, we should be able to access it at http://192.168.64.16/productpage, suppose the IP address is 192.168.64.16. We will simply use HTTP because certificate is not the point of this demo. Note that when we install Chaos Mesh we have to specify the runtime (containerd). Otherwise, we might run into issues when running experiments (e.g. “unable to flush ip sets” for network faults). Before fault injections, we’ll start a curl command and observe the output:

while true; do
 curl -I http://192.168.64.16/productpage --connect-timeout 4 --max-time 5
 sleep 5
done

We can watch the output of the command above as we inject each of the following faults later on.

Chaos Mesh UI

Since we installed chaos mesh as cluster IP service, we need to expose it using Istio Gateway and Virtual Service with the following manifest. If you have your own choice of Ingress, use a corresponding ingress resource.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: chaos-dashboard-gateway
 namespace: chaos-testing
spec:
 selector:
 istio: ingressgateway
 servers:
 - port:
 number: 8080
 name: http
 protocol: HTTP
 hosts:
 - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: chaos-dashboard-vs
 namespace: chaos-testing
spec:
 hosts:
 - "*"
 gateways:
 - chaos-dashboard-gateway
 http:
 - route:
 - destination:
 port:
 number: 2333
 host: chaos-dashboard

Once we configured Virtual Service and Gateways, and waited all Pods in chaos-testing namespace to come up, then we should be able to access the web portal of chaos mesh at 192.168.64.16:8080. Follow the on-screen instruction to fetch the token for access at Cluster level.

Chaos Mesh Web Portal

Now, we can create experiments, either using web portal, or using their CRDs. We will use CRDs in this demo.

Network Latency Injection

In this section we artificially introduce a network latency of 3 seconds for a period of 45 seconds. It takes effect as soon as we create the NetworkChaos CRD as below:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
 name: network-delay
spec:
 action: delay # the specific chaos action to inject
 mode: one # the mode to run chaos action; supported modes are one/all/fixed/fixed-percent/random-max-percent
 selector: # pods where to inject chaos actions
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage
 delay:
 latency: '3s'
 duration: '45s'

Watch for the curl output, which normally returns 200 code every 5 seconds, now fails for about 45 second window.

After the window of latency injection, the curl test returns normal. During the experiment, we can check the status of the fault injection by describing the NetworkChaos object. Alternatively, we can look at Experiments and Events in Chaos Mesh web portal. When completed, we can delete the NetworkChaos object. Alternatively, we can archive the experiment from web portal.

HTTP Failure Injection

The NetworkChaos API injects faults at Pod’s network layer. HTTPChaos API introduces HTTP error. We can configure an HTTPChaos object following the instruction:

apiVersion: chaos-mesh.org/v1alpha1
kind: HTTPChaos
metadata:
 name: test-http-chaos
spec:
 mode: all
 selector:
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage
 target: Request
 port: 9080
 #method: GET
 path: '*'
 abort: true
 duration: 2m

This object intercepts HTTP calls to the Pods as labelled and aborts the request. If we monitor the curl output at the same time, we can see that it starts to receive 503 error as soon as we injected the fault.

Once we remove the HTTPChaos object, the return code becomes 200 again.

Pod Failure Injection

We can emulate a fault that kills a Pod with the following manifest:

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
 name: pod-failure-example
spec:
 action: pod-kill
 mode: one
 duration: '30s'
 selector:
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage

In the mean time, watch for the Pods in the bookinfo namespace, we’ll notice the old Pod being terminated.

Because the Pod is managed by a replica set, the replica set brings up a new Pod to ensure the desired number of replicas.

Stressor

In this experiment, we add an HPA (Horizontal Pod Autoscaler) to the productpage deployment, emulate a memory stress to the Pod and watch it scale up. Before the testing, we need to first configure a resource request for memory for the deployment. To configure memory request, edit the deployment productpage-v1 by adding memory request:

Then we add an HPA as below:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
 name: productpage-deploy
 namespace: bookinfo
spec:
 maxReplicas: 4
 minReplicas: 1
 scaleTargetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: productpage-v1
 metrics:
 - type: Resource
 resource:
 name: memory
 target:
 type: Utilization
 averageUtilization: 70

Now if we describe the HPA, we shall see that HPA is active:

Then we create a StressChaos object as below:

apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
 name: memory-stress-example
spec:
 mode: one
 selector:
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage
 stressors:
 memory:
 workers: 2
 size: 128MiB

It will take some time for the HPA to detect the memory stress and scale up. We can tweak this with scaling policy. After scaling activity, the HPA status will report with the new desired number of replica:

We can also confirm that by checking the deployment size. If we want the size to reduce after we remove the stressor, we need to configure HPA accordingly with scale down triggers.

Chaos Mesh uses stress-ng utility to emulate memory or cpu stress on a Pod.

Summary

Chaos mesh is a cloud native resilience testing tool. It supports many fault experiments not only on Kubernetes, but also on virtual machines. I wish I came across this tool sooner in my professional life. Had I had chaosd in the past, I could have incorporate it in our quality assurance process on Linux platform. Without Chaos Mesh, we’d have to use different utilities for different types of faults (e.g. stress or stress-ng for cpu and memory stress). In the domain of platform engineering on Kubernetes, it helps not only software developers, but also platform engineers with correct configuration for workloads on Kubernetes. In this post we explored four types of faults. For more about all fault types and their attributes, go to their documentation for more details.

FluxCD: Continuous Deployment with GitOps

Sat, 15 Jan 2022 18:49:00 -0400

This post explains why I land on FluxCD GitOps for my project. Let’s star

Background

In the Korthweb project, I landed on Istio for the Ingress Gateway technology. I first attempted to expand the orthanc Helm Chart to bring Istio as dependency (sub-chart). One of the external chart for Istio gateway needs to be referenced multiple times (for ingress and egress). However, it cannot even be used as dependency (sub-chart) because of this issue. Istio didn’t re-introduce Helm Chart as a supported deployment until September 2021. So I’m not too confident about it.

This leads to a second thought of the “Big Helm Chart” approach to deploy all tiers in Korthweb. Helm is based on templating, and having two layers of charts brings complexity. I do need to bring a number of Helm charts together, but not necessarily by another Helm Chart.

The Kubernetes documentation also mentions Kustomize as an alternative to Helm. Below is a quick exploration of it.

Kustomize

A challenge with managing declarative object is to organize numerous manifest files to maintain consistency and readability. Kustomize is a standalone tool to customize Kubernetes objects through a Kustomization file. This page provides a good list of fields that can be used:

vars and replacements: copy fields from one source into any number of specified targets.
namePrefix, namespace, nameSuffix: customize namespace and names.
configMapGenerator, secretGenerator, and generatorOptions: create configuration entries from literal, files, or environment variables.
resources: indicates another kustomization directory (e.g. as base)
patches (also called overlays) add or override fields on resources.
patchesStrategicMerge: modifies values in known (loaded) resources
images: modifies the name, tags and/or digest for images, without creating patches.
openapi: use Kubernetes OpenAPI data to get merge key and patch strategy information about resource types.

The base -> overlay pattern ensures readability as well as portability. Typical use pattern is create one base kustomization, and several overlay kustomizations each representing an environment, such as dev, qa and production. Here is a good tutorial.

Helm and Kustomize are two approaches to deploy Kubernetes workload. Both tackle the challenge of managing many YAML declarations. Helm is template driven, and is commonly used by application developers as a means of packaging application releases while orchestrating the dependencies. Kustomize follows a base-overlay pattern, and is more commonly used by cluster operators for re-using manifests across multiple environments (e.g. dev, staging and prod).

In my use case, my deployment needs both. I need third party Helm chart to configure components such as PostgreSQL for HA. I also need Kustomize for the orthanc workload. With the need for both, there should be a higher level deployment technology that ingrate with both mechanisms.

GitOps

GitOps is originally brought up by Weaveworks in 2017. It is a methodology to deploy workload continuously, using a Git repository as source of truth. The point of GitOps is not about tooling, or specific platform, but rather to ensure the workload state matches the declaration in repository. For example, you can implement GitOps with Ansible for VM environment.

When it comes to managing Kubernetes workload, a GitOps tool must handle the challenge with managing many YAML declarations. Therefore most GitOps tools seek to support many well-adopted mechanisms such as Helm and Kustomize as discussed above, instead of simply taking an enormous amount of raw YAML declarations. The amount of deployment mechanisms supported, is a key indicator of how powerful a GitOps tool is.

The most notable tools are ArgoCD and FluxCD. Both are currently CNCF incubating projects. ArgoCD is powerful with many tools supported, such as Kustomize, Helm, Ksonnet, Jsonnet, etc. It also contains a user interface and aims to manage an entire deployment workflow. On the other hand, FluxCD has controllers mainly for Kustomize and Helm, with a jsonnet extension from third party. This post draws a comparison of them (along with JenkinsX) based on earlier versions (from mid 2020). There is even a standard (OpenGitOps) in CNCF landscape but still at Sandbox level.

For Korthweb project, I chose FluxCD. It is simple, and provides just enough types of controller for what I do.

FluxCD

The diagram below illustrates the components:

The configurations starts with a bootstrapping process, which creates directory in Git repository (if not exist), and installs flux-system components in the target Kubernetes cluster. The sync process starts as soon as bootstrapping is completed. The process in charge of syncing declarations to target cluster, confusingly, is also called Kustomization. Therefore there are two Kustomizations. According to the FAQ on FluxCD website:

There are two Kustomization types. the kustomization.kustomize.toolkit.fluxcd.io is a Kubernetes custom resource while kustomization.kustomize.config.k8s.io is the type used to configure a Kustomize overlay. The kustomization.kustomize.toolkit.fluxcd.io object refers to a kustomization.yaml file path inside a Git repository or Bucket source.

Inside of the Git repository, with a kustomization.kustomize.toolkit.fluxcd.io obejct, the flux-system points to Kustomization file (representing kustomization.kustomize.config.k8s.io object) at root level. The kustomization.yaml file organizes resources in the same directory. A kustomize directory may also reference other kustomize directory, forming a hierarchy.

Implementation

FluxCD has a command “flux check –pre” to check the prerequisite, such as kubectl. The code is stored in the GitOps directory of Korthweb repository.

To configure deployment, we need to first create a personal access token. For GitHub, here is the instruction. Export the token to environment variable, and launch bootstrapping:

$ export GITHUB_TOKEN=xxx_yyy55555XXXodr7ABBBB234CCccw
$ flux bootstrap github \
 --owner=digihunch \
 --repository=korthweb \
 --branch=main \
 --personal \
 --path=gitops/environment/dev

A deploy key is configured during the bootstrapping process. As soon as bootstrapping is completed, the sync (aka kustomization, or reconciliation) has started, which can be monitored using:

flux get kustomizations --watch

Running this command without –watch switch returns the overview of all kustomizations. Once reconciliation is completed, it should display something like:

NAME 	READY	MESSAGE 	REVISION 	SUSPENDED
application 	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False
dependency 	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False
flux-system 	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False
infrastructure	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False

FluxCD introduced a number of CRDS. For example, to check configured source repository, check gitrepositories CRD:

kubectl get gitrepositories -n flux-system

To check the detail of one kustomization (e.g. infrastructure), check kusomization CRD in flux-system namespace:

kubectl -n flux-system get kustomizations flux-system -o yaml | less

Other commonly used custom resources include:

helmcharts
helmreleases
helmrepositories

A Helm release can be created imperatively, for example:

$ flux create source helm istio \
 --interval=1h \
 --url=https://istio-release.storage.googleapis.com/charts
$ flux create helmrelease istio-base \
 --interval=1h \
 --release-name=istio-base \
 --target-namespace=istio-system \
 --create-target-namespace=true \
 --source=HelmRepository/istio \
 --chart=base \
 --chart-version="1.12.0" 


$ flux create helmrelease istiod \
 --interval=1h \
 --release-name=istiod \
 --target-namespace=istio-system \
 --source=HelmRepository/istio \
 --chart=istiod \
 --chart-version="1.12.0" \
 --values=istiod-values.yaml

However, in a GitOps approach, they should be stored as code (use –export to export declaration). For example, the infrastructure kustomization keeps HelmReleases for installing Istio and PostgreSQL. This kustomization is referenced by a Flux Kustomization from higher level.

You can also manually reconcile one of the kustomizations (or other CRDs) with flux reconcile command, for example:

flux reconcile kustomization dependency

Since Nov 2021, FluxCD (0.20) supports reconciliation based on server-side apply. This increases performance and help address issues such as applying large config map, which is the equivalent of adding –server-side flag to the kubectl apply command.

Limitation

Troubleshooting the FluxCD repo can be involving and counter-intuitive. I had to commit a lot of changes to the repo because it serves as source of truth. Even though the commits can be made to a branch, it still involves a lot of code pushes. Traditionally I commit a change after testing. In the GitOps workflow, I commit a change then to test.

More flexible troubleshooting options are still to be desired. For example, there is no way to run one (FluxCD’s) Kustomization object at a time (and disable the rest), unless you remove their YAML files from the repo.

The next limitation is the ordering of resources in Kustomization. Arguably this is a limitation from Kustomize, instead of FluxCD’s. For example, I need the following manifest to be executed:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: selfsigned-issuer
spec:
 selfSigned: {}

The resource is a CRD that needs to be first installed using Helm chart (cert-manager). However there’s no way to control the sequence (Helm release executed first before declaration using CRD). Although there’s some workaround, it is not convenient. Alternatively, we can separate the resource creation and CRD creation into separate kustomization objects with dependency relationship, as suggested in this example.

Summary

Deployment in Kubernetes can get complicated with a lot of manifests. Helm, Kustomize and the like are means to handle the complexity due to numerous manifests. GitOps tools such as FlexCD brings these tools under a single framework, and more importantly, implements the idea of using Git repository as source of truth for continuous deployment.

Getting started with GitHub Actions

Thu, 27 May 2021 13:52:31 -0400

In my orthweb project, I had to compile a library on my own. In search for free computing resources I realized that GitHub action can meet all my needs.

CI/CD pipeline

As a development project grows, there are many operational tasks demanding automation. Prior to pipeline technology, developers used to use Makefile to organize command execution locally. Today, its role has declined, but Makefile is a good choice in certain situations. In most cases though, to offload the build command execution to a shared system, automation engines like Jenkins came around. Then Jenkins evolved into pipelines.

In strict terms, CI pipeline is the build pipeline; and CD pipeline is release pipeline. The two types of pipelines use pretty much the same pool of building blocks, with different purposes. The build pipeline focuses on producing quality artifact in a consistent manner. The release pipelines focus on system stability while deploying an artifact across different environments. Because release pipelines may connect to different environment, it has to deal with various situations. It is very common to have multiple stages in release pipeline, each stage pointing to a different environment (e.g. DEV, TEST and PROD). At workplace both could be loosely referred to as CI/CD pipeline, or even simply pipeline.

A lot of projects provide pipeline capability: BitBucket, Bamboo, TeamCity, Jenkins, Azure DevOps, AWS CodePipeline, TravisCI etc. Since late 2018, GitHub also joined the game with GitHub actions. It is openly free for public repositories, and has a free tier for private repositories. It executes task as defined in .github/workflow/action.yaml in the code project. I will take my own project as an example.

Runners

You can run jobs in self-hosted runners or GitHub managed runners, similar to other pipeline solutions (e.g. self-hosted agent vs managed agent from Azure DevOps). The GitHub hosted runners only have three operating systems to support: Windows, Ubuntu and MacOS. The Ubuntu and Windows runners are built from Standard_DS2_v2 VMs in Microsoft Azure. They are pre-installed with a virtual environment with packages required for common build tasks. The same virtual environment is also used in hosted agents by Azure DevOps. While they are free and you can elevate privilege on the runner, you cannot SSH or RDP to it for further troubleshooting.

The self-hosted runners require users to manage the instance on their own, including configuring virtual environment, installing GitHub Action Runner, etc.

Workflow file

Most pipeline declaration uses YAML or JSON, such as Jenkinsfile, AWS CodePipeline. GitHub refers to an automation process as a “workflow” and you can program the workflow in YAML (.github/workflow/action.yaml). Here is the reference and an example with environmental variable and versioning:

The handling of environment is documented here. There are a lot of custom actions available in GitHub Marketplace. For example, the versioning in the above example, uses an action by Einar Egilsson, which is open source itself.

Example pipeline

My example pipeline consists of two phases:

Build Library: spin up a docker container to build source code, and publish the artifact
Publish Image: add the artifact to an existing Docker image, and publish the result as my own image.

The status of the pipeline is also open, and can be found here. The retention period of artifact is 90 days by default but can be customized. To persist the artifact, I add it to my own Docker image and publish it to DockerHub, hence the second phase.

When building the second phase, I need to generate secret from my DockerHub account and store that encrypted secrets in GitHub settings, so that the secret value can be referenced in workflow file.

Failures in Actions are displayed in error steps and by default the rest of the steps are skipped.

I use the Docker build & push plugin to build and push my own docker image to DockerHub. Apart from DockerHub as my choice, GitHub also has its own artifactory GitHub Packages with a small free tier. It supports NPM, Docker, Maven, Gradle, etc.

Triggers of Action

Most of the times, GitHub action are triggered upon commit to main branch of the repo. In GitHub, this is known as a workflow_dispatch event. This is not the only event that can trigger GitHub action. All the available events are listed here on its documentation. This makes it very flexible to trigger action at many points in the workflow.

One example is to trigger GitHub action during PR review. When a developer opens a PR with a few commits in the proposed branch, the PR can preemptively check linting, style, etc and even build the application. These activities can also be defined in a GitHub action manifest with pull_request as triggering event.

Troubleshooting

In general, it is painful to troubleshoot activities happening inside of runners. I often had to write a few steps for the sake of printing variables, and trigger a run to see what their value is. This requires a lot of time especially when I have to wait for available runners.

To help troubleshooting pipeline runs there is an open-source utility called act. You can run GitHub actions locally from a Docker container on your MacBook. You can deliver environment variables and secrets via files. If you ever need to troubleshoot the runner environment, you have the option to connect to the Shell environment inside of the runner container. This tool is extremely helpful.

Closing remarks

GitHub action really makes the CI/CD pipeline capability available to any developers who stores their code on GitHub. GitHub expands from a code repository solution to a full CI/CD solution with a free tier sufficient for personal projects.

Docker Compose, Docker Stack and Docker Swarm

Sun, 24 May 2020 21:58:03 -0400

This posting covers some basic docker orchestration tools.

Docker Compose

Docker Compose’s predecessor is a tool called Fig developed by Orchard, which was acquired by Docker in 2014, with Fig renamed to Docker Compose. Docker Compose is the official container management tool. It is essentially a python script that parses yaml file, to make Docker API calls to manage containers dynamically. It is installed along with Docker on MacOS and Windows. On Linux, you will have to download package with curl command and install manually. Docker Compose has three versions so far and we should create new template with v3.

The Docker compose yaml template consists of three parts:

services: similar to docker run
- build: specify Dockerfile to build image
- cap_add, cap_drop: specify kernel capabilities (e.g. NET_ADMIN, SYS_ADMIN)
- command: override default startup command by container
- container_name
- depends_on
- devices: map host device to container
- dns
- dns_search:
- entryppoint: override entry point from image
- env_file: specify file that stores environment variable
- environment: specify environment variable
- image: specify the location of image
- pid: share the PID namespace with host
- ports: expose network ports. HOST:CONTAINER
- networks
- volumes: mount host volume to container
networks: similar to docker network create
volumes: similar to docker volume create

Here is a typical structure of docker compose yaml template (wordpress):

version: "3.8"
services:
 mysql:
 image:mysql:5.7
 volumes:
 - mysql_data:/var/lib/mysql
 restart: always
 environment:
 MYSQL_ROOT_PASSWORD:root
 MYSQL_DATABASE:mywordpress
 MYSQL_USER:digihunch
 MYSQL_PASSWORD:hunchdigi
 wordpress:
 depends_on:
 - mysql
 image: wordpress:php7.4
 ports:
 - "8080:80"
 restart:always
 environment:
 WORDPRESS_DB_HOST:mysql:3306
 WORDPRESS_DB_USER:digihunch
 WORDPRESS_DB_PASSWORD: hunchdigi
 WORDPRESS_DB_NAME: digihunch 
networks:
 frontend:
 backend:
volumes
 mysql-data: {}

In summary, Docker Compose is an orchestration tool for single host, typically seen in development and test environment with dependencies between services.

Docker Stack

A stack is a set of related services and infrastructure that gets deployed and managed as a unit. A docker stack file has the same format as Docker Compose file, with the only requirement that the version: key specify a value of 3.0. The other difference between Docker Stacks and Docker Compose, is that stacks do not support builds. All images have to be built prior to deploying the stack.

From the stack file, Docker first executes the network section and create networks that do not exist. Then it goes through other elements. A service is a JSON collection(dictionary) that contains a bunch of keys. The image key is the only mandatory key in the service objects, which will be pulled from Docker Hub by default. Ports key maps the port of Swarm to the port of each service replica. By default, all ports are mapped using ingress mode. This means they’ll be mapped and accesible from every node in the Swarm -even nodes not running a replica. The alternative is host mode, where ports are only mapped on Swarm nodes running replicas for the service.

The environment key lets you inject environment variables into services replica.

The secrets key defines two secrets – revprox_cert and revprox_key. These must be defined in the top-level secrets key, and must exist on the system. Secrets get mounted into service replicas as a regular file. The secrets defined in this service will be mounted in each service replica as /run/secrets/revprox_cert and /run/secrets/revprox_key, unless otherwise specified.

The volumes key is used to mount pre-created volumes and host directories into a service replica.

The networks key ensures that all replicas for the service will be attached to the front-tier network. The network specified here must be defined in the networks top-level key, and if it doesn’t already exist, Docker will create it as an overlay.

The service also defines a placement constraint under the deploy key. This ensures that replicas for this service will always run on Swarm worker nodes. Placement constraints are a form of topology-aware scheduling, and can be a great way of influencing scheduling decisions.

When Docker stops a container, it issues a SIGTERM to the process with PID 1 inside the container. The container (its PID 1 process) then has a 10-second grace period to perform any clean-up operations. If it doesn’t handle the signal, it will be forcibly terminated after 10 seconds with a SIGKILL. The stop_grace_period property overrides this 10 second grace period.”
Although you may scale a docker service as part of a stack with scale command, it is not recommended. Instead, stack file should be used as the ultimate source of truth (declarative method vs imperative method). All changes to the stack should be made to the stack file, and the updated stack file used to redeploy the app.

Docker Swarm

For multi-host cluster, Docker Swarm facilitates the deployment of micro-services. Docker Swarm is:

a cluster of Docker hosts: enterprise-grade, secure communication, PKI with automation, dynamic addition of nodes
an orchestration engine, with deployment automation, deploying native swarm apps (using Docker API) and Kubernetes apps.

A Docker nodes can be physical servers, VMs, cloud instances, etc. Nodes are configured as managers or workers. Managers look after the control plane of the cluster, and dispatches tasks to workers. Managers forms a distributed management cluster on its own, and they use Raft protocol to ensure consistency. Workers accept tasks from managers and execute them. Swarm mandatorily uses TLS to encrypt communications, authenticate nodes, and authorize roles, with Automatic key rotation.

The atomic unit of scheduling on a swarm is the service. When a container is wrapped in a service, we call it a task or a replica, and the service construct adding things like scaling, rolling updates and simple rollbacks.

To initialize a swarm, we need to have the following ports open. Then we can initialize the first manager node, join additional manager nodes, and then join workers.

2377/tcp: for secure client-to-swarm communication
7946/tcp & udp: for control plane gossip
4789/udp: for VXLAN-based overlay networks

A Docker node can exist either in single-engine mode as stand alone, or in swarm mode as part of a swarm. Service only exist in swarm mode. Running docker swarm init on a Docker host in single-engine mode will switch that node into swarm mode, create a new swarm, and make the node the first manager of the swarm. Then additional nodes can be joined as managers or workers.

Swarm managers have native support for high availability, through an active-passive, multi-manager HA. Only one manager is considered active (the leader), which is the only one that will ever issue live commands against the swarm. If a passive manager receives commands for the swarm, it proxies them across to the leader.

Managers are either leaders or followers. This is Raft terminalogy because swarm uses an impelementation of the Raft consensus althorithm to power manager HA. As to HA, the following two best practices apply:

deploy an odd number of managers
don’t deploy too many managers (3 or 5 recommended, never more than 7)

Having an odd number of managers reduces the chances of split-brain conditions. Having less than 7 managers ensures that achieving consensus is quick.

With a service, we can specify name, port mappings, network to attach to, and images, as well as desired state for an application service. It is recommended in production environment to use docker-compose template to specify service. Services have replication mode, and the default is replicated. This will deploy a desired number of replicas and distribute them as evenly as possible across the cluster. The other mode is global, which runs a single replica on every node in the swarm.

Running “docker service scale” command can scale the number of service replicas from 5 to 10, which in the background updates the service’s desired state to the newly specified number of replicas. Behind the scenes, Swarm also runs a scheduling algorithm that defaults to balancing replicas as evenly as possible across the node in the swarm. Docker makes it super easy to push updates to deployed applications. With rolling update, you may specify number of replicas to update at a time, and cool-off period per update.

Here is a great article on the difference between Docker Swarm and Kubernetes.

AWS Certified DevOps Engineer Exam Tips

Sat, 17 Aug 2019 23:07:59 -0400

The last 30 days have been exhausting for me. I studied hard on the new AWS Certified DevOps Engineer exam and thank goodness I passed (750 out of 1000 is required). This was the hardest professional certification I ever worked on. The exam was re-launched recently in March 2019 so there is still a shortage of information around the community. I was hoping to share my experience to help demystify this new exam.

I had a somewhat solid background to begin with, having taken the AWS Certified Solution Architect Professional exam (before the 2019 update), and worked quite a bit on CloudFormation, automation and Git. However, I still did not anticipate the exam to be this difficult until I was halfway through and already had the exam and materials paid for.

Without clear guideline on study material other than the white papers, I first skimmed through the ACloudGuru course, which helped me form a high level sense of exam coverage. Nonetheless it does not cover any topic in-depth and therefore by no means makes an essential part of my study. I checked out LinuxAcademy course and they are much more in-depth for the major topics indeed. I like the course material in Lucid chart. However, the LinuxAcademy course along does not cover everything you need to know.

What I found extremely helpful is the free training videos from the official training website. I strongly recommend the 7-hour course Exam Readiness: AWS Certified DevOps Engineer – Professional. The instructor did a great job outlining the services and knowledge areas in the assessment . The course also comes with quality sample questions with answers and explanations on what the thinkings are behind the correct answers or why some choices are obviously wrong. I went through these questions twice and feel much better at not only understanding the question, but also understanding the intent of the question.

Apart from the Exam Readiness course, other free introductory courses from the official training website are helpful as well especially for those services that you only need to know the basics. Most of those courses are 5 ~ 10 minutes long, with brief but sufficient introduction and a demo session. The other extremely helpful resource is the official practice questions. The practice exam is harder than the actual exam but they closely resemble the actual question style in the exam. Unfortunately, no answer is provided but they made me spend time finding answers across the documentations and blogs. It is worth-noting that the AWS blogs provides plenty of use cases that are covered in the questions.

When it comes to the real exam, it covers a lot more topics than its predecessor. Many questions are long and confusing. And I wish I could run a diff command to highlight the differences between choices. During exam preparation you really need to train yourself on reading efficiently. I found myself sometimes eyeball through all four choices at the same time, which get my mind scattered.

During my study, I divide all services into three categories based on the level of familiarity, and here is my list:

Category 1. Know these services very well, in and out:

AWS ElasticBeanstalk, OpsWorks Stacks, OpsWorks Chef Automate, CloudFormation, CloudWatch, CodeBuild, CodeCommit, CodeDeploy, CodePipeline, CodeStar, Lambda, API Gateway, Config, Trusted Advisor, CloudTrail, Systems Manager, Autoscaling Group in EC2, DynamoDB

Category 2. Know these services well, but not necessarily down to every single detail:

Amazon Kinesis Firehose, Kinesis Analytics, Kinesis Streams, Step Functions, Elastic Load Balancer, Secrets Manager, Serverless Application Model (SAM), Route53, RDS, Certificate Manager, ElasticSearch, ECS, ECR

Category 3. Know about these services at a high level, but do not skip any:

AWS Organization, X-Ray, GuardDuty, Macie, Inspector, Service Catalog, KMS, Batch, Athena, Single-Sign-On, Data LifeCycle Manager, CloudSearch, Health Dashboard, Glue, QuickSight, LightSail

Although I did not mention much about the white papers, I want to highlight their importance again. I would not attempt the exam without reading and understanding the required white papers.

If you aspire to take the AWS certified DevOps Engineer exam I hope this helps you a little bit. Good luck.