argocd on Digi Hunch

Kick the tires on ArgoCD

Sun, 10 Jul 2022 00:10:00 -0400

Background

In January, I wrote about FluxCD, and adopted it in the Korthweb project. I like the simple design of FluxCD, and I am comfortable with commands without using a web UI. Half a year later, I am re-visiting this choice, with ArgoCD in mind.

After reading numerous recent posts that compare the two (such as this one from the new stack), I started to give more thoughts on ArgoCD, for a couple reasons. First, ArgoCD has more contributing companies and public references. Red Hat adopted Argo CD as the underlying technology for OpenShift GitOps (and Tekton for pipeline). Second, ArgoCD has a mature UI. In corporate collaboration, especially with those not well-versed with command line, a UI is extremely helpful. Bundled with the UI, Argo CD also uses its own RBAC independent of Kubernetes RBAC, making it a great choice for continuous deployment for enterprise applications.

User Interface

You can choose to install just the core components, without UI, SSO and multi-cluster features. To me, it does not make sense because those features are exactly the reason to choose Argo CD. Argo CD also has an eponymous CLI tool, similar to flux for Flux CD. Since Argo CD uses its own RBAC independent of Kubernetes RBAC, Argo CD will have its own credential. The default username is admin and password needs to be retrieved from Secrets. In the tutorial we use this user’s credential to interact with the target cluster. This is different from Flux where the client simply uses kubectl configuration.

In the “Getting Started” guide, we create a namespace “argocd” and install it with manifests. Alternatively, we can use CLI tool to install it. With Argo CD, we can consider hosting it on a different port than the workload’s port, such as 8443. This port serves as “GitOps management port” separated from the application port (e.g. 443) for business workload. Just like any application on Kubernetes, we usually need to configure an Ingress on the management port.

In the argocd namespace, we can see a few services. The application set controller acts as the main controller that reconciles between actual state and desired state. There is also a redis server, for storing data in Argo CD. Another service to note is the dex server, indicating that the SSO capability is provided by the Dex project.

Features

In FluxCD, they use a CRD Kustomization with kustomize.toolkit.fluxcd.io/v1beta2 as version. This name is confusing. Luckily, in ArgoCD, the CRDs are Application (with argoproj.io/v1alpha1 version) and ApplicationSet (with argoproj.io/v1alpha1 version). The controllers monitors CRs created with these CRDs. ArgoCD defines Application as a group of Kubernetes resources as defined by a manifest. Argo CD Application resource deploys resources from a single Git repository to a single destination cluster/namespace. On the other hand, ApplicationSet uses templated automation to create, modify, and manage multiple Argo CD applications at once. ApplicationSet controllers monitors ApplicationSet resources. We can define a template within the declaration of an ApplicationSet and it allows for parameter substitution.

Another key capability for a GitOps utility is the support of multiple templating tools. FluxCD supports Helm and Kustomize. So do Argo CD. In addition, ArgoCD also supports jsonnet (ksonnet is not supported anymore). Jsonnet is a templating tool with many useful operators. Kubectl can directly consume Jsonnet’s JSON output as if it were YAML. In FluxCD, there are different CRDs for Helm and Kustomize (HelmRelease and Kustomization). In ArgoCD, the Application CRD covers Helm, Kustomize, and Jsonnet by embedding them as attributes in the declaration. This makes it even simpler than Flux CD.

Secret Management

It’s a big no-no to put secret in a code repository. Therefore any GitOps solution must solve the problem with secret logistics. What baffles me is ArgoCD remains un-opinionated on this matter. There appears to be some context for this stance. As a result, ArgoCD only points to a few third-party secret management solutions.

Secret logistics is a tricky problem with GitOps workflow. One of the techniques is sealed secret. In the repo we store the secret as encrypted by public key. The controller keeps the private key which decrypts the secret at the time of deployment. A few engineers don’t find it feasible for operation, as explained in this post and this post.

An alternative is to use external secret operator. Such operators will be able to sync a secret from external secret store such as Hashicorp Vault, AWS secret manager, Azure Key Vault, etc. This is much neater because we only store reference to secret in the repository. However, when the cluster assesses external secret store, it still requires either a secret, or a permission.

Another technique that drawing attentions is using sops for secret and kustomize-sops to integrate with ArgoCD. Mozilla’s sops (SecretOPerationS) project releases a binary utility called sops to encrypt and decrypt YAML manifest. It can encrypt only the values in YAML manifest and not the attributes (keys). We use sops to encrypt YAML files in order to store them in Git repo. We also use sops to decrypt YAML right before we deploy it with kubectl. The key pair for encryption and decryption can be stored in Azure Key Vault, AWS KMS, GCP KMS, and even age and pgp. Here is a simple tutorial.

Example

In a few command we can configure a minimal example. I first use KinD or Minikube to create a cluster following the steps in real-quicK-cluster repo. Then we configure an application using YAML manifest.

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
$ kubectl port-forward svc/argocd-server -n argocd 8443:443

From MacOS, I can now browse to https://localhost:8443/ for ArgoCD UI, and login with admin user and the password as printed above. We can then use the UI to create an Application. Alternatively, we can configure an application using CLI utility:

$ brew install argocd
$ argocd login localhost:8443
$ argocd app create guestbook --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace default
$ argocd app sync guestbook

On the GUI we should see the application is sync’ed.

On the left panel, click on the gear icon for settings. Under Clusters, we can see the cluster URL, which we use as dest-server value above. We can add more clusters here.

Pick the right open-source project

My experience with ArgoCD and FluxCD raised an issue to reflect on. For the same problem, there are multiple choices of open-source project. What would be a good methodology to choose the right open-source technology? I would first look at the following factors:

History
Community size
License model
Enterprise support
Governing model

The history and community size give a good idea of the technical maturity level. The License model has legal implications as to where you can use the technology. License model can change even after the project has been launched and even become not open-source any more. One example is Elasticsearch license change in Jan 2021, from the very open APLv2 to under dual license of both Elastic License (by Elastic) and Server Side Public License (SSPL, introduced by MongoDB). Neither license is certified by OSI as compliant with open source definition. For enterprises, apart from licensing, I’d also consider if any organization provides commercial support for the open-source product.

What is often neglected is the governing model. The governing model determines how product decisions are made that will shape the future of the project. For example, CNCF is a vendor neutral foundation for cloud computing and if an organization contributes a project to CNCF, then the member projects have to align with CNCF’s governing practices. Google also introduced an OUC (Open Usage Common) governing model.

Another good way is to just follow Red Hat’s choice. Red Hat built its business model around open-source technologies. They have to pick open-source project to provide enterprise level support so presumably they have a rigour selection process that one can piggyback off. Products supported by Red Hat are composed of open source components often vetted by multiple upstream communities, and changes made to these components are pushed to their respective upstream projects, often before they land in a supported product from Red Hat. Here are more insights in this business model.

Had I gone with the Red Hat rule, I would have picked Argo CD in the first place because that is the base of OpenShift GitOps!

Verdict

ArgoCD and FluxCD are tools for GitOps. However, just using a GitOps tool does not guarantee that all workload are deployed continuously. First, these GitOps controllers cannot replace Operators, which has domain knowledge about how to orchestrate their workloads. Second, resources not directly managed by these operators (such as those installed by Helm) are not being monitored continuously. For continuous deployment over all workloads, I recommend use operator to install third party applications, and use Argo CD or Flux CD to manage the operators.

GitOps makes use of the controller pattern to manage deployment in a continuous matter. Controller works closely with CRDs. ArgoCD uses similar set of CRDs to manage continuous deployment. What sets it apart is the user friendly web UI, the IAM integration and multi-cluster capabilities. These capabilities make ArgoCD well adapted in enterprise IT eco-system.

Kubernetes Operator

Thu, 07 Apr 2022 09:39:00 -0400

Kubernetes has a number of tools to automate the deployment of a single workload. In previous posts, we had covered Helm and Kustomize. What are left unresolved is how to maintain the status of workload after deployment is completed. In this post, I will give an introduction to Kubernetes Operator. Compared with Helm (templating approach) and Kustomize (patching approach), Kubernetes Operator follows the operator pattern. Operators are usually provided by the developer of the application.

Operator Pattern

In Kubernetes, we know that a controller takes care of routine tasks to ensure that desired state expressed by Kubernetes resource types matches the current state. One example is that the Deployment controller ensures the number of pods running matches the amount specified in the replica field. Controller is the key to ensure that resources can be managed by declarative manifests for Kubernetes resources.

Kubernetes makes use of controller pattern throughout its own design. One of its key component, Controller Manager, is a collection of many controllers. Each controller is in charge of a control loop, responsible for listening the object it manages. Another component, Kube-scheduler, is also a special type of Controller. The kube-scheduler monitors unscheduled Pod and health of nodes and determines the best Node to schedule the new Pod to. Then it writes the decision to etcd store for kubelet to execute.

This controller pattern is fairly successful in what it does and we can extend the use of it. Beyond the built-in resource types, we can create our own custom resource definitions (CRDs), and create controllers that watches for the manifest that declares custom resources (CRs). The controller ensures that the resource status matches their specifications. This is also known as reconciliation, which is implemented as a control loop. Operator pattern can be illustrated in the diagram below:

Operator Pattern

Technically, there is no difference between a controller and an operator. What makes an Operator (used to install workload) different than a native Kubernetes controller, are two things. First, an Operator usually needs CRDs because the built-in resource types are insufficient. Second, the operator reflects the domain knowledge to keep the target workload running. For example, stateful workloads such as database needs their operational steps executed in certain orders.

On Operator Pattern, CNCF published a whitepaper with a deeper review. This white paper is the best reference for a good understanding of the Operator Pattern.

Custom Resource Definition

The built-in controllers work with built-in objects (pre-defined APIs). Custom operators usually need their own APIs to function. To extend Kubernetes API, we define the schema of these APIs in the form of CRDs (custom resource definitions) using OpenAPIv3 standard. Then, we can declare Custom Resources (CRs) in compliance with the schema. The OpenAPIv3 schema in the CRD resource tells validating web hook (admission control) how to validate the schema when we send an CR in to API server.

When we work with third-party operators, they usually provide CRDs along with the operator implementation. For example, in my operator example project, we have a minimalist CRD WordPress with one property: sqlRootPassword and we can declare a CR as in this example. For a more realistic use case, we can take a look at Kiali CRD. In the next section, we’ll use it along with Kiali operator to install Kiali.

Operator Usage

Like Artifact Hub to Helm, OperatorHub is a public registry of most used Kubernetes Operators. In this section, we will take an example of using Operators. We will install Kiali as an add-on to Istio using Kiali CR and operator, which also depends on Prometheus to be installed using Prometheus Operator first. Note that the Kiali installation outlined in this section is not the the quick-start install manifests from Istio’s sample directory. For Kiali on production system we have to customize the installation.

Suppose we have installed Istio, we can then install Prometheus operator using Helm. The Prometheus operator will install Prometheus. Then we use Helm again to install Kiali operator. The Kiali operator will watch for creation of Kiali CRD, to deploy services:

$ helm install -f prometheus-values.yaml --namespace istio-system --repo https://prometheus-community.github.io/helm-charts --version 13.6.0 istio-prometheus prometheus --insecure-skip-tls-verify
$ helm install -f kiali-operator-values.yaml --namespace kiali-operator --repo https://kiali.org/helm-charts --version 1.45.0 kiali-op kiali-operator --create-namespace
$ kubectl apply -f kiali-cr.yaml

I include example content for each file in the commands above on Github gist (prometheus-values.yalm, kiali-operator-values.yaml and kiali-cr.yaml). For more options for installing Kiali, refer to their documentation.

I use this example to install Kiali and it includes two Operators, the Prometheus Operator and the Kiali Operator. The Prometheus Operator is one of the first ever written Kubernetes Operator. As soon as the operator is deployed, it starts to deploy the operator service. For the Kiali operator, we need to deploy Kiali CR after the Kiali Operator has been deployed. Both are valid patterns.

Operator Development

Operator is powerful. However, authoring an Operator is not a trivial effort. One usually start with a framework. A framework creates a body of boiler plate code that has the pattern implemented and allows developers to enrich the functions following the pattern. The white paper introduced three frameworks:

CNCF Operator Framework – aims at Operator Developers with an SDK, a scaffolding tool and a test harness. It currently supports three project types: Golang, Helm and Ansible. CNCF Operator framework consists of SDK and OLM.
Kopf (Kubernetes Operator Pythonic Framework) – an easy-to-use framework in Python that abstracts away most of the low-level Kubernetes API communications hassle.
kubebuilder – helps build a Manager similar to the native kube-controller-manager. For difference with OperatorSDK, read here.
Metacontroller: lightweight Kubernetes Controller as a Service

In CNCF Operator Framework, the Operator SDK supports development using Ansible, Helm and Golang. The author of this post makes a general comparison as follows:

Type	Best use case	Underlying technology	Amt of Effort
Helm	Stateless workload	Helm Charts	Med
Ansible	Stateless workload	Ansible Roles and Playbooks	Med
Golang	Stateful workload	Code developed in Golang	High

The aforementioned Kiali operator is an example of Operator developed in Ansible. The prometheus operator, is developed in Golang as the workload can be stateful depending on configuration. One needs to know how to develop operator in Golang in order to tackle the most complicated situations. This is requires some serious development effort. The documentation with a quick start section is available here. Even that is not very straightforward. RedHat, the maintainer of the CNCF Operator framework has a good blog post on how to develop an Operator in Golang.

The example requires some development knowledge to go through. On my MacOS (Intel) I have to configure the following prerequisites:

Install gcc, using command: xcode-select –install
Install the right version of golang. You can find the version here. The MacOS has a version of golang installed already so I had to install version 1.17 and link to it: brew install go@1.17 && brew link –force go@1.17
Install operator-sdk with home brew: brew install operator-sdk
When you run “operator-sdk version”, ensure the result shows a golang version that matches your installation.
If you need to push docker image, also connect to docker registry by running: docker login

Then we can create our working directory, initialize the repository and create boilerplate code (scaffolding) with these commands:

$ mkdir wordpress-operator && cd wordpress-operator
$ operator-sdk init --domain digihunch.com --repo github.com/digihunch/wordpress-operator
$ operator-sdk create api --group wordpress --version v1 --kind WordPress --resource --controller

With the repo initialized, we can go to the section “Defining the API” and “Implementing the Controller”. The blog post does not cover every code editing needed to bring up wordpress. You are supposed to go to the author’s repository to fit the changes into your own repo. The author’s repo has a few more controllers such as common.go and mysql.go.

At the end of the lab, you should be able to run the controller and bring up wordpress. I used my own repository for this lab and have made the code changes for this lap in a couple commits. To test locally with the code:

$ git clone git@github.com:digihunch/wordpress-operator.git
$ cd wordpress-operator
$ make install run

Then we can validate wordpress install from a new terminal as the instruction shows:

$ kubectl create -f config/samples/wordpress_v1_wordpress.yaml
$ minikube service wordpress --url

For Developers that requires more details, RedHat has an eBook for Kubernetes Operators, in supplement to the documentation. As DevOps professional, I’m mainly concerned with understanding how Operator works and using Operators correctly.

Too many Tools?

Now we seem to have too many choice of tools when it comes to deploying workload on Kubernetes. Kustomize and Helm can deploy simple workloads. Operator can deploy stateful workloads, as well as keep the workload status in check. Further, we have FluxCD and ArgoCD based on GitOps workflow.

When assessing a tool, we should think about the complexity of the workload deployed. If it is a single stateless workload, Kustomize or Helm should be sufficient. If it is not very simple but still stateless, we can consider using Helm charts developed by the community. For multiple workloads, we can build our own top-level chart to combine existing sub-charts created by the community.

Helm is essentially a package manager. It does not follow controller pattern and therefore will not monitor the current status of deployment. Helm has other limitations compared to Operator. For example, as a templating scheme, it reaches limitation when dealing with complex logic, even with the help of its helper functions. It is also hard to reason through the template code when we have to troubleshoot a deployment. Refer to this blog post for the author’s experience with Helm.

If we want our deployment to be fully declarative and continuous, then we will follow the Operator pattern by using a Kubernetes Operator. When we have many workloads of different levels of complexity, we can combine them with GitOps tool. Operator is one of the underlying technologies behind GitOps.

Workload profile	Just Installation	Installation and Maintain Status
Single stateless workload	Helm or Kustomize	Operator (using Ansible or Helm)
Single stateful workload	Helm or Kustomize	Operator (using Golang)
Multiple workloads	Helm (e.g. build parent chart)	GitOps in combination with Operator, Helm and Kustomize

The table above helps refine deployment requirement. It’s not a recommendation, but rather a model of analyzing deployment requirement.

FluxCD: Continuous Deployment with GitOps

Sat, 15 Jan 2022 18:49:00 -0400

This post explains why I land on FluxCD GitOps for my project. Let’s star

Background

In the Korthweb project, I landed on Istio for the Ingress Gateway technology. I first attempted to expand the orthanc Helm Chart to bring Istio as dependency (sub-chart). One of the external chart for Istio gateway needs to be referenced multiple times (for ingress and egress). However, it cannot even be used as dependency (sub-chart) because of this issue. Istio didn’t re-introduce Helm Chart as a supported deployment until September 2021. So I’m not too confident about it.

This leads to a second thought of the “Big Helm Chart” approach to deploy all tiers in Korthweb. Helm is based on templating, and having two layers of charts brings complexity. I do need to bring a number of Helm charts together, but not necessarily by another Helm Chart.

The Kubernetes documentation also mentions Kustomize as an alternative to Helm. Below is a quick exploration of it.

Kustomize

A challenge with managing declarative object is to organize numerous manifest files to maintain consistency and readability. Kustomize is a standalone tool to customize Kubernetes objects through a Kustomization file. This page provides a good list of fields that can be used:

vars and replacements: copy fields from one source into any number of specified targets.
namePrefix, namespace, nameSuffix: customize namespace and names.
configMapGenerator, secretGenerator, and generatorOptions: create configuration entries from literal, files, or environment variables.
resources: indicates another kustomization directory (e.g. as base)
patches (also called overlays) add or override fields on resources.
patchesStrategicMerge: modifies values in known (loaded) resources
images: modifies the name, tags and/or digest for images, without creating patches.
openapi: use Kubernetes OpenAPI data to get merge key and patch strategy information about resource types.

The base -> overlay pattern ensures readability as well as portability. Typical use pattern is create one base kustomization, and several overlay kustomizations each representing an environment, such as dev, qa and production. Here is a good tutorial.

Helm and Kustomize are two approaches to deploy Kubernetes workload. Both tackle the challenge of managing many YAML declarations. Helm is template driven, and is commonly used by application developers as a means of packaging application releases while orchestrating the dependencies. Kustomize follows a base-overlay pattern, and is more commonly used by cluster operators for re-using manifests across multiple environments (e.g. dev, staging and prod).

In my use case, my deployment needs both. I need third party Helm chart to configure components such as PostgreSQL for HA. I also need Kustomize for the orthanc workload. With the need for both, there should be a higher level deployment technology that ingrate with both mechanisms.

GitOps

GitOps is originally brought up by Weaveworks in 2017. It is a methodology to deploy workload continuously, using a Git repository as source of truth. The point of GitOps is not about tooling, or specific platform, but rather to ensure the workload state matches the declaration in repository. For example, you can implement GitOps with Ansible for VM environment.

When it comes to managing Kubernetes workload, a GitOps tool must handle the challenge with managing many YAML declarations. Therefore most GitOps tools seek to support many well-adopted mechanisms such as Helm and Kustomize as discussed above, instead of simply taking an enormous amount of raw YAML declarations. The amount of deployment mechanisms supported, is a key indicator of how powerful a GitOps tool is.

The most notable tools are ArgoCD and FluxCD. Both are currently CNCF incubating projects. ArgoCD is powerful with many tools supported, such as Kustomize, Helm, Ksonnet, Jsonnet, etc. It also contains a user interface and aims to manage an entire deployment workflow. On the other hand, FluxCD has controllers mainly for Kustomize and Helm, with a jsonnet extension from third party. This post draws a comparison of them (along with JenkinsX) based on earlier versions (from mid 2020). There is even a standard (OpenGitOps) in CNCF landscape but still at Sandbox level.

For Korthweb project, I chose FluxCD. It is simple, and provides just enough types of controller for what I do.

FluxCD

The diagram below illustrates the components:

The configurations starts with a bootstrapping process, which creates directory in Git repository (if not exist), and installs flux-system components in the target Kubernetes cluster. The sync process starts as soon as bootstrapping is completed. The process in charge of syncing declarations to target cluster, confusingly, is also called Kustomization. Therefore there are two Kustomizations. According to the FAQ on FluxCD website:

There are two Kustomization types. the kustomization.kustomize.toolkit.fluxcd.io is a Kubernetes custom resource while kustomization.kustomize.config.k8s.io is the type used to configure a Kustomize overlay. The kustomization.kustomize.toolkit.fluxcd.io object refers to a kustomization.yaml file path inside a Git repository or Bucket source.

Inside of the Git repository, with a kustomization.kustomize.toolkit.fluxcd.io obejct, the flux-system points to Kustomization file (representing kustomization.kustomize.config.k8s.io object) at root level. The kustomization.yaml file organizes resources in the same directory. A kustomize directory may also reference other kustomize directory, forming a hierarchy.

Implementation

FluxCD has a command “flux check –pre” to check the prerequisite, such as kubectl. The code is stored in the GitOps directory of Korthweb repository.

To configure deployment, we need to first create a personal access token. For GitHub, here is the instruction. Export the token to environment variable, and launch bootstrapping:

$ export GITHUB_TOKEN=xxx_yyy55555XXXodr7ABBBB234CCccw
$ flux bootstrap github \
 --owner=digihunch \
 --repository=korthweb \
 --branch=main \
 --personal \
 --path=gitops/environment/dev

A deploy key is configured during the bootstrapping process. As soon as bootstrapping is completed, the sync (aka kustomization, or reconciliation) has started, which can be monitored using:

flux get kustomizations --watch

Running this command without –watch switch returns the overview of all kustomizations. Once reconciliation is completed, it should display something like:

NAME 	READY	MESSAGE 	REVISION 	SUSPENDED
application 	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False
dependency 	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False
flux-system 	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False
infrastructure	True 	Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7	False

FluxCD introduced a number of CRDS. For example, to check configured source repository, check gitrepositories CRD:

kubectl get gitrepositories -n flux-system

To check the detail of one kustomization (e.g. infrastructure), check kusomization CRD in flux-system namespace:

kubectl -n flux-system get kustomizations flux-system -o yaml | less

Other commonly used custom resources include:

helmcharts
helmreleases
helmrepositories

A Helm release can be created imperatively, for example:

$ flux create source helm istio \
 --interval=1h \
 --url=https://istio-release.storage.googleapis.com/charts
$ flux create helmrelease istio-base \
 --interval=1h \
 --release-name=istio-base \
 --target-namespace=istio-system \
 --create-target-namespace=true \
 --source=HelmRepository/istio \
 --chart=base \
 --chart-version="1.12.0" 


$ flux create helmrelease istiod \
 --interval=1h \
 --release-name=istiod \
 --target-namespace=istio-system \
 --source=HelmRepository/istio \
 --chart=istiod \
 --chart-version="1.12.0" \
 --values=istiod-values.yaml

However, in a GitOps approach, they should be stored as code (use –export to export declaration). For example, the infrastructure kustomization keeps HelmReleases for installing Istio and PostgreSQL. This kustomization is referenced by a Flux Kustomization from higher level.

You can also manually reconcile one of the kustomizations (or other CRDs) with flux reconcile command, for example:

flux reconcile kustomization dependency

Since Nov 2021, FluxCD (0.20) supports reconciliation based on server-side apply. This increases performance and help address issues such as applying large config map, which is the equivalent of adding –server-side flag to the kubectl apply command.

Limitation

Troubleshooting the FluxCD repo can be involving and counter-intuitive. I had to commit a lot of changes to the repo because it serves as source of truth. Even though the commits can be made to a branch, it still involves a lot of code pushes. Traditionally I commit a change after testing. In the GitOps workflow, I commit a change then to test.

More flexible troubleshooting options are still to be desired. For example, there is no way to run one (FluxCD’s) Kustomization object at a time (and disable the rest), unless you remove their YAML files from the repo.

The next limitation is the ordering of resources in Kustomization. Arguably this is a limitation from Kustomize, instead of FluxCD’s. For example, I need the following manifest to be executed:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: selfsigned-issuer
spec:
 selfSigned: {}

The resource is a CRD that needs to be first installed using Helm chart (cert-manager). However there’s no way to control the sequence (Helm release executed first before declaration using CRD). Although there’s some workaround, it is not convenient. Alternatively, we can separate the resource creation and CRD creation into separate kustomization objects with dependency relationship, as suggested in this example.

Summary

Deployment in Kubernetes can get complicated with a lot of manifests. Helm, Kustomize and the like are means to handle the complexity due to numerous manifests. GitOps tools such as FlexCD brings these tools under a single framework, and more importantly, implements the idea of using Git repository as source of truth for continuous deployment.