observability on Digi Hunch

Kick the tires on ArgoCD

Sun, 10 Jul 2022 00:10:00 -0400

Background

In January, I wrote about FluxCD, and adopted it in the Korthweb project. I like the simple design of FluxCD, and I am comfortable with commands without using a web UI. Half a year later, I am re-visiting this choice, with ArgoCD in mind.

After reading numerous recent posts that compare the two (such as this one from the new stack), I started to give more thoughts on ArgoCD, for a couple reasons. First, ArgoCD has more contributing companies and public references. Red Hat adopted Argo CD as the underlying technology for OpenShift GitOps (and Tekton for pipeline). Second, ArgoCD has a mature UI. In corporate collaboration, especially with those not well-versed with command line, a UI is extremely helpful. Bundled with the UI, Argo CD also uses its own RBAC independent of Kubernetes RBAC, making it a great choice for continuous deployment for enterprise applications.

User Interface

You can choose to install just the core components, without UI, SSO and multi-cluster features. To me, it does not make sense because those features are exactly the reason to choose Argo CD. Argo CD also has an eponymous CLI tool, similar to flux for Flux CD. Since Argo CD uses its own RBAC independent of Kubernetes RBAC, Argo CD will have its own credential. The default username is admin and password needs to be retrieved from Secrets. In the tutorial we use this user’s credential to interact with the target cluster. This is different from Flux where the client simply uses kubectl configuration.

In the “Getting Started” guide, we create a namespace “argocd” and install it with manifests. Alternatively, we can use CLI tool to install it. With Argo CD, we can consider hosting it on a different port than the workload’s port, such as 8443. This port serves as “GitOps management port” separated from the application port (e.g. 443) for business workload. Just like any application on Kubernetes, we usually need to configure an Ingress on the management port.

In the argocd namespace, we can see a few services. The application set controller acts as the main controller that reconciles between actual state and desired state. There is also a redis server, for storing data in Argo CD. Another service to note is the dex server, indicating that the SSO capability is provided by the Dex project.

Features

In FluxCD, they use a CRD Kustomization with kustomize.toolkit.fluxcd.io/v1beta2 as version. This name is confusing. Luckily, in ArgoCD, the CRDs are Application (with argoproj.io/v1alpha1 version) and ApplicationSet (with argoproj.io/v1alpha1 version). The controllers monitors CRs created with these CRDs. ArgoCD defines Application as a group of Kubernetes resources as defined by a manifest. Argo CD Application resource deploys resources from a single Git repository to a single destination cluster/namespace. On the other hand, ApplicationSet uses templated automation to create, modify, and manage multiple Argo CD applications at once. ApplicationSet controllers monitors ApplicationSet resources. We can define a template within the declaration of an ApplicationSet and it allows for parameter substitution.

Another key capability for a GitOps utility is the support of multiple templating tools. FluxCD supports Helm and Kustomize. So do Argo CD. In addition, ArgoCD also supports jsonnet (ksonnet is not supported anymore). Jsonnet is a templating tool with many useful operators. Kubectl can directly consume Jsonnet’s JSON output as if it were YAML. In FluxCD, there are different CRDs for Helm and Kustomize (HelmRelease and Kustomization). In ArgoCD, the Application CRD covers Helm, Kustomize, and Jsonnet by embedding them as attributes in the declaration. This makes it even simpler than Flux CD.

Secret Management

It’s a big no-no to put secret in a code repository. Therefore any GitOps solution must solve the problem with secret logistics. What baffles me is ArgoCD remains un-opinionated on this matter. There appears to be some context for this stance. As a result, ArgoCD only points to a few third-party secret management solutions.

Secret logistics is a tricky problem with GitOps workflow. One of the techniques is sealed secret. In the repo we store the secret as encrypted by public key. The controller keeps the private key which decrypts the secret at the time of deployment. A few engineers don’t find it feasible for operation, as explained in this post and this post.

An alternative is to use external secret operator. Such operators will be able to sync a secret from external secret store such as Hashicorp Vault, AWS secret manager, Azure Key Vault, etc. This is much neater because we only store reference to secret in the repository. However, when the cluster assesses external secret store, it still requires either a secret, or a permission.

Another technique that drawing attentions is using sops for secret and kustomize-sops to integrate with ArgoCD. Mozilla’s sops (SecretOPerationS) project releases a binary utility called sops to encrypt and decrypt YAML manifest. It can encrypt only the values in YAML manifest and not the attributes (keys). We use sops to encrypt YAML files in order to store them in Git repo. We also use sops to decrypt YAML right before we deploy it with kubectl. The key pair for encryption and decryption can be stored in Azure Key Vault, AWS KMS, GCP KMS, and even age and pgp. Here is a simple tutorial.

Example

In a few command we can configure a minimal example. I first use KinD or Minikube to create a cluster following the steps in real-quicK-cluster repo. Then we configure an application using YAML manifest.

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
$ kubectl port-forward svc/argocd-server -n argocd 8443:443

From MacOS, I can now browse to https://localhost:8443/ for ArgoCD UI, and login with admin user and the password as printed above. We can then use the UI to create an Application. Alternatively, we can configure an application using CLI utility:

$ brew install argocd
$ argocd login localhost:8443
$ argocd app create guestbook --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace default
$ argocd app sync guestbook

On the GUI we should see the application is sync’ed.

On the left panel, click on the gear icon for settings. Under Clusters, we can see the cluster URL, which we use as dest-server value above. We can add more clusters here.

Pick the right open-source project

My experience with ArgoCD and FluxCD raised an issue to reflect on. For the same problem, there are multiple choices of open-source project. What would be a good methodology to choose the right open-source technology? I would first look at the following factors:

History
Community size
License model
Enterprise support
Governing model

The history and community size give a good idea of the technical maturity level. The License model has legal implications as to where you can use the technology. License model can change even after the project has been launched and even become not open-source any more. One example is Elasticsearch license change in Jan 2021, from the very open APLv2 to under dual license of both Elastic License (by Elastic) and Server Side Public License (SSPL, introduced by MongoDB). Neither license is certified by OSI as compliant with open source definition. For enterprises, apart from licensing, I’d also consider if any organization provides commercial support for the open-source product.

What is often neglected is the governing model. The governing model determines how product decisions are made that will shape the future of the project. For example, CNCF is a vendor neutral foundation for cloud computing and if an organization contributes a project to CNCF, then the member projects have to align with CNCF’s governing practices. Google also introduced an OUC (Open Usage Common) governing model.

Another good way is to just follow Red Hat’s choice. Red Hat built its business model around open-source technologies. They have to pick open-source project to provide enterprise level support so presumably they have a rigour selection process that one can piggyback off. Products supported by Red Hat are composed of open source components often vetted by multiple upstream communities, and changes made to these components are pushed to their respective upstream projects, often before they land in a supported product from Red Hat. Here are more insights in this business model.

Had I gone with the Red Hat rule, I would have picked Argo CD in the first place because that is the base of OpenShift GitOps!

Verdict

ArgoCD and FluxCD are tools for GitOps. However, just using a GitOps tool does not guarantee that all workload are deployed continuously. First, these GitOps controllers cannot replace Operators, which has domain knowledge about how to orchestrate their workloads. Second, resources not directly managed by these operators (such as those installed by Helm) are not being monitored continuously. For continuous deployment over all workloads, I recommend use operator to install third party applications, and use Argo CD or Flux CD to manage the operators.

GitOps makes use of the controller pattern to manage deployment in a continuous matter. Controller works closely with CRDs. ArgoCD uses similar set of CRDs to manage continuous deployment. What sets it apart is the user friendly web UI, the IAM integration and multi-cluster capabilities. These capabilities make ArgoCD well adapted in enterprise IT eco-system.

Chaos Mesh – Cloud Native Chaos Engineering

Thu, 30 Jun 2022 21:49:00 -0400

In this post, we discuss the resilience test problem and why chaos mesh emerged. Then we go over a lab of chaos mesh with a few experiments.

The Problem

I used to support a server application installed in customer’s data centre. The server application receives data from client application, cleanse the data and put them on customer-managed storage. Unfortunately, the customer-managed storage system hang up from time to time. When this happens, the application does not get an I/O failure to begin with so it continues to serve client traffic with many threads pending for I/O completion. All the threads will fail eventually after a long, unresponsive pause.

What is really bad, is that the server application first acknowledges the client application of the recipient of their data, and then asynchronously archive them to the customer’s storage. In this case, the client applications think they securely send out the data upon receiving acknowledgement. However, the server application lost the data afterwards because it does not interact with storage system in a fail-safe fashion. As a result, I often find myself in the business of restoring client data from the cache directory on the servers, and grumbling about how the quality process allowed this issue slipped into production.

To be fair, this is not an easy catch in the quality assurance process. It is caused by a hardware issue, which is technically the customer’s own problem. However, it is the vendor’s responsibility to design fault-tolerant software. Further, if we put on the site reliability engineering goggle, we see vendor’s software and customer’s hardware as a whole, instead of two disparate silos. The problem is however, how we emulate this kind of fault in house?

Chaos Engineering

A software engineer could have unmounted the storage while the software is active during test. This approach is faulty because it has a couple of issues. First, unmounting the storage does not produce exactly the same symptom. Unmounting the storage in most cases gives a clear failure upfront and the application would have caught it. What we need is a delay in storage long enough to cause a time out. Second, we’d have to perform this activity with an operating system command with sufficient privileges. The software being tested usually do not have such privileges. We need to systematically inject failures of particular kind.

Chaos are interruptions along with the cascading effects, that happen in production but are impossible or extremely difficult to emulate in lower environments through automated quality process. To improve robustness, we need the capability to emulate various failure scenarios in a systematic manner. Chaos Engineering aims to improve such capability.

Chaos Engineering emerged as a discipline to improve software’s ability to tolerate failures while still delivery adequate quality of service. This is known as the resiliency of the software. Suppose the storage has 99.9999% SLA, or 31 seconds of downtime every year. Chaos Engineer requires the functional parts of our software do what they can to protect the data, during the 31 seconds of turbulence each year, instead of just falling apart. In regulated industries, this is critical. Resilience testing is difficult in quality assurance process, but we can’t avoid it.

By definition, Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. It is particularly useful in distributed system, where network faults are normal. The one-page chaos engineering website has a good summary of such intent. At Netflix, they built Chaos Monkey, a tool to randomly cause failures on certain computing instances in the cloud. It is one of the first Chaos Engineering tools and it is a big step forward. Chaos Monkey can perform one type of experiment (faulty server) and requires writing custom code.

Chaos Mesh

Moving to Kubernetes platform, where everything operates on network, it is even more important to have a suite of resilience testing tools. Thankfully, we have Chaos Mesh, a cloud-native Chaos Engineering platform that helps platform engineers and software engineers simulate faults in a variety of scenarios. There are other chaos engineering tools (such as Litmus, also a CNCF project) but I will just focus on Chaos Mesh in this post because it has longer development history.

With Chaos Mesh, we can run experiments to simulate certain types of faults, including faults on Pod, Network, File I/O, DNS, Time, JVM, Linux Kernel, HTTP. We can also emulate incidents on cloud platforms (Azure, AWS, GCP) and emulate stress. We can even orchestrate multiple experiments into a workflow. Experiments can start ad hoc or on schedule. In addition to Kubernetes platform, the Chaos Mesh team also provides a tool Chaosd to emulate faults on a physical nodes. In the rest of this post, we will install Chaos Mesh and perform four simple experiments using the bookinfo sample application from Istio’s installer.

Demo Platform

In this lab, we install chaos mesh on Minikube on my Mac environment. I have install the platform in a particular way so the experiments will all be successful. I use this script in my real-quick-cluster project to create minikube cluster. This script configures a 3-node minikube cluster, with containerd as CRI, and enables plugins metallb and metrics server. Metallb provides a load balancer for the cluster. Metrics server allows scaling based on metrics, a test scenario for stressor in Chaos Mesh.

I will also install Istio in this lab, because I use it for Ingress and service routing. I also use a sample application called bookinfo which comes with Istio code repository. It is highly recommended to have an ingress as it is used in production, although you may use your own choice of ingress and service routing tool such as ingress-nginx. To install istio, we run this simple script:

#! /bin/bash
curl -L https://istio.io/downloadIstio | sh -
export PATH=$(realpath istio*/bin):$PATH
if istioctl x precheck; then
 echo ready to install istio and label namespace for istio-injection
 istioctl install -f istio-operator.yaml -y --verify
 kubectl label namespace default istio-injection=enabled
else
 echo failed precheck
 exit 1
fi

Below is the content of istio-operator.yaml

apiVersion: install.istio.io/v1alpha2
kind: IstioOperator
metadata:
 name: istio-operator
spec:
 profile: default
 hub: docker.io/istio
 tag: 1.13.2
 namespace: istio-system
 meshConfig:
 accessLogFile: /dev/stdout
 components:
 pilot:
 k8s:
 hpaSpec:
 maxReplicas: 7
 minReplicas: 1
 nodeSelector:
 beta.kubernetes.io/os: linux
 overlays:
 - kind: Deployment
 name: istiod
 patches:
 - path: spec.template.metadata.labels.version
 value: 1.13.2
 ingressGateways:
 - name: istio-ingressgateway
 enabled: true
 label:
 istio: ingressgateway
 k8s:
 hpaSpec:
 maxReplicas: 5
 minReplicas: 1
 service:
 ports:
 - name: http
 port: 80
 targetPort: 20080
 protocol: TCP
 - name: http-dashboard
 port: 8080
 targetPort: 28080
 protocol: TCP
 overlays:
 - kind: Deployment
 name: istio-ingressgateway
 patches:
 - path: spec.template.metadata.labels.version
 value: 1.13.2
 - path: spec.template.spec.containers[name:istio-proxy].lifecycle
 value:
 preStop:
 exec:
 command: ["sh", "-c", "sleep 5"]

For more about istio installation, refer to my previous post. Next, we’ll create the namespaces, install bookinfo and install chaos mesh using Helm:

kubectl create ns chaos-testing
kubectl create ns bookinfo

kubectl -n bookinfo apply -f istio/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl -n bookinfo apply -f istio/samples/bookinfo/networking/bookinfo-gateway.yaml

helm install chaos-mesh chaos-mesh/chaos-mesh -n chaos-testing --version 2.2.0 --set dashboard.service.type=ClusterIP --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock

As soon as we install bookinfo, we should be able to access it at http://192.168.64.16/productpage, suppose the IP address is 192.168.64.16. We will simply use HTTP because certificate is not the point of this demo. Note that when we install Chaos Mesh we have to specify the runtime (containerd). Otherwise, we might run into issues when running experiments (e.g. “unable to flush ip sets” for network faults). Before fault injections, we’ll start a curl command and observe the output:

while true; do
 curl -I http://192.168.64.16/productpage --connect-timeout 4 --max-time 5
 sleep 5
done

We can watch the output of the command above as we inject each of the following faults later on.

Chaos Mesh UI

Since we installed chaos mesh as cluster IP service, we need to expose it using Istio Gateway and Virtual Service with the following manifest. If you have your own choice of Ingress, use a corresponding ingress resource.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: chaos-dashboard-gateway
 namespace: chaos-testing
spec:
 selector:
 istio: ingressgateway
 servers:
 - port:
 number: 8080
 name: http
 protocol: HTTP
 hosts:
 - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: chaos-dashboard-vs
 namespace: chaos-testing
spec:
 hosts:
 - "*"
 gateways:
 - chaos-dashboard-gateway
 http:
 - route:
 - destination:
 port:
 number: 2333
 host: chaos-dashboard

Once we configured Virtual Service and Gateways, and waited all Pods in chaos-testing namespace to come up, then we should be able to access the web portal of chaos mesh at 192.168.64.16:8080. Follow the on-screen instruction to fetch the token for access at Cluster level.

Chaos Mesh Web Portal

Now, we can create experiments, either using web portal, or using their CRDs. We will use CRDs in this demo.

Network Latency Injection

In this section we artificially introduce a network latency of 3 seconds for a period of 45 seconds. It takes effect as soon as we create the NetworkChaos CRD as below:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
 name: network-delay
spec:
 action: delay # the specific chaos action to inject
 mode: one # the mode to run chaos action; supported modes are one/all/fixed/fixed-percent/random-max-percent
 selector: # pods where to inject chaos actions
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage
 delay:
 latency: '3s'
 duration: '45s'

Watch for the curl output, which normally returns 200 code every 5 seconds, now fails for about 45 second window.

After the window of latency injection, the curl test returns normal. During the experiment, we can check the status of the fault injection by describing the NetworkChaos object. Alternatively, we can look at Experiments and Events in Chaos Mesh web portal. When completed, we can delete the NetworkChaos object. Alternatively, we can archive the experiment from web portal.

HTTP Failure Injection

The NetworkChaos API injects faults at Pod’s network layer. HTTPChaos API introduces HTTP error. We can configure an HTTPChaos object following the instruction:

apiVersion: chaos-mesh.org/v1alpha1
kind: HTTPChaos
metadata:
 name: test-http-chaos
spec:
 mode: all
 selector:
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage
 target: Request
 port: 9080
 #method: GET
 path: '*'
 abort: true
 duration: 2m

This object intercepts HTTP calls to the Pods as labelled and aborts the request. If we monitor the curl output at the same time, we can see that it starts to receive 503 error as soon as we injected the fault.

Once we remove the HTTPChaos object, the return code becomes 200 again.

Pod Failure Injection

We can emulate a fault that kills a Pod with the following manifest:

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
 name: pod-failure-example
spec:
 action: pod-kill
 mode: one
 duration: '30s'
 selector:
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage

In the mean time, watch for the Pods in the bookinfo namespace, we’ll notice the old Pod being terminated.

Because the Pod is managed by a replica set, the replica set brings up a new Pod to ensure the desired number of replicas.

Stressor

In this experiment, we add an HPA (Horizontal Pod Autoscaler) to the productpage deployment, emulate a memory stress to the Pod and watch it scale up. Before the testing, we need to first configure a resource request for memory for the deployment. To configure memory request, edit the deployment productpage-v1 by adding memory request:

Then we add an HPA as below:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
 name: productpage-deploy
 namespace: bookinfo
spec:
 maxReplicas: 4
 minReplicas: 1
 scaleTargetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: productpage-v1
 metrics:
 - type: Resource
 resource:
 name: memory
 target:
 type: Utilization
 averageUtilization: 70

Now if we describe the HPA, we shall see that HPA is active:

Then we create a StressChaos object as below:

apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
 name: memory-stress-example
spec:
 mode: one
 selector:
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage
 stressors:
 memory:
 workers: 2
 size: 128MiB

It will take some time for the HPA to detect the memory stress and scale up. We can tweak this with scaling policy. After scaling activity, the HPA status will report with the new desired number of replica:

We can also confirm that by checking the deployment size. If we want the size to reduce after we remove the stressor, we need to configure HPA accordingly with scale down triggers.

Chaos Mesh uses stress-ng utility to emulate memory or cpu stress on a Pod.

Summary

Chaos mesh is a cloud native resilience testing tool. It supports many fault experiments not only on Kubernetes, but also on virtual machines. I wish I came across this tool sooner in my professional life. Had I had chaosd in the past, I could have incorporate it in our quality assurance process on Linux platform. Without Chaos Mesh, we’d have to use different utilities for different types of faults (e.g. stress or stress-ng for cpu and memory stress). In the domain of platform engineering on Kubernetes, it helps not only software developers, but also platform engineers with correct configuration for workloads on Kubernetes. In this post we explored four types of faults. For more about all fault types and their attributes, go to their documentation for more details.

Istio Operation Gotchas

Sat, 19 Mar 2022 11:09:00 -0400

In this post I discuss a few aspects when putting istio in operation.

Installation

Istio installation can be confusing, due to architectural and guideline changes as well as renaming of operator CRDs since its release, and especially since 2020. This left lots of information outdated on the web, adding to Istio’s perceived complexity.

Currently, the recommended installation methods are istioctl or Helm. Using Istio Operator manifest (with out istioctl) is discouraged. As to the Helm chart installer, it was once deprecated (around early 2020), but was later re-introduced in the fall of 2021. This back-and-forth had caused some aversion. As of Nov 2021, their re-introduced Helm charts version dropped alpha tag. The Helm repo consists of separate Helm charts, for Istio CRD (base), control plane (istiod), each gateway and CNI respectively. A typical deployment therefore requires multiple Helm Releases (example here). Istio document still considers Helm support as alpha, so I assume the most reliable method to install Istio is istioctl.

The istioctl utility can be used with many options to customize Istio install. For example, we can supply a YAML declaration input (to -f switch) to customize installation behaviours. The YAML file declares a CRD. Two different CRDs have been used: IstioOperator and IstioControlPlane. According to this blog and this post, since Istio 1.5 in early 2020, we’re supposed to IstioOperator CRD exclusively. The IstioControlPlane CRD is left only for legacy support. As stated in the current documentation:

The istioctl command supports the full IstioOperator API via command-line options for individual settings or for passing a yaml file containing an IstioOperator custom resource (CR).

With IstioOperator CRD, we still have a number of options to tweak the install behaviours. Here is a summary of potential options:

use IstioOperator API via IstioOperator CRD (without using “values” or overlay fields)
specify an attribute value in argument, including a pre-built profile e.g. –set meshConfig.accessLogFile=/dev/stdout, –set profile=demo
use K8sObjectOverlay by using “k8s/overlays/patches” field in IstioOperatorCRD
use Helm API by using “values” field in IstioOperatorCRD

The key-value specified in –set switch overrides the same key-value supplied in the IstioOperator CRD. So option 2 overrides option 1. The value for profile can also be empty if you’d rather start from scratch. However too many –set switches makes the command wordy so we should build our own IstioOperator CRD

For option 4, the document hyper-links Helm API to a section from version istio 1.4, and I appears to exist only for legacy (pre-2020 Helm support) compatibility. Option 3 (K8sObjectOverlay) would be helpful when a field cannot be conveniently customized with option 1 and we have to patch the object like in Kustomization.

So the most practical approach is IstioOperator CRD for per-component customization, potentially with K8sObjectOverlay. No matter which option, istioctl compiles the installation manifest before applying it against Kubernetes API. This manifest can be previewed using “istioctl manifest” command, so that you can take a look before installation.

Thank you Isito for so much confusion just to land on a working installation method. Below is the content of az-istio-operator.yaml file that I use for my installation:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
 name: istio-install-customization
spec:
 profile: default
 hub: docker.io/istio
 tag: 1.13.1
 revision: 1-13-1
 namespace: istio-system
 meshConfig:
 accessLogFile: /dev/stdout
 outboundTrafficPolicy:
 mode: "REGISTRY_ONLY"
 components:
 pilot:
 k8s:
 hpaSpec:
 maxReplicas: 7
 minReplicas: 1
 nodeSelector:
 beta.kubernetes.io/os: linux
 ingressGateways:
 - name: istio-ingressgateway
 #namespace: istio-ingress
 enabled: true
 label:
 istio: ingressgateway
 k8s:
 hpaSpec:
 maxReplicas: 11
 minReplicas: 1
 serviceAnnotations:
 service.beta.kubernetes.io/azure-load-balancer-internal: "true"
 service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "my-lb-subnet"
 egressGateways:
 - name: istio-egressgateway
 enabled: true

I can preview the install, run the install and validate installation status:

$ istioctl manifest generate -f az-istio-operator.yaml
$ istioctl install -f az-istio-operator.yaml -y --verify
$ istioctl verify-install -f az-istio-operator.yaml
$ kubectl -n istio-system get IstioOperator installed-state-istio-install-customization-1-13-1 -o yaml | less

Note that in the IstioOperator declaration I marked the revision. This is helpful when I run multiple versions of control plane (e.g. during upgrade). I can check revisions of istiod with:

$ istioctl x revision list

We can delete installed istio components

$ istioctl x uninstall -f az-istio-operator.yaml
$ istioctl x uninstall --revision 1-11-5

In practice, it is helpful to use separate operators each for a different component (pilot, ingressGateways, egressGateways). This makes maintenance and upgrade easier.

Debugging

The documentation has a page for common problems that one needs to be familiar with. It covers not only problems, but also steps to troubleshoot each kind of problem (e.g. authorization policy).

Although istioctl is pretty confusing as an installation tool, it is a good utility for many troubleshooting activities. We should probably add its path to PATH environment variable and add the export command (e.g. export PATH=”$HOME/istio/bin:$PATH“) to .zshrc or .bashrc. Istioctl has a few useful subcommands, some of which are only available as experimental and therefore needs to be following an x. Some common commands are given below:

To analyze Istio problems:

$ istioctl analyze -n istio-system

To look at proxy configuration of an Envoy instance at different levels, use proxy-config sub-command or pc for shorthand:

$ istioctl proxy-config <clusters|listeners|routes|endpoints|bootstrap|log|secret|stats>
$ istioctl pc cluster deploy/istio-ingressgateway -n istio-system # see what Envoy cluster an ingress gateway knows about
$ istioctl proxy-config log deploy/httpbin --level "rbac:debug"
$ istioctl pc log <pod_name> -n <namespace> --level connection:debug
$ istioctl pc secret -n istio-system deploy/istio-ingressgateway # check certificates loaded to a gateway
$ istioctl proxy-config listeners deploy/istio-ingressgateway -n istio-system # query envoy listener configuration
$ istioctl pc routes deploy/istio-ingressgateway -n istio-system --name http.8080 # query envoy route configuration

The last two commands set logging level to the specified workload. If we want to set logging level at mesh level, we can use these commands:

$ istioctl admin log --level authorization:debug
$ istioctl admin log

To look at synchronization status of each envoy in the mesh, use proxy-status sub-command, or ps for shorthand:

$ istioctl ps # ensure data plane is in sync

If an item for a workload shows STALE instead of SYNCED, it means that the configuration has not been pushed from control plane to that instance of Envoy proxy. Check if the Istio configuration change is valid. If the system is newly installed and there is no ingress or egress gateway resources declared, the RDS column for ingress or egress may show “NOT SENT”. The far right column displays the version of istiod connected.

To describe applied istio config:

$ istioctl describe <pod|service>
$ istioctl describe po workload1 -n my-workload # detect misconfigurations on workload

To view dashboard:

$ istioctl dashboard <envoy|grafana|prometheus>

To check authorization policy on a Pod,

$ istioctl x authz check mypod -n workload

To validate istio configuration in a file:

$ istioctl validate -f resource_authorization_policy.yaml

Sometimes we need to turn on access logging just on the envoy proxy on the Gateway Pod. In that case, we will need to apply Istio’s Envoy filter object. This filter is applied to Pods labelled as gateway. It patches the existing filter chain with the additional defined in the manifest:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
 name: hcm-gw-access-log
 namespace: istio-system
spec:
 workloadSelector:
 labels:
 istio: ingressgateway
 configPatches:
 - applyTo: NETWORK_FILTER
 match:
 context: GATEWAY
 listener:
 filterChain:
 sni: demo.digihunch.com
 filter:
 name: "envoy.filters.network.http_connection_manager"
 patch:
 operation: MERGE
 value:
 typed_config:
 "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
 access_log:
 - name: envoy.access_loggers.file
 typed_config:
 "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
 path: /dev/stdout
 format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"

The object above enables access logging on the Gateway Pods only, without impacting other Envoy proxies.

Multi-tenancy

In large enterprises, Istio is usually installed by a platform/operation team. The entire platform is shared by multiple application teams. Istio should not be seen as a responsibility of a single party. It is necessary to break down the Istio CRDs and define the responsibility of each CRD. For example:

Gateways are placed in the istio-system namespace (or a dedicated istio-ingress namespace in some case). Platform team can decide whether a Gateway is considered a shared infrastructure, or each tenant (application team) uses their own Gateway. In the servers/hosts field of Gateway declaration, add namespace before host to suggest that the routing behaviour of that host must be defined in a certain namespace.
Virtual Services can be managed in two models as well. They can be managed individually and are all placed in each tenant namespace. Alternatively in a shared responsibility model, Virtual Service can be created in istio-system namespace, and the processing of each match can be delegated to a Virtual Service in each tenant namespace, using the Delegate feature of Virtual Service. Another field that can help with multi-tenancy is the “gateways” field, you can specify a value of mesh to indicate the virtual service is available to the entire mesh.
Destination Rules are usually placed in each Tenant’s workspace
Peer Authentication also depends on the configuration. If a mesh level configuration is enforced, it is easier for the Platform Team to manage it and this should be the setup for new clusters. If for historical reasons Peer Authentication is enforced per tenant namespace, it can be delegated to each application team. It can make communication troubleshooting more complex.
Control Plane and observability workloads such as Kiali are usually the responsibility of Platform team.
Request Authentication and Authorization Policy should be the responsibility of individual application team.

In a multi-tenancy management model, Istio related resource should be subject to admission control policy, as well as scrutiny by the security and platform teams. Google Anthos has a good page on the constraint templates for Istio resources.

As to restricting traffic between namespaces, apart from the measures from this previous post, we can also use Sidecar CRD to restrict outbound traffic. As its documentation states:

Sidecar describes the configuration of the sidecar proxy that mediates inbound and outbound communication to the workload instance it is attached to. By default, Istio will program all sidecar proxies in the mesh with the necessary configuration required to reach every workload instance in the mesh, as well as accept traffic on all the ports associated with the workload. The Sidecar configuration provides a way to fine tune the set of ports, protocols that the proxy will accept when forwarding traffic to and from the workload. In addition, it is possible to restrict the set of services that the proxy can reach when forwarding outbound traffic from workload instances.

The documentation page also includes two examples. The first is a sidecar at the mesh level that restricts outbound traffic to the same namespace that the sidecar is in, and the istio-system namespace:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
 name: default
 namespace: istio-config
spec:
 egress:
 - hosts:
 - "./*"
 - "istio-system/*"

The second example overrides the mesh level default above, and allows egress traffic to three specified namespaces:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
 name: default
 namespace: prod-us1
spec:
 egress:
 - hosts:
 - "prod-us1/*"
 - "prod-apis/*"
 - "istio-system/*"

As to inbound control, we can expose a Virtual Service to other namespaces by using the exportTo field to specify which other namespaces the Virtual Service should be exported to. If no namespaces are specified then the virtual service is exported to all namespaces by default.

Sometimes we want a virtual services to expose to both outside of the mesh via Ingress, and within the mesh, and we hope to use the same hostname. For this requirement, we can use the ServiceEntry CRD. ServiceEntry enables adding additional entries into Istio’s internal service registry.

From Ingress to CRD: why my solution needs Istio Gateways on Kubernetes platforms

Wed, 29 Dec 2021 22:50:16 -0400

Update: also read my other article here on the different generations of ingress technologies.

In my Korthweb project I was researching for the best ingress mechanism for HTTP and TCP workload, both of which need to be secured. I started with Kubernetes Ingress but eventually decided to go with Istio Gateway. This blog post is about the justification. In this essay, I will make the distinction between Ingress and Gateway and explain why a Kubernetes platform needs the latter going forward.

The word ingress can be used either to indicate ingress resource (in conjunction with ingress controller) in the context of Kubernetes cluster, or more generally to indicate the technology (e.g. provided by service mesh) to direct north-south traffic, originated from the outside into the workload running within the cluster. I will use both meanings of the word ingress throughout the article.

Background

In my previous post, I discussed Kubernetes Service object and native Ingress. In short, a Service object addresses the problem of exposing a workload (Pod or Deployment), operating at layer 3-4. A ClusterIP type of service exposes workload within the cluster only, and therefore is only used by internal services that are consumed by workload in the same cluster. A NodePort type of service exposes workload to outside of the cluster, using the node’s IP address and port range available on the node, a great step forward but still quite inconvenient and subject to the limitation of using Node’s IP and Port. A LoadBalancer type of Service brings separate IP address and port for running a service. To back up this type of service, the platform has to provide a load balancer with its own IP address and port range to manage. The implementation is platform specific e.g. Metal LB for Minikube, Azure Load Balancer for AKS, NLB for AWS EKS. In this type of service, Node Ports still exists but are not exposed to the outside world. Instead, they are only exposed to the backend of Load Balancer, whose front-end port is exposed to the outside world. In both NodePort and LoadBalancer types of Service, the kube-proxy process plays a role on each node to direct traffic into NodePort to target Pods.

Kubernetes Ingress, on the other hand, targets issues above layer 4, for example, path-based routing and is therefore mostly used for HTTP and HTTPS traffic. It is implemented by an Ingress Controller and now there has been a diverse ecosystem of those ingress controllers. In many implementations (e.g. Azure Application Gateway Ingress Controller, AGIC), the ingress themselves usually come with a load balancer managed by themselves. This eliminates the need for a separate LoadBalancer type of Service just for L4 capability. As a result, Ingress (with both L4 and L7 capabilities) is usually deployed along with ClusterIP type Service. Here is an example. The declaration of the Ingress in this case usually contains numerous lines of annotations in order to communicate the specification to the implementation.

Requirement

Using Kubernetes Ingress along with Service of ClusterIP type seems perfect to address the majority of use cases. However, it has its blind spots. My Korthweb project deals with DICOM traffic (a protocol on top of TCP) over TLS as well as HTTP traffic, which can be broken down as:

The ability to proxy TCP traffic over an arbitrary TCP port
The ability to terminate TLS/SSL encryption for TCP traffic

The Ingress documentation of Kubernetes clearly states that: an Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

The stance of community-driven Nginx ingress controller can be found in the documentation, which suggests the use of Service object for arbitrary TCP port. There is some unofficial claims of workaround available but I’m not confident.

I also went through a number of other Ingress providers but to my disappointment, the only product that supports it seems to be Traefik:

Product	Requirement #1	Requirement #2
F5 driven Nginx Ingress Controller	Supported	There have also been requests for TLS termination for TCP traffic but the request has not been closed as of yet.
HA Proxy Ingress controller	Supported	There’s no mention of this in documentation.
Kong’s Kubernetes Controller (with TCPIngress CRD)	Supported	Documentation claims support through SNI-based routing. However, the example demonstrates it using self-signed certificate only. BYO cert not supported according to this GitHub issue.
Traefik Lab’s IngressRoute (with IngressRouteTCP CRD)	Supported	Documentation seems to suggest that it is supported by IngressRoute.

My general impression is that Req #2 isn’t very popular so the providers either don’t support it or delaying the implementation. Even if they do, the CRD used for TCP ingress is different than HTTP’s. For example with Traefik Lab, the CRD for TCP is IngressRouteTCP, and for HTTP it is IngressRoute.

Introduction to Gateway

In addition to functional shortages, according to this blog post, there are signs of fragmentation into different but strikingly similar CRDs and overloaded annotations. In Kubecon 2019, a group of contributors discussed the evolution of Ingress into Gateway. Below are the two key slides stolen from their presentation:

This sums up how the concept of Gateway is different from Ingress. Gateway is an instantiation of a given LB. It works along with Route to achieve the functions brought by an Ingress. Gateway as a resource type of its own makes management easier at L4-L6 by a separate team. Routing at L7 is offloaded to “Route” resource type.

This conceptual evolution gives rise to the Gateway API open source project managed by the SIG-NETWORK community. Gateway API is a collection of resources that model service networking in Kubernetes, including GatewayClass, Gateway, HTTPRoute, TCPRoute, Service etc. The aim of this initiative is to evolve Kubernetes service networking through expressive, extensive and role-oriented interfaces that are implemented by many vendors and have broad industry support.

The diagram above stolen from Gateway API website illustrate the management model for each resource.

Kubernetes Gateway Implementations

The Gateway API is still a fairly young initiative and all the reference implementations are either work-in-progress or in early stages. In the last section, we learned that in the journey from Ingress to Gateway, the standardization initiative comes a bit behind the implementation efforts. For many supporters, the natural strategy is to continue with existing proprietary ingress controller implementation, and retrofit their technology to the emerging Gateway API along the way.

Because of that, we can see many market players with multiple flavours of implementations, typically one that evolves from their original product offering, with higher adoption rate and maturity level, and one that conforms to Gateway API specification.

For example, Traefik Labs has its Gateway API implementation in experimental stage. Its own implementation consists of (standard) Kubernetes Ingress and Kubernetes IngressRoute (based on custom resource with support of advanced features such as TCP route). Another example is the HAProxy Ingress, a community driven ingress controller implementation for HAProxy. Starting from its version 0.13, it partially supports the Gateway API’s v1alpha1 specification. Here is the conformance statement. Contour as a CNCF project for ingress controller has support for Gateway API at alpha version. Kong’s Kubernetes Ingress Controller follows the same path.

The Gateway functionality is also provided as part of Service Mesh product offering, and we can see some reference implementation by service mesh providers. Istio has its own Gateway implementation but tries to adapt to Kubernetes Gateway API. Hashicorp Consul also claims to be building support for Kubernetes Gateway API.

Until the Gateway API project matures, it is not recommended to use Gateway implementations that conforms to it, unless you intend to be their Guinea Pig. For my project, I chose Istio Gateway.

Istio Gateway implementation

As discussed, when the Gateway standard is still in its infancy, I chose a non-standard implementation of Gateway, even though it may include some CRDs. Out of the many Gateway implementations, I choose Istio mainly for two reasons.

First, I need a Gateway as part of service mesh because service mesh provides many other features that are needed in the platform. One of the goals of Service Mesh is to provide commonly used features (traffic management, observability, security, extensibility) in a commodity layer on top of Kubernetes. Gateway is not the only thing I need out of this commodity layer. With one install of Istio, many common platform-level problems are also addressed (e.g. mTLS, traceability, etc). I’ve compared three major service mesh technologies as below:

Linkderd	Istio	Consul
– lightweight – uses linkerd2-proxy – the original service mesh – no Gateway implementation (deal breaker in my use case)	– feature rich – uses Envoy proxy – complex but getting better	– initially a service discovery and distributed key-value store – lacks observability features but getting better – uses Envoy proxy

It is worth-noting that there are some initiatives to standardize service mesh (e.g. SMI) with reference implementation such as OpenServiceMesh. The standard is too weak to be considered important for now compared with maturity and stability. Istio has strong community support. The risk with Istio to acknowledge, is that it is not following an open-governance model (unlike many other CNCF projects), which could potentially causes vendor lock-in.

Second, The Istio Gateway features the separation between Gateway and Virtual Service CRDs. The former defines entry point and the latter defines routing rules. This design separates entry points from routing rules, enabling the flexibility of reusing the same Virtual Service for different gateways.

Istio uses Envoy proxy to manage traffic between Pods. Unlike the kube-proxy pattern, the Envoy proxies are side-car containers centrally managed by Istio Control Plane, which also enables other features such as traceability, as illustrated in the diagram below (stolen from this post).

In addition to Virtual Service, Istio Gateway also has the concept of Destination Rules. Virtual Service defines how to route traffic to different destinations. Destination rule defines how to split traffic to different subset at the routing destination. The concepts are documented on this page. For HTTP traffic, the relevant entities can be represented in the diagram below:

Here is an example of using Gateway and Virtual Service resources (from Korthweb sample project). Also note that in the Istio literature, there is neither a CRD name called “Ingress”, nor a resource type named “Ingress Gateway”, although they may be loosely used to refer to “Gateway resources configured to manage ingress traffic”.

Istio Gateway Installation

There are more than one ways to install Istio. In the past, Istio Operator was used to install Istio. The Operator calls IstioOperator API. Today the use of Istio Operator is not recommended anymore but the IstioOperator API is used implicitly by Istioctl installer. The two recommended approaches to install Istio today is by Istioctl and Helm.

With istioctl, you can specify an option imperatively, or using an overlay file as did in this lab. With Helm, istio has multiple charts, and requires multiple steps:

install CRDs using the base chart
install istiod using the istiod chart. Here is an example of values provided to Helm installer.
install ingress gateway using the gateway chart. Here is an example of values provided to Helm installer for ingress gateway. Note that egress gateway also requires the same helm chart with different value definition.

This instruction contains how to manually install Istio gateways using Helm, including installation of multiple Helm charts. I attempted to create a single chart to consolidate the multiple charts required for istio. It was not successful because of an error when trying to reference the same gateway chart dependency for multiple times.

From Microservice to Service Mesh

Thu, 25 Nov 2021 00:31:19 -0400

We all know what microservice is now but how does service mesh assist with microservice development.

Microservice

Microservice as an architecture was firstly conceptualized in this article by Martin Fowler in 2014. It covers the pros (strong module boundaries, independent deployment, technology diversity) and cons (dealing with distributed system, eventual consistency, operational complexity). The reality is, many teams develops their product with the microservice architectural pattern.

The implementation of microservice architecture involves a lot of programming patterns, and tools. The creation of these patterns and tools are usually done in a separate dedicated project so developers can focus on business logics. When building software, developers only need to interact with libraries and frameworks. Libraries (e.g. log4j) provides dependencies, and developers needs to write code to call the libraries. On the other hand, frameworks (e.g. Spring, Flask) not only provides tools, but also implements a pattern. It addresses a set of common problems such as authentication, expose http service, logger and database connectivity. Once set up, the framework will call the code that developers write (unlike in libraries).

Spring – an example of Microservice framework

When it comes to microservice, a well-known appliction framwork is the Spring framework. It solves problems such as:

Application context and dependency injection (for Inversion of Control, or IOC)
Database access and transaction management
Expose rest APIs (using spring MVC)

There’s an entire ecosystem of projects under Spring framework. This framework is a huge system requiring a lot of configuration efforts. This is where Spring Boot helps.

Spring Boot makes it easy to create stand-alone, production-grade Spring based applications that you can just run. It features the “convention over configuration” paradigm to save programmers from boiler plate configuration. SpringBoot gives you a standalone application ready to run without complicated deployment steps.

Managing configuration in property files does not scale in the time of microservice. Spring Cloud provides configuration as a service (in line with everything else microservice framework). It doesn’t necessarily have to be hosted in the cloud. It is comparable to Apachee Zookeeper, Etcd (distributed key value store), Hashicorp Consul and Netflix OSS (Eureka, Ribbon, Hystrix). You can pull from Git repo. The mission of Spring Cloud is to eliminate boilerplate associated with distributed systems problems for Spring Boot applications.

Many developers use Spring Boot along with Spring Cloud to build microservices. This page contains a diagram for such architecture.

In this architecture, Spring cloud helps with service discovery, traffic routing, circuit-breaking, distributed tracing and monitoring. It can also act as API gateway (in place of Nginx).

API Gateway and API Management

Microservices relies on API. Let’s distinguish API gateway, and API management (this long post has some good information).

API Gateway is a microserivce pattern. The idea is a single point of entry for all clients. The API gateway either proxy an incoming request to the appropriate service, or it may fan out a request to multiple services. The other important aspect is the API gatway can expose a different API for reach client.

API Gateway Pattern

A variation of this pattern is the Backends for frontends pattern, where it defines a separate API gateway for each kind of client. The API Gateway may authenticate user and pass an Access Token containing information about the user to the services. It may use a circuit breaker to invoke services. To summarize, the key functions of an API Gateway in this pattern is:

Unified entry point for multiple API implementations
Protocol transformation
Request morphing
Client specific logics

This page from Azure has a good comparison between API Gateway pattern vs direct connection between client and microservice. Note that API Gateway can also refers to API Gateway product, which implements the functions above. For example

Spring Cloud Gateway
Solo.io Gloo
Netflix Zuul

API management acts as a proxy for an existing API implementations. Typical functions include:

AuthN and AuthZ
Service Discovery Ingreation
Load balancing (e.g. L7 path based routing)
Logging, tracing (track user), correlation
Response Caching
Retry policies, circuit breaker, QoS
Enforce policy
Track usage and monetization
metrics (duration)
rate limiting and throttling
Request morphing (header, query string and claims transformation)
IP whitelisting

API management are usually implemented as tightly controlled shared infrastructure owned by either a “platform team”, “integration team”, or other API infrastructure teams. Examples of API management product (including SaaS) are:

Google Cloud Apigee
Mulesoft
Kong

In real world, people often use API management produce and API gateway product interchangeably.

Service Mesh

Many dub Service Mesh the next generation of Microservice. So what is the relationship between microservice and service mesh. The Microservice architectural pattern creates the need for API gateway pattern. To address this pattern, the API Gateway products first emerged. Service mesh emerged later. Service mesh and API gateway have a common set of features.

In this presentation (a tale of two frameworks) from 2018 (early days of service mesh), two teams discussed microservice (spring cloud) and service mesh (istio) approaches. There is a slide about when to use which. Many teams since have moved to Service Mesh for feature richness. This is a case study from 2021.

There are a number of service mesh technologies, such as Consul, Isito and Linkerd. Here is a comparison chart. Even though all those projects are open-source, there is some competition already. Linkerd is the first to bring up the concept of service mesh in this blog. It also purportedly has better performance than Istio. However, it does not use Envoy proxy. Istio is good at marketing. It has higher adoption rate and is feature rich. However, Google did not donate Istio project to CNCF as many expected. Instead, it created its own governing body, the Open Usage Commons. The Istio is not an open-governance project, which potentially diverge from CNCF in the future [1]. Hashicorp Consul initially was built for service discovery and distributed key/value store. It supports Kubernetes and VM. However, it still lacks observability features.

Apart from the three major technologies, other players tries to push for standardization of service mesh. The most influential initiative is the SMI (service mesh interface), pushed by Microsoft. The idea is a separation of standard and implementation, so late players will have a chance. OpenServiceMesh is Microsoft’s reference implementation of SMI. The SMI is something to watch for but it remains pretty week thus far. Google’s platform has Anthos Service Mesh which is a commercial distribution based on Istio. AWS has its own AppMesh technology, also using Envoy proxy.

[1] Update from Apr 25, 2022 – Istio applied to become CNCF project.

Istio Lab – Ingress and Egress

Tue, 16 Nov 2021 21:28:00 -0400

Istio is a popular open-source service mesh implementation using Envoy proxy. One of the benefit of using Istio is the ingress and egress it brings to native Kubernetes platform. This article is a hands-on guide to test Istio ingress and egress gateways on Minikube. It was tested on my MacBook. All the information in this post can be found in several different pages from Istio and Minikube documentation. The purpose of this article is a single cheat sheet for myself to quickly set up Istio ingress and egress testing.

Install Istio on Minikube

We start with configuring Minikube on MacBook (or change the command accordingly on other platform), and install metal LB load balancer. The steps with Istio also works with a regular Kubernetes platform.

minikube start --memory=12288 --cpus=6 --kubernetes-version=v1.20.2 --nodes 3 --container-runtime=containerd --driver=hyperkit
minikube addons enable metallb

Then we need to give load balancer an IP address range before setting up the load balancer. Minikube adds an interface named bridge100 to MacOS host environment. The IP address of the host can be found via the following command:

minikube ssh "ping host.minikube.internal -c 1"

For example, my host IP address is 192.168.64.1, so I dedicate the IP range 192.168.64.64 to 192.168.64.80 to the MetalLB load balancer. Then I need to provide this range when configuring the load balancer:

minikube addons configure metallb
kubectl -n metallb-system get po

After the configuration, make sure the MetalLB pod is up. Then we can install istio to the Kubernetes cluster. The steps are completed using istioctl following the instruction, I created a quick and dirty script to put the steps together:

#! /bin/bash
curl -L https://istio.io/downloadIstio | sh -
export PATH=$(realpath istio*/bin):$PATH
ln -sfn $(ls -d -- istio-*) istio
if istioctl x precheck; then
 echo ready to install istio and lable namespace for istio-injection
 istioctl install --set profile=demo -y --verify --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY
 kubectl label namespace default istio-injection=enabled
else
 echo failed precheck
 exit 1
fi

The script above download the istio zip file, unzips it, installs Istio (in demo mode and set outboundTrafficPolicy to REGISTRY_ONLY), labels default namespace for proxy injection. To use istioctl without spelling out full path, we can run the following command again separately outside of script:

export PATH=$(realpath istio/bin):$PATH

To validate, check istioctl version command.

Install Sample application

For simplicity, the rest of this article uses httpbin to test Istio’s ingress and egress. For completeness, this section gives the steps to install bookinfo application as it is used in other Istio testing as well. It also help understand how to ingress to an application with Istio using an IP provided by the load balancer. Skip this section if just need to test with the simple httpbin application. The Bookinfo application comes within the istio 1.12 directory. In the following steps from the istio directory, we install some observability add-ons as well as the bookinfo application. Then we test access to the website.

kubectl apply -f samples/addons/prometheus.yaml
kubectl apply -f samples/addons/kiali.yaml
kubectl apply -f samples/addons/jaeger.yaml
kubectl apply -f samples/addons/grafana.yaml

kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

kubectl exec "$(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}')" -c ratings -- curl -sS productpage:9080/productpage | grep -o "<title>.*</title>"

export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT
curl -s "http://${GATEWAY_URL}/productpage" | grep -o "<title>.*</title>"

On my laptop, the value of $GATEWAY_URL is 192.168.64.64:80. Therefore I can browse to site http://192.168.64.64:80/productpage with local browser. We can also check kiali dashboard:

istioctl dashboard kiali

Then we are good to examine ingress and egress.

Prep Work

The following custom resources are related to traffic management in istio:

Virtual Service (vs)
Destination Rule (dr)
Service Entry (se)
Gateway (gw, Istio’s ingress or egress)

We will expand on the role of each in a separate article. To execute command from within a Pod, we also use sleep application which is essentially a Pod with a sleeper thread for the purpose allowing user to run troubleshooting command. The sleep application is provided in istio directory:

kubectl apply -f samples/sleep/sleep.yaml
kubectl get po -l app=sleep

Find out the Pod name to execute command. Then we use httpbin website to emulate an external service, and use the sleep Pod (suppose the name is sleep-557747455f-q99cz) to access the service:

kubectl exec -it sleep-557747455f-q99cz -c sleep -- curl http://httpbin.org/headers

As we previously set outboundTrafficPolicy.mode=REGISTRY_ONLY, this step is expected to output nothing. If this is not the case, it means the cluster may have ALLOW_ALL as outbount traffic policy, and you will need to follow this guide to set it to REGISTRY_ONLY. With REGISTRY_ONLY, in order to allow access, we need to register a service entry, by applying the document below:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
 name: httpbin
spec:
 hosts:
 - httpbin.org
 ports:
 - number: 80
 name: http
 protocol: HTTP
 resolution: DNS

Now we should be able to access httpbin service as it is registered:

kubectl get se httpbin -o yaml

Using the exec command above again, we should see output from the curl command.

Ingress Testing

We deploy an httpbin service in order to test ingress. The YAML declaration is prepared by Istio installation package:

kubectl apply -f samples/httpbin/httpbin.yaml

Then we deploy the ingress:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: httpbin-gateway
spec:
 selector:
 istio: ingressgateway
 servers:
 - port:
 number: 80
 name: http
 protocol: HTTP
 hosts:
 - "httpbin.digihunch.com"

The selector above indicates that the gateway deployed to a Pod labeled with istio: ingressgateway. Then we create a corresponding virtual service for this gateway:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: httpbin
spec:
 hosts:
 - "httpbin.digihunch.com"
 gateways:
 - httpbin-gateway
 http:
 - match:
 - uri:
 prefix: /status
 - uri:
 prefix: /delay
 route:
 - destination:
 port:
 number: 8000
 host: httpbin

Note that the hosts setup must match the configuration in gateway. The virtual service defines two services at URIs /status and /delay to expose, and we can confirm the creation of virtual service:

kubectl get vs httpbin -o yaml

We can then test with curl command from MacOS host to verify that the ingress is working. We need to use the IP address of the service (192.168.64.64 with port 80 as previously indicated):

curl -I -HHost:httpbin.digihunch.com http://192.168.64.64/status/200
curl -I -HHost:httpbin.digihunch.com http://192.168.64.64/delay/2

We use -HHost to emulate this request going to httpbin.digihunch.com (otherwise we’d have to fake DNS entry in /etc/hosts). Both command should give their return without error. The second command will give return after a delay of 2 seconds.

This validates the success of ingress setup.

Egress Testing

To visit an external service, there are four options:

Just access external service from Pods, when you have global.outboundTrafficPolicy.mode=ALLOW_ANY
Use ServiceEntry (as illustrated above in Prep Work)
Bypass envoy proxy (not recommended, because there is no role of service mesh)
Configure an Egress Gateway

In the Prep Work, we’ve seen how to use Service Entry. However, there is no control of how Pods access the external service. We are going to configure an Egress Gateway to manage the outgoing traffic flow. This allows you to manage route, and apply observability features to the outgoing traffic, something the Service Entry alone will not provide. The egress gateway have two typical use cases as explained on Istio website.

First, we want to confirm that the egress pod exists, with a quick command:

kubectl get po -n istio-system -l app=istio-egressgateway && kubectl get se

It should exist as we installed istio as demo version. We then also need a service entry, as we have already configured in Prep Work. At this point, the egress gateway resource has not been created.

Now, in a separate terminal, we monitor the log of egress pod

kubectl logs -f istio-egressgateway-7d5d69dcfd-kvwxv -n istio-system

With the -f (follow) session on, if we visit external service as we did during Prep Work, there is no new entry pop up at the tail of the log. This is because the external access is still not managed by egress. Now we need to add egress, with the following resources applied:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: istio-egressgateway
spec:
 selector:
 istio: egressgateway
 servers:
 - port:
 number: 80
 name: http
 protocol: HTTP
 hosts:
 - httpbin.org

Then we setup the virtual service with related rules:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: vs-for-egressgateway
spec:
 hosts:
 - "httpbin.org"
 gateways:
 - istio-egressgateway
 - mesh
 http:
 - match:
 - gateways:
 - mesh
 port: 80
 route:
 - destination:
 host: istio-egressgateway.istio-system.svc.cluster.local
 subset: httpbin
 port:
 number: 80
 weight: 100
 - match:
 - gateways:
 - istio-egressgateway
 port: 80
 route:
 - destination:
 host: httpbin.org
 port:
 number: 80
 weight: 100

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
 name: dr-for-egressgateway
spec:
 host: istio-egressgateway.istio-system.svc.cluster.local
 subsets:
 - name: httpbin

With all applied, we can repeat the access command using curl, and watch for the log tail.

kubectl exec -it sleep-557747455f-q99cz -c sleep -- curl http://httpbin.org/ip

The log should output a line saying curl command access, which indicates that this outbound access is managed by istio egress gateway. The log line looks like below and in the example the IP address 10.244.2.4 is the Pod IP address.

[2021-11-22T01:22:47.547Z] "GET /ip HTTP/2" 200 - via_upstream - "-" 0 45 818 816 "10.244.2.9" "curl/7.80.0-DEV" "9ec0a193-77f0-90a5-b132-21c3259307da" "httpbin.org" "3.216.167.140:80" outbound|80||httpbin.org 10.244.2.4:42372 10.244.2.4:8080 10.244.2.9:52710 - -

Looking at the virtual service declaration, you might noticed multiple routes in the virtual service. The first route (mesh) tells sidecars to route request to egress gateway. The second route (istio-egressgateway) tells egress gateway to route traffic to external service as destination.

In real life, there’s more to consider. For example, we usually want the host to allow on egress to be a wild card host such as *.api.google.com. While the hosts field in virtual service can be wild card, the route destination in virtual service has to be a list of fully qualified domains. The other situation is the destination website may issue an HTTP redirect to a different host, which also needs to be registered as a destination. These details can add up to administrative effort.

Egress example

An other real life scenario is the use of authorization policy. Authorization policy rules use SPIFEE identity of a Pod. If a Pod initiates egress connection to destination host in TLS, then the Pod’s SPIFFEE identity and its identity for external TLS connection are in conflict.

With Azure AKS

The steps above also works on other cloud platform such as AKS. This is an AKS cluster that I tested the steps. The Terraform code provisions a private cluster, as well as a Bastion Host. From the Bastion host, we can perform the same steps as above.

When Istio gets installed, it will created a Service object called istio-ingressgateway in the istio-system namespace. The service is of load balancer type and by default will create a public load balancer in Azure AKS.

The test steps will also work for private load balancers. To tell Azure to create private load balancer on a specific subnet, the Service object needs to include annotation. We have to customize Istio installation in order to do that, using the following overlay file (azoverlay.yaml):

apiVersion: install.istio.io/v1alpha2
kind: IstioControlPlane
metadata:
 namespace: istio-operator
 name: custom-istiocontrolplane
spec:
 profile: demo
 values:
 gateways:
 istio-ingressgateway:
 serviceAnnotations:
 service.beta.kubernetes.io/azure-load-balancer-internal: "true"
 service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "suitable-porpoise-lb-subnet"

This is inspired by this post. Then the command to install Istio becomes:

istioctl install -f azoverlay.yaml -y --verify --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY

This concludes the hand-on guide for Istio ingress and egress labs. I will need to write another post explaining the related concepts.

Logging and Monitoring in Kubernetes with PLG stack

Wed, 13 Oct 2021 21:29:00 -0400

We’ve checked out the the actors in PLG stack (Promtail, Loki, Node Exporter, Prometheus, Grafana) and whipped up a quick pipeline on MacOS. Now I’m going a little further to implement the same PLG stack (Prometheus Loki and Grafana) in a Kubernetes cluster. This setup is for demo only, therefore no persistent storage is enabled.

Test Workload

I host a deployment of Flog with three pods running on Minikube. Flog is an open-source emulating log generation behaviour of an application. On the Minikube cluster we start the deployment as below:

minikube start --driver=hyperkit --container-runtime=containerd --memory=12288 --cpus=2
kubectl create ns obsv
kubectl -n obsv create deployment flog --image=mingrammer/flog --replicas=3 -- flog -f rfc3164 -l -d 300ms
kubectl -n obsv get po

The Pods will come up in a heartbeat.

I will use helm to install the objects required for logging and metrics pipelines. There are multiple Helm charts for each components. Some high-level charts (usually with a name suffix of -stack) contain several other resource as sub-charts. They are created as one-stop-shop for multiple components but I found none of them serve my exact purpose. For example, both loki-stack and kube-prometheus-stack include Grafana. But I only need one instance of Grafana. Therefore I stick to the low-level charts.

Log Shipping

Add a helm repo, and install loki and promtail. Note that we need to specify correct loki address when installing Promtail.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --namespace obsv --install loki grafana/loki
helm upgrade --namespace obsv --install promtail grafana/promtail --set "config.lokiAddress=http://loki.obsv.svc.cluster.local:3100/loki/api/v1/push"

When installing Promtail, a DaemonSet is created on the Node. The default configuration applies appropriate configuration and tagging strategy for Kubernetes Pod and Node. So the only customization I specified is Loki address. We can then check logging with Loki. To do so, first expose port 3100 to host, and then use logcli (e.g. on MacOS) to query for logs:

kubectl -n obsv port-forward service/loki 3100:3100
logcli labels
logcli labels pod
logcli query '{pod="flog-775d5fc5c8-p4rlx"}'

Log lines should be pumped to Loki a minute after Loki comes up. The logcli labels command should display the tags. The logcli query command should return the log lines.

Metrics

I use Premetheus with node exporter. In its architecture, Prometheus contain the server, the pushgateway, and alertmanager. The helm chart for Prometheus contains all of those components. It also has a dependency repo for kube-state-metrics. To install:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm --namespace=obsv install prometheus prometheus-community/prometheus
kubectl -n obsv get svc

Once installed, a DaemonSet for Prometheus Node Exporter is created. The exporter is already configured by default for Kubernetes monitoring. The Prometheus server is also configured, on port 80 by default. it needs to be forwarded in order to access from Browser:

kubectl --namespace obsv port-forward service/prometheus-server 9100:80

To verify installation of Prometheus, browse to localhost:9100 to examine the metrics.

Visualization

Last but not least, I will configure Grafana. The repo has been added already so we’ll just install the chart:

helm upgrade --namespace obsv --install grafana grafana/grafana
kubectl get secret --namespace obsv grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

The second command retrieves the credential. To access the web portal, we need port forwarding again:

kubectl port-forward --namespace obsv service/grafana 3000:80

To verify installation, browse to http://localhost:3000 and log in as user admin with the password above. Then add two data sources:

Type: Prometheus, URL: http://prometheus-server.obsv.svc.cluster.local:80
Type: Loki, URL: http://loki.obsv.svc.cluster.local:3100

Then we can explore data using both data sources.

PLG stack

Summary

In the last two posts I reviewed the setups for PLG stack in Kubernetes, from a regular environment to k8s cluster. Fluentd, Prometheus are both CNCF projects. The PLG stack seems to be more adopted than EFK but both have their own advantages. Welcome to the PLG vs EFK debate.

Intro to PLG stack -Prometheus, Loki and Grafana

Sun, 03 Oct 2021 12:59:00 -0400

Last month we discussed log shipping with EFK. This week I spent sometime checking out its alternative Loki. Having been exposed to the ELK stack extensively, I am also interested in exploring the counterparts in this new stack, such as Prometheus, Loki and Grafana. So I need to address the issues of shipping both metrics and logs.

Let’s start by clarifying the terms:

Grafana is a visualizer. It supports many backends such as Prometheus, Loki, Elasticsearch, CloudWatch and Azure Monitor. It is the flagship product of Grafana Labs.

Prometheus is a time-series database and alerting platform. To push metrics to Premethus, you can either integrate your application with client library (in their term, instrumenting), or configure an existing exporters for a third party application such as PostgreSQL. Prometheus collects and stores its metrics as time series data ( i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels) and it comes with basic visualization capability. Premetheus is a CNCF project since 2016 and is maintained by Grafana Labs.

Loki is a log aggregation system, also developed by Grafana Labs. Loki does not index the contents of the logs. Instead it groups entries into streams, and indexes a set of labels for each log stream. You can use grafana or logcli to consume the logs. Loki supports clients such as Fluentd, Fluentbit, Logstash and Promtail.

Promtail is a log collection agent built for Loki.

The roles of the components such as Prometheus, Loki, Grafana and Promtail are similar to the ELK stack. Grafana resembles Kibana. Promtail resembles Filebeat. Premethus exporters resemble Metricbeat. Both Premetheus and Loki resemble Elasticsearch in some aspects. Premetheus keeps metrics and Loki persists log streams.

These tools are heavily used in Kubernetes. For a simple start, I’d like to just configure two minimally working pipelines on my MacBook without any containerization. My example setup is to achieve the followings:

Fake up some log lines and ship them to Loki
Ship OS metrics to Prometheus
Display the metrics and logs with Grafana

Here’s the diagram of what we want to implement:

This is simple enough to build a quick and dirty setup

Configure Node Exporter

We can use home brew to install node_exporter on MacOS. For Linux, use the equivalent package management tool:

brew install node_exporter
brew services start node_exporter
curl http://localhost:9100/metrics | grep "node_"

The node_exporter collect metrics and make it available for scrape using port 9100 as shown above.

Configure Prometheus

Install Prometheus with homebrew and start the service:

brew install prometheus
brew services start prometheus

If curl to port 9090 with GET returns “Found”, then Prometheus is successfully installed. We also want to configure it so it scrapes node exporter for metrics. Edit the configuration file, in my case, /usr/local/etc/prometheus.yml, by adding the followings:

 - job_name: node
 static_configs:
 - targets: ['localhost:9100']

This tells Prometheus to scrape metrics at specified interval. Restart Prometheus and browse to http://localhost:9090/ for Prometheus UI. Click on Status -> Targets and you can see the node as an export. Click on Graph and Execute a query for example “node_memory_free_bytes” and click on Graph. You should see a plot of the metric value.

We can later configure to display the chart in Grafana.

Generate log lines

We use a tool called flog to generate fake logs:

brew tap mingrammer/flog
brew install flog
flog -f rfc3164 -l -d 300ms -t log -o /tmp/test.log -w

The command above produces log in RFC3164 format to /tmp/test.log, one line every 300ms. The log does not rotate.

Configure Loki

Install Loki with homebrew and start the service. We want to configure Loki before Promtail so it is ready to receive logs. You might as well install LogCLI to interact with Loki.

brew install loki
brew install logcli
brew services start loki

If nc to port 3100 returns success, then Loki is successfully installed.

Configure Promtail

Install promtail using homebrew:

brew install promtail

Note that Promtail is not installed as a homebrew service. Although it can be manually configured as a service, I’d rather stay focused and use command line just for the time being. A copy of configuration file is located in /usr/local/etc/promtail-local-config.yaml but it needs to be modified first with a job to tell it where to scrap the log lines. The configuration looks like this:

server:
 http_listen_port: 9080
 grpc_listen_port: 0

positions:
 filename: /tmp/positions.yaml

clients:
 - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: app
 static_configs:
 - targets:
 - localhost
 labels:
 job: applogs
 __path__: /tmp/test.log

Then, use the command as in the documentation to start promtail:

promtail -config.file /usr/local/etc/promtail-local-config.yaml

Port 9080 will be open when Promtail is running. Promtail will push logs to Loki. You can use LogCLI to interact with Loki and see the latest log lines pushed to Loki.

logcli labels job
logcli query '{job="applogs"}'

However, unlike Prometheus, Loki itself does not have any visualization capabilities. We will need Grafana to display the log nicely on the web.

Configure Grafana

We use Grafana to visualize both the logs and metrics. To install Grafana on Mac:

brew install grafana
brew services start grafana

If curl to port 3000 with GET returns “Found”, then Grafana is successfully installed. Browse to localhost:3000, with default credential admin and admin. From the UI, add two data sources. For the first data source, specify Prometheus as type and localhost:9000 as destination. For the second, specify Loki as the destination http://localhost:3100

The steps to see logs in Loki is pretty much the same as on this page of its documentation.

Exploring logs

Exploring metrics data is similar. Click on Explore on the side bar, then select Prometheus from the dropdown as data source. Then execute a query such as “node_memory_free_bytes”.

Explore Metrics

With this quick and dirty configuration, we established a good understanding of what Prometheus, Loki and Grafana do. Next, we will move all these configurations to K8s cluster and understand some specific points of configurations.

Log Shipping in Kubernetes with EFK stack

Sat, 04 Sep 2021 21:50:00 -0400

I first worked on log shipping with ELK stack three years ago. In the context of Kubernetes cluster, log shipping has similar challenges. In this post I will discuss the set up of log shipping with Kubernetes cluster using EFK stack

Logging Architecture

As discussed, if the Kubernetes cluster has a runtime in compliant with CRI (e.g. containerd), then the stdout and stderr of the Pod is stored on the node, in the location /var/log/containers/.

When creating log shipping solution, it is important to use a compliant cluster (e.g. minikube) to ensure what you develop will work across environments.

The Kubernetes document has a section on logging architecture which is a good start point. It outlines several different patterns. Logging at the node level is turned on by default and does not require special configuration, as explained in the section above. The EFK pattern is close to the diagram under using a node logging agent for cluster-level logging.

Node logging agent

The diagram above is stolen from Kubernetes documentation. In EFK stack, the agent is a daemonset running fluentd Pod.

EFK stack

While we can use ELK (Elasticsearch, Logstash, Kibana) stack for log shipping, EFK (Elasticsearch, Fluentd, Kibana) is generally recommended in Kubernetes cluster. We compare the two in the following table:

	ELK	EFK
Development language	All in Java. Logstash and Filebeat both require JVM. Managed as an open-source project by Elastic company	Fluentd in Ruby and does not require JVM to run. Fluentd is a CNCF project built to integrate with Kubernetes.
Typical pattern	Filebeat acts as a lightweight collector to monitor the source log. Logstash as aggregator to receive from filebeat, and push to Elasticsearch	The fluentd Pod can be configured to serve as forwarder and aggregator based on configuration. fluentd-forwarder is deployed as daemonset on node, and ship the result to fluentd-aggregator, which may run in a separate cluster. The fluentd-aggregator pushes processed results to Elasticsearch.
Metrics	use metric beat for data collection	scrape metrics from prometheus server
Event Routing	tag-based	if-then statement

ELK and EFK stacks comparison

The main problems in log shipping are:

column mapping : identify column patterns in each log line and map them to appropriate column in Elasticsearch.
multi-line processing: identify when a logging entry spread across multiple lines and process accordingly.

Suppose we want to congregate the logs from stdout and stderr of PostgreSQL pods. The raw output in /var/log/container on the node, looks like this:

2021-08-28T15:27:46.75370563Z stdout F server stopped
2021-08-28T15:27:46.757173069Z stderr F postgresql-repmgr 15:27:46.75 INFO ==> Starting PostgreSQL in background...
2021-08-28T15:27:46.883126928Z stderr F postgresql-repmgr 15:27:46.88 INFO ==> Registering Primary...
2021-08-28T15:27:47.017164653Z stderr F postgresql-repmgr 15:27:47.01 INFO ==> Loading custom scripts...
2021-08-28T15:27:47.023334611Z stderr F postgresql-repmgr 15:27:47.02 INFO ==> Loading user's custom files from /docker-entrypoint-initdb.d ...
2021-08-28T15:27:47.026169813Z stderr F postgresql-repmgr 15:27:47.02 INFO ==> Starting PostgreSQL in background...
2021-08-28T15:27:47.343607487Z stderr F postgresql-repmgr 15:27:47.34 INFO ==> Stopping PostgreSQL...
2021-08-28T15:27:47.448111425Z stdout F waiting for server to shut down.... done
2021-08-28T15:27:47.448172479Z stdout F server stopped
2021-08-28T15:27:47.453722807Z stderr F postgresql-repmgr 15:27:47.45 INFO ==> ** PostgreSQL with Replication Manager setup finished! **
2021-08-28T15:27:47.453829953Z stdout F
2021-08-28T15:27:47.503516746Z stderr F postgresql-repmgr 15:27:47.50 INFO ==> Starting PostgreSQL in background...
2021-08-28T15:27:47.532558987Z stdout F waiting for server to start....2021-08-28 15:27:47.532 GMT [273] LOG: pgaudit extension initialized
2021-08-28T15:27:47.533307459Z stdout F 2021-08-28 15:27:47.533 GMT [273] LOG: listening on IPv4 address "0.0.0.0", port 5432
2021-08-28T15:27:47.533466407Z stdout F 2021-08-28 15:27:47.533 GMT [273] LOG: listening on IPv6 address "::", port 5432
2021-08-28T15:27:47.537987947Z stdout F 2021-08-28 15:27:47.537 GMT [273] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-08-28T15:27:47.547956465Z stdout F 2021-08-28 15:27:47.547 GMT [273] LOG: redirecting log output to logging collector process
2021-08-28T15:27:47.548005463Z stdout F 2021-08-28 15:27:47.547 GMT [273] HINT: Future log output will appear in directory "/opt/bitnami/postgresql/logs".
2021-08-28T15:27:47.551741571Z stdout F 2021-08-28 15:27:47.551 GMT [275] LOG: database system was shut down at 2021-08-28 15:27:47 GMT
2021-08-28T15:27:47.558012894Z stdout F 2021-08-28 15:27:47.557 GMT [273] LOG: database system is ready to accept connections
2021-08-28T15:27:47.618577092Z stdout F done
2021-08-28T15:27:47.618708978Z stdout F server started
2021-08-28T15:27:47.630065958Z stderr F postgresql-repmgr 15:27:47.62 INFO ==> ** Starting repmgrd **
2021-08-28T15:27:47.638116348Z stderr F [2021-08-28 15:27:47] [NOTICE] repmgrd (repmgrd 5.2.1) starting up
2021-08-28T15:27:47.65317627Z stderr F INFO: set_repmgrd_pid(): provided pidfile is /opt/bitnami/repmgr/tmp/repmgr.pid
2021-08-28T15:27:47.653232015Z stderr F [2021-08-28 15:27:47] [NOTICE] starting monitoring of node "orthweb-postgresql-ha-postgresql-0" (ID: 1000)
2021-08-28T15:27:47.681683703Z stderr F [2021-08-28 15:27:47] [NOTICE] monitoring cluster primary "orthweb-postgresql-ha-postgresql-0" (ID: 1000)
2021-08-28T15:28:11.742865958Z stderr F [2021-08-28 15:28:11] [NOTICE] new standby "orthweb-postgresql-ha-postgresql-1" (ID: 1001) has connected

From this snippet of log, we can see each line in stdout or stderr is appended with a timestamp. There are multi-line log entries but each is still appended with a timestamp. This is just how kubernetes keeps the log file for Pod stdout and stderr. To handle that, we need to first take out the real log line, and then process multi-line.

We will go over the installation of EFK stack and the mechanism to address the two challenges above.

Install Elasticsearch and Kibana

To install Elasticsearch, we use the helm chart provided by the official repository:

helm repo add elastic https://helm.elastic.co

If we run multiple pods on the same hosts, then we need some customized values in order to get the installation to work. The values.yaml file looks like this:

---

antiAffinity: "soft"

esJavaOpts: "-Xmx128m -Xms128m"

# Allocate smaller chunks of memory per pod.
resources:
 requests:
 cpu: "100m"
 memory: "512M"
 limits:
 cpu: "1000m"
 memory: "512M"

Then we can “preview” what values are used for installation, with helm’s template command:

helm template elasticsearch elastic/elasticsearch -f values.yaml

The effect of the antiAffinity property allows multiple Elasticsearch Pod to be scheduled on the same node. This is not required in production with multiple nodes. To install Elasticsearch and Kibana, run:

helm install elasticsearch elastic/elasticsearch -f values.yaml
helm install kibana elastic/kibana

The Kibana service is exposed on port 5601 of the cluster. To access the port on cluster, we need to run port-forward command as below:

kubectl port-forward deployment/kibana-kibana 5601

Install Fluentd

There are different ways to configure Fluentd. For example, in the forwarder-aggregator pattern, a forwarder Pod is a Daemonset on each Kubernetes node. The forwarder pushes to the aggregator, at port 24224. This page has some configuration details. This pattern is similar to filebeat -> logstash pattern in ELK stack.

In our case, we use a simplified pattern, with a fluentd daemonset acting as collector and then forward to Elasticsearch. Similarly, in ELK stack we can use filebeat to push to Elasticsearch without Logstash. The only reason is the entire architecture isn’t as complicated as requiring an aggregator.

We need to create configmap as below:

kind: ConfigMap
apiVersion: v1
metadata:
 name: fluentd-cm
 namespace: default
 labels:
 app.kubernetes.io/component: forwarder
 app.kubernetes.io/instance: fluentd
 app.kubernetes.io/managed-by: Helm
 app.kubernetes.io/name: fluentd
 helm.sh/chart: fluentd-1.3.0
 annotations:
 meta.helm.sh/release-name: fluentd
 meta.helm.sh/release-namespace: default
data:
 fluentd.conf: |
 # Ignore fluentd own events
 <match fluent.**>
 @type null
 </match>

 # HTTP input for the liveness and readiness probes
 <source>
 @type http
 port 9880
 </source>

 # Throw the healthcheck to the standard output instead of forwarding it
 <match fluentd.healthcheck>
 @type null
 </match>

 # Get the logs from the containers running in the node
 <source>
 @type tail
 read_from_head true
 tag kubernetes.*
 path /var/log/containers/orthweb-postgresql-ha-postgresql-**.log
 pos_file /opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos
 <parse>
 @type regexp
 expression ^(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.[^Z]*Z)\s(?<stream>[^\s]+)\s(?<character>[^\s])\s(?<message>.*)$
 </parse>
 </source>
 # enrich with kubernetes metadata
 <filter kubernetes.**>
 @type kubernetes_metadata
 @id filter_kube_metadata
 kubernetes_url "#{ENV['FLUENT_FILTER_KUBERNETES_URL'] || 'https://' + ENV.fetch('KUBERNETES_SERVICE_HOST') + ':' + ENV.fetch('KUBERNETES_SERVICE_PORT') + '/api'}"
 verify_ssl "#{ENV['KUBERNETES_VERIFY_SSL'] || true}"
 ca_file "#{ENV['KUBERNETES_CA_FILE']}"
 skip_labels "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_LABELS'] || 'false'}"
 skip_container_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_CONTAINER_METADATA'] || 'false'}"
 skip_master_url "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_MASTER_URL'] || 'false'}"
 skip_namespace_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_NAMESPACE_METADATA'] || 'false'}"
 </filter>


 <match kubernetes.var.log.containers.orthweb-postgresql-ha-postgresql-**.log>
 @type elasticsearch
 include_tag_key true
 host "elasticsearch-master.default.svc.cluster.local"
 port "9200"
 index_name "postgresql-logs"
 <buffer>
 @type file
 path /opt/bitnami/fluentd/logs/buffers/orthanc-logs.buffer
 flush_thread_count 2
 flush_interval 5s
 </buffer>
 </match>

Then we can create the resource, with helm chart pointing to the config map:

helm install fluentd bitnami/fluentd --set aggregator.enabled=false --set forwarder.configMap=fluentd-cm

We can validate the index creation on Elasticsearch:

kubectl port-forward service/elasticsearch-master 9200
curl -XGET localhost:9200/_cat/indices

From Kibana, we can forward the port as instructed above, and browse to the UI. Once logged on to Kibana, we need to add index pattern first before being able to view the content of index.

How about Fluent Bit

Fluentd has an even more lightweight brother Fluent Bit, also a CNCF project, designed by the same team, for environments with tighter resource restrictions. The technical differences are outlined on this page outlines the technical differences. In terms of use case, Fluentd is a log aggregator and Fluent Bit is simply a forwarder. In terms of ecosystem, Fluentd has a stronger ecosystem whereas Fluent Bit is more seen in IoT devices. Read this post for more comparison.

Summary

EFK stack (Elasticsearch, Fluentd and Kibana) on Kubernetes is a natural choice for ELK users. Fluentd is a CNCF project created for integration with Kubernetes. It is good alternative to enterprise solution such as Splunk. There are lots of plugins supported and articles on configuration. While developing a solution I had to spend time reading the input plugin documentations.

Log shipping through ELK

Wed, 19 Sep 2018 17:26:51 -0400

A common devops task is build logging pipeline with ELK stack (Elasticsearch, Logstash, Kibana). Suppose the application is written in Java and currently use log4j’s RollingFileAppender to generate log files locally. We can use log4j’s socket appender to write to Logstash, which further pushes the log stream to Elasticsearch cluster.

In this model, failure to push a log line results in the following in log4j output:

2018-09-06 11:05:39,778,ERROR,stderr - [AsyncAppender-Dispatcher-Thread-672] log4j:WARN Detected problem with connection: java.net.SocketException: Broken pipe (Write failed)

The logstash log displays the socket exception as well:

[2018-09-06T10:12:21,935][DEBUG][logstash.inputs.log4j ] Accepted connection {:client=>"192.168.111.56:58118", :server=>"0.0.0.0:4560"}
[2018-09-06T10:12:21,963][DEBUG][logstash.pipeline ] filter received {"event"=>{"method"=>"?", "thread"=>"676774870@qtp-1804103302-6", "message"=>"Unable to resolve session ID from SessionKey [org.apache.shiro.web.session.mgt.WebSessionKey@71da9a4]. Returning null to indicate a session could not be found.", "priority"=>"DEBUG", "type"=>"log4j", "path"=>"org.apache.shiro.session.mgt.DefaultSessionManager", "@timestamp"=>2018-09-06T15:12:21.950Z, "file"=>"?:?", "@version"=>"1", "host"=>"192.168.111.56:58118", "logger_name"=>"org.apache.shiro.session.mgt.DefaultSessionManager", "class"=>"?", "timestamp"=>1536246741950}}
[2018-09-06T10:12:21,963][DEBUG][logstash.pipeline ] output received {"event"=>{"method"=>"?", "thread"=>"676774870@qtp-1804103302-6", "message"=>"Unable to resolve session ID from SessionKey [org.apache.shiro.web.session.mgt.WebSessionKey@71da9a4]. Returning null to indicate a session could not be found.", "priority"=>"DEBUG", "type"=>"log4j", "path"=>"org.apache.shiro.session.mgt.DefaultSessionManager", "@timestamp"=>2018-09-06T15:12:21.950Z, "file"=>"?:?", "@version"=>"1", "host"=>"192.168.111.56:58118", "logger_name"=>"org.apache.shiro.session.mgt.DefaultSessionManager", "class"=>"?", "timestamp"=>1536246741950}}
[2018-09-06T10:12:22,041][DEBUG][logstash.inputs.log4j ] Closing connection {:client=>"192.168.111.56:58118", :exception=>java.io.InvalidObjectException: Object type java.util.Hashtable is not allowed.}

The troubleshooting isn’t very straightforward. So its alternative is preferred. The alternative is to keep the existing RollingFileAppender as well as the local log files, but use a filebeat agent for each application node. Here is the diagram:

ELK Log Shipping Pipeline

Filebeat is a very light agent to be installed with the application on the same server or container.
Streams of text information congregate to logstash nodes. These log stash service pushes the converged stream to ElasticSearch cluster
ElasticSearch cluster ingest the stream for indexing
Kibana is responsible for viewing.

This is an example architecture with many potential variations. For example, log stash service may be deployed on the same hosts with ElasticSearch cluster. If there aren’t many application nodes, filebeat may directly push its output to ElasticSearch. This architecture provides a lot of flexibility and scalability.

With log4j format, two challenges to address are:

identifying multi-line entry in the log
mapping sections of a log entry into fields in Elastisearch

For example, the following log entry reflects both challenges above:

2018-09-06 11:52:18,022,DEBUG,service.dataportal.web.retrieve - [687285925@qtp-564051174-3350] creating streaming output request 9868594 and requid=fb9a2cad-e24b-4eb8-ad62-b027184e7b9b
2018-09-06 11:52:18,054,ERROR,org.glassfish.jersey.server.ServerRuntime$Responder - [687285925@qtp-564051174-3350] An I/O error has occurred while writing a re
sponse message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: org.mortbay.jetty.EofException
 at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:91)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
 at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135)
 at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662)
 at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395)
 at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280)
 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
 at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
 at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
 at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
 at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
 at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
 at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
 at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
 at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
 at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
 at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
 at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
 at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
 at org.mortbay.jetty.handler.HandlerList.handle(HandlerList.java:49)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.Server.handle(Server.java:326)
 at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
 at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:926)
 at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
 at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
 at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
 at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
 at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.mortbay.jetty.EofException
 at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:789)
 at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:568)
 at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1010)
 at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.flush(ResponseWriter.java:330)
 at org.glassfish.jersey.message.internal.CommittingOutputStream.flush(CommittingOutputStream.java:287)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.flush(WriterInterceptorExecutor.java:305)
 at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79)
 at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
 at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
 at org.glassfish.jersey.spi.ContentEncoder.aroundWriteTo(ContentEncoder.java:137)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
 at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85)
 ... 40 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
 at java.net.SocketOutputStream.socketWrite0(Native Method)
 at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
 at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
 at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368)
 at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:122)
 at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:723)
 ... 58 more

To address multi-line entry we need to tell filebeat how to identify the start of a line through its multiline.pattern configuration option:

- type: log
 enabled: true
 paths:
 - /var/log/dhunch/app.log
 multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}'
 multiline.negate: true
 multiline.match: after
 fields:
 product:
 name: dhunchapp
 log:
 type: app_log
 content: diagnostic

When filebeat pushes log entries to logstash, there will be an additional “time” column added, which reflects the time the log is ingested. The entire log4j message line is all put in a single field called “message”, this message field contains the timestamp from application, the thread, class, logging level and the actual diagnostic message. These are not mapped to separate sections making it difficult to search in Elasticsearch.

To address section mapping, the configuration is made on the logstash side. We will take advantage of a plugin called grok filter. The function of grok filter here, is to parse the original “message” field, map different sections into separate columns, respectively called logtime, loglevel, logclass, logthread and logmsg. The grok filter expression is as below:

filter {
 grok {
 match => { "message" => "%{TIMESTAMP_ISO8601:logtime},%{LOGLEVEL:loglevel},%{NOTSPACE:logclass} - \[%{DATA:logthread}\] %{GREEDYDATA:logmsg}" }
 }
 date {
 match => [ "logtime" , "yyyy-MM-dd HH:mm:ss,SSS" ]
 timezone => "America/Chicago"
 target => "@timestamp"
 }
 mutate {
 replace => [ "message" , "%{logmsg}" ]
 }
 mutate {
 remove_field => [ "logmsg" ]
 }
}

The date plugin indicates to Elasticsearch to treat this field as datetime with specified time zone. The mutate plugin below essentially renames the “logmsg” column to “message”, which allows Elasticsearch to understand this field as log message.