event-driven on Digi Hunch

Knative Eventing Introduction

Fri, 29 Apr 2022 10:16:00 -0400

In the previous post, I mentioned that Knative Serving and Knative Eventing should be seen as two different projects. The former is supposed to be widely used as a serving layer for microservices, whereas the latter has a narrower customer base. There are a dozen companies who need to build Platform as a Service, and will benefit from event-driven architecture. They are the niche customer for Knative Eventing. In this post, we go through two demos for Eventing.

Event Driven Architecture

Before jumping into Knative eventing, let’s first understand the value of event-driven architecture. As covered in Knative presentation in 2020, a growing micro-service architecture involves lots of service-to-service communications that looks like a spider web in a diagram. Two services communicating with each other are tightly coupled: changing one requires a change of the other. Event-driven architecture evolves to decouple those interdependent services. Knative eventing tries to standardize the event data with Cloud Event and also proposes a framework to drive this architecture.

The documentation uses different words to express the same concepts. So let’s take a note first: source = producer which emits events, and sink = subscriber = consumer which is the destination of events. Knative Eventing supports two models of in even-driven architecture.

Brokers and Triggers provides an “event mesh”. Event producers delivers events to a Broker. Triggers then distribute them uniformly by consumers.

Broker is a “hub” for events. It is a central location to receive and send events for delivery. We can choose from the following broker types:

MT (multi-tenant) channel-based broker (which a channel implementation, such as Kaka channel or an in-memory channel)
GCP broker
Apache Kafka broker
RabbitMQ broker

Triggers represents a desire to subscribe to events from a specific broker. It can act as a filter

channel-subscriber model

Channels and Subscriptions provide a “event pipe” model which transforms and routes events between Channels using Subscriptions. This model makes more sense when events out of one system needs to be transformed and then routed to another process.

A channel can be either a generic channel object, or a custom channel implementation (such as Kafka channel or in-memory channel)

The subscription consists of a Subscription object, which specifies the Channel and the Sink (aka the Subscriber) to deliver events to. You can also specify some Sink-specific options, such as how to handle failures.

The key difference is Broker-Trigger model allows you to filter events, where as Channel-Subscription model allows you to transform events.

Note that a Knative Service can act as both a Source and a Sink for events, and for good reason. You may want to consume events from the Broker and send modified events back to the Broker, as you would in any pipeline use-case. Between these components, Knative Eventing uses CloudEvents to send information back and forth.

Knative Eventing Lab

We can create a lab cluster using Kind as per the instruction in real-quicK-cluster. In docker-desktop settings, I configured 6 CPU and 8GB memory as the lab involves a number of nodes. The creation takes a couple minutes. This Knative eventing lab has four paths one can take, based on different model and different implementations:

Lab A demonstrates the configuration of broker-trigger model using Kafka based broker.
Lab B demonstrates the configuration of channel-subscription model, using in-memory channel.

The two labs share some common steps. We first install Knative Eventing using the provided manifests:

$ kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.3.0/eventing-crds.yaml
$ kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.3.0/eventing-core.yaml
$ kubectl get pods -n knative-eventing

The Pods should come to READY fairly quickly. In the mean time, we can configure the consumer as well using the manifests below:

apiVersion: v1
kind: Namespace
metadata:
 name: event-example
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: hello-display
 namespace: event-example
spec:
 replicas: 1
 selector:
 matchLabels: &labels
 app: hello-display
 template:
 metadata:
 labels: *labels
 spec:
 containers:
 - name: event-display
 image: gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
---
kind: Service
apiVersion: v1
metadata:
 name: hello-display
 namespace: event-example
spec:
 selector:
 app: hello-display
 ports:
 - protocol: TCP
 port: 80
 targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: goodbye-display
 namespace: event-example
spec:
 replicas: 1
 selector:
 matchLabels: &labels
 app: goodbye-display
 template:
 metadata:
 labels: *labels
 spec:
 containers:
 - name: event-display
 image: gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
---
kind: Service
apiVersion: v1
metadata:
 name: goodbye-display
 namespace: event-example
spec:
 selector:
 app: goodbye-display
 ports:
 - protocol: TCP
 port: 80
 targetPort: 8080

The consumers (aka sinks) will be able to process events at the validation step at the end.

Lab A – Install Kafka and configure Kafka-based Broker

This lab is for broker-trigger model. We first configure a Kafka Broker, which requires a Kafka cluster. The document uses Strimzi operator to quickly install Kafka cluster. Instead, here we use Helm to install the Kafka cluster:

helm install my-cluster-kafka bitnami/kafka -n kafka --create-namespace --set volumePermissions.enabled=true --set replicaCount=3

At the end of installation, the Helm notes indicate that the bootstrap server is my-cluster-kafka.kafka.svc.cluster.local:9092. All Kafka pods should come ready in about 5 minutes. After cluster install, we need to install Kafka controller and the Broker-layer data plane which allow you to map a Kafka instance to Broker CRD later:

kubectl apply -f https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.3.0/eventing-kafka-controller.yaml
kubectl apply -f https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.3.0/eventing-kafka-broker.yaml
kubectl get pods -n knative-eventing
kubectl -n knative-eventing edit cm kafka-broker-config

As the command above suggests, once data plane is installed, we edit the default broker config by updating the bootstrap.servers with the bootstrap server address noted above. We also need to make sure the replication factor is no greater than number of partitions, which should not be a problem in our lab as the replica count was set to 3.

Now we can create a Broker in the same namespace with consumer:

apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
 name: default
 namespace: event-example
 annotations:
 eventing.knative.dev/broker.class: Kafka
spec:
 config:
 apiVersion: v1
 kind: ConfigMap
 name: kafka-broker-config
 namespace: knative-eventing

Once the manifest above is applied, we should see the status of broker (named default) to be READY. We can also validate broker creation with kn CLI command:

 $ kubectl -n event-example get broker default
NAME URL AGE READY REASON
default http://kafka-broker-ingress.knative-eventing.svc.cluster.local/event-example/default 19m True
$ kn broker list -n event-example
NAME URL AGE CONDITIONS READY REASON
default http://kafka-broker-ingress.knative-eventing.svc.cluster.local/event-example/default 19m 7 OK / 7 True

Then we create Triggers that connects both to the broker with filters, and to the consumers by referencing service name. The manifest to create triggers is provided below:

apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
 name: hello-display
 namespace: event-example
spec:
 broker: default
 filter:
 attributes:
 type: greeting
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: hello-display
---
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
 name: goodbye-display
 namespace: event-example
spec:
 broker: default
 filter:
 attributes:
 source: sendoff
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: goodbye-display

We can confirm the setups above by monitoring the resources (like we did for broker), or use knative CLI:

$ kn trigger list -n event-example
NAME BROKER SINK AGE CONDITIONS READY REASON
goodbye-display default service:goodbye-display 3m24s 6 OK / 6 True
hello-display default service:hello-display 3m24s 6 OK / 6 True

Now we can go to the validation step to send event and validate result.

Lab B – Install in-memory channel and configure Channel

This lab is for channel-subscription model. we first install an in-memory channel:

kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.3.0/in-memory-channel.yaml

Then we can map this channel to a Channel resource in Eventing. We can do so by applying the manifest below:

apiVersion: messaging.knative.dev/v1
kind: Channel
metadata:
 name: demo-channel-1
 namespace: event-example
---
apiVersion: messaging.knative.dev/v1
kind: Channel
metadata:
 name: demo-channel-2
 namespace: event-example

The creation of Channel resources can be verified as below

$ kn -n event-example channel list
NAME TYPE URL AGE READY REASON
demo-channel-1 InMemoryChannel http://demo-channel-1-kn-channel.event-example.svc.cluster.local 6m34s True
demo-channel-2 InMemoryChannel http://demo-channel-2-kn-channel.event-example.svc.cluster.local 6m34s True

Now we need to create Subscription which connects Channels to Consumers. Subscriptions can be created with the manifest below:

apiVersion: messaging.knative.dev/v1
kind: Subscription
metadata:
 name: demo-subscription-1
 namespace: event-example
spec:
 channel:
 apiVersion: messaging.knative.dev/v1
 kind: Channel
 name: demo-channel-1
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: hello-display
---
apiVersion: messaging.knative.dev/v1
kind: Subscription
metadata:
 name: demo-subscription-2
 namespace: event-example
spec:
 channel:
 apiVersion: messaging.knative.dev/v1
 kind: Channel
 name: demo-channel-2
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: goodbye-display

We can confirm subscription:

$ kn -n event-example subscription list
NAME CHANNEL SUBSCRIBER REPLY DEAD LETTER SINK READY REASON
demo-subscription-1 Channel:demo-channel-1 service:hello-display True
demo-subscription-2 Channel:demo-channel-2 service:goodbye-display True

Now we can go to validation step.

Validation

For validation, we need to first configure an event source. The source in our lab is a throw-away Pod with curl utility. I use my favourite nicolaka netshoot to fire off the events using. To launch the Pod and get to command shell:

$ kubectl run tmp-shell -n event-example --rm -i --tty --image nicolaka/netshoot -- /bin/bash
If you don't see a command prompt, try pressing enter.
bash-5.1#

We can POST message to the corresponding endpoint URL to fire events. The endpoint is different for each lab:

Lab	URL
A (Kafka-based broker)	http://kafka-broker-ingress.knative-eventing.svc.cluster.local/event-example/default
B (channel)	http://demo-channel-1-kn-channel.event-example.svc.cluster.local http://demo-channel-2-kn-channel.event-example.svc.cluster.local

With the correct URL, we can build a payload for POST method. Below is three curl commands with different payload POST to the same endpoint (taking lab A as an example):

# curl -v "http://broker-ingress.knative-eventing.svc.cluster.local/event-example/default" \
-X POST \
-H "Ce-Id: say-hello" \
-H "Ce-Specversion: 1.0" \
-H "Ce-Type: greeting" \
-H "Ce-Source: not-sendoff" \
-H "Content-Type: application/json" \
-d '{"msg":"Hello Knative!"}'

# curl -v "http://broker-ingress.knative-eventing.svc.cluster.local/event-example/default" \
-X POST \
-H "Ce-Id: say-goodbye" \
-H "Ce-Specversion: 1.0" \
-H "Ce-Type: not-greeting" \
-H "Ce-Source: sendoff" \
-H "Content-Type: application/json" \
-d '{"msg":"Goodbye Knative!"}'

# curl -v "http://broker-ingress.knative-eventing.svc.cluster.local/event-example/default" \
-X POST \
-H "Ce-Id: say-hello-goodbye" \
-H "Ce-Specversion: 1.0" \
-H "Ce-Type: greeting" \
-H "Ce-Source: sendoff" \
-H "Content-Type: application/json" \
-d '{"msg":"Hello Knative! Goodbye Knative!"}'

All the requests should receive 202 return code. As highlighted above, we specified type and source for each event fired to the broker. On the consumer side, we can simply check the pod log to verify the events received. Below is an example of events received as a result of the events sent above (take broker-trigger model as example):

$ kubectl -n event-example logs -l app=hello-display --tail=100
2022/03/09 17:05:59 Failed to read tracing config, using the no-op default: empty json tracing config
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: greeting
 source: not-sendoff
 id: say-hello
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:19.196589801Z
Data,
 {
 "msg": "Hello Knative!"
 }
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: greeting
 source: sendoff
 id: say-hello-goodbye
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:42.977803608Z
Data,
 {
 "msg": "Hello Knative! Goodbye Knative!"
 }
$ kubectl -n event-example logs -l app=goodbye-display --tail=100
2022/03/09 17:05:58 Failed to read tracing config, using the no-op default: empty json tracing config
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: not-greeting
 source: sendoff
 id: say-goodbye
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:27.430702588Z
Data,
 {
 "msg": "Goodbye Knative!"
 }
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: greeting
 source: sendoff
 id: say-hello-goodbye
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:42.977803608Z
Data,
 {
 "msg": "Hello Knative! Goodbye Knative!"
 }

This is the gist of Knative evening. Note that the format of HTTP payload when we fire off events needs to conform to CloudEvents format. The container image (event_display) used in the container is also created for the purpose of processing events received and display them in stdout.

Conclusion

Persistent storage allows for stateful applications on Kubernetes. It also enables architectural patterns that requires persistent storage. Knative eventing is a framework that enables event driven architecture. It supports many messaging channels (including NATs, a cloud-native messaging system) and broker types (MT channel based, GCP, RabbitMQ).

Knative Serving Introduction

Sun, 17 Apr 2022 11:33:00 -0400

Background

As per IBM‘s definition, Knative enables serverless workloads to run on Kubernetes clusters, and makes building and orchestrating containers with Kubernetes faster and easier. It has drawn a lot of attention recently. It released version 1.0 in November 2021, and was accepted as a CNCF incubating project in March 2022.

Glories aside, the value it delivers is the ability to go serverless on a Kubernetes platform. Originally, Knative was built with three components: build, serving and eventing. The build component was deprecated in favour of the Tekton project. Tekton is cloud native CI/CD pipeline. It connects to source code repo and build artifacts. It can also deploy applications with pipeline as code. Tekton’s documentation includes a quality tutorial here linked to interactive terminal. On the other hand, we’ll focus on Knative Serving in this post, and Knative Eventing in the next.

Knative can be seen as a serverless framework with a number of open-source technologies as building blocks, such as Istio for ingress gateway, Kafka or Google Pub/Sub as event-streaming engine, and Prometheus for observability to name a few. Despite of using the same CLI tool (named kn), Knative Serving and Eventing are considered separate capabilities using different groups of CRDs. They are installed separately, using their respective YAML files, or operator.

Serving vs Eventing ?

After reading the article “Did we market Knative wrong” from Ahmet Balkan, my impression is the two are not related. They could have been two separate projects without sharing a common name. As the author puts, these two shared some core logics. But beyond that, they don’t have anything in common.

Serving is lightweight and requires Istio. It provides the capability to scale from N to 0 when the system is idle, and from 0 to N when requests come in. It is the missing serving layer for running microservices on Kubernetes and Ahmet position it as the first thing that people installed after creating a cluster.

The target user of Eventing is much smaller. Eventing is for event-driven architecture and is more complex than Serving. Ahmet admits that they over-estimated how many people on the planet want to build a Heroku-like PaaS layer on top of Knative. There are a couple of dozen companies who would work through the complexity and build their own Kubernetes-based internal PaaS or even public-facing FaaS using Knative. They are the niche audience of Knative Eventing.

For most platform builders who just need to run micro-service, Knative Serving is all they need.

A Demo on Serving

The definition of term “serverless” is very loose. Instead, Knative documentation promotes the ability to scale to zero and refer to it as “some people call this Serverless”. This suggests that “the ability to scale to zero” should be the better term to describe this capability. To deploy it, we need the Service object with apiVersion serving.knative.dev/v1. Below is a guide for a quick hands-on, with some steps modified from the Knative Serving tutorial and installation guide, with optional steps (extensions, DNS) skipped.

Follow the first part of this post or this guide in real-quicK-cluster project, to install minikube with metal LB and configure istio (using istioctl as per instruction). Then Knative serving can be installed with two YAML manifests:

$ kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.2.0/serving-crds.yaml
$ kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.2.0/serving-core.yaml

Then we can install Knative istio controller to integrate them so we can see the knative gateways:

$ kubectl apply -f https://github.com/knative/net-istio/releases/download/knative-v1.2.0/net-istio.yaml
$ kubectl -n knative-serving get gateway

Now we can install the dummy service, either using kn CLI tool or using YAML manifest. The manifest is given below:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
 name: hello
 namespace: default
spec:
 template:
 metadata:
 name: hello-world
 annotations:
 autoscaling.knative.dev/class: "kpa.autoscaling.knative.dev"
 autoscaling.knative.dev/metric: "rps"
 autoscaling.knative.dev/target: "50"
 autoscaling.knative.dev/min-scale: "0"
 autoscaling.knative.dev/max-scale: "10"
 spec:
 containerConcurrency: 0
 containers:
 - env:
 - name: TARGET
 value: World
 image: gcr.io/knative-samples/helloworld-go
 name: user-container
 ports:
 - containerPort: 8080
 protocol: TCP

We can notice that in the annotations there are some settings to overwrite the default configuration. They are explained in the autoscaling section of the Knative serving documentation. Here we use request per second (rps) as metric, and have 50 as target. The scale range is between 3 and 10 replicas. Once the resource is created, we can validate it with one of the two commands:

$ kubectl get services.serving.knative.dev
$ kn service describe hello

The creation of knative service also created a number of Kubernetes resources, such as deployment and services:

$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello ExternalName <none> knative-local-gateway.istio-system.svc.cluster.local 80/TCP 52s
hello-world ClusterIP 10.110.127.231 <none> 80/TCP 108s
hello-world-private ClusterIP 10.111.27.73 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP 108s
kubernetes ClusterIP 10.96.0.1 <none>
$ kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
hello-world-deployment 1/1 1 1 71s

Before testing, let’s mock the DNS entry to resolve to the Istio ingress service IP like below in /etc/hosts:

192.168.64.16 hello.default.example.com

Now we can confirm that the service returns the expected result (after a brief pause):

$ curl hello.default.example.com
Hello World!

Wait for a short period without curl or connection from browser, the service should scale down to zero:

$ kubectl get deploy -l serving.knative.dev/service=hello -w
NAME READY UP-TO-DATE AVAILABLE AGE
hello-world-deployment 0/0 0 0 78s

From a separate terminal, emulate HTTP request with hey command, so the service receives more than 50 rps per pod:

$ hey -n 200 -c 20 -z 20s -m GET http://hello.default.example.com

We can watch the deployment get scaled up to 10. Once the hey command completes its task, the deployment will scale back down:

$ kubectl get deploy -l serving.knative.dev/service=hello -w
NAME READY UP-TO-DATE AVAILABLE AGE
hello-world-deployment 0/0 0 0 78s
hello-world-deployment 0/1 0 0 3m54s
hello-world-deployment 0/1 0 0 3m54s
hello-world-deployment 0/1 0 0 3m54s
hello-world-deployment 0/1 1 0 3m54s
hello-world-deployment 1/1 1 1 4m2s
hello-world-deployment 1/10 1 1 4m4s
hello-world-deployment 1/10 1 1 4m4s
hello-world-deployment 1/10 1 1 4m4s
hello-world-deployment 1/10 10 1 4m5s
hello-world-deployment 2/10 10 2 4m9s
hello-world-deployment 3/10 10 3 4m10s
hello-world-deployment 4/10 10 4 4m10s
hello-world-deployment 7/10 10 7 4m10s
hello-world-deployment 9/10 10 9 4m10s
hello-world-deployment 10/10 10 10 4m11s
hello-world-deployment 10/5 10 10 5m18s
hello-world-deployment 10/5 10 10 5m18s
hello-world-deployment 5/5 5 5 5m18s
hello-world-deployment 5/2 5 5 5m20s
hello-world-deployment 5/2 5 5 5m20s
hello-world-deployment 2/2 2 2 5m20s
hello-world-deployment 2/1 2 2 5m22s
hello-world-deployment 2/1 2 2 5m22s
hello-world-deployment 1/1 1 1 5m22s
hello-world-deployment 1/0 1 1 5m52s
hello-world-deployment 1/0 1 1 5m52s
hello-world-deployment 0/0 0 0 5m52s

Autoscaling in Knative serving is backed by KPA (Knative Pod Autoscaler, used in the example above) or HPA. Also note that the trigger to scale up and down is client connection (RPS). This is different than KEDA. KEDA also supports the capability to scale to zero. However, with KEDA, it is an event that scales a deployment up from zero. In Knative serving, it is the connection to the Service itself that wakes up the service.

Cost of scale-to-zero

As discussed, both KEDA and Knative serving supports the ability to scale to zero and wake up from zero. The trigger to wake up from zero is different. With KEDA, workload wakes up by an event. With Knative Serving, workload has to wake up upon receiving a connection to the service. In the design to solve this problem, Knative has four sub-components :

Activator – When a service is scaled to zero, its request are routed to the activator which waits for a Pod to become ready (wake up) and proxies the traffic while editing the underlying Endpoints to route the traffic directly to the Pod(s)
Autoscaler – KPA
Controller – The main component responsible for watching API objects for Knative CRDs (KService, Configuration, Route, Revision) and manage their lifecycle, create the underlying Kubernetes resources and garbage-collect old objects.
Webhook – a Kubernetes Admission Webhook acting both as validating admission controller and mutating admission controller.

The key player is activator, which receives requests when a service is scaled to zero. The wake-up process is summarized as follows:

After receiving request, activator will buffer (hold onto) the request
Then, it will look at request’s hostname to find which KService it is for.
Then, it will scale up the Kubernetes Deployment and wait for a Pod to become ready.
In the meanwhile, it updates Kubernetes Service to point to Pod IP addresses (so that activator gets out of the network path if the KService is awake).
Finally, activator proxies the request to the started pod

So the “plumbing” of the service connection managed by the Activator pod is the cost of scale to zero. This is also called Load Balancing, and the behaviour can be tweaked using two parameters: activator capacity and target burst capacity:

Target burst capacity: once the target deploy has one more more Pod, service requests may still route via activator or bypassing the activator. Target burst capacity determines at what point service request should start to bypass activator.
Activator capacity: determines how many requests can activator hold on to. Considering the service-to-service connection also routes via Service CRD, there can be a lot of connections via Activator so its capacity needs to be adjusted.

The additional overhead of managing and optimizing the “Load Balancer” is also part of operational cost for the ability to scale to zero.

In comparison, KEDA now has an HTTP add-on still at beta but allows connection-based wake-up. The design is a little different. As discussed in this previous post.

Conclusion

Knative consists of two disparate components: Serving and Eventing. Serving uses KPA to provide scaling based on service request, and the ability to scale to zero. It is often compared with KEDA, a single-purpose lightweight tool for autoscaling. Here is a blurb on their difference by KEDA. The takeaways is that KEDA is more focused on scalability, whereas Knative serving covers more aspects. For example, the Service object of Knative also supports Traffic Splitting. Knative serving can integrate with istio.

Autoscaling on Kubernetes Platform

Mon, 28 Mar 2022 13:14:00 -0400

Introduction

The concept of autoscaling on Kubernetes platform dates from the era where virtualization first became widespread and the overhead of provisioning a new server became lightweight through the use of cloud-init. With public cloud, customers operate on usage-based billing. Autoscaling allows workload to scale down during idle times to reduce cost, and scale up during peak time to meet the demand of business traffic. Vertical autoscaling replaces a VM with one of higher capacity, which is usually interruptive. Horizontal autoscaling adds or removes VMs to adjust capacity, and works in conjunction with load balancing mechanism to assign load to a specific target in the group. Unless otherwise specified, we simply refer to horizontal autoscaling as autoscaling. Depending on what triggers autoscaling, it can be metrics based or event driven.

Metric-based autoscaling is based on VM (or Pod) metrics in the autoscaling group. The metrics are mostly about CPU usage, memory, IOPS, number of connections, etc. For example, when the average CPU utilization across all VMs in the last five minutes hits 70% threshold, then the scaler introduces a new VM into the autoscaling group. The trigger can factor in a variety of metrics. Advanced autoscaling APIs can also support lifecycle hooks, i.e. custom activities upon creation of new VMs during scale-up, or upon deletion of existing VMs during scale-down. Other aspects of custom behaviours include a cool-off period, i.e a no-activity window after the previous scaling activity. Since scaling activities are re-active.

Using real-time metrics as a trigger of scaling is not always a good idea. For example, a buggy order processing program may consume 100% of CPU due to an infinite loop, or 99% of memory due to memory leak. Metric-based scaling may fire off even though there is currently no order pending in the queue. Event-driven approach is more flexible. Event can fire from any type of source. For example, in Kubernetes, when scheduler fails to schedule a Pod due to constraints, it is an event This event can trigger scaling. In some case, a metric hitting a threshold fires an event. For example, scale up when size of order queue reaches 20. In this sense, metric-based autoscaling is a special case of event-driven autoscaling.

With Kubernetes, let’s examine node group autoscaling (aka cluster autoscaling) and workload autoscaling (Pod autoscaling).

Node Group Autoscaling

Cluster autoscaler is the mechanism to auto-scale node groups for Kubernetes. As per its documentation, any metric-based cluster/node group autoscalers are NOT compatible with CA. They are also not particularly suitable for Kubernetes in general. Take AKS for example, the events to trigger scale-up and scale-down are as below:

The cluster autoscaler component can watch for pods in your cluster that can’t be scheduled because of resource constraints. The cluster then automatically increases the number of nodes.
The cluster autoscaler decreases the number of nodes when there has been unused capacity for a period of time. Pods on a node to be removed by the cluster autoscaler are safely scheduled elsewhere in the cluster.

Both are in essence event driven. The behaviours can be fine-tuned with a number of parameters as below:

scan-interval
scale-down-delay-after-add
scale-down-delay-after-delete
scale-down-delay-after-failure
scale-down-unneeded-time
scale-down-unready-time
scale-down-utilization-threshold
max-graceful-termination-sec
balance-similar-node-groups
expander: random, most-pods, least-waste, priority
skip-nodes-with-local-storage
skip-nodes-with-system-pods
max-empty-bulk-delete
new-pod-scale-up-delay
max-total-unready-percentage
max-node-provision-time
ok-total-unready-count

The parameters above constitute the autoscaler profiler, and are effective if cluster autoscaler is enabled. For many implementations, cluster autoscaler can be enabled and disabled even after the cluster has been created, and the parameters can be changed. The overhead of provisioning a new node should not be overlooked, because that is usually the window that a Pod needs to wait to get scheduled. As stated in CA’s FAQ, the main purpose of CA is to get pending pods a place to run, instead of pre-emptively accommodating to increasing workload.

The delay in pod scheduling while adding a new node can be controlled to a certain degree with one of the two workarounds below:

With HPA or KEDA, set lower threshold so the workload level scaling acts more aggressive than the increase of demand. This buys some buffer time
Use a tool to puff up utilization, such as cluster overprovisioner, which deploys pods that
- request enough resources to reserve virtually all resources for a node
- consume no actual resources
- use a priority class that causes them to be evicted as soon as any other Pod needs it.

In practice the cluster autoscaler setup should be conservative and keep node size as stable as it can. For example, a 20 minutes idle-window (low utilization) on a node is not worth the overhead to remove a node and add it back in 20 minute later.

When the cluster do need to scale down by removing a node, one common symptom is failing to scale down because some Pods have nowhere else to schedule to. Here is a list of possible causes as troubleshooting tips.

If node scaling should be triggered sparsely, then pod scaling is by design very dynamic. Cloud native applications should assume that pod scaling occurs very frequently.

In late 2021, AWS released the open-source project Karpenter for cluster autoscaler. Karpenter addresses some challenges with native Cluster Autoscaler on EKS. Karpenter is gaining momentum and now adding support for other cloud service providers including Azure.

Workload Autoscaling

Stateless workload are controlled by a Deployment object, which is associated with a replicaSet object. For stateless workload we can use HorizontalPodAutoscaler, or HPA. There is a VerticalPodAutoscaler (VPA) which is much less common. HPA is metrics based with flexible options such as specifying an object, depending on what metrics are available via metrics API. There are two versions of HorizontalPodAutoscaler: autoscaling/v1 and autoscaling/v2. The latter supports scaling policies, such as adjusting downscale stabilization window, and limiting scale down rate. No matter which API version, the metric-based triggers in HPA are fairly limited.

We already know that metrics are not always the best indicator to trigger scaling. We need an option to trigger scaling based on the status of other components such as queue size. KEDA (Kubernetes Event Driven Autoscaling) is a great option to consider for horizontal workload scaling. KEDA works with HPA, and significantly enriches trigger options. Apart from metrics, KEDA can use a number of external mechanisms as triggers, for example:

RabbitMQ/Kafka/SQS: scale based on queue size
Azure Log Analytics: scale based on a kusto query result against Azure Log Analytics
AWS CloudWatch, Azure Monitor: scale based on metrics from Azure Monitor/AWS CloudWatch
Azure Pipelines: scale based on agent pool queues of Azure Pipeline
Elasticsearch: scale based on elasticsearch query result
Kubernetes Workload: scale based on the count of running pods of a specified workload
MSSQL, MySQL, Postgres, Cassandra: scale based on a query result
Prometheus: scale based on prometheus query result

KEDA is a single-purpose and lightweight component. With KEDA, we don’t need to explicitly define HPA. It allows us to select from a longer list of triggering mechanisms for our auto scaler. We shall not underestimate the work needed to select the most suitable trigger because having an incorrect trigger (e.g. bad metrics) is costly. Let’s take Java applications as an example. Java workload operates in a JVM inside of the container. JVM request the entire heap size from operating system. The garbage collection activities also consumes a good portion of CPU cycles. This pattern makes CPU and memory metrics inaccurate as an indicator for scaling activity. Because of this we need to find out what is the best scaler for Java application, based on understanding of how the entire solution stack works as a whole.

The other aspect that KEDA beats HPA is its ability to scale to 0. This can be helpful when a service is idle most of the time but cannot shut down.

KEDA lab

let’s use Kafka as an example to configure KEDA for a dummy workload. We create a mock cluster using Kind with a simple configuration file. Then, let’s start with the following dummy workload with replica count set to 1:

apiVersion: v1
kind: Namespace
metadata:
 name: workload
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: aks-helloworld-one
 namespace: workload
spec:
 replicas: 1
 selector:
 matchLabels:
 app: aks-helloworld-one
 template:
 metadata:
 labels:
 app: aks-helloworld-one
 spec:
 containers:
 - name: aks-helloworld-one
 image: neilpeterson/aks-helloworld:v1
 ports:
 - containerPort: 80
 env:
 - name: TITLE
 value: "Welcome to Azure Kubernetes Service (AKS)"
---
apiVersion: v1
kind: Service
metadata:
 name: aks-helloworld-one
 namespace: workload
spec:
 type: LoadBalancer
 ports:
 - port: 80
 selector:
 app: aks-helloworld-one

We need to install KEDA and Kafka using Helm:

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda -n keda --create-namespace
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install kafka bitnami/kafka -n kafka --create-namespace --set volumePermissions.enabled=true --set replicaCount=3

Watch for all Pods to come up. Also read the notes from Kafa installation and confirm the Kafka service address. Now, we will apply KEDA scaled object, defined as below:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: kafka-scaledobject
 namespace: workload
spec:
 scaleTargetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: aks-helloworld-one
 pollingInterval: 10
 cooldownPeriod: 30
 idleReplicaCount: 0
 minReplicaCount: 2
 maxReplicaCount: 5
 fallback:
 failureThreshold: 3
 replicas: 1
 advanced:
 restoreToOriginalReplicaCount: true
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 300
 policies:
 - type: Percent
 value: 100
 periodSeconds: 15
 triggers:
 - type: kafka
 metadata:
 bootstrapServers: kafka-0.kafka-headless.kafka.svc.cluster.local:9092
 consumerGroup: my-group # Make sure that this consumer group name is the same one as the one that is consuming topics
 topic: test
 # Optional
 lagThreshold: "5"
 offsetResetPolicy: latest

The field definitions are on KEDA deploy documentation. Items under Kafka trigger are on the trigger documentation. In this lab we set the idelReplicaCount to 0. It will scale up with average lag of all partitions reaching 5. In the next few steps, we’ll mock up some messages posted to the Kafka topic named “test” for consumer group my-group. We can watch deployment size grow along with the growth of lags.

To emulate Kafka client activity, we can spin up a Kafka test Pod:

kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:2.8.1-debian-10-r73 --namespace kafka --command -- sleep infinity
kubectl exec --tty -i kafka-client --namespace kafka -- bash

From within the Pod, we can leverage the client-side scripts located in /opt/bitnami/kafka/bin/. For example, to post message to a topic (e.g. named test):

kafka-console-producer.sh --topic test --broker-list kafka-0.kafka-headless.kafka.svc.cluster.local:9092,kafka-1.kafka-headless.kafka.svc.cluster.local:9092,kafka-2.kafka-headless.kafka.svc.cluster.local:9092

The Helm installer also gives the command with broker list. To consume messages from a topic (e.g. test) to a given consumer group:

kafka-console-consumer.sh --topic test --bootstrap-server kafka.kafka.svc.cluster.local:9092 --group my-group

We can have two command terminals, post test messages on one terminal and watch it consumed nearly immediately on the other terminal.

The Kafka trigger documentation suggests that the number of replicas will not exceed the number of partitions on a topic when a topic is specified. To make this lab work, we need to have set 5 partitions:

kafka-topics.sh --alter --bootstrap-server kafka.kafka.svc.cluster.local:9092 --topic test --partitions 5
kafka-topics.sh --describe --bootstrap-server kafka.kafka.svc.cluster.local:9092 --topic test

Once we confirm five partitions, we can spin up two command terminals, one to produce message and the other to consume messages. If working, we can stop the consumer and use the command below to watch for the lag for each partition.

kafka-consumer-groups.sh --bootstrap-server kafka.kafka.svc.cluster.local:9092 --describe --group my-group

Now we can artificially trigger scaling by increasing average lag. We keep posting messages on the producer (each carriage return posts a message), and we can check the lag after posting:

Growth of average lags

The size of deployment starts with 0 as defined in the scaled object. As the average exceeds 5, we can see deployment size growing.

Growth of deployment size

This lab is an oversimplified scenario to illustrate the idea of scaling. Kafka is a typical queue construct and other queue configuration such as RabbitMQ or AWS SQS works in very similar ways. Real life use case involves more aspects to consider, such as multiple topics, and authentication.

Connection triggered wake-up

KEDA uses Events to scale workload from zero to one (wake up). There is no way to scale (wake up) based on an incoming web request. In many cases, such as serverless configuration, we need to scale the deployment size from zero to N once the service receives incoming web request. This is not supported by KEDA. By definition KEDA uses Events to wake up.

There is an HTTP-add-on for KEDA still at beta but it is trying to address this problem. This page shows the design. Suppose a service has scaled down to zero, the followings will happen to wake it up:

The incoming request is routed to an interceptor behind the service
interceptor keeps track of number of pending HTTP request
The scaler periodically watches for the size of the pending queue on the interceptor
Based on the queue size, the scaler reports scaling metrics as appropriate to KEDA
As the queue size increases, the scaler instructs KEDA to scale up as appropriate

The periodical check activity is the key to make it work and also what makes it a pseudo-trigger.

Summary

Fine-tuning autoscaling is important to the performance of workload on Kubernetes. At node level, we briefed on cluster autoscaler and suggest that we only use it sparsely. At pod level, we introduced native HPA as well as KEDA, with an example. We also discussed KEDA has limitations and the HTTP-add-on. In the next post, we’ll explore Knative’s autoscaling capability.

Getting started with GitHub Actions

Thu, 27 May 2021 13:52:31 -0400

In my orthweb project, I had to compile a library on my own. In search for free computing resources I realized that GitHub action can meet all my needs.

CI/CD pipeline

As a development project grows, there are many operational tasks demanding automation. Prior to pipeline technology, developers used to use Makefile to organize command execution locally. Today, its role has declined, but Makefile is a good choice in certain situations. In most cases though, to offload the build command execution to a shared system, automation engines like Jenkins came around. Then Jenkins evolved into pipelines.

In strict terms, CI pipeline is the build pipeline; and CD pipeline is release pipeline. The two types of pipelines use pretty much the same pool of building blocks, with different purposes. The build pipeline focuses on producing quality artifact in a consistent manner. The release pipelines focus on system stability while deploying an artifact across different environments. Because release pipelines may connect to different environment, it has to deal with various situations. It is very common to have multiple stages in release pipeline, each stage pointing to a different environment (e.g. DEV, TEST and PROD). At workplace both could be loosely referred to as CI/CD pipeline, or even simply pipeline.

A lot of projects provide pipeline capability: BitBucket, Bamboo, TeamCity, Jenkins, Azure DevOps, AWS CodePipeline, TravisCI etc. Since late 2018, GitHub also joined the game with GitHub actions. It is openly free for public repositories, and has a free tier for private repositories. It executes task as defined in .github/workflow/action.yaml in the code project. I will take my own project as an example.

Runners

You can run jobs in self-hosted runners or GitHub managed runners, similar to other pipeline solutions (e.g. self-hosted agent vs managed agent from Azure DevOps). The GitHub hosted runners only have three operating systems to support: Windows, Ubuntu and MacOS. The Ubuntu and Windows runners are built from Standard_DS2_v2 VMs in Microsoft Azure. They are pre-installed with a virtual environment with packages required for common build tasks. The same virtual environment is also used in hosted agents by Azure DevOps. While they are free and you can elevate privilege on the runner, you cannot SSH or RDP to it for further troubleshooting.

The self-hosted runners require users to manage the instance on their own, including configuring virtual environment, installing GitHub Action Runner, etc.

Workflow file

Most pipeline declaration uses YAML or JSON, such as Jenkinsfile, AWS CodePipeline. GitHub refers to an automation process as a “workflow” and you can program the workflow in YAML (.github/workflow/action.yaml). Here is the reference and an example with environmental variable and versioning:

The handling of environment is documented here. There are a lot of custom actions available in GitHub Marketplace. For example, the versioning in the above example, uses an action by Einar Egilsson, which is open source itself.

Example pipeline

My example pipeline consists of two phases:

Build Library: spin up a docker container to build source code, and publish the artifact
Publish Image: add the artifact to an existing Docker image, and publish the result as my own image.

The status of the pipeline is also open, and can be found here. The retention period of artifact is 90 days by default but can be customized. To persist the artifact, I add it to my own Docker image and publish it to DockerHub, hence the second phase.

When building the second phase, I need to generate secret from my DockerHub account and store that encrypted secrets in GitHub settings, so that the secret value can be referenced in workflow file.

Failures in Actions are displayed in error steps and by default the rest of the steps are skipped.

I use the Docker build & push plugin to build and push my own docker image to DockerHub. Apart from DockerHub as my choice, GitHub also has its own artifactory GitHub Packages with a small free tier. It supports NPM, Docker, Maven, Gradle, etc.

Triggers of Action

Most of the times, GitHub action are triggered upon commit to main branch of the repo. In GitHub, this is known as a workflow_dispatch event. This is not the only event that can trigger GitHub action. All the available events are listed here on its documentation. This makes it very flexible to trigger action at many points in the workflow.

One example is to trigger GitHub action during PR review. When a developer opens a PR with a few commits in the proposed branch, the PR can preemptively check linting, style, etc and even build the application. These activities can also be defined in a GitHub action manifest with pull_request as triggering event.

Troubleshooting

In general, it is painful to troubleshoot activities happening inside of runners. I often had to write a few steps for the sake of printing variables, and trigger a run to see what their value is. This requires a lot of time especially when I have to wait for available runners.

To help troubleshooting pipeline runs there is an open-source utility called act. You can run GitHub actions locally from a Docker container on your MacBook. You can deliver environment variables and secrets via files. If you ever need to troubleshoot the runner environment, you have the option to connect to the Shell environment inside of the runner container. This tool is extremely helpful.

Closing remarks

GitHub action really makes the CI/CD pipeline capability available to any developers who stores their code on GitHub. GitHub expands from a code repository solution to a full CI/CD solution with a free tier sufficient for personal projects.