cloud run on Digi Hunch

Computing services: from PaaS to Serverless

Fri, 21 Oct 2022 19:31:00 -0400

Silicon Valley startups in mid-2000s likely do not run their own IT operations (i.e. renting their own data centre spaces, purchasing their own rack-mounted servers). Since the launch of EC2, AWS has been renting extra computing capacity to those startups, in the IaaS model. The leased infrastructure requires maintenance work, and AWS realized that many of these customers cannot afford specialized database admins, network admins, storage admins, or even server admins. As a result, they created a handful of managed services aiming to cut out admin overhead and let their customer focus on coding. This is how Platform-as-a-service (PaaS) came about.

Let’s take a look at what are exactly operation activities.

Ops activities

IT operation team manages server provisioning, installation of operating system, tuning performance, configuring auto scaling and load balancing, configure networking and storage systems, etc. Networking can be so complex that many infrastructure teams have a dedicated Network Operation Center (NOC), who along with security team, manages key aspects of networking, such as segmentation, router configuration, load balancing, firewall configuration.

For client-server application, the client-side code will make outgoing connections, utilizing the TCP/IP stack on the host through an ephemeral port. The server-side code has to be wrapped as a service. A daemon ensures the process running this service stays up and listens to a TCP port in order to respond to request by invoking the functions. Application team usually assumes these activities.

If database is involved, then the patching, upgrade, replication, data protection are all Ops problems. If storage is involved, then Ops has to manage mass data accumulated over years, the integration between storage and database and applications, performance, replication, etc. Some larger organizations have full-time database administrator and storage administrators.

Then comes container. Containers have their benefits but it increases the operation overhead by an order of magnitude. Running container application at scale warrants its own platform, most likely a Kubernetes platform, to address all of the problems above again at the cluster level. Some organization created platform team to manage container and VM platforms.

It is the Ops, that turns functional code into a running business. It is also the Ops, that becomes a pain point as a startup scales. With IaaS and PaaS models, AWS managed to convince many small businesses to delegate their IT operations to AWS. This is the humble start of cloud computing.

Elastic Beanstalk

At first, I wasn’t too impressed with Elastic Beanstalk, since it abstracts away too many details. However, I later realized that it has been surprisingly popular in the developer community, especially with individual developers and SMBs. It simplifies deployment to the point that their users don’t need to know other AWS services, allowing them to focus on coding application logic.

You configure Applications and Environments (one application may have multiple environments). In the Environment layer, you can specify code platform (e.g. Python 3.8 on 64bit Amazon Linux 2, Java, Go, PHP, Ruby) and even container platform (Docker on EC2 or ECS). Behind the scene, Elastic Beanstalk configures EC2 instances, Elastic Load Balancers, etc on the selected VPC and integrate with logging and monitoring services. In the console, Elastic Beanstalk exposes a list of configurations options (e.g. AMI, instance type). This centralized configuration page is dummied down for those who don’t want to deal with Ops.

The downside of Elastic Beanstalk is it takes away a lot of flexibility. Many developers find Elastic Beanstalk limit their choices of deployment, as their applications scale. Elastic Beanstalk does not suit for applications that demand extensive operation efforts. Its niche market is individual developers and SMB. Few enterprise applications run on Elastic Beanstalk.

Containerization with ECS and EKS

When we containerize an application, we build container images. Then we run these images with container runtimes, is a core feature of container platform. Container platform also provides orchestration engine since we frequently take containers up and down. In addition, container platform provides mechanisms for container networking and storage.

AWS has a couple options for container platform. ECS (Elastic Container Service) came out earlier. It organizes a group of EC2 instances as a cluster. You can manage autoscaling, networking, and persistent storage (EFS, FSx etc) on ECS. EKS (Elastic Kubernetes Service) is the managed Kubernetes service by AWS. Just like AKS, it provides a managed control plane along with computing nodes.

I see ECS as a proprietary and simplified container platform, and Kubernetes as an open-source standard for full-fledged container platform with an entire ecosystem. EKS includes an upstream-certified Kubernetes distribution with a set of tools specific to AWS. Since Kubernetes is the de-facto standard container platform, I prefer EKS by default, unless I can justify the use of ECS. In fact, ECS and Kubernetes have many concepts in common. For example, a “Task” in ECS is equivalent to a Pod in Kubernetes.

Whether it is ECS or EKS, right-sizing the computing node group is always challenging especially when the application traffic load is irregular. AWS Fargate is a technology that provides on-demand, right-sized compute capacities. It works with ECS and EKS. When integrated with EKS, we delegate the node management (e.g. scaling) to Fargate and forget about sizing the node pool.

Using ECS and Fargate involves quite a bit of configurations. To simplify that, we can use App Runner, which builds ECS cluster and uses Fargate to execute the container behind the scenes. App Runner helps client in a way similar to Elastic Beanstalk, but concentrate on Container workload.

Serverless with Lambda and API Gateway

The services above have their limitations when it comes to scaling capability. First, they cannot scale to zero. You still pay for idling resources. Also, it is not easy to find the optimal autoscaling setting. Lambda and API Gateway together solves these challenges. AWS refers to it as serverless, which has since become a buzzword. To understand what it is, let’s examine two concepts:

Function as a Service: service with the ability to execute code on demand. Users only pay for code execution time and do not care where the underlying runtime is
Backend as a Service: service with the ability to listen to a port and respond to web request

Lambda itself is a function as a service. Triggered by events, it only incurs a charge when it’s invoked. It does not stay up and listening to a TCP port for incoming web request, as does a backend service. In order to act as a backend, Lambda needs to pair up with API gateway. In this configuration, API gateway listens to a web request, and it fires an event to trigger the execution of Lambda function. Lambda and API gateway together makes a backend as a service. In AWS, the coupling of API gateway and Lambda function ensures an idle service does not incur computing cost.

Since Lambda supports many types of events as trigger, it is also used in event-driven architecture, either standalone or from a VPC. Under the hood, Lambda runs code in a container (with a quick startup time relative to a VM).

Developers can release Lambda code by uploading zip package to S3 bucket, or just packaging code into container image. For deployment, apart from AWS console and CLI, one can leverage CloudFormation, SAM (serverless application model), or CDK.

Lambda vs Fargate

Both Lambda and Fargate are serverless capabilities, at least from a marketing perspective. Both can be used to back web service but there are differences. They provision computing resource at different granularity.

In a web service, the execution duration of a Lambda function is the response duration to an API request, in terms of seconds. While the server is waiting for a request, there is no usage of the computing resource so you’re not paying for waiting for a request. However, this also creates the delay of cold-start, especially when the code size is large. There are several ways to optimize the cold start (e.g. SnapStart for Java), but none of those can completely get rid of the cold-start delay with a once-after-a-while request. A light GET call could take 5 seconds with cold start.

On the Fargate side, the resource provisioning is based on container lifecycle, instead of request lifecycle. As a result, you’re still paying for wait time, and it is not per-request billing. Since the container remains up, your request is not going to experience the cold-start if it’s been idle for a while. Although, Fargate saves you from the effort to right-sizing the computing nodes for container execution, it is not quite the idea of “scale-to-zero when idle” by itself.

Serverless Architecture

In the white paper AWS Serverless Multi-Tier Architectures with Amazon API Gateway and AWS Lambda, AWS advocates the serverless architecture as a modern alternative to the traditional widely adopted three-tier architecture (presentation, logic and data tiers). In the three tier architecture, the scalability of three tier are managed separately. The modern serverless architecture that AWS whitepaper proposes uses API Gateway and Lambda function in place of Load Balancer and EC2 instances (e.g. in an Auto Scaling Group), as illustrated below:

Both API Gateway and Lambda scale automatically to support the need of application workload. It assumes the role of logic tier in three-tier architecture but requires minimal maintenance work. For presentation tier, AWS has serverless alternatives such as CloudFront, S3. For data tier, AWS has serverless alternatives such as Amazon Aurora for relational database and DynamoDB for NoSQL. However, the “no request, no pay” model for Lambda does not apply to the data tier in serverless architecture.

Whilst this paradigm benefits small shop IT who wants to minimize infrastructure cost, it has downsides. There is no ability for infrastructure optimization. Since you do not manage where the code runs, client may have concerns over security (e.g. multi-tenant runtime). As business grows, keep using Lambda can result in technology lock-in. Also, a less used application usually requires warm-up time. A code start (downloading the code and preparing the environment behind the scene) can take 100ms to over a second.

Conclusion

PaaS attempts to help startups simplify the “grunt work” of IT operation. Serverless takes it even further. The semantics of serverless computing is confusing and the Wikipedia page acknowledges it as a misnomer. The nature of serverless model, is the cloud users delegate server capacity management to cloud platforms. The users don’t need to manage servers, VMs, instances, containers, etc on their own. In a previous post, I discussed the ability to scale to zero, which is just one of the many enabling technologies of serverless. Also, “no request, no pay” is neither an inherent nature of serverless model. Serverless service may involve storage (e.g. data service, S3, Aurora serverless) which incurs storage cost. There is a whitepaper on choosing the right AWS service to deploy your website or web application, with a decision tree.

AWS pioneered serverless with Lambda release in 2014 but competitor follows. In the Azure landscape, there is an entire suite of computing services from virtual machine to serverless (also with a decision tree in documentation). Azure’s counterpart for serverless architecture is Azure Function (released in 2016 for GA) with API Management. As for GCP, the serverless suite includes the event-driven Cloud Function (introduced in 2017) and Knative-based FaaS Cloud Run (introduced in 2019).

There are voices in advocacy of standardization of serverless model, and CNCF had since made minuscule efforts such as CloudEvents. The status quo, unfortunately, is anything but standardized.

Lastly, here is a table that summarizes the pros and cons of each computing service model.

Computing Service Model	Pro	Con
EC2	– Most straightforward and widespread legacy model – Legacy	– Ops tasks can be heavy (e.g. patch and vulnerability management of OS) – Utilization can be low
ECS	– Container orchestration is managed – Convenient to scale – Well integrated with other AWS services	– Limited advanced features – Vendor lock-in
EKS	– Highly scalable and flexible – Advanced, platform-neutral deployment tools available (e.g. Helm, ArgoCD, etc) – Custom configurations (e.g. operators)	– Significant operation overhead – Steep learning curve (especially for teams)
Lambda	– automatically scale – low ops overhead – pay per use	– limited choices of runtime – subject to latency due to cold start; yet warm start incurs cost – not suitable for long running tasks (batch processing jobs etc)

In summary, the Lambda-based serverless model is good for stateless server-side workload with short response time (<15s), tolerance of cold-start, no need for portability across platforms, and no complex package dependency.

Knative Eventing Introduction

Fri, 29 Apr 2022 10:16:00 -0400

In the previous post, I mentioned that Knative Serving and Knative Eventing should be seen as two different projects. The former is supposed to be widely used as a serving layer for microservices, whereas the latter has a narrower customer base. There are a dozen companies who need to build Platform as a Service, and will benefit from event-driven architecture. They are the niche customer for Knative Eventing. In this post, we go through two demos for Eventing.

Event Driven Architecture

Before jumping into Knative eventing, let’s first understand the value of event-driven architecture. As covered in Knative presentation in 2020, a growing micro-service architecture involves lots of service-to-service communications that looks like a spider web in a diagram. Two services communicating with each other are tightly coupled: changing one requires a change of the other. Event-driven architecture evolves to decouple those interdependent services. Knative eventing tries to standardize the event data with Cloud Event and also proposes a framework to drive this architecture.

The documentation uses different words to express the same concepts. So let’s take a note first: source = producer which emits events, and sink = subscriber = consumer which is the destination of events. Knative Eventing supports two models of in even-driven architecture.

Brokers and Triggers provides an “event mesh”. Event producers delivers events to a Broker. Triggers then distribute them uniformly by consumers.

Broker is a “hub” for events. It is a central location to receive and send events for delivery. We can choose from the following broker types:

MT (multi-tenant) channel-based broker (which a channel implementation, such as Kaka channel or an in-memory channel)
GCP broker
Apache Kafka broker
RabbitMQ broker

Triggers represents a desire to subscribe to events from a specific broker. It can act as a filter

channel-subscriber model

Channels and Subscriptions provide a “event pipe” model which transforms and routes events between Channels using Subscriptions. This model makes more sense when events out of one system needs to be transformed and then routed to another process.

A channel can be either a generic channel object, or a custom channel implementation (such as Kafka channel or in-memory channel)

The subscription consists of a Subscription object, which specifies the Channel and the Sink (aka the Subscriber) to deliver events to. You can also specify some Sink-specific options, such as how to handle failures.

The key difference is Broker-Trigger model allows you to filter events, where as Channel-Subscription model allows you to transform events.

Note that a Knative Service can act as both a Source and a Sink for events, and for good reason. You may want to consume events from the Broker and send modified events back to the Broker, as you would in any pipeline use-case. Between these components, Knative Eventing uses CloudEvents to send information back and forth.

Knative Eventing Lab

We can create a lab cluster using Kind as per the instruction in real-quicK-cluster. In docker-desktop settings, I configured 6 CPU and 8GB memory as the lab involves a number of nodes. The creation takes a couple minutes. This Knative eventing lab has four paths one can take, based on different model and different implementations:

Lab A demonstrates the configuration of broker-trigger model using Kafka based broker.
Lab B demonstrates the configuration of channel-subscription model, using in-memory channel.

The two labs share some common steps. We first install Knative Eventing using the provided manifests:

$ kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.3.0/eventing-crds.yaml
$ kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.3.0/eventing-core.yaml
$ kubectl get pods -n knative-eventing

The Pods should come to READY fairly quickly. In the mean time, we can configure the consumer as well using the manifests below:

apiVersion: v1
kind: Namespace
metadata:
 name: event-example
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: hello-display
 namespace: event-example
spec:
 replicas: 1
 selector:
 matchLabels: &labels
 app: hello-display
 template:
 metadata:
 labels: *labels
 spec:
 containers:
 - name: event-display
 image: gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
---
kind: Service
apiVersion: v1
metadata:
 name: hello-display
 namespace: event-example
spec:
 selector:
 app: hello-display
 ports:
 - protocol: TCP
 port: 80
 targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: goodbye-display
 namespace: event-example
spec:
 replicas: 1
 selector:
 matchLabels: &labels
 app: goodbye-display
 template:
 metadata:
 labels: *labels
 spec:
 containers:
 - name: event-display
 image: gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
---
kind: Service
apiVersion: v1
metadata:
 name: goodbye-display
 namespace: event-example
spec:
 selector:
 app: goodbye-display
 ports:
 - protocol: TCP
 port: 80
 targetPort: 8080

The consumers (aka sinks) will be able to process events at the validation step at the end.

Lab A – Install Kafka and configure Kafka-based Broker

This lab is for broker-trigger model. We first configure a Kafka Broker, which requires a Kafka cluster. The document uses Strimzi operator to quickly install Kafka cluster. Instead, here we use Helm to install the Kafka cluster:

helm install my-cluster-kafka bitnami/kafka -n kafka --create-namespace --set volumePermissions.enabled=true --set replicaCount=3

At the end of installation, the Helm notes indicate that the bootstrap server is my-cluster-kafka.kafka.svc.cluster.local:9092. All Kafka pods should come ready in about 5 minutes. After cluster install, we need to install Kafka controller and the Broker-layer data plane which allow you to map a Kafka instance to Broker CRD later:

kubectl apply -f https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.3.0/eventing-kafka-controller.yaml
kubectl apply -f https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.3.0/eventing-kafka-broker.yaml
kubectl get pods -n knative-eventing
kubectl -n knative-eventing edit cm kafka-broker-config

As the command above suggests, once data plane is installed, we edit the default broker config by updating the bootstrap.servers with the bootstrap server address noted above. We also need to make sure the replication factor is no greater than number of partitions, which should not be a problem in our lab as the replica count was set to 3.

Now we can create a Broker in the same namespace with consumer:

apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
 name: default
 namespace: event-example
 annotations:
 eventing.knative.dev/broker.class: Kafka
spec:
 config:
 apiVersion: v1
 kind: ConfigMap
 name: kafka-broker-config
 namespace: knative-eventing

Once the manifest above is applied, we should see the status of broker (named default) to be READY. We can also validate broker creation with kn CLI command:

 $ kubectl -n event-example get broker default
NAME URL AGE READY REASON
default http://kafka-broker-ingress.knative-eventing.svc.cluster.local/event-example/default 19m True
$ kn broker list -n event-example
NAME URL AGE CONDITIONS READY REASON
default http://kafka-broker-ingress.knative-eventing.svc.cluster.local/event-example/default 19m 7 OK / 7 True

Then we create Triggers that connects both to the broker with filters, and to the consumers by referencing service name. The manifest to create triggers is provided below:

apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
 name: hello-display
 namespace: event-example
spec:
 broker: default
 filter:
 attributes:
 type: greeting
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: hello-display
---
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
 name: goodbye-display
 namespace: event-example
spec:
 broker: default
 filter:
 attributes:
 source: sendoff
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: goodbye-display

We can confirm the setups above by monitoring the resources (like we did for broker), or use knative CLI:

$ kn trigger list -n event-example
NAME BROKER SINK AGE CONDITIONS READY REASON
goodbye-display default service:goodbye-display 3m24s 6 OK / 6 True
hello-display default service:hello-display 3m24s 6 OK / 6 True

Now we can go to the validation step to send event and validate result.

Lab B – Install in-memory channel and configure Channel

This lab is for channel-subscription model. we first install an in-memory channel:

kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.3.0/in-memory-channel.yaml

Then we can map this channel to a Channel resource in Eventing. We can do so by applying the manifest below:

apiVersion: messaging.knative.dev/v1
kind: Channel
metadata:
 name: demo-channel-1
 namespace: event-example
---
apiVersion: messaging.knative.dev/v1
kind: Channel
metadata:
 name: demo-channel-2
 namespace: event-example

The creation of Channel resources can be verified as below

$ kn -n event-example channel list
NAME TYPE URL AGE READY REASON
demo-channel-1 InMemoryChannel http://demo-channel-1-kn-channel.event-example.svc.cluster.local 6m34s True
demo-channel-2 InMemoryChannel http://demo-channel-2-kn-channel.event-example.svc.cluster.local 6m34s True

Now we need to create Subscription which connects Channels to Consumers. Subscriptions can be created with the manifest below:

apiVersion: messaging.knative.dev/v1
kind: Subscription
metadata:
 name: demo-subscription-1
 namespace: event-example
spec:
 channel:
 apiVersion: messaging.knative.dev/v1
 kind: Channel
 name: demo-channel-1
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: hello-display
---
apiVersion: messaging.knative.dev/v1
kind: Subscription
metadata:
 name: demo-subscription-2
 namespace: event-example
spec:
 channel:
 apiVersion: messaging.knative.dev/v1
 kind: Channel
 name: demo-channel-2
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: goodbye-display

We can confirm subscription:

$ kn -n event-example subscription list
NAME CHANNEL SUBSCRIBER REPLY DEAD LETTER SINK READY REASON
demo-subscription-1 Channel:demo-channel-1 service:hello-display True
demo-subscription-2 Channel:demo-channel-2 service:goodbye-display True

Now we can go to validation step.

Validation

For validation, we need to first configure an event source. The source in our lab is a throw-away Pod with curl utility. I use my favourite nicolaka netshoot to fire off the events using. To launch the Pod and get to command shell:

$ kubectl run tmp-shell -n event-example --rm -i --tty --image nicolaka/netshoot -- /bin/bash
If you don't see a command prompt, try pressing enter.
bash-5.1#

We can POST message to the corresponding endpoint URL to fire events. The endpoint is different for each lab:

Lab	URL
A (Kafka-based broker)	http://kafka-broker-ingress.knative-eventing.svc.cluster.local/event-example/default
B (channel)	http://demo-channel-1-kn-channel.event-example.svc.cluster.local http://demo-channel-2-kn-channel.event-example.svc.cluster.local

With the correct URL, we can build a payload for POST method. Below is three curl commands with different payload POST to the same endpoint (taking lab A as an example):

# curl -v "http://broker-ingress.knative-eventing.svc.cluster.local/event-example/default" \
-X POST \
-H "Ce-Id: say-hello" \
-H "Ce-Specversion: 1.0" \
-H "Ce-Type: greeting" \
-H "Ce-Source: not-sendoff" \
-H "Content-Type: application/json" \
-d '{"msg":"Hello Knative!"}'

# curl -v "http://broker-ingress.knative-eventing.svc.cluster.local/event-example/default" \
-X POST \
-H "Ce-Id: say-goodbye" \
-H "Ce-Specversion: 1.0" \
-H "Ce-Type: not-greeting" \
-H "Ce-Source: sendoff" \
-H "Content-Type: application/json" \
-d '{"msg":"Goodbye Knative!"}'

# curl -v "http://broker-ingress.knative-eventing.svc.cluster.local/event-example/default" \
-X POST \
-H "Ce-Id: say-hello-goodbye" \
-H "Ce-Specversion: 1.0" \
-H "Ce-Type: greeting" \
-H "Ce-Source: sendoff" \
-H "Content-Type: application/json" \
-d '{"msg":"Hello Knative! Goodbye Knative!"}'

All the requests should receive 202 return code. As highlighted above, we specified type and source for each event fired to the broker. On the consumer side, we can simply check the pod log to verify the events received. Below is an example of events received as a result of the events sent above (take broker-trigger model as example):

$ kubectl -n event-example logs -l app=hello-display --tail=100
2022/03/09 17:05:59 Failed to read tracing config, using the no-op default: empty json tracing config
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: greeting
 source: not-sendoff
 id: say-hello
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:19.196589801Z
Data,
 {
 "msg": "Hello Knative!"
 }
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: greeting
 source: sendoff
 id: say-hello-goodbye
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:42.977803608Z
Data,
 {
 "msg": "Hello Knative! Goodbye Knative!"
 }
$ kubectl -n event-example logs -l app=goodbye-display --tail=100
2022/03/09 17:05:58 Failed to read tracing config, using the no-op default: empty json tracing config
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: not-greeting
 source: sendoff
 id: say-goodbye
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:27.430702588Z
Data,
 {
 "msg": "Goodbye Knative!"
 }
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: greeting
 source: sendoff
 id: say-hello-goodbye
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:42.977803608Z
Data,
 {
 "msg": "Hello Knative! Goodbye Knative!"
 }

This is the gist of Knative evening. Note that the format of HTTP payload when we fire off events needs to conform to CloudEvents format. The container image (event_display) used in the container is also created for the purpose of processing events received and display them in stdout.

Conclusion

Persistent storage allows for stateful applications on Kubernetes. It also enables architectural patterns that requires persistent storage. Knative eventing is a framework that enables event driven architecture. It supports many messaging channels (including NATs, a cloud-native messaging system) and broker types (MT channel based, GCP, RabbitMQ).

Knative Serving Introduction

Sun, 17 Apr 2022 11:33:00 -0400

Background

As per IBM‘s definition, Knative enables serverless workloads to run on Kubernetes clusters, and makes building and orchestrating containers with Kubernetes faster and easier. It has drawn a lot of attention recently. It released version 1.0 in November 2021, and was accepted as a CNCF incubating project in March 2022.

Glories aside, the value it delivers is the ability to go serverless on a Kubernetes platform. Originally, Knative was built with three components: build, serving and eventing. The build component was deprecated in favour of the Tekton project. Tekton is cloud native CI/CD pipeline. It connects to source code repo and build artifacts. It can also deploy applications with pipeline as code. Tekton’s documentation includes a quality tutorial here linked to interactive terminal. On the other hand, we’ll focus on Knative Serving in this post, and Knative Eventing in the next.

Knative can be seen as a serverless framework with a number of open-source technologies as building blocks, such as Istio for ingress gateway, Kafka or Google Pub/Sub as event-streaming engine, and Prometheus for observability to name a few. Despite of using the same CLI tool (named kn), Knative Serving and Eventing are considered separate capabilities using different groups of CRDs. They are installed separately, using their respective YAML files, or operator.

Serving vs Eventing ?

After reading the article “Did we market Knative wrong” from Ahmet Balkan, my impression is the two are not related. They could have been two separate projects without sharing a common name. As the author puts, these two shared some core logics. But beyond that, they don’t have anything in common.

Serving is lightweight and requires Istio. It provides the capability to scale from N to 0 when the system is idle, and from 0 to N when requests come in. It is the missing serving layer for running microservices on Kubernetes and Ahmet position it as the first thing that people installed after creating a cluster.

The target user of Eventing is much smaller. Eventing is for event-driven architecture and is more complex than Serving. Ahmet admits that they over-estimated how many people on the planet want to build a Heroku-like PaaS layer on top of Knative. There are a couple of dozen companies who would work through the complexity and build their own Kubernetes-based internal PaaS or even public-facing FaaS using Knative. They are the niche audience of Knative Eventing.

For most platform builders who just need to run micro-service, Knative Serving is all they need.

A Demo on Serving

The definition of term “serverless” is very loose. Instead, Knative documentation promotes the ability to scale to zero and refer to it as “some people call this Serverless”. This suggests that “the ability to scale to zero” should be the better term to describe this capability. To deploy it, we need the Service object with apiVersion serving.knative.dev/v1. Below is a guide for a quick hands-on, with some steps modified from the Knative Serving tutorial and installation guide, with optional steps (extensions, DNS) skipped.

Follow the first part of this post or this guide in real-quicK-cluster project, to install minikube with metal LB and configure istio (using istioctl as per instruction). Then Knative serving can be installed with two YAML manifests:

$ kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.2.0/serving-crds.yaml
$ kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.2.0/serving-core.yaml

Then we can install Knative istio controller to integrate them so we can see the knative gateways:

$ kubectl apply -f https://github.com/knative/net-istio/releases/download/knative-v1.2.0/net-istio.yaml
$ kubectl -n knative-serving get gateway

Now we can install the dummy service, either using kn CLI tool or using YAML manifest. The manifest is given below:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
 name: hello
 namespace: default
spec:
 template:
 metadata:
 name: hello-world
 annotations:
 autoscaling.knative.dev/class: "kpa.autoscaling.knative.dev"
 autoscaling.knative.dev/metric: "rps"
 autoscaling.knative.dev/target: "50"
 autoscaling.knative.dev/min-scale: "0"
 autoscaling.knative.dev/max-scale: "10"
 spec:
 containerConcurrency: 0
 containers:
 - env:
 - name: TARGET
 value: World
 image: gcr.io/knative-samples/helloworld-go
 name: user-container
 ports:
 - containerPort: 8080
 protocol: TCP

We can notice that in the annotations there are some settings to overwrite the default configuration. They are explained in the autoscaling section of the Knative serving documentation. Here we use request per second (rps) as metric, and have 50 as target. The scale range is between 3 and 10 replicas. Once the resource is created, we can validate it with one of the two commands:

$ kubectl get services.serving.knative.dev
$ kn service describe hello

The creation of knative service also created a number of Kubernetes resources, such as deployment and services:

$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello ExternalName <none> knative-local-gateway.istio-system.svc.cluster.local 80/TCP 52s
hello-world ClusterIP 10.110.127.231 <none> 80/TCP 108s
hello-world-private ClusterIP 10.111.27.73 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP 108s
kubernetes ClusterIP 10.96.0.1 <none>
$ kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
hello-world-deployment 1/1 1 1 71s

Before testing, let’s mock the DNS entry to resolve to the Istio ingress service IP like below in /etc/hosts:

192.168.64.16 hello.default.example.com

Now we can confirm that the service returns the expected result (after a brief pause):

$ curl hello.default.example.com
Hello World!

Wait for a short period without curl or connection from browser, the service should scale down to zero:

$ kubectl get deploy -l serving.knative.dev/service=hello -w
NAME READY UP-TO-DATE AVAILABLE AGE
hello-world-deployment 0/0 0 0 78s

From a separate terminal, emulate HTTP request with hey command, so the service receives more than 50 rps per pod:

$ hey -n 200 -c 20 -z 20s -m GET http://hello.default.example.com

We can watch the deployment get scaled up to 10. Once the hey command completes its task, the deployment will scale back down:

$ kubectl get deploy -l serving.knative.dev/service=hello -w
NAME READY UP-TO-DATE AVAILABLE AGE
hello-world-deployment 0/0 0 0 78s
hello-world-deployment 0/1 0 0 3m54s
hello-world-deployment 0/1 0 0 3m54s
hello-world-deployment 0/1 0 0 3m54s
hello-world-deployment 0/1 1 0 3m54s
hello-world-deployment 1/1 1 1 4m2s
hello-world-deployment 1/10 1 1 4m4s
hello-world-deployment 1/10 1 1 4m4s
hello-world-deployment 1/10 1 1 4m4s
hello-world-deployment 1/10 10 1 4m5s
hello-world-deployment 2/10 10 2 4m9s
hello-world-deployment 3/10 10 3 4m10s
hello-world-deployment 4/10 10 4 4m10s
hello-world-deployment 7/10 10 7 4m10s
hello-world-deployment 9/10 10 9 4m10s
hello-world-deployment 10/10 10 10 4m11s
hello-world-deployment 10/5 10 10 5m18s
hello-world-deployment 10/5 10 10 5m18s
hello-world-deployment 5/5 5 5 5m18s
hello-world-deployment 5/2 5 5 5m20s
hello-world-deployment 5/2 5 5 5m20s
hello-world-deployment 2/2 2 2 5m20s
hello-world-deployment 2/1 2 2 5m22s
hello-world-deployment 2/1 2 2 5m22s
hello-world-deployment 1/1 1 1 5m22s
hello-world-deployment 1/0 1 1 5m52s
hello-world-deployment 1/0 1 1 5m52s
hello-world-deployment 0/0 0 0 5m52s

Autoscaling in Knative serving is backed by KPA (Knative Pod Autoscaler, used in the example above) or HPA. Also note that the trigger to scale up and down is client connection (RPS). This is different than KEDA. KEDA also supports the capability to scale to zero. However, with KEDA, it is an event that scales a deployment up from zero. In Knative serving, it is the connection to the Service itself that wakes up the service.

Cost of scale-to-zero

As discussed, both KEDA and Knative serving supports the ability to scale to zero and wake up from zero. The trigger to wake up from zero is different. With KEDA, workload wakes up by an event. With Knative Serving, workload has to wake up upon receiving a connection to the service. In the design to solve this problem, Knative has four sub-components :

Activator – When a service is scaled to zero, its request are routed to the activator which waits for a Pod to become ready (wake up) and proxies the traffic while editing the underlying Endpoints to route the traffic directly to the Pod(s)
Autoscaler – KPA
Controller – The main component responsible for watching API objects for Knative CRDs (KService, Configuration, Route, Revision) and manage their lifecycle, create the underlying Kubernetes resources and garbage-collect old objects.
Webhook – a Kubernetes Admission Webhook acting both as validating admission controller and mutating admission controller.

The key player is activator, which receives requests when a service is scaled to zero. The wake-up process is summarized as follows:

After receiving request, activator will buffer (hold onto) the request
Then, it will look at request’s hostname to find which KService it is for.
Then, it will scale up the Kubernetes Deployment and wait for a Pod to become ready.
In the meanwhile, it updates Kubernetes Service to point to Pod IP addresses (so that activator gets out of the network path if the KService is awake).
Finally, activator proxies the request to the started pod

So the “plumbing” of the service connection managed by the Activator pod is the cost of scale to zero. This is also called Load Balancing, and the behaviour can be tweaked using two parameters: activator capacity and target burst capacity:

Target burst capacity: once the target deploy has one more more Pod, service requests may still route via activator or bypassing the activator. Target burst capacity determines at what point service request should start to bypass activator.
Activator capacity: determines how many requests can activator hold on to. Considering the service-to-service connection also routes via Service CRD, there can be a lot of connections via Activator so its capacity needs to be adjusted.

The additional overhead of managing and optimizing the “Load Balancer” is also part of operational cost for the ability to scale to zero.

In comparison, KEDA now has an HTTP add-on still at beta but allows connection-based wake-up. The design is a little different. As discussed in this previous post.

Conclusion

Knative consists of two disparate components: Serving and Eventing. Serving uses KPA to provide scaling based on service request, and the ability to scale to zero. It is often compared with KEDA, a single-purpose lightweight tool for autoscaling. Here is a blurb on their difference by KEDA. The takeaways is that KEDA is more focused on scalability, whereas Knative serving covers more aspects. For example, the Service object of Knative also supports Traffic Splitting. Knative serving can integrate with istio.