rosa on Digi Hunch

Authentication to kube-apiserver via OIDC

Fri, 28 Jul 2023 09:20:00 -0400

Background

There are many benefits of using OIDC to authenticate to kube-api server, especially with multiple clusters that need consistent log-in experience. With the last post on how OIDC Authorization Code Flow works, now I will discuss options with authentication for kubectl to connect to kube API server.

To start, let’s look at the anatomy of kubeconfig file. The full schema is in the documentation. Looking at my kubeconfig file, there are three sections:

clusters: each entry specifies a cluster’s name, server address and certificate authority data (in base64 encoding or a file location).
users: each entry specifies a username. Some users are identified with client key and certificate. Some specify a command to provide client authentication. Refer to the authentication strategies.
contexts: each entry links a user to a cluster

Therefore, the key to use OIDC integration, is to use command to provide client authentication.

Vanilla Kubernetes

The documentation on authenticating has a diagram on how to use OpenID Connect tokens. The diagram does not give details on how access_token and id_token were obtained. So it could be any OIDC flow (Authorization Code Flow, Implicit Flow, etc) as we have discussed.

Although the instruction does not mandate which OIDC flow to use, we should use Authorization Code Flow in this architecture. The API server needs to trust the OIDC issuer, and the document covers how to configure API server.

In the diagram, step 2 and step 3 are required by kubectl itself does not perform these activities. All kubectl does is carry the JWT token in the Authorization Bearer. Vanilla Kubernetes does not provide a solution for OIDC integration. It only provides some instructions and we still need some helper scripts to glue all these instruction steps together.

There are many open-source project for this purpose. For example, Jetstack has kube-oidc-proxy and Int128 developed kubelogin. Other projects such as k8s-auth-client, k8s-oidc-helper, and gangway are no longer being updated. The kubelogin project remains influential. It has a clear diagram too:

From the diagram we can see kubelogin proposes authorization code flow.

Also, one design concern with Kubernetes control plane is the placement of endpoint. From this diagram we can see that even if the cluster endpoint is on private subnet, OIDC integration should still work. The control plane (specifically kube-apiserver) initiates outbound connection to OIDC Provider. There is no inbound connection to it from the OIDC provider.

In the kubelogin setup, the redirect URI is set to localhost:8080 because it stands up a server on the same host where browser is running. The browser can always resolve localhost. For a full configuration steps, Okta has this blog post on how to use kubelogin as helper, and Okta as Authorization Server to authenticate kubectl via OIDC.

Step-by-step with kubelogin

We’ll go through an example with int128/kubelogin because it works with any Kubernetes flavour, including managed Kubernetes services. It is also fairly simple. The instruction covers a few types of Authorization Servers (Google Identity Platform, KeyCloak, Dex with GitHub, Okta and Ping Identity). I’ll take KinD cluster as an example and use Azure AD as Authorization Server.

First, we’ll register an App in Azure Portal. Go to “App Registrations” and “New registration”. Give it a name “kubeoidc” and set Redirect URI to “Web” with URL “localhost:8000”. Click on Register.

The next page shows the details for this app. The Application (client) ID is important for next steps. Click on “Add a certificate or secret”, then “New client secret”, put in expiry and description. The secret value is generated and displayed on the next page, which is important for our next step.

We also need to find out the issuer URL. From the App page above, click on “Endpoints” and find out the URL from field “OpenID Connect metadata document”. My metadata document URL looks like:

https://login.microsoftonline.com/xx8x8xx8-7777-66yy-55b5-444aaaaa3322/v2.0/.well-known/openid-configuration

The OIDC Issuer URL is the part before .well-known. In this case, it is:

https://login.microsoftonline.com/xx8x8xx8-7777-66yy-55b5-444aaaaa3322/v2.0

Now we have collected what we need for the next few steps: ClientID, ClientSecret and OIDC Issuer URL. We can then create the KinD cluster, and reference ClientID and IssuerURL in the cluster configuration:

cat << EOF > kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
 - role: control-plane
 kubeadmConfigPatches:
 - |
 kind: ClusterConfiguration
 apiServer:
 extraArgs:
 oidc-issuer-url: https://login.microsoftonline.com/xx8x8xx8-7777-66yy-55b5-444aaaaa3322/v2.0
 oidc-client-id: 99999e88-e777-6666-c5c5-c444444d3d22
 - role: worker
 - role: worker
 - role: worker
EOF

kind create cluster --config=kind-config.yaml

The command will create cluster and configure kubeconfig file with a user named kind-kind as admin, which isn’t what we need. Now we use kubelogin helper command:

kubectl oidc-login setup \
 --oidc-issuer-url=ISSUER_URL \
 --oidc-client-id=YOUR_CLIENT_ID \
 --oidc-client-secret=YOUR_CLIENT_SECRET

This command will act as the client in the OIDC flow and prompts you to log in to Azure in a Browser. Once logged on, it gives you the next few commands to run. Since we’ve already created a cluster with the Issuer URL and Client ID, we can skip creating cluster, and run the steps to:

bind the Azure user to a ClusterRole
set up the kubeconfig locally with a user oidc, which needs to execute the oidc-login command

Lastly, we can test the oidc user with kubectl --user=oidc get nodes. We can also set the context to use oidc user by default. Voila.

From this example, we learned how to configure OIDC integration for any Kubernetes distros. The steps that we need to take are:

On the cluster side, we customize kube-api-server flag with OIDC provider info. Specifically, we provided Issuer URL and client ID in this example. In the OIDC Flow diagram, this step establish a trust from the Resource Server (K8s cluster) to the OIDC provider.
On the kubectl side, kubectl itself cannot fulfill all the duties of a client app in the OIDC Flow. It needs a helper script and we’ve made friend with int128/kubelogin.
On the Identity Store side, we expect it to be an OIDC-compliant Authorization Server. Otherwise, we consider using Dex as a broker in between.

When it comes to managed service, many allows us to customize the OIDC related flags for kube-api-server. Let’s look at how some managed services get this to work.

Azure Kubernetes Service

I find the kubectl authentication with AKS highly opinionated in its documentation. The recommendation is using Azure Active Directory as identity store but I don’t find it work with other OIDC providers.

To work with Azure Active Directory, you would configure the cluster and specify Azure role-based access control with the Group UUID in Azure AD. On the client side, you will need to install Azure’s kubelogin utility. Do not confuse it with int128’s kubelogin, which is for any cluster. This kubelogin is just for Azure. Once installed, you can use az-cli command to update your kubeconfig file, which call this utility from kubeconfig. To examine details about client configuration and AAD integration, check out the Terraform template in the azure directory of my cloudkube project.

The document only covers Azure AD integration and I tried to find if there’s a way to integrate with third-party OIDC providers. Unfortunately I have no luck. ChatGPT points me to a page about enabling OIDC provider but it is in the context of workload identity and it does not allow you to customize the issuer. So it’s completely irrelevant. Because you cannot customize OIDC issuer, etc, it simply won’t work with any third-party OIDC provider. Sure enough, most of Azure’s client use Azure AD anyways.

Azure makes it streamlined to configure OIDC integration of AKS with Azure AD, its own identity store. To my disappointment, it is currently not possible to integrate with third-party OIDC provider for authentication at cluster endpoint.

ROSA

RedHat OpenShift on AWS (ROSA) is a. However it reflects how OpenShift configures third-party identity provider.

Let’s first create a cluster off custom VPC and private endpoint. This is covered in my previous post about ROSA. After the cluster creation, we’re at the point where we can run oc command against cluster endpoint from Bastion host because it is a private cluster. However, being a private cluster is irrelevant to how we configure OIDC integration.

We’ll use Azure AD again as OIDC provider. So we need to register an app the same way as I did above in the kubelogin example. We need to have Client ID, Secret and OIDC issuer URL. For RedirectURI, go to your OpenShift console, and under Cluster, click on the cluster name → access control → identity providers → select OpenID. Note the page clearly states that this is Authorization Code Flow, and the OAuth Callback URL is provided. Use it to regiter App in Azure, and fill in the page with Client ID, Secret and OIDC issuer URL. Also fill in other fields accordingly and click on Add. Now you should have it configured!

Identity Providers for OpenShift cluster

The next step is trying to login. From Bastion host, run oc login and it will give me an URL to use. The URL contains the cluster endpoint, which resolvable from the Bastion Host itself. However I need a Browser session here, so I have to run Bastion host as SOCKS5 proxy and tell Chrome on my MacBook to use it:

~ open /Applications/Google\ Chrome.app --args --proxy-server="socks5://localhost:1080"

The browser session redirects to Azure AD for log in. Once completed the webpage will display a token that I can use with oc login command. Run this command with token from Bastion, I’m logged in:

oc login --token=sha256~3ZT5JGWELOcBzfohftEm9D2UwoOVFvATASuZk3_uxps --server=https://api.dhc.62q3.p1.openshiftapps.com:6443
oc whoami
oc get no

At this point, if I run oc whoami, I get the user name. However, this user cannot do anything. This is because it is not associated with a role yet. You grant more permission to this user: go back to OpenShift console, Clusters → ClusterName → Access Control → Cluster Roles and Access → Add user. Here you can map the user name to a role (let’s say ClusterAdmin). Then this user will have its priviledge:

The whole Flow works with private cluster, because the redirect URI is resolvable from the Bastion host. If you chose to expose cluster endpoint publicly (not recommended), you can perform the above steps directly from your MacBook or Laptop.

So the ROSA experience has been smooth. Unlike kubectl, the oc utility can act as the Client App in Authorization Code Flow. The other part of the configuration such as client secret and issuer URL are made in OpenShift console. Good job!

Elastic Kuberentes Service

EKS allows you to specify OIDC issuers from console or CLI to set up third-party OIDC configuration. There is a blog post from Okta on this, which works for private clusters. In the instruction, the author first manually created kubeconfig file with int128 kubelogin, and then bind ClusterRole with the user.

The blog post is very detailed. Instead of repeating it, I would like to discuss two SSO models available in AWS. I summarize them as below:

Both are SSO models for EKS. Model 1 (IAM Identity Center) is home grown as AWS using IAM Identity Center (formerly AWS SSO). Users start with an IAM principal (AWS construct) and use the aws-auth config map to tie them to Kubernetes groups. This AWS blog post and this support article are based on the IAM Identity Center model. On the other hand, Model 2 (Direct OIDC) is the vanilla Kubernetes approach. It takes group claim from OIDC identity token. The Okta blog post is based on this model.

The IAM identity center model works with SAML compliant identity providers, oftentimes Active Directory, although there seems to be a plan to support OIDC-compliant identity provider as well in the future. Even if it was supported today, I’d still prefer model 2 for working with any OIDC compliant identity provider because it’s simple. Why not leverage K8s’ native capability? For identity providers that do not support OIDC natively, or does not issue group claim (e.g. Google workspace), as we discussed, we can also consider alternatives such as Dex as identity broker. However, this model comes handy when a pipeline user with IAM role needs to authenticate into EKS.

One should always go for the direct OIDC model, if the upstream identity provider supports OIDC. The provider itself can even be AWS Cognito User pool. Cognito User Pool itself supports federated identity but again, I would directly connect EKS cluster to the OIDC compatible identity provider, instead of going through Cognito User Pool. As a result, the only use case where Cognito user pool is absolutely necessary, is when we need the Cognito user pool itself as the identity provider, as we have in CloudKube‘s eks directory.

Summary

Configuring OIDC provider for cluster endpoint can be confusing and we need to understand how OIDC flows work. I dived into OIDC in the previous post and in this post, I explained how to get it to work with vanilla Kubernetes. I summarized the three requirements and looked at the OIDC provider option with some managed services. Then I went through OIDC options for AKS, ROSA and EKS.

Kubernetes Platform as a Service and Red Hat OpenShift

Sun, 25 Jun 2023 11:10:15 -0400

The Three-layer model

Kubernetes is so complex that it becomes a buzz word itself. I categorize the related work into three layers: a cluster layer, a platform layer and an application layer, by their purposes. The three layers are illustrated as below:

Let’s examine each layer in this model and where the Kubernetes Platform as a Service fits in.

The Kubernetes Cluster Layer

At the bottom, the Kubernetes Cluster layer is the foundational layer. It focus on using self-hosted VMs or cloud resources to build a functional Kubernetes cluster and worker node groups. A functional cluster includes a highly available control plane, as well as scalable node groups that all communicate with the control plane. Cloud Service Providers like AWS and Azure provides managed Kubernetes service, which takes away the complexity (and flexibility as well) of managing control plane components such as etcd store and API server. The managed services also automatically provisions computing nodes and join them into the cluster. The cluster layer may also involve integration with of CNI and CSI, to ensure Pod-to-Pod communication and available storage classes.

Professionals working at this layer are infrastructure experts who understand networking, storage, as well as how to manage cloud resources or VMs, infrastructure as code. On a daily basis, they deal with VPCs/V-Nets, subnets, EBS/Azure Disk, File storage, EC2/Azure VMs, etc. When the team is doing a bad job at this layer, you might see symptoms like unresponsive cluster API, orphaned worker nodes, or kubectl failing to connect to cluster endpoint.

The tenants (applications) of the Kubernetes platform does not directly interact with this layer. If you decide to switch CSP vendor, this layer requires 100% re-engineering because the managed Kubernetes service by each CSP is different.

The Kubernetes Platform Layer

The Platform layer sits in the middle. When organization decides to adopt Kubernetes, they often underestimate the efforts required in this layer. This layer works on a functional cluster, without directly interacting with the underlying cloud resources. This layer involves any Kubernetes abstractions that do not creates tangible business value. Rather, this layer is an enabler. It allows the applications to deploy smoothly, evolve quickly, and more importantly, focus on the business.

Teams working on this layer needs to be Kubernetes experts. On a daily basis, they play with common CNCF toolings, such as Prometheus, ArgoCD, Istio, Cilium, Tekton, Open Policy Agent, etc. They are comfortable with Operators, Helm Charts, Ingress, etc. Inside of the Kubernetes cluster, they also manage the foundational services such as Event streaming (e.g. Kafka), PostgreSQL database (e.g. PostgreSQL), software-defined storage (e.g. Ceph), service mesh (e.g. Istio), Authentication (e.g. Keykloak) , etc. These services act as the infrastructure layer to the business workload. If the team is doing a bad job, you would see data loss with database, observability service not populating data, ingress does not process request, etc.

The tenants (application) share services in this layer. If you decide to switch CSP vendor. I estimate 80% of the work at this layer is portable, and 20% requires re-engineering. That is because each CSP offers different external resources, therefor the low level Kubernetes objects in this layer, such as storage classes, load balancers, supported CNIs are different. High level objects such as Kafka remains portable across platforms.

The Application Layer

The next layer at the top is application layer. Workloads in this layer are directly linked to the business value. The applications are very diverse. Most of the time, the release team is the main player at this layer. If the organization develops its own application, the software development team also work at this layer.

In terms of knowledge, the members of development team are experts in software engineering, and Software Development Life Cycle (SDLC), etc. On a daily basis, they deal with programming languages, product development, build and release. If they screw up their work, expect business errors, such as orders sent to wrong client, incorrect balance sheet, etc. This team has high visibility in the organization due to its direct link to business value.

This layer of work involves multiple tenants. Each tenant is isolated within their own namespace. When you switch CSP vendor, this layer should be readily portable with minimal effort.

It is also worth noting that, with solid platform and cluster layers, the team working at this layer do not write bespoke code for networking, observability, authentication and authorization, encryption and many other aspects not relevant to the core business. Once deployed, the application services are resilient, scale to demands, and cost efficient. This layer reaps the benefits of Kubernetes.

Kubernetes Platform as a Service

As the Kubernetes dust is still settling, a builder’s title may not always reflect which layer she or he focuses on. Today it is pretty common for infrastructure engineers to expand their role into the platform layer, or likewise, a software engineer to drill down to the platform layer.

The boundary between platform layer and cluster layer is clear. The cluster layer deals with underlying infrastructure, either in the cloud or on premise. They abstract away the complex infrastructure world from those working with the platform layer.

The boundary between platform layer and application layer is a little tricky to articulate. The application layer focuses on implementing the business logics. The platform layer takes care of the functions that are not part of business logic but essential to the business application. Take an HTTP request for example, application developer should not have to write code to terminate TLS (not part of business logic). They should only write the code to process the HTTP request (business logic). TLS termination is delegated to an Ingress, to be configured by platform builders.

The folks working at the Platform layer needs to interface with both sides. They provide Platform as a Service to the Application teams. However, their work appears mostly invisible in an organization. Their effort is oftentimes underestimated. There are several reasons for that. First, the platform layer does not directly create tangible business value. They are just someone else’s enabler. Second, their building blocks involve a lot of abstractions by Kubernetes API. Third, the idea of platform engineering is newly emerged. There hasn’t been a populous recognition of its value.

Red Hat OpenShift Container Platform

The platform team builds the platform with their choice of open-source tools. For clusters using OpenShift Kubernetes Engine, Red Hat introduces Open Shift container platform consisting of Red Hat’s opinionated (but validated) choice of toolings, for example:

OpenShift Service Mesh: Istio
OpenShift Streams: Apache Kafka
OpenShift GitOps: ArgoCD
OpenShift Container Platform Pipelines: Tekton
OpenShift Serverless: Knative
OpenShift Data Foundation: Ceph

Clients building their clusters with OpenShift Kubernetes Engine may build their own platform with the toolings in the OpenShift enterprise Kubernetes container platform. For more services, check out the documentation for OpenShift Container Platform. For customers with OpenShift Kubernetes Engine, their options to DIY platform are:

Entry-Level: Red Hat OpenShift Kubernetes Engine: Enterprise Kubernetes distribution on RHEL CoreOS
Mid-Level: Red Hat OpenShift Container Platform (RHOCP):
Plus-Level: Red Hat OpenShift Platform Plus: RHOCP + advanced cluster management, security, data management essentials, enterprise container registry

OpenShift runs the business model of Kubernetes PaaS.This is a unique business model that I do not find a matching competitor. Even if you choose to DIY your own platform, the Red Hat’s choices are still a great reference. The OpenShift enterprise Kubernetes container platform maps perfectly to the platform layer of the three-layer model, aiming to simplify the work in the platform layer.

Managed RedHat OpenShift

At first, the OpenShift container platform started as a value add-on to the Kubernetes Engine. Now it’s a separate product line in their business model. In the mean time, OpenShift partners with major CSPs, to develop the cloud service editions, including:

Red Hat OpenShift on AWS (ROSA)
Microsoft Azure Red Hat OpenShift (ARO)
Red Hat OpenShift Dedicated – on AWS and GCP
Red Hat OpenShift on IBM Cloud

These offerings are managed Kubernetes Platform as a Service in the cloud. Since RedHat is the only player in this model, we can refer to them as managed OpenShift services. In addition to an already-confusing world of Kubernetes platform portfolios, these offerings gives consumers even more options. On AWS for example, users have the following options:

Managed Platform: OpenShift Dedicated, managed by Red Hat
Managed Platform: Red Hat OpenShift Service on AWS (ROSA), managed by Red Hat and AWS
Self-built cluster: OpenShift Container Platform

This video discussed more details about these options, such as support model. It is also worth noting that these options tend to be much pricier than managed clusters such as EKS and AKS.

Since a Managed RedHat Platform makes it easy to deploy, let’s take ROSA as an example and create a cluster. To enable ROSA in AWS console, click on “Getting Started”. The next page ensures ROSA is enabled and checks other prerequisite such as meeting service quotas and creating ELB service-linked role, as show below:

Now, with an AWS account (and ROSA enabled), a RedHat account, and the rosa-cli utility, we can create a cluster with just a few commands. As a note, be wary of the cost and do not forget to delete the cluster afterwards.

Create a ROSA cluster

With the following set of commands, we can kick off cluster creation, using STS. We can bring our own VPC, so long as it meets certain prerequisites. I use the Terraform template in the vpc-base project, to create the underlying VPC. We’ll need the followings from this template:

The CIDR range of the VPC: as input with a default
The subnet Ids of the private subnet to place, printed in the output

The subnets are private subnets, because we want to provision the cluster with private node and private endpoint. When we use rosa CLI, we provide the CIDR and subnet IDs.

# start with AWS cli configured to the correct profile
rosa login # with redhat account and past token
rosa create account-roles --mode auto -y # this command creates the IAM roles ManagedOpenShift-*-Role, with RedHat account as trust entity
rosa verify permissions # optional
rosa verify quota # optional

export ROSA_CLUSTER_NAME="dhc" \
OPENSHIFT_VERSION=4.13.4 \
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) \
AWS_DEFAULT_REGION=us-east-1

rosa create cluster --sts --private \
 --cluster-name $ROSA_CLUSTER_NAME \
 --multi-az \
 --private-link \
 --region $AWS_DEFAULT_REGION \
 --version $OPENSHIFT_VERSION \
 --enable-autoscaling \
 --min-replicas 3 \
 --max-replicas 3 \
 --compute-machine-type m5.xlarge \
 --machine-cidr 147.206.0.0/16 \
 --subnet-ids subnet-052852a1fb4d7d2ad,subnet-06d8d40ae39d55c47,subnet-0f67ce08bc588012c

The CLI will pick up the correct VPC by CIDR, and prompt you to confirm creation of private cluster. After the command kicks off, it will wait for OIDC provider creation, and role creation. Then it uses a Terraform template to create the related resources including VPC. Use this command to check status:

rosa list clusters
rosa describe cluster -c dhc

When the second command displays the state of waiting (Waiting for OIDC configuration), we can create OIDC provider:

rosa create operator-roles -c $ROSA_CLUSTER_NAME --mode auto --yes
rosa create oidc-provider -c $ROSA_CLUSTER_NAME --mode auto --yes

Throughout the process, we can monitor the install log (terraform output) with:

rosa logs install -c dhc --watch

In the log, you might see errors with terminals connecting to the terraform backend, which doesn’t necessarily indicate a cluster creation error. Always check the cluster state until it reports success.

Kick the tires

Eventually the describe cluster command will show ready state. We can now create an admin user:

rosa create admin -c $ROSA_CLUSTER_NAME

The command above prints an oc command (OpenShift CLI, equivalent to kubectl) with password to log in. Let’s examine the cluster with oc. Because it is a private cluster, the endpoint is not available publicly. However, it is accessible from the Bastion host. Use the SSM Session Manager technique from my previous post to SSH to the Bastion Host, which should have oc installed. To install oc yourself, use HomeBrew on Mac. On Linux or Windows, log on to OpenShift console, go to Downloads on the left pannel and find it out under CLI tools.

The oc command may report insecure TLS on the login URL. Wait for a few minutes for the certificate to come off as safe. Once you run the oc command with password, it should return “Login successful” and then we can connect to the cluster:

$ oc get node # or kubectl get node
NAME STATUS ROLES AGE VERSION
ip-147-206-135-41.ec2.internal Ready,SchedulingDisabled infra,worker 3m5s v1.26.5+7d22122
ip-147-206-155-141.ec2.internal Ready control-plane,master 25m v1.26.5+7d22122
ip-147-206-156-81.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-164-21.ec2.internal Ready infra,worker 3m3s v1.26.5+7d22122
ip-147-206-179-90.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-191-118.ec2.internal Ready,SchedulingDisabled control-plane,master 26m v1.26.5+7d22122
ip-147-206-192-232.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-193-198.ec2.internal Ready infra,worker 3m20s v1.26.5+7d22122
ip-147-206-218-114.ec2.internal Ready control-plane,master 26m v1.26.5+7d22122

You can use oc the same way you’d use kubectl. Both works through SOCK5 proxy. In the meantime, log in to the RedHat console with your Red Hat credential, you can see the cluster in Ready state as well:

The rosa create admin command creates a htpasswd type (username-password) of identity provider (IdP) with a user named cluster-admin and a preset password. In real life however, we often configure third party IdP with OIDC integration. I’ll have to leave this to the next blog post.

We shall see the nodes as EC2 instances from AWS console as well. Note that there are three roles: control-plane, worker and infra. The infra nodes are for infrastructure services. These services (Ingress Controller, GitOps, Pipeliens) are the ones in the platform player as we discussed above.

There are many customizations you can make in this installation process and I’d have to defer to the ROSA documentation. To clean up, use the following ROSA command:

rosa remove cluster -c dhc

The output also gives you the command to delete operator roles and OIDC provider, for example:

rosa delete operator-roles -c 23o4u3j98tqmlbtjo612opb7a4bbim5f --mode auto --yes
rosa delete oidc-provider -c 23o4u3j98tqmlbtjo612opb7a4bbim5f --mode auto --yes

Then we can destroy the VPCs using terraform.

ROSA with HCP

Update Oct 2023:

The deployment above provisioned a few nodes for control plane, which add to the overall time to provision a cluster. In Aug 2023, there is a new option Hosted Control Plane (HCP) that came to allow users to provision a hosted control plane. This results in cost savings and shorter time to provision a cluster. Here is a table of comparison between the ROSA with HCP and ROSA classic.

Final words

In this post, I discussed the three-layer model and pointed out that platform layer isn’t as visible as the other two. I also experimented ROSA as a turn-key Kubernetes platform with its opinionated stack of services.

Some misinformed organizations even skip the entire platform layer in their estimate of effort. They build a cluster, ran a hello-world service and assumes they can start putting applications on the Kubernetes cluster. There are also customers who purchased the entire Managed OpenShift platform but only use it as a cluster. Yikes!

The concept of Kubernetes platform, or generally platform engineering is still spreading. The consulting team that I worked in full-time last year re-branded itself as platform engineering. Marketings are pushing it. Builders are doing it. We’ll keep an eye, on whether customers are buying it.