Infrastructure as Code (IaC) on Digi Hunch

Kubernetes Platform as a Service and Red Hat OpenShift

Sun, 25 Jun 2023 11:10:15 -0400

The Three-layer model

Kubernetes is so complex that it becomes a buzz word itself. I categorize the related work into three layers: a cluster layer, a platform layer and an application layer, by their purposes. The three layers are illustrated as below:

Let’s examine each layer in this model and where the Kubernetes Platform as a Service fits in.

The Kubernetes Cluster Layer

At the bottom, the Kubernetes Cluster layer is the foundational layer. It focus on using self-hosted VMs or cloud resources to build a functional Kubernetes cluster and worker node groups. A functional cluster includes a highly available control plane, as well as scalable node groups that all communicate with the control plane. Cloud Service Providers like AWS and Azure provides managed Kubernetes service, which takes away the complexity (and flexibility as well) of managing control plane components such as etcd store and API server. The managed services also automatically provisions computing nodes and join them into the cluster. The cluster layer may also involve integration with of CNI and CSI, to ensure Pod-to-Pod communication and available storage classes.

Professionals working at this layer are infrastructure experts who understand networking, storage, as well as how to manage cloud resources or VMs, infrastructure as code. On a daily basis, they deal with VPCs/V-Nets, subnets, EBS/Azure Disk, File storage, EC2/Azure VMs, etc. When the team is doing a bad job at this layer, you might see symptoms like unresponsive cluster API, orphaned worker nodes, or kubectl failing to connect to cluster endpoint.

The tenants (applications) of the Kubernetes platform does not directly interact with this layer. If you decide to switch CSP vendor, this layer requires 100% re-engineering because the managed Kubernetes service by each CSP is different.

The Kubernetes Platform Layer

The Platform layer sits in the middle. When organization decides to adopt Kubernetes, they often underestimate the efforts required in this layer. This layer works on a functional cluster, without directly interacting with the underlying cloud resources. This layer involves any Kubernetes abstractions that do not creates tangible business value. Rather, this layer is an enabler. It allows the applications to deploy smoothly, evolve quickly, and more importantly, focus on the business.

Teams working on this layer needs to be Kubernetes experts. On a daily basis, they play with common CNCF toolings, such as Prometheus, ArgoCD, Istio, Cilium, Tekton, Open Policy Agent, etc. They are comfortable with Operators, Helm Charts, Ingress, etc. Inside of the Kubernetes cluster, they also manage the foundational services such as Event streaming (e.g. Kafka), PostgreSQL database (e.g. PostgreSQL), software-defined storage (e.g. Ceph), service mesh (e.g. Istio), Authentication (e.g. Keykloak) , etc. These services act as the infrastructure layer to the business workload. If the team is doing a bad job, you would see data loss with database, observability service not populating data, ingress does not process request, etc.

The tenants (application) share services in this layer. If you decide to switch CSP vendor. I estimate 80% of the work at this layer is portable, and 20% requires re-engineering. That is because each CSP offers different external resources, therefor the low level Kubernetes objects in this layer, such as storage classes, load balancers, supported CNIs are different. High level objects such as Kafka remains portable across platforms.

The Application Layer

The next layer at the top is application layer. Workloads in this layer are directly linked to the business value. The applications are very diverse. Most of the time, the release team is the main player at this layer. If the organization develops its own application, the software development team also work at this layer.

In terms of knowledge, the members of development team are experts in software engineering, and Software Development Life Cycle (SDLC), etc. On a daily basis, they deal with programming languages, product development, build and release. If they screw up their work, expect business errors, such as orders sent to wrong client, incorrect balance sheet, etc. This team has high visibility in the organization due to its direct link to business value.

This layer of work involves multiple tenants. Each tenant is isolated within their own namespace. When you switch CSP vendor, this layer should be readily portable with minimal effort.

It is also worth noting that, with solid platform and cluster layers, the team working at this layer do not write bespoke code for networking, observability, authentication and authorization, encryption and many other aspects not relevant to the core business. Once deployed, the application services are resilient, scale to demands, and cost efficient. This layer reaps the benefits of Kubernetes.

Kubernetes Platform as a Service

As the Kubernetes dust is still settling, a builder’s title may not always reflect which layer she or he focuses on. Today it is pretty common for infrastructure engineers to expand their role into the platform layer, or likewise, a software engineer to drill down to the platform layer.

The boundary between platform layer and cluster layer is clear. The cluster layer deals with underlying infrastructure, either in the cloud or on premise. They abstract away the complex infrastructure world from those working with the platform layer.

The boundary between platform layer and application layer is a little tricky to articulate. The application layer focuses on implementing the business logics. The platform layer takes care of the functions that are not part of business logic but essential to the business application. Take an HTTP request for example, application developer should not have to write code to terminate TLS (not part of business logic). They should only write the code to process the HTTP request (business logic). TLS termination is delegated to an Ingress, to be configured by platform builders.

The folks working at the Platform layer needs to interface with both sides. They provide Platform as a Service to the Application teams. However, their work appears mostly invisible in an organization. Their effort is oftentimes underestimated. There are several reasons for that. First, the platform layer does not directly create tangible business value. They are just someone else’s enabler. Second, their building blocks involve a lot of abstractions by Kubernetes API. Third, the idea of platform engineering is newly emerged. There hasn’t been a populous recognition of its value.

Red Hat OpenShift Container Platform

The platform team builds the platform with their choice of open-source tools. For clusters using OpenShift Kubernetes Engine, Red Hat introduces Open Shift container platform consisting of Red Hat’s opinionated (but validated) choice of toolings, for example:

OpenShift Service Mesh: Istio
OpenShift Streams: Apache Kafka
OpenShift GitOps: ArgoCD
OpenShift Container Platform Pipelines: Tekton
OpenShift Serverless: Knative
OpenShift Data Foundation: Ceph

Clients building their clusters with OpenShift Kubernetes Engine may build their own platform with the toolings in the OpenShift enterprise Kubernetes container platform. For more services, check out the documentation for OpenShift Container Platform. For customers with OpenShift Kubernetes Engine, their options to DIY platform are:

Entry-Level: Red Hat OpenShift Kubernetes Engine: Enterprise Kubernetes distribution on RHEL CoreOS
Mid-Level: Red Hat OpenShift Container Platform (RHOCP):
Plus-Level: Red Hat OpenShift Platform Plus: RHOCP + advanced cluster management, security, data management essentials, enterprise container registry

OpenShift runs the business model of Kubernetes PaaS.This is a unique business model that I do not find a matching competitor. Even if you choose to DIY your own platform, the Red Hat’s choices are still a great reference. The OpenShift enterprise Kubernetes container platform maps perfectly to the platform layer of the three-layer model, aiming to simplify the work in the platform layer.

Managed RedHat OpenShift

At first, the OpenShift container platform started as a value add-on to the Kubernetes Engine. Now it’s a separate product line in their business model. In the mean time, OpenShift partners with major CSPs, to develop the cloud service editions, including:

Red Hat OpenShift on AWS (ROSA)
Microsoft Azure Red Hat OpenShift (ARO)
Red Hat OpenShift Dedicated – on AWS and GCP
Red Hat OpenShift on IBM Cloud

These offerings are managed Kubernetes Platform as a Service in the cloud. Since RedHat is the only player in this model, we can refer to them as managed OpenShift services. In addition to an already-confusing world of Kubernetes platform portfolios, these offerings gives consumers even more options. On AWS for example, users have the following options:

Managed Platform: OpenShift Dedicated, managed by Red Hat
Managed Platform: Red Hat OpenShift Service on AWS (ROSA), managed by Red Hat and AWS
Self-built cluster: OpenShift Container Platform

This video discussed more details about these options, such as support model. It is also worth noting that these options tend to be much pricier than managed clusters such as EKS and AKS.

Since a Managed RedHat Platform makes it easy to deploy, let’s take ROSA as an example and create a cluster. To enable ROSA in AWS console, click on “Getting Started”. The next page ensures ROSA is enabled and checks other prerequisite such as meeting service quotas and creating ELB service-linked role, as show below:

Now, with an AWS account (and ROSA enabled), a RedHat account, and the rosa-cli utility, we can create a cluster with just a few commands. As a note, be wary of the cost and do not forget to delete the cluster afterwards.

Create a ROSA cluster

With the following set of commands, we can kick off cluster creation, using STS. We can bring our own VPC, so long as it meets certain prerequisites. I use the Terraform template in the vpc-base project, to create the underlying VPC. We’ll need the followings from this template:

The CIDR range of the VPC: as input with a default
The subnet Ids of the private subnet to place, printed in the output

The subnets are private subnets, because we want to provision the cluster with private node and private endpoint. When we use rosa CLI, we provide the CIDR and subnet IDs.

# start with AWS cli configured to the correct profile
rosa login # with redhat account and past token
rosa create account-roles --mode auto -y # this command creates the IAM roles ManagedOpenShift-*-Role, with RedHat account as trust entity
rosa verify permissions # optional
rosa verify quota # optional

export ROSA_CLUSTER_NAME="dhc" \
OPENSHIFT_VERSION=4.13.4 \
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) \
AWS_DEFAULT_REGION=us-east-1

rosa create cluster --sts --private \
 --cluster-name $ROSA_CLUSTER_NAME \
 --multi-az \
 --private-link \
 --region $AWS_DEFAULT_REGION \
 --version $OPENSHIFT_VERSION \
 --enable-autoscaling \
 --min-replicas 3 \
 --max-replicas 3 \
 --compute-machine-type m5.xlarge \
 --machine-cidr 147.206.0.0/16 \
 --subnet-ids subnet-052852a1fb4d7d2ad,subnet-06d8d40ae39d55c47,subnet-0f67ce08bc588012c

The CLI will pick up the correct VPC by CIDR, and prompt you to confirm creation of private cluster. After the command kicks off, it will wait for OIDC provider creation, and role creation. Then it uses a Terraform template to create the related resources including VPC. Use this command to check status:

rosa list clusters
rosa describe cluster -c dhc

When the second command displays the state of waiting (Waiting for OIDC configuration), we can create OIDC provider:

rosa create operator-roles -c $ROSA_CLUSTER_NAME --mode auto --yes
rosa create oidc-provider -c $ROSA_CLUSTER_NAME --mode auto --yes

Throughout the process, we can monitor the install log (terraform output) with:

rosa logs install -c dhc --watch

In the log, you might see errors with terminals connecting to the terraform backend, which doesn’t necessarily indicate a cluster creation error. Always check the cluster state until it reports success.

Kick the tires

Eventually the describe cluster command will show ready state. We can now create an admin user:

rosa create admin -c $ROSA_CLUSTER_NAME

The command above prints an oc command (OpenShift CLI, equivalent to kubectl) with password to log in. Let’s examine the cluster with oc. Because it is a private cluster, the endpoint is not available publicly. However, it is accessible from the Bastion host. Use the SSM Session Manager technique from my previous post to SSH to the Bastion Host, which should have oc installed. To install oc yourself, use HomeBrew on Mac. On Linux or Windows, log on to OpenShift console, go to Downloads on the left pannel and find it out under CLI tools.

The oc command may report insecure TLS on the login URL. Wait for a few minutes for the certificate to come off as safe. Once you run the oc command with password, it should return “Login successful” and then we can connect to the cluster:

$ oc get node # or kubectl get node
NAME STATUS ROLES AGE VERSION
ip-147-206-135-41.ec2.internal Ready,SchedulingDisabled infra,worker 3m5s v1.26.5+7d22122
ip-147-206-155-141.ec2.internal Ready control-plane,master 25m v1.26.5+7d22122
ip-147-206-156-81.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-164-21.ec2.internal Ready infra,worker 3m3s v1.26.5+7d22122
ip-147-206-179-90.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-191-118.ec2.internal Ready,SchedulingDisabled control-plane,master 26m v1.26.5+7d22122
ip-147-206-192-232.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-193-198.ec2.internal Ready infra,worker 3m20s v1.26.5+7d22122
ip-147-206-218-114.ec2.internal Ready control-plane,master 26m v1.26.5+7d22122

You can use oc the same way you’d use kubectl. Both works through SOCK5 proxy. In the meantime, log in to the RedHat console with your Red Hat credential, you can see the cluster in Ready state as well:

The rosa create admin command creates a htpasswd type (username-password) of identity provider (IdP) with a user named cluster-admin and a preset password. In real life however, we often configure third party IdP with OIDC integration. I’ll have to leave this to the next blog post.

We shall see the nodes as EC2 instances from AWS console as well. Note that there are three roles: control-plane, worker and infra. The infra nodes are for infrastructure services. These services (Ingress Controller, GitOps, Pipeliens) are the ones in the platform player as we discussed above.

There are many customizations you can make in this installation process and I’d have to defer to the ROSA documentation. To clean up, use the following ROSA command:

rosa remove cluster -c dhc

The output also gives you the command to delete operator roles and OIDC provider, for example:

rosa delete operator-roles -c 23o4u3j98tqmlbtjo612opb7a4bbim5f --mode auto --yes
rosa delete oidc-provider -c 23o4u3j98tqmlbtjo612opb7a4bbim5f --mode auto --yes

Then we can destroy the VPCs using terraform.

ROSA with HCP

Update Oct 2023:

The deployment above provisioned a few nodes for control plane, which add to the overall time to provision a cluster. In Aug 2023, there is a new option Hosted Control Plane (HCP) that came to allow users to provision a hosted control plane. This results in cost savings and shorter time to provision a cluster. Here is a table of comparison between the ROSA with HCP and ROSA classic.

Final words

In this post, I discussed the three-layer model and pointed out that platform layer isn’t as visible as the other two. I also experimented ROSA as a turn-key Kubernetes platform with its opinionated stack of services.

Some misinformed organizations even skip the entire platform layer in their estimate of effort. They build a cluster, ran a hello-world service and assumes they can start putting applications on the Kubernetes cluster. There are also customers who purchased the entire Managed OpenShift platform but only use it as a cluster. Yikes!

The concept of Kubernetes platform, or generally platform engineering is still spreading. The consulting team that I worked in full-time last year re-branded itself as platform engineering. Marketings are pushing it. Builders are doing it. We’ll keep an eye, on whether customers are buying it.

Landing Zone in AWS – An Introduction

Thu, 08 Dec 2022 22:37:00 -0400

Cloud adoption has gone through phases. Hashicorp’s CTO Armon Dadgar has a great stream on Hashcorp’s narrative of the three Phases of Cloud Adoption:

Phase 1, with main focus on agility, app teams in wild west, account sprawl, inconsistent configuration, security & compliance challenge
Phase 2. consistent platform layer providing an opinionated way of configurations, security and compliance control, introduction of platform team, whose customers are application team, scaling challenges with platform team capacity not keeping up with application team’s demand
Phase 3. self-service platform at scale for many application teams. CICD, infra-as-code

New greenfield clients today should aim at phase 2 or phase 3 depending on their target operation size. Either way, they need a landing zone for consistency. This post focuses on the landing zone options on AWS and the key constructs.

Overview

AWS Prescriptive Guidance defines landing zone as a well-architected (secure, scalable, compliant, etc), multi-account AWS environment that is a secure baseline from which you can deploy workloads and applications. A landing zone may consists of:

Multi-account architecture
Identity and Access management solution
Governance, compliance, logging and auditing solutions
Security and networking design

Landing zone reflects an enterprise opinion on how to configure networking and IAM. Establishing a landing zone can be a very involving process. In early days of cloud operation people used build landing zone by “clickops” on AWS console, which cannot keep up with the growth of landing zone and associated security services. Compared to SMB clients, some sectors such healthcare and national security have much more regulations and compliance frameworks.

In fact, landing zone is such a buzzword that I have learned to be very sensitive to the context. If a solution has words “landing zone” in its name, given the complexity and loose use of the words, there is a good chance that the solution only delivers some of the aspects above. Cloud consultants are still to address the gaps. Therefore I decided to write this post about what I learned about landing zone in AWS.

Multi-account and Organization

Since 2017, AWS has been officially advocating the use of multiple accounts as a best practice and security strategy. They encourage clients to view an account as a resource container, just like Resource Group in Azure (here is an article on the account structure between AWS and Azure) Client builds an account for security boundary and financial container. Many blog post came along (such as this one) on the implementation details. From tooling perspective, AWS launched AWS Organization in 2017 to facilitate multi-account management. This presentation from re:Inforce 2019 is a good material to understand multi-account environment with AWS Organizations. In 2021, AWS published a new white paper on best practices with multiple accounts. My previous coworkers authored two blog posts here and here to reflect multi-account setup as of late 2020.

AWS Organization is an account management service that provides a vehicle to centrally manage AWS accounts by groups, which brings many benefits, such as centralized logging, compliance management, consolidated billing, etc. As an administrator, you can create accounts in your organization and invite existing accounts to join the organization.

Typical account structure involves:

A root organization named Root
A organization hierarchy with one or more OUs under root, with each OU having one or more child OUs.
Each OU can have multiple accounts, with each account having one email address.

You can also configure service control policies (SCPs), a type of organization-level policy that you can use to manage permissions in your organization. SCPs offer central control over the maximum available permissions for all accounts in your organization. SCPs are a means of implementing guardrails in your AWS organization.

To vend multiple account automatically, AWS introduced the Account Factory (which became part of Control Tower later in 2019) and let users create new account from AWS console and specify which OU it belongs to. It also allows users to implement customization after account creation with Service Catalog products. Some legacy orchestration solution (e.g. Augmented Account Factory) were based on this mechanism. Another orchestration solution prior to the launch of Control Tower was the AWS Landing Zone solution (ALZ, introduced in 2018), which uses AWS CodePipeline to provision accounts and deploy resources.

From Landing Zone Solution (ALZ) to Control Tower

The launch of Control Tower was a game changer in 2019 to bring users to multi-account best practices. Control Tower, as an AWS product (instead of a solution by some service teams at AWS) matured over several years. It gradually deprecated the previous generation orchestration solutions. Newer orchestration solutions all have to support Control Tower. According to this page, we can customize Control Tower based Landing Zone in these ways:

AWS Control Tower console:
- Instead of creating OU in AWS organization, do it under Control Tower in the console, under “create required OUs”.
Outside of AWS Control Tower console
- using Account Factory for Terraform (AFT): Terraform-based account provisioning pipeline, for heavy Terraform shops.
- using Customizations for AWS Control Tower (CfCT) solution. The LZ created remains in sync with Control Tower.
  - You first launch a standardized CF stack to set up the mechanisms for customization.
  - Then you create a custom package to define the customization. This includes a manifest file.

On the other hand, the original ALZ solution is currently in long-term support and will not receive any additional features. It is deprecated. AWS advises its customers migrate to AWS Control Tower based landing zone. The ALZ page is redirected to a page about customizing Control Tower landing zone.

The Control Tower based landing zone configures OUs, accounts, SSO and guardrails. It however does not entail networking design except for the guardrails.

Security Reference Architecture (SRA)

SRA is just a reference architecture and accompanying recommendations on AWS security services, and how they work together in a multi-account environment to host a single-page application. It comes with a repository to demonstrate how to configure a secure multi-account environment with Control Tower, CfCT, as well as security services. Alternatively, you can deploy it with CloudFormation StackSets. For fully automated deployment of this architecture, check out Landing Zone Accelerator down below.

One important feature that reflects the notion of centralized security in a multi-VPC topology, is the use of a dedicated VPC for centralized interface endpoints. This pattern is seen in all generations of security reference architectures.

AWS Security Environment Accelerator (ASEA)

Regulated customers often find they need to add additional controls and capabilities to be defined and setup outside of Control Tower. ASEA, as an orchestration solution, aims to remove the complexity of having to develop and maintain a separate codebase to manage the additional customizations, by providing a tool to help deploy and operate secure multi-account, multi-region AWS environments on an ongoing basis.

Overtime, as Control Tower introduces new capabilities to support the customizations required in heavily regulated environments, the capabilities will be removed from ASEA and enabled directly within the Control Tower managed service, further reducing operational burden. Read this for its relationship with ALZ and Control Tower.

ASEA was first released in late 2020. It covers more on networking design and has a fairly comprehensive installation process. ASEA primarily cater to government of Canada‘s PBMM Security Configuration Profile with an opinionated configuration. The first few revisions were referred to as PMBB architecture. While it provides a great reference architecture for highly regulated landing zone, it has not gotten much traction elsewhere. The recommendation going forward, is to use Landing Zone Accelerator, which incorporates the features and lessons learned from ASEA.

Landing Zone Accelerator (LZA)

First released in May 2022, the Landing Zone Accelerator on AWS solution deploys a cloud foundation that is architected to align with AWS best practices and multiple global compliance frameworks. LZA operates on top of Control Tower managed landing zone. This page in the documentation brings a good explanation of how it works. Basically it employs CodeBuild as an orchestration engine, and leverages CDK to drive resource deployment. There are two repositories: the GitHub repository for Landing Zone Accelerator itself, and a CodeCommit configuration repository provisioned during preparation.

As document states, LZA is a fully automated implementation of the architecture guidelines documented in the SRA. LZA also incorporates features and lessons learned from ASEA and Compliant Framework for Federal and DoD Workloads in GovCloud (US), neither of which are recommended for new deployment. In addition, LZA aims to enable iterations and extensions of a secure environment over time. The vision is to eventually replace AFT, CfCT, and ASEA. As AWS releases newer versions of LZA, client should be able to upgrade it in a pipeline run.

Baseline architecture for LZA

The Landing Zone Accelerator project also provides a samples configurations in each regulated frameworks. For example, the healthcare best practice come up in Oct 2022 for healthcare industry. The healthcare best practice sample incorporates healthcare specific configurations, such as the detective guardrails defined in the Operational Best Practices for HIPAA Security conformance pack. To deploy the best practices, modify the configuration in config repo, and run the pipeline again.

Landing Zone Orchestration Options

To summarize, we have the following landing zone options in AWS:

Orchestration Layer	Lifecycle	Summary
Solutions based on account factory with Service Catalog (e.g. ALZ, Augmented Account Factory)	All solutions pre-dates the launch of control tower have been or will soon be deprecated.	Pre-Control Tower solutions should migrate to current alternatives.
AWS Control Tower	GA in June 2019	To customize the landing zone, use AWS console, or alternatively, one of the following three options: – CfCT: Customization for Control Tower – AFT: Account Factory Terraform – AFC: Account Factory Customizations For example, CfCT can be used to deploy Security Reference Architecture (SRA)
AWS Secure Environment Accelerator (ASEA)	Released 2020	Canadian Centre for Cyber Security (CCCS) Medium Cloud Control Profile, formerly known as PBMM. This approach will be replaced by LZA
AWS Landing Zone Accelerator (LZA)	Released in 2022	A low-code deployment option. Samples provided in support of reference architectures that align with industry best practices or compliance frameworks. Examples for industry best practices: – general best practice – healthcare – finance and tax – education Example for compliance frameworks: – US state local government – FedRAMP for US Federal and Department of Defence (DoD) – CCCS Medium for Canadian government – AWS Trusted Secure Enclave (TSE) Sensitive Edition (SE), which also aligns with other medium level security profiles such as NIST 800-53, ITSG-33, FedRAMP moderate, CCCS-Medium, IRAP, etc

If your organization has a compliance framework that LZA supports, it makes sense to start with LZA. Otherwise, it is sufficient to use Control Tower for multi-account setup.

Networking constructs

Landing zone involves multiple VPCs so it’s important to understand VPC peering and Transit Gateway. VPC peering can only be setup between two VPCs and transitive peering relationship is not supported. For 6 VPCs to talk to all each other, we’d need 15 peering setups, which is not sustainable. We have two options:

We can connect many VPCs to a single Transit Gateway and the VPCs will be able to talk to each other. We can also connect Transit Gateway to site-to-site VPN or Direct Connect. This is a good explanation.
We can still use peering, but dedicate one VPC as “Transit VPC” in a hub and spoke model.

When you attach a VPC to a transit gateway, you must specify one subnet from each AZ to be used by the transit gateway to route traffic. Specifying one subnet from an AZ enables traffic to reach resources in every subnet in that AZ.

Here is a comparison table. Read the white-paper “Building a Scalable and Secure Multi-VPC AWS Network Infrastructure” for more about network design. For example, Some clients need to inspect traffic. When deploying multiple VPCs, we also need to ensure DNS resolution works across VPCs, and between on-prem networks, by configuring Amazon DNS server.

Our VPCs also need to connect to managed AWS services. For services like S3 or DynamoDB (of the same or different account), workload in VPC can access them via public DNS. The traffic goes through Internet Gateway of VPC and then public Internet. It is neither secure nor economical. We often want all network traffic to stay on the global AWS backbone. There are three types of Endpoint under VPC to help us.

	Gateway VPC Endpoint or Gateway Endpoint	Interface VPC Endpoint or Interface Endpoint	Gateway Load Balancer Endpoint
Purpose	VPC access native AWS services	VPC access native AWS services	VPC access your own service (aka Endpoint Service)
Enabled by Private Link	No	Yes	Yes
Traffic remain on AWS network	Yes	Yes	Yes
Mechanism	Use the public IP address of the service along with configuration in routing table to access target resource. The routing table acts as gateway. You can use the public DNS name of the service.	Use private IP address from the VPC to access the target service. Require endpoint-specific DNS name for the target service. Incur extra charge. S3 as example. More secure because there is ENI in the VPC controlled by security group.	Your Endpoint Services is hosted in front of a fleet of network virtual appliances. You can select endpoint type as you create it in console under VPC
Access from clients on premise or from other region	Does NOT allow access from clients on premise or in VPC from other region	Allows access from clients on premise or clients in VPC from other region	Allowed

Private Link integrates with a subset of AWS services. To check the list of interface endpoint, use:

aws ec2 describe-vpc-endpoint-services --query "ServiceDetails[?ServiceType[0].ServiceType=='Interface'].ServiceName"

Given the inter-VPC connectivity, most deployment centralizes interface endpoint into a dedicated VPC.

Centralized Interface Endpoints

There are several benefits to use a single VPC as dedicated provider of interface endpoint in a multi-VPC topology. First, the interface endpoints incurs a standing charge and it makes financial sense to consolidate them in one VPC. Second, this setup centralizes the configuration and security aspects as well.

In most cases, interface endpoint services follow the format of com.amazonaws.<region>.<endpoint>, with the dns name looking like: <endpoint>.<region>.amazonaws.com. For example:

Endpoint Service Name: com.amazonaws.us-east-1.ssm
Private Hosted Zone: ssm.us-east-1.amazonaws.com
Zone Record (A alias): ssm.us-east-1.amazonaws.com

However, there are a few exceptions to that which can make it tricky to implement interface endpoint with infrastructure as code.

Exception 1. Private DNS name suffix is api.aws instead of amazonaws.com:

Endpoint Service Name: com.amazonaws.us-east-1.eks-auth
Private Hosted Zone: eks-auth.us-east-1.api.aws
Zone Record (A alias): eks-auth.us-east-1.api.aws

Exception 2. Endpoint Service Name doesn’t start with com.amazonaws

Endpoint Service Name: aws.sagemaker.us-east-1.notebook
Private Hosted Zone: notebook.us-east-1.sagemaker.aws
Zone Record (A alias): notebook.us-east-1.sagemaker.aws

Exception 3. Two A-records are required under the same PHZ

Endpoint Service Name: com.amazonaws.us-east-1.ecr.dkr
Private Hosted Zone: dkr.ecr.us-east-1.amazonaws.com
Zone Record (A alias): dkr.ecr.us-east-1.amazonaws.com and *.dkr.ecr.us-east-1.amazonaws.com

Exception 4. Two PHZs are required:

Endpoint Service Name: com.amazonaws.us-east-1.ec2
Private Hosted Zone: ec2.us-east-1.amazonaws.com and ec2.us-east-1.api.aws
Zone Record (A alias): ec2.us-east-1.amazonaws.com and ec2.us-east-1.api.aws

The infrastructure code that handles interface endpoints should be able to take care of the exceptions.

Security Constructs

AWS Network firewall is a configuration under VPC. You associate a Network Firewall with one or more subnets in the VPC. You also associate the Network Firewall with Firewall Policies. Each policy consists of rule groups. Each rule blocks or filters traffic. The log can be published to CloudWatch or S3 via Kinesis. Usecases of AWS Network firewall include:

inspect VPC-to-VPC traffic;
filter outbound traffic;
prevent inbound internet traffic;
secure AWS Direct Connect and VPN traffic

When you associate a firewall to your VPC, you must provide a subnet for each Availability Zone where you want to place a firewall endpoint to filter traffic

There are many points of configuration for firewall rules in AWS. AWS Firewall Manager is a place for central management. It connects with other services such as:

AWS WAF (L7)
AWS Network Firewall (L4)
AWS Shield (DDos)
Amazon Route 53 Resolver DNS Firewall
Security Groups
Third party firewall support

IAM Identity Center (previously AWS SSO) is for logging into AWS portal, giving each identity from an external identity store (such as AD) an identity on an AWS account. It addresses several challenges as a result of having multiple AWS accounts.

First, a user from AD needs to access multiple accounts. Second, in each account with access, s/he needs to have an IAM role. These IAM roles are based on attributes of his/her external identity, and can be different per AWS account. With Permission Set IAM Identity Center:

Allow you to login on different AWS account using the same credential
Allow you to federate with external identity store( e.g. using SAML)
Manage permission set for each account.

This video has a good walk-through of how to configure Azure AD as identity store, and tie it to specific AWS accounts and specify permission set.

Note that IAM Identity Center has a different purpose than Cognito. Cognito is to connect your app with an IdP. Your app can be an EC2-hosted application, serverless application on API gateway, or container-based application on Kubernetes, as long as the support open identity standards (e.g. OAuth 2.0, SAML 2.0 and OIDC). Cognito has two pools:

User pool for authentication. With a user pool, your app
Identity pool for authorization

So Cognito is for AuthN & AuthZ to your own app’s endpoint (business traffic) and it supports a number of standards. IAM Identity Center is for AuthN & AuthZ to your AWS account (management traffic). If you use it for your own application, your application user will have direct access your AWS resources. It mainly supports SAML (and OIDC but less used) and is commonly connected with Active Directory (Microsoft or AWS).

One can use AWS Directory Service to host a compatible and managed directory service. AWS Directory Service supports four modes:

AWS managed Microsoft AD: actual Microsoft Active Directory in AWS Cloud
Simple AD, powered by Linux-Samba Active Directory-compatible server
AD Connector: a proxy for redirecting directory request to your existing Microsoft AD without caching any information in the cloud
Cognito user pools

If you combine two AD domains, you will need domain trust.

Other security services

In addition to the core services above, LZA involves the following four services as well.

AWS Macie is a fully managed data security and privacy service based on ML and pattern matching. It continually evaluates your S3 environments to discovery PII and act on them. It also reports alerts on unencrypted buckets, publicly accessible buckets, etc.

AWS GuardDuty continuously analyze S3, container & instance workloads, user and accounts for potential threads, across account, based on ML for intelligent threat detection. In addition, GuardDuty also acts on findings.

AWS Config is an essential service that traces resource inventory, their changes and monitors for compliance:

When you turn on AWS Config, it first discovers the supported AWS resources in your account and generates a configuration item (poin-in-time view of attributes) for each resource. AWS Config also generates configuration items when the configuration of a resource changes, and it maintains historical records of the configuration items of your resources from the time you start the configuration recorder.
On an ongoing basis, AWS Config keeps track of all changes to your resources, whether or not it is initiated by the API
If you are using AWS Config rules, AWS Config continuously evaluates your AWS resource configurations for desired settings. You can deploy several related rules in a pre-built compliance pack.

As you may have noticed, there are many AWS services related to firewall, security and compliance. AWS Security Hub aims to be a a consolidated view of your security status in AWS. You can automate security checks, manage security findings, and identify the highest priority security issues across accounts in client environment. It can also:

consolidate security findings from GuardDuty, Inspector (vulnerability scanner and management), Macie, Config, Systems Manager, Firewall Manager, IAM Access Analyzer and other Integrated APN solutions
Check findings against best practices
Client can take action: i.e. investigate findings or take response and remediation actions

Summary

Landing zone deployment can take numerous iterations to finalize the requirement. It is important to have a vision of the client’s cloud operating model, which is usually discussed prior to implementation. At the implementation phase, I discuss the topic on two more separate posts on control tower and landing zone accelerator.

AWS serverless services and developer tools

Wed, 09 Nov 2022 12:19:00 -0400

As discussed, serverless simply means cloud services that delegate autoscaling management to cloud platform. In my mind, the word “serverless” translates into “managed autoscaling”. As long as a service’s capacity is managed automatically, we can consider it as serverless. Given that capacity scaling accounts for a good amount of work in IT operation, moving to serverless significantly reduces operation overhead.

Unlike containers, the serverless ecosystem lacks standard. Since AWS is leading the charge in this field, let’s take a look at its offerings.

Lambda Function

Lambda functions have triggers. We can configure a trigger from an AWS service, or even a non-AWS service which supports AWS event bridge, to invoke Lambda function. Based on the trigger (e.g. SQS, S3, DynamoDB), you then specify event source mapping. For example, if trigger is SQS, you need to specify the queue name, batch size and batch window in event source mapping. If trigger is S3, the event source mapping needs to specify bucket, S3 action, etc. Event source mappings vary significantly among trigger types.

Depending on the trigger, a Lambda function may be invoked in one of the two ways:

	Synchronous invocation	Asynchronous invocation
Summary	Requestor fires request and waits until it receives response before closing connection.	Lambda places triggering event in a queue and immediately returns a success code (202). Then a separate process reads events off the queue and sends them to your Lambda function. When the function returns a success response or exits without throwing an error, Lambda sends a record of the invocation to an EventBridge event bus.
Pros	Requestor get invocation result as soon as function run is complete	The queue decouples the request and invocation.
Cons	When load is high the function requires higher capacity for concurrency	Involves more parties at play and can be quite complex to troubleshoot.

Here is a chart with the triggers that support each invocation mode.

Real-life applications often need to include libraries and dependencies that requires language-specific steps. For example, NodeJS applications need bundling (often using Webpack or esbuild). We can use Lambda layers to simplify the management. The additional libraries often need to be packaged in S3 bucket or as container image.

Lambda functions are subject to cold start to cope with. When a lot of invocations occur about the same time, the order of executions might be different than the order of upstream events that invokes the function.

API Gateway

AWS API Gateway is an API gateway implementation for REST and WebSocket APIs. It couples with Lambda in the classic multi-tier serverless architecture pattern.

In this classic pattern Lambda function often need to work with a database (e.g. DynamoDB etc), to perform CRUD operations and other custom business logics. The CRUD operations are so commonplace that it makes sense to use mapping template to configure CRUD operation instead of writing similar set of functions for each new data model. API Gateway supports such a mapping template called Velocity Template Language (VTL), a technology from Apache Velocity Project.

API Gateway integrate with many other AWS services. Here are the available integration types:

Integration Type	Description	Integration Mode	How it works
AWS integration	connects gateway to an AWS service action as the end point.	AWS_PROXY	this mode only supports only one action with one service: the function invoking action for Lambda service. Therefore it is available only for Lambda integration and no other AWS services. For that reason, it is also known as Lambda proxy integration. This mode is recommended for Lambda integration and is the default mode for LambdaIntegrationOption CDK construct. It connects a method (PUT, GET, etc) to a Lambda function and pass along the request on the way in, and the response on the way out. You do not set integration request or integration response. Even if you do, there’s no effect. This “pass-along” mode is easier to understand and configure.
AWS integration	connects gateway to an AWS service action as the end point.	AWS	this mode connects an API method to a broad range of supported AWS service action. Function invoking for Lambda service is a common example but not the only service action supported in this mode. When used in Lambda integration, it is also referred to as “Lambda custom integration”, or “normal (request/response mapping) integration”. This mode is good for advanced use cases (e.g. header modification) but involves more management effort. You have to control the mapping between method request and integration request, and between integration response and method response.
HTTP integration	for generic HTTP service endpoint	HTTP_PROXY	the pass-along mode for upstream HTTP endpoint
HTTP integration	for generic HTTP service endpoint	HTTP	the request/response mapping mode for upstream HTTP endpoint
MOCK integration	the API gateway itself serves as the endpoint.	MOCK	the API gateway itself acts endpoint without an upstream. One example use case is to return CORS-related headers upon a pre-flight OPTIONS query.

The table above is a summary of API Gateway integration types as covered here. In the AWS context, proxy mode suggests that the request is not being morphed (transformed). In non-proxy mode, request or response may be modified on their ways in or out.

For API gateway to invoke lambda function, synchronous invocation is used by default. You can also configure API gateway to invoke Lambda function asynchronously by using headers.

As with other API gateway implementations, AWS API Gateway can also connects to an authorizer (either another Lambda function or Cognito service) in order to authorize the incoming request.

AppSync

In a previous post, I discussed GraphQL as a modern and efficient alternative to REST API. AppSync to GraphQL is the same as API Gateway to REST API. One of the advantages that GraphQL has over REST API is more information in the response. Oftentimes, we use a proxy that supports GraphQL in front of REST API service. AppSync can act as such proxy. I think of it as a managed Apollo since they play the the role in the architecture.

Another benefit of GraphQL is the support of subscription, obviating WebSocket configuration. AppSync supports pub/sub API for real-time experience. Client application can get near real-time update as the data on the server is changed.

We can configure AppSync to connect to different data sources to formulate GraphQL response. The data source can be an HTTP endpoint, a Lambda function, a database (Relational or DynamoDB), or OpenSearch.

App Sync

Between data source and the request, AppSync uses resolver to convert GraphQL payload to the underlying protocols and executes if the caller is authorized to invoke it. Resolvers are comprised of request and response mapping templates, which contain transformation and execution logic. AppSync also uses VTL as the mapping template for resolvers.

Messaging Services

Messaging services have three patterns: queues, pub/sub, and event buses. In AWS, the corresponding services are SQS, SNS and EventBridge. Here is a good post about how to choose among them.

Queues are temporary storage to decouple the source and destination systems. The expectation is that the actions in response to the message can be delayed. If that is not the case and the response needs to be immediate, that is by definition an event-driven pattern. A common solution is to have the AWS service invoke Lambda function. If that is not supported, we can use SNS as an intermediary.

SNS is for simple event-driven pattern. In a more complicated event-driven architecture, we often need an event bus to route events in certain ways. We also need to support various event sources by different software providers. Moreover, we want the capability to register our own event schema. These are the scenarios where EventBridge can help.

Developer tools

In application development with serverless stack, the line between application and infrastructure is somewhat blurred. Developers often find themselves making repeated configuration on cloud resources while writing application code. For example, to test Python code (application), developer has to upload the code to S3, create Lambda function referencing the code, etc. This would require too much work with AWS CLI, or CloudFormation. The AWS SAM (AWS Serverless Application Model) is a better utility for serverless development workflow.

SAM comes with its own CLI and developers can feed it with template that interacts with a number of serverless resources (e.g. API, SimpleTable, Function, etc). I view SAM as one layer of abstraction on top of CloudFormation that handles some resources in serverless stack. It saves developers from re-writing resources in CloudFormation templates and keeping them consistent, which would have been tedious. Once you deployed your application using SAM, it will appear as a Lambda Application in the console.

SAM templates are also declarative. It is simple but limited in feature. AWS CDK is also a powerful utility that works with general purpose programming language for IaC and interacts with all AWS resources. It is very powerful for building serverless applications.

AWS Amplify

We have CDK, SAM and CloudFormation but we still have to ensure integration between those resources and our applications. For example, when creating a S3 buckets, they have to get the endpoint and reference it in the application code. So is Cognito. As a result, developers still have to spend time on resource integration. They still can’t focus on business logic. We need a tool that can make opinionated configuration of cloud resources and automatically reference them from the application code. This is the purpose of AWS Amplify.

Amplify not only help application developers create backend resources with opinionated configurations. It also provides libraries for the application to use and connect to those resources seamlessly. Developers will need to include Amplify libraries in the application code. To create cloud resources, they can use Amplify CLI or Studio (a web portal from AWS console).

Amplify natively supports a number of serverless resources such as API (using API gateway or AppSync), Storage (S3 and CloudFront), Lambda function, Cognito, etc. Developer may use Amplify CLI command to create such supported resources. Amplify will prompt some guiding questions in order to configure them correctly. In addition to the natively supported resources, developers can also create custom resources. They have to declare those custom resources with CloudFormation or CDK. I came across this project as a good illustration of how Amplify works.

The Amplify Studio can save developers from using Amplify CLI commands. It also can integrate with Figma, providing developers with a framework for frontend development. However, in my experience, it is still glitchy. For now I stick to Amplify CLI.

Amplify also supports hosting capability, providing users with opinionated and customizable CI/CD pipeline configuration.

How about “clientless”

Lambda function can also run client-side logics. System administrators have to create lots of client-side scripting. While Lambda functions can encapsulate those logics, we’d still need a script orchestrator to invoke those functions. AWS Step Function comes to rescue. It was even regarded as the most under-utilized service. As the author states, a state machine (as design pattern) is simply a flow of actions with decision making logics. AWS Step function helps you groom the logic flow of existing actions with Amazon States Language. There are standard and express workflows. Each step is a state. A state can be of several different types, such as Choice, Task, Succeed, Fail, End, Map, Wait, Parallel. The task can be a Lambda function, and even AWS API calls.

AWS Step function integrate with Lambda functions. It is typically used for patterns with long process and the need to orchestrate the execution of several Lambda functions. For specific use cases, look at these sample projects. For example, a network professional needs to run a lot of connectivity testing, reusing the same Python script but run it from different subnets. We need to create a Lambda function for commands like “nc -vz” then invoke the lambda function from VPC, multiple times from different VPCs. We should use step function to drive this. It works like a Makefile on Linux, without requiring your own computer to run.

AWS Step function also integrates with other AWS services, such as SNS, SQS, Dynamo, Batch, Glue, EMR, EKS, API gateway, event bridge.

Whenever we need to create a script, involving custom actions, or AWS API calls, we should consider using AWS step function to organize the actions. The benefits are: it saves you a laptop or bastion host (“client-less”), many ways to invoke them (not just cron”).

Summary

In this post, we started with some serverless services in AWS. For server-side application, we can use API Gateway to invoke Lambda function. For client-side, we can use step function to invoke Lambda function. We then discussed some tools to speed up application development in serverless pattern.

AWS categorizes both Amplify and AppSync under Front-end Mobile. Amplify does not represent any computing resources in AWS cloud. It is a library and CLI tools to enhance developer experience. AppSync on the other hand is a cloud computing resource acting as GraphQL API for mobile or web application.

Build and Manage Kubernetes Clusters

Fri, 23 Sep 2022 11:50:00 -0400

There are numerous options to build a Kubernetes cluster. If your company has a multi-cloud strategy, most likely you will have to deal with cluster creation on multiple cloud platform or on virtual machines on premise.

Most likely, the chosen cloud platform already make it simple for us. However, it is still important to understand what it really takes to build a Kubernetes cluster. In general, we need to figure out these tasks:

Decide where to host the computing infrastructure (i.e. Node) : on premise or public cloud;
Choose a Kubernetes release: either the vanilla release or one of the third-party distributions;
Install Kubernetes to the computing environment, and integrate it with the cloud platform;
Determine required add-ons (e.g. Istio or Linkerd for Service Mesh, dashboard utility, etc);
Deploy application workload to Kubernetes platform;

A public cloud platform provider usually can assist you with task 1 through 3, and partially 4, depending on the provider. If your Kubernetes resides on private cloud or on-prem environment, you can use a Platform solution such as VMware Tanzu or Openshift, which usually covers task 1, 3 and 4. There is no standard about what task these platform solution must address. Therefore it is important to have this list of tasks in mind in order to make a good comparison. I will discuss each of the tasks in this post.

Hosting environment

Nodes are the building blocks of a Kubernetes cluster. We need master nodes as well as worker nodes. In addition, a working cluster also requires storage, and networking infrastructure.

Public cloud platforms typically provides control plane as a service, obviating administrator’s effort to provision master nodes. For example, the control plane of Azure AKS has two levels of uptime commitment: a free tier of 99.5% SLO and a paid tier with an SLA of 99.95% (using AZs) and 99.9% (without using AZs). This uptime commitment applies to control plane only and do not apply to worker nodes. The management of etcd store is also a responsibility of the cloud provider, which frees up the cluster administrator from managing etcd store. However, they cannot access etcd store either. This is not very convenient because as the size of the cluster grows it is a common requirement to connect to etcd store for troubleshooting purpose.

The deployment APIs for public cloud allow the cluster administrator to define the instance size, count and availability zone for the worker nodes. They also automatically register the worker nodes to control plane so that the cluster administrators do not have to do so by themselves. As to storage, the public cloud usually provide some default storage classes based on their storage as service. For networking device, the cluster provision process automatically configures the cloud API so the cluster can manage cloud resources such as network load balancer.

With private cloud or data centre, we usually use virtual machines, or bare-metal servers. Cluster administrators will need to make their own control plane with master nodes. and install worker nodes and register them to the master nodes. The Kubernetes Installation section below will discuss this.

Kubernetes release

If you have to install Kubernetes, you have to think about the Kubernetes release being used. You can use the binary from official Github repository. For example, the release note of version 1.24.3 points to the change log file for download links to server binaries, node binaries. This is the vanilla Kubernetes release.

Apart from the vanilla release, many developers build their own distributions, based off forks of the Kubernetes project. CNCF has a page to keep track of certified Kubernetes distributions. Some of the distributions are open source and can be used for on-prem infrastructure. Here is a list of top players:

Distribution Name	Repo	Description
EKS Distro	Link	Used in EKS managed service or EKS Anywhere for on-prem infrastructure
AKS Engine	Link	Used in Azure Stack for on-prem infrastructure.
Google Kubernetes Engine	N/A	Used in GKE managed service only.
OpenShift Kubernetes Engine	Link	Community distribution (OKD, or OpenShift Kubernetes Distribution) is the open-source upstream.
Rancher Kubernetes Engine (RKE)	Link	still using Docker as container runtime. Supported CNI include: Canal, Flannel, Calico and Weave
K3s	Link	Lightweight distro without small resource requirement. Great for Edge, IoT, ARM etc
RKE2	Link	Originally named RKE government. Supports deployment via Cluster API. Supports containerd as container runtime. Supported CNI include: Cillium, Calico, Canal and Multus. Lightweight
VMware Tanzu	Link	VMWare Tanzu Grid and VMWare Tanzu Community

Above is just a very incomplete list of Kubernetes distributions. There are many more distributions that are not on this list, such as CoreOS Tectonic, Docker Kubernetes, Heptio, Mesosphere, Mirantis, Platform9, Stackube, Telekube. For full details of how each distribution is different, you will need to go over their documents.

With the selected distribution, we still need to deploy the binaries to the nodes. We can do this with a cluster management platform, or standalone installers. Cluster management platform can also help us with baseline configuration (e.g. IAM integration, CNI plugin), in addition to the binary installation.

Cluster Management Platform

These platforms are also sometimes referred to as container management platform.

For example, OpenShift container platform is a self-managed platform based on OpenShift Kubernetes Engine and can run on a variety of hosting environment, public cloud, or private cloud. The installation steps varies depending on the hosting environment. When running on public cloud such as AWS (aka ROSA), the public cloud only provides computing nodes and associated infrastructure. Many corporate with multi-cluster strategy use this option on public cloud to keep their Kubernetes cluster fleet consistent across cloud vendors.

The Openshift container platform also packages some useful open-source add-ons with corporate support, for example:

Red Hat’s strategy is to pick the most renowned open-source project in each domain and add enterprise support to it. However, for management portal, Red Hat developed its own Advanced Cluster Management tool for Kubernetes, and open-sourced it in 2020 in the upstream project Open Cluster Management.

Similar to OpenShift, VMware Tanzu also attempts to cover the domains, with a smaller product portfolio:

Service Mesh: compatible with Istio
Mission Control: management portal
Observability

Google Anthos is also a container platform. Their product line include, but not limited to:

Anthos Config Management (ACM)
Anthos Service Mesh (ASM, an Istio distribution)

SUSE, the developer of RKE, RKE2, and K3s) offers Rancher as multi-cluster management platform. Apart from the engines, SUSE also offers Lonhorn as a storage solution. However, they do not have offerings for service mesh or GitOps. So there is no doubt that Red Hat OpenShift has the most complete portfolio for Kubernetes.

There are also companies that only offers management platforms without their own Kubernetes distribution. For example:

Product capabilities in this category vary a lot and you should refer to their specific documentation to understand. You will probably see a stack chart from each of the platform provider (e.g. SUSE Enterprise Container, OpenShift, Tanzu, Anthos, Rafay) with all technology integrations.

Cluster Installation Tools

As we saw in the installation steps for OpenShift, they are highly dependent on platform. With public cloud, the provisioning process also applies only to a specific platform. Since Kubernetes Installation process is tedious, some tools emerged to help, for example: kubespray, kubeadm, kops and Cluster API. These are governed by SIG cluster lifecycle special interest group.

Here are some traditional options to install a Kubernetes clusters:

kube-up: the first tool to build cluster from 2015. It has been deprecated.
Kubeadm: a tool built to provide best-practice “fast paths” for creating Kubernetes clusters that are minimum viable, and secure. Kubeadm’s scope is limited to the local node filesystem and the Kubernetes API, and it is intended to be a composable building block of higher level tools. It is first released in Sep 2016. The high level configuration steps goes through initialization (kubeadm init), control plane (kubeadm join control plane), and node (kubeadm join node). Kubeadm does not integrate with cloud providers and it does not install addons (auth, monitoring, CNI, storage class)
Kubespray: runs on bare metal or VMs using Ansible for provisioning and orchestration. The first release was in Oct 2015. Since v2.3 (Oct 2017) kubespray started to use kubeadm internally. In addition to kubeadm, kubespray configures CNI, storage class, other CRI. It supports cloud providers and air-gap environment. However it does not support infrastructure management.

The options above are official options. You may use kubeadm and kubespray to quickly (i.e. in an hour) spin up clusters for education purposes. However, with their limitations, it typically requires a lot of efforts to build a production-grade cluster with the needed addons and integrations.

Apart from the official options, there are also unofficial tools such as kubicorn, which was first introduced in 2018 as a cluster management framework with modular support for cloud providers. However it appears to be short-lived.

In the next two sections, we introduce kops and cluster API, two most recent projects to install cluster.

Kops

The kops utility directly perform the provisioning and orchestration via API to the cloud deployment engine. Kops, with first release in Oct 2016, is tightly integrated with the unique features of the cloud providers (e.g. AWS: ASG, ELB, EBS, KMS, S3, IAM). However, kops is only CLI without controller-style reconciliation. It does not support baremetal or vsphere. It also bundles addons with fixed version.

When picking a tool to install cluster, we need to strike a balance between how much simplification the tool brings, and how many different platform the installer can work with. Kops appears to be such a good compromise. It works with a number of cloud platforms using different set of APIs, although most are in alpha and beta stages today. Here is how to install cluster on AWS.

Both kops and Cluster API have good momentum but they work differently. Cluster API was first released in Mar 2019, and is currently less mature than kops. However, it is declarative and may reflect the direction of where cluster lifecycle management is heading.

Cluster API

Cluster API focuses on following areas:

Manage cluster lifecycle declaratively
Infrastructure abstraction (e.g. computing, storage, networking, security, etc)
Utilizing existing tools (e.g. kubeadm, cloud-init)
Modular and pluggable: to be adaptable to different infrastructure providers.

It involves a number of CRs as illustrated in its diagram. We should be clear on the providers for Bootstrap, Infrastructure and Control Plane.

The biggest benefit is the controller pattern to manage the entire lifecycle of a cluster. This allows managing clusters with GitOps, and rolling upgrade of the cluster. It also allows for declarative node scaling, self healing and multi-cluster management.

The client utility for is clusterctl, and with that along with the manifest, we can create a cluster in a few commands. A lot of workflows are still in development but we can take a look at its quick start guide to get a taste of how it works. The installation steps vary a lot based on the environment and the cluster. Also it introduces the separation of management cluster and workload cluster.

Workload cluster is the target cluster being created, as per the manifests.
Management cluster is where you keep track of the workload cluster being managed. You can manage multiple workload clusters from a single management cluster. Note that this management cluster will store credentials about workload clusters, and may become a single point of failure.

Although Cluster API reflects a great initiative to standardize the provisioning of Kubernetes cluster, whether it will succeed has to do with the level of complexity. In the next section, we will get a taste of how it looks to deploy a Kubernetes cluster in a lab.

Management cluster vs workload cluster

In the lab, I use my MacBook to create a management cluster with KinD. Then we configure a workload cluster in AWS from the management cluster.

Cluster API Lab

Note that the steps here are based on the quick start guide on Cluster API document. Also, there is a bug with the AWS provider so the end of the lab will report a warning. The main purpose of this lab is to demonstrate how Cluster API is supposed to work, even though it still has yet to mature.

To start, I install clusterctl (the cluster API client utility), clusterawsadm (the utility specific for AWS) on MacBook, then start a simple KinD cluster.

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.2.0/clusterctl-darwin-amd64 -o clusterctl
chmod +x ./clusterctl
sudo mv ./clusterctl /usr/local/bin/clusterctl
clusterctl version

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/download/v1.4.1/clusterawsadm-darwin-amd64 -o clusterawsadm
chmod +x clusterawsadm
sudo mv clusterawsadm /usr/local/bin
clusterawsadm version

kind create cluster

So far, I installed the required utility and a KinD cluster on MacBook. Then I use clusterawsadm to create InstanceProfile, ManagedPolicy and IAM Roles required for cluster creation. The AWS region and access are configured as environment variables:

export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID=AKIAXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=J8ByduiofpwuisDjDoijOISDs
clusterawsadm bootstrap iam create-cloudformation-stack

This runs a CloudFormation stack to create the permission related resources:

Then I initialize the management cluster with the clusterctl utility, specifying AWS as a provider. I also need to assign the environment variable AWS_B64ENCODED_CREDENTIALS with proper value:

export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)
clusterctl init --infrastructure aws

Now I use clusterctl to generate the manifest for the workload cluster. In environment variables, I specify cluster and node sizes, SSH key name, control plane machine type and node machine type:

export AWS_SSH_KEY_NAME=cskey
export AWS_CONTROL_PLANE_MACHINE_TYPE=t3.large
export AWS_NODE_MACHINE_TYPE=t3.large
clusterctl generate cluster myekscluster --kubernetes-version 1.24.3 --control-plane-machine-count=3 --worker-machine-count=3 > capi-quickstart.yaml

kubectl apply -f capi-quickstart.yaml

At the end I tell the management cluster to create a workload cluster as per the manifest, by simply declaring the CRs. It will take some time for the cluster to create, and there are a number of ways to monitor the progress. You can monitor the log on the controller pods in their respect namespaces. You can also check the cluster status with:

kubectl get kubeadmcontrolplane
clusterctl describe cluster myekscluster

Currently there is a bug and the commands at the end will report as below:

Hopefully the bug will be fixed shortly. To delete the cluster, simply delete the resources in the manifest with kubectl delete -f capi-quickstart.yaml

Summary

There are numerous ways to build a Kubernetes cluster. Before deciding on the approach, I recommend having a full understanding of the hosting environment. This is because installation approach and hosting environment are still tightly coupled. This is the status quo and is not going to change in the near future. Both kops and cluster API reflects initiative to decouple the two but both are still in early stage and already facing growing complexity. Cluster API manages complexity with CRDs to abstract system resources and infrastructure, as illustrated here:

CRDs and providers to abstract system resources and infrastructure

The diagram is from the “Cluster API and declarative Kubernetes Management” white paper. Here is a stream with more about the same topic.

Infrastructure deployment in Terraform 1/2

Wed, 11 Aug 2021 21:44:00 -0400

Terraform is an excellent Infrastructure-as-Code (IaC) tool based on Hashicorp Configuration Language (HCL). Compared to JSON or YAML based declarative templates (e.g. CloudFormation and ARM), HCL is more concise, thanks to the flexibility of HCL. On the other hand, HCL is not as flexible as general purpose languages. For that sake, I see HCL as semi-declarative IaC. This post is my notes about best practices with Terraform development, from the context of AWS, but also applies to other cloud platforms.

Complex Types

There are three primitive types (string, number and bool) that forms collection types and structural types. Here are some common ones:

list: element may repeat, and order is maintained:

["orange", "banana", "orange", "apple"]

set: elements are unique and unordered

["apple", "banana", "orange"]

tuple: each element has its own type

["a", 15, true]

object: defined by a schema with named attributes each with its own type

{
 name = "John"
 age = 52
}

list of object

[
 {
 alloc_id = "0b7271a3219bc1fc2"
 subnet_id = "0c02af76c2c3e46fa"
 },
 {
 alloc_id = "0440c334c48d4247f"
 subnet_id = "02652e69fa2a71de8"
 }
]

map of string

{
 property = "foo"
 attribute = "bar"
}

map of object

{
 objkey1 = {
 alloc_id = "0b7271a3219bc1fc2"
 subnet_id = "0c02af76c2c3e46fa"
 }
 objkey2 = {
 alloc_id = "0440c334c48d4247f"
 subnet_id = "02652e69fa2a71de8"
 }
}

Whenever applicable, Terraform converts types implicitly or explicitly. For example, when a list or tuple is converted to set, all elements are converted to string and duplicates are removed. Object and map are very similar. Map of string can be converted to object if the attributes comply with the schema. Additional attributes not in the schema are discarded.

HCL Types is similar to Python

Although being totally different beasts, the complex types between HCL and Python are similar, to the point I suspect the HCL design is influenced by Python. I summarize the similarities as such:

Python Types	list []	tuple ()	set {}	dict {}
Terraform Types	list []	tuple []	set []	map {}
ordered	Y You can access item by index	Y	N you cannot access an item by index or key; however you can loop over all itmes	N key-value pair that allows you to access item by key
changeable (mutable)	Y	N You cannot update, add or remove items	Y Add or remove only. no change to existing elements	Keys must remain unique or the values get overwritten; Values are mutable
allow duplicate elements	Y	Y	N	Keys must be unique; values don’t have to

In Python, list and tuple allow elements of mixed types but in IaC like Terraform we mostly don’t need mixed types. In Terraform, an object is a map without a defined type. In most situations, lists and tuples behave identically, as do maps and objects.

Another area of similarity is with the comprehension of list and dict/maps. In Python for example,

## Supposed you need to create a list:
lst=[]
for x in range(10):
 lst.append(x**2)

print(lst)

### that can be simplified as the following to create the list:
lst = [x**2 for x in range(10)]

## You can even add contidion
even_numbers=[num for num in range(10) if num%2==0]

## You can introduce function calls:
words = ["hello", "world", "python", "list", "comprehension"]
lengths = [len(word) for word in words]

## You can even combine two lists
lst1=[1,2,3,4]
lst2=['a','b','c','d']

pair=[[i,j] for i in lst1 for j in lst2]

print(pair)

## With dict, it's similar
evens={x:x**2 for x in range(10) if x%2==0}
print(evens)

In Terraform, we use similar techniques:

[for s in var.list : upper(s)] # build a tuple/list from a list

[for k, v in var.map : length(k) + length(v)] # build a list from a map

{for s in var.list : s => upper(s)} # build a map from a list

[for s in var.list : upper(s) if s != ""] # build a tuple/list from a list with condition

Note that the documentation of Terraform doesn’t explicitly call them out as comprehensions. However, it’s exactly the same idea as comprehensions in Python. Even the range() function exists both in Python and Terraform.

Modularization

Modules allows you to group related resources together. They can also be re-used and called by other modules. It is fairly straightforward to create a module:

put the resource declarations into a sub-directory
define input and output in the directory

However, the introduction of module complicates the directory structure and variable referencing, which is important to take into account before starting creating modules. This guideline has further discussion about when to create a module. I re-wrote the terraform templates in Orthweb project to leverage modularization wherever possible, but there is still some stand-alone resource (e.g. random_id) not belonging to any module.

To reference resources across modules, you need to import those resources (using data source) from within the module. There are a couple of ways. You may pass the argument of data source as input variable, or you can leverage the filter capability of data source. Let’s look at one example of each mechanism.

In the example below, we import a subnet by subnet id:

data "aws_subnet" "private_subnet" {
 id = var.private_subnet_id
}

In the example below, we import a subnet by filtering from all subnets in the VPC by tag:

data "aws_subnet" "private_subnet" {
 vpc_id = var.vpc_id
 filter {
 name = "tag:Name"
 values = ["Private"]
 }
}

There are pros and cons of each approach. A module with mechanism 1 is more transferrable across different environment, because the ID of subnet is explicitly provided. However, authors needs to manage those explicit variables with code. Mechanism 2 fetches target resources with filter. It depends on a well-implemented tagging policy in the resource farm.

The Terraform Registry (since 2017) contains a lot of pre-built modules for each backend platform (e.g. AWS). If you find any module that can be used in your project, the module repo can be referenced directly by Git repository URL. You should be aware of the risk of this practice though. Many platforms are keen to publish modules for their platform. Anyone can publish their own modules to the community as well.

You can quickly generate module documentation with terraform-docs.

Local Execution

Local execution is the basic workflow mode which is mostly seen with very small collaboration team. In this mode, the developer executes terraform binary (Terraform CLI) from their workstation (e.g. Laptop). The Terraform CLI converts code into API calls to interface cloud provider. The most frequently used commands (from terraform directory) are:

terraform init
terraform plan
terraform apply

The init command initializes the working directory. The plan command figures out the delta between code and infrastructure. It outlines the changes it is about to make. The apply command commits the change. The documentation of Terraform CLI commands is here.

Terraform keeps track of the infrastructure it manages in state file. This article explains the purpose of state. State management collaboration difficult with local execution because the state file by default is created in the working directory on user’s workstation. Although the state file can be configured to be stored in a shared location such as S3, it still requires a mechanism to lock the state in a multi-developer collaboration.

In large operations, the same code base in Terraform, is usually used to created several different sets of infrastructures, for example, in different geographic regions. So it is a 1-to-many relationship between the code repo and the infrastructure state. To further complicate things, each state might have been deployed using different revisions of the code. To overcome that challenge, Terraform introduced the concept of workspace, which is essentially an instance of state describing a particular group of infrastructure being managed by the same source code. When there are many workspaces, it becomes tricky to manage them with CLI commands.

State management is a major challenge that needs to be solved for team collaboration in local execution workflow. Each state must use the same revision of Terraform code. You can use Git in combination as a workaround to that limitation but the point is you cannot tie a workspace to a commit with the workspace commands.

In some enterprise environment, the execution is from a VM (e.g. ADO agent on-premise) without Internet access, which poses another challenge. First, we need to pre-load required providers manually. The enterprise needs a proxy solution to safely download packages from Hashicorp website. One good option is Nexus Repository, with both open-source and pro supports. It is a full-function artifactory repo that can host helm repo, apt repo, yum repo, etc. Second, we also need to configure Terraform so it picks up providers locally. Managing plugins without Internet access requires understanding of the order in which Terraform tries to load plugins during initialization.

Remote Execution

In remote execution, the code is executed in Terraform Enterprise or Terraform Cloud. Both are remote web servers. The difference is that Terraform Enterprise is self-hosted service, requiring IT specialist to install and maintain Terraform Enterprise. Terraform Cloud on the other hand, is a managed SaaS service. The pricing model includes a free plan for small number of users.

Terraform workspace configuration

In Terraform Enterprise or Cloud, the remote execution is organized in workspaces. You need to create an organization, and then create workspace under the organization in order to execute code. With each workflow, you can specify version control system (VCS) and subdirectory, to tell the workspace where to fetch Terraform code from. The workspace also allows you to define secrets and variables specific to the workspace. When you execute a workspace plan, the secrets and variables are passed from workspace to the execution logic.

Terraform Workspace Variable configuration

You will also need to design the Terraform code in a way to work seamlessly with the secrets and variables loaded from the workspace. The variable declaration in code should match the definition in workspace. There are already a number of variables that came in handy. Check out this guide.

Each execution is referred to as a “run”, with its own run id. A workspace involves may runs, which may succeed or fail. Each run pulls a specific commit of the source repository, and goes through stages such as plan, and apply. The UI from each run result list out the status of each result, in a very easy to read format.

The state data is persisted in the web server as they were generated. Therefore the collaborator do not need to worry about managing state with CLI tools. If there are files that you do not want picked up by the execution engine, their locations can be added to a file .terraformignore. Refer to this guide.

AWS profile

Local execution still has a lot of use cases in enterprises such as testing with temporary resources. A common challenge is authentication. As discussed, Terraform CLI picks up identity information from AWS CLI and authenticates its way into the backend to run API calls against. So AWS CLI must be configured correctly with the sufficient permission to provision resources. On the other hand, enterprises usually offload IAM to an identity store, such as AzureAD, Okta, etc. Putting those together, the pattern of authentication and authorization usually looks like this:

User logs on via SSO (e.g. SAML). The validation response gives a name of an IAM role.
Upon successful authentication, user takes the IAM role. The role does not have any capability, except for assuming a second IAM role.
The second IAM role (the functional role) grants user the permission to do its business.

The steps above, can be carried out in AWS console, or with AWS cli using assume-role command. However, when we put Terraform in the picture, it becomes a little involving because the credential information is updated whenever the functional role is assumed, and the assume-role command takes a pretty long argument.

To skip typing the long command every time, there are some handy tools, such as aws-azure-login. An even better tool that works with a variety of identity stores is saml2aws. The tool allows you to configure identity backend, assume the functional role, and update credential information in aws credential file, all with a single command. The AWS CLI configuration reads:

[default]
region = us-east-1
output = json
cli_history = enabled
cli_pager =
role_session_name = functional_operation

[profile function_user]
source_profile = default
role_session_name = functional_operation
role_arn = arn:aws:iam::9998887766:role/admin-access
region = us-east-1

In Terraform provider, we need to tell it to assume that role as well:

provider "aws" {
 region = "us-east-1"
 assume_role {
 role_arn = "arn:aws:iam::9998887766:role/admin-access"
 session_name = "terraform"
 }
}

This will ensure Terraform assumes appropriate role before doing its job.

AWS EC2 SSH Key Pair

RSA key authentication for SSH should be used for Linux Instances. When creating an EC2 instance, we give it our public key so we can then later authenticate through SSH. If the key is already stored in AWS, we just need to tell EC2 the name of the key, in the key_name property. If the code is likely to be executed from several different places by different users, then we can write the code so it picks up public key from user’s workstation (~/.ssh/id_rsa.pub). Here is an example:

variable "local_pubkey_file" {
 type = string
 default = "~/.ssh/id_rsa.pub"
}
data "local_file" "pubkey" {
 filename = pathexpand(var.local_pubkey_file)
}
resource "aws_key_pair" "user-pubkey" {
 key_name = "runner-pubkey"
 public_key = data.local_file.pubkey.content
}
resource "aws_instance" "bastion" {
 instance_type = "t2.micro"
 key_name = aws_key_pair.user-pubkey.key_name
 ......
}

For remote execution, we can even add an option to pass public key in as variable, to override the key file variable. For an example, check out my orthweb project.

To upload files to EC2 instance from Terraform execution environment, we can use the file provisioner with ssh as connection type.

AWS CDK example in Python – provision Kubernetes Nodes

Thu, 03 Dec 2020 21:14:00 -0400

There are two mechanisms to initialize instances in AWS. Cloud init and CloudFormation Init. Both are widely used and we discuss each of them in this posting. Then we will give an example of using AWS CDK in Python.

Cloud-Init

Cloud-Init is a service originally built for Ubuntu, as a bootstrapping utility to customize a Linux VM as it boots for the first time. It has evolved to be an industry standard multi-distribution method for cross-platform (public or private) cloud instance initialization, or even bare-metal installation. In cloud-init you can install packages and write files, or configure users and security. Because cloud-init is called during the initial boot process, there are no additional steps or required agents to apply your configuration.

Cloud-Init uses UserData, which is part of instance metadata. With AWS, you can pass two types of user data to Amazon EC2: shell scripts and cloud-init directives.

The cloud-config files are text files encoded in base64, with more details covered in the documentation here. cloud-init also works across distributions. For example, you don’t use apt-get install or yum install to install a package. Instead you can define a list of packages to install. cloud-init automatically uses the native package management tool for the distro you select.

CloudFormation Init

The cloudformation init mechanism does not only initialize instance, it also provides a mechanism for the resource being created to communicate with other resources. It allows an instance to emit signal to a different resource (via cfn-signal). It can also monitor changes to external resource and invoke local action (using cfn-hup with hooks). CloudFormation Init requires several components to work together:

The cloudformation resource should have metadata. The metadata must have a key AWS::CloudFormation::Init in which configsets are declared.
The UserData must use helper script (cfn-init) to invoke configuration jobs
The UserData can use helper script (cfn-signal) to signal with a CreationPolicy or WaitCondition (of the same or different resource), so you can synchronize other resources in the stack when the prerequisite resource or application is ready.
The cfn-hup service on the instance can be configured, to check for updates to metadata and execute custom hooks when changes are detected.

Comparison

While there are overlaps between the functionalities of Cloud Init and CloudFormation Init, the major difference is the latter support extended features (signal, update, etc); whereas the former is vendor neutral. The table below summarized some the differences:

	Cloud Init	CloudFormation Init
Works on	Linux OS distribution	CloudFormation resource, in combination with cfn helper scripts
Usecase	Initialization only	Both initialization and resource update
Trigger	cloud-init systemd service	Initial: from UserData Update: by cfn hook
Adoption	Multiple cloud vendors and bare-metal system	AWS cloud instances
Action Playbook	/var/lib/cloud/ Instance Metadata -> User Data, encoded in base 64	CloudFormation Resource -> Metadata section -> AWS::CloudFormation::Init -> configSets and configs
Log file and stdout	/var/log/cloud-init.log /var/log/cloud-init-output.log	/var/log/cfn-init.log /var/log/cfn-init-cmd.log

Comparison between cloud-init and cfn-init

AWS Cloud Development Toolkit (CDK)

Traditionally, AWS CloudFormation uses template in YAML or JSON for resource declaration. As the size of system grows, the amount of resource involved grows quickly and the size of such declaration file may grow beyond manageable. Nested stacks and export of output are mechanisms designed to combat the template sprawling, but to a very limited extent. Two reasons it is hard to control template size are:

In declarative statements, each line carries very small piece of information. Without flow controls such as if-else, loops, object oriented structure, the level of code reusability is very low;
Some auxiliary resources (such as AWS::EC2::VPCGatewayAttachment) must be declared explicitly, even though they are insignificant to the stack functionality

To address these challenges, AWS introduced AWS CDK (cloud development tookkit), which supports multiple languages (JavaScript, TypeScript, Python, Java, and C#). The CDK was natively developed in TypeScript, which is supposed to be the preferred development language. A tutorial is provided here, with detailed API documentation here.

To install aws cdk and create a hello world project, follow this example.

An Example in Python

I have created an example for AWS CDK in Python. The purpose is to create some EC2 instance to complete a lab for Kubernetes (without using managed EKS service). The example provisions the followings:

VPC, a public and private subnets, Internet and NAT gateways;
Relevant security groups and permissions
Bastion host, public instances in public subnet
Private instances in private subnet, with public route through NAT gateway

The private instances forms a cluster for Kubernetes lab. We will use kubespray to initialize these instances. During the bootstraping, we download kubespray, install ansible, etc.

Here is the code repo for this example. With CloudFormation only, the single template could go well beyond 1000 lines. With CDK, the code are organized into several different python files, each representing a stack. The stacks can be stood up with command:

cdk deploy vpc-stack
cdk deploy security-stack
cdk deploy bastion-stack
cdk deploy private-stack

Although the documentation in Python is available, there are generally not a lot of examples built out on the Internet. The pypi site provides some Python specific examples for each module (e.g. core and aws-ec2). Given these libraries are available for only 2 years (since 2018), many advocates TypeScript as the language. However, I have implemented some CloudFormation init, used helper script, and UserData in this example, without running into any language specific issues.It should be noted that the EC2 instance by default will call cfn-init. So there is no need to explicitly run cfn-signal or cfn-init from user data in python code (example). This can be verified in file /var/lib/cloud/instances/<instance-id>/user-data.txt which automatically includes the following lines:

# fingerprint: e1b32ead13878deb
(
 set +e
 /opt/aws/bin/cfn-init -v --region us-east-1 --stack bastion-stack --resource bastionhost5F466975da9934ba490de456 -c config_set_1,config_set_2
 /opt/aws/bin/cfn-signal -e $? --region us-east-1 --stack bastion-stack --resource bastionhost5F466975da9934ba490de456
 cat /var/log/cfn-init.log >&2
)

In addition to Python, AWS CDK also supports other languages. In the next post, we will discuss use of CDK in Typescript.