EC2 Instances on Digi Hunch

EKS impression

Fri, 23 Dec 2022 18:18:19 -0400

I’ve worked on a few AKS projects previously. Since I joined AWS I wanted to put aside some time to check out EKS (Elastic Kubernetes Service). Here in this post, I put down my first impression on EKS, and also share my Terraform template in cloudkube project to create an EKS cluster.

Similar to AKS, EKS exposes API endpoint and the control plane components are hidden from AWS users. When creating EKS cluster it does not create the underlying VPC and subnets. Therefore, you have create an existing VPC and at least two subnets ahead of time, and specify them during EKS creation. Bear in mind that there is a list of requirement for the VPC and subnets.

In the cluster, the CNI that EKS officially supports is Amazon VPC CNI plugin. It is available as an add-on. Similar to Azure CNI, each Pod gets its own IP address. In addition, EKS supports other compatible CNI plugins such as Calico, Cilium, Weave Net and Antrea.

Computing Nodes in EKS

There are three modes to address computing capacity: self-managed nodes, EKS managed node groups and AWS Fargate. The documentation has a comparison table.

With self-managed nodes, users create EC2 instances separately and then register them to the control plane. The instances must use the same IAM role and AMI. You can use Auto Scaling groups of Bottlerocket (AWS-sponsored purpose-built Linux distro for container host) nodes. The self-managed node option is mostly for AWS outpost customers who bring in their own computing capacity from data centre.

If you provision computing capacity from AWS, it makes sense to assign EKS managed node groups when creating EKS cluster. We can turn on Cluster Autoscaler, a Kubernetes construct to manage the auto scaling of node groups. Sometimes we want to have more than one node groups. For example, to build a multi-architecture cluster, we need one node group with amd64 nodes and the other with arm64 nodes (e.g. instances with Graviton processor). In general, arm-based CPU delivers better performance with less power consumption and the industry is slowly moving towards more arm-based CPU architecture.

Fargate is what I call managed computing service for EKS. With Fargate you do not need to tweak Cluster Autoscaler to self-manage computing capacity. The Fargate documentation has a long list of considerations. For example, Pods must match a Fargate profile (here‘s an example) at the time that they’re scheduled to run on Fargate. So we need to build Fargate profile and Pod labelling properly. Also, Fargate does not support DaemonSet. Another big consideration is that Fargate does not support non-VPC CNI. In my opinion these are pretty significant limitations. Many workloads (system-level or application-level) would need Daemonset (e.g. kube-proxy, some CNI or CSI drivers, Dynatrace monitoring).

The pro of Fargate is the serverless computing model. The construct of a Fargate profile isn’t complicated. You just specify subnets, namespace and labels. However, the downside is the long list of considerations. Some teams may consider these restrictions too much. The other overhead is the need to manage Fargate profile to ensure all Pods are scheduled somewhere.

To me, using Fargate alone impairs portability of workload. The good thing is that Fargate and Managed Node Group are not mutually exclusive on a cluster. In most cases, we can go partially serverless, and reap the benefits of both of them.

Node AutoScaling

For workloads that don’t have a matching Fargate profile, we have to figure out node autoscaling ourselves. I touched on Cluster Autoscaler in “Autoscaling on Kubernetes Platform“. CA works on AWS as well and is triggered upon a Pod coming to unschedulable status in Scheduler. There is some limitations though. For example, CA interacts with Autoscaling Group (instead of EC2 instances directly). When it determines it’s time to scale up, it bumps up the desired capacity by one at a time in the Autoscaling group. The configurations in Autoscaling group may also be at play and CA do not have direct control. For example, the “scaling cooldown“. The pool of nodes is homogenous as per the pre-configured launch template and CA has no control. If a Pod requires a different type of node (e.g. ARM64 CPU, spot instance, etc), then we’d first have to create a node group with the desired node type. Moreover, in the worst cases, one-at-a-time scale-up does not meet the increase of demand driven by Pod increases, causing nuances such as racing conditions.

Because the Cluster Autoscaler doesn’t really deal with the nodes themselves, this kind of integration is clunky and slow. Nearly half of Kubernetes customers on AWS report that configuring cluster auto scaling using the Kubernetes Cluster Autoscaler is challenging and restrictive, according to this blog post. As a result, AWS launched an open-source cluster autoscaler project, Karpenter. Karpenter first only supported EKS but now the support includes other CSPs. For EKS, Karpenter directly interact with different types of EC2 instances.

Karpenter makes node scaling work in a more cloud-native manner. In the presence of unschedulable Pods, Karpenter bypasses the Kubernetes scheduler and works directly with the Cloud provider, to launch the minimal compute resources needed to fit those Pods and immediately binds the Pods to the newly provisioned Nodes without waiting for scheduler. As Pods are removed or rescheduled to other nodes, Karpenter looks for opportunities to terminate under-utilized nodes. Karpender defines a CR called Provisioner to specify node provisioning configuration, such as instance size, zone, CPU architecture, etc. It is a manifest that describes a node group so the node scaler is aware of all the available node types. You can have multiple Provisioners for different needs, just like node groups. The Provisioner CR can also set TTL for empty Nodes, such that once a Node has no pods other than DaemonSet, Karpenter will terminate the Node on TTL expiry.

Karpenter’s idea is similar to the idea of AutoPilot cluster in GKE. The new EKS workshop has an section on how to set up CA and Karpenter in practice.

Identity Management for EKS

For IAM, we need to be concerned with three aspects. The management traffic to the cloud service, the management traffic for Kubernetes cluster and business traffic.

Traffic type	AWS	Azure
I. Cloud Service Endpoint (Management Traffic for Cloud Service)	AWS IAM identity	Azure RBAC
II. Kubernetes API (Management Traffic for K8s Cluster)	IAM mapping or OIDC	Azure RBAC (implementation of OIDC)
III. Business traffic	Up to Kubernetes Ingress	Up to Kubernetes Ingress

For business traffic (type III), it is all up to the Ingress. I’ve written another article on managing ingress traffic on Kubernetes platforms. We interact with cloud service endpoint (type II) with either AWS CLI or Terraform, to create any object, including resources needed for a cluster. This is generally how we work with cloud service, not specific to Kubernetes. Usually the IAM identity assumes another IAM role, which empowers it with a lot of permissions.

For access to Kubernetes API (type III), EKS supports OIDC and IAM mapping. AWS documentation refers to this as “Cluster Authentication“. There is one special scenario where your identity for type II access inherits your identity for type I access. As the document puts:

When you create an Amazon EKS cluster, the AWS Identity and Access Management (IAM) entity user or role, such as a federated user that creates the cluster, is automatically granted system:masters permissions in the cluster’s role-based access control (RBAC) configuration in the Amazon EKS control plane. This IAM entity doesn’t appear in any visible configuration, so make sure to keep track of which IAM entity originally created the cluster.

This special scenario (I call it the “implicit master user“) allows us to perform critical activities on the cluster, such as creating IAM mapping, or OIDC configuration.

The above addressed how AWS resource access Kubernetes resource. On the other hand, to address how a Kubernetes resource access AWS resources, we need IRSA (IAM Roles for Service Account). We have a service account in Kubernetes and map it to an IAM role.

AppMesh

AppMesh is AWS’ Envoy based service-mesh offering supporting Kubernetes cluster, ECS service and even EC2 instance. AppMesh’s control plane is a managed AWS service, with a controller running on the Kubernetes cluster. To install AppMesh on the cluster:

On the EKS cluster, install AppMesh Controller using Helm
Associate the cluster with IAM OIDC provider
Create an IAM role for the appmesh-controller service account

After these steps, you can create a mesh using CloudFormation, Terraform, etc. The data plane (Envoy proxy) can run on Kubernetes (as sidecar). Traffic between control plane and data plane can go through private link (Interface VPC endpoint) for added security. Like Istio, AppMesh enables mTLS. For observability, you can export Envoy metrics with Prometheus. Coupled with XRay, AppMesh also supports distributed tracing.

AppMesh uses a different set of CRDs than Istio. Key CRDs are:

Mesh: represents an entire service mesh. At mesh level you can configure Egress filter (to allow or deny external traffic) and set IP version (v4 vs v6)
VirtualGateway: a CRD that represents an Ingress in to the Mesh. A virtual gateway allows resources that are outside of your mesh to communicate to resources that are inside of your mesh. A virtual gateway references Envoy proxy deployment by podSelector. It references GatewayRoutes by namespaceSelector, and optionally gatewayRouteSelector. You also specify listeners in the manifest to reference Envoy proxy Service (LoadBalancer Type).
GatewayRoute: A gateway route is attached to a virtual gateway and routes traffic to an existing virtual service. If a route matches a request, it can distribute traffic to a target virtual service. In the manifest, you specify a list of httpRoute, each with matching condition and action. In the action section you can specify virtualService as target.
VirtualService: an abstraction of a real service provided by a virtual node directly or indirectly by means of a virtual router. Dependent services call your virtual service by its virtualServiceName, and those requests are routed to the VirtualNode or VirtualRouter that is specified as the provider for the VirtualService.
VirtualRouter: Virtual routers handle traffic for virtual services. In a virtual router manifest, you can define Route to direct incoming requests to virtual nodes as target.
VirtualNode: A virtual node acts as a logical pointer to a particular task group (i.e. ECS service, Kubernetes deployment). It represent a Service in the AppMesh. In the manifest, you reference Pods by podSelector, specify listeners for any inbound traffic that your virtual node expects, and specify serviceDiscovery for your task group.

You can configure those Custom Resources using YAML manifests (and check the API reference a lot). Alternatively, you can configure them from AWS CLI or AWS console. The console will help you visualize what can be configured. For further details on how these CRs play together, there is a workshop for AppMesh.

EKS cluster using Terraform

Officially, there is an EKS blueprint project for provisioning EKS cluster in Terraform.

I also keep my own Terraform code in the AWS directory of cloudkube project. It works out to be a little more complex than my Terraform template to create Azure Kubernetes Cluster (Azure directory). Because I had to create Cognito resources with initial credential to allow users to connect to cluster without using the implicit master account.

Below is the diagram of the processes.

Create EKS cluster with Terraform module

The template configures kubectl access on a Bastion host, which assumed the same role that our IAM user uses to create the Kubernetes cluster. Therefore, the IAM role is the master identity.

Note that the IAM user (power-user) has very powerful permissions. Usually it is ideal to assign lots of permission to IAM Roles (temporary credential) instead of IAM user (long-term credential). So the role chaining would look like:

The IAM user that Terraform uses has no permission other than assuming a “PowerUser” role
The PowerUser role trusts the IAM user. It also has the permission to assume the “EKS-Manager” role
The EKS-Manager role trusts PowerUser’s role session.

However, the role chaining scenario above is not currently supported in Terraform.

I use a Bastion host because the cluster endpoint is on private subnet. The bastion host is on a public subnet. However, if we do not like public subnet and public IP, we can place the bastion host on a private subnet, and use SSM system manager agent with SSH tunnel plugin to have SSH access to private bastion host.

Summary

I first came across this article about EKS and its awfulness and then decided to check out EKS. I’m not sure all points are still valid but it’s generally real-life experiences. There are also many peripheral services, such as AMP (AWS Managed Prometheus), AMG (AWS Managed Grafana), ADOT (AWS Distro for Open Telemetry), AppMesh (Another Envoy-based Service Mesh, easier to manage than Istio but less Powerful), with a lot to explore.

AWS CDK example in Typescript – provision an AWX server

Sat, 19 Dec 2020 17:14:00 -0400

This post provides an example of using AWS CDK in Typescript.

Ansible Tower and AWX

We have used open-source Ansible extensively in the past. While the automation is convenient, the lack of UI makes it not as suitable as a team collaboration tool. One way to allow team collaboration with open-source Ansible, is to use Jenkins to glue the components together, as discussed in the Automated Deployment Pipeline series. In this setup, the open-source Ansible remains command-line driven, with Jenkins building up the command, rather than a human user. There are many upsides in this configuration, but it is not built specifically for Ansible. Ansible is agent-less, and can be run from any host. This sounds appealing and can work well in smaller server fleet. However, since it requires some configuration on the controlling host for Ansible to function properly, it become unnecessary to configure Ansible environment on every single host (e.g. production). A typically environment only has Ansible environment configured on the bastion host. This brings the need for a dedicated controller server to drive all Ansible tasks.

Ansible Tower is Red Hat’s commercial enhancement to the open source Ansible, providing web-based console, REST API and other services such as Role-based Access Control (RBAC). Managing Ansible via REST API is still somewhat involving but this also enables other open-source contributions to simplify the use of API. For example Tower CLI allows you to use Ansible Tower with simplified command. Ansible Tower has an open-source upstream project called AWX, maintained by Red Hat. AWX is essentially a preview release of Ansible Tower without commercial support. AWX can serve as an engine for all Ansible related task. AWX server is essentially an Ansible control server. AWX, or Ansible Tower, also brings several concepts on top of Ansible:

Job template: defines how an Ansible playbook should be executed, including details such as machine credential, project, inventory, and playbook file.
Job: the actual execution of job template
Project: connects Ansible Tower to source control such as BitBucket. It is tied to a Git repository and a branch within that repository

To deploy AWX on EC2 instances, there is a reference deployment by AWS. However, it is provided as CloudFormation template and appears to be outdated (from 2018). In our project (late 2020, named ansible tower lab, or dubbed as “atlab”), we provide the infrastructure in AWS CDK (written in typescript), to provision the AWX environment.

The goal is that once the configuration is completed, you can run ansible ping against a target EC2 instance. The steps are as automated as possible. However, a number of key steps are purposefully left manual for learning purpose, such as the installation of AWX on EC2 instance.

Infrastructure as Code

In previous posting, I created infrastructure as code in AWS CDK with Python, so I decided to change to typescript in this project, with the assumption it is just a matter of syntax mapping. However, I underestimated the transition to a new language I never learned before. A fuzzy understanding of little details such as when to use let a=4 vs this.a=4, may produce elusive errors that takes hours to troubleshoot. I would therefore strongly recommend reading the basic syntax guide for typescript, before getting started.

In typescript, this example project provides an implementation of configuring autoscaling groups, including cloud init, user data, etc on AWS. Other than the language, everything else is very similar to the project in this post, which was developed in Python. Also, note that the project directory structure varies slightly based on the language being used.

If you are absolutely new to AWS cdk, start with this app. It is beyond the scope of this post, to cover extensively the installation and environment configuration of AWS CDK.

In the provision process for Bastion host, the cloudformation init script pulls a specific version from AWX repository, then makes slight modification. User will need to install it manually. Note that AWX can be installed on three types of platforms:

OpenShift
Kubernetes
Docker Compose

All are documented in their README file. For simplicity in this project, the installation is on standalone docker compose. This is the default mode so there is no need to modify the inventory file.

The code repo

The repository is version controlled here. To run the project, you need to have aws cli environment, then install the required packages including node js, and npm packages such as aws cdk. Once configured, validate with command:

cdk ls

This should display the stacks available. Use cdk deploy to deploy each stack.

When BastionStack is deployed, dependent packages should be installed with user data and cloud init. You will just need to SSH on to the server to manually install AWX, as explained in the instruction, to manually install AWX:

ansible-playbook -i inventory install.yml

Then you can browse to the server (at port 80 by default). Before the log-in page for the first time, the AWX will upgrade itself, with the following screen presented:

Now log on with default credential (in README.md), you will have the UI for AWX:

AWX Web Console

From here, you can edit inventory by adding the host. Or use the helper script (~/awxcompose-helper.sh) from bastion host to create a new inventory (named Private Instance Inventory), and populate it with the hosts in the stack. The helper script does so by querying aws resource, and isssue rest API calls to AWX. After executing the script, you can see a new inventory, and the Private instance inventory should contain all hosts in the stack:

Automatically populated inventory

You can then run Ansible ping against the host to validate connectivity. Note that during inventory creation, the ansible_user is already set to ec2-user (by the helper script):

Ping result

Some technical details

The initialization process on Bastion host creates an RSA key pair, stores the public key to AWS, for the upcoming private instances to uses. It keeps the private key locally in order to make outgoing SSH connection to the private instances. To ensure connectivity between AWX and private instances, there are a couple of (bash) helper scripts involved. Both reflects some technical details that I had to work through.

awxcompose-helper.sh: the initialization process pulls AWX installation file from git repo. The installation process will build a docker-compose file in ~/.awx/awxcompose, based on a template (~/awx-*/installer/roles/local_docker/templates/docker-compose.yml.j2). When user tells AWX to connect to private instance, the connection was made out of a docker container (instead of from the OS of bastion host), we need this script to map SSH key file from host to container, by modifying the template file. Without this helper, outgoing SSH connection will fail with error (Permission denied (publickey,gssapi-keyex,gssapi-with-mic)). This script is invoked in the cloud init process without requiring manual execution.
awxinvt-helper.sh: once the private stack is up and the installation has completed, we need to add the hosts to AWX inventory. This script gets the instance ID and IP addresses of the private instances, and uses Rest API calls to create inventory and populate it with hosts. Ansible has multiple ways of authentication. This script uses the non-stateful basic authentication with each curl command requiring credential. Ansible Rest API guide is provided here and be wary of the convention where URI must end with a slash.

This project is just a start of AWX on AWS CDK project using Typescript. In real life scenarios, there are some work to do to make this even more automated. For example, use cfn-hup service to monitor changes of private stack, and therefore update inventories accordingly.

AWS CDK example in Python – provision Kubernetes Nodes

Thu, 03 Dec 2020 21:14:00 -0400

There are two mechanisms to initialize instances in AWS. Cloud init and CloudFormation Init. Both are widely used and we discuss each of them in this posting. Then we will give an example of using AWS CDK in Python.

Cloud-Init

Cloud-Init is a service originally built for Ubuntu, as a bootstrapping utility to customize a Linux VM as it boots for the first time. It has evolved to be an industry standard multi-distribution method for cross-platform (public or private) cloud instance initialization, or even bare-metal installation. In cloud-init you can install packages and write files, or configure users and security. Because cloud-init is called during the initial boot process, there are no additional steps or required agents to apply your configuration.

Cloud-Init uses UserData, which is part of instance metadata. With AWS, you can pass two types of user data to Amazon EC2: shell scripts and cloud-init directives.

The cloud-config files are text files encoded in base64, with more details covered in the documentation here. cloud-init also works across distributions. For example, you don’t use apt-get install or yum install to install a package. Instead you can define a list of packages to install. cloud-init automatically uses the native package management tool for the distro you select.

CloudFormation Init

The cloudformation init mechanism does not only initialize instance, it also provides a mechanism for the resource being created to communicate with other resources. It allows an instance to emit signal to a different resource (via cfn-signal). It can also monitor changes to external resource and invoke local action (using cfn-hup with hooks). CloudFormation Init requires several components to work together:

The cloudformation resource should have metadata. The metadata must have a key AWS::CloudFormation::Init in which configsets are declared.
The UserData must use helper script (cfn-init) to invoke configuration jobs
The UserData can use helper script (cfn-signal) to signal with a CreationPolicy or WaitCondition (of the same or different resource), so you can synchronize other resources in the stack when the prerequisite resource or application is ready.
The cfn-hup service on the instance can be configured, to check for updates to metadata and execute custom hooks when changes are detected.

Comparison

While there are overlaps between the functionalities of Cloud Init and CloudFormation Init, the major difference is the latter support extended features (signal, update, etc); whereas the former is vendor neutral. The table below summarized some the differences:

	Cloud Init	CloudFormation Init
Works on	Linux OS distribution	CloudFormation resource, in combination with cfn helper scripts
Usecase	Initialization only	Both initialization and resource update
Trigger	cloud-init systemd service	Initial: from UserData Update: by cfn hook
Adoption	Multiple cloud vendors and bare-metal system	AWS cloud instances
Action Playbook	/var/lib/cloud/ Instance Metadata -> User Data, encoded in base 64	CloudFormation Resource -> Metadata section -> AWS::CloudFormation::Init -> configSets and configs
Log file and stdout	/var/log/cloud-init.log /var/log/cloud-init-output.log	/var/log/cfn-init.log /var/log/cfn-init-cmd.log

Comparison between cloud-init and cfn-init

AWS Cloud Development Toolkit (CDK)

Traditionally, AWS CloudFormation uses template in YAML or JSON for resource declaration. As the size of system grows, the amount of resource involved grows quickly and the size of such declaration file may grow beyond manageable. Nested stacks and export of output are mechanisms designed to combat the template sprawling, but to a very limited extent. Two reasons it is hard to control template size are:

In declarative statements, each line carries very small piece of information. Without flow controls such as if-else, loops, object oriented structure, the level of code reusability is very low;
Some auxiliary resources (such as AWS::EC2::VPCGatewayAttachment) must be declared explicitly, even though they are insignificant to the stack functionality

To address these challenges, AWS introduced AWS CDK (cloud development tookkit), which supports multiple languages (JavaScript, TypeScript, Python, Java, and C#). The CDK was natively developed in TypeScript, which is supposed to be the preferred development language. A tutorial is provided here, with detailed API documentation here.

To install aws cdk and create a hello world project, follow this example.

An Example in Python

I have created an example for AWS CDK in Python. The purpose is to create some EC2 instance to complete a lab for Kubernetes (without using managed EKS service). The example provisions the followings:

VPC, a public and private subnets, Internet and NAT gateways;
Relevant security groups and permissions
Bastion host, public instances in public subnet
Private instances in private subnet, with public route through NAT gateway

The private instances forms a cluster for Kubernetes lab. We will use kubespray to initialize these instances. During the bootstraping, we download kubespray, install ansible, etc.

Here is the code repo for this example. With CloudFormation only, the single template could go well beyond 1000 lines. With CDK, the code are organized into several different python files, each representing a stack. The stacks can be stood up with command:

cdk deploy vpc-stack
cdk deploy security-stack
cdk deploy bastion-stack
cdk deploy private-stack

Although the documentation in Python is available, there are generally not a lot of examples built out on the Internet. The pypi site provides some Python specific examples for each module (e.g. core and aws-ec2). Given these libraries are available for only 2 years (since 2018), many advocates TypeScript as the language. However, I have implemented some CloudFormation init, used helper script, and UserData in this example, without running into any language specific issues.It should be noted that the EC2 instance by default will call cfn-init. So there is no need to explicitly run cfn-signal or cfn-init from user data in python code (example). This can be verified in file /var/lib/cloud/instances/<instance-id>/user-data.txt which automatically includes the following lines:

# fingerprint: e1b32ead13878deb
(
 set +e
 /opt/aws/bin/cfn-init -v --region us-east-1 --stack bastion-stack --resource bastionhost5F466975da9934ba490de456 -c config_set_1,config_set_2
 /opt/aws/bin/cfn-signal -e $? --region us-east-1 --stack bastion-stack --resource bastionhost5F466975da9934ba490de456
 cat /var/log/cfn-init.log >&2
)

In addition to Python, AWS CDK also supports other languages. In the next post, we will discuss use of CDK in Typescript.